rkt/Documentation/stage1-implementors-guide.md

Stage1 ACI implementor's guide

Background

Rocket's execution of containers is divided roughly into three separate stages:

0. Stage 0: discovering, fetching, verifying, storing, and compositing of both application (stage 2) and stage 1 images for execution.
1. Stage 1: execution of the stage 1 image from within the composite image prepared by stage 0.
2. Stage 2: execution of individual application images within the containment afforded by stage 1.

This separation of concerns is reflected in the file-system and layout of the composite image prepared by stage 0:

0. Stage 0: rkt executable, and the Container Runtime Manifest created at "/var/lib/rkt/containers/$uuid/container".
1. Stage 1: "stage1.aci", made available at "/var/lib/rkt/containers/$uuid/stage1" by rkt run.
2. Stage 2: "$app.aci", made available at "/var/lib/rkt/containers/$uuid/stage1/rootfs/opt/stage2/$imageid" by rkt run.

The stage 1 implementation is what creates the execution environment for the contained applications. This occurs via entrypoints from stage 0 on behalf of rkt run and rkt enter. These entrypoints are nothing more than executable programs located via Annotations from within the stage 1 ACI manifest, and executed from within the stage 1 of a given container at "/var/lib/rkt/containers/$uuid/stage1/rootfs".

Stage 2 is the destination application images and stage 1 is the vehicle for getting us there from stage 0. For any given container instance, the stage 1 may be completely different, allowing for flexibility in containment strategies employed within the same host while utilizing reusable application ACIs.

Entrypoints

rkt run => "coreos.com/rocket/stage1/run"

0. rkt prepares the container's stage 1 and stage 2 images and Container Runtime Manifest under "/var/lib/rkt/containers/$uuid", acquiring an exclusive advisory lock on the directory.
1. chdirs to "/var/lib/rkt/containers/$uuid"
2. resolves the "coreos.com/rocket/stage1/run" entrypoint via Annotations found within "/var/lib/rkt/containers/$uuid/stage1/manifest"
3. executes the resolved entrypoint relative to "/var/lib/rkt/containers/$uuid/stage1/rootfs"

It is the responsibility of this entrypoint to consume the Container Runtime Manifest and execute the constituent apps in the appropriate environments as specified by the Container Runtime Manifest.

The environment variable "RKT_LOCK_FD" contains the file descriptor number of the open directory handle for "/var/lib/rkt/containers/$uuid". It is necessary that stage 1 leave this file descriptor open and in its locked state for the duration of the rkt run.

In the bundled rocket stage 1 which includes systemd-nspawn and systemd, the entrypoint is a static Go program found at "/init" within the stage 1 ACI rootfs. The majority of its execution entails generating a systemd-nspawn argument list and writing systemd unit files for the constituent apps before executing systemd-nspawn. Systemd-nspawn then boots the stage 1 systemd with the just-written unit files for launching the contained apps. The "/init" program is essentially a Container Runtime Manifest to systemd-nspawn + systemd.service translator.

An alternative stage 1 could forego systemd-nspawn and systemd altogether, or retain these and introduce something like novm or qemu-kvm for greater isolation by first starting a VM. All that is required is an executable at the place indicated by the "coreos.com/rocket/stage1/run" entrypoint which knows how to apply the Container Runtime Manifest and prepared ACI file-systems to good effect.

Arguments

• --debug to activate debugging
• --private-net to trigger the creation of a private network

rkt enter => "coreos.com/rocket/stage1/enter"

0. rkt verifies the container and image to enter are valid and running
1. chdirs to "/var/lib/rkt/containers/$uuid"
2. resolves the "coreos.com/rocket/stage1/enter" entrypoint via Annotations found within "/var/lib/rkt/containers/$uuid/stage1/manifest"
3. executes the resolved entrypoint relative to "/var/lib/rkt/containers/$uuid/stage1/rootfs"

In the bundled rocket stage 1 the entrypoint is a statically-linked C program found at "/enter" within the stage 1 ACI rootfs. This program enters the namespaces of the systemd-nspawn container's PID 1 before executing the "/diagexec" program which then chroots into the specific application's rootfs loading the application's environment variables as well.

An alternative stage 1 would need to do whatever is appropriate for entering the application's environment created by its own "coreos.com/rocket/stage1/run" entrypoint.

Arguments

1. image id of the specific application to enter
2. cmd to execute
3. any cmd arguments

Examples

Stage1 ACI manifest

{
    "acKind": "ImageManifest",
    "acVersion": "0.2.0",
    "name": "foo.com/rocket/stage1",
    "labels": [
        {   
            "name": "version",
            "value": "0.0.1"
        },
        {   
            "name": "arch",
            "value": "amd64"
        },
        {   
            "name": "os",
            "value": "linux"
        }
    ],
    "annotations": [
        {   
            "name": "coreos.com/rocket/stage1/run",
            "value": "/ex/run"
        },
        {   
            "name": "coreos.com/rocket/stage1/enter",
            "value": "/ex/enter"
        }
    ]
}