How Docker works

What it can see runs commands in a container on a small made-up host with a PostgreSQL server, an SSH daemon and a user's shell. Each output is worked out from the namespaces that are switched on, and the clone() call underneath shows the flags runc would pass.

• Run ps with the Docker defaults, then switch the PID namespace off: the host's processes appear, with their real PIDs.
• Switch the network namespace off and run ip: the container now has the host's interfaces, as with --network host.
• Try mount and reboot, then turn on --privileged. With the PID namespace still on, reboot only ends the container; with it off, it would reboot the machine.

What it can use applies cgroup limits. Allocate memory until the OOM killer fires, then set a CPU limit of half a CPU with 4 busy threads and see how long a single 30 ms request takes.

A Linux container is a process, or a tree of processes, that the kernel isolates from the rest of the system without running a separate operating system. There is no single "container" object in the Linux kernel. A container runtime combines several independent features: namespaces to give the process its own view of process IDs, network, files and other resources; control groups (cgroups) to limit and measure the memory, CPU and other resources it uses; a root filesystem built from the image's layers; and reduced privileges through capabilities and a seccomp system-call filter.

Because the processes run directly on the host kernel, a container starts as fast as a process and has almost no overhead. The same fact sets the limit of its isolation: everything is enforced by one shared kernel.

A namespace wraps a global resource so that processes inside see their own instance of it. Linux has eight types; Docker uses seven of them, and the user namespace only when the daemon is configured for it:

The Docker daemon asks containerd to run the container, and containerd starts runc, the reference implementation of the Open Container Initiative runtime specification, with a config.json describing it. runc then:

1. creates the new namespaces (with clone() and unshare());
2. creates a cgroup for the container and writes the limits into files such as memory.max, cpu.max and pids.max, then moves the process into it;
3. sets up the mounts: the overlay filesystem assembled from the image layers, plus /proc, /sys and /dev, and switches root to it with pivot_root();
4. drops capabilities to the default set, applies the seccomp filter, sets the user, and calls execve() on the image's entry point, which becomes PID 1 in the new PID namespace.

The network namespace starts with only a loopback interface. Docker's default bridge network adds one end of a virtual Ethernet (veth) pair to it and attaches the other end to the docker0 bridge on the host, with NAT rules for published ports.

Control groups organize processes in a tree and apply resource controllers to each node. Modern systems use the unified cgroup v2 hierarchy under /sys/fs/cgroup, where a container's limits are plain files.

Memory. memory.max is a hard limit. When the cgroup reaches it the kernel first tries to reclaim memory, such as file cache; if nothing can be reclaimed, the cgroup's OOM killer kills a process in it. With a 256 MB limit and allocations of 64 MB, the program on this page is killed when it asks for 280 MB, and the container exits with status 137, which is 128 plus SIGKILL's number 9. memory.high is a softer limit that throttles and reclaims instead of killing.

CPU. cpu.max holds a quota and a period, by default 100 ms. The cgroup's threads may use at most the quota of CPU time in each period, summed over all CPUs; when it is used up, every thread in the cgroup waits for the next period. Kubernetes turns a CPU limit of 500m into cpu.max = 50000 100000. For a multithreaded program this has a surprising effect on latency:

Linux splits root's powers into 41 capabilities. Docker keeps 14 by default (CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_MKNOD, CAP_NET_RAW, CAP_SETGID, CAP_SETUID, CAP_SETFCAP, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_SYS_CHROOT, CAP_KILL, CAP_AUDIT_WRITE) and drops the rest, including CAP_SYS_ADMIN (mounting, and much else), CAP_SYS_BOOT, CAP_SYS_MODULE and CAP_NET_ADMIN. Its default seccomp profile also blocks around 44 of the 300-plus system calls and allows some, such as mount and reboot, only to processes holding the matching capability.

--privileged gives back every capability, disables seccomp and AppArmor or SELinux confinement and exposes the host's devices; it keeps the namespaces, but a privileged container can usually reach the host. Without a user namespace, which Docker enables only with the daemon's userns-remap setting, UID 0 in a container is UID 0 on the host, so these restrictions are what stand between a root process and the machine. Rootless Docker and Podman run the whole engine inside a user namespace.

A virtual machine runs its own kernel on virtual hardware provided by a hypervisor; a container shares the host's kernel. Containers therefore start in milliseconds and add little overhead, but a kernel vulnerability reachable from inside a container can compromise the host. Where that matters, such as running untrusted code from many customers, there are middle grounds: gVisor intercepts the container's system calls in a user-space kernel, and Firecracker and Kata Containers wrap each container or function in a lightweight virtual machine.

The idea of changing a process's view of the system goes back to chroot in Version 7 Unix (1979), which changes only the root directory. FreeBSD jails (2000) and Solaris Zones (2005) isolated processes more fully. In Linux, mount namespaces arrived in 2002, and the rest followed over the next decade, with user namespaces completed in Linux 3.8 in 2013. Cgroups were developed at Google as "process containers" and merged in Linux 2.6.24 in 2008; cgroup v2 was declared stable in Linux 4.5 in 2016. LXC (2008) combined them into containers, and Docker (2013) made them easy to package and ship with layered images. In 2015 Docker donated its runtime as runc to the new Open Container Initiative, whose image and runtime specifications other tools, including containerd, CRI-O and Podman, also follow.