Migrate running containers by checkpoint-restoring using CRIU
When containers were first introduced as a way to package microservices, they were designed to be entirely stateless and ephemeral. Typically, stateless applications are microservices or containerised applications that aren’t required to store data. In order to make the containers stateless, state is usually maintained in the external persistent storage like a database.
However, there are stateful applications — including use cases in database, analytics, machine learning (ML) and deep learning (DL) applications that store/process data; state is essential for these kind of tasks.
Why we need container migration?
Container management solution Kubernetes employs preemption which removes existing Pods from a cluster under resource pressure to make room for higher priority pending Pods. Lower priority tasks often experience frequent preemptions. Such tasks can have a state in memory and run for hours doing a lot of computation. Cost of losing the state will be significant, if these tasks are restarted and all those computations have to be done again.
Being able to move those stateful containers to new machines for higher resources or load balancing is called stateful migration. The basic steps to migrate running containers from one node to another are to checkpoint the container on the source node, transfer the checkpoint image to the destination node, and restore the container on the destination node. This way, the container is migrated without losing its state.
Use cases for checkpoint and restore containers
Following are the possible use cases to checkpoint and restore containers
Slow-boot services speed up
If some service starts up too long (it can perform complex state initialization for example) we can checkpoint it after it finishes starting up and on the 2nd and subsequent starts restore it from the image.
Reboot without losing state
Sometimes, it is necessary to reboot a node for important security updates. With the help of checkpoint and restore, a slow starting container can be checkpointed before the reboot. Then, after the reboot, the container can be restored from the checkpoint without losing any state and without long service downtimes.
Preemption/Eviction
Similar to the first use case, Checkpointing a container on one node and restoring it on another node for higher resources or load balancing etc.
Snapshots of apps
Save a series of app’s states (all but first incremental) and revert later to any of them.
One of examples when this snapshot might be useful is debugging. One might need to bring an application into a “desired” state fast, and having a dump at that state would speed things up.
Another example, take periodic snapshots of running applications and transfer them on another machine for debugging or behavior and performance analysis.
Checkpoint/Restore In Userspace (CRIU)
CRIU is a project to implement checkpoint/restore functionality for Linux tasks. CRIU provides utility to freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. Then use the files to restore and run the application from the point it was frozen at.
CRIU uses kernel interface ptrace to seize the process and injects parasite code to dump the memory pages of the process into image files from within the process’s address space.
For each checkpointed part of the process, separate image files are created. Information about memory pages, for example, is collected from /proc/$PID/smaps, /proc/$PID/mapfiles/ and from /proc/$PID/pagemap
CRIU uses the information gathered during checkpointing to restore a process. The process is transformed into the same state the process was in before being checkpointed. Files are opened and positioned as they were before, memory is restored to the same state, and all other remaining information from the image files is used to restore the process. Once the state is restored, the restored process resumes control and continues from the point at which it was previously checkpointed.
Refer project home at http://criu.org or the project github https://github.com/checkpoint-restore/criu
Live migration using CRIU
Live migration attempts to provide a seamless transfer of service between physical machines without impacting client processes or applications. The criu utility can be used to perform live migration of apps or containers. This can be achieved by using a shared file-system such as NFS.
CRIU integration with Docker
Docker provides a feature in the experimental mode(since Docker 1.13, so every later version, like Docker 17.03, containerd version 1.5.0 or newer) that allows you to freeze a running container by checkpointing it, which turns its state into a collection of files on disk. Later, the container can be restored from the point it was frozen.
For demonstration, I am using Ubuntu 18.04
sudo apt-get update -ysudo apt-get install -y criu
Enable experimental features
echo “{\”experimental\”: true}” >> /etc/docker/daemon.json
systemctl restart dockercheckpoint
There’s a top level checkpoint sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.
Here’s an example of creating a checkpoint, from a container that simply logs an integer in a loop.
First, we create container:
docker run -d — name looper busybox /bin/sh -c ‘i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done’You can verify the container is running by printings its logs:
docker logs looper
If you do this a few times you’ll notice the integer increasing. Now, we checkpoint the container:
docker checkpoint create looper checkpoint1You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
restore
Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container start call. Here’s an example:
docker start — checkpoint checkpoint1 looper
If we then print the logs, you should see they start from where we left off and continue to increase.
For more details, refer https://criu.org/Docker or https://docs.docker.com/engine/reference/commandline/checkpoint/
CRIU integration with Podman
Similar to Docker, Podman is also a container technology for building container images and managing containers. In contrast with Docker, Podman is a daemonless container engine for developing, managing, and running OCI containers.
Install Podman
Add project repository
. /etc/os-release
echo “deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /” | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L “https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | sudo apt-key add -Once the repository is added, proceed to install Podman.
sudo apt update
sudo apt -y install podmanYou can verify the podman version with
podman — versionpodman version 3.0.1
checkpoint
To checkpoint the container use
podman run -dt — name looper busybox /bin/sh -c ‘i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done’You can verify the container is running by printings its logs:
podman logs looperIf you do this a few times you’ll notice the integer increasing. Now, we checkpoint the container:
podman container checkpoint looperrestore
podman container restore looperthe container should start from where it left off and continue to increase the count.
podman logs looperFor more details, refer https://criu.org/Podman or https://podman.io/getting-started/checkpoint
Checkpoint/Restore Support in Kubernetes
The are some initiatives related to checkpoint/restore in Kubernetes and it seems that it will make way into upstream Kubernetes soon.
https://github.com/kubernetes/kubernetes/pull/104907
It is planned to be merged as an Alpha feature soon.
At KubeCon + CloudNative North America 2021, RedHat presented details about Kubernetes and checkpoint restore.
