criu: time and space travel for linux containers -- kir kolyshkin
TRANSCRIPT
![Page 1: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/1.jpg)
CRIU:
time and space travel
for Linux containers
CRIU:
time and space travel
for Linux containers
Kirill Kolyshkin
ContainerDays NYC, 30 Oct 2015
![Page 2: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/2.jpg)
AgendaAgenda
• Why would we want to migrate containers
• Why wouldn't we want to migrate containers
• How complex is to migrate containers
2
![Page 3: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/3.jpg)
Live migration at a glanceLive migration at a glance
• Save the state
• Transfer the state
• Restore the state
3
![Page 4: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/4.jpg)
Container live migrationContainer live migration
4
![Page 5: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/5.jpg)
Why would we want to migrate containers?Why would we want to migrate containers?
• It's awesome!
• Load balancing in a cluster
• Kernel upgrade
– Can be done without migration
• Hardware upgrade
5
![Page 6: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/6.jpg)
Why wouldn't we want to live migrate containers?Why wouldn't we want to live migrate containers?
6
![Page 7: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/7.jpg)
How to avoid live migrating containersHow to avoid live migrating containers
• Incoming traffic load balancing
• Microservices
• Crash-driven upgrades
• Scheduled downtimes
7
![Page 8: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/8.jpg)
How to make live migration really live?How to make live migration really live?
• Need to get rid of migrating memory while the container is frozen
• Two ways:
– Pre-copy the memory
– Post-copy the memory
8
![Page 9: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/9.jpg)
Live migration in more detailsLive migration in more details
• Pre-copy: collect and transfer the memory (might be iterative)
• Freeze the container
• Save its state
• Copy the state
• Restore
• Unfreeze
• Post-copy: swap in the memory over the network
9
![Page 10: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/10.jpg)
Obstacles, booby traps, and rakesObstacles, booby traps, and rakes
10
VS
![Page 11: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/11.jpg)
What do we need to migrateWhat do we need to migrate
• Virtual Machine
– Environment (i.e. virtual hardware)
– CPU state
– Memory
• Container
– Environment (cgroups, namespaces)
– Processes and stuff
– Memory11
![Page 12: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/12.jpg)
Collect and copy the memoryCollect and copy the memory
• Virtual Machine
– All memory is at hand
• Container
– Memory is spread through the processes
– Different types of memory (shared/private, backed by a file or not)
– Need to collect the processes first
● Only then collect the memory
12
![Page 13: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/13.jpg)
FreezingFreezing
• Virtual Machine
– Suspend all CPUs
• Container
– Walk the tree (/proc), catch the processes and freeze those
– Freeze cgroup helps a bit
13
![Page 14: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/14.jpg)
Saving the stateSaving the state
• Virtual Machine
– Hardware state, tree, 300K, ~70 objects
• Container
– State of all objects, graph, 160K, ~1000 objects
– Not all objects have decent API to get the state
14
![Page 15: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/15.jpg)
Copying the stateCopying the state
• Virtual Machine
– Can read and copy at once, easy to serialize
• Container
– Not easy to serialize as it's a graph not a tree
15
![Page 16: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/16.jpg)
Restoring the stateRestoring the state
• VM: recreate the memory, state of CPUs and virtual hardware
• Containers
– In-kernel: create a myriad of small objects
– In CRIU: same, but there might not be a convenient API
● Over 1000 syscalls
● Need to sort it all out
16
![Page 17: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/17.jpg)
FreezeFreeze
• VM: resume the virtual CPUs
• Container
– Either SIGCONT through the tree
– Or “unfreeze” the cgroup
– Problem: need to wake processes in the proper order
17
![Page 18: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/18.jpg)
Post-memory migration: network swap devicePost-memory migration: network swap device
• Not yet ready for neither VMs nor CTs
• userfaultfd by Andrea Arcangeli of Red Hat
– a file descriptor to inform about page fault and get a memory back
– merged into 4.2 kernel
– work in progress to use it for KVM/QEMU
• Container
– Userfault FD is not sufficient for CRIU case
18
![Page 19: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/19.jpg)
ImplementationImplementation
• https://criu.org
• plus.google.com/+CriuOrg
• @__criu__
• github: xemul/criu
19
![Page 20: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/20.jpg)
CRIU uses beyond the live migrationCRIU uses beyond the live migration
• HPC jobs: periodic checkpoints
• Slow boot services speed up
• That magical SAVE button e.g. in games
• Software testing speed up
• Reverse debugging
20
![Page 21: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/21.jpg)
Live migrationLive migration
• P.Haul
– Process hauler
– http://criu.org/P.Haul
– Uses CRIU for c/r
21
![Page 22: CRIU: time and space travel for Linux containers -- Kir Kolyshkin](https://reader031.vdocument.in/reader031/viewer/2022030305/587458f91a28abd53e8b92d1/html5/thumbnails/22.jpg)
That's all Folks!
Kirill Kolyshkin
That's all Folks!
Kirill Kolyshkin