Making GPU Snapshot and Restore Faster
GPU snapshotting reduces inference cold starts by restoring an initialized process instead of loading weights, initializing CUDA, and rebuilding runtime state on every startup.
For Qwen2.5-14B on vLLM, the checkpoint is roughly 56 GB. At that size, extra copies and serial reads become expensive quickly.
Design
pod-snapshotter separates orchestration from execution. The manager handles the Kubernetes lifecycle, while the node agent performs checkpoint and restore operations on the host.
The original paths were:
Snapshot:
Manager → Kubelet API → containerd → runc → CRIU → TAR
Restore:
Manager → Placeholder Pod → Node Agent → runc restore
Snapshot creation used kubelet's checkpoint API, which ultimately returned a tar archive.
Restore already followed a different path because kubelet has no equivalent restore endpoint. The manager creates a placeholder pod for GPU allocation, networking, volumes, and other Kubernetes-managed state, and the node agent restores the process into it.
A ~56 GB snapshot took 1,279 seconds to create, while stock CRIU took 72.6 seconds to restore it. Both looked like I/O bottlenecks, but the causes were different: snapshot creation was moving the same bytes too many times, while restore was processing necessary bytes too serially.
Snapshot: keep CRIU's native format
CRIU naturally writes a directory of checkpoint images. The kubelet path converted that directory into a tar, after which the agent extracted it back into a directory on NVMe.
CRIU images → TAR → Extract → CRIU images on NVMe
The tar existed because kubelet returned one. runc restore --image-path ultimately wants the CRIU image directory again.
Since the node agent already had the privileges needed to invoke runc directly, snapshot creation moved there as well.
Agent → runc checkpoint → CRIU → NVMe
CRIU now writes directly into the final artifact directory, with a manifest written last to mark the snapshot complete. This removes the intermediate archive while keeping the artifact close to the format CRIU consumes during restore.
Observation — snapshot creation dropped from 1,279s to 297s, a 4.3× improvement.
Kubelet + TAR 1,279 s
Agent direct 297 s
The I/O counters make the difference clearer:
Kubelet + TAR Agent Direct
NVMe written 59.1 GB 56.5 GB
OS disk written 257.9 GB 0.2 GB
The direct path writes 56.5 GB for a 56.49 GB checkpoint, effectively one copy of the artifact. In the kubelet path, CRIU itself took roughly 121 seconds. Around 793 seconds were spent creating the archive.
The optimization was not making tar faster. It was removing tar and preserving CRIU's image-directory format end to end. Removing tar only changes snapshot creation. Once both approaches produce the same CRIU image directory on NVMe, restore is effectively identical: 43s vs 42s.
That isolates the next bottleneck inside CRIU.
Restore: parallelize CRIU
A large part of the vLLM checkpoint consists of shared-memory and memfd-backed objects.
Stock CRIU restores much of this state serially, limiting the amount of I/O in flight against NVMe.
Stock: Serial
Patched: 8-thread parallel
The fork is based on CRIU 4.2.1 and contains six changes to the restore path.
What changed in CRIU
1 Make shared restore state thread-safe
2 Add native async page reads
3 Parallelize shared-memory restore
4 Parallelize memfd restore
5 Enable async reads for memfd + shmem pages
6 Bound read coalescing
Patches 3 and 4 account for most of the controlled improvement. They remove serial loops around shared-memory and memfd restore and allow independent objects to be reconstructed concurrently.
Patches 5 and 6 improve the page-read path. The async path existed, but 205 of 208 page images in this workload were bypassing it. Patch 5 moves those pages onto async I/O, while patch 6 limits read coalescing so enough independent requests remain in flight.
Observation — CRIU restore dropped from 72.6s to 35.9s, while end-to-end restore dropped from 79s to 43s.
Stock Patched
CRIU restore 72.6 s 35.9 s
Restore → serving 79.0 s 43.0 s
NVMe read 56.4 GB 56.4 GB
That is 2.02× faster inside CRIU and 1.84× faster end to end.
Both versions still read the same 56.4 GB. What changed was how quickly CRIU consumed it:
Stock CRIU 0.71 GB/s
Patched CRIU 1.31 GB/s
The logs make the attribution clearer. Both versions restore 205 memfd objects. Stock CRIU processes them serially, while the patched version distributes them across eight threads.
After that phase, the remaining restore time is almost unchanged:
Stock 15.6 s
Patched 16.0 s
Most of the ~37-second improvement comes from removing the serial memfd bottleneck.
Async I/O helped further, but less than expected. Moving the relevant pages onto the async path improved the read phase by roughly 2 seconds, and queue depth stopped helping beyond 16. Once the serial restore bottleneck was removed, NVMe itself was no longer the main constraint.
Overall learning
Snapshot creation improved by moving fewer bytes and preserving CRIU's native image-directory format end to end.
Before: CRIU → TAR → Extract → CRIU directory
After: CRIU → NVMe
Result: 1,279s → 297s
Restore was different. Those bytes still had to be read, so the improvement came from increasing concurrency.
Before: Serial
After: 8-thread parallel
Result: 79s → 43s end to end
The useful distinction is simple: first check whether the bytes need to move at all. If they do, check whether the software above the storage is issuing enough parallel work to keep the device busy.
All measurements were run on Azure Standard_NC24ads_A100_v4 with an A100 80 GB GPU, 216 GB RAM, local NVMe, Qwen2.5-14B-Instruct, vLLM 0.9.2, and CRIU 4.2.1. Cold restores were confirmed by observing the full 56.4 GB read from NVMe, and each restore was validated with token generation.