Beyond Lazy Pulling: Why Signal-Driven Prewarming Is Better for Cold Starts

Container image cold starts remain a major challenge in serverless GPU infrastructure. Inference images often contain gigabytes of CUDA libraries, PyTorch dependencies, compiled kernels, and runtime binaries, even though applications may access only a fraction of that data during startup.

Nydus addresses this through lazy loading, while Dragonfly accelerates distribution using peer-to-peer caching. Both are useful optimizations. But they raise a more fundamental question:

If we know which workloads are about to run, why wait until they start to fetch the data they need?

Lazy loading shifts cold-start costs rather than eliminating them

Traditional container runtimes download and extract image layers before starting a container. Nydus improves this by fetching filesystem data on demand. This significantly reduces the upfront image-pulling time. However, the data still needs to be fetched eventually.

Lazy loading often shifts cold-start latency from the image-pull phase into application initialization. It doesn't necessarily eliminate that latency.

For example, an inference container might start quickly but later block while loading PyTorch libraries, initializing CUDA, loading model dependencies, or accessing a previously untouched runtime component.

Dragonfly improves this by fetching missing data from nearby peers instead of repeatedly accessing the origin registry. But even a fast peer request can introduce network latency on the critical execution path.

This creates two different optimization goals:

• Lazy loading: Reduce the data required before container creation.
• Prewarming: Ensure startup-critical data is available before the application needs it.

The important metric isn't simply container startup time. It's time to application readiness and first successful inference.

Cold-cache miss illustrated. Warm hits avoid remote fetching; Dragonfly also supports prefetching.

A better approach: manifest-driven, signal-based prewarming

Instead of relying entirely on reactive loading, we can analyze images ahead of time and proactively populate caches using workload signals.

The strategy combines two ideas:

Manifest-driven warming — what to fetch

Analyze an image using static analysis or representative startup traces to identify frequently accessed files and startup-critical chunks. Generate a manifest describing that working set, associated with the immutable image digest so it can be reused across deployments.

Signal-driven warming — when and where to fetch

Use workload signals to trigger prewarming:

• Deployments and new image revisions
• Autoscaling predictions and traffic spikes
• GPU node provisioning and scheduling decisions
• Frequently used models and runtime images

The cache coordinator distributes the required content to target nodes before execution. When the application starts, its critical data is already available on local NVMe. Unexpected accesses continue through Nydus lazy loading.

Rather than eliminating data transfer, this approach moves predictable transfers outside the startup critical path whenever the platform has sufficient advance notice.

How fuse-cache enables this

I built fuse-cache around a distributed, three-tier caching architecture:

fuse-cache integrates with Nydus through a local registry-compatible backend, serving image-blob cache misses through local NVMe, peers, or cloud storage.

OCI blobs are addressed by content digest, enabling reuse across images. The system also provides coordinator-driven cache warming, allowing selected data to be distributed across GPU nodes before execution.

By combining these primitives with image-analysis manifests and workload signals, cache population becomes part of workload orchestration rather than something driven only by application reads.

Proactive prewarming — Nydus + fuse-cache

Proposed end-to-end workflow: coordinator-driven warmup already exists; image-manifest generation and signal-driven orchestration are proposed integration work.

Why this can outperform reactive peer-assisted pulling

Consider launching 100 inference containers across a GPU cluster. With reactive Nydus and Dragonfly, containers may start quickly but block later on missing filesystem chunks during library initialization or the first inference request.

Signal-driven prewarming populates the local cache ahead of execution, reducing blocking network reads during initialization and making readiness latency more predictable.

However, the benefit depends on having enough time to warm the cache before execution. If prewarming begins only after a request arrives, the total time to readiness may not improve.

Dragonfly vs. fuse-cache

Dragonfly supports targeted preheating of container images by manifest URL and files by URL. Its current preheat controls select clusters and peers, including peer IPs, counts, or percentages. An image tarball exposed as a downloadable file can therefore be warmed as file content; caching that archive alone does not import or unpack it into a container runtime.

Node targeting and working-set selection are separate capabilities. Dragonfly’s documented preheat interface selects image manifests or file URLs and destination peers; it does not itself analyze an image to identify only the application’s required startup files or chunks. An OCI image manifest describes image content, while the startup manifest proposed here describes the workload’s critical working set. Nydus separately supports file-level prefetch hints and startup profiling, so selective prefetch is also possible in a Nydus + Dragonfly stack. The proposed fuse-cache integration connects that working-set selection to workload signals and pre-execution placement.

For predictable workloads, preventing a remote read on the startup critical path can be more valuable than making that remote read faster.

Why lazy loading still matters

Prewarming cannot predict every application access. Different GPU architectures, runtime options, and dynamic code paths may require dependencies that weren't identified during image analysis.

Nydus remains essential for fetching those files on demand. P2P remains valuable for distributing cache content efficiently.

Manifest-driven warming determines what to fetch. Signal-driven warming determines when and where. Lazy loading handles everything else.

Conclusion

With fuse-cache, the goal is to move predictable data transfers ahead of execution, combining proactive NVMe warming, distributed reuse, and Nydus lazy loading.

The fastest remote read isn't one served by a nearby peer. It's the one the application never has to wait for.

Implementation: fuse-cache on GitHub