• Wenbo Qi(Gaius), Dragonfly/ModelPack Maintainer
• Chenyu Zhang(Chlins), Harbor/ModelPack Maintainer
• Feynman Zhou, ORAS Maintainer, CNCF Ambassador

Reviewer:

• Sascha Grunert, CRI-O Maintainer
• Wei Fu, containerd Maintainer

The weight of AI models: Why infrastructure always arrives slowly​

As AI adoption accelerates across industries, organizations face a critical bottleneck that is often overlooked until it becomes a serious obstacle: reliably managing and distributing large model weight files at scale. A model's weights serve as the central artifact that bridges both training and inference pipelines — yet the infrastructure surrounding this artifact is frequently an afterthought.

This article addresses the operational challenges of managing AI model artifacts at enterprise scale, and introduces a cloud-native solution that brings software delivery best practices — versioning, immutability, and GitOps, to the world of large model files.

The gap nobody talks about — until it breaks production​

The cloud native gap: Most existing ML model storage approaches were not designed with Kubernetes-native delivery in mind, leaving a critical gap between how software artifacts are managed and how model artifacts are managed.

Today, enterprises operate AI infrastructure on Kubernetes yet their model artifact management lags behind. Software containers are pulled from OCI registries with full versioning, security scanning, and rollback support. Model weights, by contrast, are often downloaded via ad hoc scripts, copied manually between storage buckets, or distributed through unsecured shared filesystems. This gap creates deployment fragility, security risks, and operational overhead at scale.

When your model weighs more than your entire app​

Modern foundation models are not small. A single model checkpoint can range from tens of gigabytes to several terabytes. For reference, a quantized LLaMA-3 70B model weighs approximately 140 GB, while frontier multimodal models can easily exceed 1 TB. These are not files you version-control with standard Git — they demand dedicated storage strategies, efficient transfer protocols, and careful access control.

The core challenges are: storage at scale, distribution speed, and reproducibility. Teams need to store multiple model versions, rapidly distribute them to GPU inference nodes across regions, and guarantee that any deployment can be traced back to an exact, immutable artifact.

Three paths forward — and why none of them are enough​

Rethinking the delivery pipeline: Models deserve better than a shell script​

The approach described here treats AI model weights as first-class OCI (Open Container Initiative) artifacts, packaging them in the same container registries used for application images. This enables model delivery to leverage the full ecosystem of container tooling: security scanning, signed provenance, GitOps-driven deployment, and Kubernetes-native pulling.

What If we shipped models the same way we ship code?​

In the cloud-native era, developers have long established a mature and efficient paradigm for software delivery.

The software delivery:

1. Develop: Developers commit code to a Git repository, manage code changes through branches, and define versions using tags at key milestones.
2. Build: CI/CD pipelines compile and test, packaging the output into an immutable Container Image.
3. Manage and deliver: Images are stored in a Container Registry. Supply chain security (scanning/signing), RBAC, and P2P distribution ensure safe delivery.
4. Deploy: DevOps engineers use declarative Kubernetes YAML to define the desired state. The Container's lifecycle is managed by Kubernetes.

The cloud native AI model delivery:

1. Develop: Algorithm engineers push weights and configs to the Hugging Face Hub, treating it as the Git Repository.
2. Build: CI/CD pipelines package weights, runtime configurations, and metadata into an immutable Model Artifact.
3. Manage and deliver: The Model Artifact is managed by an Artifact Registry, reusing the existing container infrastructure and toolchain.
4. Deploy: Engineers use Kubernetes OCI Volumes or a Model CSI Driver. Models are mounted into the inference Container as Volumes via declarative semantics, decoupling the AI model from the inference engine (vLLM, SGLang, etc.).

By applying software delivery paradigms and supply chain thinking to model lifecycle management, we constructed a granular, efficient system that resolves the challenges of managing and distributing AI models in production.

Walking the pipeline: A build story in four steps​

Build​

modctl is a CLI tool designed to package AI models into OCI artifacts. It standardizes versioning, storage, distribution and deployment, ensuring integration with the cloud-native ecosystem.

Step 1: Auto-generate Modelfile​

Run the following in the model directory to generate a definition file.

modctl modelfile generate .

Step 2: Customize Modelfile​

You can also customize the content of the Modelfile.

# Model name (string), such as llama3-8b-instruct, gpt2-xl, qwen2-vl-72b-instruct, etc.
NAME qwen2.5-0.5b

# Model architecture (string), such as transformer, cnn, rnn, etc.
ARCH transformer

# Model family (string), such as llama3, gpt2, qwen2, etc.
FAMILY qwen2

# Model format (string), such as onnx, tensorflow, pytorch, etc.
FORMAT safetensors

# Specify model configuration file, support glob path pattern.
CONFIG config.json

# Specify model configuration file, support glob path pattern.
CONFIG generation_config.json

# Model weight, support glob path pattern.
MODEL *.safetensors

# Specify code, support glob path pattern.
CODE *.py

Step 3: Login to Artifact Registry (Harbor)​

modctl login -u username -p password harbor.registry.com

Step 4: Build OCI Artifact​

modctl build -t harbor.registry.com/models/qwen2.5-0.5b:v1 -f Modelfile .

A Model Manifest is generated after the build. Descriptive information such as ARCH, FAMILY, and FORMAT is stored in a file with the media type application/vnd.cncf.model.config.v1+json.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.cncf.model.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.cncf.model.config.v1+json",
    "digest": "sha256:d5815835051dd97d800a03f641ed8162877920e734d3d705b698912602b8c763",
    "size": 301
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.model.weight.v1.raw",
      "digest": "sha256:3f907c1a03bf20f20355fe449e18ff3f9de2e49570ffb536f1a32f20c7179808",
      "size": 4294967296
    },
    {
      "mediaType": "application/vnd.cncf.model.weight.v1.raw",
      "digest": "sha256:6d923539c5c208de77146335584252c0b1b81e35c122dd696fe6e04ed03d7411",
      "size": 5018536960
    },
    {
      "mediaType": "application/vnd.cncf.model.weight.config.v1.raw",
      "digest": "sha256:a5378e569c625f7643952fcab30c74f2a84ece52335c292e630f740ac4694146",
      "size": 106
    },
    {
      "mediaType": "application/vnd.cncf.model.weight.code.v1.raw",
      "digest": "sha256:15da0921e8d8f25871e95b8b1fac958fc9caf453bad6f48c881b3d76785b9f9d",
      "size": 394
    },
    {
      "mediaType": "application/vnd.cncf.model.doc.v1.raw",
      "digest": "sha256:5e236ec37438b02c01c83d134203a646cb354766ac294e533a308dd8caa3a11e",
      "size": 23040
    }
  ]
}

Step 5: Push​

modctl push harbor.registry.com/models/qwen2.5-0.5b:v1

Management​

Current AI infrastructure workflows focus heavily on model distribution performance, often ignoring model management standards. Manual copying works for experiments, but in large-scale production, lacking unified versioning, metadata specs, and lifecycle management is poor practice. As the standard cloud-native Artifact Registry, Harbor is ideally suited for model storage, treating models as inference artifacts.

Harbor standardizes AI model management through:

• Versioning: Models are OCI Artifacts with immutable Tags and Sha256 Digests. This guarantees deterministic inference environments. Meanwhile, it visually presents the model's basic attributes, parameter configurations, display information, and the file list, which not only reduce the risks of unknown versions but also achieves full transparency of the model.

• RBAC: Fine-grained access control. Control who can PUSH (e.g., Algorithm Engineers), who can only PULL (e.g., Inference Services), and who has administrative privileges.

• Lifecycle management: Tag retention policies automatically purge non-release versions while locking active versions, balancing storage costs with stability.

• Supply chain security: Integration with Cosign/Notation for signing. Harbor enforces signature verification before distribution, preventing model poisoning attacks.

• Replication: Automated, incremental synchronization between central and edge registries or active-standby clusters.

• Audit: Comprehensive logging of all artifact operations (pull/push/delete) for security compliance and traceability.

Delivery​

Downloading terabyte-sized model weights directly from the origin introduces bandwidth bottlenecks. We utilize Dragonfly for P2P-based distribution, integrated with Harbor for preheating.

Dragonfly P2P-based distribution​

For large-scale distribution scenarios, Dragonfly has been deeply optimized based on P2P technology. Taking the example of 500 nodes downloading a 1TB model, the system distributes the initial download tasks of different layers across nodes to maximize downstream bandwidth utilization and avoid single-point congestion. Combined with a secondary bandwidth-aware scheduling algorithm, it dynamically adjusts download paths to eliminate network hotspots and long-tail latency. For individual model weight, Dragonfly splits individual model weights into pieces and fetches them concurrently from the origin. This enables streaming-based downloading, allowing users to share models without waiting for the complete file. This solution has been proven in high-performance AI clusters, utilizing 70%-80% of each node's bandwidth and improving model deployment efficiency.

Preheating​

For latency-sensitive inference services, Harbor triggers Dragonfly to distribute and cache data on target nodes before service scaling. When the instance starts, the model loads from the local disk, achieving zero network latency.

Deployment​

Deployment focuses on decoupling the Model (Data) from the Inference Engine (Compute). By leveraging Kubernetes declarative primitives, the Engine runs as a Container, while the Model is mounted as a Volume. This native approach not only enables multiple Pods on the same node to share and reuse the model, saving disk space, but also leverages the preheating and P2P capabilities of Harbor & Dragonfly to eliminate the latency of pulling large model weights, significantly improving startup speed.

OCI Volumes (Kubernetes 1.31+)​

Native support for mounting OCI artifacts as volumes via CRI-O/containerd. This feature was introduced as alpha in Kubernetes 1.31 (requires enabling the ImageVolume feature gate) and promoted to beta in Kubernetes 1.33 (enabled by default, no feature gate configuration needed). CRI-O specifically enhances this for LLMs by avoiding decompression overhead at mount time by storing layers uncompressed, resulting in superior performance when mounting large model files.

Step 1: Build YAML​

apiVersion: v1
kind: Pod
metadata:
  name: vllm-cpu-inference
  labels:
    app: vllm
spec:
  containers:
    - name: vllm
      image: openeuler/vllm-cpu:latest
      command:
        - 'python3'
        - '-m'
        - 'vllm.entrypoints.openai.api_server'
      args:
        - '--model'
        - '/models'
        - '--dtype'
        - 'float32'
        - '--host'
        - '0.0.0.0'
        - '--port'
        - '8000'
        - '--max-model-len'
        - '1024'
        - '--disable-log-requests'
      env:
        - name: VLLM_CPU_KVCACHE_SPACE
          value: '1'
        - name: VLLM_WORKER_MULTIPROC_METHOD
          value: 'spawn'
      resources:
        requests:
          memory: '2Gi'
          cpu: '1'
        limits:
          memory: '16Gi'
          cpu: '8'
      volumeMounts:
        - name: model-volume
          mountPath: /models
          readOnly: true
      ports:
        - containerPort: 8000
          protocol: TCP
          name: http
      livenessProbe:
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 60
        periodSeconds: 10
        timeoutSeconds: 5
      readinessProbe:
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 30
        periodSeconds: 5
  volumes:
    - name: model-volume
      image:
        reference: ghcr.io/chlins/qwen2.5-0.5b:v1
        pullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000
      protocol: TCP
      name: http
  type: ClusterIP

Step 2: Deploy inference Workload​

Step 3: Call Inference Workload​

Model CSI Driver​

For compatibility with Kubernetes 1.31 and older, we offer the Model CSI Driver as an interim solution to mount and deploy models as volumes. As OCI Volumes are slated for GA in Kubernetes 1.36, shifting to native OCI Volumes is recommended for the long term.

Step 1: Build YAML​

apiVersion: v1
kind: Pod
metadata:
  name: vllm-cpu-inference
  labels:
    app: vllm
spec:
  containers:
    - name: vllm
      image: openeuler/vllm-cpu:latest
      command:
        - 'python3'
        - '-m'
        - 'vllm.entrypoints.openai.api_server'
      args:
        - '--model'
        - '/models'
        - '--dtype'
        - 'float32'
        - '--host'
        - '0.0.0.0'
        - '--port'
        - '8000'
        - '--max-model-len'
        - '1024'
        - '--disable-log-requests'
      env:
        - name: VLLM_CPU_KVCACHE_SPACE
          value: '1'
        - name: VLLM_WORKER_MULTIPROC_METHOD
          value: 'spawn'
      resources:
        requests:
          memory: '2Gi'
          cpu: '1'
        limits:
          memory: '16Gi'
          cpu: '8'
      volumeMounts:
        - name: model-volume
          mountPath: /models
          readOnly: true
      ports:
        - containerPort: 8000
          protocol: TCP
          name: http
      livenessProbe:
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 60
        periodSeconds: 10
        timeoutSeconds: 5
      readinessProbe:
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 30
        periodSeconds: 5
  volumes:
    - name: model-volume
      csi:
        driver: model.csi.modelpack.org
        volumeAttributes:
          model.csi.modelpack.org/reference: ghcr.io/chlins/qwen2.5-0.5b:v1
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000
      protocol: TCP
      name: http
  type: ClusterIP

Step 2: Deploy Inference Workload​

Step 3: Call Inference Workload​

Future​

• Enhanced Preheating: Allow models to be preheated to specified nodes and querying cache distribution across nodes for model-aware pod scheduling.
• Dragonfly RDMA Acceleration: Enable Dragonfly to utilize InfiniBand or RoCE to improve the speed of distribution.
• Lazy Loading: Implement on-demand downloading of model weights to reduce startup latency.
• containerd Optimization: Enhance the OCI Volumes implementation to reduce decompression overhead for large layers.
• Model Security Scanning: Introduce deep scanning capabilities specifically designed for model weights to detect embedded malicious payloads.

Collaborative Projects​

• Kubernetes: https://github.com/kubernetes/kubernetes
• Harbor: https://github.com/goharbor/harbor
• Dragonfly: https://github.com/dragonflyoss/dragonfly
• CRI-O: https://github.com/cri-o/cri-o
• containerd: https://github.com/containerd/containerd
• modctl: https://github.com/modelpack/modctl
• Model CSI Driver: https://github.com/modelpack/model-csi-driver
• Model Spec: https://github.com/modelpack/model-spec
• ORAS: https://github.com/oras-project/oras

References​

• Kubernetes - Read Only Volumes Based On OCI Artifacts: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/
• Harbor - AI Model Processor: https://github.com/goharbor/community/blob/main/proposals/new/AI-model-processor.md
• Dragonfly - Load-Aware Scheduling Algorithm: https://d7y.io/docs/next/operations/deployment/applications/scheduler/#bandwidth-aware-scheduling-algorithm
• CRI-O - Add OCI Volume/Image Source Support: https://github.com/cri-o/cri-o/pull/8317
• containerd - Add OCI/Image Volume Source support: https://github.com/containerd/containerd/pull/10579

Dragonfly now natively speaks hf:// and modelscope:// — turning the world's two largest model hubs into P2P-accelerated sources for enterprise AI infrastructure.

The Problem: AI Model Distribution Is Broken at Scale​

The AI industry has a dirty secret: downloading models is painfully slow and absurdly wasteful.

Consider a typical scenario. Your ML platform team manages a Kubernetes cluster with 200 GPU nodes. A new version of a 70B parameter model drops — say, DeepSeek-V3 at ~130 GB. Every node needs a local copy. That's 26 TB of data pulled from a single model hub, all funneling through the same origin servers, the same egress bandwidth, and the same rate limits.

The numbers get worse:

• Hugging Face Hub serves over 1 million models, with individual files regularly exceeding 10 GB (safetensors, GGUF quantizations).
• ModelScope Hub hosts over 10,000 models — including some of the largest Chinese-origin models like Qwen, Yi, and inclusionAI's Ling series — serving a rapidly growing global user base.
• Git LFS, which backs large file storage on these platforms, was never designed for mass fan-out distribution.
• Rate limits throttle unauthenticated requests; even authenticated tokens hit ceilings under burst traffic.
• Network costs compound — every byte crosses the internet boundary N times instead of once.

Existing workarounds — NFS mounts, pre-baked container images, manual S3 mirrors — all introduce operational complexity, stale-model risk, or storage overhead.

What if the infrastructure itself could solve this? What if downloading a model to the 200th node was as fast as downloading it to the first — regardless of which model hub it came from?

That's exactly what the new hf:// and modelscope:// protocol support in Dragonfly delivers.

What Is Dragonfly?​

Dragonfly is a CNCF Graduated project that provides a P2P-based file distribution system. Originally built for container image distribution at Alibaba-scale (processing billions of requests daily), Dragonfly turns every downloading node into a seed for its peers.

Core Architecture:

Figure 1: End-to-end flow of the P2P model distribution in Dragonfly. The Seed Peer fetches the model from the origin hub once (Step 1), the Dragonfly Scheduler computes the P2P topology (Step 3), and GPU nodes share pieces via micro-task distribution (Step 5) — reducing origin traffic from 26 TB to ~130 GB across a 200-node cluster.

The magic: Dragonfly splits files into small pieces and distributes them across the P2P mesh. The origin (Hugging Face Hub or ModelScope Hub) is hit once by the seed peer. Critically, the Seed Peer does not need to finish downloading the entire model before sharing with other peers — as soon as any single piece is downloaded, it can be shared immediately. This piece-based streaming download means distribution begins in parallel with the initial fetch, dramatically reducing total transfer time. For a 130 GB model across 200 nodes, origin traffic drops from 26 TB to ~130 GB — a 99.5% reduction.

Until now, Dragonfly supported HTTP/HTTPS, S3, GCS, Azure Blob Storage, Alibaba OSS, Huawei OBS, Tencent COS, and HDFS backends. But the two largest sources of AI model artifacts — Hugging Face and ModelScope — required users to pre-resolve hub URLs into raw HTTPS links, losing authentication context, revision pinning, and repository structure awareness.

Not anymore.

Introducing Native Model Hub Protocols in Dragonfly​

With two new backends merged into the Dragonfly client, dfget (Dragonfly's download tool) now natively understands both Hugging Face and ModelScope URLs. No proxies. No URL rewriting. No wrapper scripts.

The hf:// Protocol — Hugging Face Hub​

Merged via PR #1665, this backend adds first-class support for downloading from the world's largest open-source model repository.

URL Format:

hf://[<repository_type>/]<owner>/<repository>[/<path>]

Components:

Usage Examples:

# Download a single model file with P2P acceleration
dfget hf://deepseek-ai/DeepSeek-R1/model.safetensors \
  -O /models/DeepSeek-R1/model.safetensors

# Download an entire repository recursively
dfget hf://deepseek-ai/DeepSeek-R1 \
  -O /models/DeepSeek-R1/ -r

# Download a specific dataset
dfget hf://datasets/huggingface/squad/train.json \
  -O /data/squad/train.json

# Access private repositories with authentication
dfget hf://owner/private-model/weights.bin \
  -O /models/private/weights.bin \
  --hf-token=hf_xxxxxxxxxxxxx

# Pin to a specific model version
dfget hf://deepseek-ai/DeepSeek-R1/model.safetensors --hf-revision v2.0 \
  -O /models/DeepSeek-R1/model.safetensors

The modelscope:// Protocol — ModelScope Hub​

Merged via PR #1673, this backend brings the same P2P-accelerated experience to ModelScope Hub — Alibaba's open model platform hosting thousands of models, with particularly strong coverage of Chinese-origin LLMs and multimodal models.

URL Format:

modelscope://[<repo_type>/]<owner>/<repo>[/<path>]

Components:

Usage Examples:

# Download a model repository with P2P acceleration
dfget modelscope://deepseek-ai/DeepSeek-R1 \
  -O /models/DeepSeek-R1/ -r

# Download a single file
dfget modelscope://deepseek-ai/DeepSeek-R1/config.json \
  -O /models/DeepSeek-R1/config.json

# Download with authentication for private repos
dfget modelscope://deepseek-ai/DeepSeek-R1/config.json \
  -O /tmp/config.json --ms-token=<token>

# Download a dataset
dfget modelscope://datasets/damo/squad-zh/train.json \
  -O /data/squad-zh/train.json

# Download from a specific revision
dfget modelscope://deepseek-ai/DeepSeek-R1/config.json --ms-revision v2.0 \
  -O /models/DeepSeek-R1/config.json

Under the Hood: Technical Deep Dive​

Both implementations live in the Dragonfly Rust client as new backend modules. Here's how they work at the systems level.

1. Pluggable Backend Architecture​

Dragonfly uses a pluggable backend system. Each URL scheme (http, s3, gs, hf, modelscope, etc.) maps to a backend that implements the Backend trait:

#[tonic::async_trait]
pub trait Backend {
    fn scheme(&self) -> String;
    async fn stat(&self, request: StatRequest) -> Result<StatResponse>;
    async fn get(&self, request: GetRequest) -> Result<GetResponse<Body>>;
    async fn put(&self, request: PutRequest) -> Result<PutResponse>;
    async fn exists(&self, request: ExistsRequest) -> Result<bool>;
}

Both hf and modelscope backends are registered as builtin backends in the BackendFactory, sitting alongside HTTP, object storage, and HDFS:

// Hugging Face backend
self.backends.insert(
    "hf".to_string(),
    Box::new(hugging_face::HuggingFace::new(self.config.clone())?),
);

// ModelScope backend
self.backends.insert(
    "modelscope".to_string(),
    Box::new(modelscope::ModelScope::new()?),
);

This means both schemes are available everywhere dfget or the Dragonfly daemon operates — no additional configuration needed.

2. URL Parsing: Same Grammar, Different Conventions​

Both backends share the same URL grammar — scheme://[type/]owner/repo[/path] — but respect each platform's conventions:

3. Two Download Modes (Both Backends)​

Single File Mode (e.g., hf://owner/repo/file.bin or modelscope://owner/repo/file.bin):

1. Parse URL → extract file path
2. Build platform-specific download URL
3. stat() performs a HEAD request to get content length and validate existence
4. get() streams the file content through Dragonfly's piece-based P2P network
5. For HF: Git LFS redirects are handled transparently by the HTTP client

Repository Mode (e.g., hf://owner/repo -r or modelscope://owner/repo -r):

1. Parse URL → no file path present
2. Call platform-specific API to list repository files
3. Deserialize the repository metadata into a file listing
4. For each file, construct a scheme-native URL (not raw HTTPS), preserving backend semantics
5. Dragonfly's recursive download engine processes each file through the P2P mesh

This is a crucial design decision: recursive downloads emit hf:// or modelscope:// URLs back into the download pipeline, not raw HTTPS URLs. This preserves authentication context and ensures every file in the recursive download goes through the correct backend — maintaining token forwarding and URL semantics.

4. Platform-Specific API Integration​

Hugging Face uses a resolve-based download pattern where the server may return the file directly or redirect to Git LFS storage for large model files. The reqwest HTTP client follows these redirects automatically, making LFS handling completely transparent.

ModelScope uses a structured REST API with explicit endpoints for file listing (/repo/files). The API returns a JSON envelope with Code, Data, and Message fields. The file listing endpoint supports recursive traversal natively via the Recursive=true parameter, returning structured RepoFile objects with name, path, type, and size metadata.

5. Authentication​

Both backends support token-based authentication via CLI flags and bearer token headers:

# Hugging Face authentication
dfget hf://owner/private-model/weights.bin \
  --hf-token=hf_xxxxxxxxxxxxx

# ModelScope authentication
dfget modelscope://owner/private-model/config.json \
  --ms-token=<token>

Tokens propagate through all operations (stat, get, exists), enabling access to private repositories and gated models on both platforms.

Real-World Impact: Where This Matters​

1. Multi-Node GPU Cluster Model Deployment​

In large-scale enterprise environments — the kind I architect and operate daily — distributing a 130 GB model like meta-llama/Llama-2-70b across 50 GPU nodes creates a debilitating network bottleneck. I've seen this pattern cripple deployment velocity firsthand.

Before: Each of your 50 GPU nodes downloads the model independently.

• Total bandwidth: 6.5 TB from the model hub
• Time: Limited by origin server throughput and rate limits
• Cost: Full internet egress x 50

After: Seed peer fetches once, P2P distributes across the cluster.

• Origin bandwidth: ~130 GB (once)
• Time: Near-wire-speed from local peers after initial seed
• Cost: Minimal egress, heavy intra-cluster traffic (free)

When you're managing self-healing, multi-cloud Kubernetes clusters at enterprise scale, this kind of origin traffic reduction isn't an optimization — it's a prerequisite for operational sanity.

2. Multi-Hub Model Sourcing​

Teams increasingly source models from multiple hubs. A team might use Llama from Hugging Face and Qwen from ModelScope. With both backends built in, Dragonfly becomes the unified distribution layer regardless of origin:

# From Hugging Face
dfget hf://meta-llama/Llama-2-7b -O /models/llama2/ -r

# From ModelScope
dfget modelscope://qwen/Qwen-7B -O /models/qwen/ -r

Same P2P mesh. Same caching layer. Same operational model. Different origins.

3. CI/CD for ML Pipelines​

Model evaluation pipelines that spin up ephemeral runners to test against specific model versions benefit from revision pinning:

# Deterministic model versions in CI — from either hub
dfget hf://org/model --hf-revision abc123def -O /workspace/model/ -r
dfget modelscope://org/model --ms-revision v1.0 -O /workspace/model/ -r

Combined with Dragonfly's caching layer, repeated CI runs across different runners pull from local P2P cache instead of remote hubs. In the enterprise CI/CD systems I've built, this eliminates one of the last remaining sources of non-deterministic pipeline failures: flaky model downloads.

4. Cross-Platform Model Sourcing​

For organizations utilizing global infrastructure, Hugging Face serves as the primary hub. Dragonfly's dual-hub support enables a single distribution platform that routes to the optimal origin:

# Global clusters pull from Hugging Face
dfget hf://deepseek-ai/DeepSeek-R1 -O /models/DeepSeek-R1/ -r

5. Air-Gapped and Edge Deployments​

For environments with limited or no internet access — common in regulated enterprise and financial services infrastructure — Dragonfly's seed peer can be pre-loaded from an internet-connected staging area. Once seeded, internal nodes use P2P to distribute models without any external connectivity.

6. Dataset Distribution for Training​

Large-scale training jobs often need the same dataset replicated across data-parallel workers:

# From Hugging Face
dfget hf://datasets/allenai/c4/en/train-00000-of-01024.json.gz \
  -O /data/c4/train-00000.json.gz

# From ModelScope
dfget modelscope://datasets/damo/squad-zh/train.json \
  -O /data/squad-zh/train.json

P2P distribution turns O(N) origin fetches into O(1) origin + O(log N) P2P propagation.

Comparison: Why Not Just Use Platform CLIs?​

Platform-specific CLIs are excellent for individual developer workflows. The native protocol support in Dragonfly is for infrastructure-scale model distribution.

Getting Started​

Prerequisites​

• Dragonfly cluster deployed (scheduler + seed peer + peer on nodes)
• dfget CLI available on target machines

Quick Start​

1. Install Dragonfly (via Helm for Kubernetes):

helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install dragonfly dragonfly/dragonfly \
  --namespace dragonfly-system --create-namespace

2. Download models with P2P from either hub:

# From Hugging Face
dfget hf://deepseek-ai/DeepSeek-R1/model.safetensors -O ./model.safetensors

# From ModelScope
dfget modelscope://deepseek-ai/DeepSeek-R1/config.json -O ./config.json

# Recursive repository download (works with both)
dfget hf://deepseek-ai/DeepSeek-R1 -O ./DeepSeek-R1/ -r --hf-token=$HF_TOKEN
dfget modelscope://deepseek-ai/DeepSeek-R1 -O ./DeepSeek-R1/ -r --ms-token=$MS_TOKEN

3. Verify P2P is working:

# Check Dragonfly daemon logs for peer transfer activity
journalctl -u dfdaemon | grep "peer task"

What's Next​

These two backends are just the beginning. The architecture is designed for extensibility — adding support for additional model hubs follows the same pattern: implement the Backend trait, register the scheme, and the entire P2P mesh instantly serves the new source. Potential future enhancements include:

• Intelligent pre-warming: Automatically seed popular models across clusters based on usage patterns
• Deduplication across revisions: Share common pieces between model versions (e.g., shared tokenizer files)
• Cross-hub deduplication: When the same model exists on both Hugging Face and ModelScope, share pieces across download sources
• Integration with Kubernetes model serving frameworks: Native support in KServe, Triton Inference Server, and vLLM for P2P model loading
• Bandwidth-aware scheduling: Prioritize P2P transfers based on GPU node topology and network proximity

Contributing​

The PRs that brought these features to life:

• dragonflyoss/client#1665 — Rust implementation of the hf:// backend (Hugging Face)
• dragonflyoss/client#1673 — Rust implementation of the modelscope:// backend (ModelScope)
• dragonflyoss/d7y.io#386 — Documentation for the hf:// protocol in dfget
• dragonflyoss/d7y.io#396 — Documentation for the modelscope:// protocol in dfget

Dragonfly is a CNCF Graduated project and welcomes contributions. If you're working on AI infrastructure and have ideas for improving model distribution, check out the Dragonfly GitHub repository and join the community.

Conclusion​

The AI industry's model distribution problem doesn't need another wrapper script or another S3 bucket. It needs infrastructure-level P2P distribution with first-class understanding of where models live — whether that's Hugging Face, ModelScope, or the next model hub that emerges.

Dragonfly now speaks both hf:// and modelscope:// natively: authenticated, revision-aware, P2P-accelerated paths from the world's two largest model hubs to every node in your cluster. One origin fetch per hub. Peer-distributed propagation. No operational overhead.

The models are getting bigger. The clusters are getting larger. The hubs are multiplying. The distribution layer needs to keep up.

Now it can.

Links​

Dragonfly community​

• Website: https://d7y.io/
• Github Repo: https://github.com/dragonflyoss/dragonfly
• Slack Channel: #dragonfly on CNCF Slack
• Discussion Group: dragonfly-discuss@googlegroups.com
• Twitter: @dragonfly_oss

Hugging Face​

• Website: https://huggingface.co/
• Github Repo: https://github.com/huggingface/huggingface_hub
• Document: https://huggingface.co/docs

ModelScope​

• Website: https://modelscope.cn/
• Github Repo: https://github.com/modelscope/modelscope
• Document: https://modelscope.cn/docs

Key Highlights:

• Dragonfly graduates from CNCF after demonstrating production readiness and widespread adoption across container and AI workloads.
• Dragonfly is used by major organizations, including Ant Group, Alibaba, Datadog, DiDi, and Kuaishou to power large-scale container and AI model distribution.
• Since joining the CNCF, Dragonfly is backed by over a 3,000% growth in code contributions and a growing contributor community spanning over 130 companies.

SAN FRANCISCO, Calif. – January 14, 2026 – The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, today announced the graduation of Dragonfly, a cloud native open source image and file distribution system designed to solve cloud native image distribution in Kubernetes-centered applications.

“Dragonfly’s graduation reflects the project’s maturity, broad industry adoption and critical role in scaling cloud native infrastructure,” said Chris Aniszczyk, CTO, CNCF. “It’s especially exciting to see the project’s impact in accelerating image distribution and meeting the data demands of AI workloads. We’re proud to support a community that continues to push forward scalable, efficient and open solutions.”

Dragonfly’s Technical Capabilities​

Dragonfly delivers efficient, stable, and secure data distribution and acceleration powered by peer-to-peer (P2P) technology. It aims to provide a best‑practice, standards‑based solution for cloud native architectures to improve large‑scale delivery of files, container images, OCI artifacts, AI models, caches, logs, and dependencies.

Dragonfly runs on Kubernetes and is installed via Helm, with its official chart available on Artifact Hub. It also includes tools like Prometheus for tracking performance, OpenTelemetry for collecting and sharing data, and gRPC for rapid communication between parts. Enhancing Harbor capability to distribute images and OCI artifacts through the preheat feature. In the GenAI era, as model serving becomes increasingly important, Dragonfly delivers even more value by distributing AI model artifacts defined by the ModelPack specification.

Dragonfly continues to advance container image distribution, supporting tens of millions of container launches per day in production, saving storage bandwidth by up to 90%, and reducing launch time from minutes to seconds, with large-scale adoption across different cloud native scenarios.

Dragonfly is also driving standards and acceleration solutions for distributing both AI model weights and optimized image layout in AI workloads. The technology reduces data loading for large-scale AI applications and enables the distribution of model weights at a hundred-terabyte scale to hundreds of nodes in minutes. As AI continues to integrate into operations, Dragonfly becomes crucial to powering large-scale AI workloads.

Milestones Driving Graduation​

Dragonfly was open-sourced by Alibaba Group in November 2017. It then joined the CNCF as a Sandbox project in October 2018. During this stage, Dragonfly 1.0 became production-ready in November 2019 and the Dragonfly subproject, Nydus, was open-sourced in January 2020. Dragonfly then reached Incubation phase in April 2020, with Dragonfly 2.0 later released in 2021.

Since then, the community has significantly matured and attracted hundreds of contributors from organizations such as Ant Group, Alibaba Cloud, ByteDance, Kuaishou, Intel, Datadog, Zhipu AI, and more, who use Dragonfly to deliver efficient image and AI model distribution.

Since joining CNCF, contributors have increased by 500%, from 45 individuals across 5 companies to 271 individuals across over 130 companies. Commit activity has grown by over 3,000%, from roughly 800 to 26,000 commits, and the number of overall participants has reached 1,890.

What’s Next For Dragonfly​

Dragonfly will accelerate AI model weight distribution based on RDMA, improving throughput and reducing end-to-end latency. It will also optimize image layout to reduce data loading time for large-scale AI workloads. A load-aware two-phase scheduling will be introduced, leveraging collaboration between the scheduler and clients to enhance overall distribution efficiency. To provide more stable and reliable services, Dragonfly will support automatic updates and fault recovery, ensuring stable operation of all components during traffic bursts while controlling back-to-source traffic.

Dragonfly’s Graduation Process​

To officially graduate from Incubation status, the Dragonfly team enhanced the election policy, clarified the maintainer lifecycle, standardized the contribution process, defined the community ladder, and added community guidelines for subprojects. The graduation process is supported by CNCF’s Technical Oversight Committee (TOC) sponsors for Dragonfly, Karena Angell and Kevin Wang, who conducted a thorough technical due diligence with Dragonfly’s project maintainers.

Additionally, a third-party security audit of Dragonfly was conducted. The Dragonfly team along with the guidance of their TOC sponsors, completed both a self-assessment and a joint assessment with CNCF TAG Security, then collaborated with the Dragonfly security team on a threat model. After this, the team improved the project’s security policy.

Learn more about Dragonfly and join the community: https://d7y.io/

Supporting Quotes​

“I am thrilled, as the founder of Dragonfly, to announce its graduation from the CNCF. We are grateful to each and every open source contributor in the community, whose tenacity and commitment have enabled Dragonfly to reach its current state. Dragonfly was created to resolve Alibaba Group’s challenges with ultra-large-scale file distribution and was open-sourced in 2017. Looking back on this journey over the past eight years, every step has embodied the open source spirit and the tireless efforts of the many contributors. This graduation marks a new starting point for Dragonfly. I hope that the project will embark on a new journey, continue to explore more possibilities in the field of data distribution, and provide greater value!”

—Zuozheng Hu, founder of Dragonfly, emeritus maintainer

“I am delighted that Dragonfly is now a CNCF graduated project. This is a significant milestone, reflecting the maturity of the community, the trust of end users, and the reliability of the service. In the future, with the support of CNCF, the Dragonfly team will work together to drive the community’s sustainable growth and attract more contributors. Facing the challenges of large-scale model distribution and data distribution in the GenAI era, our team will continue to explore the future of data distribution within the cloud native ecosystem.”

—Wenbo Qi (Gaius), core Dragonfly maintainer

“Since open-sourcing in 2020, Nydus, alongside Dragonfly, has been validated at production scale. Dragonfly’s graduation is a key milestone for Nydus as a subproject, allowing the project to continue improving the image filesystem’s usability and performance. It will also allow us to further explore ecosystem standardization and AGI use cases that will advance the underlying infrastructure.”

—Song Yan, core Nydus maintainer

“The combination of Dragonfly and Nydus substantially shortens launch times for container images and AI models, enhancing system resilience and efficiency.”

—Jiang Liu, Nydus maintainer

“Thanks to the community’s collective efforts, Dragonfly has evolved from a tool for accelerating container images into a secure and stable distribution system widely adopted by many enterprises. Continuous improvements in usability and stability enable the project to support a variety of scenarios, including CI/CD, edge computing, and AI. New challenges are emerging for the distribution of model weights and data in the age of AI. Dragonfly is becoming a key infrastructure in mitigating these challenges. With the support of the CNCF, Dragonfly will continue to drive the future evolution of cloud native distribution technologies.”

—Yuan Yang, Dragonfly maintainer

TOC Sponsors​

“We’re grateful to the TOC members who dedicated significant time to the technical due diligence required for Dragonfly’s advancement, as well as the technical leads and community members who supported governance and technical reviews. We also thank the project maintainers for their openness and responsiveness throughout this process, and the end users who met with TOC and TAB members to share their experiences with Dragonfly. This level of collaboration is what helps ensure the strength and credibility of the CNCF ecosystem.”

—Karena Angell, chair, TOC, CNCF

“Dragonfly’s graduation is a testament to the project’s technical maturity and the community’s consistent focus on performance, reliability, and tangible impact. It’s been impressive to see Dragonfly evolve to meet the needs of large-scale production environments and AI workloads. Congratulations to the maintainers and contributors who’ve worked hard to reach this milestone.”

—Kevin Wang, vice chair, TOC, CNCF

Project End Users​

“Over the past few years, as part of Ant Group’s container infrastructure, the Dragonfly project has accelerated container image and code package delivery across several 10K-node Kubernetes clusters. It has significantly saved image transmission bandwidth, and the Nydus subproject has additionally helped us to reduce image pull time to near zero. The project also supports the delivery of large language models within our AI infrastructure. It is a great honor to have contributed to Dragonfly and to have shared our practices with the community.”

—Xu Wang, head of the container infra team, Ant Group, and co-launcher of Kata Containers Project

“Dragonfly has become a key infrastructure component of the container image and data distribution system for Alibaba’s large-scale distributed systems. In ultra-large-scale scenarios such as the Double 11 (Singles’ Day) shopping festival, Dragonfly has provided stable and efficient distribution capabilities and has improved the efficiency of system scaling and delivery. Facing the new technological challenges of the AI era, Dragonfly has played an important role in model data distribution and cache acceleration, helping us to build a more efficient, intelligent computing platform. We are happy to see Dragonfly graduate, which represents an enhancement in community maturity and validates its reliability in large-scale production environments.”

—Li Yi, director of engineering for container service, Alibaba Cloud & Tao Huang, director of engineering for cloud native transformation project, Alibaba Group

“Datadog recently adopted the Dragonfly subproject, Nydus, and it has helped significantly reduce time spent pulling images. This includes AI workloads, where image pulls previously took 5 minutes, and node daemonsets, which have startup speeds directly related to how quickly applications can be scheduled on nodes. We have seen significant improvements using Nydus, now everything starts in a matter of seconds. We are thrilled to see Dragonfly graduate and hope to continue to contribute to this impressive ecosystem!”

—Baptiste Girard-Carrabin, Datadog

“DiDi uses a distributed cloud platform to handle a large number of user requests and quickly adjust resources, which requires very efficient and stable management of resource distribution and image synchronization. Dragonfly is a core component of our technical architecture due to its strong cloud native adaptability, excellent P2P acceleration capabilities, and proven stability in large-scale scenarios. We believe that Dragonfly’s graduation is a strong testament to its technical maturity and industry value. We also look forward to its continued advancement in the field of cloud native distribution, providing more efficient solutions for large-scale file synchronization, image distribution, and other enterprise scenarios.”

—Feng Wu, head of the Elastic Cloud, DiDi & Rapier Yang, head of the Elastic Cloud, DiDi

“At Kuaishou, Dragonfly is considered the cornerstone of our container infrastructure, and it will soon provide stable and reliable image distribution capabilities for tens of thousands of services and hundreds of thousands of servers. Integrated with its subproject Nydus, Dragonfly dramatically enhances application startup efficiency while significantly alleviating disk I/O pressure—ensuring stability for services. In the era of AI large models, Dragonfly also functions as a critical component of our AI infrastructure, providing exceptional acceleration capabilities for large model distribution. We are deeply honored to partner with the vibrant Dragonfly community in collectively exploring future innovations for cloud native distribution technologies.”

—Wang Yao, head of container registry service of Kuaishou

About Cloud Native Computing Foundation​

Cloud native computing empowers organizations to build and run scalable applications with an open source software stack in public, private, and hybrid clouds. The Cloud Native Computing Foundation (CNCF) hosts critical components of the global technology infrastructure, including Kubernetes, Prometheus, and Envoy. CNCF brings together the industry’s top developers, end users, and vendors and runs the largest open source developer conferences in the world. Supported by nearly 800 members, including the world’s largest cloud computing and software companies, as well as over 200 innovative startups, CNCF is part of the nonprofit Linux Foundation. For more information, please visit www.cncf.io.

New features and enhancements​

load-aware scheduling algorithm​

A two-stage scheduling algorithm combining central scheduling with node-level secondary scheduling to optimize P2P download performance based on real-time load awareness.

For more information, please refer to the Scheduling.

Vortex Protocol Support for P2P File Transfer​

Dragonfly provide the new Vortex transfer protocol based on TLV to improve the download performance in internal network. Use the TLV (Tag-Length-Value) format as a lightweight protocol to replace gRPC for data transfer between peers. TCP-based Vortex reduces large file download time by 50% and QUIC-based Vortex by 40% compared to gRPC, both effectively reducing peak memory usage.

For more information, please refer to the TCP Protocol Support for P2P File Transfer and QUIC Protocol Support for P2P File Transfer.

Request SDK​

A SDK for routing User requests to Seed Peers using consistent hashing, replacing the previous Kubernetes Service load balancing approach.

For more details, please refer to Request SDK.

Simple Multi‑Cluster Kubernetes Deployment with Scheduler Cluster ID​

Dragonfly supports a simplified feature for deploying and managing multiple Kubernetes clusters by explicitly assigning a schedulerClusterID to each cluster. This approach allows users to directly control cluster affinity without relying on location‑based scheduling metadata such as IDC, hostname, or IP.

Using this feature, each Peer, Seed Peer, and Scheduler determines its target scheduler cluster through a clearly defined scheduler cluster ID. This ensures precise separation between clusters and predictable cross‑cluster behavior.

For more information, please refer to the Create Dragonfly Cluster Simple.

Performance and Resource Optimization for Manager and Scheduler Components​

Enhanced service performance and resource utilization across Manager and Scheduler components while significantly reducing CPU and memory overhead, delivering improved system efficiency and better resource management.

Enhanced Preheating​

• Support for IP-based peer selection in preheating jobs with priority-based selection logic where IP specification takes highest priority, followed by count-based and percentage-based selection

• Support for preheating multiple URLs in a single request.

• Support for preheating file and image via Scheduler gRPC interface.

Calculate task ID based on image blob SHA256 to avoid redundant downloads​

The Client now supports calculating task IDs directly from the SHA256 hash of image blobs instead of using the download URL. This enhancement prevents redundant downloads and data duplication when the same blob is accessed from different registry domains.

Cache HTTP 307 redirects for split downloads​

Support for caching HTTP 307 (Temporary Redirect) responses to optimize Dragonfly's multi-piece download performance. When a download URL is split into multiple pieces, the redirect target is now cached, eliminating redundant redirect requests and reducing latency.

Go Client Deprecated and Replaced by Rust Client​

The Go client has been deprecated and replaced by the Rust Client. All future development and maintenance will focus exclusively on the Rust client, which offers improved performance, stability, and reliability.

For more information, please refer to the dragonflyoss/client.

Additional Enhancements​

• Enable 64K page size support for ARM64 in the Dragonfly Rust client.
• Fix missing git commit metadata in dfget version output.
• Support for config_path of io.containerd.cri.v1.images plugin for containerd V3 configuration.
• Replaces glibc DNS resolver with hickory-dns in reqwest to implement DNS caching and prevent excessive DNS lookups during piece downloads.
• Support for the --include-files flag to selectively download files from a directory.
• Add the --no-progress flag to disable the download progress bar output.
• Support for custom request headers in backend operations, enabling flexible header configuration for HTTP requests.
• Refactored log output to reduce redundant logging and improve overall logging efficiency.

Significant bug fixes​

• Modified the database field type from text to longtext to support storing the information of preheating job.
• Fixed panic on repeated seed peer service stops during Scheduler shutdown.
• Fixed broker authentication failure when specifying the Redis password without setting a username.

Nydus​

New features and enhancements​

• Nydusd: Add CRC32 validation support for both RAFS V5 and V6 formats, enhancing data integrity verification.
• Nydusd: Support resending FUSE requests during nydusd restoration, improving daemon recovery reliability.
• Nydusd: Enhance VFS state saving mechanism for daemon hot upgrade and failover.
• Nydusify: Introduce Nydus-to-OCI reverse conversion capability, enabling seamless migration back to OCI format.
• Nydusify: Implement zero-disk transfer for image copy, significantly reducing local disk usage during copy operations.
• Snapshotter: Builtin blob.meta in bootstrap for blob fetch reliability for RAFS v6 image.

Significant bug fixes​

• Nydusd: Fix auth token fetching for access_token field in registry authentication.
• Nydusd: Add recursive inode/dentry invalidation for umount API.
• Nydus Image: Fix multiple issues in optimize subcommand and add backend configuration support.
• Snapshotter: Implement lazy parent recovery for proxy mode to handle missing parent snapshots.

Others​

You can see CHANGELOG for more details.

Links​

• Dragonfly Website: https://d7y.io/
• Dragonfly Repository: https://github.com/dragonflyoss/dragonfly
• Dragonfly Client Repository: https://github.com/dragonflyoss/client
• Dragonfly Console Repository: https://github.com/dragonflyoss/console
• Dragonfly Charts Repository: https://github.com/dragonflyoss/helm-charts
• Dragonfly Monitor Repository: https://github.com/dragonflyoss/monitoring

Dragonfly GitHub​

Title​

Empowering Dragonfly: A Lightweight and Scalable Distribution System for Large Models with High Concurrency

Link​

IEEE Xplore

Authors​

Zhou Xu, Dalian University of Technology Lizhen Zhou, Dalian University of Technology Zichuan Xu, Dalian University of Technology Wenbo Qi, Ant Group Jinjing Ma, Ant Group Song Yan, Ant Group Yuan Yang, Alibaba Cloud Min Huang, Dalian University of Technology Haomiao Jiang, Dalian University of Technology Qiufen Xia, Dalian University of Technology Guowei Wu, Dalian University of Technology

Background​

With the continuous evolution of Artificial Intelligence Generated Content (AIGC) technologies, the distribution of container images and large models has become a challenge. Traditional centralized registry often faces single-node bandwidth bottlenecks during peak concurrent downloads, leading to severe network congestion and excessively long download times. Conversely, while Content Delivery Networks (CDNs) or private links can mitigate some hotspot loads, they fail to fully leverage the idle bandwidth of cluster nodes and introduce additional overhead. Consequently, cloud-native applications and AI services urgently need a dynamic, efficient, scalable, and non-intrusive distribution system for large-scale images and models, compatible with mainstream formats like OCI spec.

Design​

Key Design 1: A Lightweight Network Measurement Mechanism​

• Network Latency Probing: Each peer proactively sends probes to a designated set of targets within the cluster, as assigned by the Scheduler, to measure end-to-end latency. This approach ensures efficient probing of all peers in the cluster while operating under constrained network resources.

• Network Bandwidth Prediction: We employ non-intrusive measurements by analyzing historical loading data to estimate network bandwidth with reasonable accuracy. When a peer initiates a model loading request, it pulls pieces of the model from other peers and reports the size and loading time of these pieces to the Scheduler. The Scheduler leverages historical data from successful loading operations to predict bandwidth.

Key Design 2: A Scalable Scheduling Framework​

• Separating Inference from Scheduling: To ensure the Scheduler has adequate resources for task scheduling, we separate inference services from scheduling operations.

• Real-Time Data Synchronization: To maintain data consistency across multiple schedulers, we store and continuously update end-to-end latency information for the entire network, as collected by the lightweight network measurement module.

Key Design 3: An Asynchronous Model Training and Inference​

• Asynchronous Model Training: Asynchronous training and inference are facilitated through collaboration between the Trainer and Triton. The Scheduler retrieves end-to-end latency and bandwidth predictions from Redis and sends them to the Trainer, which then initiates training and persists the updated model. Triton periodically polls for updates and loads the new model for inference in the subsequent cycle.

• Graph Learning Algorithm: This algorithm aggregates feature parameters from peers, modeling each sample as an interaction between a peer and its parent. It also incorporates information from neighboring peers to capture similarities within the cluster, thereby improving the accuracy of bandwidth predictions.

Summary​

This paper presents an efficient and scalable peer-to-peer (P2P) model distribution system that optimizes resource utilization and ensures data synchronization through a multi-layered architecture.

• First, it introduces a lightweight network measurement mechanism that probes latency and infers bandwidth to predict real-time network conditions.

• Second, it proposes a scalable scheduling framework that separates inference services from scheduling, enhancing both resource efficiency and system responsiveness.

• Finally, it designs the Trainer module, which integrates asynchronous model training and inference with graph learning algorithms to support incremental learning for bursty tasks.

GitHub Repository​

Features​

Persistent Cache Task​

It designs to provide persistent caching for tasks. This tool can import file and export file in P2P network. The solution is specifically engineered for high-speed read and write operations. This makes it particularly advantageous for scenarios involving large files, such as machine learning model checkpoints, where rapid, reliable access and distribution across the network are critical for training and inference workflows. By leveraging P2P distribution and persistent caching, dfcache significantly reduces I/O bottlenecks and accelerates the lifecycle of large data assets.

For documentation on how to use the dfcache command-line tool, please refer to the following link: dfcache.

$ dfcache import /tmp/file.txt
⣷ Done: 2229733261

dfcache export 2229733261 -O /tmp/file.txt

Resource Search (Tasks and Persistent Cache Tasks)​

The Resource Search feature enables seamless querying of tasks, including files, images, and persistent cache tasks. It optimizes resource access, improving task management and retrieval efficiency.

Vortex: A P2P File Transfer Protocol Based on TLV​

Vortex protocol is a high-performance peer-to-peer (P2P) file transfer protocol implementation in Rust, designed as part of the Dragonfly project. It utilizes the TLV (Tag-Length-Value) format for efficient and flexible data transmission, making it ideal for large-scale file distribution scenarios.

Packet Format:

• Packet Identifier (8 bits): Uniquely identifies each packet
• Tag (8 bits): Specifies data type in value field
• Length (32 bits): Indicates Value field length, up to 4 GiB
• Value (variable): Actual data content, maximum 1 GiB Protocol Format:

-------------------------------------------------------------------------------------------------
|                            |                   |                    |                         |
| Packet Identifier (8 bits) |    Tag (8 bits)   |  Length (32 bits)  |   Value (up to 4 GiB)   |
|                            |                   |                    |                         |
-------------------------------------------------------------------------------------------------

For more information, please refer to the Vortex Protocol.

Enhanced Large File Distribution​

This release significantly enhances Dragonfly's large file distribution capabilities, delivering improved efficiency and performance. We've revamped our scheduling algorithms for large file scenarios to ensure smarter resource and task allocation. Additionally, new mechanisms now more effectively balance the load across peers during large file transfers. Optimizations to the peer-to-peer (P2P) protocol and network transport layers further boost transmission efficiency.

These improvements include performance optimizations for both Client and Scheduler. You can find more details in the project's pull request.

Support scopes for Personal Access Tokens (PATs)​

By enabling users to define specific access rights (scopes) for each PAT, we significantly enhance the security of Open API interactions. Instead of granting broad permissions, PATs can now be limited to only the necessary privileges required for a particular integration or task.

Enhanced Preheating​

Implement Distributed Rate Limiting for Preheating Tasks​

By limiting the rate at which preheating requests are initiated across the distributed system, it prevents excessive preheating activities from stressing the origin. This enhancement ensures a more stable preheating.

Support to set piece length for preheating​

By allowing adjustment of the piece size, users can optimize data transfer efficiency, particularly in scenarios involving large files.

Flexible Preheating: Set Peer scope by Percentage or Count​

This feature enhances preheating capabilities by allowing users to specify the preheating scope more precisely.

Implement Audit Logging for User Operations​

This feature introduces comprehensive audit logging capabilities to track user operations within the system. Audit logs will record critical actions performed by users, such as initiating preheating tasks, deleting task caches, and other significant system interactions.

Garbage Collection​

Dragonfly supports Garbage Collection (GC) Audit Logs and GC Job Records to track and manage garbage collection activities. The Manager enables automated GC retention, allowing records to be preserved for a configurable time period. Additionally, it provides the capability to manually trigger forced GC operations as needed.

This feature ensures efficient monitoring and management of GC processes, offering flexibility through automated retention policies and manual intervention for immediate GC execution.

Optimized File Download with Hard Link​

File download needs to be done in a way that is efficient and secure. If users are downloading a large file, it is not efficient to download the file and copy to the output path. Instead, we can create a hard link to the file and send the link to the user. This way, we can avoid copying the file and save time and resources. If hard link fails (e.g. due to different file systems), dfdaemon will fall back to copying the file.

For more information, please refer to the file download workflow.

Hardware Acceleration for Piece Hash Computation​

This feature enables hardware-accelerated Piece hash computation, significantly boosting performance and efficiency. By utilizing specialized hardware, the hash computation process is accelerated, allowing faster processing of large file.

Advanced Storage Management​

Disk Space Validation for Operations​

This feature enhances the client's storage functionality by implementing disk space validation. When insufficient disk space is detected, the client will return a failure response, preventing potential data corruption or incomplete operations.

Disk Garbage Collection Management​

This feature enhances Peer's disk management by introducing configurable garbage collection (GC) thresholds based on disk usage. The distThreshold parameter allows users to define a specific disk capacity (e.g., 10TiB) as the base for calculating GC trigger points. If set, the distHighThresholdPercent (e.g., 80%) and distLowThresholdPercent (e.g., 60%) are applied relative to this capacity. If distThreshold is not provided or set to 0, these percentages are calculated based on the total actual disk space. When disk usage exceeds the high threshold, Dragonfly triggers GC(LRU) to reclaim space. GC stops when usage falls below the low threshold. This enables efficient management of a logical disk portion for caching, improving resource utilization and system performance.

Support for OpenTelemetry Tracing​

Dragonfly supports for tracing based on OpenTelemetry, covering the Manager, Scheduler, and Peers. This enables end-to-end visibility into the download process, allowing users to query detailed information, such as overall download latency, using a specific task ID. The integration ensures efficient monitoring and performance analysis across the entire system.

Add tracing configuration as follows(in Manager, Scheduler and Peer):

You can access the Jaeger UI to visualize the traces.

For more information, please refer to the Tracing.

Security Enhancements​

We extend our sincere gratitude to the CNCF TAG Security for their collaboration on a joint security audit. Their expertise and thorough review were invaluable in helping us identify areas for security improvement within Dragonfly. For detailed information on the specific security issues addressed and the corresponding fixes, please refer to the following issue: #3811

Nydus​

Significant bug fixes​

• Fixed memory leaks and file descriptor leaks caused by sysinfo library.
• Cleans up the Unix domain socket (UDS) to prevent dfdaemon startup crashes.
• Prevent client from repeatedly downloading the same piece from multiple parents.

Others​

You can see CHANGELOG for more details.

Links​

• Dragonfly Website: https://d7y.io/
• Dragonfly Repository: https://github.com/dragonflyoss/dragonfly
• Dragonfly Client Repository: https://github.com/dragonflyoss/client
• Dragonfly Console Repository: https://github.com/dragonflyoss/console
• Dragonfly Charts Repository: https://github.com/dragonflyoss/helm-charts
• Dragonfly Monitor Repository: https://github.com/dragonflyoss/monitoring

Dragonfly GitHub​

Dragonfly v2.2.0 is released! 🎉🎉🎉 Thanks the contributors who made this release happen and welcome you to visit d7y.io website.

Features​

Client written in Rust​

The client is written in Rust, offering advantages such as ensuring memory safety, improving performance, etc. The client is a submodule of Dragonfly, refer to dragonflyoss/client.

Client supports bandwidth rate limiting for prefetching​

Client now supports rate limiting for prefetch requests, which can prevent network overload and reduce competition with other active download tasks, thereby enhancing overall system performance. Refer to the documentation to configure the proxy.prefetchRateLimit option.

The following diagram illustrates the usage of download rate limit, upload rate limir, and prefetch rate limit for the client.

Client supports leeching​

If the user configures the client to disable sharing, it will become a leech.

Optimize client’s performance for handling a large number of small I/Os by Nydus​

• Add the X-Dragonfly-Prefetch HTTP header. If X-Dragonfly-Prefetch is set to true and it is a range request, the client will prefetch the entire task. This feature allows Nydus to control which requests need prefetching.
• The client’s HTTP proxy adds an independent cache to reduce requests to the gRPC server, thereby reducing request latency.
• Increase the memory cache size in RocksDB and enable prefix search for quickly searching piece metadata.
• Use the CRC-32-Castagnoli algorithm with hardware acceleration to reduce the hash calculation cost for piece content.
• Reuse the gRPC connections for downloading and optimize the download logic.

Defines the V2 of the P2P transfer protocol​

Define the V2 of the P2P transfer protocol to make it more standard, clearer, and better performing, refer to dragonflyoss/api.

Enhanced Harbor Integration with P2P Preheating​

Dragonfly improves its integration with Harbor v2.13 for preheating images, includes the following enhancements:

• Support for preheating multi architecture images.
• User can select the preheat scope for multi-granularity preheating. (Single Seed Peer, All Seed Peers, All Peers)
• User can specify the scheduler cluster ids for preheating images to the desired Dragonfly clusters.

Refer to documentation for more details.

Task Manager​

User can search all peers of cached task by task ID or download URL, and delete the cache on the selected peers, refer to the documentation.

Peer Manager​

Manager will regularly synchronize peers’ information and also allows for manual refreshes. Additionally, it will display peers’ information on the Manager Console.

Add hostname regexes and CIDRs to cluster scopes for matching clients​

When the client starts, it reports its hostname and IP to the Manager. The Manager then returns the best matching cluster (including schedulers and seed peers) to the client based on the cluster scopes configuration.

Supports distributed rate limiting for creating jobs across different clusters​

User can configure rate limiting for job creation across different clusters in the Manager Console.

Support preheating images using self-signed certificates​

Preheating requires calling the container registry to parse the image manifest and construct the URL for downloading blobs. If the container registry uses a self-signed certificate, user can configure the self-signed certificate in the Manager’s config for calling to the container registry.

Support mTLS for gRPC calls between services​

By setting self-signed certificates in the configurations of the Manager, Scheduler, Seed Peer, and Peer, gRPC calls between services will use mTLS.

Observability​

Dragonfly is recommending to use prometheus for monitoring. Prometheus and grafana configurations are maintained in the dragonflyoss/monitoring repository.

Grafana dashboards are listed below:

Nydus​

Nydus v2.3.0 is released, refer to Nydus Image Service v2.3.0 for more details.

• builder: support –parent-bootstrap for merge.
• builder/nydusd: support batch chunks mergence.
• nydusify/nydus-snapshotter: support OCI reference types.
• nydusify: support export/import for remote images.
• nydusify: support –push-chunk-size for large size image.
• nydusd/nydus-snapshotter: support basic failover and hot upgrade.
• nydusd: support overlay writable mount for fusedev.

Console​

Console v0.2.0 is released, featuring a redesigned UI and an improved interaction flow. Additionally, more functional pages have been added, such as preheating, task manager, PATs(Personal Access Tokens) manager, etc. Refer to the documentation for more details.

Document​

Refactor the website documentation to make Dragonfly simpler and more practical for users, refer to d7y.io.

Significant bug fixes​

The following content only highlights the significant bug fixes in this release.

• Fix the thread safety issue that occurs when constructing the DAG(Directed Acyclic Graph) during scheduling.
• Fix the memory leak caused by the OpenTelemetry library.
• Avoid hot reload when dynconfig refresh data from Manager.
• Prevent concurrent download requests from causing failures in state machine transitions.
• Use context.Background() to avoid stream cancel by dfdaemon.
• Fix the database performance issue caused by clearing expired jobs when there are too many job records.
• Reuse the gRPC connection pool to prevent redundant request construction.

AI Infrastructure​

Model Spec​

The Dragonfly community is collaboratively defining the OCI Model Specification. OCI Model Specification aims to provide a standard way to package, distribute and run AI models in a cloud native environment. The goal of this specification is to package models in an OCI artifact to take advantage of OCI distribution and ensure efficient model deployment, refer to CloudNativeAI/model-spec for more details.

Support accelerated distribution of AI models in Hugging Face Hub(Git LFS)​

Distribute large files downloaded via the Git LFS protocol through Dragonfly P2P, refer to the documentation.

Maintainers​

The community has added four new Maintainers, hoping to help more contributors participate in the community.

• Han Jiang: He works for Kuaishou and will focus on the engineering work for Dragonfly.
• Yuan Yang: He works for Alibaba Group and will focus on the engineering work for Dragonfly.

Other​

You can see CHANGELOG for more details.

Links​

• Dragonfly Website: https://d7y.io/
• Dragonfly Repository: https://github.com/dragonflyoss/dragonfly
• Dragonfly Client Repository: https://github.com/dragonflyoss/client
• Dragonfly Console Repository: https://github.com/dragonflyoss/console
• Dragonfly Charts Repository: https://github.com/dragonflyoss/helm-charts
• Dragonfly Monitor Repository: https://github.com/dragonflyoss/monitoring

Project post by Yufei Chen, Miao Hao, and Min Huang, Dragonfly project

This document will help you experience how to use dragonfly with TritonServe. During the downloading of models, the file size is large and there are many services downloading the files at the same time. The bandwidth of the storage will reach the limit and the download will be slow.

Dragonfly can be used to eliminate the bandwidth limit of the storage through P2P technology, thereby accelerating file downloading.

Installation​

By integrating Dragonfly Repository Agent into Triton, download traffic through Dragonfly to pull models stored in S3, OSS, GCS, and ABS, and register models in Triton. The Dragonfly Repository Agent is in the dragonfly-repository-agent repository.

Prerequisites​

Notice: Kind is recommended if no kubernetes cluster is available for testing.

Dragonfly Kubernetes Cluster Setup​

For detailed installation documentation, please refer to  quick-start-kubernetes.

Prepare Kubernetes Cluster​

Create kind multi-node cluster configuration file kind-config.yaml, configuration content is as follows:

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:  - role: control-plane  - role: worker  - role: worker

Create a kind multi-node cluster using the configuration file:

kind create cluster --config kind-config.yaml

Switch the context of kubectl to kind cluster:

kubectl config use-context kind-kind

Kind loads dragonfly image​

Pull dragonfly latest images:

docker pull dragonflyoss/scheduler:latest
docker pull dragonflyoss/manager:latest
docker pull dragonflyoss/dfdaemon:latest

Kind cluster loads dragonfly latest images:

kind load docker-image dragonflyoss/scheduler:latest
kind load docker-image dragonflyoss/manager:latest
kind load docker-image dragonflyoss/dfdaemon:latest

Create dragonfly cluster based on helm charts​

Create helm charts configuration file charts-config.yamland set dfdaemon.config.agents.regx to match the download path of the object storage. Example: add regx:.*models.* to match download request from object storage bucket models. Configuration content is as follows:

scheduler:
  image: dragonflyoss/scheduler
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
seedPeer:
  image: dragonflyoss/dfdaemon
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
dfdaemon:
  image: dragonflyoss/dfdaemon
  tag: latest
  metrics:
    enable: true
  config:    verbose: true
    pprofPort: 18066
    proxy:
      defaultFilter: 'Expires&Signature&ns'
      security:
        insecure: true
        cacert: ''
        cert: ''
        key: ''
      tcpListen:
        namespace: ''
        port: 65001
      registryMirror:
        url: https://index.docker.io
        insecure: true
        certs: []
        direct: false
      proxies:
        - regx: blobs/sha256.*
        # Proxy all http download requests of model bucket path.
        - regx: .*models.*manager:
  image: dragonflyoss/manager
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
jaeger:  enable: true

Create a dragonfly cluster using the configuration file:

helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install --wait --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f charts-config.yaml

Example output:

LAST DEPLOYED: Wed Nov 29 21:23:48 2023
NAMESPACE: dragonfly-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the scheduler address by running these commands:
  export SCHEDULER_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=scheduler" -o jsonpath={.items[0].metadata.name})
  export SCHEDULER_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $SCHEDULER_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  kubectl --namespace dragonfly-system port-forward $SCHEDULER_POD_NAME 8002:$SCHEDULER_CONTAINER_PORT
  echo "Visit http://127.0.0.1:8002 to use your scheduler"

2. Get the dfdaemon port by running these commands:
  export DFDAEMON_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=dfdaemon" -o jsonpath={.items[0].metadata.name})
  export DFDAEMON_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $DFDAEMON_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  You can use $DFDAEMON_CONTAINER_PORT as a proxy port in Node.

3. Configure runtime to use dragonfly:
  https://d7y.io/docs/getting-started/quick-start/kubernetes/

4. Get Jaeger query URL by running these commands:
  export JAEGER_QUERY_PORT=$(kubectl --namespace dragonfly-system get services dragonfly-jaeger-query -o jsonpath="{.spec.ports[0].port}")
  kubectl --namespace dragonfly-system port-forward service/dragonfly-jaeger-query 16686:$JAEGER_QUERY_PORT
  echo "Visit http://127.0.0.1:16686/search?limit=20&lookback=1h&maxDuration&minDuration&service=dragonfly to query download events"

Check that dragonfly is deployed successfully:

kubectl get pods -n dragonfly-systemNAME                                 READY   STATUS    RESTARTS       AGEdragonfly-dfdaemon-8qcpd             1/1     Running   4 (118s ago)   2m45sdragonfly-dfdaemon-qhkn8             1/1     Running   4 (108s ago)   2m45sdragonfly-jaeger-6c44dc44b9-dfjfv    1/1     Running   0              2m45sdragonfly-manager-549cd546b9-ps5tf   1/1     Running   0              2m45sdragonfly-mysql-0                    1/1     Running   0              2m45sdragonfly-redis-master-0             1/1     Running   0              2m45sdragonfly-redis-replicas-0           1/1     Running   0              2m45sdragonfly-redis-replicas-1           1/1     Running   0              2m7sdragonfly-redis-replicas-2           1/1     Running   0              101sdragonfly-scheduler-0                1/1     Running   0              2m45sdragonfly-seed-peer-0                1/1     Running   1 (52s ago)    2m45s

Expose the Proxy service port​

Create the dfstore.yaml configuration file to expose the port on which the Dragonfly Peer’s HTTP proxy listens. The default port is 65001 and settargetPort to 65001.

kind: ServiceapiVersion: v1metadata:  name: dfstorespec:  selector:    app: dragonfly    component: dfdaemon    release: dragonfly  ports:    - protocol: TCP      port: 65001      targetPort: 65001  type: NodePort

Create service:

kubectl --namespace dragonfly-system apply -f dfstore.yaml

Forward request to Dragonfly Peer’s HTTP proxy:

kubectl --namespace dragonfly-system port-forward service/dfstore 65001:65001

Install Dragonfly Repository Agent​

Set Dragonfly Repository Agent configuration​

Create the dragonfly_config.jsonconfiguration file, the configuration is as follows:

{
  "proxy": "http://127.0.0.1:65001",
  "header": {},
  "filter": [
    "X-Amz-Algorithm",
    "X-Amz-Credential",
    "X-Amz-Date",
    "X-Amz-Expires",
    "X-Amz-SignedHeaders",
    "X-Amz-Signature"
  ]
}

• proxy: The address of Dragonfly Peer’s HTTP Proxy.
• header: Adds a request header to the request.
• filter: Used to generate unique tasks and filter unnecessary query parameters in the URL.

In the filter of the configuration, set different values when using different object storage:

Set Model Repository configuration​

Create cloud_credential.json cloud storage credential, the configuration is as follows:

```json
{
  "gs": {
    "": "PATH_TO_GOOGLE_APPLICATION_CREDENTIALS",
    "gs://gcs-bucket-002": "PATH_TO_GOOGLE_APPLICATION_CREDENTIALS_2"
  },
  "s3": {
    "": {
      "secret_key": "AWS_SECRET_ACCESS_KEY",
      "key_id": "AWS_ACCESS_KEY_ID",
      "region": "AWS_DEFAULT_REGION",
      "session_token": "",
      "profile": ""
    },
    "s3://s3-bucket-002": {
      "secret_key": "AWS_SECRET_ACCESS_KEY_2",
      "key_id": "AWS_ACCESS_KEY_ID_2",
      "region": "AWS_DEFAULT_REGION_2",
      "session_token": "AWS_SESSION_TOKEN_2",
      "profile": "AWS_PROFILE_2"
    }
  },
  "as": {
    "": {
      "account_str": "AZURE_STORAGE_ACCOUNT",
      "account_key": "AZURE_STORAGE_KEY"
    },
    "as://Account-002/Container": {
      "account_str": "",
      "account_key": ""
    }
  }
}

In order to pull the model through Dragonfly, the model configuration file needs to be added following code in config.pbtxt file:

model_repository_agents{  agents [    {      name: "dragonfly",    }  ]}

The densenet_onnx example contains modified configuration and model file. Modified config.pbtxt such as:

name: "densenet_onnx"platform: "onnxruntime_onnx"max_batch_size : 0input [  {    name: "data_0"    data_type: TYPE_FP32    format: FORMAT_NCHW    dims: [ 3, 224, 224 ]    reshape { shape: [ 1, 3, 224, 224 ] }  }]output [  {    name: "fc6_1"    data_type: TYPE_FP32    dims: [ 1000 ]    reshape { shape: [ 1, 1000, 1, 1 ] }    label_filename: "densenet_labels.txt"  }]model_repository_agents{  agents [    {      name: "dragonfly",    }  ]}

Triton Server integrates Dragonfly Repository Agent plugin​

Install Triton Server with Docker​

Pull dragonflyoss/dragonfly-repository-agent image which is integrated Dragonfly Repository Agent plugin in Triton Server, refer to Dockerfile.

docker pull dragonflyoss/dragonfly-repository-agent:latest

Run the container and mount the configuration directory:

docker run --network host --rm \  -v ${path-to-config-dir}:/home/triton/ \  dragonflyoss/dragonfly-repository-agent:latest tritonserver \  --model-repository=${model-repository-path}

• path-to-config-dir: The files path of dragonfly_config.json&cloud_credential.json.
• model-repository-path: The path of remote model repository.

The correct output is as follows:

=============================== Triton Inference Server ===============================
successfully loaded 'densenet_onnx'
I1130 09:43:22.595672 1 server.cc:604]
+------------------+------------------------------------------------------------------------+
| Repository Agent | Path                                                                   |
+------------------+------------------------------------------------------------------------+
| dragonfly        | /opt/tritonserver/repoagents/dragonfly/libtritonrepoagent_dragonfly.so |
+------------------+------------------------------------------------------------------------+

I1130 09:43:22.596011 1 server.cc:631]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend     | Path                                                            | Config                                                                                                                                                        |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| pytorch     | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so         | {}                                                                                                                                                            |
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1130 09:43:22.596112 1 server.cc:674]
+---------------+---------+--------+
| Model         | Version | Status |
+---------------+---------+--------+
| densenet_onnx | 1       | READY  |
+---------------+---------+--------+

I1130 09:43:22.598318 1 metrics.cc:703] Collecting CPU metrics
I1130 09:43:22.599373 1 tritonserver.cc:2435]
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                                                                                           |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                                                                                          |
| server_version                   | 2.37.0                                                                                                                                                                                                          |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0]         | s3://192.168.36.128:9000/models                                                                                                                                                                                 |
| model_control_mode               | MODE_NONE                                                                                                                                                                                                       |
| strict_model_config              | 0                                                                                                                                                                                                               |
| rate_limit                       | OFF                                                                                                                                                                                                             |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                                       |
| min_supported_compute_capability | 6.0                                                                                                                                                                                                             |
| strict_readiness                 | 1                                                                                                                                                                                                               |
| exit_timeout                     | 30                                                                                                                                                                                                              |
| cache_enabled                    | 0                                                                                                                                                                                                               |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1130 09:43:22.610334 1 grpc_server.cc:2451] Started GRPCInferenceService at 0.0.0.0:8001
I1130 09:43:22.612623 1 http_server.cc:3558] Started HTTPService at 0.0.0.0:8000
I1130 09:43:22.695843 1 http_server.cc:187] Started Metrics Service at 0.0.0.0:8002

Execute the following command to check the Dragonfly logs:

kubectl exec -it -n dragonfly-system dragonfly-dfdaemon-<id> -- tail -f /var/log/dragonfly/daemon/core.log

Check downloaded successfully through Dragonfly:

{
  "level": "info",
  "ts": "2024-02-02 05:28:02.631",
  "caller": "peer/peertask_conductor.go:1349",
  "msg": "peer task done, cost: 352ms",
  "peer": "10.244.2.3-1-4398a429-d780-423a-a630-57d765f1ccfc",
  "task": "974aaf56d4877cc65888a4736340fb1d8fecc93eadf7507f531f9fae650f1b4d",
  "component": "PeerTask",
  "trace": "4cca9ce80dbf5a445d321cec593aee65"
}

Verify​

Call inference API：

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.08-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

Check the response successful:

Request 01Image '/workspace/images/mug.jpg':    15.349563 (504) = COFFEE MUG    13.227461 (968) = CUP    10.424893 (505) = COFFEEPOT

Performance testing​

Test the performance of single-machine model download by Triton API after the integration of Dragonfly P2P. Due to the influence of the network environment of the machine itself, the actual download time is not important, but The proportion of download speed in different scenarios is more meaningful:

• Triton API: Use signed URL provided by Object Storage to download the model directly.
• Triton API & Dragonfly Cold Boot: Use Triton Serve API to download model via Dragonfly P2P network and no cache hits.
• Hit Remote Peer: Use Triton Serve API to download model via Dragonfly P2P network and hit the remote peer cache.
• Hit Local Peer: Use Triton Serve API to download model via Dragonfly P2P network and hit the local peer cache.

Test results show Triton and Dragonfly integration. It can effectively reduce the file download time. Note that this test was a single-machine test, which means that in the case of cache hits, the performance limitation is on the disk. If Dragonfly is deployed on multiple machines for P2P download, the models download speed will be faster.

Resources​

Dragonfly Community​

• Website: https://d7y.io/
• GitHub Repo: https://github.com/dragonflyoss/dragonfly
• Dragonfly Repository Agent GitHub Repo: https://github.com/dragonflyoss/dragonfly-repository-agent
• Slack Channel: #dragonfly on CNCF Slack
• Discussion Group: dragonfly-discuss@googlegroups.com
• Twitter: @dragonfly_oss

NVIDIA Triton Inference Server​

• Website: https://developer.nvidia.com/triton-inference-server
• GitHub Repo: https://github.com/triton-inference-server/server

What is Git LFS?​

Git LFS (Large File Storage) is an open-source extension for Git that enables users to handle large files more efficiently in Git repositories. Git is a version control system designed primarily for text files such as source code and it can become less efficient when dealing with large binary files like audio, videos, datasets, graphics and other large assets. These files can significantly increase the size of a repository and make cloning and fetching operations slow.

Git LFS addresses this issue by storing these large files on a separate server and replacing them in the Git repository with small placeholder files (pointers). When a user clones or pulls from the repository, Git LFS fetches the large files from the LFS server as needed rather than downloading all the large files with the initial clone of the repository. For specifications, please refer to the Git LFS Specification. The server is implemented based on the HTTP protocol, refer to Git LFS API. Usually Git LFS’s content storage uses object storage to store large files.

Git LFS Usage​

Git LFS manages large files​

GitHub and GitLab usually manage large files based on Git LFS.

• GitHub uses Git LFS refer to About Git Large File Storage.
• GitLab uses Git LFS refer to Git Large File Storage.

Git LFS manages AI models and AI datasets​

Large files of models and datasets in AI are usually managed based on Git LFS. Hugging Face Hub and ModelScope Hub manage models and datasets based on Git LFS.

• Hugging Face Hub uses Git LFS refer to Getting Started with Repositories.
• ModelScope Hub uses Git LFS refer to Getting Started with ModelScope.

Hugging Face Hub’s Python Library implements Git LFS to download models and datasets. Hugging Face Hub’s Python Library distributes models and datasets to accelerate, refer to Hugging Face accelerates distribution of models and datasets based on Dragonfly.

Dragonfly eliminates the bandwidth limit of Git LFS’s content storage​

This document will help you experience how to use dragonfly with Git LFS. During the downloading of large files, the file size is large and there are many services downloading the larges files at the same time. The bandwidth of the storage will reach the limit and the download will be slow.

Dragonfly can be used to eliminate the bandwidth limit of the storage through P2P technology, thereby accelerating large files downloading.

Dragonfly accelerates downloads with Git LFS​

By proxying the HTTP protocol file download request of Git LFS to Dragonfly Peer Proxy, the file download traffic is forwarded to the P2P network. The following documentation is based on GitHub LFS.

Get the Content Storage address of Git LFS​

Add GIT_CURL_VERBOSE=1 to print verbose logs of git clone and get the address of content storage of Git LFS.

GIT_CURL_VERBOSE=1 git clone git@github.com:{YOUR-USERNAME}/{YOUR-REPOSITORY}.git

Look for the trace git-lfs keyword in the logs and you can see the log of Git LFS download files. Pay attention to the content of actions and download in the log.

15:31:04.848308 trace git-lfs: HTTP: {"objects":[{"oid":"c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29","size":5242880,"actions":{"download":{"href":"https://github-cloud.githubusercontent.com/alambic/media/376919987/c0/36/c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20231221%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231221T073104Z&X-Amz-Expires=3600&X-Amz-Signature=4dc757dff0ac96eac3f0cd2eb29ca887035d3a6afba41cb10200ed0aa22812fa&15:31:04.848403 trace git-lfs: HTTP: X-Amz-SignedHeaders=host&actor_id=15955374&key_id=0&repo_id=392935134&token=1","expires_at":"2023-12-21T08:31:04Z","expires_in":3600}}}]}

The download URL can be found in actions.download.href in the objects. You can find that the content storage of GitHub LFS is actually stored at github-cloud.githubusercontent.com. And query parameters include X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-Signature and X-Amz-SignedHeaders. The query parameters are AWS Authenticating Requests parameters. The keys of query parameters will be used later when configuring Dragonfly Peer Proxy.

Information about Git LFS :

1. The content storage address of Git LFS is github-cloud.githubusercontent.com.
2. The query parameters of the download URL include X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-Signature and X-Amz-SignedHeaders.

Installation​

Prerequisites​

Notice: Kind is recommended if no kubernetes cluster is available for testing.

Install dragonfly​

For detailed installation documentation based on kubernetes cluster, please refer to quick-start-kubernetes.

Setup kubernetes cluster​

Create kind multi-node cluster configuration file kind-config.yaml, configuration content is as follows:

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:  - role: control-plane  - role: worker    extraPortMappings:      - containerPort: 30950        hostPort: 65001  - role: worker

Create a kind multi-node cluster using the configuration file:

kind create cluster --config kind-config.yaml

Switch the context of kubectl to kind cluster:

kubectl config use-context kind-kind

Kind loads dragonfly image​

Pull dragonfly latest images:

docker pull dragonflyoss/scheduler:latest
docker pull dragonflyoss/manager:latest
docker pull dragonflyoss/dfdaemon:latest

Kind cluster loads dragonfly latest images:

kind load docker-image dragonflyoss/scheduler:latest
kind load docker-image dragonflyoss/manager:latest
kind load docker-image dragonflyoss/dfdaemon:latest

Create dragonfly cluster based on helm charts​

Create helm charts configuration file charts-config.yaml. Add the github-cloud.githubusercontent.com rule to dfdaemon.config.proxy.proxies.regx to forward the HTTP file download of content storage of Git LFS to the P2P network. And dfdaemon.config.proxy.defaultFilter adds X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-Signature and X-Amz-SignedHeaders parameters to filter the query parameters. Dragonfly generates a unique task id based on the URL, so it is necessary to filter the query parameters to generate a unique task id. Configuration content is as follows:

scheduler:
  image: dragonflyoss/scheduler
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

seedPeer:
  image: dragonflyoss/dfdaemon
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

dfdaemon:
  image: dragonflyoss/dfdaemon
  tag: latest
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    proxy:
      defaultFilter: 'X-Amz-Algorithm&X-Amz-Credential&X-Amz-Date&X-Amz-Expires&X-Amz-Signature&X-Amz-SignedHeaders'
      security:
        insecure: true
        cacert: ''
        cert: ''
        key: ''
      tcpListen:
        namespace: ''
        port: 65001
      registryMirror:
        url: https://index.docker.io
        insecure: true
        certs: []
        direct: false
      proxies:
        - regx: blobs/sha256.*
        - regx: github-cloud.githubusercontent.com.*

manager:
  image: dragonflyoss/manager
  tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

jaeger:
  enable: true

Create a dragonfly cluster using the configuration file:

helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install --wait --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f charts-config.yaml

Output:

NAME: dragonfly
LAST DEPLOYED: Thu Dec 21 17:24:37 2023
NAMESPACE: dragonfly-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the scheduler address by running these commands:
  export SCHEDULER_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=scheduler" -o jsonpath={.items[0].metadata.name})
  export SCHEDULER_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $SCHEDULER_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  kubectl --namespace dragonfly-system port-forward $SCHEDULER_POD_NAME 8002:$SCHEDULER_CONTAINER_PORT
  echo "Visit http://127.0.0.1:8002 to use your scheduler"

2. Get the dfdaemon port by running these commands:
  export DFDAEMON_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=dfdaemon" -o jsonpath={.items[0].metadata.name})
  export DFDAEMON_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $DFDAEMON_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  You can use $DFDAEMON_CONTAINER_PORT as a proxy port in Node.

3. Configure runtime to use dragonfly:
  https://d7y.io/docs/getting-started/quick-start/kubernetes/

4. Get Jaeger query URL by running these commands:
  export JAEGER_QUERY_PORT=$(kubectl --namespace dragonfly-system get services dragonfly-jaeger-query -o jsonpath="{.spec.ports[0].port}")
  kubectl --namespace dragonfly-system port-forward service/dragonfly-jaeger-query 16686:$JAEGER_QUERY_PORT
  echo "Visit http://127.0.0.1:16686/search?limit=20&lookback=1h&maxDuration&minDuration&service=dragonfly to query download events"

Check that dragonfly is deployed successfully:

kubectl get po -n dragonfly-systemNAME                                 READY   STATUS    RESTARTS       AGEdragonfly-dfdaemon-cttxz             1/1     Running   4 (116s ago)   2m51sdragonfly-dfdaemon-k62vd             1/1     Running   4 (117s ago)   2m51sdragonfly-jaeger-84dbfd5b56-mxpfs    1/1     Running   0              2m51sdragonfly-manager-5c598d5754-fd9tf   1/1     Running   0              2m51sdragonfly-mysql-0                    1/1     Running   0              2m51sdragonfly-redis-master-0             1/1     Running   0              2m51sdragonfly-redis-replicas-0           1/1     Running   0              2m51sdragonfly-redis-replicas-1           1/1     Running   0              106sdragonfly-redis-replicas-2           1/1     Running   0              78sdragonfly-scheduler-0                1/1     Running   0              2m51sdragonfly-seed-peer-0                1/1     Running   1 (37s ago)    2m51s

Create peer service configuration file peer-service-config.yaml, configuration content is as follows:

apiVersion: v1kind: Servicemetadata:  name: peer  namespace: dragonfly-systemspec:  type: NodePort  ports:    - name: http-65001      nodePort: 30950      port: 65001  selector:    app: dragonfly    component: dfdaemon    release: dragonfly

Create a peer service using the configuration file:

kubectl apply -f peer-service-config.yaml

Git LFS downloads large files via dragonfly​

Proxy Git LFS download requests to Dragonfly Peer Proxy(http://127.0.0.1:65001) through Git configuration. Set Git configuration includes http.proxy, lfs.transfer.enablehrefrewrite and url.{YOUR-LFS-CONTENT-STORAGE}.insteadOf properties.

git config --global http.proxy http://127.0.0.1:65001git config --global lfs.transfer.enablehrefrewrite truegit config --global url.http://github-cloud.githubusercontent.com/.insteadOf https://github-cloud.githubusercontent.com/

Forward Git LFS download requests to the P2P network via Dragonfly Peer Proxy and Git clone the large files.

git clone git@github.com:{YOUR-USERNAME}/{YOUR-REPOSITORY}.git

Verify large files download with Dragonfly​

Execute the command:

# find podskubectl -n dragonfly-system get pod -l component=dfdaemon# find logspod_name=dfdaemon-xxxxxkubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/log/dragonfly/daemon/core.log

Example output:

2023-12-21T16:55:20.495+0800INFOpeer/peertask_conductor.go:1326peer task done, cost: 2238ms{"peer": "30.54.146.131-15874-f6729352-950e-412f-b876-0e5c8e3232b1", "task": "70c644474b6c986e3af27d742d3602469e88f8956956817f9f67082c6967dc1a", "component": "PeerTask", "trace": "35c801b7dac36eeb0ea43a58d1c82e77"}

Performance testing​

Test the performance of single-machine large files download after the integration of Git LFS and Dragonfly P2P. Due to the influence of the network environment of the machine itself, the actual download time is not important, but the ratio of the increase in the download time in different scenarios is very important.

• Git LFS: Use Git LFS to download large files directly.
• Git LFS & Dragonfly Cold Boot: Use Git LFS to download large files via Dragonfly P2P network and no cache hits.
• Hit Dragonfly Remote Peer Cache: Use Git LFS to download large files via Dragonfly P2P network and hit the remote peer cache.
• Hit Dragonfly Remote Local Cache: Use Git LFS to download large files via Dragonfly P2P network and hit the local peer cache.

Test results show Git LFS and Dragonfly P2P integration. It can effectively reduce the file download time. Note that this test was a single-machine test, which means that in the case of cache hits, the performance limitation is on the disk. If Dragonfly is deployed on multiple machines for P2P download, the large files download speed will be faster.

Links​

Dragonfly community​

• Website: https://d7y.io/
• GitHub Repo: https://github.com/dragonflyoss/dragonfly
• Slack Channel: #dragonfly on CNCF Slack
• Discussion Group: dragonfly-discuss@googlegroups.com
• Twitter: @dragonfly_oss

Git LFS​

• Website: https://git-lfs.com/
• GitHub Repo: https://github.com/git-lfs/git-lfs
• Document: https://github.com/git-lfs/git-lfs/tree/main/docs

This document will help you experience how to use dragonfly with TorchServe. During the downloading of models, the file size is large and there are many services downloading the files at the same time. The bandwidth of the storage will reach the limit and the download will be slow.

Dragonfly can be used to eliminate the bandwidth limit of the storage through P2P technology, thereby accelerating file downloading.

Architecture​

Dragonfly Endpoint plugin forwards TorchServe download model requests to the Dragonfly P2P network.

The models download steps:

1. TorchServe sends a model download request and the request is forwarded to the Dragonfly Peer.
2. The Dragonfly Peer registers tasks with the Dragonfly Scheduler.
3. Return the candidate parents to Dragonfly Peer.
4. Dragonfly Peer downloads model from candidate parents.
5. After downloading the model, TorchServe will register the model.

Installation​

By integrating Dragonfly Endpoint into TorchServe, download traffic through Dragonfly to pull models stored in S3, OSS, GCS, and ABS, and register models in TorchServe. The Dragonfly Endpoint plugin is in the dragonfly-endpoint repository.

Prerequisites​

Notice: Kind is recommended if no kubernetes cluster is available for testing.

Dragonfly Kubernetes Cluster Setup​

For detailed installation documentation, please refer to  quick-start-kubernetes.

Prepare Kubernetes Cluster​

Create kind multi-node cluster configuration file kind-config.yaml, configuration content is as follows:

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:  - role: control-plane  - role: worker  - role: worker

Create a kind multi-node cluster using the configuration file:

kind create cluster --config kind-config.yaml

Switch the context of kubectl to kind cluster:

kubectl config use-context kind-kind

Kind loads dragonfly image​

Pull dragonfly latest images:

docker pull dragonflyoss/scheduler:latest
docker pull dragonflyoss/manager:latest
docker pull dragonflyoss/dfdaemon:latest

Kind cluster loads dragonfly latest images:

kind load docker-image dragonflyoss/scheduler:latest
kind load docker-image dragonflyoss/manager:latest
kind load docker-image dragonflyoss/dfdaemon:latest

Create dragonfly cluster based on helm charts​

Create helm charts configuration file charts-config.yaml and set dfdaemon.config.agents.regx to match the download path of the object storage, configuration content is as follows:

scheduler:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

seedPeer:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

dfdaemon:
  hostNetwork: true
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    proxy:
      defaultFilter: 'Expires&Signature&ns'
      security:
        insecure: true
        cacert: ''
        cert: ''
        key: ''
      tcpListen:
        namespace: ''
        port: 65001
      registryMirror:
        url: https://index.docker.io
        insecure: true
        certs: []
        direct: false
      proxies:
        - regx: blobs/sha256.*
        - regx: .*amazonaws.*

manager:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

jaeger:
  enable: true

Create a dragonfly cluster using the configuration file:

$ helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
$ helm install --wait --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f charts-config.yaml
LAST DEPLOYED: Mon Sep  4 10:24:55 2023
NAMESPACE: dragonfly-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the scheduler address by running these commands:
  export SCHEDULER_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=scheduler" -o jsonpath={.items[0].metadata.name})
  export SCHEDULER_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $SCHEDULER_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  kubectl --namespace dragonfly-system port-forward $SCHEDULER_POD_NAME 8002:$SCHEDULER_CONTAINER_PORT
  echo "Visit http://127.0.0.1:8002 to use your scheduler"
2. Get the dfdaemon port by running these commands:
  export DFDAEMON_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=dfdaemon" -o jsonpath={.items[0].metadata.name})
  export DFDAEMON_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $DFDAEMON_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  You can use $DFDAEMON_CONTAINER_PORT as a proxy port in Node.
3. Configure runtime to use dragonfly:
  https://d7y.io/docs/getting-started/quick-start/kubernetes/
4. Get Jaeger query URL by running these commands:
  export JAEGER_QUERY_PORT=$(kubectl --namespace dragonfly-system get services dragonfly-jaeger-query -o jsonpath="{.spec.ports[0].port}")
  kubectl --namespace dragonfly-system port-forward service/dragonfly-jaeger-query 16686:$JAEGER_QUERY_PORT
  echo "Visit http://127.0.0.1:16686/search?limit=20&lookback=1h&maxDuration&minDuration&service=dragonfly to query download events"

Check that dragonfly is deployed successfully:

$ kubectl get po -n dragonfly-system
NAME                                 READY   STATUS    RESTARTS      AGE
dragonfly-dfdaemon-7r2cn             1/1     Running   0          3m31s
dragonfly-dfdaemon-fktl4             1/1     Running   0          3m31s
dragonfly-jaeger-c7947b579-2xk44     1/1     Running   0          3m31s
dragonfly-manager-5d4f444c6c-wq8d8   1/1     Running   0          3m31s
dragonfly-mysql-0                    1/1     Running   0          3m31s
dragonfly-redis-master-0             1/1     Running   0          3m31s
dragonfly-redis-replicas-0           1/1     Running   0          3m31s
dragonfly-redis-replicas-1           1/1     Running   0          3m5s
dragonfly-redis-replicas-2           1/1     Running   0          2m44s
dragonfly-scheduler-0                1/1     Running   0          3m31s
dragonfly-seed-peer-0                1/1     Running   0          3m31s

Expose the Proxy service port​

Create the dfstore.yaml configuration to expose the port on which the Dragonfly Peer’s HTTP proxy listens. The default port is 65001 and settargetPort to 65001.

kind: Service
apiVersion: v1
metadata:
  name: dfstore
spec:
  selector:
    app: dragonfly
    component: dfdaemon
    release: dragonfly
  ports:
    - protocol: TCP
      port: 65001
      targetPort: 65001
  type: NodePort

Create service:

kubectl --namespace dragonfly-system apply -f dfstore.yaml

Forward request to Dragonfly Peer’s HTTP proxy:

kubectl --namespace dragonfly-system port-forward service/dfstore 65001:65001

Install Dragonfly Endpoint plugin​

Set environment variables for Dragonfly Endpoint configuration​

Create config.json configuration，and set DRAGONFLY_ENDPOINT_CONFIG environment variable for config.json file path.

export DRAGONFLY_ENDPOINT_CONFIG=/etc/dragonfly-endpoint/config.json

The default configuration path is:

• linux: /etc/dragonfly-endpoint/config.json
• darwin: ~/.dragonfly-endpoint/config.json

Dragonfly Endpoint configuration​

Create the config.json configuration to configure the Dragonfly Endpoint for S3, the configuration is as follows:

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "filter": [
    "X-Amz-Algorithm",
    "X-Amz-Credential",
    "X-Amz-Date",
    "X-Amz-Expires",
    "X-Amz-SignedHeaders",
    "X-Amz-Signature"
  ],
  "object_storage": {
    "type": "s3",
    "bucket_name": "your_s3_bucket_name",
    "region": "your_s3_region",
    "access_key": "your_s3_access_key",
    "secret_key": "your_s3_secret_key"
  }
}

In the filter of the configuration, set different values when using different object storage:

In the filter of the configuration, set different values when using different object storage:

Object storage configuration​

In addition to S3, Dragonfly Endpoint plugin also supports OSS, GCS and ABS. Different object storage configurations are as follows:

OSS(Object Storage Service)

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "filter": ["Expires", "Signature"],
  "object_storage": {
    "type": "oss",
    "bucket_name": "your_oss_bucket_name",
    "endpoint": "your_oss_endpoint",
    "access_key_id": "your_oss_access_key_id",
    "access_key_secret": "your_oss_access_key_secret"
  }
}

GCS(Google Cloud Storage)

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "object_storage": {
    "type": "gcs",
    "bucket_name": "your_gcs_bucket_name",
    "project_id": "your_gcs_project_id",
    "service_account_path": "your_gcs_service_account_path"
  }
}

ABS(Azure Blob Storage)

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "object_storage": {
    "type": "abs",
    "account_name": "your_abs_account_name",
    "account_key": "your_abs_account_key",
    "container_name": "your_abs_container_name"
  }
}

TorchServe integrates Dragonfly Endpoint plugin​

For detailed installation documentation, please refer to TorchServe document.

Binary installation​

The Prerequisites​

Install TorchServe dependencies and torch-model-archiver：

python ./ts_scripts/install_dependencies.py
conda install torchserve torch-model-archiver torch-workflow-archiver -c pytorch

Clone TorchServe repository：

git clone https://github.com/pytorch/serve.gitcd serve

Create model-store directory to store the models：

mkdir model-storechmod 777 model-store

Create plugins-path directory to store the binaries of the plugin：

mkdir plugins-path

Package Dragonfly Endpoint plugin​

Clone dragonfly-endpoint repository：

git clone https://github.com/dragonflyoss/dragonfly-endpoint.git

Build the dragonfly-endpoint project to generate file in the build/libs directory:

cd ./dragonfly-endpointgradle shadowJar

Note: Due to the limitations of TorchServe’s JVM, the best Java version for Gradle is 11, as a higher version will cause the plugin to fail to parse.

Move the Jar file into the plugins-path directory:

mv build/libs/dragonfly_endpoint-1.0-all.jar  <your plugins-path>

Prepare the plugin configuration config.json, and use S3 as the object storage:

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "filter": [
    "X-Amz-Algorithm",
    "X-Amz-Credential",
    "X-Amz-Date",
    "X-Amz-Expires",
    "X-Amz-SignedHeaders",
    "X-Amz-Signature"
  ],
  "object_storage": {
    "type": "s3",
    "bucket_name": "your_s3_bucket_name",
    "region": "your_s3_region",
    "access_key": "your_s3_access_key",
    "secret_key": "your_s3_secret_key"
  }
}

Set the environment variables for the configuration:

export DRAGONFLY_ENDPOINT_CONFIG=/etc/dragonfly-endpoint/config.json

–model-storesets the previously created directory to store the models and –plugins-path sets the previously created directory to store the plugins. Start the TorchServe with Dragonfly Endpoint plugin:

torchserve --start --model-store <path-to-model-store-file> --plugins-path=<path-to-plugin-jars>

Verify​

Prepare the model. Download a model from Model ZOO or package the model refer to Torch Model archiver for TorchServe. Use squeezenet1_1_scripted.mar model to verify：

wget https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar

Upload the model to object storage. For detailed uploading the model to S3, please refer to S3。

# Download the command line tool
pip install awscli
# Configure the key as prompted
aws configure
# Upload file
aws s3 cp < local file path > s3://< bucket name >/< Target path >

TorchServe plugin is named dragonfly, please refer to TorchServe Register API for details of plugin API. The url parameter are not supported and add the file_name parameter which is the model file name to download.

Download the model:

curl -X POST  "http://localhost:8081/dragonfly/models?file_name=squeezenet1_1.mar"

Verify the model download successful:

{
  "Status": "Model \"squeezenet1_1\" Version: 1.0 registered with 0 initial workers. Use scale workers API to add workers for the model."
}

Added model worker for inference:

curl -v -X PUT "http://localhost:8081/models/squeezenet1_1?min_worker=1"

Check the number of worker is increased:

* About to connect() to localhost port 8081 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8081 (#0)
> PUT /models/squeezenet1_1?min_worker=1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 202 Accepted
< content-type: application/json
< x-request-id: 66761b5a-54a7-4626-9aa4-12041e0e4e63
< Pragma: no-cache
< Cache-Control: no-cache; no-store, must-revalidate, private
< Expires: Thu, 01 Jan 1970 00:00:00 UTC
< content-length: 47
< connection: keep-alive
<
{  "status": "Processing worker updates..."}
* Connection #0 to host localhost left intact

Call inference API:

# Prepare pictures that require reasoning
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/kitten_small.jpg
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/dogs-before.jpg

# Call inference API
curl http://localhost:8080/predictions/squeezenet1_1 -T kitten_small.jpg -T dogs-before.jpg

Check the response successful:

{
  "lynx": 0.5455784201622009,
  "tabby": 0.2794168293476105,
  "Egyptian_cat": 0.10391931980848312,
  "tiger_cat": 0.062633216381073,
  "leopard": 0.005019133910536766
}

Install TorchServe with Docker​

Docker configuration​

Pull dragonflyoss/dragonfly-endpoint image with the plugin. The following is an example of the CPU version of TorchServe, refer to Dockerfile.

docker pull dragonflyoss/dragonfly-endpoint

Create model-store directory to store the model files：

mkdir model-storechmod 777 model-store

Prepare the plugin configuration config.json, and use S3 as the object storage:

{
  "addr": "http://127.0.0.1:65001",
  "header": {},
  "filter": [
    "X-Amz-Algorithm",
    "X-Amz-Credential",
    "X-Amz-Date",
    "X-Amz-Expires",
    "X-Amz-SignedHeaders",
    "X-Amz-Signature"
  ],
  "object_storage": {
    "type": "s3",
    "bucket_name": "your_s3_bucket_name",
    "region": "your_s3_region",
    "access_key": "your_s3_access_key",
    "secret_key": "your_s3_secret_key"
  }
}

Set the environment variables for the configuration:

export DRAGONFLY_ENDPOINT_CONFIG=/etc/dragonfly-endpoint/config.json

Mount the model-store and dragonfly-endpoint configuration directory. Run the container:

sudo docker run --rm -it --network host \
  -v $(pwd)/model-store:/home/model-server/model-store \
  -v ${DRAGONFLY_ENDPOINT_CONFIG}:${DRAGONFLY_ENDPOINT_CONFIG} \
  dragonflyoss/dragonfly-endpoint:latest

How to Verify​

Prepare the model. Download a model from Model ZOO or package the model refer to Torch Model archiver for TorchServe. Use squeezenet1_1_scripted.mar model to verify：

wget https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar

Upload the model to object storage. For detailed uploading the model to S3, please refer to S3。

# Download the command line tool
pip install awscli

# Configure the key as prompted
aws configure

# Upload file
aws s3 cp <local file path> s3://<bucket name>/<Target path>

TorchServe plugin is named dragonfly, please refer to TorchServe Register API for details of plugin API. The url parameter are not supported and add the file_name parameter which is the model file name to download.

Download a model：

curl -X POST  "http://localhost:8081/dragonfly/models?file_name=squeezenet1_1.mar"

Verify the model download successful:

{
  "Status": "Model \"squeezenet1_1\" Version: 1.0 registered with 0 initial workers. Use scale workers API to add workers for the model."
}

Added model worker for inference:

curl -v -X PUT "http://localhost:8081/models/squeezenet1_1?min_worker=1"

Check the number of worker is increased:

* About to connect() to localhost port 8081 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8081 (#0)
> PUT /models/squeezenet1_1?min_worker=1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 202 Accepted
< content-type: application/json
< x-request-id: 66761b5a-54a7-4626-9aa4-12041e0e4e63
< Pragma: no-cache
< Cache-Control: no-cache; no-store, must-revalidate, private
< Expires: Thu, 01 Jan 1970 00:00:00 UTC
< content-length: 47
< connection: keep-alive
<
{  "status": "Processing worker updates..."}
* Connection #0 to host localhost left intact

Call inference API:

# Prepare pictures that require reasoning
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/kitten_small.jpg
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/dogs-before.jpg

# Call inference API
curl http://localhost:8080/predictions/squeezenet1_1 -T kitten_small.jpg -T dogs-before.jpg

Check the response successful:

{
  "lynx": 0.5455784201622009,
  "tabby": 0.2794168293476105,
  "Egyptian_cat": 0.10391931980848312,
  "tiger_cat": 0.062633216381073,
  "leopard": 0.005019133910536766
}

Performance testing​

Test the performance of single-machine model download by TorchServe API after the integration of Dragonfly P2P. Due to the influence of the network environment of the machine itself, the actual download time is not important, but the ratio of the increase in the download time in different scenarios is very important.

• TorchServe API: Use signed URL provided by Object Storage to download the model directly.
• TorchServe API & Dragonfly Cold Boot: Use TorchServe API to download model via Dragonfly P2P network and no cache hits.
• Hit Remote Peer: Use TorchServe API to download model via Dragonfly P2P network and hit the remote peer cache.
• Hit Local Peer: Use TorchServe API to download model via Dragonfly P2P network and hit the local peer cache.

Test results show TorchServe and Dragonfly integration. It can effectively reduce the file download time. Note that this test was a single-machine test, which means that in the case of cache hits, the performance limitation is on the disk. If Dragonfly is deployed on multiple machines for P2P download, the models download speed will be faster.

Links​

Dragonfly community​

• Website: https://d7y.io/
• GitHub Repo: https://github.com/dragonflyoss/dragonfly
• Slack Channel: #dragonfly on CNCF Slack
• Discussion Group: dragonfly-discuss@googlegroups.com
• Twitter: @dragonfly_oss

Pytorch​

• Website: https://pytorch.org/
• GitHub Repo: https://github.com/pytorch/pytorch
• TorchServe Document: https://pytorch.org/serve/

TorchServe GitHub Repo: https://github.com/pytorch/serve

This document will help you experience how to use dragonfly with hugging face. During the downloading of datasets or models, the file size is large and there are many services downloading the files at the same time. The bandwidth of the storage will reach the limit and the download will be slow.

Dragonfly can be used to eliminate the bandwidth limit of the storage through P2P technology, thereby accelerating file downloading.

Prerequisites​

Notice: Kind is recommended if no kubernetes cluster is available for testing.

Install dragonfly​

For detailed installation documentation based on kubernetes cluster, please refer to quick-start-kubernetes.

Setup kubernetes cluster​

Create kind multi-node cluster configuration file kind-config.yaml, configuration content is as follows:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
    extraPortMappings:
      - containerPort: 30950
        hostPort: 65001
  - role: worker

Create a kind multi-node cluster using the configuration file:

kind create cluster --config kind-config.yaml

Switch the context of kubectl to kind cluster:

kubectl config use-context kind-kind

Kind loads dragonfly image​

Pull dragonfly latest images:

docker pull dragonflyoss/scheduler:latest
docker pull dragonflyoss/manager:latest
docker pull dragonflyoss/dfdaemon:latest

Kind cluster loads dragonfly latest images:

kind load docker-image dragonflyoss/scheduler:latest
kind load docker-image dragonflyoss/manager:latest
kind load docker-image dragonflyoss/dfdaemon:latest

Create dragonfly cluster based on helm charts​

Create helm charts configuration file charts-config.yaml and set dfdaemon.config.proxy.registryMirror.url to the address of the Hugging Face Hub’s LFS server, configuration content is as follows:

scheduler:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

seedPeer:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

dfdaemon:
  metrics:
    enable: true
  hostNetwork: true
  config:
    verbose: true
    pprofPort: 18066
    proxy:
      defaultFilter: 'Expires&Key-Pair-Id&Policy&Signature'
      security:
        insecure: true
      tcpListen:
        listen: 0.0.0.0
        port: 65001
      registryMirror:
        # When enable, using header "X-Dragonfly-Registry" for remote instead of url.
        dynamic: true
        # URL for the registry mirror.
        url: https://cdn-lfs.huggingface.co
        # Whether to ignore https certificate errors.
        insecure: true
        # Optional certificates if the remote server uses self-signed certificates.
        certs: []
        # Whether to request the remote registry directly.
        direct: false
        # Whether to use proxies to decide if dragonfly should be used.
        useProxies: true
      proxies:
        - regx: repos.*
          useHTTPS: true

manager:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

Create a dragonfly cluster using the configuration file:

$ helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
$ helm install --wait --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f charts-config.yaml
NAME: dragonfly
LAST DEPLOYED: Wed Oct 19 04:23:22 2022
NAMESPACE: dragonfly-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the scheduler address by running these commands:
  export SCHEDULER_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=scheduler" -o jsonpath={.items[0].metadata.name})
  export SCHEDULER_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $SCHEDULER_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  kubectl --namespace dragonfly-system port-forward $SCHEDULER_POD_NAME 8002:$SCHEDULER_CONTAINER_PORT
  echo "Visit http://127.0.0.1:8002 to use your scheduler"
2. Get the dfdaemon port by running these commands:
  export DFDAEMON_POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=dfdaemon" -o jsonpath={.items[0].metadata.name})
  export DFDAEMON_CONTAINER_PORT=$(kubectl get pod --namespace dragonfly-system $DFDAEMON_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  You can use $DFDAEMON_CONTAINER_PORT as a proxy port in Node.
3. Configure runtime to use dragonfly:
  https://d7y.io/docs/getting-started/quick-start/kubernetes/

Check that dragonfly is deployed successfully:

$ kubectl get po -n dragonfly-system
NAME                                 READY   STATUS    RESTARTS       AGE
dragonfly-dfdaemon-rhnr6             1/1     Running   4 (101s ago)   3m27s
dragonfly-dfdaemon-s6sv5             1/1     Running   5 (111s ago)   3m27s
dragonfly-manager-67f97d7986-8dgn8   1/1     Running   0              3m27s
dragonfly-mysql-0                    1/1     Running   0              3m27s
dragonfly-redis-master-0             1/1     Running   0              3m27s
dragonfly-redis-replicas-0           1/1     Running   1 (115s ago)   3m27s
dragonfly-redis-replicas-1           1/1     Running   0              95s
dragonfly-redis-replicas-2           1/1     Running   0              70s
dragonfly-scheduler-0                1/1     Running   0              3m27s
dragonfly-seed-peer-0                1/1     Running   2 (95s ago)    3m27s