Cache-to-Cache
Let independent LLMs exchange semantics directly through projected KV caches instead of generated text.
- Source repo
- thu-nics/C2C
- Stars
- ★ 686
- Last updated
- 7d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 43/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
- Not a fit if
- Teams that only need conventional text-based agent messaging
- Users unable to provide model weights, checkpoints, and inference compute
- Teams needing production-mature multi-sharer serving today
- Source review
- 43/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
Cache-to-Cache (C2C) is a Python project for direct semantic communication between large language models, packaged under the name `rosetta`. Its `C2CProjector` maps a sharer model's KV cache into a receiver model's representation space, while `RosettaModel` fuses that state into generation without requiring an intermediate text response. The repository includes downloadable pretrained fusers, an interactive chat script, a Gradio demo, supervised training scripts, and a unified evaluator; the receiver still produces the final text output. During training, the source and target models remain frozen and only the projector parameters are updated, with extension points for custom projectors, datasets, and benchmarks. The project reports 8.5–10.5% higher accuracy than individual models, 3.0–5.0% better performance than text-based communication, and a 2.0× latency speedup. It is best suited to research or engineering teams that can run local models and want model-level latent communication, rather than users seeking a ready-made ChatGPT, Claude, or agent-protocol integration.
The flow starts by loading receiver and sharer models through AutoModelForCausalLM or load_rosetta_model. A C2CProjector transforms each selected sharer's KV cache, and RosettaModel.set_projector_config maps source layers, target layers, and projector instances. When RosettaModel.generate runs, kv_cache_index determines where in the sequence each sharer's projected cache is applied; the receiver fuses those representations and generates the final text. Published checkpoints can be retrieved from nics-efc/C2C_Fuser with huggingface_hub.snapshot_download, while script/playground/live_chat_example.py supports one or multiple sharers. script/train/SFT_train.py reads recipes from recipe/train_recipe/ and optimizes only the projector, and script/evaluation/unified_evaluator.py executes configured datasets and metrics. A local script/playground/gradio_demo.py and a separate hosted Hugging Face demo are also documented.
- Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
- Engineers with local Hugging Face causal models who need to train a dedicated fuser for a chosen receiver–sharer pair.
- Researchers reproducing experiments with the supplied training recipes and
unified_evaluator.pyon configured datasets. - Prototype builders demonstrating how one or several sharer models can influence a receiver through
live_chat_example.py. - Model-architecture researchers adding a custom
Projector, dataset adapter, or evaluation benchmark through the registries and configuration system.
How do you install or deploy this agent?
Create the documented Python 3.10 environment and install the core package in editable mode:
conda create -n rosetta python=3.10
conda activate rosetta
pip install -e .Install the additional dependency groups for training and evaluation:
pip install -e ".[training,evaluation]"The published inference example explicitly selects torch.device("cuda"), so running it unchanged requires a working CUDA environment. Downloading a published fuser also requires network access to Hugging Face. No mandatory API credential is documented in the example.
How do you use this agent?
The shortest documented pretrained path downloads a fuser, loads the wrapped model, and uses kv_cache_index to control when the sharer's cache is injected:
import torch
from huggingface_hub import snapshot_download
from script.playground.inference_example import load_rosetta_model, run_inference_example
checkpoint_dir = snapshot_download(
repo_id="nics-efc/C2C_Fuser",
allow_patterns=["qwen3_0.6b+qwen2.5_0.5b_Fuser/*"],
)
model_config = {
"rosetta_config": {
"base_model": "Qwen/Qwen3-0.6B",
"teacher_model": "Qwen/Qwen2.5-0.5B-Instruct",
"checkpoints_dir": f"{checkpoint_dir}/qwen3_0.6b+qwen2.5_0.5b_Fuser/final",
}
}
rosetta_model, tokenizer = load_rosetta_model(model_config, eval_config={}, device=torch.device("cuda"))
device = rosetta_model.device
prompt = [{"role": "user", "content": "Say hello in one short sentence."}]
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(input_text, return_tensors="pt").to(device)
instruction_index = torch.tensor([1, 0], dtype=torch.long).repeat(inputs['input_ids'].shape[1] - 1, 1).unsqueeze(0).to(device)
label_index = torch.tensor([-1, 0], dtype=torch.long).repeat(1, 1).unsqueeze(0).to(device)
kv_cache_index = [instruction_index, label_index]
with torch.no_grad():
sampling_params = {
'do_sample': False,
'max_new_tokens': 256
}
outputs = rosetta_model.generate(**inputs, kv_cache_index=kv_cache_index, **sampling_params)
output_text = tokenizer.decode(outputs[0, instruction_index.shape[1] + 1:], skip_special_tokens=True)
print(f"C2C output text: {output_text}")An existing checkpoint can drive the interactive chat. Passing several checkpoint paths enables the preliminary multi-sharer mode:
python script/playground/live_chat_example.py --checkpoint_dir path/to/checkpointpython script/playground/live_chat_example.py --checkpoint_dir path/to/ckpt1 path/to/ckpt2Training and evaluation use the supplied configuration recipes:
python script/train/SFT_train.py --config recipe/train_recipe/C2C_0.6+0.5.jsonpython script/evaluation/unified_evaluator.py --config recipe/eval_recipe/unified_eval.yamlWhat are this agent's strengths and limitations?
- It projects and fuses KV caches directly, avoiding intermediate text generation—the defining distinction from text-based model communication.
- Seven pretrained Qwen/Llama receiver–sharer pairs are listed, while the framework also handles differing hidden sizes, layer counts, attention heads, and tokenizers.
- Training freezes both foundation models and updates only the C2C projectors, limiting the parameters that must be optimized.
- The repository covers inference, interactive chat, a Gradio demo, single- and multi-GPU training, unified evaluation, and extension hooks.
- Published checkpoints cover only the listed Qwen and Llama pairings; other combinations generally require a custom configuration and projector training.
- The examples load multiple local models and checkpoints and use CUDA, creating a higher deployment burden than ordinary text API orchestration.
- Multi-sharer support is explicitly described as preliminary and still under active development.
- The agent-managed KV-cache feature and its serving system are only announced as forthcoming, so a complete production serving path is not documented.
- There is no documented native integration with ChatGPT, Codex, Claude, the OpenAI API, the Claude API, or MCP.
How does this agent compare with similar options?
Unlike text-based model communication, C2C does not first decode a sharer's knowledge into a textual message. It projects the sharer's KV cache and fuses it into the receiver instead. The project reports 3.0–5.0% better performance and a 2.0× latency speedup over text communication, plus 8.5–10.5% higher accuracy than individual models. The supplied material does not provide per-model or per-dataset breakdowns for those aggregate figures, nor does it compare C2C with another named framework.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Cache-to-Cache This agent | 43 · Major gaps | Library / SDKFree | ★ 686 | 7d ago | Python | — |
| JARVIS / HuggingGPT | 36 · Major gaps | Self-hosted serviceFree + model costs | ★ 25k | 1y ago | Python | OpenAI API |
| AgileRL | 71 · Some gaps | Library / SDKFreemium | ★ 951 | 1d ago | Python | — |
| AutoResearch | 52 · Major gaps | CLIFree + model costs | ★ 97k | 6mo ago | Python | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Runtime examples are primarily local model inference, and the workflow scopes its declared permission to contents: write; however, that permission automatically edits and pushes to main after a push, without human confirmation or rollback guidance. The README discloses Hugging Face model/checkpoint downloads, but does not fully map flows involving datasets, prompts, generated content, or optional OpenAI/W&B integrations, and provides no sensitive-data or credential-handling guidance. Core dependencies are exactly pinned, while many optional dependencies are loosely or not versioned; the action uses a mutable tag and no vulnerability or supply-chain controls are shown. Paper authors, citation metadata, and linked artifacts provide meaningful attribution, but placeholder your-org package URLs weaken its completeness.
The C2C/Rosetta explanation, examples, and repository layout are broadly coherent, and concrete model/checkpoint availability is documented. Scores are reduced because the project description, homepage links, Python support declarations, and license metadata conflict. Core dependencies and model sources are identified, supporting ordinary availability, but external model, dataset, and network dependencies are not mitigated. No exception behavior, diagnostic messages, common-error documentation, or recovery guidance is present, so failure_messages receives zero.
Documentation covers pretrained inference, chat, training, evaluation, and extension of projectors, datasets, and benchmarks, giving researchers and developers several clear scenarios. kv_cache_index, model roles, and configuration points are illustrated concretely, supporting reasonably precise activation. However, the claimed support for arbitrary LLM pairs lacks compatibility conditions or counterexamples, and multi-sharer support is explicitly preliminary. Setup covers Python 3.10 and installation, but examples assume CUDA and omit CPU support, memory requirements, platforms, offline operation, and hardware minimums; Python-version configuration is also inconsistent.
The README is well organized around setup, use, training, evaluation, extension, and code layout, with several actionable examples. Deductions reflect the absence of an FAQ or troubleshooting section, sparse limitations, and unstable C2C/Rosetta/Unified Memory naming and placeholder URLs. The repository LICENSE is Apache-2.0 while pyproject declares MIT in both license text and classifier; this material conflict makes the license criterion zero. Versioning consists only of 0.1.0/Alpha and a News section rather than a formal changelog. Authors and an organization are visible, but no designated maintainer, support route, release policy, or security-update owner is stated.
Examples produce directly decodable text and expose chat, training, and evaluation paths, making outputs reasonably usable for research. KV-cache semantic communication presents a clear potential advantage over individual models and text communication, and the README gives accuracy and latency figures. Those figures remain summary assertions in the supplied files without experiment tables, statistical detail, or resource accounting. The compute, storage, training, GPU, and checkpoint costs of operating multiple models are not quantified, limiting the cost-benefit score.
The main performance claims point to a named paper, while reproduction entry points, configuration locations, and supported model pairs are stated concretely, providing reasonable traceability. The supplied evidence does not include the paper body, test results, or independent sources that corroborate the numbers, and README/pyproject conflicts affect cross-source consistency. Broad claims such as support for arbitrary LLM pairs and performance gains are not clearly separated into verified facts, inference, and intended capability, so fact-inference separation remains thin.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- There is a direct licensing conflict: the root LICENSE is Apache-2.0, while pyproject declares MIT. Obtain maintainer clarification before redistribution or integration.
- A push to main triggers a contents: write workflow that can rewrite and push files automatically. Constrain its permissions, pin actions to immutable commits, and add review and recovery controls before adoption.
- Do not expose prompts, private datasets, or sensitive outputs to optional remote services until the actual OpenAI, W&B, Hugging Face, and dataset-loading data flows and retention policies are verified.
- Treat the accuracy gains, 2x latency claim, and arbitrary-LLM compatibility as project claims requiring independent validation, including GPU/memory cost and architecture-specific compatibility checks.