Cache-to-Cache
Let independent LLMs exchange semantics directly through projected KV caches instead of generated text.
- Source repo
- thu-nics/C2C
- Stars
- ★ 782
- Last updated
- 24d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 39/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
- Not a fit if
- Teams that only need conventional text-based agent messaging
- Users unable to provide model weights, checkpoints, and inference compute
- Teams needing production-mature multi-sharer serving today
- Website
- fuvty.github.io ↗ set on GitHub
- Maintenance
- Commits in the last 30 dayslast commit 2026-10-28+0.9 ★/day over 7 days
- Source review
- 39/100 · Major gaps 3 safety controls not found
What does this agent do, and when should you use it?
Cache-to-Cache (C2C) is a Python project for direct semantic communication between large language models, packaged under the name rosetta. Its C2CProjector maps a sharer model's KV cache into a receiver model's representation space, while RosettaModel fuses that state into generation without requiring an intermediate text response. The repository includes downloadable pretrained fusers, an interactive chat script, a Gradio demo, supervised training scripts, and a unified evaluator; the receiver still produces the final text output. During training, the source and target models remain frozen and only the projector parameters are updated, with extension points for custom projectors, datasets, and benchmarks. The project reports 8.5–10.5% higher accuracy than individual models, 3.0–5.0% better performance than text-based communication, and a 2.0× latency speedup. It is best suited to research or engineering teams that can run local models and want model-level latent communication, rather than users seeking a ready-made ChatGPT, Claude, or agent-protocol integration.
The flow starts by loading receiver and sharer models through AutoModelForCausalLM or load_rosetta_model. A C2CProjector transforms each selected sharer's KV cache, and RosettaModel.set_projector_config maps source layers, target layers, and projector instances. When RosettaModel.generate runs, kv_cache_index determines where in the sequence each sharer's projected cache is applied; the receiver fuses those representations and generates the final text. Published checkpoints can be retrieved from nics-efc/C2C_Fuser with huggingface_hub.snapshot_download, while script/playground/live_chat_example.py supports one or multiple sharers. script/train/SFT_train.py reads recipes from recipe/train_recipe/ and optimizes only the projector, and script/evaluation/unified_evaluator.py executes configured datasets and metrics. A local script/playground/gradio_demo.py and a separate hosted Hugging Face demo are also documented.
- Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
- Engineers with local Hugging Face causal models who need to train a dedicated fuser for a chosen receiver–sharer pair.
- Researchers reproducing experiments with the supplied training recipes and
unified_evaluator.pyon configured datasets. - Prototype builders demonstrating how one or several sharer models can influence a receiver through
live_chat_example.py. - Model-architecture researchers adding a custom
Projector, dataset adapter, or evaluation benchmark through the registries and configuration system.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pip install -epip install -e ".[training,evaluation]"
- Config files
0.5.jsonrecipe/eval_recipe/unified_eval.yaml- Runtime
- Python 3.10
How do you install or deploy this agent?
Create the documented Python 3.10 environment and install the core package in editable mode:
conda create -n rosetta python=3.10
conda activate rosetta
pip install -e .Install the additional dependency groups for training and evaluation:
pip install -e ".[training,evaluation]"The published inference example explicitly selects torch.device("cuda"), so running it unchanged requires a working CUDA environment. Downloading a published fuser also requires network access to Hugging Face. No mandatory API credential is documented in the example.
How do you use this agent?
The shortest documented pretrained path downloads a fuser, loads the wrapped model, and uses kv_cache_index to control when the sharer's cache is injected:
import torch
from huggingface_hub import snapshot_download
from script.playground.inference_example import load_rosetta_model, run_inference_example
checkpoint_dir = snapshot_download(
repo_id="nics-efc/C2C_Fuser",
allow_patterns=["qwen3_0.6b+qwen2.5_0.5b_Fuser/*"],
)
model_config = {
"rosetta_config": {
"base_model": "Qwen/Qwen3-0.6B",
"teacher_model": "Qwen/Qwen2.5-0.5B-Instruct",
"checkpoints_dir": f"{checkpoint_dir}/qwen3_0.6b+qwen2.5_0.5b_Fuser/final",
}
}
rosetta_model, tokenizer = load_rosetta_model(model_config, eval_config={}, device=torch.device("cuda"))
device = rosetta_model.device
prompt = [{"role": "user", "content": "Say hello in one short sentence."}]
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(input_text, return_tensors="pt").to(device)
instruction_index = torch.tensor([1, 0], dtype=torch.long).repeat(inputs['input_ids'].shape[1] - 1, 1).unsqueeze(0).to(device)
label_index = torch.tensor([-1, 0], dtype=torch.long).repeat(1, 1).unsqueeze(0).to(device)
kv_cache_index = [instruction_index, label_index]
with torch.no_grad():
sampling_params = {
'do_sample': False,
'max_new_tokens': 256
}
outputs = rosetta_model.generate(**inputs, kv_cache_index=kv_cache_index, **sampling_params)
output_text = tokenizer.decode(outputs[0, instruction_index.shape[1] + 1:], skip_special_tokens=True)
print(f"C2C output text: {output_text}")An existing checkpoint can drive the interactive chat. Passing several checkpoint paths enables the preliminary multi-sharer mode:
python script/playground/live_chat_example.py --checkpoint_dir path/to/checkpointpython script/playground/live_chat_example.py --checkpoint_dir path/to/ckpt1 path/to/ckpt2Training and evaluation use the supplied configuration recipes:
python script/train/SFT_train.py --config recipe/train_recipe/C2C_0.6+0.5.jsonpython script/evaluation/unified_evaluator.py --config recipe/eval_recipe/unified_eval.yamlWhat are this agent's strengths and limitations?
- It projects and fuses KV caches directly, avoiding intermediate text generation—the defining distinction from text-based model communication.
- Seven pretrained Qwen/Llama receiver–sharer pairs are listed, while the framework also handles differing hidden sizes, layer counts, attention heads, and tokenizers.
- Training freezes both foundation models and updates only the C2C projectors, limiting the parameters that must be optimized.
- The repository covers inference, interactive chat, a Gradio demo, single- and multi-GPU training, unified evaluation, and extension hooks.
- Published checkpoints cover only the listed Qwen and Llama pairings; other combinations generally require a custom configuration and projector training.
- The examples load multiple local models and checkpoints and use CUDA, creating a higher deployment burden than ordinary text API orchestration.
- Multi-sharer support is explicitly described as preliminary and still under active development.
- The agent-managed KV-cache feature and its serving system are only announced as forthcoming, so a complete production serving path is not documented.
- There is no documented native integration with ChatGPT, Codex, Claude, the OpenAI API, the Claude API, or MCP.
How does this agent compare with similar options?
Unlike text-based model communication, C2C does not first decode a sharer's knowledge into a textual message. It projects the sharer's KV cache and fuses it into the receiver instead. The project reports 3.0–5.0% better performance and a 2.0× latency speedup over text communication, plus 8.5–10.5% higher accuracy than individual models. The supplied material does not provide per-model or per-dataset breakdowns for those aggregate figures, nor does it compare C2C with another named framework.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Cache-to-Cache This agent | 39 · Major gaps | Library / SDKFree | ★ 782 | 24d ago | Python | — |
| JARVIS / HuggingGPT | 42 · Major gaps | Self-hosted serviceFree + model costs | ★ 26k | 1y ago | Python | OpenAI API |
| Gamma-World: Multi-Agent World Model | 38 · Major gaps | — | ★ 579 | 5mo ago | Python | — |
| AgileRL | 62 · Some gaps | Library / SDKFreemium | ★ 970 | today | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
The README shows local model loading, training, and evaluation examples, but does not explain model or input-data handling boundaries, sensitive-data protections, or confirmation before actions; these areas receive only limited credit. Dependencies are version-pinned, but no security review evidence is supplied. A GitHub Actions workflow has contents:write permission and can commit and push path rewrites after pushes, with no approval or rollback guidance, so external effects and recovery score zero. The README cites a paper, project page, and model resources, providing some attribution evidence, while publisher identity remains unknown.
The README, package metadata, and license conflict: the README describes C2C, while package metadata and repository links still identify unified_memory; metadata declares MIT, while the supplied LICENSE is Apache-2.0. Dependencies are pinned and optional dependency groups are documented, but failure handling and error messages are not described.
The README targets developers and researchers and covers inference, chat, training, evaluation, and adding components with examples, so intended scenarios are fairly clear. It claims support for arbitrary model pairs but the supplied material does not define compatibility limits. Environment guidance mostly covers Python 3.10, CUDA examples, and basic installation, leaving fit details incomplete.
The repository structure, installation steps, configuration locations, examples, and supported model pairs are documented, giving the material a useful organization. Multi-sharer support is explicitly marked preliminary, but other known limitations are sparse. The LICENSE is Apache-2.0 while pyproject declares MIT; the package version is 0.1.0, while homepage and documentation links retain placeholder or legacy project names. News provides some update history, but maintainer and support responsibilities are unclear.
The project describes transferring semantics between models through KV caches and provides pretrained fusers, examples, and training/evaluation workflows, demonstrating usable functionality. Accuracy and performance benefits are asserted in the README without corroboration in the supplied material. Resource costs for using two models and a projector are not quantified, so value and cost-benefit merit only partial credit.
The README includes a paper citation, model-pair list, and configuration examples, but the supplied material lacks checkable experimental detail for the accuracy, latency, and speed claims. Conflicts among the README, pyproject, and LICENSE prevent strong cross-source corroboration. Performance figures are presented as claims without a clear separation between evidence and inference.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- License metadata conflicts: LICENSE is Apache-2.0, while pyproject.toml declares MIT; do not assume they are interchangeable without resolving the discrepancy.
- The GitHub Actions workflow can automatically commit and push rewrites after a push with contents:write permission; review this write behavior and its permissions when adopting the repository.
- Accuracy and speed improvements are README claims; the supplied static material does not include experimental detail supporting those figures.
FAQ
Can it be used with any pair of language models?
Does training alter the foundation-model weights?
Is Hugging Face hosting required at inference time?
Is multi-sharer fusion ready for production deployment?
Does the project include a complete long-running serving system?
Is Cache-to-Cache free?
Is Cache-to-Cache open source?
Is Cache-to-Cache safe to use?
Is Cache-to-Cache still maintained?
Who makes Cache-to-Cache?
What is Cache-to-Cache's official website?
Thanks — we use this to improve the page.