Cache-to-Cache

Let independent LLMs exchange semantics directly through projected KV caches instead of generated text.

Source repo
thu-nics/C2C
Stars
★ 782
Last updated
24d ago
License
Apache-2.0
Primary language
Python

At a glance

How it runs
Library / SDKCLIWeb app
Works with
Universal · cross-platform
Cost
Free, no paid service needed
Setup effort
High · needs real infrastructure
You'll need
Python 3.10condaPyTorchTransformershuggingface_hubCUDA-capable runtime for the documented inference exampleShell / CLINetwork accessLocal filesystem
Typical use
Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
Not a fit if
  • Teams that only need conventional text-based agent messaging
  • Users unable to provide model weights, checkpoints, and inference compute
  • Teams needing production-mature multi-sharer serving today
Website
fuvty.github.io ↗ set on GitHub
Maintenance
Commits in the last 30 dayslast commit 2026-10-28+0.9 ★/day over 7 days

What does this agent do, and when should you use it?

Cache-to-Cache (C2C) is a Python project for direct semantic communication between large language models, packaged under the name rosetta. Its C2CProjector maps a sharer model's KV cache into a receiver model's representation space, while RosettaModel fuses that state into generation without requiring an intermediate text response. The repository includes downloadable pretrained fusers, an interactive chat script, a Gradio demo, supervised training scripts, and a unified evaluator; the receiver still produces the final text output. During training, the source and target models remain frozen and only the projector parameters are updated, with extension points for custom projectors, datasets, and benchmarks. The project reports 8.5–10.5% higher accuracy than individual models, 3.0–5.0% better performance than text-based communication, and a 2.0× latency speedup. It is best suited to research or engineering teams that can run local models and want model-level latent communication, rather than users seeking a ready-made ChatGPT, Claude, or agent-protocol integration.

The flow starts by loading receiver and sharer models through AutoModelForCausalLM or load_rosetta_model. A C2CProjector transforms each selected sharer's KV cache, and RosettaModel.set_projector_config maps source layers, target layers, and projector instances. When RosettaModel.generate runs, kv_cache_index determines where in the sequence each sharer's projected cache is applied; the receiver fuses those representations and generates the final text. Published checkpoints can be retrieved from nics-efc/C2C_Fuser with huggingface_hub.snapshot_download, while script/playground/live_chat_example.py supports one or multiple sharers. script/train/SFT_train.py reads recipes from recipe/train_recipe/ and optimizes only the projector, and script/evaluation/unified_evaluator.py executes configured datasets and metrics. A local script/playground/gradio_demo.py and a separate hosted Hugging Face demo are also documented.

  1. Multi-model researchers comparing KV-cache communication with conventional text exchanges between Qwen or Llama models.
  2. Engineers with local Hugging Face causal models who need to train a dedicated fuser for a chosen receiver–sharer pair.
  3. Researchers reproducing experiments with the supplied training recipes and unified_evaluator.py on configured datasets.
  4. Prototype builders demonstrating how one or several sharer models can influence a receiver through live_chat_example.py.
  5. Model-architecture researchers adding a custom Projector, dataset adapter, or evaluation benchmark through the registries and configuration system.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • pip install -e
  • pip install -e ".[training,evaluation]"
Config files
0.5.json recipe/eval_recipe/unified_eval.yaml
Runtime
Python 3.10

How do you install or deploy this agent?

Create the documented Python 3.10 environment and install the core package in editable mode:

conda create -n rosetta python=3.10
conda activate rosetta
pip install -e .

Install the additional dependency groups for training and evaluation:

pip install -e ".[training,evaluation]"

The published inference example explicitly selects torch.device("cuda"), so running it unchanged requires a working CUDA environment. Downloading a published fuser also requires network access to Hugging Face. No mandatory API credential is documented in the example.

How do you use this agent?

The shortest documented pretrained path downloads a fuser, loads the wrapped model, and uses kv_cache_index to control when the sharer's cache is injected:

import torch
from huggingface_hub import snapshot_download
from script.playground.inference_example import load_rosetta_model, run_inference_example

checkpoint_dir = snapshot_download(
    repo_id="nics-efc/C2C_Fuser",
    allow_patterns=["qwen3_0.6b+qwen2.5_0.5b_Fuser/*"],
)

model_config = {
    "rosetta_config": {
        "base_model": "Qwen/Qwen3-0.6B",
        "teacher_model": "Qwen/Qwen2.5-0.5B-Instruct",
        "checkpoints_dir": f"{checkpoint_dir}/qwen3_0.6b+qwen2.5_0.5b_Fuser/final",
    }
}

rosetta_model, tokenizer = load_rosetta_model(model_config, eval_config={}, device=torch.device("cuda"))
device = rosetta_model.device

prompt = [{"role": "user", "content": "Say hello in one short sentence."}]
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(input_text, return_tensors="pt").to(device)

instruction_index = torch.tensor([1, 0], dtype=torch.long).repeat(inputs['input_ids'].shape[1] - 1, 1).unsqueeze(0).to(device)
label_index = torch.tensor([-1, 0], dtype=torch.long).repeat(1, 1).unsqueeze(0).to(device)
kv_cache_index = [instruction_index, label_index]

with torch.no_grad():
    sampling_params = {
        'do_sample': False,
        'max_new_tokens': 256
    }
    outputs = rosetta_model.generate(**inputs, kv_cache_index=kv_cache_index, **sampling_params)
    output_text = tokenizer.decode(outputs[0, instruction_index.shape[1] + 1:], skip_special_tokens=True)
    print(f"C2C output text: {output_text}")

An existing checkpoint can drive the interactive chat. Passing several checkpoint paths enables the preliminary multi-sharer mode:

python script/playground/live_chat_example.py --checkpoint_dir path/to/checkpoint
python script/playground/live_chat_example.py --checkpoint_dir path/to/ckpt1 path/to/ckpt2

Training and evaluation use the supplied configuration recipes:

python script/train/SFT_train.py --config recipe/train_recipe/C2C_0.6+0.5.json
python script/evaluation/unified_evaluator.py --config recipe/eval_recipe/unified_eval.yaml

What are this agent's strengths and limitations?

Pros
  • It projects and fuses KV caches directly, avoiding intermediate text generation—the defining distinction from text-based model communication.
  • Seven pretrained Qwen/Llama receiver–sharer pairs are listed, while the framework also handles differing hidden sizes, layer counts, attention heads, and tokenizers.
  • Training freezes both foundation models and updates only the C2C projectors, limiting the parameters that must be optimized.
  • The repository covers inference, interactive chat, a Gradio demo, single- and multi-GPU training, unified evaluation, and extension hooks.
Limitations
  • Published checkpoints cover only the listed Qwen and Llama pairings; other combinations generally require a custom configuration and projector training.
  • The examples load multiple local models and checkpoints and use CUDA, creating a higher deployment burden than ordinary text API orchestration.
  • Multi-sharer support is explicitly described as preliminary and still under active development.
  • The agent-managed KV-cache feature and its serving system are only announced as forthcoming, so a complete production serving path is not documented.
  • There is no documented native integration with ChatGPT, Codex, Claude, the OpenAI API, the Claude API, or MCP.

How does this agent compare with similar options?

Unlike text-based model communication, C2C does not first decode a sharer's knowledge into a textual message. It projects the sharer's KV cache and fuses it into the receiver instead. The project reports 3.0–5.0% better performance and a 2.0× latency speedup over text communication, plus 8.5–10.5% higher accuracy than individual models. The supplied material does not provide per-model or per-dataset breakdowns for those aggregate figures, nor does it compare C2C with another named framework.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
Cache-to-Cache This agent 39 · Major gaps Library / SDKFree ★ 782 24d ago Python —
JARVIS / HuggingGPT 42 · Major gaps Self-hosted serviceFree + model costs ★ 26k 1y ago Python OpenAI API
Gamma-World: Multi-Agent World Model 38 · Major gaps — ★ 579 5mo ago Python —
AgileRL 62 · Some gaps Library / SDKFreemium ★ 970 today Python —

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
39/ 100 5-point scale 2.0 / 5
Trust 7/29
Reliability 5/14
Adaptability 9/18
Convention 8/18
Effectiveness 7/13
Verifiability 3/8
Why each dimension lost points
Trust7 / 29 · 1.2/5

The README shows local model loading, training, and evaluation examples, but does not explain model or input-data handling boundaries, sensitive-data protections, or confirmation before actions; these areas receive only limited credit. Dependencies are version-pinned, but no security review evidence is supplied. A GitHub Actions workflow has contents:write permission and can commit and push path rewrites after pushes, with no approval or rollback guidance, so external effects and recovery score zero. The README cites a paper, project page, and model resources, providing some attribution evidence, while publisher identity remains unknown.

Reliability5 / 14 · 1.8/5

The README, package metadata, and license conflict: the README describes C2C, while package metadata and repository links still identify unified_memory; metadata declares MIT, while the supplied LICENSE is Apache-2.0. Dependencies are pinned and optional dependency groups are documented, but failure handling and error messages are not described.

Adaptability9 / 18 · 2.5/5

The README targets developers and researchers and covers inference, chat, training, evaluation, and adding components with examples, so intended scenarios are fairly clear. It claims support for arbitrary model pairs but the supplied material does not define compatibility limits. Environment guidance mostly covers Python 3.10, CUDA examples, and basic installation, leaving fit details incomplete.

Convention8 / 18 · 2.2/5

The repository structure, installation steps, configuration locations, examples, and supported model pairs are documented, giving the material a useful organization. Multi-sharer support is explicitly marked preliminary, but other known limitations are sparse. The LICENSE is Apache-2.0 while pyproject declares MIT; the package version is 0.1.0, while homepage and documentation links retain placeholder or legacy project names. News provides some update history, but maintainer and support responsibilities are unclear.

Effectiveness7 / 13 · 2.7/5

The project describes transferring semantics between models through KV caches and provides pretrained fusers, examples, and training/evaluation workflows, demonstrating usable functionality. Accuracy and performance benefits are asserted in the README without corroboration in the supplied material. Resource costs for using two models and a projector are not quantified, so value and cost-benefit merit only partial credit.

Verifiability3 / 8 · 1.9/5

The README includes a paper citation, model-pair list, and configuration examples, but the supplied material lacks checkable experimental detail for the accuracy, latency, and speed claims. Conflicts among the README, pyproject, and LICENSE prevent strong cross-source corroboration. Performance figures are presented as claims without a clear separation between evidence and inference.

Risks and how to mitigate them
  • Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
  • Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
  • Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
  • License metadata conflicts: LICENSE is Apache-2.0, while pyproject.toml declares MIT; do not assume they are interchangeable without resolving the discrepancy.
  • The GitHub Actions workflow can automatically commit and push rewrites after a push with contents:write permission; review this write behavior and its permissions when adopting the repository.
  • Accuracy and speed improvements are README claims; the supplied static material does not include experimental detail supporting those figures.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 5eac72b90faf
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/c2c.svg)](https://followagents.com/en/agents/c2c)
<a href="https://followagents.com/en/agents/c2c"><img src="https://followagents.com/badge/c2c.svg" alt="FARS review"></a>

FAQ

Can it be used with any pair of language models?
The framework claims arbitrary-pair support and handles differences in hidden dimensions, layers, attention heads, and tokenizers. Only seven listed pairs have published fusers, so other pairs will generally require configuration and training.
Does training alter the foundation-model weights?
No. The documented training process freezes both source and target models and updates only the C2C projector parameters.
Is Hugging Face hosting required at inference time?
The core delivery is a local Python library. The example downloads models and fusers from Hugging Face, but the supplied material does not say that inference must call a hosted API after the files are available.
Is multi-sharer fusion ready for production deployment?
A chat example can fuse caches from multiple sharers into one receiver, but the feature is explicitly described as preliminary and under active development.
Does the project include a complete long-running serving system?
It currently documents a library, scripts, a Gradio demo, and a hosted demonstration. Agent-managed KV cache and its corresponding serving system are announced as forthcoming.
Is Cache-to-Cache free?
Yes. Cache-to-Cache is free to use and does not need a paid service.
Is Cache-to-Cache open source?
Yes. The source code is on GitHub (thu-nics/C2C) under the Apache-2.0 licence.
Is Cache-to-Cache safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored Cache-to-Cache 7/29 on trust and safety. The review did not find these safety controls in the source: sensitive-data handling, disclosed external effects, rollback or recovery path. Before giving it real accounts or data: Use dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
Is Cache-to-Cache still maintained?
Yes. The latest commit on GitHub is from 2026-10-28, within the last 30 days.
Who makes Cache-to-Cache?
Cache-to-Cache is published on GitHub by the account thu-nics. FollowAgents has not verified which person or company is behind that account.
What is Cache-to-Cache's official website?
The project's GitHub repository lists https://fuvty.github.io/thinking_yard_project_page/projects/c2c/ as its website. That link is set by the repository owner; the source code itself is on GitHub (thu-nics/C2C).
Was this page helpful?
View on GitHub ↗ Install ↓

Related concepts