ToolBench & ToolLLaMA
Train, run, and evaluate API-using models with real REST API data, retrieval, and DFSDT.
Per-dimension scores and reasoning
Evidence shows: The repository provides no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust-related criteria are unmet, hence score 0.
Evidence shows: README provides detailed installation and run instructions, but no error handling or failure messages are documented. Dependencies are listed in requirements.txt, but availability is not guaranteed. Thus, self_consistency and dependency_availability score 1, failure_messages score 0.
Evidence shows: README clearly identifies target audience (researchers and developers) and use cases (tool learning, training, evaluation). API customization and Web UI instructions are provided, but trigger precision (e.g., conditions for API calls) is not explicit. Thus, audience_and_scenarios, capability_boundaries, and environment_fit score 2, trigger_precision score 1.
Evidence shows: README provides clear information architecture, installation notes, and examples. Naming stability is not explicit, known limitations are partially mentioned, license is Apache-2.0, versioning/changelog is partially mentioned, and maintenance responsibility is not explicit. Thus, information_architecture, install_notes, examples_and_faq, and license score 2; naming_stability, known_limitations, versioning_changelog, and maintenance_responsibility score 1.
Evidence shows: Output format (e.g., JSON) is described in README, but no actual output examples are provided. Marginal value is high (provides dataset and models), but cost-benefit is not explicit. Thus, output_usability and marginal_value score 2, cost_benefit score 1.
Evidence shows: Claims in README (e.g., performance) lack specific data or citations, but a paper is referenced. Cross-source verification is limited, and facts vs. inferences are not clearly separated. Thus, claim_traceability, cross_source_corroboration, and fact_inference_separation score 1.
- The repository lacks permission management or user confirmation mechanisms; caution is advised when using external APIs.
- Dependencies are outdated and may have known security vulnerabilities; consider updating.
- Performance claims in README lack specific data support; treat with caution.
What does this agent do, and when should you use it?
ToolBench is a data, training, inference, and evaluation project for language-model tool learning, accompanied by ToolLLaMA models and the ToolEval evaluator. Its dataset reports 3,451 tools, 16,464 REST APIs, 126,486 instances, and 469,585 real API calls across single-tool and two multi-tool settings. The repository includes retriever training, ToolLLaMA fine-tuning, closed-domain and open-domain inference, and a local tool environment for custom APIs. Inference can use the ToolBench RapidAPI service or a user's own RapidAPI account; a forked Chatbot UI front end and Python backend server are also documented. ToolEval measures pass rate and preference/win rate from model predictions, using the OpenAI API in its evaluation workflow.
The project reads API descriptions and implementations from data/toolenv/tools/ and uses training, annotation, and retrieval data from data/instruction, data/answer, and data/retrieval. preprocess/preprocess_retriever_data.py produces retriever data and toolbench/retrieval/train.py trains a bert-base-uncased retriever; preprocess/preprocess_toolllama_data.py processes solution-path annotations and toolbench/train/train_mem.py or train_lora.py fine-tunes ToolLLaMA. toolbench/inference/qa_pipeline.py runs DFS_woFilter_w2 inference and writes answer files, while qa_pipeline_open_domain.py retrieves APIs from corpus.tsv first; documented backbone_model values include toolllama, chatgpt_function, and davinci. toolbench/inference/toolbench_server.py exposes a local :5000/stream endpoint that accepts JSON fields text, top_k, and method. ToolEval uses convert_to_answer_format.py, eval_pass_rate.py, and eval_preference.py to convert predictions and calculate pass-rate and preference outputs.
- A research group training an open model to call APIs from real RapidAPI documentation and execution traces.
- A machine-learning engineer reproducing ToolLLaMA fine-tuning experiments on a machine with the documented two A100 80GB GPUs.
- A model developer who needs to retrieve candidate APIs before running open-domain API-use inference.
- An application prototyping team that wants to add a documented custom API, such as hello_world, with its own api.py implementation to closed-domain inference.
- An evaluator comparing predictions from multiple methods across the six test subsets using pass rate and preference win rate.
- A team demonstrating ToolLLaMA with the separately run Chatbot UI fork and the local :5000/stream backend.
What are this agent's strengths and limitations?
- It supplies an end-to-end research workflow—from API data and path annotations to retrieval training, inference, and evaluation—rather than model weights alone.
- The documented dataset covers single-tool, intra-category multi-tool, and intra-collection multi-tool tasks, including reasoning, tool execution, and execution results.
- It supports both a closed API list and open-domain API retrieval using corpus.tsv and a retrieval model.
- ToolEval provides both pass-rate and pairwise preference evaluation, along with prediction-conversion scripts.
- Custom APIs can be added to closed-domain inference through an API documentation JSON file and api.py.
- API execution is tightly tied to RapidAPI: the hosted path requires a ToolBench key, while the alternative requires a personal RapidAPI key and subscriptions.
- The documented full ToolLLaMA-7b training example uses two A100 80GB GPUs, creating a substantial hardware requirement.
- Open-domain inference has several additional deployment pieces: a trained retriever, corpus.tsv, and tool-environment data.
- ToolEval and OpenAI-model inference require OpenAI API keys; pass-rate and preference evaluation perform repeated evaluation calls.
- Custom API usage is currently documented only for closed-domain inference; open-domain custom API support is described as planned.
How do you install or deploy this agent?
Clone the repository and install dependencies:
git clone [email protected]:OpenBMB/ToolBench.git
cd ToolBench
pip install -r requirements.txtBefore training or inference, download and unpack the data so that data/ is under ToolBench/:
wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1XFjDxVZdUY7TXYF2yvzx3pJlS2fy78jk&confirm=yes' -O data.zip
unzip data.zipFor ToolEval alone, install:
pip install -r toolbench/tooleval/requirements.txtThe web interface is separate: clone chatbot-ui-toolllama, then run npm install and npm run dev.
How do you use this agent?
For the ToolBench RapidAPI service, obtain a key and set it first:
export TOOLBENCH_KEY="your_toolbench_key"
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
--tool_root_dir data/toolenv/tools/ \
--backbone_model toolllama \
--model_path ToolBench/ToolLLaMA-7b \
--max_observation_length 1024 \
--observ_compress_method truncate \
--method DFS_woFilter_w2 \
--input_query_file data/test_instruction/G1_instruction.json \
--output_answer_file toolllama_dfs_inference_result \
--toolbench_key $TOOLBENCH_KEYFor an OpenAI model, use --backbone_model chatgpt_function and provide --openai_key. For a personal RapidAPI account, provide --rapidapi_key and add --use_rapidapi_key. Open-domain inference additionally needs --corpus_tsv_path, --retrieval_model_path, and --retrieved_api_nums 5.
How does this agent compare with similar options?
The training code is based on FastChat. StableToolBench is presented as a local server based on simulated API responses; ToolEval can compare outputs from approaches such as ReACT and DFSDT, but the supplied material does not establish that these options are interchangeable across deployment scenarios.