Agent Memory Benchmark (AMB)

The README describes document ingestion, retrieval, generation, judging, result storage, and the Gemini key environment variable, so data flow is partly visible. Least privilege, sensitive data handling, and external effects score 1–2. The supplied material shows no confirmation step, permission boundaries, key protection details, or rollback process; user_confirmation and rollback score 0, and related safety criteria receive no higher marks. Dependencies have minimum versions and one compatibility note but no security review evidence, so dependency_security scores 1. The README names the upstream PrecisionMemBench project but gives no maintainer or source governance details. Publisher identity is unknown; source_attribution scores 1.

README usage generally matches the command entry point, package name, and Python requirement in pyproject.toml, so self_consistency scores 2. Dependencies have minimum versions and a limited compatibility explanation, but no lockfile or availability guarantees, so dependency_availability scores 1. The files do not describe failure messages or error handling, so failure_messages scores 0.

The README describes conversation memory, document research, and cross-tool tasks, and gives examples for datasets, memory providers, modes, and query limits; audience_and_scenarios scores 2. It distinguishes ordinary answer evaluation from PrecisionMemBench retrieval mode, but does not systematically define capability boundaries or triggering rules, so capability_boundaries and trigger_precision score 1 each. Python, environment variables, the MemBench data path, and command line steps are documented, so environment_fit scores 2.

The README is organized into overview, problem, evaluation flow, setup, usage, results, and requirements, and includes command examples. Setup and naming guidance are clear enough for 2 in install_notes, naming_stability, and examples_and_faq. It gives only limited discussion of benchmark shape and cost rather than a complete limitations list, so known_limitations scores 1. The supplied files contain no license, changelog, or maintenance owner evidence, so license, versioning_changelog, and maintenance_responsibility score 0. Information architecture is assessable only from the README, so it scores 1.

The tool offers dataset and provider listings, statistics, benchmark runs, result files, and browser viewing, making outputs reasonably usable; output_usability scores 2. The README gives a concrete motivation about benchmark discrimination with million-token context windows and identifies accuracy, speed, and token cost as measures, so marginal_value scores 2. It emphasizes cost measurement but supplies no concrete estimate of run costs or resource needs, so cost_benefit scores 1.

The README names the evaluation flow, modes, result path, and materials it says are needed for reproduction, and identifies the PrecisionMemBench upstream project; this makes some claims traceable, so claim_traceability scores 2. The available evidence is limited to README and project configuration. The configuration corroborates the entry point, dependencies, and Python requirement but cannot independently establish results, dataset contents, or reproducibility claims, so cross_source_corroboration scores 1. The README separates the evaluation steps from PrecisionMemBench retrieval scoring and presents performance statements as project motivation; however, the claims cannot be independently checked from the supplied material, so fact_inference_separation scores 2 rather than full marks.