PaperJury
A pre-submission AI reviewer that first adjudicates whether each critique is valid, then revises your paper under deterministic guardrails — not every AI comment gets applied.
Evidence shows explicit human sign-off rules ('never edit without explicit author sign-off', auto entered only via /goal or config), a local no-backend architecture, an append-only revert journal (journal.js revert covered by tests), and full MIT/arXiv/CITATION.bib attribution — hence full marks for user_confirmation, rollback, source_attribution. Deducted: the deterministic guard scripts under scripts/ are not present in the evidence files, so least_privilege, external_effects, and data-flow claims rest on README assertions; the update check makes an outbound GitHub request by default (opt-out exists), and cloud-side handling of manuscripts is outside the project's control, so sensitive_data_handling and dependency_security score 2.
Evidence shows a clear node>=18 requirement, npm run doctor health check, honest degradation to compiled:null without LaTeX, and tests covering CRLF, revert round-trips, and ambiguity rejection — dependency_availability earns 3. Deducted: the tested script sources (apply-patch, compile-guard, etc.) are absent from the evidence, so implementation-vs-docs consistency cannot be statically confirmed (self_consistency 2); failure messages are only visible as regexes inside test assertions, not as user-facing text (failure_messages 2).
Mode triggers are precisely specified (auto is explicitly never self-detected), and capability boundaries ('does not replace peer review', no fabricated experiments) are declared — trigger_precision and capability_boundaries earn 3. Deducted: the real audience is locked to Claude Code users working on LaTeX/Markdown papers, with limited portability (a separate Codex port exists) — audience_and_scenarios 2; the product is bound to Claude Code skill/workflow/memory primitives, environment_fit 2.
Information architecture, dual-path install notes with doctor and a disable env var, a shipped dogfood sample, FAQ, honest limitation statements, and MIT license all earn 3. Deducted: no CHANGELOG file; version history is scattered in the News section with v1.0.0 vs v1.2.1 requiring reader reconstruction — versioning_changelog 2; naming (engine v3) is stable but layered — naming_stability 2; maintenance responsibility is implied only by four named authors with no governance or maintenance statement — maintenance_responsibility 1.
Outputs (issue ledger, reviewable patches, verification report) are clearly defined and backed by a real dogfood sample; cost figures (2.47 hours, 6.76M tokens/paper) are concrete. Deducted: usability, marginal value, and cost-benefit all rest on self-reported README/paper numbers that static review cannot execute or verify; baseline comparisons (F1 0.656, ESVR 0.025) are uncorroborated here, so all three criteria score 2 rather than 3.
Every major claim points to the arXiv paper, the dogfood RUN_REPORT, or code tests; tests distinguish snapshot regression from behavioral assertions, and compile results explicitly separate compiled:true/null. Deducted: key script sources are not in the evidence and sample file contents were not provided, so headline numbers cannot be independently corroborated within this repository — all three criteria score 2: traceable but not independently re-checkable here.
- Static review only; nothing was executed. All performance figures (F1, ESVR, cost) are self-reported — verify against the arXiv paper and dogfood sample before relying on them.
- The deterministic guard scripts under scripts/ were not part of the reviewed evidence; safety boundaries rest mainly on README descriptions. Run npm run doctor and read the scripts before use.
- Although auto mode is declared explicit-only, avoid granting broad tool permissions on projects containing sensitive or unpublished manuscripts.
- The update check contacts GitHub by default; set PAPERJURY_DISABLE_UPDATE_CHECK=1 for fully offline use.
- No CHANGELOG and no maintenance statement; long-term upkeep and security response are uncertain.
What does this agent do, and when should you use it?
PaperJury, from the Spark-To-Paper-Skills/paperjury repository, is a Claude Code skill for pre-submission self-review of research papers. It organizes review as a closed loop — review → verdict → revise → verify — where N domain reviewers read the full manuscript, disputed critiques go to an isolated 5-member jury (escalating to 12 on no clear majority), and every issue ends in one of three verdicts: valid-fixable, author-required, or invalid-drop. Models handle reading, judgment and drafting, while deterministic scripts in scripts/ manage the ledger, patch application, anchor tracking, LaTeX compilation and submission-compliance checks. Outputs include an evidence-backed issue list, reviewable minimal patches and a verification report; the repo ships a dogfood sample with before/after PDFs and a human-audited run report. All project files stay in your local paper project — the tool has no backend server.
Runs as a Claude Code skill, auto-discovered via SKILL.md. Its pipeline is assign-reviewers → reading-check → coverage-auditor → merge → {trial ‖ polish} → recall-audit → drafter → {edit-audit | meaning-audit} → clerk: 2–4 (default 3) field reviewers read the whole paper and report weaknesses with verbatim citations and confidence; mechanical issues go through the polish fast path while major disputes enter a 5/12-member trial; a judge agent routes upheld issues as valid-fixable or author-required and sets a close_criterion. Deterministic scripts include decompose, extract-docx (one-shot Word→Markdown), ledger, apply-patch, anchor-diff, cross-ref, compile-guard (real LaTeX compilation; degrades to structural lint with compiled:null without a toolchain), compliance-check (anonymization leaks, margins, page limits) and the doctor self-check. Three modes are supported: direct-edit, review and auto; the first two confirm every patch, while auto requires explicit mode: auto configuration plus a /goal.
- A researcher with a fresh first draft: have the panel read it like reviewers and separate fatal flaws from minor fixes.
- An author doing a final pre-submission check: verify claims aren't overstated, whether experiments support conclusions, and run the submission-readiness compliance screen.
- An author tweaking one paragraph: in direct-edit mode say "tighten this section but don't change the claims" — the patch is drafted, confirmed by you, then applied without ballooning into a rewrite.
- An author needing unattended multi-round revision: explicitly enable auto mode with a verifiable /goal; safe edits apply automatically, high-risk ones return to you.
- An author with a Word (.docx) paper: it converts once to a Markdown working copy, reviews it fully, never touches the original, and leaves merging back to you.
What are this agent's strengths and limitations?
- Adjudication before revision: every critique passes an isolated jury trial and receives one of three explicit verdicts; blind expert agreement reached 0.887 on final verdicts and 0.913 on routing decisions.
- Deterministic guardrails cap edit risk: frozen anchors, per-segment edit limits, edit/meaning audits and recall-audit keep the unsafe-edit rate (ESVR) at 0.025 — about 4.4× lower than the strongest baseline.
- Verifiable evidence: evaluated on 12 held-out papers against four baselines (arXiv:2606.16322), with a bundled dogfood sample (152 critiques deduplicated to 55 issues: 26 safely applied, 10 author-required, 19 dropped) whose revised draft compiled with 0 errors / 0 warnings.
- Real compilation and compliance: runs LaTeX locally and reports errors, undefined references, overfull boxes and page counts; scripts screen for anonymization leaks, documentclass drift and other submission risks, and it never claims verification it cannot perform.
- Claude Code ecosystem dependency: core operation relies on the Claude Code skill mechanism and runs models through your own session, whose terms govern content handling; Codex users must switch to the separate paperjury-codex repository.
- Non-trivial runtime cost: the paper reports ~2.47 hours and 6.76M tokens per paper — far above a forward-only rewriter's 0.31 hours — and full trial depth may be excessive for quick checks (quick mode is still on the Roadmap).
- Explicitly not a substitute for peer review or author judgment: issues requiring new experiments, missing evidence or research judgment are always returned to the author, with no conclusions produced by the tool.
- Degraded without a LaTeX toolchain: real compilation and layout checks become unavailable, leaving only structural lint with reported compiled:null.
How do you install or deploy this agent?
Option 1 (Claude Code plugin, recommended): run /plugin marketplace add Spark-To-Paper-Skills/paperjury then /plugin install paperjury@Spark-To-Paper-Skills inside Claude Code. Option 2 (clone as a skill): git clone https://github.com/Spark-To-Paper-Skills/paperjury ~/.claude/skills/paperjury (Windows PowerShell: $env:USERPROFILE\.claude\skills\paperjury), or place it under <project>/.claude/skills/ for project-only scope. Prerequisites: node is required (deterministic checks run in Node); a LaTeX toolchain is optional — without it, compilation checks degrade to structural lint. After installing, run npm run doctor in the skill directory to verify repo integrity, required tools and manuscript detection.
How do you use this agent?
After installation, state your needs in natural language inside your paper project — no commands to memorize. "Please review the paper, focusing on whether experiments support the claims" starts a review; "Tighten this introduction paragraph without changing the claims" uses direct-edit; "Run submission-readiness / compliance check" triggers the deterministic format screen plus compile-driven layout checks. Unattended auto mode requires both mode: auto in configuration and a /goal; enabling neither will not start a multi-round loop. See docs/AGENT-GUIDE.md for deeper agent-driving instructions.
How does this agent compare with similar options?
The paper benchmarks four baselines: a forward-only rewriter (ESVR 0.240), an LLM critic (F1 0.446), an LLM-as-judge loop (F1 0.519, ESVR 0.110) and a naive unbounded generator (8.37 hours/paper); PaperJury beats all issue-generating baselines on F1 (0.656) on every one of the 12 papers and cuts unsafe edits to 0.025. Mechanically, the README credits PaperSpine as the inspiration for its spine and anti-drift design, but PaperSpine focuses on motivation-driven drafting and rewriting, whereas PaperJury centers on adversarial review and trial-style adjudication.