Jev Drone
A MuJoCo drone testbed combining onboard vision, conventional flight control, and low-rate model judgment for autonomous obstacle courses.
Per-dimension scores and reasoning
The evidence confines control to a MuJoCo simulation, places the model in an advisory layer, and gives the 50 Hz safety reflex and measured climb limits veto authority. The README also describes camera-to-symbolic-JSON flow and discloses API calls, video, and replay effects, so least privilege, data-flow transparency, and external-effect handling are meaningful but incomplete. Deductions apply because the autonomous loop has no per-action user confirmation; key handling is limited to copying an environment file and sourcing variables, with no evidence of redaction, leak-resistant logging, or revocation guidance; and dependencies have only minimum versions with no lockfile, hashes, audit, or vulnerability process. No-Jev operation, caching, and offline replay provide useful fallback or reversibility, but not formal state rollback. TypeSafe and MuJoCo Menagerie are attributed and the separate model license is acknowledged, while the copyright holder remains the generic 'authors' and publisher responsibility is unclear.
The README gives a largely coherent account of control rates, responsibility boundaries, call suppression, safety overrides, failure cases, and experimental caveats. Deductions reflect varying descriptions of the judgment rate as about 2.5 Hz and about 3 Hz, plus both 77.5 m and a later 88 m result without explicit version mapping. requirements.txt identifies the main packages, but open-ended minimum versions and reliance on an externally fetched model and TypeSafe service weaken dependency availability. Numerous failure modes and diagnostic lessons are documented, yet there is no source-level evidence of actionable exception messages, retry policy, timeout behavior, or recovery messaging.
The intended setting is clearly defined as camera-only quadrotor autonomy in MuJoCo, with an obstacle course and a separately identified partial tunnel experiment. Model-enabled, model-disabled, video, replay, and trace modes cover the principal research scenarios. Capability boundaries are unusually explicit: Jev does not consume pixels, perform perception, or operate at control rate; it makes tactical judgments while ordinary code owns safety. The incomplete tunnel outcome is also labeled honestly. Trigger rules are precise, including scene fingerprinting, no-call conditions, and lost-target gating. GLFW/EGL guidance and API-free replay improve environment fit, but operating-system, hardware, Python-version, fully headless setup, and adaptation to other vehicles or courses are not documented.
The README has strong information architecture across concept, course, results, limitations, execution, file layout, and engineering lessons; the file-responsibility table is particularly clear. Installation and usage commands are useful, but setup.sh and .env.example are absent from the supplied evidence, so their behavior cannot be checked, and platform prerequisites are incomplete. Names are internally consistent but carry no public-interface stability promise. Examples and troubleshooting material are extensive, and known limitations are handled thoroughly through disclosure of the single Jev run, variance, earlier negative comparison, and incomplete tunnel experiment. The complete MIT text is present and the third-party model license is flagged. There is no versioning scheme, release history, or changelog, and no identifiable maintainer, support route, security contact, or ownership of future updates.
The described outputs include flight metrics, station telemetry, optional video, judgment traces, and tape-based replay without further API calls, making the product's outputs potentially useful for analysis and demonstration. The Jev layer adds a climb maneuver that the stated greedy baseline cannot represent, with an ablation supporting some marginal value. Full credit is withheld because the main successful Jev column is one 65-second run, an earlier three-seed comparison found no advantage, and the tunnel behavior remains unreliable. Calls, latency, token usage, and a free ablation path are quantified, but monetary price, hardware demand, setup time, and scaled operating cost are not.
The README ties claims to rates, latencies, call counts, telemetry excerpts, hand-built scene judgments, and an ablation, while carefully narrowing the conclusion it claims to support. Deductions apply because no raw logs, test files, experiment-script contents, per-seed Jev records, or machine-checkable result artifacts are supplied; most empirical evidence is self-reported in one document. LICENSE and requirements.txt corroborate licensing and dependency claims only, not flight performance. Fact and inference are separated exceptionally well: perception, control, safety, and model contributions are distinguished, and single-run evidence, negative results, and partial experiments are explicitly labeled.
- The principal Jev success result is a single 65-second run and does not establish stable benefit across seeds, environments, or physical aircraft.
- Dependencies use unbounded minimum versions without a lockfile or hashes, so installations may drift and supply-chain security is not demonstrated.
- An API key is placed in and sourced from a local .env file, but the supplied material shows no log redaction, permission guidance, rotation, or leak-response procedure.
- Do not generalize the simulated safety reflex, zero-collision record, or advisory architecture into a safety assurance for real drones.
- setup.sh, .env.example, implementation code, and raw experiment artifacts were not supplied, so runtime behavior, error handling, and numerical claims could not be statically cross-checked.
- The MuJoCo Menagerie Skydio X2 model has a separate license whose exact terms should be reviewed before redistribution or reuse.
What does this agent do, and when should you use it?
Jev Drone is a MuJoCo quadrotor experiment in which a Skydio X2 flies a five-station obstacle course using only onboard depth and segmentation imagery. At 15 Hz, flight.py converts camera buffers into a symbolic scene; tactics.py then calls TypeSafe Jev at roughly 2.5–3 Hz to choose among tactical actions such as going around, climbing, braking, or reacquiring the target. Attitude, thrust, and motor mixing remain in a 500 Hz geometric controller, while run.Guidance retains a 50 Hz safety veto, so the model neither consumes pixels nor owns low-level safety. run.py executes episodes and records metrics, while video runs produce an MP4 and a tape.npy file that replay.py can recut offline. The repository reports one collision-free, full-course Jev run, but that evidence is a single 65-second trial and an earlier three-seed comparison showed no advantage, making this a research prototype rather than a validated real-world flight system.
run.py starts the MuJoCo course, Skydio X2 dynamics, moving target, episode loop, and metric collection. flight.Eye processes depth and segmentation buffers into a compact JSON scene containing five forward range sectors, obstruction-top information, target bearing, and visibility; perception does not use simulator ground truth. When code determines that judgment is needed and the scene fingerprint has changed, tactics.py calls Jev once for maneuver, risk, and target_truly_lost; unchanged scenes reuse the previous answer. flight.Pilot runs geometric control, thrust-priority mixing, and slew-limited commands at 500 Hz, while run.Guidance applies a 50 Hz hard reflex and rejects climb proposals that fail measured clearance constraints. A run can emit console telemetry and metrics; --video additionally creates course.mp4 and course.mp4.tape.npy, which replay.py can render again without physics or API calls. The separate tunnel.py maps two Score judgments directly to steering and altitude commands, but its 620 m tunnel experiment is explicitly documented as partial and not reliably completed.
- Robotics researchers evaluating a layered architecture that places low-rate model judgment above high-rate conventional control.
- MuJoCo developers reproducing camera-only target tracking, obstacle traversal, and safety-reflex behavior with a quadrotor.
- Safety-oriented autonomy teams studying how model proposals can remain advisory and subject to measurable code-level vetoes.
- Experimenters comparing Jev tactical choices against a greedy wider-side steering ablation.
- Simulation engineers investigating camera-coordinate errors, depth clipping, quadrotor mixing, inverted-flight recovery, or lost-target estimation.
- Demo creators who want to record one paid flight and repeatedly recut its telemetry footage without further model calls.
What are this agent's strengths and limitations?
- The responsibility split is explicit: Jev handles tactical judgment at about 3 Hz, while 15 Hz perception, 50 Hz safety, and 500 Hz flight control remain ordinary code.
- A measurable safety layer can veto the model, including checking obstruction-top height and the aircraft's climb ceiling before accepting climb.
- On-demand calls and scene fingerprinting avoid repeated requests in open corridors or unchanged situations; measured median latency is also reported.
- The repository includes a no-Jev ablation, operational metrics, recorded video, and replay.py for analysis without rerunning physics or spending additional API credits.
- It documents concrete simulator and controller failures involving camera axes, depth clipping, attitude error, motor mixing, and world-frame target prediction.
- The tactical path depends on TypeSafe Jev and its API credentials; no alternative provider, offline model, or standard protocol adapter is documented.
- The main success evidence is one 65-second Jev run rather than a seed-matched average, and an earlier three-seed trial showed no advantage over baseline.
- The demonstrated system is a MuJoCo Skydio X2 simulation; no real-aircraft deployment, hardware interface, or operational safety certification is provided.
- Jev runs consume API usage: the reported result used 80 calls and 96k tokens, but monetary pricing and a budget limit are not documented.
- The separate tunnel experiment is not yet reliable and can over-commit a dodge until the aircraft ends up against a wall.
- Setup fetches an external airframe asset and requires network access plus a working GLFW or EGL environment, while exact Python and MuJoCo versions are not stated.
How do you install or deploy this agent?
From the repository root, run ./setup.sh to create the virtual environment and fetch the Skydio X2 model from MuJoCo Menagerie. Then run cp .env.example .env and place the required TypeSafe Jev key in .env. Load it with set -a && . ./.env && set +a. Set export MUJOCO_GL=glfw for a displayed run, or use egl on a headless machine. The supplied material does not specify supported operating systems, a Python version, or a complete system-package list.
How do you use this agent?
For a first Jev-enabled episode, run .venv/bin/python run.py --seconds 65 --seeds 1. Run the free three-seed ablation with .venv/bin/python run.py --no-jev --fast --seconds 65 --seeds 0 1 2. Create telemetry video with .venv/bin/python run.py --seconds 65 --seeds 1 --video course.mp4, or prefix a run with TRACE=1 to print every judgment. A video run also creates course.mp4.tape.npy; recut it offline with .venv/bin/python replay.py course.mp4.tape.npy out.mp4 --from 12 --to 30. Review the three questions, option rubrics, and reaction thresholds at the top of tactics.py before changing judgment behavior.
How does this agent compare with similar options?
The documented alternative is the same stack with Jev disabled and a greedy “steer toward the wider side” heuristic substituted. Across three baseline runs it remained collision-free but stopped at station 2 at 17.7 m because it could not represent flying over the low beam. The reported single Jev run completed the 77.5 m course, kept the target visible 82% of the time, spent 9% of the run pinned in the reflex layer, and also had zero collisions. This is not a seed-matched statistical result: an earlier, simpler three-seed comparison found no advantage for Jev.