Guide to Loop Engineering: How ‘autoresearch’ and ‘Bilevel Autoresearch’ Turn AI Agents Into Autonomous Machine Learning ML Research Loops

Most people still use AI like a 2015 search box. You type, you read, you type again. A newer pattern replaces that manual back-and-forth with a loop. This guide explains loop engineering using two verified artifacts. The sources are Andrej Karpathy’s autoresearch repository and the Bilevel Autoresearch paper. The framing follows a write-up by @0xCodila.

What is Loop Engineering?

To start, compare two modes. A prompt is one instruction, after which you decide the next step. A loop, by contrast, is a goal the model pursues until it arrives. The model plans, acts, checks its own result, then repeats. You define the objective once, and the loop handles iteration. Crucially, a loop only earns its cost when the work is measurable.

The Three Parts That Make A Loop Work

So what separates a real loop from a chatbot on repeat? Every reliable loop has three components.

1. A verifier grades each attempt. That check can be a passing test, a moving metric, or a build. Without a verifier, the agent simply agrees with itself on repeat.
2. State records what was tried, what failed, and what remains. A small side file lets the next run resume instead of restarting.
3. A stop condition prevents runaway cost. The loop halts when the goal is met, or after N attempts.

The Karpathy Loop: Inside ‘autoresearch’

These three parts are not theoretical. On March 7, 2026, Karpathy released autoresearch, an open-source repository under the MIT license. It ships three core files and about 630 lines of code. The project went viral within days and now sits near 90,000 GitHub stars. It was latter presented as the pattern “the Karpathy Loop.”

The design is deliberately small, yet strict. The agent edits only train.py, which holds the GPT model, optimizer (Muon and AdamW), and training loop. It cannot touch the evaluation utilities in prepare.py. That separation stops the agent from making the test easier instead of the model better. Meanwhile, a human writes program.md, the instructions the agent must respect.

Each cycle runs one experiment. The agent reads the code, proposes a change, trains five minutes, then keeps or rolls back. The scoring metric is val_bpb, validation bits per byte, where lower is better. That budget yields roughly 12 experiments per hour, so about 100 run overnight.

The reported outcomes are concrete. Karpathy pointed it at his already-optimized nanochat GPT-2 training code. It ran for two days and completed about 700 experiments, keeping 20 genuine improvements. Stacked together, those fixes cut GPT-2-quality training time by 11%, from 2.02 to 1.80 hours. One finding was a QK-Norm implementation missing a scalar multiplier, which had left attention too diffuse across heads.

Notably, humans tire after roughly a dozen experiments, whereas the loop does not. Separately, Shopify CEO Tobi Lütke ran autoresearch overnight on an internal model. He reported a 19% improvement after 37 experiments. Karpathy’s takeaway: if you have an objective metric, you are the bottleneck.

Prompt vs Loop vs Bilevel Loop

The differences become clearer side by side.

The Five Building Blocks

Consequently, AI engineering teams now assemble working loops from five reusable pieces:

• Automation fires the loop on a schedule, event, or trigger.
• A skill stores project knowledge in a markdown file, read on every run.
• Sub-agents split the writer from the reviewer, since one model grades itself too generously.
• Connectors let the loop act inside real tools, like an issue tracker or Slack.
• Finally, a verifier remains the gate that rejects bad work. Claude Code and Codex now ship all five.

Bilevel Autoresearch: A Loop On Top Of The Loop

Next, researchers asked a sharper question. If autoresearch is research, can you autoresearch autoresearch? The research paper Bilevel Autoresearch: Meta-Autoresearching Itself answers yes.

The inner loop matches Karpathy’s original: propose, train, evaluate, keep or discard. The outer loop watches the inner loop and reads its code and traces. It identifies where the search itself keeps stalling. Then it writes new Python mechanisms, injects them at runtime, and reruns the inner loop.

The result held on Karpathy’s GPT pretraining benchmark. The outer loop cut val_bpb 5x more than the single loop (-0.045 vs -0.009). Notably, both loops used the same LLM, so the gain came from architecture, not a smarter model. In practice the design splits into three levels. Level 1 runs the base loop. Level 1.5 tunes search parameters every five iterations. Level 2 generates mechanisms through a four-round session. The reported experiments used an RTX 5090 32GB and a 300-second budget.

The reason is worth noting. The inner loop kept returning to the same priors, even after they stopped working. The outer loop broke those patterns by forcing unfamiliar exploration.

Use Cases With Examples

These ideas transfer well beyond pretraining. For model work, a loop searches hyperparameters until val_bpb drops. For software, it refactors until tests, types, and the build pass. For content, it rewrites until every rubric score clears a threshold. For data, it tunes a pipeline until schema checks hold. Each case shares one trait: an automatic gate that can fail the work.

Try It Yourself: A Loop In One Prompt

Theory aside, you can feel the mechanic without Claude Code or Codex. Paste this into any capable model and watch it self-correct.

You will work in a loop until the task meets the bar.

TASK:
[describe exactly what you want produced]

SUCCESS CRITERIA (be strict):
- [criterion 1]
- [criterion 2]
- [criterion 3]

LOOP PROTOCOL, repeat every turn:
1. PLAN   - state the single next step.
2. DO     - produce or improve the work.
3. VERIFY - score the result 1-10 on each criterion. Be honest.
4. DECIDE - if every criterion is 8+, print FINAL and stop.
            Otherwise print ITERATING and fix the weakest point first.

RULES:
- Never call it done until every criterion is 8 or higher.
- Each pass must fix the weakest score from the last VERIFY.
- Do not ask questions. Make a sensible assumption and continue.
Begin.

Underneath, the control flow is small. The skeleton below shows those three parts in Python: a verifier, a decision, and two stop conditions.

current = baseline
best = evaluate(current)                 # verifier: lower val_bpb is better
for step in range(MAX_STEPS):            # stop condition 1: experiment budget
    candidate = propose_change(current)  # agent edits train.py
    score = train_and_eval(candidate)    # train 5 min, then verify
    if score < best:                     # keep only real improvements
        current, best = candidate, score # commit
    # else: discard candidate, restore baseline
    if best <= TARGET:                   # stop condition 2: goal met
        break

Both versions are limited. You are still the trigger, and closing the tab erases the state. Adding automation, a state file, and a real gate turns this into an autonomous loop.

See It Run

The interactive demo below animates one full loop: propose, train, verify, then keep or roll back. Adjust the target and step limit, and watch val_bpb fall until the stop condition fires.

Key Takeaways

• A loop needs three parts: a verifier, persistent state, and a stop condition.
• autoresearch lets an agent edit only train.py and never the evaluator.
• Karpathy’s overnight runs kept 20 fixes from 700 experiments, for an 11% speedup.
• Bilevel Autoresearch adds an outer loop for a 5x val_bpb gain.
• Loops shift the work to design and review; they do not remove thinking.