Reinforcement Learning, Recursive Improvement, and On Practice

Imitation lets a system inherit the past. Reinforcement learning subjects it to the test of outcomes. Recursive improvement makes every round of practice transform the actor itself.

1. A blue line that crossed the human ceiling

In 2017, DeepMind published a seemingly simple chart in the AlphaGo Zero paper. It compressed much of the central problem of artificial intelligence into two curves.

The purple curve represents supervised learning. By studying games played by professional Go players, the model predicts what an expert would play in a given position. It starts quickly because thousands of years of openings, patterns, life-and-death problems, and tactical knowledge have already been compressed into the game records. The model does not explore from scratch; it begins from human knowledge.

But the purple curve soon plateaus.

The blue curve represents reinforcement learning through self-play. It begins far below, like a novice who does not yet understand Go. As self-play continues, however, it catches the purple curve, crosses it, and keeps climbing.

AlphaGo Zero learned from no human game records. It knew only the rules and the result of each game. The paper reports that after roughly 36 hours it surpassed AlphaGo Lee, the system that defeated Lee Sedol. After 72 hours of training, it beat AlphaGo Lee 100–0.

The chart raises a basic question:

Why could a system that never imitated human games eventually surpass the accumulated skill of human players?

2. “Most like an expert” and “actually strongest” are different goals

The objective of supervised learning can be sketched as:

$$\max_\theta P(a_{\text{human}}\mid s;\theta)$$

Given board state $s$, the model learns to predict the action $a$ that a human expert would choose. Its direct objective is to reproduce expert behavior more accurately, not to find the highest-win-rate strategy among every possible strategy.

This does not mean a supervised model is mathematically incapable of outperforming every individual in its data. It can combine many people’s experience and generalize beyond particular examples. The limitation is that an imitation objective ties the system to the distribution of expert data. The farther it travels from paths humans have already taken, the less supervisory signal it receives.

That is both the power and the boundary of imitation learning: it can compress the past efficiently, but it may not open the future.

Reinforcement learning changes the objective:

$$\max_\theta \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]$$

$\tau$ is the trajectory of a complete game and $R(\tau)$ is its final reward. The system stops asking, “What would a human expert play?” and asks instead, “Which actions are most likely to produce a win?”

The two objectives look similar, but the difference is fundamental:

One learns answers humans already know. The other tests every possible new answer against outcomes.

3. Move 37: leaving the distribution of human experience

In game two of AlphaGo’s 2016 match against Lee Sedol, AlphaGo played the now-famous Move 37.

It did not look like a good move. Professionals rarely played there, and commentators initially thought AlphaGo had made a mistake. Only as the game unfolded did its long-term value become clear.

In his lectures on large language models, Andrej Karpathy uses Move 37 to illustrate a defining property of reinforcement learning: it need not stay forever within the distribution of human data. If an unfamiliar path improves the win rate through repeated trials, the system can reinforce it even if that move never appeared among the standard answers of human experts.

Strictly speaking, Move 37 came from AlphaGo Lee, not the tabula-rasa AlphaGo Zero. AlphaGo Lee first learned from human records and then improved through reinforcement learning and search. Move 37 therefore does not prove that teachers have no value. It proves something subtler:

Learning can begin from human knowledge without treating known human methods as the boundary of all possible methods.

4. From reinforcement learning to recursive improvement

Learning in AlphaGo Zero was not a one-time event.

1. The current neural network produced move probabilities and a position value.
2. Monte Carlo tree search used the network as a starting point to find a stronger policy.
3. The system played itself using that improved policy.
4. Wins and losses supplied feedback about its position evaluations.
5. Search results and game outcomes trained the next network.
6. The stronger network returned to search and self-play.

The loop can be written as:

$$\pi_t \xrightarrow{\text{self-play}} D_t \xrightarrow{\text{training}} \pi_{t+1} \xrightarrow{\text{stronger self-play}} D_{t+1}$$

Policy $\pi_t$ produces new experience $D_t$, and that experience produces policy $\pi_{t+1}$. Each generation is both the result of the previous round of practice and the starting point for the next.

The recursive improvement now discussed in Silicon Valley and AI research goes one step further. Ordinary reinforcement learning primarily improves policy:

$$\pi_t \rightarrow \pi_{t+1}$$

A self-improving agent may change its policy, memory, tools, workflow, and even the improvement mechanism itself:

$$(\pi_t, M_t, T_t, E_t) \rightarrow (\pi_{t+1}, M_{t+1}, T_{t+1}, E_{t+1})$$

$M$ denotes memory, $T$ tools and workflow, and $E$ evaluation and improvement. The system no longer learns only how to do the current task better. It begins learning how to learn and improve more effectively next time.

5. The arrival of self-iterating agents

In 2025, Google DeepMind released AlphaEvolve. Large language models propose programs; automated evaluators run, verify, and score them. Better programs remain in a database and become starting points for later generations. AlphaEvolve has been used to optimize Google data centers, chip design, and AI training.

That year, researchers from Sakana AI and the University of British Columbia introduced the Darwin Gödel Machine. The coding agent can read and modify its own code, then use programming benchmarks to test whether the modified version is genuinely stronger. It independently added better file-viewing tools, patch verification, ranking across multiple candidates, and records of failed attempts. On SWE-bench, its performance rose from 20% to 50%.

The path toward the best agent was not monotonically upward. Some versions that performed worse in the short term later became the starting points for decisive advances. The Darwin Gödel Machine therefore keeps a diverse archive of agents rather than preserving only the current leader, allowing exploration to continue along several evolutionary branches.

In 2026, Meta’s HyperAgents pushed the idea up another level. It combined a task agent that solves problems with a meta-agent that edits agents inside one editable program. The meta-agent can modify both the task agent and its own method of improvement.

The distinction matters:

The system is not only searching for better answers. It is improving how it searches for better answers.

These systems are not the completed “intelligence explosion” of science fiction. They usually modify tools, workflows, and code only where results can be verified automatically, and they still depend on human-defined environments, evaluators, computing resources, and safety boundaries.

As of August 2026, Anthropic reported that Claude could take a leading role, from high-level instructions, in roughly 26% of Anthropic’s AI R&D work. Yet none of the task categories it measured could be performed fully autonomously.

What we are seeing is not completed recursive self-improvement, but its early engineering form.

6. Crossing the Chishui four times: online decisions in an open world

Move 37 mattered because it departed from conventional human play and revealed its value over a longer reward horizon. In real history, the Red Army’s four crossings of the Chishui River show a more complex process of strategic updating.

The campaign was not designed in advance as a complete four-move sequence. When the battle at Tucheng could not continue, the original plan changed quickly and the army withdrew across the Chishui. It later turned east after learning that northern Guizhou was lightly defended, crossed again, feinted toward southern Sichuan to force redeployment, and then suddenly crossed for a fourth time and broke south out of encirclement.

Its decision structure can be abstracted as:

$$\text{observe} \rightarrow \text{plan} \rightarrow \text{act} \rightarrow \text{change the situation} \rightarrow \text{observe again} \rightarrow \text{revise the plan}$$

Seen locally, repeated changes of direction can look indecisive or contradictory. Seen across the movement of opposing forces, changes in the operational space, and the final breakout, those conflicting moves form a continuous process of changing the environment, gathering new information, and replanning.

The campaign should not simply be called a historical Move 37. Move 37 was an unconventional action outside the distribution of human game records. The Chishui campaign was more like an agent operating in an open world: every action changed the opponent’s behavior, and the opponent’s response became new input for the next decision.

The analogy has clear limits. Go has fully knowable states, fixed rules, and an unambiguous win signal. War is an open system with incomplete information, changing opponents, and actions that cannot be replayed. A Go program can run millions of self-play games; real-world decisions often offer one chance.

They cannot be equated, but both expose the same principle:

High-level decision-making is not faithful repetition of actions that worked before. It is the continuous rewriting of the next move in response to a changing reality.

7. On Practice: knowledge is an iterative system, not an answer

Mao Zedong’s On Practice does not merely say that practice matters. It proposes a cyclical structure of knowledge:

Practice, knowledge, again practice, and again knowledge.

Knowledge emerges from practice and is summarized as theory. Theory returns to practice, where it is tested while transforming reality. New results then revise and develop prior knowledge, guiding another round of practice.

$$P_t \rightarrow K_t \rightarrow P_{t+1} \rightarrow K_{t+1}$$

$P_t$ is current practice and $K_t$ the knowledge formed from it. New knowledge guides practice $P_{t+1}$, whose results update knowledge to $K_{t+1}$.

This is not a circle that returns to the same place. If feedback is real and abstraction effective, each round should move the actor to a higher level of understanding.

From this perspective, On Practice rejects not just the absence of practice but two forms of learning that cannot iterate:

• Dogmatism copies existing theory without exposing it to feedback from a new reality.
• Empiricism accumulates local experience without abstracting it into knowledge that can guide the next round.

The former has imitation without updating; the latter has experience without a learning algorithm.

Its deepest similarity to recursive improvement is therefore not an emphasis on repeated attempts, but this:

Each round of practice not only solves the current problem; it transforms the actor that will solve the next one.

8. But who defines the reward?

Go is ideal for reinforcement learning because its rules are fixed and victory is clear. At the end of a game, the system knows exactly who won.

In most real-world problems, what counts as winning is itself part of the problem.

A recommender optimized for clicks may learn to produce outrage and addiction. A coding agent rewarded only for benchmark scores may exploit test loopholes. An organization focused only on near-term profit may recursively amplify its own short-termism.

Reinforcement learning does not automatically lead to truth. It efficiently searches for strategies that maximize the current reward. If the reward diverges from the real objective, recursive improvement can amplify a small misalignment into systematic error.

This is also the boundary that must remain when comparing machine learning with On Practice:

• Reinforcement learning optimizes a policy against a human-defined reward.
• On Practice concerns how people form, test, and develop knowledge while transforming the objective world through social practice.

They are not the same, but they share a structural view of learning: knowledge should not be a statically preserved answer. It should be a system that can receive feedback from reality, revise itself, and return to reality.

9. From imitating the past to improving how we improve

Karpathy compares the stages of large-language-model training to school. Pretraining resembles reading textbooks; supervised fine-tuning resembles studying worked solutions; reinforcement learning resembles solving exercises when you know only the problem and final answer and must discover the path yourself.

But “doing more exercises” does not capture the full meaning of recursive improvement. A higher-order learning system can change not only its answers but how it takes notes, which tools it uses, how it decomposes problems, how it checks errors, and how it selects the next exercise. It improves the system that produces answers.

From AlphaGo to self-iterating agents, from On Practice to the Chishui campaign, one thread connects them:

Imitation gets us where our predecessors arrived. Practice tests existing knowledge against outcomes. Recursive improvement turns every round of practice into the starting point for the next.

Teachers can show us roads already traveled, but cannot draw every road that has yet to exist. Innovation is not a refusal to learn. It is what happens after learning, when judgment is submitted to reality and the result rewrites the method of the next action.

Reinforcement learning frees a system from merely imitating what people did in the past. Recursive improvement lets it modify how it will think, act, and learn next time. On Practice describes the more general form of the same process:

Practice produces knowledge; knowledge returns to practice; and new practice changes knowledge itself.

---

10. References

1. David Silver et al., Mastering the Game of Go without Human Knowledge, Nature, 2017.
2. Andrej Karpathy, Deep Dive into LLMs like ChatGPT, 2025. The reinforcement-learning discussion begins around 2:10:59; AlphaGo and Move 37 around 2:42:07.
3. Google DeepMind, AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms, 2025.
4. Sakana AI, The Darwin Gödel Machine: AI that improves itself by rewriting its own code, 2025.
5. Meta AI, HyperAgents, 2026.
6. Anthropic, Measurements for understanding the pace of AI development inside frontier labs, 2026.
7. Anthropic, When AI builds itself, 2026.
8. Mao Zedong, On Practice, 1937.
9. People’s Daily Party History Channel, Crossing the Chishui Four Times (Part I).
10. Cyberspace Administration of China, Why did Mao call the Four Crossings of the Chishui his “proudest stroke”?.