AlphaZero’s training is driven by a reinforcement learning process that iteratively improves the neural network parameters $\theta$ to minimize a specific loss function. This function combines error minimization for game outcome predictions with policy optimization to match the improved move probabilities generated by Monte Carlo Tree Search (MCTS). Acquisition of knowledge in chess by AlphaZero

The Loss Function

The network is trained on a composite loss function $l$ that sums three distinct components: mean-squared error (value loss), cross-entropy (policy loss), and weight regularization.

The formula is defined as:

$$l = (z - v)^2 - \pi^T \log p + c||\theta||^2$$

Here is the detailed breakdown of each component:

• Value Loss $(z - v)^2$: This term minimizes the error between the predicted value $v$ and the actual game outcome $z$.

  • $z$ (Actual Outcome): The game result from the perspective of the current player, encoded as $+1$ for a win, $0$ for a draw, and $-1$ for a loss. Unlike AlphaGo Zero, which assumed binary outcomes, AlphaZero optimizes for expected value to account for draws, which are common in Chess.
  • $v$ (Predicted Value): The network’s estimate of the expected outcome from the current board position $s$.

• Policy Loss $-\pi^T \log p$: This term aligns the neural network’s move probabilities with the stronger search probabilities found by MCTS.

  • $p$ (Raw Policy): The vector of move probabilities output by the neural network $f_\theta(s)$.
  • $\pi$ (Target Policy): The improved probability distribution over moves generated by the MCTS. The MCTS selects moves based on a combination of the network’s priors and actual simulation results, making $\pi$ a stronger policy than $p$. The network attempts to predict this MCTS distribution.

• Regularization $c||\theta||^2$: This is an L2 weight regularization term (controlled by parameter $c$) used to prevent overfitting and ensure the stability of the neural network weights.

The Training Loop

The training proceeds through a continuous cycle of self-play and parameter updates, distinct from previous iterations like AlphaGo Zero in its efficiency and continuous nature.

1. Self-Play Data Generation: The system plays games against itself using the latest parameters $\theta$. In each position $s$, an MCTS search is executed to determine the move probabilities $\pi$. The game proceeds until a terminal state $s_T$ determines the winner $z$.

2. Continuous Updates: Unlike AlphaGo Zero, which waited to evaluate a new model against the best previous version before adopting it, AlphaZero maintains a single neural network that is updated continually.

   1. It removed the evaluating against previous version part from AlphaGO

3. Optimization: The parameters are updated using stochastic gradient descent (SGD) on mini-batches of data (positions sampled from self-play games) to minimize the loss function $l$.

   • Hyperparameters: Training typically uses a large batch size (e.g., 4,096) and momentum (0.9). The learning rate is annealed (reduced) over time, starting at 0.2 and dropping eventually to 0.0002.
   • Exploration Noise: To ensure the system explores a wide variety of positions, Dirichlet noise is added to the prior probabilities in the root node of the search tree. This noise is scaled inversely to the number of legal moves typical for the game (e.g., chess has fewer moves than Go, so the noise scaling differs).

Raw Policy –> Improved Policy?

1. NN first outputs the probability vector taking in the state s.
   • To ensure system also explores, Dirichlet noise is added these probabilities at the root before the search begins
2. The system runs 800 simulations from this root node guided by the raw policy p. The search traverses the tree using a variant of the PUCT algorithm (Predictor + Upper Confidence Bound applied to Trees). For every step in the simulation, the algorithm selects the move that maximizes the following combination:$$ \text{Score} = Q(s, a) + U(s, a) $$

• $Q(s, a)$ (Exploitation): The mean value (expected win rate) of the move $a$ so far, derived from previous simulations that went down this path. (need to store these)
• $U(s, a)$ (Exploration): A value derived from the raw policy $p$. It is $$U(s, a) = C_{puct} \cdot P(s, a) \cdot \frac{\sqrt{\sum_b N(s, b)}}{1 + N(s, a)}$$
  • Here, $P(s, a)$ is the raw probability from the network.
  • $N(s, a)$ is the visit count (how many times this move has been tried in the search).

3. Once the 800 simulations are done are complete, the search stops. The “superior policy” $\pi$ is generated solely based on the visit counts of the root node, not the raw probabilities or the win rates directly. The probability assigned to each move $a$ in the superior policy $\pi$ is proportional to its visit count exponentiated by a temperature parameter $\tau$:

   $$\pi(a|s_t) = \frac{N(s_t, a)^{1/\tau}}{\sum_b N(s_t, b)^{1/\tau}}$$
   • Why is $\pi$ superior? The raw policy $p$ might miss a tactical trap that appears 5 moves later. The MCTS explores that future path, realizes the danger (lowering the $Q$ value), and stops visiting that node. Consequently, the final visit count $N$ for that blunder will be low, and the superior policy $\pi$ will assign it a near-zero probability, effectively “correcting” the network’s blind spot.
   • During training $\tau = 1$ is a good choice but during eval it would be $\tau \to 0$.

4. $(s_{t},\pi_{t},z)$ is stored in the replay buffer.

   • z is the final game outcome
   • $\pi_{t}$ is the probability distribution over the moves for this state.

Key Insight: The “Training Policy” is effectively using MCTS to create a “teacher” for the neural network. The network provides the intuition ($p$), MCTS does the heavy lifting to find a better move ($\pi$), and then the network is updated to make that better move its new intuition.

Some Other Details

• Structure: AlphaZero uses a deep residual neural network (ResNet) with 20 ResNet blocks forming the “torso” of the network.
• Heads: The torso branches into two specialized output heads: a policy head (predicting move probabilities) and a value head (predicting the game outcome).
• Input Representation: The network takes an $8\times8\times119$ tensor that encodes the current board position and the history of the last eight plies.
• Orientation: The board is always oriented from the perspective of the playing side.

Training Process

Self-Play: Training data is generated exclusively through self-play, where the system plays against itself without any human data.

Iterations: The network parameters are updated over a total of 1,000,000 gradient descent steps.

Optimization: Stochastic gradient descent is used with a batch size of 4096 and a synchronous decaying optimizer.

Learning Rate: The initial learning rate is 0.2, which is reduced by a factor of 0.1 at 100k, 300k, 500k, and 700k steps.

Data Handling & Exploration

Buffer: Positions are sampled from a self-play buffer containing the previous 1 million positions.

Sampling: To avoid overfitting, no more than 30 positions are sampled from any single game on average.

Checkpointing: The self-play networks are refreshed with the latest training network every 1,000 steps.

Search: Every move is decided using 800 Monte Carlo Tree Search (MCTS) simulations.

Game Diversity & Termination

Exploration Noise: Dirichlet(0.3) noise is added to 25% of all move priors to ensure variety during self-play.

Stochastic vs. Deterministic: Moves are sampled stochastically for the first 30 plies of a game and deterministically thereafter.

Early Resignation: While 20% of games are played to completion, 80% are terminated early if the value head predicts an expected score of 5% or less.

Game Cap: Maximum game length is capped at 512 plies.