I Trained My First Deep RL Agent to Land on the Moon

In my last post I wrote down all the theory from Unit 1. This time it’s the hands-on, and this is where reinforcement learning stopped being just words for me. You train an agent, a lunar lander, to land on the Moon. I did it, and the trained model is on the Hugging Face Hub as prem-ml/LunarLander-v3.

I want to go through my notebook the way I understood it, every piece of code, and connect each one back to the RL ideas from the first post. If the loop, the policy and the reward felt abstract before, this is the part that made them real.

First, a fake screen

The agent doesn’t need to see the game to learn, but the replay video does. A notebook has no screen, so we create a virtual one. That’s the whole reason for this block of installs.

!apt install swig cmake
!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit1/requirements-unit1.txt

!sudo apt-get update
!sudo apt-get install -y python3-opengl
!apt install ffmpeg
!apt install xvfb
!pip3 install pyvirtualdisplay

swig and cmake are needed to build Box2D, the physics engine behind Lunar Lander. The requirements file installs Gymnasium with Box2D, Stable-Baselines3 and huggingface_sb3. Then xvfb is the fake display, ffmpeg turns the frames into a video, and python3-opengl helps with rendering.

from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()

This starts the virtual screen. visible=0 means nothing is shown, it just exists in the background so the frames can be captured.

The imports

import gymnasium
!pip install pygame gymnasium[box2d] stable-baselines3 huggingface_sb3
from huggingface_sb3 import load_from_hub, package_to_hub
from huggingface_hub import notebook_login

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.monitor import Monitor

Each import is one job. Gymnasium gives the environment. PPO is the algorithm. make_vec_env runs many copies of the environment at once. evaluate_policy and Monitor test the agent. package_to_hub, load_from_hub and notebook_login are for sharing and loading models on the Hub.

Meeting the environment

Before training anything, the course makes you run the RL loop by hand with random actions. I liked this, because it’s the loop from my last post written as code.

import gymnasium as gym

env = gym.make("LunarLander-v3")
observation, info = env.reset()

for _ in range(20):
  action = env.action_space.sample()
  print("Action taken:", action)

  observation, reward, terminated, truncated, info = env.step(action)

  if terminated or truncated:
      print("Environment is reset")
      observation, info = env.reset()

env.close()

env.reset() gives the first state. env.action_space.sample() picks a random action, so right now the agent knows nothing. env.step(action) does the action and hands back the new observation and the reward, which is state, action, reward, next state, exactly the loop.

It also returns two flags. terminated means the episode really ended, like the lander landed or crashed. truncated means it was cut off, like a time limit. When either one happens, we reset and start a new episode.

What the lander sees, and what it can do

env = gym.make("LunarLander-v3")
env.reset()
print("_____OBSERVATION SPACE_____ \n")
print("Observation Space Shape", env.observation_space.shape)
print("Sample observation", env.observation_space.sample())

print("\n _____ACTION SPACE_____ \n")
print("Action Space Shape", env.action_space.n)
print("Action Space Sample", env.action_space.sample())

The observation has shape (8,), so it’s eight numbers. They are the lander’s horizontal and vertical position, its horizontal and vertical speed, its angle, its angular speed, and two true or false values for whether each leg is touching the ground.

The action space is discrete with four actions: do nothing, fire the left engine, fire the main engine, fire the right engine. That’s the discrete action space from my last post, now with real numbers.

Then there’s the reward, and this is the part I kept thinking about. The reward goes up when the lander is closer to the pad, moving slower and sitting level. It goes up by 10 for each leg touching the ground. It goes down a little every frame an engine fires, 0.03 for the side engines and 0.3 for the main one. Crashing costs 100 points and landing safely gives 100. A score of 200 or more counts as solved.

Nobody tells the lander how to land. We only write down what we care about, and it finds the rest.

That felt like the reward hypothesis in real life. The engine costs are basically a fuel bill, and the agent has to work out when spending fuel is worth it.

Sixteen landers at once

env = make_vec_env('LunarLander-v3', n_envs=16)

This creates 16 copies of the environment that run side by side. The agent gets more varied experience from each step of training, because 16 different games are happening at the same time.

Building the brain

model = PPO(
    policy="MlpPolicy",
    env=env,
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.999,
    gae_lambda=0.98,
    ent_coef=0.005,
    clip_range=0.2,
    verbose=1,
    seed=42
)

This one block is the agent. PPO stands for Proximal Policy Optimization, and the course describes it as a mix of the two approaches from Unit 1, learning a policy and learning values. MlpPolicy means a plain neural network, which fits because the input is just a vector of eight numbers. If the input were game frames, you’d use CnnPolicy instead.

The settings were the part I had to slow down for. Here is how I understood each one.

learning_rate

How big a step the network takes when it updates itself.

n_steps

How many steps each environment collects before an update.

batch_size

How many of those steps are used in one small update.

n_epochs

How many times it goes over the collected data per update.

gamma

The discount rate from the first post. 0.999 means it cares a lot about the future.

gae_lambda

A smoothing setting for estimating how good an action was.

ent_coef

A bonus for staying a bit random, which is a push toward exploration.

clip_range

Limits how much the policy can change in one update. That’s the “proximal” in PPO.

seed

Makes the run repeatable.

I liked seeing ent_coef in there. The exploration and exploitation tradeoff I wrote about before isn’t only a story, it’s a number you can change.

Training

from stable_baselines3.common.env_util import make_vec_env
env = make_vec_env('LunarLander-v3', n_envs=16)
env.reset()

model.set_env(env)

model.learn(total_timesteps=2500000)

model_name = "ppo-LunarLander-v3"
model.save(model_name)

I re-create the environment first to release a lock Box2D can hold, then attach it to the model with set_env. After that it’s one line, model.learn, with total_timesteps=2500000. That one line runs the whole RL loop millions of times. At the start the agent is basically random, so it crashes, and over time the rewards push the policy toward actions that land. Then model.save writes the trained agent to a file.

Reading the training logs

Because I set verbose=1, training prints a small table after every iteration. At first it looked like noise to me, but every line tells you something about how the agent is doing. Here is the very first one.

---------------------------------
| rollout/           |          |
|    ep_len_mean     | 91.6     |
|    ep_rew_mean     | -199     |
| time/              |          |
|    fps             | 2407     |
|    iterations      | 1        |
|    time_elapsed    | 13       |
|    total_timesteps | 32768    |
---------------------------------

ep_len_mean

The average length of an episode, in steps. Short episodes early on usually mean the lander crashes quickly.

ep_rew_mean

The average total reward per episode. It starts very negative, because a random lander crashes.

fps

Steps per second. It’s a speed check, not a score.

iterations

How many rounds of collecting data and then updating have happened.

time_elapsed

Seconds since training started.

total_timesteps

Total steps taken so far across all environments.

One thing I noticed is that 32768 is exactly 16 environments times 2048 steps. So one iteration is every environment playing 2048 steps, and my settings show up right there in the log. This first table also has no train/ part, because the agent hasn’t been updated yet. It has only collected experience.

By iteration 2, the table gets longer.

-----------------------------------------
| rollout/                |             |
|    ep_len_mean          | 88.8        |
|    ep_rew_mean          | -126        |
| time/                   |             |
|    fps                  | 2016        |
|    iterations           | 2           |
|    time_elapsed         | 32          |
|    total_timesteps      | 65536       |
| train/                  |             |
|    approx_kl            | 0.008190928 |
|    clip_fraction        | 0.0986      |
|    clip_range           | 0.2         |
|    entropy_loss         | -1.38       |
|    explained_variance   | 0.000366    |
|    learning_rate        | 0.0003      |
|    loss                 | 933         |
|    n_updates            | 10          |
|    policy_gradient_loss | -0.00772    |
|    value_loss           | 3.05e+03    |
-----------------------------------------

Now there’s a train/ part, which is about the learning itself.

approx_kl

How much the policy changed in the last update. A small number means a gentle, stable change.

clip_fraction

The share of samples where PPO had to clip the update. 0.07 would mean about 7%.

clip_range

The clip limit I mentioned earlier, 0.2.

entropy_loss

How random the policy still is, shown as a negative number. With 4 actions the most random it can be is about 1.39, so a value near -1.39 means the agent is still almost fully random. It moves toward 0 as the agent gets surer.

explained_variance

How well the agent predicts the returns it will get. 1 is perfect and 0 is no better than guessing.

learning_rate

The step size, 0.0003.

loss

The combined loss. It’s large early on mostly because value_loss is large.

policy_gradient_loss

The loss on the policy part, how much the policy wants to change.

value_loss

How wrong the agent’s value estimates are. It’s high early because rewards like -100 and +100 are big numbers.

n_updates

How many training epochs have been done so far. 10 is 1 finished training round times 10 epochs, which matches n_epochs=10.

The number I watch most is ep_rew_mean. It went from -199 to -126 after just one round of learning, so the lander was already crashing less, even though it’s still very negative. Around 200 means the lander is landing properly.

The other numbers say it’s still very early. entropy_loss of -1.38 is almost the most random a 4-action policy can be, and explained_variance near 0 means the value estimates haven’t started working yet. That’s normal at this point.

Early on the numbers are ugly. That’s fine, they’re supposed to be.

Checking if it actually works

eval_env = Monitor(gym.make("LunarLander-v3", render_mode='rgb_array'))
mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)
print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")

The course says to never evaluate on the training environment, so this makes a fresh one. Monitor tracks the rewards, and evaluate_policy plays 10 full episodes. deterministic=True means the agent always picks its best action instead of drawing from probabilities. That’s the deterministic and stochastic policy idea from the first post, now with a real switch.

The course counts the result as mean reward minus standard deviation, and it has to be 200 or more to pass. Mine works out to 275.31, which is above the line.

Sharing it on the Hub

notebook_login()
!git config --global credential.helper store

You make a token with write access on the Hugging Face site and paste it here. Then comes the upload. This part has my own workaround in it, because I recorded the replay video myself and handed it to the upload function.

import glob
from stable_baselines3.common.vec_env import VecEnv, DummyVecEnv, VecVideoRecorder

env_id = "LunarLander-v3"
model_architecture = "PPO"
repo_id = "prem-ml/LunarLander-v3"
commit_message = "Upload PPO LunarLander-v3 trained agent with working video"

base_env = DummyVecEnv([lambda: Monitor(gym.make(env_id, render_mode="rgb_array"))])
eval_env = VecVideoRecorder(
    base_env,
    video_folder="/tmp",
    record_video_trigger=lambda step: step == 0,
    video_length=2000,
    name_prefix="real-lander-replay"
)

obs = eval_env.reset()
for _ in range(2000):
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = eval_env.step(action)
eval_env.close()

recorded_videos = glob.glob("/tmp/real-lander-replay-step-0-to-*.mp4")
actual_video_path = recorded_videos[0] if recorded_videos else ""

class RealVideoRecorder:
    def __init__(self, path):
        self.path = path

gym.Env.video_recorder = RealVideoRecorder(actual_video_path)
VecEnv.video_recorder = RealVideoRecorder(actual_video_path)
DummyVecEnv.video_recorder = RealVideoRecorder(actual_video_path)

package_to_hub(
    model=model,
    model_name=model_name,
    model_architecture=model_architecture,
    env_id=env_id,
    eval_env=eval_env,
    repo_id=repo_id,
    commit_message=commit_message
)

Here is what it does, step by step. First it records the trained agent playing for 2000 steps into a video file. Then it finds that file on disk and puts its path on the classes that package_to_hub looks at, so the upload picks up the real recording. The last call, package_to_hub, evaluates the agent, writes a model card, and pushes everything to the repo.

Loading it back

!pip install shimmy

from huggingface_sb3 import load_from_hub
repo_id = "prem-ml/LunarLander-v3"
filename = "ppo-LunarLander-v2.zip"

custom_objects = {
            "learning_rate": 0.0,
            "lr_schedule": lambda _: 0.0,
            "clip_range": lambda _: 0.0,
}

checkpoint = load_from_hub(repo_id, filename)
model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)

This is how anyone can download my agent. load_from_hub takes the repo and the file name and fetches the zip. PPO.load rebuilds the model from it. The custom_objects dictionary is a compatibility trick for models saved with a different Python version, and shimmy helps older Gym models run on Gymnasium.

What I took from it

Before this, I knew the words: agent, policy, reward, return. After this, I’ve seen them as lines of code. The environment is gym.make, the policy is the neural network inside PPO, and the reward is that list of rules about legs, engines and crashes.

The part I keep coming back to is how little I told the agent. I never wrote a single rule about landing. I wrote what counts as good and what counts as bad, and a neural network worked the rest out. I’m still thinking about what that means for everything else we might teach this way.

The notebook ends with some challenges: train for more steps, try different PPO settings, try another algorithm like DQN, or try a different environment. The course goes on to Q-Learning next, where the table I mentioned last time finally shows up.

The model is on the Hub at prem-ml/LunarLander-v3 and the full notebook is in my GitHub repo.