How I Plan to Climb the Lunar Lander Leaderboard

After I uploaded my first Lunar Lander agent, I went to the Hugging Face leaderboard to see where I stood. My older model, lunar_lander_v3, is at rank 5764, with a mean reward of 258.99, a std of 16.28 and a result of 242.71. It passes the course’s 200 line, but thousands of agents are above it.

So this post is me thinking out loud about what I can do to move up. No code today, just the options in plain words.

What the score really is

The result isn’t just the mean reward. It’s the mean minus the std, so 258.99 minus 16.28 gives 242.71. That means there are two ways up: raise the average landing score, or make the landings more consistent.

The agent at rank 5 on the board has a mean of 370.25 and a std of only 8.94, so its result is 361.31. That agent lands almost the same way every time. I hadn’t thought about it like that.

One amazing landing doesn’t help much if the next one crashes.

Train it for longer

This is the simplest option. My agent has only seen a limited number of steps, and more steps means more practice. The cost is time, and the gains slow down after a while, so I don’t think this alone fixes everything.

Keep the best version, not the last one

While an agent trains, its score goes up and down. The final model isn’t always the best one it ever was. Stable-Baselines3 can test the agent during training and save the best version, which costs almost nothing, so I like this one a lot.

Change the settings, but only one at a time

There are a few settings I want to look at: a learning rate that shrinks over time, a bigger batch size, fewer epochs, and a smaller entropy bonus. All of them affect how steady the agent is. If I change five things at once I’ll never know which one helped, so it has to be one at a time.

Test with more episodes

I evaluated my agent on only 10 episodes, which is a small test. The score can look better or worse just by luck. Since the std is part of the result, I need a fair number for it, so I’d evaluate on 100 episodes before comparing anything.

Try different seeds

Same settings, different random start, and you get a different agent. I could train a few and keep the best one. It feels like the exploration and exploitation tradeoff again: keep improving one agent, or start fresh and see what happens.

Try a different algorithm

PPO isn’t the only option. The course mentions DQN, for example. I’d keep this for last, because it means learning something new, not just tuning what I already have.

What I’m not going to chase

At the very top of the board there are scores like a mean of 2000. I can’t explain that with a normal evaluation, so I’m not treating it as the target. A realistic goal is the 300 to 400 range, where a lot of regular PPO agents sit.

Funny thing, this is the RL loop again. I’m the agent, the leaderboard is the reward, and every experiment is an action. I don’t know which action gives the biggest reward yet, and the only way to find out is to try them.