I'm Starting to Learn Deep Reinforcement Learning

Introduction

I've been reading about AI for a while, but I always skipped one part: reinforcement learning. I heard the name everywhere and never understood it. So I'm starting from zero now.

The best resource I found is the Hugging Face Deep RL course. It's free and it goes step by step. This post is me writing down what I learned in the first unit, in my own words, so I actually understand it and don't just copy it.

What is Reinforcement Learning?

Think about how you learn to ride a bike. Nobody gives you a list of correct movements. You try, you fall, you adjust, and slowly you get better.

That's reinforcement learning. A learner tries things, sees what happens, and uses that to do better next time. In the course's words, it's learning from interaction with the environment. The learner is called the agent. It learns by trial and error, and it gets rewards for good results and negative rewards for bad ones.

The idea behind it is called the reward hypothesis. It says every goal can be described as maximizing the total reward we collect. That sounds too simple to me, and I'm not sure it fits everything in life. But for training an agent, it works.

The Reinforcement Learning Framework

This is the part with all the terms. Once I understood each one, the rest became easier.

The agent is the learner or decision maker. The environment is the world the agent lives in. The state is what the agent can see about the world right now. An action is something the agent can do. A reward is the feedback the environment gives after an action.

It runs as a loop. The agent sees a state, takes an action, and the environment gives back a new state and a reward. Then it repeats.

A small thing I learned: the state and the observation are not always the same. In chess you see the whole board, so state and observation are the same. In a game like Super Mario, you only see part of the level, so you get an observation, not the full state.

The agent's goal isn't to get the biggest reward right now. It's to get the biggest total reward over time. That total is called the return, or cumulative reward.

There's also the discount rate, called gamma. Rewards that come sooner count more than rewards that come later, because the future is less certain. A gamma close to 1 means the agent cares about the future. A gamma close to 0 means it only cares about what's right in front of it. I like this idea a lot, because people do the same thing.

The policy is the agent's brain. It tells the agent which action to take in each state. The whole point of RL is to find the best policy, the one that gives the most reward.

The type of tasks

There are two kinds of tasks.

An episodic task has a start and an end. A game level is a good example. It begins, it ends, and then you start over.

A continuing task never ends. Think of an agent trading stocks, which just keeps going. It has to learn without a clear finish line.

The Exploration/Exploitation tradeoff

This one is my favourite, and it sounds philosophical to me.

Exploration means trying new things to find out more about the environment. Exploitation means using what you already know to get the reward you know about.

Picture a restaurant. You've found one place you like. You can keep going back, which is exploitation, and you'll always get a good meal. Or you can try new places, which is exploration, and you might find something better or get a bad meal. If you only exploit, you never discover anything better. If you only explore, you never enjoy what you found.

So the agent has to balance both. I realised this isn't only a machine problem. We all deal with it when we pick a career, a habit, or a path.

The two main approaches for solving RL problems

We want the agent to find the best policy. There are two ways to get there.

The first is policy-based methods. You teach the agent the policy directly. It learns “in this state, take this action.” A policy can be deterministic, so the same state always gives the same action, or stochastic, where it gives probabilities for different actions.

The second is value-based methods. Here the agent learns a value function, which says how good it is to be in a state. It doesn't pick actions directly. It moves toward the states with the highest value. So the policy comes from the values.

Put simply, policy-based says “learn what to do,” and value-based says “learn what is worth reaching, then act on that.”

The “Deep” in Deep Reinforcement Learning

So where does “deep” come from?

Plain reinforcement learning can store values in a simple table. That works when there are only a few states. But real problems, like a video game, have a huge number of states, and a table can't hold them all.

Deep reinforcement learning replaces the table with a neural network. The network learns to estimate the values or the policy, even for states it hasn't seen. That's the “deep”: deep learning inside reinforcement learning.

Where I am now

I've only finished the first part, so I know the vocabulary and not much more. But it already changed how I think about learning. An agent isn't told the answer. It tries, gets feedback, and slowly gets better. That sounds a lot like us.

I'll keep going through the course and write about what I learn.