Quick start: Reinforcement Racer

~ admin

Reinforcement learning is the often forgotten sibling of supervised and unsupervised machine learning. But now, with its important role in generative AI learning, I wanted to touch on the subject with a fun introductory example. Fun projects like these can always teach you a lot.

I was inspired by YouTuber Nick Renotte (video: https://www.youtube.com/watch?v=Mut_u40Sqz4) and retried his Project 2 from the two-year-old tutorial. In this tutorial, we teach a model to race a car.

F1 drivers learn from experience, and so does a reinforcement learning agent. Let’s look at teaching such an agent to race through training and making it learn from repeated experience.

A quick recap of Reinforcement Learning (RL)

The docs of RL library, called “Stable Baselines3”, states the following,

“Reinforcement Learning differs from other machine learning methods in several ways. The data used to train the agent is collected through interactions with the environment by the agent itself (compared to supervised learning where you have a fixed dataset for instance).”

https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html

So in RL, there is the concept of an agent/model interacting with its surroundings and learning through rewards. Actions that lead to achieving the goal are rewarded and so are reinforced as being “good” actions.

For the environment, i.e. our race track, we can use the already created, “CarRacing” environment from the “Gymnasium” project – the actively developed fork of OpenAI’s project called “Gym”. Gymnasium helps to get started quickly with reinforcement learning with many pre-made environments.

CarRacing environment. The agent controls a car.

In CarRacing, the reward is negative (-0.1) for every frame and positive (+1000/N) for every track tile visited, where N is the total number of tiles visited in the track. Hence, the car is encouraged to move as quickly as possible and visit new parts of the track. The negative reward penalises staying still as the agent is encouraged to get to the finish asap. The positive reward encourages the car to visit new parts of the track as quickly as possible.

For our agent, I will use the PPO algorithm from the Stable Baselines3 library which comes with a general but good choice of hyperparameters for its models.

PPO: Proximal Policy Optimization

PPO is an algorithm for training RL agents. It has been used before in the fine-tuning of large language models via Reinforcement learning from human feedback (RLHF: https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback).

In PPO, the agent selects random actions which get less random over time as it learns. PPO is conservative in its learning process and makes sure not to change its policy (essentially its brain) by changing too much in each update. Constraining updates in this way leads to consistent performance, stability and speedy learning. However, this may also cause the policy to get stuck in a sub-optimal method for success. PPO seems to work well for this task in a short amount of time without requiring huge memory resources. It’s great to get started.

Read more about it: https://spinningup.openai.com/en/latest/algorithms/ppo.html

Stable baselines3 guide to choosing agents https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html.

Whereas Nick used the stable baslines3 library directly, I will use Stable Baseline’s “RL zoo” – a framework for using Stable Baselines3 in an arguably quicker and more efficient way – all on the command line. RL Zoo docs: https://stable-baselines3.readthedocs.io/en/master/guide/rl_zoo.html

Training with RL Zoo – Reinforcement boogaloo

I found an easy way to get training our PPO agent (see: https://github.com/DLR-RM/rl-baselines3-zoo). I followed the installation from source (this requires git). I also recommend you create a virtual environment for all the dependencies. 

git clone https://github.com/DLR-RM/rl-baselines3-zoo

Your system is probably different from mine and my commands for the command line are aimed at Linux/MacOS so you may need to translate some commands for your system.

Really make sure to have all the dependencies!

pip install -r requirements.txt
pip install -e .

Now let’s start training. Specify the algorithm (PPO), the environment (CarRacing-v2) and the number of time-steps (try different numbers!). For example you can try 500000 time steps.

python -m rl_zoo3.train --algo ppo --env CarRacing-v2 --n-timesteps 500000

This may take a few minutes. If you’re on Linux and get error: legacy-install-failure you may also need to install a few more packages.

sudo dnf install python3-devel<br>sudo dnf install swig<br>sudo dnf groupinstall "Development Tools"

Once training is complete, you can visualise the learning process by plotting the reward over time/training episodes.

python scripts/plot_train.py -a ppo -e CarRacing-v2 -y reward -f logs/ -x steps

Increasing rewards with episodes indicates that the agent is learning.

Record a video of your latest agent with

python -m rl_zoo3.record_video --algo ppo --env CarRacing-v2 -n 1000 --folder logs/

I tested the agent after 4,000,000 time steps of training (~2hr for me) and got this episode below – what a good run!

RL zoo allows you to continue training if you want to improve it.

python train.py --algo ppo --env CarRacing-v2 -i logs/ppo/CarRacing-v2_1/CarRacing-v2.zip -n 50000

Thanks for reading!

95 thoughts on “Quick start: Reinforcement Racer”

  3. Great breakdown! Strategic thinking in poker mirrors smart moves in games like those on Jili777. Knowing when to fold or go all-in is key, just like picking the right slot at the right time.

    Reply

  4. Lovart AI sounds like a game-changer for designers looking to blend creativity with AI efficiency. Its tri-modal interaction could really streamline workflows, especially with Figma and Photoshop integration. Worth keeping an eye on. Lovart AI

    Reply

  5. Blackjack strategy can feel overwhelming at first, but breaking it down step-by-step really helps! Seeing platforms like 68win login prioritize a smooth user experience – even with registration – is a great sign for new players too. It’s all about accessibility!

    Reply

  6. Keno’s probability is fascinating, isn’t it? Seeing patterns emerge is key! Platforms like jboss are making gaming more accessible, though responsible play is always first. Enjoy the thrill, but set limits!

    Reply

  9. It’s smart to prioritize secure online platforms – verification is key these days! Seeing innovations like streamlined banking with GCash (via jl boss games) is reassuring. Responsible gaming & solid security go hand-in-hand, don’t you think?

    Reply

  10. Interesting article! Thinking about games of strategy really highlights how much planning goes into both winning and responsible play. Speaking of strategy, have you checked out the ph987 app? It’s designed for players who like a calculated approach! It’s cool to see platforms emphasizing secure, thoughtful gaming experiences.

    Reply

  11. Interesting points about responsible gaming! Platforms like jljl77 app games are gaining traction in the Philippines, focusing on both entertainment and secure transactions – a good sign for players seeking value & trust.

    Reply

  12. Interesting points about adapting to table dynamics! It reminds me of the secure, player-focused experience at phswerte casino – they really emphasize trust & a smooth journey, like their verification process. Solid analysis!

    Reply

  13. Dice games are all about understanding probability – even simple ones! It’s cool to see platforms like 777pinas game offering diverse options, but always remember responsible gaming & knowing your odds! A verified account is key, too. 😉

    Reply

  15. Interesting analysis! The focus on local payment options like GCash & PayMaya at ss777 apk is smart for the Filipino market. A playful interface, like a ‘game zoo’, can really boost engagement too!

    Reply

  16. It’s so important to remember gambling should be fun, not a source of stress. Seeing platforms like spintime online casino offer data insights is interesting – transparency can help players stay informed & in control. Let’s all gamble responsibly! 😊

    Reply

  18. Scratch cards always feel like a little burst of fun, don’t they? Reminds me of exploring new online platforms like jlboss login – so many games to try! Convenient apps like that are a game-changer for quick entertainment. Definitely checking it out! ✨

    Reply

  19. Smart bankroll management is key with any online gaming, and understanding the odds is crucial. Seeing platforms like JL Boss offer convenient access via the jlboss app is interesting, but always gamble responsibly! It’s about entertainment, not just winning.

    Reply

  21. It’s so important to remember gambling should be fun, not a source of stress. Seeing platforms like nustargame app focus on classic games with modern security is a good sign. Balance is key, folks! Let’s all play responsibly. ✨

    Reply

  23. That’s a great point about accessibility in shooting games! Seeing platforms like wk777 login game prioritize quick registration (under 2 mins!) & local payment options like GCash is smart for wider appeal. It’s all about lowering the barrier to entry, right? 🤔

    Reply

  24. Fascinating to see how gaming evolved in the Philippines! Platforms like arina plus games are really adapting to local preferences with options like GCash & Tagalog support – smart move for responsible play! It’s a long way from cockfighting…

    Reply

  26. It’s fascinating how gaming platforms now blend tradition with tech – like preserving cultural roots while innovating! Seeing that reflected in Voslot777’s approach, and easy access via voslot777 club, is really interesting. It’s about more than just games, it’s heritage!

    Reply

  65. I have been using this platform for a while now and the payouts are consistently fast. Definitely one of the most reliable sites I have encountered. Check out idrpvip5 if you want a premium feel.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *