Solving Reinforcement Learning Racetrack Exercise with Off-policy Mon…

In the section Off-policy Monte Carlo Control of the book Reinforcement Learning: An Introduction 2nd Edition (page 112), the author left us with an interesting exercise: using the weighted importance sampling off-policy Monte Carlo method to find the fastest way driving on both tracks. This exercise is comprehensive that asks us to consider and build almost every component of a reinforcement learning task, like the environment, agent, reward, actions, conditions of termination, and the algorithm. Solving this exercise is fun and helps us build a solid understanding of the interaction between algorithm and environment, the importance of a correct episodic task definition, and how the value initialization affects the training outcome. Through this post, I hope to share my understanding and solution to this exercise with everyone interested in reinforcement learning.

As mentioned above,…