Parallel Reinforcement Learning Environments on FPGA 🌍

Introduction | Environments | Template | Quickstart｜Training Example｜Other Great Tools

Introduction

💎 FPGA-Gym is the first diverse suite of scalable reinforcement learning environments based on FPGA technology.

It features very fast environment computation speed⚡️, large-scale parallelism🏅, and high flexibility🌊.

🏠 FPGA-Gym provides a Template for parallel reinforcement learning environments on FPGA. Its modular and parameterized features allow users to conveniently customize new environments without extensive FPGA knowledge.

🌌 It now features 7 environments based on this template!

Environments

FPGA-Gym currently offers a variety of environments, including classical control, gridworld, and strategy games🌍. We are committed to continually expanding and enhancing the range of environments available.

Environment	Category	Registered Version(s)	Source	Reference
🍎 CartPole	Classic Control	CartPole-v1	code	Click
🪵 Pendulum	Classic Control	Pendulum-v1	code	Click
⛰️MountainCar	Classic Control	MountainCar-v0	code	Click
🧊 FrozenLake	gridworld	FrozenLake-v1	code	Click
🕳️ CliffWalking	gridworld	CliffWalking-v0	code	Click
🚕 Taxi	gridworld	Taxi-v3	code	Click
🃏 Blackjack	strategy games	Blackjack-v1	code	Click
🚢 Seaquest	Atari games	Seaquest-v0	code	Click

Template

we provide the verilog template and python template, The hierarchical structure of the file is illustrated as follows:

• FPGA-Top.v This file is just a wrapper.
  • Pipline.v This file defines our carefully designed parallel template.
    • Compute.v This file defines the number of hardware compute resources.
      • Compute_Single.v This file defines a single environment compute logic. This corresponds to a computational resource on the development board. If the user wants to add a new environment, they need to customize this file. We add CartPole as example in the template.
• FPGAEnv.py This file defines the parent class. After inheriting the parent class, different environment son classes need to modify the main places as follows. Please see the "How to Add New Environments Using the FPGA-Gym Template" for details on how to add your own environment to the template.

Quickstart

If the environment you need is already available in the FPGA-Gym library, we provide a guide on "How to Use the FPGA-Gym Library".

If the environment you need is not available in the FPGA-Gym library, we provide instructions on ["How to Add New Environments Using the FPGA-Gym Template"](User_Guide/Write _Single_Environment_Compute_in_Verilog.md).

Please see the "How to Write Single Environment Compute in Verilog.md" for details on how to add your own environment to the template.

You can see an example in this Video. We demo an example of PPO algorithm training based on FPGA-Gym.

You can see an example in this Video. We demo an example of how to write an atari game seaquest based on FPGA-Gym in verilog.

Preliminary software and hardware preparations:

1. An FPGA development board with PCIe Connect the FPGA development board with PCIe to the PCIe slot on the computer motherboard, just like the GPU.
2. PCIe driver: Download the PCIe driver from AMD's official website. Follow the steps in the readme under the XDMA file.
3. Vivado software(A comprehensive design suite to turn our hardware behavior description language Verilog into the actual computing circuit inside the FPGA): Download the software Vivado from AMD's official website. We used version 2018.2 in our experiment, please open it with a later version of Vivado.

Training Example

We provide examples of training, including CartPole + DQN, CartPole + PPO, CliffWalking + DQN.

Other Great Tools

Other works have embraced the approach of parallel RL environments. In particular, we suggest users check out the following sister repositories:

In building

Type	Name	Paper	Feature
CPU	VectorEnv	Click	The base class for vectorized environments to run multiple independent copies of the same environment in parallel.
CPU	EnvPool	Click	A C++-based batched environment pool with pybind11 and thread pool.