SpikeRL

SpikeRL: A Highly Scalable Framework for Spiking Reinforcement Learning

Table of Contents

• Introduction
• Features
• Installation
• Requirements
• Contact

Introduction

SpikeRL is a highly scalable framework designed for implementing and experimenting with spiking reinforcement learning algorithms. Leveraging the power of spiking neural networks, SpikeRL aims to bridge the gap between traditional reinforcement learning and biologically plausible neural computations, offering researchers and developers a robust platform for advanced AI experiments.

Features

• Scalability: Efficiently handles large-scale spiking neural networks.
• Efficiency: Utilizes mixed-precision techniques for optimization and energy efficiency.
• Extensibility: Easily extendable to accommodate new models and environments.

Installation

Follow the steps below to set up SpikeRL on your local machine.

Prerequisites

• Python 3.7 or higher
• pip package manager

Steps

1. Clone the Repository

   git clone https://github.com/icl-utk-edu/spike-rl.git
   cd spike-rl

2. Create a Virtual Environment (Optional but Recommended)

   python3 -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install Dependencies

   For mpi support with spikeRL, compile PyTorch from source (instructions at https://github.com/pytorch/pytorch) with either openmpi or mpich installation in your system. Ensure you have the required version of numpy and mujoco as specified in requirements.txt.

   pip install -r requirements.txt

Requirements

The project dependencies are listed in the requirements.txt file:

wheel
setuptools
pybind11
ipython
notebook
pyglet
six
pillow
imageio
scipy
scikit-learn
pandas
matplotlib
gymnasium
networkx
torch
gym
mpi4py
mujoco==2.3.3
numpy==1.24.0

Usage

# Using NCCL backend
torchrun --nproc_per_node=8 main.py --env Ant-v4 --hidden_sizes 100 100 --batch_size 100 --epochs 1 --backend nccl

# Using MPI backend
mpiexec -n 8 python main.py --env Ant-v4 --hidden_sizes 100 100 --batch_size 100 --epochs 1 --backend mpi

Contact

For any questions or support, please contact [email protected].

---

Acknowledgements

This work was supported by the U.S. Department of Energy, Office of Science, ASCR under Award Number DE-SC0021419. This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Assistant Secretary of Defense for Research and Engineering.