Gradient Boosting Reinforcement Learning

🌈 Abstract

The paper introduces Gradient Boosting Reinforcement Learning (GBRL), a framework that extends the advantages of Gradient Boosting Trees (GBT) to the reinforcement learning (RL) domain. GBRL implements various actor-critic algorithms and compares their performance with neural network (NN) counterparts. The paper also presents a GPU-accelerated GBRL implementation that integrates with popular RL libraries.

🙋 Q&A

[01] GBT as RL Function Approximator

1. Can GBT-based AC algorithms effectively solve complex high-dimensional RL tasks? The results show that GBT-based AC algorithms, such as PPO GBRL, can effectively solve complex high-dimensional RL tasks. GBRL achieves competitive performance across a diverse array of tasks, including classic control tasks, high-dimensional vectorized problems, and categorical tasks.

2. How does GBRL compare with NN-based training in various RL algorithms? The performance of GBRL varied across RL algorithms. In some environments, such as MountainCar, GBRL outperformed NN with all AC methods. However, in other environments, like Pendulum, NN performed better. The results suggest that the choice of RL method depends on the task characteristics, with GBT thriving in complex, yet structured environments.

[02] Comparison to NNs

1. Do the benefits of GBT in supervised learning transfer to the realm of RL? Yes, the results suggest that the benefits of GBT in supervised learning, such as handling structured and categorical data, can effectively transfer to the RL domain. In environments like MiniGrid, which have structured and categorical features, GBRL outperformed or performed on par with NN-based methods.

2. Can traditional GBT libraries be used instead of GBRL for RL tasks? No, the results show that traditional GBT libraries, such as CatBoost and XGBoost, are unable to solve RL tasks in a realistic timeframe. GBRL, however, efficiently solves RL tasks while remaining competitive with NN across a range of environments.

[03] Benefits in Categorical Domains

1. How does GBRL perform in categorical environments compared to NN-based methods? In categorical environments, such as the MiniGrid domain, GBRL outperforms or performs on par with NN-based methods. Specifically, PPO GBRL significantly outperforms PPO NN in most MiniGrid tasks. These results emphasize that GBRL is a strong candidate for problems characterized by structured and categorical data.

[04] Comparison to Traditional GBT Libraries

1. How does GBRL compare to traditional GBT libraries like CatBoost and XGBoost in RL tasks? The results show that traditional GBT libraries, such as CatBoost and XGBoost, are unable to solve RL tasks in a realistic timeframe. In contrast, GBRL efficiently solves RL tasks while remaining competitive with NN-based methods across a range of environments.

[05] Evaluating the Shared AC Architecture

1. How does sharing the tree structure between the actor and the critic impact GBRL's performance? The shared tree structure in GBRL's actor-critic architecture significantly reduces memory consumption and increases training speed, without negatively impacting the resulting policy's performance. By sharing the tree structure, GBRL requires less than half the memory and almost triples the training FPS compared to non-shared architectures.