100x faster initial training

Effective machine learning model training often requires meticulous hyperparameter tuning to prevent overfitting. This process typically involves employing hyperparameter optimization tools like Optuna, which iteratively trains the model with various hyperparameter combinations. A common practice is to perform 100 trials, essentially training the model 100 times with different hyperparameter sets to identify the optimal configuration.

However, what if you could bypass this time-consuming step altogether? Perpetual ML introduces Perpetual Learning, a novel approach that achieves precisely this feat. By eliminating the need for hyperparameter tuning, Perpetual Learning boasts a 100x speedup compared to traditional gradient boosting methods, while maintaining state-of-the-art accuracy on tabular data.

Perpetual Learning is currently applied to gradient boosting machines (GBM, GBDT, GBRT, GBT or MART), which have consistently produced outstanding results for tabular data tasks. This blog post delves into a comparative analysis of PerpetualBooster and the widely-used LightGBM package, utilizing both the California Housing (regression) and Forest Cover Types (classification) datasets.

Table 1 shows the comparison results for the California Housing (regression) dataset. LightGBM employs the n_estimators parameter to enhance model complexity and improve test performance. Similarly, PerpetualBooster utilizes the budget parameter for performance optimization. We compared the models at equivalent error rates across three scenarios, and in each instance, PerpetualBooster demonstrated a clear advantage in speed, exceeding LightGBM by a factor of 100x.

Table 2 presents a comparison of results on the Forest Cover Types dataset, a widely used benchmark for classification tasks. This comparison focuses solely on a single case due to the extended execution time associated with LightGBM in this specific context. The results reveal that PerpetualBooster demonstrates superior efficiency, achieving 76 times faster execution speed compared to LightGBM while maintaining equivalent log loss performance. This significant speed advantage makes PerpetualBooster a compelling option for real-world applications where rapid model training and deployment are crucial.

Perpetual ML represents a significant advancement in the realm of machine learning by eliminating the need for hyperparameter tuning. This translates to dramatically faster training times while upholding or even surpassing the accuracy of traditional approaches like LightGBM. This blog post provides preliminary insights into Perpetual ML's capabilities, and future explorations will delve deeper into its applications and advantages across diverse machine learning tasks.