HPO4TabPFN

How do you scale down a TabPFN? And which hyperparameters matter most at different scales?

TabPFN is a transformer-based model that solves small tabular classification problems in under a second—but it's large. This study investigates how to create smaller, more efficient versions (NanoTabPFN) while maintaining competitive performance.

Using random search with no prior assumptions, we systematically explored how scaling strategy (width vs. depth vs. compound), model size, and optimizer choice affect both performance and hyperparameter sensitivity. Our evaluation used the TabArena benchmark alongside classic datasets (Iris, Wine, Breast Cancer).

The key finding: smaller and deeper models can outperform larger, shallower configurations—challenging the assumption that more parameters always means better performance.

We tested four model scales with different width/depth tradeoffs:

• Compound Small (1.35M) outperformed larger models—the Pareto-optimal configuration wasn't the biggest one
• Muon optimizer dominated—consistently beat AdamW, Lion, and AdamW Schedule-Free across all scales
• Deeper > Wider—at equal parameter counts, deeper models slightly outperformed wider ones
• Batch size correlates with scale—larger models benefited from larger batch sizes
• Learning rate is critical—SHAP analysis showed LR and model scale as the most important hyperparameters
• Smaller models need higher LR—inverse relationship between model size and optimal learning rate

Performance was evaluated on the TabArena benchmark. The Compound Small configuration achieved the best accuracy-to-parameters ratio.

SHAP analysis across all model configurations revealed the relative importance of each hyperparameter:

• Learning Rate (LR): Highest importance scores (46.4 average)—the single most critical hyperparameter
• Optimizer: Second most important (36.6 average)—Muon's superiority was consistent
• Batch Size: Lower but scale-dependent importance (17.0 average)—matters more for larger models

Interestingly, smaller models showed higher sensitivity to learning rate, while larger models were more robust to LR variations but more sensitive to batch size choices.

The experimental pipeline consisted of:

• Search Space: Learning rate [0.00001, 0.01] log-scale, batch size {32, 64, 128, 256}, optimizer {AdamW, Muon, Lion, AdamW Schedule-Free}
• HPO Framework: Random search via NePS with no prior assumptions—ensuring unbiased exploration
• Validation: Synthetic validation dataset (1600 samples) for HPO, real datasets (Iris, Wine, Breast Cancer) for final evaluation
• Benchmark: TabArena for standardized comparison across configurations