Abstract
Shared compute clusters are routinely used to run large numbers of training jobs for hyperparameter optimization, yet the resulting contention, preemption, and heterogeneous hardware conditions complicate efficient search under fixed budgets. Early-stopping methods reduce waste by terminating unpromising configurations, but naïve application can amplify cluster interference, bias performance estimates, and degrade fairness when multiple users compete for limited accelerators. This paper studies resource-efficient hyperparameter optimization on shared clusters by coupling multi-fidelity early-stopping with predictive allocation that anticipates learning-curve outcomes and cluster-side execution costs. The approach treats a trial as a stochastic process whose partial observations support calibrated predictions of final quality, remaining time, and marginal utility, while the scheduler solves a constrained allocation problem that balances expected improvement against latency, energy, and queueing externalities. The resulting design integrates (i) backprop-friendly surrogate models over configuration embeddings and partial training traces, (ii) decision rules for early termination that account for censoring and interference, and (iii) cluster-aware placement that adapts to dynamic resource fragmentation and preemption risk. The paper formalizes objectives, derives optimization and approximation properties, analyzes complexity and sources of error, and describes a reproducible evaluation methodology emphasizing workload mixes, contention regimes, and robustness to nonstationary cluster conditions. The aim is to characterize when predictive allocation yields measurable gains over purely algorithmic early-stopping and when additional modeling is required to avoid pathological interactions on shared infrastructure.