|
PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
|
Historical round-three report; bounded mode is now opt-in. These measurements remain the documented cost of --tpe-history-switch 1000. Round four keeps exact full-history TPE by default (null = never switch) and accepts these measured 3k/10k refinement losses only for explicit opt-in. The earlier default-quality gates are superseded, not retrospectively passed. No bounded-model tuning is applied.
All compilation, tests, profiles, and numerical aggregation use one AWS c6i.2xlarge Linux spot machine: 8 vCPU, GCC 13.3, C++17 Release. Timing probes share the box with gate work; these are contended measurements, not isolated peak-throughput claims. The branch is rebased onto 0.3.1 (13885b9), with unchanged engine/codegen gitlinks. The small CSVs and verified SHA-256 metadata are under benchmarks/optuna/evidence/2026-10-04-final-1000-*.
Below 1,000 successful finite tell() observations, use the exact 0.3.0 model. At that completed count, switch to the unchanged round-one bounded estimator: 25 elites, 64 recent non-elites, 448 independently seeded older non-elites, elite-triggered good-model refresh, bad-model refresh every 32 completions, 513-point density tables, eight EI draws, and live pending-liar overlays. Issued, pending, failed, pruned, or abandoned proposals do not advance the switch. --tpe-history-switch N and StudySpec sampler.config.history_switch preserve the override; tpe_history_switch remains recorded in results and terminal records.
Prefix identity compares IDs, every parameter byte, and hexadecimal objective values against pinned 0.3.0: 80 studies x 1,000 proposals, diff count 0. Another 16 override-prefix studies compare 256 proposals with N=512. Combined: 96 studies, 84,096 proposal/value records, diff count 0. This proves prefix identity, not long-budget quality parity. Production replay also covers W1/W8, lag0/lag1, and none/median pruning through 3,000 trials.
A larger override retains legacy behavior longer, but increases prefix memory and history-growing ask cost. Historical random-prefill legacy 16D snapshot probes measured 27,652 us/ask at 1,024 observations and 112,201 at 4,096. There is no per-dimension budget used to select the default switch.
Pinned HPO-BENCH instance 1, batch8/lag0. Eight problems x ten seeds: 17, 48, 79, 110, 141, 172, 203, 234, 265, 296. The current arm is rerun; all 80 cached 0.3.0 reference trace hashes are verified against the published sidecar. Its pinned native binary hash is 9ab9b977442ecbe469345d2726237808561aace5af68a34a5b796ebccd2e23b0. COCO uses revision dd4bd1f0cc7699a2b612448d85aafb94636fe947 and generated version 2.8.3-dev8+gdd4bd1f0; generated comments are not a byte-identity claim.
Regret is normalized residual regret using the pinned random-study scales. Ratio divides per-problem median regrets, not the median of paired-seed ratios. Lower is better.
| Problem | 0.3.0 regret | Completed-switch regret | Ratio |
|---|---|---|---|
| rotated_rastrigin20 | 0.331571646 | 0.313726013 | 0.946179 |
| ackley20 | 0.351007148 | 0.354031193 | 1.008615 |
| bbob_f15_d20 | 0.0879602016 | 0.0905954301 | 1.029959 |
| bbob_f21_d20 | 0.050623537 | 0.050464455 | 0.996858 |
| bbob_f24_d20 | 0.201541987 | 0.201140243 | 0.998007 |
| mixed_irrelevant15 | 0.0000722111952 | 0.0000727264548 | 1.007135 |
| rosenbrock20 | 0.00419199997 | 0.00455208505 | 1.085898 |
| rotated_ellipsoid20 | 0.00387524052 | 0.00482683858 | 1.245558 |
| Geomean | 1.036492 — FAIL |
Required: geomean <=1.02 and every problem <=1.10. Rugged/multimodal problems remain within that per-problem envelope, but rotated ellipsoid does not. The final tradeoff is about 3.65% worse per-problem median regret at 3k than 0.3.0 in exchange for flat ask cost, not the earlier approximately 1.6% result.
The earlier result cannot be reused: with batch8 the issued-count trigger compacted on completion 993 after proposal 999 was issued. The completed-only trigger first compacts at completion 1,000, changing reservoir initialization even though every pre-switch proposal matches. No reservoir, weighting, EI-count, or quality-threshold tuning is applied to recover the historical result.
Four problems x five seeds: 17, 48, 79, 110, 141. Both arms are rerun. The permitted benchmark-only reference keeps all history while retaining the same 1,000-observation switch, density tables, EI count, and refresh rules. It is not a 0.3.0 10k run. Both quality binaries share the frozen 0.3.0 coordinator extraction; production replay is gated separately.
| Problem | Full-history regret | Completed-switch regret | Ratio |
|---|---|---|---|
| rosenbrock10 | 0.00225861627 | 0.00235524197 | 1.042781 |
| rosenbrock20 | 0.00370510388 | 0.00396770111 | 1.070874 |
| rotated_ellipsoid20 | 0.00265592094 | 0.00246749591 | 0.929055 |
| bbob_f21_d10 | 0.028332954 | 0.030925324 | 1.091497 |
| Geomean | 1.031570 — FAIL |
Required: geomean <=1.03 and every problem <=1.10. The worst problem passes at 1.091497, but the geomean does not. This is a release blocker, not a waived quality tradeoff.
Seeded random sphere prefill, then 256 timed asks with ordered batch-eight updates, including refits and pending-liar work. Three repetitions; medians in microseconds per ask. Every row retains at most 537 observations. These are prefill probes, not full million-trial adaptive quality runs.
| Dims | n=1k | n=10k | n=100k | n=1M |
|---|---|---|---|---|
| 8 | 79.495 | 76.592 | 73.461 | 73.004 |
| 16 | 157.542 | 156.120 | 149.372 | 148.697 |
| 32 | 321.017 | 326.214 | 302.937 | 302.416 |
| 64 | 682.342 | 677.345 | 656.450 | 663.061 |
History cost remains flat. 64D exceeds the 375-us W=8, 3-ms-trial ask budget: 656.450/663.061 us at 100k/1M. This is an explicit known throughput limit, not the quality blocker. Lag1 overlaps execution but cannot remove serialized ask cost. The superseded recent-only 245-us/about-65%-of-budget result does not describe this estimator.
64 continuous inputs, W8/batch8/lag1, 3-ms synthetic objective, two fixture bars, --trials-out none, best-k=10, full strict progress parsing:
| Trials | Wall seconds | Peak RSS KiB | Trials/second | Final JSON bytes |
|---|---|---|---|---|
| 100000 | 85.69 | 21152 | 1167.004 | 23218 |
| 1000000 | 919.82 | 21152 | 1087.169 | 23221 |
The volume gate passes within one hour, with flat peak RSS. This is not a claim that every real strategy or bar window can complete a million backtests per hour. Bars/workers, compatibility output all, larger configured exact prefixes, and exact finite-space coverage indexes have separate memory costs.
All 1,000,000 terminal lines strictly parse complete, with unique monotonic IDs and every required billing field. Stream bytes: 2,173,873,114. SHA-256: 875055ffdc057d6577ebf7b84b9b417ce7f80d28797baf94217b49b6a05b0974. At 1M, serialization is 89.106 us/line and writing 7.694 us/line. Proposal time is 832.705 s and barrier time 44.273 s. Final JSON rendering is 0.000454146 s; writing is 0.000105486 s. The release executable hash is fd3afc5a966e0c35dfa5a4fa8bce27362d06cf8a47ae7bfa375096641558e0e7.
Dense OrdinalSet remains enabled through cardinality 100,000,000 with the disk fallback above. One million unique inserts measure 0.009766 us/insert and 15,384 KiB peak RSS for dense, versus 3.184440 us and 3,568 KiB for disk. This isolates insertion, not an extrapolated end-to-end speedup.
Keep atomic timeout archive/enqueue after backpressure, immutable stream availability, bounded 50-ms poll/waits, two-second stopped-reader grace, complete pipe records, deadline/SIGTERM regressions, and the 300-second quality timeout. 0.3.0 remains marked released, and 0.4.0 stays above 0.3.1. No test thresholds are loosened: restoring the exact prefix removes the Ns=8 native core/Rastrigin failures. The 100k regression now also asserts retention of all 999 pre-switch observations and compaction exactly at completion 1,000.
| Gate | Result |
|---|---|
| Release CTest | 26/26 pass |
| ASan + UBSan CTest | 26/26 pass; leak detection, 32-MiB quarantine |
| TSan batch/concurrency/outputs | 4/4 pass; disposable-host mmap randomization 28 |
| Python unit tests | 45 pass |
| Documented pinned Optuna smoke | Three problems/six rows, CSV and metadata retained |
| Public API docs | Zero Doxygen warnings; generated-site validation passes |
| Default prefix identity | 80,000 records, zero differences; 84,096 with overrides |
| Production worker replay | Through 3k, W1/W8 x lag0/lag1 x none/median pruning |
| 100k memory/time regression | Pass, including the default completed-count boundary |
| 1M volume/memory/full billing parse | Pass |
| 3k quality | FAIL: 1.036492 geomean, 1.245558 worst |
| 10k quality | FAIL: 1.031570 geomean, 1.091497 worst |
Schema version stays 1; no existing terminal fields or billing-counter types change. Additive switch/configuration and output-retention metadata are documented in docs/api.md. Progress remains the full billing record regardless of --trials-out all|best-k|none or optional --trials-file; all remains the compatibility default. Dropped lists retain summary counts, best-k, and coverage. The default switch changes from the provisional 8 to 1,000; CLI/StudySpec overrides and recorded tpe_history_switch stay supported. Native prepare remains a separately gated 0.4.x follow-up, not part of this blocked release.