PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
Loading...
Searching...
No Matches
Completed-count switch, 0.4.0

Historical round-three report; bounded mode is now opt-in. These measurements remain the documented cost of --tpe-history-switch 1000. Round four keeps exact full-history TPE by default (null = never switch) and accepts these measured 3k/10k refinement losses only for explicit opt-in. The earlier default-quality gates are superseded, not retrospectively passed. No bounded-model tuning is applied.

All compilation, tests, profiles, and numerical aggregation use one AWS c6i.2xlarge Linux spot machine: 8 vCPU, GCC 13.3, C++17 Release. Timing probes share the box with gate work; these are contended measurements, not isolated peak-throughput claims. The branch is rebased onto 0.3.1 (13885b9), with unchanged engine/codegen gitlinks. The small CSVs and verified SHA-256 metadata are under benchmarks/optuna/evidence/2026-10-04-final-1000-*.

Design and exact prefix

Below 1,000 successful finite tell() observations, use the exact 0.3.0 model. At that completed count, switch to the unchanged round-one bounded estimator: 25 elites, 64 recent non-elites, 448 independently seeded older non-elites, elite-triggered good-model refresh, bad-model refresh every 32 completions, 513-point density tables, eight EI draws, and live pending-liar overlays. Issued, pending, failed, pruned, or abandoned proposals do not advance the switch. --tpe-history-switch N and StudySpec sampler.config.history_switch preserve the override; tpe_history_switch remains recorded in results and terminal records.

Prefix identity compares IDs, every parameter byte, and hexadecimal objective values against pinned 0.3.0: 80 studies x 1,000 proposals, diff count 0. Another 16 override-prefix studies compare 256 proposals with N=512. Combined: 96 studies, 84,096 proposal/value records, diff count 0. This proves prefix identity, not long-budget quality parity. Production replay also covers W1/W8, lag0/lag1, and none/median pruning through 3,000 trials.

A larger override retains legacy behavior longer, but increases prefix memory and history-growing ask cost. Historical random-prefill legacy 16D snapshot probes measured 27,652 us/ask at 1,024 observations and 112,201 at 4,096. There is no per-dimension budget used to select the default switch.

Quality at 3,000

Pinned HPO-BENCH instance 1, batch8/lag0. Eight problems x ten seeds: 17, 48, 79, 110, 141, 172, 203, 234, 265, 296. The current arm is rerun; all 80 cached 0.3.0 reference trace hashes are verified against the published sidecar. Its pinned native binary hash is 9ab9b977442ecbe469345d2726237808561aace5af68a34a5b796ebccd2e23b0. COCO uses revision dd4bd1f0cc7699a2b612448d85aafb94636fe947 and generated version 2.8.3-dev8+gdd4bd1f0; generated comments are not a byte-identity claim.

Regret is normalized residual regret using the pinned random-study scales. Ratio divides per-problem median regrets, not the median of paired-seed ratios. Lower is better.

Problem 0.3.0 regret Completed-switch regret Ratio
rotated_rastrigin20 0.331571646 0.313726013 0.946179
ackley20 0.351007148 0.354031193 1.008615
bbob_f15_d20 0.0879602016 0.0905954301 1.029959
bbob_f21_d20 0.050623537 0.050464455 0.996858
bbob_f24_d20 0.201541987 0.201140243 0.998007
mixed_irrelevant15 0.0000722111952 0.0000727264548 1.007135
rosenbrock20 0.00419199997 0.00455208505 1.085898
rotated_ellipsoid20 0.00387524052 0.00482683858 1.245558
Geomean 1.036492 — FAIL

Required: geomean <=1.02 and every problem <=1.10. Rugged/multimodal problems remain within that per-problem envelope, but rotated ellipsoid does not. The final tradeoff is about 3.65% worse per-problem median regret at 3k than 0.3.0 in exchange for flat ask cost, not the earlier approximately 1.6% result.

The earlier result cannot be reused: with batch8 the issued-count trigger compacted on completion 993 after proposal 999 was issued. The completed-only trigger first compacts at completion 1,000, changing reservoir initialization even though every pre-switch proposal matches. No reservoir, weighting, EI-count, or quality-threshold tuning is applied to recover the historical result.

Quality at 10,000

Four problems x five seeds: 17, 48, 79, 110, 141. Both arms are rerun. The permitted benchmark-only reference keeps all history while retaining the same 1,000-observation switch, density tables, EI count, and refresh rules. It is not a 0.3.0 10k run. Both quality binaries share the frozen 0.3.0 coordinator extraction; production replay is gated separately.

Problem Full-history regret Completed-switch regret Ratio
rosenbrock10 0.00225861627 0.00235524197 1.042781
rosenbrock20 0.00370510388 0.00396770111 1.070874
rotated_ellipsoid20 0.00265592094 0.00246749591 0.929055
bbob_f21_d10 0.028332954 0.030925324 1.091497
Geomean 1.031570 — FAIL

Required: geomean <=1.03 and every problem <=1.10. The worst problem passes at 1.091497, but the geomean does not. This is a release blocker, not a waived quality tradeoff.

History-flat acquisition and known 64D limit

Seeded random sphere prefill, then 256 timed asks with ordered batch-eight updates, including refits and pending-liar work. Three repetitions; medians in microseconds per ask. Every row retains at most 537 observations. These are prefill probes, not full million-trial adaptive quality runs.

Dims n=1k n=10k n=100k n=1M
8 79.495 76.592 73.461 73.004
16 157.542 156.120 149.372 148.697
32 321.017 326.214 302.937 302.416
64 682.342 677.345 656.450 663.061

History cost remains flat. 64D exceeds the 375-us W=8, 3-ms-trial ask budget: 656.450/663.061 us at 100k/1M. This is an explicit known throughput limit, not the quality blocker. Lag1 overlaps execution but cannot remove serialized ask cost. The superseded recent-only 245-us/about-65%-of-budget result does not describe this estimator.

Million-trial volume, memory, and billing

64 continuous inputs, W8/batch8/lag1, 3-ms synthetic objective, two fixture bars, --trials-out none, best-k=10, full strict progress parsing:

Trials Wall seconds Peak RSS KiB Trials/second Final JSON bytes
100000 85.69 21152 1167.004 23218
1000000 919.82 21152 1087.169 23221

The volume gate passes within one hour, with flat peak RSS. This is not a claim that every real strategy or bar window can complete a million backtests per hour. Bars/workers, compatibility output all, larger configured exact prefixes, and exact finite-space coverage indexes have separate memory costs.

All 1,000,000 terminal lines strictly parse complete, with unique monotonic IDs and every required billing field. Stream bytes: 2,173,873,114. SHA-256: 875055ffdc057d6577ebf7b84b9b417ce7f80d28797baf94217b49b6a05b0974. At 1M, serialization is 89.106 us/line and writing 7.694 us/line. Proposal time is 832.705 s and barrier time 44.273 s. Final JSON rendering is 0.000454146 s; writing is 0.000105486 s. The release executable hash is fd3afc5a966e0c35dfa5a4fa8bce27362d06cf8a47ae7bfa375096641558e0e7.

P2/P3 and gates

Dense OrdinalSet remains enabled through cardinality 100,000,000 with the disk fallback above. One million unique inserts measure 0.009766 us/insert and 15,384 KiB peak RSS for dense, versus 3.184440 us and 3,568 KiB for disk. This isolates insertion, not an extrapolated end-to-end speedup.

Keep atomic timeout archive/enqueue after backpressure, immutable stream availability, bounded 50-ms poll/waits, two-second stopped-reader grace, complete pipe records, deadline/SIGTERM regressions, and the 300-second quality timeout. 0.3.0 remains marked released, and 0.4.0 stays above 0.3.1. No test thresholds are loosened: restoring the exact prefix removes the Ns=8 native core/Rastrigin failures. The 100k regression now also asserts retention of all 999 pre-switch observations and compaction exactly at completion 1,000.

Gate Result
Release CTest 26/26 pass
ASan + UBSan CTest 26/26 pass; leak detection, 32-MiB quarantine
TSan batch/concurrency/outputs 4/4 pass; disposable-host mmap randomization 28
Python unit tests 45 pass
Documented pinned Optuna smoke Three problems/six rows, CSV and metadata retained
Public API docs Zero Doxygen warnings; generated-site validation passes
Default prefix identity 80,000 records, zero differences; 84,096 with overrides
Production worker replay Through 3k, W1/W8 x lag0/lag1 x none/median pruning
100k memory/time regression Pass, including the default completed-count boundary
1M volume/memory/full billing parse Pass
3k quality FAIL: 1.036492 geomean, 1.245558 worst
10k quality FAIL: 1.031570 geomean, 1.091497 worst

Contract changes in 0.4.0

Schema version stays 1; no existing terminal fields or billing-counter types change. Additive switch/configuration and output-retention metadata are documented in docs/api.md. Progress remains the full billing record regardless of --trials-out all|best-k|none or optional --trials-file; all remains the compatibility default. Dropped lists retain summary counts, best-k, and coverage. The default switch changes from the provisional 8 to 1,000; CLI/StudySpec overrides and recorded tpe_history_switch stay supported. Native prepare remains a separately gated 0.4.x follow-up, not part of this blocked release.