PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
Loading...
Searching...
No Matches
History-switch review, 0.4.0

Historical; superseded by the 1,000-observation report. The startup-only switch and per-dimension selection budget below are no longer defaults.

BLOCKED: do not merge or tag this head. A full-history prefix followed by the round-one bounded estimator is implemented, but no fitted legacy prefix meets the selection budget. The provisional startup-only switch of 8 fails both required long-budget quality gates. No reservoir, weighting, EI-count, or quality-threshold tuning is included in this round.

All compilation, tests, measurements, and numerical aggregation run on an AWS c6i.2xlarge Linux spot machine (8 vCPU, GCC 13.3). Timing probes share the machine; these are contended measurements, not isolated peak-throughput claims. The branch is rebased onto 0.3.1 (13885b9); engine and codegen gitlinks are unchanged.

Switch selection

The applied linear budget is 375 microseconds at 64D, or 5.859375 microseconds per dimension: 93.75 microseconds at 16D. The budget is not increased. With the legacy ten-observation random startup, the largest qualifying power of two is N_s = 8, below 4,096. It is a startup-only value, not evidence that a fitted full-history model is affordable. Even forcing fitting with one observation costs 227.65 microseconds at 16D.

16D completed history Legacy default-startup ask us Forced fitted-model ask us
1 1.918 227.65
2 1.948 278.23
4 1.960 379.39
8 1.898 570.84
16 920.719 897.33
32 1552.078 1474.53
64 2456.822 2506.91
128 3980.667 4311.15
256 not measured 7766.598
512 not measured 14118.751
1024 not measured 27652.270
2048 not measured 55351.041
4096 not measured 112201.335

At the selected count, separate legacy snapshot probes measure:

Dims N_s Default-startup ask us Forced fitted-model ask us
8 8 0.8685654297 284.9714258
16 8 1.926037109 570.9768945
32 8 4.718483398 1167.558957
64 8 13.78279492 2380.35027

Below history_switch, estimation, EI draws, pending-liar handling, and proposal RNG consumption stay on the exact 0.3.0 path. A successful finite tell() increments the completed usable observation count, then activates bounded retention at N_s. Issued, outstanding, failed/abandoned proposals, worker count, and wall time never advance the switch. Reset restores the exact-prefix state.

After the switch, retain 25 elites, 64 recent non-elites, and 448 separately seeded older non-elites. Keep elite-triggered good-model refresh, 32-completion bad-model epochs, 513-point density tables, live pending overlays, and eight EI draws. --tpe-history-switch N and StudySpec sampler.config.history_switch override the positive uint64 threshold. Larger thresholds increase prefix memory and ask cost.

Below-switch identity and 3k quality

The pinned identity executable compares each proposal's ID, parameter bytes, and hexadecimal objective value, not just best-so-far regret. 96 studies, 4,736 proposal/value records, diff count 0: 80 studies compare the eight-proposal default prefix; another 16 compare 256 proposals with override N_s=512, exercising fitted full-history models. This is not a 3,000-trial identity claim. Since default N_s is less than 3,000, the unchanged regret gate applies instead.

Instance 1, batch8/lag0, eight problems and ten seeds: 17, 48, 79, 110, 141, 172, 203, 234, 265, 296. The 0.3.0 reference is the verified round-one trace set; current sampler studies are rerun. Ratios divide the two per-problem median normalized residual regrets, not paired-seed medians.

Problem 0.3.0 regret Switch regret Ratio at 3k
rotated_rastrigin20 0.331571646 0.333565007 1.006012
ackley20 0.351007148 0.341552435 0.973064
bbob_f15_d20 0.0879602016 0.0944494069 1.073774
bbob_f21_d20 0.050623537 0.163116261 3.222143
bbob_f24_d20 0.201541987 0.20455969 1.014973
mixed_irrelevant15 7.22111952e-05 4.60731211e-05 0.638033
rosenbrock20 0.00419199997 0.00440170257 1.050024
rotated_ellipsoid20 0.00387524052 0.00567790711 1.465175
Geomean 1.164189 — FAIL

Required geomean <=1.01 and every problem <=1.10. Worst problem is 3.222143; rugged/multimodal parity is not preserved on bbob_f21_d20.

Above-switch 10k quality

Four problems and five seeds: 17, 48, 79, 110, 141. Both current and reference studies are rerun. The permitted benchmark-only reference replaces compaction with unbounded retention, keeping the switch, density tables, EI count, and refresh rules unchanged. It is not a 0.3.0 10k run. Both quality binaries use the same frozen 0.3.0 coordinator extraction; production worker replay is gated separately.

Problem Full-history regret Switch regret Ratio at 10k
rosenbrock10 0.00223023893 0.00234853567 1.053042
rosenbrock20 0.00374331028 0.00384532402 1.027252
rotated_ellipsoid20 0.00244626049 0.00345003271 1.410329
bbob_f21_d10 0.0310354955 0.0318328076 1.025690
Geomean 1.118446 — FAIL

Required geomean <=1.03 and every problem <=1.10. Worst problem is 1.410329. The measured refinement cost of flat history is therefore outside the accepted quality envelope; the implementation is not release-ready.

History-flat ask and memory

Random sphere prefill, then 256 timed asks with ordered batch-eight feedback, including model refits and pending-liar work. Every bounded row retains at most 537 completed observations. Microseconds per ask:

Dims n=1k n=10k n=100k n=1M
8 74.784 74.074 70.351 68.948
16 149.552 149.044 139.849 140.489
32 308.362 307.294 289.121 288.795
64 641.470 657.913 612.889 625.503

History cost is flat, but 64D exceeds the 375-us budget. This is a documented known limit, not the blocking gate in this round. The native million-trial run also measures 699.469842 s of total proposal time. Lag1 overlaps execution but does not eliminate serialized sampling cost. The superseded recent-only 245-us measurement (about 65% of the budget) does not describe this reservoir estimator.

64 continuous dimensions, W=8, batch8/lag1, 3-ms synthetic objective, --trials-out none, best-k=10, and every billing line strict-parsed:

Trials Wall seconds Peak RSS KiB Trials/second Final JSON bytes
100000 71.15 14264 1405.432467 23182
1000000 780.00 14780 1282.026787 23185

The million-trial run satisfies the one-hour volume gate. All 1,000,000 received terminal lines are complete with unique, monotonic IDs and required billing fields. The stream is 2,170,866,734 bytes, SHA-256 dcc33d4881e838a4ed4c9c0a418c68b284eca2455ecbe5868993d1d4f56c336b. Serialization costs 66.880047 us/line and writes cost 4.922178 us/line. Final JSON rendering costs 0.000480054 s and writing costs 0.000089701 s. Dataset/engine state still scales with bars and workers; exact finite coverage and output all are outside the default bounded-memory guarantee.

Round-one fixes retained

Dense OrdinalSet membership applies through cardinality 100,000,000; larger spaces retain the disk fallback. A rerun with one million unique pseudo-random ordinals measures 0.0120772 us/insert and 15,464 KiB peak RSS for dense storage, versus 3.07594 us/insert and 3,536 KiB for disk. This isolates insertion; it does not claim a corresponding end-to-end trial speedup.

Archive insertion and writer enqueue remain atomic after the backpressure wait. Trial-timeout archive/stream replay, bounded 50-ms polling, stalled-reader stop, complete-line deadline/SIGTERM behavior, and cached immutable stream availability remain covered. The quality ctest timeout stays 300 s. Changelog 0.3.0 is released, and 0.4.0 remains above rebased 0.3.1.

Gates

Gate Result
Release native suite 24/26; core and TPE quality fail
ASan + UBSan native suite 24/26; same two quality assertions fail; no sanitizer diagnostic
TSan batch/concurrency/native/output suites 4/4 pass
Python suite 45/45 pass
100k bounded memory/ask native test Pass
W1/W8 replay through 3k, lag0/lag1 and none/median pruning Pass
Exact-prefix proposal/value diff 0
Required 3k quality Fail
Required 10k quality Fail
Rugged/multimodal parity **Fail (bbob_f21_d20)**
1M volume, memory, full billing parse Pass
Pinned Optuna smoke integration Pass; CSV and metadata retained
Doxygen and generated-site validation Zero warnings; 223 HTML files pass link/path checks

The native core assertion is best > 2.95; the Rastrigin-2 quality median is 1.038 against the unchanged 1.0 target. These failures are not suppressed or weakened. The native prep CLI is not implemented: its prerequisite release gates are blocked.

Evidence and reproduction

Compact public CSVs and verified adjacent metadata sidecars are under benchmarks/optuna/evidence/2026-10-03-switch-{quality,ask,native,ordinal,identity}.csv. They include per-seed regrets, trace/binary/source hashes, budget and configuration, the benchmark-only full-history patch, platform/compiler, and dependency pins. The COCO objective source revision is pinned: 52 of 53 source hashes reproduce exactly; generated coco_version.c comments differ while the version string is 2.8.3-dev8+gdd4bd1f0. This difference is recorded, not represented as 53-file byte identity. Full traces and gate logs stay in the supplemental evidence archive.

Use quality_long.py with the pinned replica, normalizers, seed lists above, --trials 3000 against label baseline, and --trials 10000 against a reference materialized by full_history_reference.py. Compile identity.cpp with the same pinned replica include directory and core/COCO libraries; define PINEFORGE_HPO_LEGACY_TPE only for the baseline and retain linker section garbage collection. identity_replay.py compares all proposal/value bytes. profile.cpp supports default-startup for the selection probe and update for bounded batch-eight acquisition. run_native.py performs the million-trial full billing parse. collect_switch.py publishes and verifies the compact evidence.