|
PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
|
Historical; superseded by the 1,000-observation report. The startup-only switch and per-dimension selection budget below are no longer defaults.
BLOCKED: do not merge or tag this head. A full-history prefix followed by the round-one bounded estimator is implemented, but no fitted legacy prefix meets the selection budget. The provisional startup-only switch of 8 fails both required long-budget quality gates. No reservoir, weighting, EI-count, or quality-threshold tuning is included in this round.
All compilation, tests, measurements, and numerical aggregation run on an AWS c6i.2xlarge Linux spot machine (8 vCPU, GCC 13.3). Timing probes share the machine; these are contended measurements, not isolated peak-throughput claims. The branch is rebased onto 0.3.1 (13885b9); engine and codegen gitlinks are unchanged.
The applied linear budget is 375 microseconds at 64D, or 5.859375 microseconds per dimension: 93.75 microseconds at 16D. The budget is not increased. With the legacy ten-observation random startup, the largest qualifying power of two is N_s = 8, below 4,096. It is a startup-only value, not evidence that a fitted full-history model is affordable. Even forcing fitting with one observation costs 227.65 microseconds at 16D.
| 16D completed history | Legacy default-startup ask us | Forced fitted-model ask us |
|---|---|---|
| 1 | 1.918 | 227.65 |
| 2 | 1.948 | 278.23 |
| 4 | 1.960 | 379.39 |
| 8 | 1.898 | 570.84 |
| 16 | 920.719 | 897.33 |
| 32 | 1552.078 | 1474.53 |
| 64 | 2456.822 | 2506.91 |
| 128 | 3980.667 | 4311.15 |
| 256 | not measured | 7766.598 |
| 512 | not measured | 14118.751 |
| 1024 | not measured | 27652.270 |
| 2048 | not measured | 55351.041 |
| 4096 | not measured | 112201.335 |
At the selected count, separate legacy snapshot probes measure:
| Dims | N_s | Default-startup ask us | Forced fitted-model ask us |
|---|---|---|---|
| 8 | 8 | 0.8685654297 | 284.9714258 |
| 16 | 8 | 1.926037109 | 570.9768945 |
| 32 | 8 | 4.718483398 | 1167.558957 |
| 64 | 8 | 13.78279492 | 2380.35027 |
Below history_switch, estimation, EI draws, pending-liar handling, and proposal RNG consumption stay on the exact 0.3.0 path. A successful finite tell() increments the completed usable observation count, then activates bounded retention at N_s. Issued, outstanding, failed/abandoned proposals, worker count, and wall time never advance the switch. Reset restores the exact-prefix state.
After the switch, retain 25 elites, 64 recent non-elites, and 448 separately seeded older non-elites. Keep elite-triggered good-model refresh, 32-completion bad-model epochs, 513-point density tables, live pending overlays, and eight EI draws. --tpe-history-switch N and StudySpec sampler.config.history_switch override the positive uint64 threshold. Larger thresholds increase prefix memory and ask cost.
The pinned identity executable compares each proposal's ID, parameter bytes, and hexadecimal objective value, not just best-so-far regret. 96 studies, 4,736 proposal/value records, diff count 0: 80 studies compare the eight-proposal default prefix; another 16 compare 256 proposals with override N_s=512, exercising fitted full-history models. This is not a 3,000-trial identity claim. Since default N_s is less than 3,000, the unchanged regret gate applies instead.
Instance 1, batch8/lag0, eight problems and ten seeds: 17, 48, 79, 110, 141, 172, 203, 234, 265, 296. The 0.3.0 reference is the verified round-one trace set; current sampler studies are rerun. Ratios divide the two per-problem median normalized residual regrets, not paired-seed medians.
| Problem | 0.3.0 regret | Switch regret | Ratio at 3k |
|---|---|---|---|
| rotated_rastrigin20 | 0.331571646 | 0.333565007 | 1.006012 |
| ackley20 | 0.351007148 | 0.341552435 | 0.973064 |
| bbob_f15_d20 | 0.0879602016 | 0.0944494069 | 1.073774 |
| bbob_f21_d20 | 0.050623537 | 0.163116261 | 3.222143 |
| bbob_f24_d20 | 0.201541987 | 0.20455969 | 1.014973 |
| mixed_irrelevant15 | 7.22111952e-05 | 4.60731211e-05 | 0.638033 |
| rosenbrock20 | 0.00419199997 | 0.00440170257 | 1.050024 |
| rotated_ellipsoid20 | 0.00387524052 | 0.00567790711 | 1.465175 |
| Geomean | 1.164189 — FAIL |
Required geomean <=1.01 and every problem <=1.10. Worst problem is 3.222143; rugged/multimodal parity is not preserved on bbob_f21_d20.
Four problems and five seeds: 17, 48, 79, 110, 141. Both current and reference studies are rerun. The permitted benchmark-only reference replaces compaction with unbounded retention, keeping the switch, density tables, EI count, and refresh rules unchanged. It is not a 0.3.0 10k run. Both quality binaries use the same frozen 0.3.0 coordinator extraction; production worker replay is gated separately.
| Problem | Full-history regret | Switch regret | Ratio at 10k |
|---|---|---|---|
| rosenbrock10 | 0.00223023893 | 0.00234853567 | 1.053042 |
| rosenbrock20 | 0.00374331028 | 0.00384532402 | 1.027252 |
| rotated_ellipsoid20 | 0.00244626049 | 0.00345003271 | 1.410329 |
| bbob_f21_d10 | 0.0310354955 | 0.0318328076 | 1.025690 |
| Geomean | 1.118446 — FAIL |
Required geomean <=1.03 and every problem <=1.10. Worst problem is 1.410329. The measured refinement cost of flat history is therefore outside the accepted quality envelope; the implementation is not release-ready.
Random sphere prefill, then 256 timed asks with ordered batch-eight feedback, including model refits and pending-liar work. Every bounded row retains at most 537 completed observations. Microseconds per ask:
| Dims | n=1k | n=10k | n=100k | n=1M |
|---|---|---|---|---|
| 8 | 74.784 | 74.074 | 70.351 | 68.948 |
| 16 | 149.552 | 149.044 | 139.849 | 140.489 |
| 32 | 308.362 | 307.294 | 289.121 | 288.795 |
| 64 | 641.470 | 657.913 | 612.889 | 625.503 |
History cost is flat, but 64D exceeds the 375-us budget. This is a documented known limit, not the blocking gate in this round. The native million-trial run also measures 699.469842 s of total proposal time. Lag1 overlaps execution but does not eliminate serialized sampling cost. The superseded recent-only 245-us measurement (about 65% of the budget) does not describe this reservoir estimator.
64 continuous dimensions, W=8, batch8/lag1, 3-ms synthetic objective, --trials-out none, best-k=10, and every billing line strict-parsed:
| Trials | Wall seconds | Peak RSS KiB | Trials/second | Final JSON bytes |
|---|---|---|---|---|
| 100000 | 71.15 | 14264 | 1405.432467 | 23182 |
| 1000000 | 780.00 | 14780 | 1282.026787 | 23185 |
The million-trial run satisfies the one-hour volume gate. All 1,000,000 received terminal lines are complete with unique, monotonic IDs and required billing fields. The stream is 2,170,866,734 bytes, SHA-256 dcc33d4881e838a4ed4c9c0a418c68b284eca2455ecbe5868993d1d4f56c336b. Serialization costs 66.880047 us/line and writes cost 4.922178 us/line. Final JSON rendering costs 0.000480054 s and writing costs 0.000089701 s. Dataset/engine state still scales with bars and workers; exact finite coverage and output all are outside the default bounded-memory guarantee.
Dense OrdinalSet membership applies through cardinality 100,000,000; larger spaces retain the disk fallback. A rerun with one million unique pseudo-random ordinals measures 0.0120772 us/insert and 15,464 KiB peak RSS for dense storage, versus 3.07594 us/insert and 3,536 KiB for disk. This isolates insertion; it does not claim a corresponding end-to-end trial speedup.
Archive insertion and writer enqueue remain atomic after the backpressure wait. Trial-timeout archive/stream replay, bounded 50-ms polling, stalled-reader stop, complete-line deadline/SIGTERM behavior, and cached immutable stream availability remain covered. The quality ctest timeout stays 300 s. Changelog 0.3.0 is released, and 0.4.0 remains above rebased 0.3.1.
| Gate | Result |
|---|---|
| Release native suite | 24/26; core and TPE quality fail |
| ASan + UBSan native suite | 24/26; same two quality assertions fail; no sanitizer diagnostic |
| TSan batch/concurrency/native/output suites | 4/4 pass |
| Python suite | 45/45 pass |
| 100k bounded memory/ask native test | Pass |
| W1/W8 replay through 3k, lag0/lag1 and none/median pruning | Pass |
| Exact-prefix proposal/value diff | 0 |
| Required 3k quality | Fail |
| Required 10k quality | Fail |
| Rugged/multimodal parity | **Fail (bbob_f21_d20)** |
| 1M volume, memory, full billing parse | Pass |
| Pinned Optuna smoke integration | Pass; CSV and metadata retained |
| Doxygen and generated-site validation | Zero warnings; 223 HTML files pass link/path checks |
The native core assertion is best > 2.95; the Rastrigin-2 quality median is 1.038 against the unchanged 1.0 target. These failures are not suppressed or weakened. The native prep CLI is not implemented: its prerequisite release gates are blocked.
Compact public CSVs and verified adjacent metadata sidecars are under benchmarks/optuna/evidence/2026-10-03-switch-{quality,ask,native,ordinal,identity}.csv. They include per-seed regrets, trace/binary/source hashes, budget and configuration, the benchmark-only full-history patch, platform/compiler, and dependency pins. The COCO objective source revision is pinned: 52 of 53 source hashes reproduce exactly; generated coco_version.c comments differ while the version string is 2.8.3-dev8+gdd4bd1f0. This difference is recorded, not represented as 53-file byte identity. Full traces and gate logs stay in the supplemental evidence archive.
Use quality_long.py with the pinned replica, normalizers, seed lists above, --trials 3000 against label baseline, and --trials 10000 against a reference materialized by full_history_reference.py. Compile identity.cpp with the same pinned replica include directory and core/COCO libraries; define PINEFORGE_HPO_LEGACY_TPE only for the baseline and retain linker section garbage collection. identity_replay.py compares all proposal/value bytes. profile.cpp supports default-startup for the selection probe and update for bounded batch-eight acquisition. run_native.py performs the million-trial full billing parse. collect_switch.py publishes and verifies the compact evidence.