This is one deterministic quality run and one machine-specific timing run. The quality comparison used identical problem definitions, search spaces, seeds, sequential minimize schedules, and trial budgets. Lower regret is better. The report is backed by checked-in raw CSV and metadata:

The quality values reproduce the earlier 2026-07-18 run. All timing values and hashes below refer to this published evidence rerun, rebuilt after the portable integer gamma_count fix. The fix did not change any recorded quality result on this platform.

Environment:

macOS 26.3.1 arm64
Apple Clang 17.0.0
CMake 4.2.3
Python 3.14.3
Optuna 4.9.0 from the isolated pinned environment
dlib 20.0.1, as pinned by pineforge-hpo

The sidecars record base revision 297df69294ac9bbfce0d59deae47724165d7ea63 and repository_dirty: true, because the double-domain gamma_count portability fix and evidence updates were not committed when this run was generated. The exact tested source identity is provided by the 20-file source manifest aggregate 12abddc13f35933acaebf80772a74e3b9fa0af3d637c05f6afc0713a616fb404, together with the configuration and native-binary hashes documented below.

Per-seed best regret

Problem	Seed	Trials	Native TPE	Optuna TPE	Native/Optuna
Branin-2	17	128	4.505393e-02	5.180082e-02	0.891
Branin-2	41	128	3.465856e-02	1.013978e-02	2.217
Branin-2	73	128	2.589337e-04	6.566929e-02	0.136
Branin-2	109	128	5.461743e-03	3.260001e-03	1.166
Branin-2	149	128	1.112884e-02	5.361489e-02	0.332
Hartmann-3	17	192	1.229579e-02	1.223354e-02	1.003
Hartmann-3	41	192	5.871779e-03	2.848561e-03	1.235
Hartmann-3	73	192	3.242738e-03	1.329583e-02	0.568
Hartmann-3	109	192	9.058661e-04	2.033688e-03	0.906
Hartmann-3	149	192	2.901265e-03	8.050801e-03	0.715
Rosenbrock-6	17	512	4.685799e+00	4.964368e+00	0.953
Rosenbrock-6	41	512	6.674520e+00	5.828791e+00	1.124
Rosenbrock-6	73	512	5.259885e+00	6.086113e+00	0.883
Rosenbrock-6	109	512	4.690910e+00	5.918463e+00	0.823
Rosenbrock-6	149	512	3.113013e+00	3.436379e+00	0.927
Rotated Rastrigin-6	17	640	2.043591e+01	1.713027e+01	1.182
Rotated Rastrigin-6	41	640	1.865834e+01	1.303038e+01	1.401
Rotated Rastrigin-6	73	640	2.296586e+01	2.759937e+01	0.838
Rotated Rastrigin-6	109	640	2.247096e+01	1.554313e+01	1.419
Rotated Rastrigin-6	149	640	1.356060e+01	2.276124e+01	0.613
Mixed log interaction	17	384	3.950369e-01	1.163804e+00	0.351
Mixed log interaction	41	384	3.561900e-01	6.045869e-01	0.602
Mixed log interaction	73	384	2.108826e-03	2.064649e+00	0.011
Mixed log interaction	109	384	1.340804e-02	1.036492e-01	0.270
Mixed log interaction	149	384	8.260307e-01	5.722133e-01	1.429
Million discrete	17	1000	8.841000e+03	0.000000e+00	8842.000
Million discrete	41	1000	2.909000e+03	1.055200e+04	0.276
Million discrete	73	1000	2.012200e+04	2.479000e+03	8.114
Million discrete	109	1000	8.711000e+03	1.567800e+04	0.556
Million discrete	149	1000	7.770000e+03	1.085300e+04	0.716

The aggregate paired record was native 19 wins, Optuna 11 wins, and 0 ties. The bounded geometric paired regret ratio (native/Optuna) was 0.825.

This small fixed-seed benchmark does not establish universal optimizer superiority. In particular, native independent TPE lost the rotated Rastrigin median and three of five paired seeds. That is consistent with the expected limitation of a product-of-one-dimensional-marginals model on a rotated, non-separable objective.

For the million-candidate problem, an independent Python full enumeration took 1.31 seconds and evaluated all 10^6 candidates. It confirmed a unique global minimum of 0 at (8, 1, 6, 3, 9, 4), with multiplicity 1; the second-best objective was 17. The objective combines five dominant modular interactions, including two product interactions, plus a non-negative squared-distance anchor that proves only the target can attain zero.

Median quality and sampler API overhead

Problem	Native regret	Optuna regret	N/O	Native us	Optuna us	Wins N/O/T
Branin-2	1.112884e-02	5.180082e-02	0.342	112.68	703.45	3/2/0
Hartmann-3	3.242738e-03	8.050801e-03	0.734	228.84	1140.97	3/2/0
Rosenbrock-6	4.690910e+00	5.828791e+00	0.833	947.90	3543.10	4/1/0
Rotated Rastrigin-6	2.043591e+01	1.713027e+01	1.182	1217.04	3912.24	2/3/0
Mixed log interaction	3.561900e-01	6.045869e-01	0.602	1117.18	5806.28	4/1/0
Million discrete	8.711000e+03	1.055200e+04	0.826	4119.76	4929.29	3/2/0

Sampler timing excludes sampler/Study construction. It measures the public proposal and feedback path: native ask/tell, versus Optuna Study.ask, parameter suggest_*, and Study.tell. Therefore the table compares native C++ API overhead with Python, Study, and in-memory storage overhead; it is not a pure estimator-kernel microbenchmark. The complete driver wall time was 91.22 seconds. Summed in-loop wall time was 31.285 seconds for native and 59.444 seconds for Optuna.

Million-candidate diagnostics

Seed	N unique	O unique	N dup	O dup	N H/L	O H/L	Hits N/O
17	718	338	282	662	3/9	0/0	0/144
41	517	386	483	614	5/21	6/26	0/0
73	305	654	695	346	5/24	6/19	0/0
109	515	433	485	567	3/5	4/16	0/0
149	740	365	260	635	4/14	5/15	0/0

Native produced a median 517 unique candidates, range [305, 740], and median 483 duplicates, range [260, 695]. Its median coverage was 0.0517%. The best-objective candidates had median Hamming distance 4 (range [3, 5]) and median L1 distance 14 (range [5, 24]). Native had no exact-optimum trial in any of the five runs.

Optuna produced a median 386 unique candidates, range [338, 654], and median 614 duplicates, range [346, 662]. Its median coverage was 0.0386%. The best-objective candidates had median Hamming distance 5 (range [0, 6]) and median L1 distance 16 (range [0, 26]). Optuna reached the exact optimum in one of five runs and proposed it 144 times in that run.

Thus native had better median regret and won three of five paired seeds on this problem, but it never reached the exact optimum. Optuna's median was worse, yet one seed found and strongly exploited the exact optimum. Both sampled far below 0.1% unique coverage, and both produced substantial duplicates; those facts are reported rather than treating 1,000 trials as broad coverage of the space.

Exploration-heavy companion (<tt>startup_trials=200</tt>)

This companion reran only million_discrete, with the same five seeds and 1,000-trial budget. Both implementations received 200 random startup trials (20% of the budget), followed by the same univariate TPE with 24 EI candidates. It is a strategy-HPO diagnostic and does not change the product default.

Metric	Native default 10	Native explore 200	Optuna default 10	Optuna explore 200
Median best regret	8711	3852	10552	8854
Paired wins	3	4	2	1
Median unique	517	985	386	439
Unique range	305-740	578-994	338-654	334-482
Median duplicates	483	15	614	561
Duplicate range	260-695	6-422	346-662	518-666
Median unique coverage	0.0517%	0.0985%	0.0386%	0.0439%
Exact hit trials	0	0	144	0
Exact-hit runs	0/5	0/5	1/5	0/5
Median Hamming	4	6	5	6
Hamming range	3-5	3-6	0-6	2-6
Median L1	14	18	16	16
L1 range	5-24	7-26	0-26	3-22
Sampler API us/trial	4119.76	3550.53	4929.29	4417.94

Exploration-profile per-seed best regret and coverage:

Seed	Native regret	Optuna regret	N unique/dup	O unique/dup	N H/L	O H/L	Hits N/O
17	2081	8854	985/15	334/666	6/17	6/16	0/0
41	3082	11976	988/12	482/518	6/18	5/16	0/0
73	3852	6816	994/6	387/613	6/26	6/22	0/0
109	6921	11188	578/422	471/529	3/7	6/20	0/0
149	8208	8105	982/18	439/561	5/20	2/3	0/0

The larger startup allocation reduced median regret for both implementations and raised native's paired record from 3/2 to 4/1. Native median unique coverage nearly doubled and duplicates fell sharply. Neither implementation reached the exact optimum under this profile. The Hamming/L1 medians did not improve because the reported candidate is selected by the deliberately coupled modular objective; coordinate distance is diagnostic and is not the optimized score. Fewer fitted-TPE trials also lowered measured sampler overhead.

Reproduction

The current commands are documented in README.md. This published evidence was produced with:

cmake -S benchmarks/optuna -B build/benchmarks/optuna-evidence \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build/benchmarks/optuna-evidence \
  --target pineforge_hpo_optuna_native --clean-first -j4
 
build/benchmarks/venv/bin/python benchmarks/optuna/run_benchmark.py \
  --native build/benchmarks/optuna-evidence/pineforge_hpo_optuna_native \
  --output benchmarks/optuna/evidence/2026-07-18-full.csv
 
build/benchmarks/venv/bin/python benchmarks/optuna/run_benchmark.py \
  --native build/benchmarks/optuna-evidence/pineforge_hpo_optuna_native \
  --problems million_discrete \
  --startup-trials 200 \
  --output benchmarks/optuna/evidence/2026-07-18-startup-200.csv

Evidence identities:

Artifact/identity	SHA-256
Full CSV, 60 rows	`7dcc63346df87367d7fe0c151faef369f103508eb3117b2f50009bcb3f83e629`
Full metadata	`2ca193efffc0c25b23948f2c2a8d7c67f819310b4cab633ef7f25cdbc01364ae`
Startup-200 CSV, 10 rows	`479c28e9f3825fb152e1b0193350af98922be6bf528c618c04091eb75c207e9a`
Startup-200 metadata	`896cd01dec6039c75057935250ae668c51772aa549b008c25c00406f1ba38214`
Source snapshot shared by both runs	`12abddc13f35933acaebf80772a74e3b9fa0af3d637c05f6afc0713a616fb404`
Native executable shared by both runs	`2b624c6bcad5b5bb9bcbb2198258054d53658fd32a5ab67198d15b354bd4d732`
Full configuration	`2c5da55f8257737d69fb3a9b676631394b68886f0c5b0f117a1deb3326b80502`
Startup-200 configuration	`5e751da2a92f90be53a444a8ff0ea1f410652855db4d75e924da8a1d45e49b2d`

The startup-200 driver's wall time was 40.14 seconds, with 1.32 seconds of pre-run exhaustive enumeration recorded separately. Summed in-loop wall time was 17.874 seconds native and 22.230 seconds Optuna. File hashes, row counts, configuration, validation results, and exact environment are machine-readable in the sidecars rather than reconstructed from this prose.