# NOTES — PGN-style flat-maxima attack vs MI-FGSM, compute-matched

## What actually ran

`bash run.sh` -> `attack_experiment.py --grad-budgets 36,72 --seeds 0,1,2 --n-images 128`.
6 (grad_budget, seed) combos x 3 arms. Total wall time 234s (see `results.json.runtime_sec`,
log timestamps 11:03:38–11:07:33). Surrogate = `cifar10_resnet20`, targets = `cifar10_vgg11_bn`,
`cifar10_mobilenetv2_x1_0`, `cifar10_shufflenetv2_x1_0`, `cifar10_repvgg_a0`, all from
`chenyaofo/pytorch-cifar-models` via `torch.hub`.

## Setup problems hit and how they were resolved

1. **`torchvision.datasets.CIFAR10(download=True)` from `cs.toronto.edu` was unusably slow**
   (~80 KB/s measured, would take ~34 min for a 163 MB file — most of the whole 45 min budget).
   Switched to the HuggingFace parquet mirror
   (`https://huggingface.co/datasets/uoft-cs/cifar10/.../test-00000-of-00001.parquet`), which
   downloaded the 24 MB test split in ~1s (27 MB/s). Needed `pyarrow` (not preinstalled) —
   installed via `uv pip`, took ~1s for 47.8 MiB, logged in `results.json.env.extra_packages`.
   Verified this gives the standard CIFAR-10 test set (10000 images, 10 classes x 1000) and
   that normalization constants `mean=(0.4914,0.4822,0.4465)`, `std=(0.2471,0.2435,0.2616)`
   reproduce ~91% clean accuracy on resnet20 vs the repo's reported 92.6% (small-sample gap,
   confirms the pipeline is correct). `run.sh` re-downloads this file if `data/` is missing.
2. Timing probe (per plan's implementation_notes): batch of 128 images, fwd+bwd on resnet20 =
   0.32s, well under the 2.5s cutoff — kept `n_images=128` as planned, no downscale needed.
3. All 5 model checkpoints and the `torch.hub` repo zip downloaded fine directly (GitHub, fast),
   no need for the direct-release-URL fallback the plan asked to have ready.
4. `bash run.sh 2>&1 | tee logs/run.log &` run in the background; used `ps`/log-tail polling
   instead of a blocking wait so the 2-process CPU limit wasn't violated (only the Python
   attack process itself was ever running at 196% CPU / ~2 threads).

## Hyperparameters not specified in the plan (had to be chosen; not grid-searched)

The plan gave `alpha = 2.5*eps/T` and grad-call accounting but never pinned down `eps`, or PGN's
neighborhood radius / prediction step / combination weight. Chose, and did not tune:
- `eps = 8/255` (standard CIFAR-10 adversarial-example budget in the robustness literature).
- PGN: `N=3` samples (per plan), neighborhood radius `zeta*eps` with `zeta=3.0`, prediction
  step `chi=eps`, combination weight `balance=0.5` (equal weight on the sample-point gradient
  and the sign-step "prediction point" gradient), momentum `decay=1.0` (same as MI-FGSM, so the
  only structural difference between arms is the flat-maxima gradient combination, not momentum).
These are defensible first-order-approximation choices consistent with the plan's description of
PGN's per-iteration structure (3 samples x 2 grad calls, weighted combine), but this is explicitly
**not** a literal reproduction of arXiv:2306.05225 (which the plan already scopes out via its
`out_of_scope` list — no claim here is about matching that paper's numbers).

## Results summary (full numbers in `results.json`, per-seed traces in `logs/run.log`)

| arm | grad_budget | transfer_success_rate (mean±std over 3 seeds) | whitebox | neighborhood_grad_norm |
|---|---|---|---|---|
| baseline-mi-equal-steps | 36 | 65.8% ± 1.1pp | 100.0% | 58.5 |
| baseline-mi-equal-budget | 36 | 66.1% ± 0.4pp | 100.0% | 61.3 |
| pgn-flat-maxima | 36 | 65.6% ± 4.16pp | 98.2% | 77.7 |
| baseline-mi-equal-steps | 72 | 66.2% ± 0.2pp | 100.0% | 57.2 |
| baseline-mi-equal-budget | 72 | 65.5% ± 0.5pp | 100.0% | 62.8 |
| pgn-flat-maxima | 72 | 73.0% ± 2.52pp | 99.2% | 71.0 |

## Prediction check (pre-registered rule, applied mechanically — see `main()` in
`attack_experiment.py` and `results.json.diagnostics`)

Part A (premise): PGN vs equal-steps MI-FGSM advantage, pooled over both grad budgets and all
3 seeds (6 runs) = **+3.29pp** — below the plan's own 5pp bar for "the literature-bias premise
even replicates here". With only `N=3` neighborhood samples (vs. the original paper's 20, an
explicit scope reduction for CPU feasibility) PGN's edge over MI-FGSM barely shows up even in
the artificially favorable equal-iteration comparison. Because Part A didn't hold, the sharper
Part B question (does the advantage survive compute-matching) can't be cleanly evaluated — the
thing we set out to shrink was already small. `prediction_outcome = "inconclusive"`,
`negative_result = false` (the prediction was neither confirmed nor refuted, per the rule; it
was undercut by the N=3 downscale, which was declared out-of-scope in the plan, not adjusted
after seeing results).

For the record, the compute-matched (Part B) comparison itself pooled over both budgets and all
3 seeds: PGN vs equal-budget MI-FGSM = **+3.45pp**, with per-run deltas
`[+5.3, -5.7, -1.4, +10.4, +4.7, +7.4]` pp — not all positive, so even taken alone this would not
meet the refutation bar (+5pp and all-positive) either. It's noisy and budget-dependent: at
grad_budget=36 PGN is statistically indistinguishable from both baselines (std 4.16pp swamps any
mean difference); at grad_budget=72 PGN pulls ahead by ~7-8pp over both baselines. This budget
sensitivity itself is a real finding, not noise-fishing — it's visible in `figs/*.png` and is
exactly the kind of pattern that "same iteration count" comparisons in papers would obscure.

## Secondary finding: flatness moved the *wrong* way

The plan's `neighborhood_grad_norm` metric was meant to test whether PGN's transferability
(such as it is) is mediated by producing flatter maxima (lower gradient norm around the AE).
Instead, PGN's adversarial examples sit in **less** flat regions than MI-FGSM's (77.7 and 71.0
vs. ~57-63 for both MI-FGSM arms, at gb=36 and gb=72 respectively) — the opposite of the
mechanism the method is named for. Caveat: this uses N=3 samples and untuned zeta/chi/balance,
so it says "this cheap approximation of PGN does not empirically produce flatter maxima under
these settings" — not "PGN's flatness mechanism is wrong" in general. Still, it means any
transfer gain we did observe at gb=72 is not explained by the flatness story as measured here.

## Completeness caveat

PGN's whitebox success rate is not saturated at 100% (98.2% at gb=36, 99.2% at gb=72), unlike
both MI-FGSM arms (100% at every budget/seed). Per the plan's own metric description, transfer
numbers for an arm that hasn't fully broken its own surrogate are harder to interpret cleanly —
flagged here rather than glossed over.

## Deviations from plan

None required — ran at full planned scale (128 images, 3 seeds, both grad budgets, all 3 arms).
`results.json.deviations` is empty for the real run (only the `--quick` smoke-test run, not
part of the reported numbers, carries a deviations note).