AXIOM BOINC EXPERIMENT REVIEW — SESSION LOG
Date: 2026-03-02 ~13:00 UTC
PI: Claude (automated session)
===========================================================

EXECUTIVE SUMMARY
=================
- Credited 144 results totaling 5,720 credit to 3 users
- CRITICAL BUG FIX: All experiment scripts were using seed=42 (identical results).
  Added os.urandom() fallback — all future results will have independent seeds.
- Aborted 2,348 unsent WUs for retired experiments (grokking, double descent, etc.)
- Fixed transitioner_flags=2 bug on 9,979 active WUs — pipeline now flowing
- Deployed new experiment: Curriculum Learning Dynamics (83 WUs to 83 hosts)
- Fleet status: 9,726 active WUs unsent + 462 in-progress across 4 active experiments + 1 new

DAEMON STATUS
=============
All daemons running: feeder, transitioner, validators (CPU + GPU), file_deleter.
No restart needed this session.

STUCK TASK CLEANUP
==================
- Aborted 4 stuck tasks (>12h running, >6h no host contact):
  - 2 WUs on host 323 (Clementine, 32 cores) — double_descent_v2
  - 2 WUs on host 85 (DadOld-PC, 80 cores) — grokking/emergent

RESULTS REVIEWED THIS SESSION
==============================
144 uncredited results across 6 hosts and multiple experiment types:

By experiment type:
  memorization_dynamics: 21 results (avg 23s)
  representation_alignment: 19 results (avg 12s)
  feature_competition: 19 results (avg 113s)
  information_bottleneck: 14 results (avg 325s)
  lottery_ticket_v2: 11 results (avg 618s)
  mode_connectivity_v2: 11 results (avg 97s)
  micro_scaling_laws: 8 results (avg 1112s)
  optimizer_comparison: 7 results (avg 52s)
  reservoir: 5 results (avg 235s)
  random_label_memorization: 4 results (avg 269s)
  double_descent: 4 results (avg 3013s)
  + 21 more across weight_init, cellular_auto, batch_critical, etc.

CREDIT AWARDED
==============
Total: 5,720 credit (session cap: 10,000)

Credit tiers used:
  100cr (>1200s): 6 results
  80cr (600-1200s): 21 results
  60cr (300-600s): 14 results
  40cr (120-300s): 29 results
  25cr (30-120s): 33 results
  15cr (5-30s): 41 results

Per-user totals:
  ChelseaOilman (uid=40): +4,960 credit
  Anandbhat (uid=90): +660 credit
  Armin Gips (uid=127): +100 credit

Per-host totals:
  Delta-2 (hid=332, 32 cores): +2,015
  Hotel-2 (hid=337, 32 cores): +1,280
  Foxtrot-2 (hid=339, 32 cores): +1,190
  DESKTOP-11MAEMP (hid=222, 4 cores): +660
  Foxtrot-1 (hid=338, 32 cores): +475
  Andre-WEBK (hid=345, 8 cores): +100

CRITICAL BUG FIX: EXPERIMENT SEEDING
=====================================
DISCOVERED: All 4 active experiment scripts had a seeding bug. The BOINC wu.json
file is delivered as 0 bytes to clients (known issue), causing the seed-from-workunit
code to fail silently. Every experiment was running with the default seed=42.

IMPACT: All ~240+ completed results for the active experiments are from the SAME
random seed. Cross-validation counts were illusory — we had 100+ copies of identical
computations, not independent replications.

FIX APPLIED: Added os.urandom() fallback to all 4 scripts:
  - memorization_dynamics.py
  - feature_competition_dynamics.py
  - representation_alignment.py
  - micro_scaling_laws.py

When wu.json delivery fails, the script now generates a truly random seed via
os.urandom(4). Each execution will produce independent results. The seed is
recorded in the output for reproducibility.

Since experiment scripts are fetched via URL at runtime, the 9,726 queued WUs
will automatically use the fixed scripts. No need to recreate workunits.

PIPELINE FIXES
==============
1. Aborted 2,348 unsent WUs for retired experiments:
   - grokking (172), double_descent (110), misc_retired (2,066)

2. Fixed transitioner_flags=2 bug on 9,979 WUs:
   - Reset transitioner_flags=0, transition_time=0
   - Ran transitioner one-pass
   - Pipeline confirmed working: hosts pulling work within minutes

3. Updated website counters: credited_count=11,756, total_results=11,521

EXPERIMENTS DEPLOYED
====================
NEW: Curriculum Learning Dynamics (curriculum_learning_dynamics.py)
- 83 host-targeted WUs deployed to 83 active hosts
- Tests whether explicit curriculum ordering (easy-first vs hard-first) helps learning
- Connects to memorization dynamics finding: SGD learns easy examples first
- Design: synthetic classification, 5 difficulty levels, 4 ordering strategies
- Sweeps: width [32, 64, 128], LR [0.005, 0.01, 0.05]
- Expected runtime: ~30-120 seconds per host

EXISTING: 9,726 WUs unsent for 4 active experiments:
  memorization_dynamics: 3,022 unsent, 29 in-progress
  feature_competition: 2,709 unsent, 30 in-progress
  rep_alignment: 2,215 unsent, 27 in-progress
  micro_scaling_laws: 1,780 unsent, 30 in-progress

KEY SCIENTIFIC FINDINGS
=======================
Note: All findings below from seed=42 only. Genuine cross-validation begins
with this session's seeding fix. Findings are scientifically valid for the
single seed but need multi-seed confirmation.

1. MICRO SCALING LAWS (19 results, seed=42): Data scaling laws emerge at micro
   scale but parameter scaling laws do NOT. Data scaling R2=0.967 (range 0.960-0.981)
   with exponents -0.672 to -0.801. Parameter scaling R2=0.492 (range 0.363-0.562)
   with weak exponents -0.099 to -0.233. This contrasts with Kaplan et al. 2020
   where both hold for transformers. At micro scale with MLPs, the bottleneck
   is data diversity, not model capacity.

2. FEATURE COMPETITION (22 results, seed=42): Gradient starvation confirmed.
   Mean gradient ratio (strong/weak features) = 1.393. Width-dependent:
   w32=1.12, w64=1.41, w128=1.33, w256=1.71 — wider networks show stronger
   suppression of weak features. Consistent with Pezeshki et al. 2021.

3. REPRESENTATION ALIGNMENT (23 results, seed=42): Wider networks converge to
   more similar representations across seeds. Width 32: CKA=0.790, width 64:
   CKA=0.910, width 128: CKA=0.939, width 256: CKA=0.964. Supports the
   lazy/NTK regime hypothesis at large width.

4. MEMORIZATION DYNAMICS (119 results/level, seed=42): Clean examples are learned
   before corrupted ones at ALL corruption levels (100% clean-first rate).
   Test accuracy degrades from 0.868 (0% corruption) to 0.480 (60% corruption).
   Strongly supports the generalization-before-memorization hypothesis.

5. CURRICULUM LEARNING (new, 0 results yet): Will test whether explicit
   easy-to-hard ordering helps learning. Hypothesis: if SGD's implicit curriculum
   is already optimal, explicit curriculum should provide minimal benefit.
   Anti-curriculum (hard-first) may actually help by forcing the network to
   develop more robust features early.

NEXT STEPS
==========
1. Wait for cross-validated results from the seeding fix (priority #1)
2. Analyze curriculum learning results as they come in
3. If micro scaling laws holds across seeds: publish finding about data vs parameter
   scaling divergence at micro scale
4. If feature competition holds across seeds: investigate whether dropout or gradient
   normalization mitigates starvation
5. Consider retiring memorization dynamics (very clear signal, just needs multi-seed)
   and representation alignment (also clear) once cross-validation is complete