The question
Continual model merging combines task experts without revisiting old data. The published pipeline linearizes CLIP ViT-B/16, merges each new expert with a Fisher-weighted average, and applies a short representation-alignment pass.
Its headline metric, final average accuracy, conflates two different behaviours: retaining old knowledge and simply learning less of each new task. I wanted to know what a task-by-task evaluation would reveal.
The setup
I built an order-aware evaluation harness that saves every post-merge checkpoint, tests it on all ten CIFAR-100 tasks, and produces a 10×10 accuracy matrix plus learning accuracy, backward transfer, and forgetting.
- 27-run main grid: 3 merge schedules × 3 seeds × 3 task orders.
- Controls: fixed λ = 0.4 vs uniform α = 1/t, a coefficient floor, Fisher-proxy normalization, EWC, naive, joint, and independent-Fisher anchors.
- About 235 GPU-hours on a single RTX 4070; 62 archived result files, including 55 complete matrices.
What I found
Uniform α = 1/t cut forgetting 3.4× (0.054 → 0.016) but lowered final accuracy by 1.74 pp. Learning accuracy fell 5.57 pp while subsequent retention improved 3.83 pp — the acquisition deficit outweighed the retention gain.
The scale of the Fisher proxy acts as an implicit merge coefficient: unit-mean normalization changes the effective coefficient, and under a 4/16-epoch stress test it raised aggregate accuracy while lowering accuracy on the four-epoch group by 3.53/2.77 pp. That group effect is exploratory; budget, identity, and position are confounded.
The lesson is not 'uniform is better'. It is that acquisition and retention need to be evaluated separately, and nominal coefficients need to be distinguished from the weighting induced by proxy scale.
What I built
Beyond the experiments, I wrote the analysis scripts that recompute every stored metric from the 55 matrices, the 14-page manuscript, and the A1 poster with A. Baggag. I also self-audited the first draft: credited CoFiMA for the 1/t identity, corrected the first-expert coefficient formula, and removed invalid t-tests.
This is an independent audit of Phan et al. (2025). I am not an author of the original CMM paper.