gssoml
GSSO — Efficient Ghost Model Training
Managed to train the speculative ghost model at a fraction of the usual cost using dynamic layer pruning and asymmetric distillation. Promising throughput results.
I'd been going back and forth for weeks on how to cut down the ghost model training cost in GSSO without sacrificing speculation accuracy.
The vanilla approach required training a full transformer (~7B) just to do speculative decoding — massive resource waste for a model that only verifies predictions.
The fix: dynamic layer pruning + asymmetric distillation.
Instead of training a full model, I train the ghost model with a dynamically pruned architecture: early layers are lightweight (low-rank attention), only the last 4-6 layers keep full resolution. Asymmetric distillation forces the ghost model to learn speculative acceptance patterns only, not the full distribution.
Results:
- Training cost down ~70%
- Inference throughput +45% vs. standard speculative decoding
- Acceptance rate dropped only 0.03 (0.92 → 0.89) — negligible given the savings
Next up: implement dynamic tree drafting with this lightweight ghost model.