Wait! You May Not Need to Lop Off So Many Heads