Rather than an incurable plague or asteroid impact, how 'bout good ol' fashioned ecological collapse?
I don't think you even have to go that far.
If there was just a small nudge to the climate, nothing major but enough so that harvests became unpredictable. Look at what happened in Europe during the Little Ice Age and actually continuing right up into the mid 1800s.
If we could not predict what areas will be productive, say four out of ten years, for a period of a century or so, what would be left?
Aslan is not a Tame Lion