Do only the fit survive? If so, then why again do humans breed defective babies and what other explanation do you have for these primates dying out?
"Survival of the fittest" is a relative term. 'Fit' does not necisseraly mean: stronger, faster, smarter, bigger, etc. 'Fit' means whatever it takes to survive in any given environments.
So, while in a desert it's probably not very benifficial to have thick layers of fur and blubber, it's a great advantage to have when you are in the Arctic. See how it works?