The effect is only minimized if Natural Selection is not involved.
On the other hand, in small populations Genetic Drift can rapidly move populations away from adaptive features provided that selection is sufficiently relaxed. If enough new niches are available, it's possible that those niches will eventually be occupied by new species even though the initial impetus went against weak selection.
So there you have it: an example of Sewell Wright's Shifting Balances Theory unfolding right in front of our noses.