'Genetic drift' refers to the changes in allele frequency between one generation and the next in a population which can be attributed to random factors. What you describe, natural random variation in mating success, could be one such factor. Being hit by a tree falling over clearly does curtail an organisms chances of reproductive success.
Your monogamy issue is pretty abstruse though, and certainly not considered a particularly important element of genetic drift as far as I know.
Perhaps successful adaptation to niches and slower changes in environment.
The obvious answer to my mind is that generation times vary greatly. A small mammal like the mouse has a generation time of 10 weeks from being born to being able to give birth. In primates it is much longer, for chimps and human it is around 15-20 years. I can certainly see how that could slow evolution down, by a factor of about 78, and the difference compared to a very primitive multicellular life form should be larger again. Brine shrimp for example have a generation time of 3 weeks giving a factor of 260.