'Genetic drift' refers to the changes in allele frequency between one generation and the next in a population which can be attributed to random factors. What you describe, natural random variation in mating success, could be one such factor. Being hit by a tree falling over clearly does curtail an organisms chances of reproductive success.
Your monogamy issue is pretty abstruse though, and certainly not considered a particularly important element of genetic drift as far as I know.
TTFN,
WK