This isn't confined to humans. Mammals, at least, all (AFAIK) have mechanisms to prevent/discourage littermates/close relatives from breeding. I seem to remember it having a lot to do with smell: a full brother should smell sort of similar, and sister won't put up with hanky-panky from someone like that. Whether this can be extended to smells that you were exposed to as a young child - like an adopted littermate or the toddler next door - I don't know. But it sounds sort of plausible.
Thanks, WK! Your second link leads to a free(!!!!!!) copy of the paper - and it is on a primate that possibly uses odor to 1) prevent incest and 2) promote nepotism.
This adds to the vague bits I posted above - smell having something to do with mating behavior in humans as well as white mice. I can imagine all sorts of research that could be done, but finding a population of undergrad students that would give up all artificial odorants could be challenging.