I see so much power in the evolutionary process that I am willing to give a bit of credance to the idea that it could have been the extremophiles that were the base for the evolution of multicellular life and then SITAPHS.
I think I'm also suggesting that there are enough "habitable" worlds that the SITAPHS can pick and choose and ignore those that aren't close enough.
It may well be that getting to intersteller travel is still very difficult. It is not at all clear, for example, that we will (in the way we mean here). So if intelligent life is only moderately common (1,000's in a galaxy), spread over perhaps 100 very different types of biosheres and only 1 in ten get to intersteller travel we end up with perhaps a handful in the galaxy who might be interested in Earth and able to get here. If we are lucky enough that all of them can show some restraint and leave other intelligences alone then we are safe.
(I'm completely discounting the possibility of successfull intergalactic travel)