They don't have to be. Swans can be aggressive and are good fighters, and the northern hemisphere ones may have developed the white colouring in order to appear larger to potential preditors. An extrovert strategy, being loud and bold. However, swan colouring could have been largely neutral evolution, as the existence of both mixed coloured and all black swans in the southern hemisphere might indicate. See link below.
It means that color of swans have no survival advantage - they can be white or black as well. Color of their plumage was not selected by "Natural selection" so in this case "Natural selection" is meaningless. It support thesis that "Natural selection" in many cases is no relevant explanation of coloration of species. Consequently "Natural selection" as explanation of evolution is in many cases only darwinistic fancy.
Modern evolutionary theory does not explain everything by natural selection. Neutral evolution (via genetic drift) also happens. The swan colours could well have been the result of natural selection (the ancestors of the northern whites and southern blacks would have evolved in different environments and in relation to different predators) but it could also be the product of genetic drift. If there's no significant advantage/disadvantage in any particular colour, entire population groups could still become entirely white, or black, or piebald.
There are two questions: 1) do predators of white (black) swans in North hemisphere (South) still exist? What species they are? If they are extinct - do we know something about them?
My guess is that preditors on their eggs and their young were more likely to have been of importance. They mate for life, so you have two large aggressive birds to defend the nest. I doubt if they made regular prey for anything as adults. A large bird which can swim and fly is not an easy target. They are certainly capable of attacking mammals as large as us if we approach their nests or young. That's an example of a behavioral characteristic which would certainly have been driven by natural selection, for reasons that should be obvious to you.
2) If coloration of swans is "product of genetic drift" without any "advantage/disatvantage" - in how many species do we see such "genetic drift"? Let say we see it in 99% of extant species (give another number if you like). Does in such case play "Natural selection" significant role in coloration of species? If not we should reject selection as driving force of coloration of species.
Of course natural selection can play a significant role in the colouration of species. Obvious examples are animals (and birds) that are clearly well camouflaged. Sexual selection also plays a role, which is why we get such weird and wonderful birds as peacocks, and so can genetic drift. Sometimes the reason for an animal's colour scheme may not seem obvious to us, but it must be remembered that we do not have the same vision as the animal's preditors, nor that of the animal itself. The zebra's stripes may be confusing to the eyes of the cat family, for example.
If you observe the colour of a species and it appears to have no advantage to that species, then it could mean that genetic drift is the story, but it could also mean that the colour had an advantage to the species' ancestors in very different circumstances.
Genetic drift characteristics are common, and you can see them in our own species. The whole point about natural selection is that it acts to eliminate disadvantageous characteristics and to promote advantageous characteristics. It does not act on neutral trivia, and it's possible that the swans' colouring is largely that.
Interestingly, you seemed to think that you had some point about swans being white that backed up your arguments in some way, but when I pointed out that they are not all white, then that seems to become something that backs up your arguments.
I doubt if I'm the only one who finds that amusing.;)