My very favorite example of a vestigial organ - and one that correlates beautifully with family trees of mammals - is the vomeronasal organ. Not exactly a household word, huh? A wide variety of mammals have 'em, though. If you've ever seen a bull check out a heifer, you may have noticed how he curls his lip and snorts in violently. What he's doing there is checking out if she's in heat - if she is releasing pheromones - by pulling in "scent" across a pair of sensory organs in the roof of his mouth. This "VNO" also connects to the lower end of the nose, but has different sensors and different connections than the normal "sense of smell."
The VNO sensors - not just in cattle but in dogs, mice, a most other mammals - are tuned to compounds that are less volatile than the standard olfactory sensors are. Many critters actually have to touch what they want to sniff - dogs licking dogs' butts, mice touching urine, etc, And these critters have what's called an "accessory olfactory bulb," separate from the olfactory bulb that connects to the nose, in their brains. Separate nerves lead from the VNO to this AOB. Separate genes code for the receptors in the VNO and those in the nose. It's a "sixth sense" in the literal sense, and one that's used in both sexes to gather information about mates and potential mates.
Snakes and lizards have VNO's, too - also called Jacobson's Organ. The use them to hunt as well as mate. Whales don't have them. Primates, however, are divided in opinion on whether VNO's are good or not. Lorises and lemurs have fully-developed VNO's. New-World monkeys do, too. Old-World monkeys, apes, and humans don't. Well, we start out to have them when we're fetuses, but they migrate upwards from the palate and shrink to tiny dead-ended tubes before or around birth. We (humans, chimps, and other Old-Worlds and apes that have been studied) also start out growing a accessory olfactory bulb, but resorb it and use the brain space for something else before/around the time of birth.
Despite anatomists looking hard, no sensory nerve connections have been found carrying signals from ape/human VNO's to the brain. That's probably good, since we have no AOB to receive the signals. Biologists have found that boucoups of the genes that code for VNO receptor proteins in mice are inactivated (pseudogenes - they're mutated so the no longer code for anything) in OldW monkeys, apes, and humans.
The pattern of VNO/no VNO matches up precisely with the subdivisions of the primates determined by other anatomical and molecular evidence. The primates that lack functional VNO's are frequently those with full-color vision and with "sophisticated" social structures. Several (think baboons) have females that advertise being in estrus by developing brightly-colored vulva.
We don't need vomeronasal organs as we evolved other ways to check out potential mates. The human/great ape VNO is a realio, trulio vestige from our ancestors that sniffed out the opposite sex. We lost the need to sniff (at least with the VNO -
Tresor perfume still smells mighty fine to my olfactory apparatus) for mates, so there was no "reason" to keep a VNO.
More info - Google or Google Scholar "vomeronasal" combined with Timothy Smith, Kunwar Bhatnagar, or Michael Meredith. Or ask me - I have a fat file folder of this stuff.