True, there's many things that are considered as facts that haven't been directly observed. But few, if any of them, are found to be as directly in conflict with people's religious beliefs as common ancestry. Therefore I can "reasonably understand" why some of those people might not regard it as a "fact", while I personally disagree with them
I would disagree , its not just evolution , much of modern science/ knowledge conflicts with religious belief .....
Just think about geology, planetolgy , cosmology, archaeology , particle physics , genetics ..etc etc