I think the major reason is the lack of interest on the part of most people on the topic or to learn enough to have any kind of real discussion.
I notice a real lack of interest to discuss Genesis on spiritual terms. Often when the matter of God's eternal plan as shown in Genesis is discussed people do not have the patience or interest to go carefully into the subject.
They often quickly want to talk about Cain's wife or what they perceive as other "problems" with Genesis. How Genesis contains all the seeds of great biblical truths in their initial and form is often of no interest to them.
The purpose of man's creastion, the opposition to God's plan, the promise of redemption, the symbols revealing the death and resurrection of Christ, and the symbols of the Triune God and His eternal plan are all crucial subjects in Genesis which are quickly discarded in favor of talking about Evolution or Ussher's chronology of the age of the earth.