I'm not so sure that he would have considered himself a Geologist. Consider for example his reasons for turning down the position of Secretary of the Geological Society in 1837, long before he published Origin.
All my geological notes are in a very rough state, none of my fossil shells worked up, and I have much to read. I have had hopes by giving up society & not wasting an hour, that I should be able to finish my geology in a year and a half, by which time the description of the higher animals by others would be completed & my whole time would then necessarily be required to complete myself the description of the invertebrate ones.
It appears that his interest in Geology, even decades before Origin, was based on the fact that the biological evidence was there, in the ground, waiting to be explained.
I always felt that the "A-ha" moment for Darwin was when he realized that he could see in the processes going on around him the explanation for the evidence he saw in the fossil record.
Quote from a letter from CD to JS Henslow in October of 1837
She also makes no mention of his early life which was marked by a fascination with bugs and flowers, not particularly rocks.
I think one important point is that at the time, there was not as great a differentiation between subject areas as there seems to be today. Like many of his time, including both of his grandparents, he was a generalist; he wanted to know about everything and how ALL of the parts fit together.
I think we also forget just how much influence the methodology of his Grandfather Josiah Wedgwood would have been. I remember looking at the copius notes that they made on different firings to create a consistent blue color and simply being astounded at the precision and documentation they went through. His experimental methods and scientific approach must have greatly influenced CD's later efforts.