I don't think it is the math that is the problem. But any theory which produces infinity for an answer somewhere has a problem with the physical interpretation of the math.
It is my weak understanding that currently the idea is that by quantizing GR we get rid of the possibility of a zero sized massive object and so get rid of infinite density.