I have no problem with the mathematics of it. On paper time, past and future, can be infinite and the math will all work out.
However, when we translate this to reality, here is the problem I see. Future time can be theoretically infinite, but in fact it never truly will be. We will advance in time and it will be an ever growing number yet we won't ever reach infinity, ie we will only tend towards it.
Similarily, past time is an ever growing number as time passes by, and in this regards also tends towards infinity. Yet to say that it is infinite means that it no longer tends towards infinity, but in a sense that it has reached it.
It's the difference then from the potential of being able to go on forever, and that we have already been going on forever.
I hope it's clear cause I don't have the feeling it isn't. As I said, I don't have any problem when discussing infinite time mathematically, only when we are supposed to apply it to reality.