In general, arguments from ignorance are very weak but infinite regression is interesting. However, isn't it really the same as a causal loop and therefore illogical? At any rate it would be "the thing that has always been there" and therefore could not be explained.
Well, I would absolutely agree that it is a weak argument because it lacks a definitive explanation and, moreover, infinite regression is simply delaying the problem the First Cause imposes. In that case it would seem that if all things are temporal, at least something about the properties of the universe is either infinite or exists outside of the time-space continuum. This is usually where a God-of-the-Gaps argument comes in to play for religious people. But this too lacks any definitive explanation which still leaves us scratching our heads both scientifically and philosophically.
"Reason obeys itself; and ignorance submits to whatever is dictated to it" -- Thomas Paine