No, that's not exactly how we can be confident that the dates are accurate.
Our confidence in the dates comes from the fact that samples are tested with several different methods and they return remarkably similar results.
How do you propose that all these different methods would return similar dates for the same rocks if they were flawed?
You might be interested in reading the following paper titled, "Radiometric Dating: A Christian Perspective".
Radiometric Dating
An excerpt:
"Radiometric dating--the process of determining the age of rocks from the decay of their radioactive elements--has been in widespread use for over half a century. There are over forty such techniques, each using a different radioactive element or a different way of measuring them. It has become increasingly clear that these radiometric dating techniques agree with each other and as a whole, present a coherent picture in which the Earth was created a very long time ago. Further evidence comes from the complete agreement between radiometric dates and other dating methods such as counting tree rings or glacier ice core layers. Many Christians have been led to distrust radiometric dating and are completely unaware of the great number of laboratory measurements that have shown these methods to be consistent. Many are also unaware that Bible-believing Christians are among those actively involved in radiometric dating.
This paper describes in relatively simple terms how a number of the dating techniques work, how accurately the half-lives of the radioactive elements and the rock dates themselves are known, and how dates are checked with one another. In the process the paper refutes a number of misconceptions prevalent among Christians today. This paper is available on the web via the American Scientific Affiliation and related sites to promote greater understanding and wisdom on this issue, particularly within the Christian community."