Distance between two objects is not a clear indicator of age.
In fact that's exactly what it is.
If you know that it takes a week for a letter to travel between point A and point B, then you know that the most recent the letter could possibly be is one week old. It can't be any younger than that, because if it is, you haven't gotten it yet.
Imagine you're getting these week-old letters from your cousin. You're reading the letter just as your own son is born, and in the letter, you read your cousin announcing the birth of his new daughter.
Because of the distance between you and your cousin, you know that your cousin's daughter is at least a week older than your new son. Not because distance
means age, but because distance means information is coming from in the past, not immediately.
Many of the stars we see are so incomprehensibly far away that the light takes millions or even billions of years to reach us. Far, far more than the 10,000 years creationists say is the maximum age of the universe. How can the universe contain something older than itself? It's an impossibility; thus, the age of the oldest known object is the youngest the universe could possibly be.