That's true, but it's a matter of degree. In the end, it comes down to what it takes to convince ourselves that something is true.
Any argument must pass the initial smell test. If you can show that a leader/expert in a field has come to the same conclusions as you have then it passes the initial smell test. It's a way of saying, "See!!! I'm not crazy!!". An appeal to authority is a way of enticing interest in your argument, and it is also a way of ensuring others that you aren't nucking futs.
For example, let's say you had never heard of quantum mechanics. I tell you that light can act as both a wave and particle, but not both at the same time. You find such an idea to be laughable. I then tell you that thousands of physicists also believe that light can act as both a particle and a wave. Now it isn't so laughable, and it might even be true. You will still want to see the evidence, and rightly so, but you are more likely to take the idea seriously now then you were before.