Why? Is it unrealistic to say that when you move your hand through water that it is you hand that is slowing down, and not the time? If you move your hand through the air with the same force it will go faster. Does that mean that time is going faster?
No, although these effects are fundamentally different to how relativistic time dilation occurs. Particularly for special relativistic time dilation where the body isn't really interacting with anything. Hopefully the examples below will help.
Couldn't gravity actually be effecting the clock's mechanism?
I think Rrhain dealt with this quite well, so I'll instead concentrate on the next statement.
Again, maybe the particels themselves are slowed down, and not the time. How do you know it's time itslef?
Well first of all, the particles are actually moving extremely fast with respect to us, so they aren't slowed down. The point is that particles moving at huge velocities live longer from our point of view.
Now you could say maybe high speeds has a physical effect on the decay rate of a particle, however this is easy to refute using Galilean relativity, which I will assume you accept. Galilean relativity says that if you are moving inertially then you can't tell you are moving. So these incredibly fast particles in their own reference frame are stationary. Since any physical effect on their decay rate would have to be real in all frames, well then the slowing down of their decay rate would be true in their own rest frame. So what about the particles which aren't moving fast with respect to us? Well they are at rest in our frame, so from the above analysis they should also have their decay rate slowed down.
However they don't, only particles moving fast with respect to us have their clocks slowed down. This refutes the possibility that the decay rate is being affected by a speed based mechanism.