If you can't express your thoughts correctly in plain English, why should I believe that you are on top of the subject you are talking about?
What about
Stephen Hawking?
He's a good example of someone who can't express their thoughts correctly in plain English but is still on top of the subject they are talking about.
Other smart people are just lazy. They could be very well on top of their subject but just don't want to take the time to dumb it down to 'plain English' so people will believe they know what they're talking about. Or maybe they are so far out there that proper english is the difficult part.
Why not just judge people by the way they look instead of the way they communicate? Its an easier form of prejudice.