OK, so the system really is very sensitive when only a few ratings are given. Which is, I have to say, very poor design (which is one of the reasons I doubted that it would work like that).
A system which relies on large numbers of ratings to produce meaningful results really isn't appropriate for this setting.
While I wouldn't expect a system as sophisticated as Boardgame Geek's game rating system, some effort to damp down the sensitivity to the first few ratings seems almost essential - if only because it is quite likely that many people will get only a few ratings and they will tend to the extremes (people are more likely to give a '1' or a '5' than a '3').