Where there are a differing number of chromosomes in the two parents species the chromosomes obviously can't pair up; this means that the gametes produced do not contain a correct number of chromosomes and are thus not viable.
But, as noted upthread, domestic and Przewalski's horses are interfertile, and they have differing chromosome numbers - diploid #'s are 64 and 66, IIRC. So a hybrid has 65, but is still interfertile. I'm guessing that's because the 2n=66 is from a recent fission in a 2n=64, and the resulting broken chromosome hasn't accumulated many mutations.
But I'm not a biologist.....