Okay think of it this way. Our vision system is based on two types of cells, rods and cones. Cone cells are responsible for our colour vision and in humans there are three types, responding to short, medium, and long wavelengths of light (which correspond to blue, green, and red light respectively). Now the majority of mammals with colour vision have only two types of cone cells, are dichromatic. As far as we know only humans and some closely related primates (and interestingly at least two species of marsupial) are trichromatic, so we're already doing quite well
Now if infrared was to be added to our repertoire of visible wavelengths, it is likely that another type of cone cell would be required. For example, birds can see the same wavelengths as us, but can also see ultraviolet, since they have four types of cone cells. By contrast bees have three cone cells like us and can see ultraviolet, but cannot see light from the red end of the spectrum that we can see.
Rod cells differentiate between light and dark, and are more sensitive to light than cone cells. In animals that are nocturnal, rod cells predominate on the retina. Don't assume that just because you stumble about in the dark, other animals have the same problem, since you are a diurnal animal so will have far fewer rod cells. Of course with fewer cone cells, nocturnal animals have poorer colour vision, and rod cells slower response to light means they are less accurate and less sensitive to movement. Note that rod cells are sensitive to light in the blue-green wavelengths and do not respond to red light.
So for IR vision to function requires an increase in these IR responsive cells would reduce either the number of cone cells, so reducing colour vision during the day, or reducing the number of rod cells and impeding general night vision. Of course it would be more likely both, since as others have said infrared wavelengths have very low energy, and any IR light from the environment will have to compete with IR in the form of heat produced by the animal. This isn't helped by the fact that the retina is backwards, so there is a network of capillaries carrying warm blood between these IR cone cells and the environment.