Would it also be correct to say that a from our position we would (most likely) only ever be able to observe one of the two mutally exclusive points?
No. (Percy, we need a sketch pad facility here)
In flat space, you can have two observers approaching each other at near c, decelerating all the way to zero relative velocity and accelerating away from each other again, and they will never be in causal contact. They will never be aware of each other's existence. But a third observer stationary wrt to the two, sitting between them, will suddenly be aware of them arriving, and from then on will be able to see both of them accelerating away for as long as he cares to watch.
This is analagous to the situation in an expanding universe, where two galaxies in opposite directions can be seen from Earth, but neither is aware, nor ever will be, of the other.
If two point are not seperated spatially but in time would there also be point where they cannot interact?
You can always just view this situation at some velocity such that they will be separated in space *and* time. But they are in one another's light cone. But they could be dragged out of each other's light cone by virtue of the expansion, if they are not at zero spatial separation wrt to the reference frame defined by the expansion (the comoving frame)