would this be a fair way to think of it.
From each persons perspective, they are travelling along the time dimension, and the other three dimensions can change in various ways, depending on the geometry of the 4D structure.
Now, if the observer changes his frame of reference, he gets a new perspective on this 4D structure. Some of what he considered to be space is now his time, and vice versa. They are just seeing the same shape from different perspectives, and they can theoretically get very different answers for how "space" is evolving through time.
For instance, imagine a sphere that is stretched out, an oblong. You can travel straight up the object's long axis, and you find that the rate of expansion of the latitude lines (space) is a certain amount.
But, if you had a different perspective and were travelling along the short axis, you'd find that the "latitude" lines were expanding at a much faster rate.
Now, i'm not sure such a radical shift in perspective is possible with the 4D structure, but is this an apt analogy?