That memorial is situated between what used to be in an area between East and West Berlin. Surrounded by walls.
The walls were in East Berlin.
The Big Wall right next to Western border (built by East Germany) was seperated from the Small Wall (also built by East Germany) by a few hundred metres, depending on the area.
In that area where the Holocaust Memorial is now, that area was around 500m in width. Nobody was allowed to be in that area without permission. They got shot by East German Border Guards for trying to cross the border. No questions asked.