Taking that one quote "you didn't build that" is not out of context. The entirety of the speech is meant to convey that exact meaning.
If you look at one interpretation of the phrase "you didn't build that" (ie "You didn't build that on your own - you needed help from other people too"), then I can agree with what you're saying.
The problem is that by quoting the phrase on its own, and out of its context, then it allows alternative interpretations of the phrase (eg "You had no part in building that" or "that was already there and you didn't add to it").
I suspect that the purpose in quoting the phrase on its own, out of context, was to allow for the alternative, more insulting interpretations, which inclusion of the context would clearly rule out.