I am wondering the following. Suppose we have some particular space $X$ and $x_1,x_2,x_3,x_4\in X$ has the law of composition that works like Klein four-group. Is it correct to say that the structure that $x_i$'s forms is the Klein four-group or is the structure isomorphic to Klein four-group? I'm confused as when I learned algebra, there was not specified that in what space does the elements of some group, like Klein four-group, belongs to.
It depends on what foundation of mathematics is being used. Certain foundations, such as Zermelo-Frankel set theory or univalent type theory, have a notion of equality between set-level algebraic structures in a universe, while other foundations, such as the Elementary Theory of the Category of Sets, do not have such a notion of equality. So one may speak of either equality or isomorphism of set-level algebraic structures in the former foundations, but only of isomorphism of set-level algebraic structures in the latter.
Since it is not true that one can speak of equality of set-level algebraic structures in all foundations, it would be better and clearer to simply use isomorphic/isomorphism in teaching for all set-level algebraic structures such as groups, since teaching is foundation-agnostic in practice.
Identifying structures with their isomorphism equivalence classes is a very useful perspective, but it can get really confusing if it's someone's primary perspective.
Specifically, when one starts talking about subgroups (e.g. how many subgroups of G are isomorphic to the Klein four group?), one runs into problems: are all subgroups equal, since they're isomorphic? Are subgroups equal if there's an automorphism of G taking one to the other? Neither of these are particularly the question we'd like the answer to.
One should teach that in some situations you care about strict equality, in some situations you care about isomorphism, and doing a lot of fussing about terminology in a situation where you don't really care about the difference won't help you figure out what to do in a situation where you do care about the difference.