Whenever "the new math" comes up, I must point out that there was no one "new math" curriculum and much of that post WWII exploration into math curriculum and pedagogy had different aims and philosophies (Davis, 2003). (This doesn't directly pertain to your question, but the actual use of new instructional methods and curricular content during this period was also not widespread or deeply penetrating into the educational system of the USA.)
So, in order to get a really good answer when talking about what was going on in the post WWII era, it may help to talk about a specific curriculum. Because, while some of the projects doing research at the time may have incorporated set theory, they may have done so for different reasons and there may not be one unifying philosophy. There possibly 30 or so projects that could be associated in some way with that era.
Angela Lynn Evans Walmsley (2003) sought to chronicle that time of reform and focus on the most prominent projects of the so-called "New Math movement," and she has this to say about the use of set theory:
Also, the language of sets and abstraction became part of the "new
mathematics". Both set theory and abstract algebra became known
synonymously with "new mathematics," but neither was a large part of
the curriculum. (p. 75)
Later in that same book, the author makes reference to Morris Kline, an outspoken opponent of curriculum reforms who specifically attacked the requirement of using precise language and the inclusion of set theory.
My own conjecture is that the prominence of set theory in the minds of people when they consider "the new math" may be, in part, attributable to the critics who focused on set theory as either too abstract, or as substituting for the arithmetic that was the focus of a lot of traditional mathematics education approaches. Possibly, people saw it as something associated with the professional mathematicians who, in part, drove some of the ideas behind new (but not necessarily research-based) approaches to mathematics education. For whatever reason, its close association does appear to be disproportionate.
As for why it is there in the first place, the question would really need to be more specific.
My general way of addressing the question (without a reference to back it up) would be that a structuralist approach to mathematics influenced the thinking of some mathematicians who wanted to see students build up their knowledge from axioms. Mathematicians might have been paying close attention to the writings of Bourbaki at the time, but to get reasons of set theory's inclusion in a curriculum, it would really be better to focus on a specific curriculum or set of related curricular materials.
Works Cited:
Davis, R. B. (2003). Changing school mathematics. In G. M. A. Stanic & J. Kilpatrick (Eds.), A history of school mathematics (Vol. 1, pp. 623–646). Reston, VA: National Council of Teachers of Mathematics.
Walmsley, A. L. E. (2003). A history of the “New Mathematics” movement and its relationship with current mathematical reform. University Press of America.
