Set theory, as well as being a subject in its own right, is a convenient language in which most of mathematics can be expressed.
English, as well as being a subject in its own right, is also a convenient language in which most of mathematics can be expressed, but nobody would ask "Why isn't English taught at the outset of math education?"
When learning a new subject, it's useful to go somewhat in order of historical development. That's one reason why we learn whole numbers before trigonometry, for example.
Expressing the fundamentals of every part of mathematics in terms of sets is something that people have only really been doing for the last century or so. It's quite a sophisticated point of view which doesn't fit very well with understanding what mathematical objects actually are.
For example, the concept of ordered pair is much easier to understand if it's presented in the usual way ($(a,b)$ is a thing with the property that $(a,b) = (c,d)$ if and only if $a=c$ and $b=d$) rather than defining $(a,b)$ as the set $\{\{a\}, \{a, b\}\}$.
On a personal note, I can remember in primary school doing some set theory with strings and objects.
You put the objects on the floor and had to put loops of string around some of the objects to create sets. It all seemed rather boring and pointless. It wasn't until I started university, or maybe the final year of high school, that I realised what sets had to do with mathematics at all.