When I first started teaching in Spanish I wrote my lectures out in full before every class, something I would never do in English (my native language). This required consulting textbooks written in Spanish and speaking to other teachers to find out what is standard usage and mathematical terminology. Such consultation is essential. Since mathematical terminology mostly comes from English, French, and German there sometimes are not native Spanish expressiosn for technical terms. Dictionaries rarely are useful for finding out what is correct mathematical usage. Much mathematical terminology in Spanish originates in French usage rather than English usage.
When actually giving class, it is important to write everything that one says on the board. If students struggle with one's accent, at least they can read what one writes. For this it is of course important that what one writes be more or less correct grammatically and orthographically. For this one sometimes has to write lectures in far more detail than one would in one's native language.
Something I have encountered frequently is that theorems are known by different names in different cultures. The (some) theorem expressing the dimension of the space of solutions of a system of linear equations in terms of its rank is known as the Rouché-Frobenius theorem in Spain. It is called differently in other countries (in the US it doesn't usually get given a name). One needs to learn these local customary names and terminology. It helps for communicating with students. Otherwise what happens is that students know some theorem as Bolzano's theorem and don't realize the professor is talking about Bolzano's theorem because he calls it something else.
If one teaches at the university level, for example calculus or linear algebra, one needs to familiarize oneself with what is taught at the high school level. Terminology used in primary education is often peculiar, highly regional, and quite different from "professional" use. For example, in Spain what most (?) mathematicians would call "convex" is called "concave up" in high school, where students are taught to speak of "concave up" and "concave down". Teaching without awareness of such issues one can generate a lot of confusion.
I found that explaining math in a second language obliged me to use a simpler, more colloquial language than I would use in my native language. My experience was that this often was positive from the pedagogical point of view. For example, I found myself explaining Cavalieri's principle in terms of presliced loaves of bread. It's a helpful metaphor I continue to use now.
When giving exams, or any graded exercise, it is absolutely essential to have a mathematically competent native speaker proofread them carefully. I once (unknowingly) wrote a problem whose meaning changed materially depending on whether como/cómo had or lacked an accent mark.
Finally, the difficulty of teaching in a second language will not last long. There is probably no better way to improve at speaking and writing a language than by giving classes in that language.