If a maths lecturer is lecturing a higher-level undergraduate course outside their area (i.e. has not taken a course in themselves or learned themselves over the course of their research), how could you justify to the students that they are suitably qualified to teach and assess them?
The comments are very interesting, but you are talking about how to justify to students. That may not always be necessary, but here is a stab at it if someone really asked (which I find somewhat unlikely).
The instructor has a PhD in mathematics, which means that s/he has learned not just how to learn independently, but how to become an expert in mathematical subject matter. Given the interconnectedness of mathematics, I am confident that Dr. X will do an excellent job not only with the subject at hand, but how it fits into your studies in general, both before and after this class.
Some may disagree with this characterization, but anyway I think it's more true than not.
Obviously the first time won't be the best, but that is something that can be said for any course when taught the first time, including freshman calculus/college algebra/finite mathematics or whatever your "first" college course happens to be. How many people really "get" the Mean Value Theorem until they have to teach it? Proving random stuff with it in some undergraduate real analysis course also likely lies years behind instructor Y or Z, not just instructor X.
Usually undergraduate level courses are not very difficult for experienced researchers/lecturers. Even without previous research experience in the specific topic, lecturers can learn the material very fast and be able to teach the topic to students well enough afterwards.
However, it may be the case that the lecturer appears to be unqualified or unfit to teach the material, but based on my personal experience I think this often reflects on the lecturer's ability to teach rather than their knowledge on the subject.
In France, my answer would be that we only hire faculty who can teach every mathematical subject at every undergraduate level (possibly with quite some work, of course). This is most usually included in the job description, and is pretty accurate in practice.
One thing that makes it relatively easy to ensure that a good majority of faculty have this broadness is that a good majority of us have taken Agrégation, a national competition intended to hire high school teachers of higher mathematical level, where one has to demonstrate a good mastery of a big chunk of undergraduate mathematics.
Another point is that the higher education in Écoles Normales Supérieures is very broad: one does not specialize too much too early; and a good majority of faculty in France went to one of these ENS. Of course, we also hire people whose higher education was at a university, and foreign mathematicians, but many of them are equally broad, and all in all I think the vast majority of faculty is able to teach any undergraduate course, if needed.
Assessment is the easy part, anyone who is well versed in the logic of mathematics should be able to vet the student's proofs and computations.
Teaching is less clear. You want the lecturer to know what the important parts are. If they aren't familiar with the subject, how are they going to know where the most fertile ideas can be found? (This can be remedied if there is a good textbook available.) Still, you can always run the class --- not as a lecture --- but as a seminar where everyone is upfront about the fact that everyone, including the professor, is learning the subject together. I've certainly had classes like that in graduate school, for emerging topics. They were fun!