Most of the research on gender and math education is focused on student gender differences. However, a few references can be found that focus on the what differences there may be based on the gender of the teacher.
One thing that appears to be common among some of these studies appears to be that perception of student performance varies based on gender of teacher and student. For instance, Ehrenberg, Goldhaber, and Brewer (1995) found that white female teachers evaluated their white female students more highly than did white male teachers. Yet they did not find learning differences.
A different analysis of the same data (Dee, 2006) appears to agree that perception differences of student performance exist. And the analysis found performance distances as well. However, the analysis also found that there were confounding circumstances in that same data set that led the author to exclude math classes from his conclusion that the gender of the teacher does matter in school. Dee referred to differences that (he determines) were probably attributable female teachers being assigned more difficult students.
In other words, there is a perception in line with what your colleague suggested, and it appears to influence the outcomes in this particular data set, namely the National Education Longitudinal Survey (NELS) from 1988. Teachers were assigned the less advanced math students, resulting in a finding of lower achievement for students of female teachers of math.
Other studies have noted:
I think we can safely say that this perception you refer to exists, but the basis for it is dubious.
I wonder what reputable evidence people would refer to if challenged in this assumption. And, of course, anyone making such a claim should be challenged to produce some sort of evidence. Based on a look at what's out there, I can say that I would not want to be the one having to defend a claim like this.
Dee, Thomas S. (2006). The Why Chromosome. Education Next. Retrieved from http://educationnext.org/the-why-chromosome/
Ehrenberg, R. G., Goldhaber, D. D., & Brewer, D. J. (1995). Do teachers’ race, gender, and ethnicity matter? Evidence from the National Education Longitudinal Study of 1988 [Electronic version]. Industrial and Labor Relations Review, 48(3), 547-561.