Some math coursecourses will have: simple applications in physics, simple applications in economics, simple applications in biology, simple applications in engineering, and so on. Beginning calculus, or differential equations, for example. So you perhaps will not ace those courses, but still you could do OK.
Now in most cases, when you work on a Ph.D. in mathematics, you will want to (or even be required to) work as a teaching assistant on low-level undergraduate courses. And it may not be possible to avoid beginning calculus courses. So in the end you will, indeed, have to know a small amount of physics.
Back when I was a student taking basic calculus, I had a TA who explained that she could get by with just a few basic notions from physics, and that she never intended to learn any more physics than that.