I left a comment a while back, but I have been mulling over this (off and on) since then, and feel that I should expand on it. As a first-term master's student, I made some off-hand comment about not liking applied mathematics. My advisor cracked down on me pretty hard at the time, and essentially explained that I lacked the mathematical maturity to have that opinion. You can't dismiss a field until you adequately understand what that field is all about. I could either take some applied mathematics classes and then make up my mind, or I could shut up.
I took a couple of applied classes and learned some nifty stuff. I still don't really dig on applied mathematics, but I have a better appreciation for applied math than I did before. I'm glad that I took those classes, and I now feel pretty confident in my opinion that applied mathematics isn't right for me.
Another anecdote: I have taken a grand total of two physics classes in my life. I took one semester of a combined precalc / physics class in high school, and I took one quarter of quantum mechanics a little over a year ago (I'm a pure math Ph.D. candidate at the moment, so I still take the occasional class). By some measures, I have been relatively successful as a mathematician without a strong background in physics.
However, there are a lot of places where I can see that a physical intuition would be really, really helpful to have. My master's advisor had an uncanny ability to know when my computations were wrong after just a second or two. It turns out that he was doing a lot of quick dimensional analysis in his head, and noticing that the units came out wrong. This struck me as very odd, since we were working in pure fractal geometry, and there were no units anywhere in any of my computations. However, he had a physical intuition about what all of those quantities represented, so it was easy for him to check units.
When I teach multivariable calculus, I can happily explain Stoke's theorem to my students, and even give a pretty convincing proof. What I cannot do is adequately explain to my students how to interpret that theorem physically---I have no intuition for what a flux or curl are (I can parrot back what I've been told, but I don't have a good internal model for these ideas). I don't think that this makes me much less effective as a mathematician (I still work in fractal geometry, after all), but I do think that it makes me somewhat less effective as an instructor in that class.
As a third example, I am currently working with a colleague to study autocorrelation and diffraction measures associated to certain kinds of sets. The project is really my colleague's idea---he is studying quasicrystals---but I am a reasonably good analyst, and there are some tricky questions of functional analysis that come up (how do you take the Fourier transform of the convolution of two measures which may not be finite?). My colleague has a much stronger background in optics, and has an intuitive grasp for what the answers should be based on his knowledge and experience with experimental data. Again, my lack of physics knowledge slows me down a bit (though I am currently learning all about optics and diffraction---yay!).
Long story short: if you have not even completed a bachelor's degree, you are too young and academically immature to have already decided that you abhor physics and don't want to learn it. You simply have not been exposed to enough physics to hold that opinion. It would be good for you to take a few classes, and will make your life easier later.
That being said, it is entirely possible to be successful as a mathematician without a strong background in physics. As I tell my students, I'm a mathematician, not a physician... er... physicalist... uh... physicist.