No, it is not too late.
Not even close.
I failed at my first attempt at college, went back at 28, initially only part-time, and eventually ended up with bachelors degrees in math, physics, and CS. Now I'm in the Ph.D program in CS. You might not be as prepared/ambitious/indecisive/whatever as I was, but it's not too late for you to learn math. I also studied with quite a few classmates who were coming back for another degree or who switched majors late, or who had worked in industry for years (or decades in a few cases). If anything, the older ones were more interested and more focused.
I'm not sure what your Ph.D. scholar friend had in mind; depending on why you're interested and how it goes it's not unlikely that getting a Ph.D. in math won't be for you. But if a Ph.D. isn't for you it won't be because of your age. If you can spend the time and money and are interested in the subject, you can learn it. And you don't even have to be a full-time student.