My job is starting to have me delve into categories that require things like regression analyses on data sets, essentially I'm being introduced to "Data Science" type material. Coming from a computer science background however, I'm aware of how easy it is to misapply statistics.
My only course on statistics was over a decade ago and it didn't include any of the theoretical underpinnings that actually explained why the formulas worked, which would help me understand when to use method X vs method Y. I'm dreadfully afraid of creating false findings.
I'm looking for teaching resources that would hopefully bridge the gap from "I can do X, Y, or Z" but would give enough math to help me understand when and where to use what. If there's anything like Roger Penrose's "The Road to Reality" for statistics, that would help. Otherwise any suggestions of what a good course plan for self-study would look like would be welcome.
[EDIT] Clarifications:
Having a comp sci background, I know for example that if the problem before me seems like a graph-theoretic problem, I can reach for my copy of "Graph Theory and its Applications" from Gross/Yellen.
Physics isn't my forte, but when I want to leverage the mathematics I do have, I can reach for "The Road to Reality" by Roger Penrose.
And if it's dealing with Data structures, I can check out "The Art of Computer Programming" from Donald Knuth, and for algorithms--although it's a textbook--there's also Cormen's "Introduction to Algorithms," or even Skeina's book "The Algorithm Design Manual." All of these works provide enough theory and proofs to make it pretty clear why these tools work. I'm fishing for items in this same category for Statistics, but judging by the comments, there isn't quite this level of organization and unity in statistics?