# Self-study Linear Algebra textbook for Machine Learning and Statistics

I am looking for a good linear/matrix algebra textbook, suitable for self-study, that covers topics relevant to statistics and machine learning. I have access to Gentle's "Matrix Algebra", but have found it to be too dry and more of a reference book for a practitioner who's already studied the subject before.

Some of the books I'm considering are:

Seber, "A Matrix Handbook for Statisticians"
Searle, "Matrix Algebra Useful for Statistics"
Harville, "Matrix Algebra from a Statistician's Perspective"
Gruber, "Matrix Algebra for Linear Models" (just came out)


and the more generally oriented:

Strang, "An Introduction to Linear Algebra"
Strang, "Linear Algebra and Its Applications"
Axler, "Linear Algebra Done Right"


The trouble is, all these texts have excellent reviews on Amazon, but so did Gentle's text and it doesn't really suit my purposes.

• If it's for self-study, and not teaching, then perhaps math.stackexchange.com is a more natural place to ask. – Dag Oskar Madsen Oct 19 '14 at 19:47
• You might also consider stats.stackexchange.com or datascience.stackexchange.com. – J W Oct 20 '14 at 5:18
• In applications to machine learning one often encounters calculus of matrix valued functions and functions of matrices. Your best bet may be to seek a book which integrates multivariable calculus and linear algebra; you will want to familiarize yourself with Lagrange multipliers and Taylor series of vector valued functions anyway. Also, there are some topics in linear algebra - like PCA and QR decompositions - which are best learned from a statistics / ML textbook once you have the proper foundations. – Paul Siegel Oct 20 '14 at 11:50
• This is a really old post, but one thing I should say is that Harville and Searle will not have what you're looking for. They do cover the linear algebra needed for stats, but they do not discuss its context within stats. – Clarinetist Oct 29 '17 at 6:16
• @DagOskarMadsen: While I agree (hence my suggestion to consider stats.stackexchange.com or datascience.stackexchange.com as well), answers could be useful for teachers of linear algebra with an eye to machine learning and statistics applications. – J W Jan 9 at 21:25

I used: Axler, "Linear Algebra Done Right" to teach myself the material in university.

I recommend this book because it focuses on understanding the concepts rather than memorizing the proofs. It also has great exercises.

Good Luck!

• A linear algebra text that doesn't mention Cramer's rule? – Gerald Edgar Jan 26 '15 at 15:51
• @GeraldEdgar The classic book Linear Algebra by Hoffman and Kunze does not cover Cramer's rule. It might not be common that a book doesn't include it, but it's not uncommon. – Chris C Jan 27 '15 at 2:27
• @ChrisC Case in point: I never even heard of Cramer's rule during my entire undergraduate and postgraduate maths degree. – DavidButlerUofA Feb 25 '15 at 22:46
• I've heard of it (as in used) Cramer's rule aplenty in high school. What I heard of it in CSE college and later... just don't use it; computationally it is very inefficient. – Fizz Feb 27 '15 at 10:41

Stephen Boyd and Lieven Vandenberghe (also authors of a well-known book on convex optimization) have recently written an introductory linear algebra book, Introduction to Applied Linear Algebra: Vectors, Matrices and Least Squares that may suit your needs. From the preface:

This book is meant to provide an introduction to vectors, matrices, and least squares methods, basic topics in applied linear algebra. Our goal is to give the beginning student, with little or no prior exposure to linear algebra, a good grounding in the basic ideas, as well as an appreciation for how they are used in many applications, including data fitting, machine learning and artificial intelligence, tomography, navigation, image processing, finance, and automatic control systems.

Also from the preface:

The book covers less mathematics than a typical text on applied linear algebra. We use only one theoretical concept from linear algebra, linear independence, and only one computational tool, the QR factorization; our approach to most applications relies on only one method, least squares (or some extension). In this sense we aim for intellectual economy: With just a few basic mathematical ideas, concepts, and methods, we cover many applications. The mathematics we do present, however, is complete, in that we carefully justify every mathematical statement. In contrast to most introductory linear algebra texts, however, we describe many applications, including some that are typically considered advanced topics, like document classification, control, state estimation, and portfolio optimization.

The book is indeed replete with applications, many of which come from machine learning (or data mining) and from statistics in the guise of data fitting. It's very much an introduction though, as you'll need another source for learning topics such as vector spaces, subspaces, LU decomposition, eigenvalues, eigenvectors, the singular value decomposition, and various other topics. It's very much at the other end of the spectrum from Axler's book, but they do have something in common - little or no emphasis on determinants. In fact, the word "determinant" does not even appear in Boyd & Vandenberghe.

On balance, I think it's worth a look as a relatively gentle introduction with a wealth of applications.

Gilbert Strang has a new book (published January 2, 2019!) that focuses on linear algebra's connection to machine learning.

G. Strang. Linear Algebra and Learning from Data SIAM, 2019.

I obviously haven't had time to review this book yet, but the TOC looks like it might suit your needs.

• Good find! One important difference between Strang's new book and Boyd & Vandenberghe's new book is that the Strang appears to be positioned after a first linear algebra course whereas B&V starts at a basic level. – J W Jan 10 at 6:22