I don't think the answer is the same for lower-division texts, upper-division texts, and graduate texts. The situation isn't the same for all of these. Therefore in this answer, I'll focus only on lower-division texts, which are what I know about. (I teach at a community college.)
For this type of class, systems like mymathlab are the publisher's way of maintaining profits when students can buy used books or download digital copies for free. They sell mymathlab to instructors as a way of not having to hand-grade homework. If solutions were in the book, or legally available, students would put the answers in mymathlab and get 100%, which would break this system. Chegg also breaks the system, but it gives plausible deniability.
Before the internet, there was a concern that less motivated students would not have the intellectual maturity to solve the problems themselves if the full solution was in the book. This no longer makes sense, because such students have other options. For a computation-based math class such as freshman calculus, they can use software to do their problems. For word problems or mathematical problems in other subjects like physics and chemistry, they will pay for a chegg membership, which is a one-stop cheating solution for all their classes. It is true, however, that many instructors are in denial about these technological realities and don't want to change their practices, so they would perceive this to be a problem with openly publishing solutions.
In a comment, Joseph O'Rourke says he thinks this has to do with the cost of preparing solutions and with page costs.
I doubt that the cost of preparing solutions is a major factor. People have generally written up solutions for their own problems as they use their own book, before it gets published (or they may get a grad student to write solutions). There is the cost of putting these in a nicer-looking format, but I think for this type of lower-division text, that has to happen anyway, because people looking to adopt the book want to know whether it has an instructor's solutions manual.
Re production costs, the cost of physically producing paper textbooks stopped being a really significant fraction of their retail price about 20-30 years ago. There has been exponential growth in prices while production costs for paper books have stayed about the same. And in any case, within the last 5 years or so I've seen about 80-90% of STEM students shift to digital books. Often they rent them. They don't want to carry around a backpack with 30 pounds of textbooks inside, and their reading habits are digital. And in any case, the solutions don't have to be included in the printed object. They can just be in a PDF file. Publishers have actually given up at this point on making money on lower-division STEM books. Their revenue stream for these courses is now from software like mymathlab and Mastering Physics.
In summary, it's not clear to me that there is a positive net value for anyone in making solutions to a lower-division STEM textbook openly available. Students have access to solutions anyway, so why bother? Having access to solutions is also probably a net negative educationally, since most students use them in an immature way. Making them openly available would increase the publisher's development costs slightly, and they can't sell the product because students would just redistribute it on the internet. It would save students the $15/month and moral degradation of using chegg, but the publishers don't care about these things.
More broadly, I think this is a situation where technological change has brought about unforseen consequences, and created a system that is dysfunctional and yet still profitable for the publishers. The system is broken and needs to be fixed, but providing solutions openly won't fix what's wrong and will also destroy the publishers' revenue stream (because it would eliminate any remaining justification for systems like mymathlab).