Some resources: Have a look at the textbooks of Wikibooks. One shall also not forget the math articles of Wikipedia, which are used by everyone.
My experience: I write an open textbook on Wikibooks (it's in German), so I can add some pros and cons from the author's point of view:
Because the textbooks is freely available (an result of being open), more people are using and reading it. The necessity to pay for a content is a big hurdle most people will not take. So having an open project is a huge advantage over a conventional textbook you need to buy.
For example I had other 800 thousand page views last year with my project, which is quite a success (you have to note, that the textbook isn't completed yet). With a conventional distributed textbook I could never reach such an audience.
Note, that the more people you reach, the more people you help. So if your reason for writing a textbook is helping students you shall write an open textbook, especially if you also want to help those, who do not have much money.
Other people are more likely to contribute to your project when it is open. In the last 5 years there were over 300 people who participated in my project. Most of them corrected typos or little mistakes, but there were also some who wrote new chapters. To archive this, you need a platform where the reader can easily change the text (such as a wiki, for example Wikibooks).
Many authors have the fear, that someone else may destroy their work, when it is open. That's not the case. I also can say: You will have the opposite problem! There are always readers correcting little mistakes, but for finding co-authors you will need to make much advertisement (e.g. write on a mailing list, ask friends...).
You also can use text materials, diagrams and pictures from other open projects such as Wikipedia, Wikimedia Commons or posts from this forum (note, that each stackexchange forum runs under CC-BY-SA), if you project is open. This will save you a lot of time. (Note: One need to have a good knowledge of the Creative Commons licenses of course!).
Whether you publish your work under a free license is always a personal question. It depends of the goals one follows. An open license may not be the best choice if one wants to make money. But then he also shall not write a textbook because you cannot make a lot of money with it. If one wants to reach a huge audience, wants to get good reputation or wants to help students he really should use open licenses because this will help him reaching his goals.
Final note: One should carefully chose the open license for his textbook. Creative Commons licenses are most suitable for textbooks. But not every CC license is open in the sense of freedomdefined (it's also the definition the Wikipedia is following). CC licenses with the modules "non derivation" or "non commercial" are not free! See The case for Free use: reasons not to use a Creative Commons -NC license for reasons.
