While I haven't used an open-source text for a semester-long university class, I have used an open-source calculus text for a summer program with high school students, and I'm currently modifying a number-theory text for a different summer program.
Summary, with extended discussion below:
Upsides:
- I can modify the text to suit my needs.
- I can modify the text in response to student reactions.
- My students don't need to pay excessive amounts of money.
Downsides:
- It is less tenable, at least from my position, to work with open-source texts with large courses that have multiple professors, for many reasons.
- The general open-source text available is supbar, in my opinion, and needs to be modified.
- Actually modifying/writing a textbook is very time-consuming and hard.
- (If you care) - it is not currently feasible to use one of the online-homework systems without using the accompanying book.
The great thing about both texts is that I can modify the source to fit the course, which I much prefer to modifying my course to fit the text/jumping around the text. In fact, this is extremely useful. It also allows me to modify and change wording, examples, and problems based on direct feedback from my students.
I must admit that I am also very happy to not force students to buy these texts, given that the information itself is readily available online.
Given the option, I would choose to use an open-source text for essentially any lower-level class (precalc, calc, linear algebra, statistics, etc.). I don't know about upper-level classes, for which there often exist great books that are extremely well-written and which don't change from year to year.
On the other hand, I don't usually have that option, at least not from my current position. And many professors I know don't have that option either. The university expects consistency, and lower-level classes are those often taught by multiple people at a time. For all of us to use a modified version of an open-source text requires all of us to agree on modifications/the text. At the very least, university policy and bureaucracy make this much harder. (I suppose it would just take the person in charge, if there is a person in charge, to make a solid decision and to commit - but this has not been my experience).
Further, in my experience, I find that most open-source textbooks need some revision for one reason or another. This limits them to the ones whose source is actually available, rather than those that are just free and which get lumped in. And the problem here is that authors get to make up their own notations/TeX conventions, which can make it very unpleasant to actually modify their text.
For that matter, writing a text at all is time-consuming and hard. It's hard to write clearly, it's hard to make pretty pictures (really hard). It really is much easier to just use some generic calculus text (the big ones are all the same, and all such classes are isomorphic) than to find, modify, and use an open-source one. Or rather, it's hard to first go out, find, modify, and use an open-source one. Having done it once, it's now very easy.
Perhaps soon a good variety of good open-source texts will become available, which will make the transition cost lower and make it more likely that a university might accept it.
I talked to the teaching head of my department (so we actually do have a guy in charge), and he's of the opinion that the future isn't in open-source, but in online homework (like webassign, or other forms). We don't actually use that at my university, but it was used at my undergrad, and I know that many other schools use online homework.