The biggest advantage of open-book exams is that they are closer to real world problems. People can look things up in the real world. So why should they not do that during exams? However, the big disadvantage is that students, because they are not used to it, have the common misconception that an open-book (or even open-internet) exam means that "I can just look everything up on the fly", which is bad, but is just that: A misconception.
Make sure to explicitly communicate how you expect them to prepare for this exam. Consider even to ask and guide them on their exam preparations. Give them a recipe for exam success (while, of course, avoiding "teaching to the test"). Also, make sure not to make the open-book exam terribly hard. If you transform a closed-book exam into an open-book exam, but the degree of difficulty stays the same, you have a slight chance of making your students' lives just a little bit easier.
The "exam taker" students might complain, so maybe compensate with "bonus" problems or other goodies (possibly, outside of exams) that compensate people who invest more time, and are actually interested in learning this stuff. What about rewarding students helping other students (outside of exams)?
Having said that, let us re-consider what an exam is and does: Exams exams are short-time, high-stakes, high-pressure turning points in a young person's career. They generate large amounts of stress and disgust in the average young (and old) person's mind, and there is tons of research showing that they barely help the student in understanding things, solving any problems or achieving any other form of goal. Exams do two things:
- Exams generate one (and only one) kind of extrinsic motivation in students. We call that kind of motivation "pressure" or "stress". In the world of "carrots and sticks", its the stick.
- They generate a grade. Institutions (and thus teachers) are required to generate a student's evaluation, usually in the form of a low-dimensional vector of "grades". As a result, generating "good grades" ends up being the only goal for institutions, and often distracts them from what should be their real goal: To help the students.
Do you think, that open- or closed-book exam can really help students? Regarding your exams, is it more important to "get it right the first time" (because else their grades, and thus their future career and chances to get scholarships will deteriorate), or do you focus on students learning from the exams and the feedback you give them on the exams?
Do you give students a chance to generate good grades, if they have the passion and put in the effort, even if they are not good "exam takers"? Do you think that the quality of being a "good exam taker" correlates with a person's ability to succeed in or contribute to society? What about those who are not good at exams? Do we know anything about those students?
So here are some suggestions: Have you tried non-one-shot (i.e. repeatable) exams? If that's too much effort (or you don't have the infrastructure to make it effortless), did you consider take-home exams? What about project work? Please don't rule it out just because students are bad project managers. No one has taught them yet. Of course, good project work also needs the right kind of structure. The same issues apply to all approaches that diverge from the traditional "lecture, homework, exam" cycle of death - It's hard. Hopefully, we will be able to come up with better approaches that we can also share more easily, soon enough.
PS: Maybe you want to take a look at Robert Talbert's awesome blog on new ideas in teaching Mathematics, flipping the classroom and a lot more?