I find that the best way for my mathematical modeling class to understand the modeling process is by having the experience of modeling some real world situation of their group's choosing. I have a number of strategies so that the project topics are interesting and of the right scope to be doable in the amount of time allotted. One of two main deliverables that I require is a 16–20 page paper about their model, focusing heavily on the evaluation of the model. (See here for Project Description and Project Expectations.)
One day of the semester, I ask the students to submit a final draft to peer review. The groups bring in three copies of their paper and the students spend the 75-minute class period reading and commenting on their peers' papers. I give the students directed questions to address about the papers. (See here for Peer Review Sheet.) The goal of this peer review process is to give the students a chance to improve upon their drafts before submitting them to me for grading a few weeks later.
I feel that I have set up the Peer Review process for success but every year there are issues with the process. For example, some groups get peer comments that their paper needs only minimal revisions but when I read the paper, it is clear that the main focus of the paper is not the model or the paper does a bad job with model evaluation. This leads to frustration and disappointment with the grades that I assign, especially with all the effort that I know that the students put into the project. (I do allow for a second revision to improve their grade afterward.) I feel that the peer review process is not doing its intended job to improve the quality of the paper and giving a false sense of security.
Do you have any suggestions for ensuring success in the Peer Review process?