It's a broad question and as such is tricky to answer specifically. Thus, broadly speaking, I would do the following, speaking from experience not from published research on the matter
- I wouldn't advertise the problems as "here are problems for Group A, here are for Group B, etc.". I've never liked to broadcast to students how they are being classified.
- Instead, I would ask "hard" problems and separately give a "hints" document. The purpose of the hints document is to guide students through the steps. For example, let's say that deep into a Calculus II class, after seeing the bulk of the integration techniques, I would ask $$\int \sqrt{x^{2} + 1}\ dx$$ and then on a hints page, I would give a series of hints, starting with a trig substitution, perhaps. The general instruction / philosophy would be, "If you get stuck, take a look at the hint and try to continue from there. If you are still stuck, look at the next hint." Thus, by going through the hints, they (ideally) automatically fall into the right bucket of ability.
- I generally don't collect homework for grading, so I tend to assign pretty much the book for homework with general instructions about what I would consider to be easy, medium, hard problems [this lets them understand their own ability without explicitly calling to attention the ability of other students --- I do my best to avoid non-self comparisons]. I tell students to answer as many problems as they feel they can with the objective that they be able to get to the hard problems.
- I don't find anything wrong with giving reach problems for the advanced students. I just don't advertise that "these problems are for advanced students" [I avoid classifying students to the students.].
If the concern is exams and grading, then I would look into if there is a possibility to have a split course. I have seen this done where if students are on, say, a BS in Math track then their course objectives are necessarily different from those on, say, a BA in Math Ed track. If you're able to do this, then I'd say it's pretty straightforward to give assignments based on which version of the course the student is enrolled. [For example, while universities may have Honors courses vs Standard courses, some universities combine these courses into one [probably for economical reasons] and those in the "Honors" version of the course have extra lab hour requirements or something of the sort.] In other words, students can take the same course for 3 credits or 4 credits depending on their major. The transferability of the course grade only goes down [4 credit hour version can count towards the 3 credit hour, but not the other way around].
My two cents. Hope this helps in part at least!
