I have been teaching introductory group theory to undergraduates. We reached cosets several weeks ago, but the combination of the textbook, my explanations and various practice questions has left the students still very uncertain about cosets. I've pretty much run out of ideas for how to try and help them.
What is it about cosets that students find hard to understand? What ways of thinking about cosets make sense to them?
I strongly suspect the difficulty is not with cosets specifically, but with working with equivalence relations generally, especially when combined with objects that they have only recently become acquainted with. So apologies in advance that this answer deals more generally with equivalence relations and my opinion that they often need more coverage than they get before cosets, rather than with cosets themselves.
Ask a student the following question: Is $\frac{534389}{832323} = \frac{3740723}{5826262}$? Give them a calculator that displays 6 decimal places (or tweak the example to make the fractions agree to a larger number of places), and how many would look at their calculators and conclude they are equal? How many would realize that they could check their answer by cross-multiplying, but then give up because their calculator could not store enough digits for the multiplications? How many would realize that you have to cross-multiply and that it is immediate that the two products do not agree in their ones digit, so they cannot be the same?
I think this example illustrates the varying levels of understanding a student may have for identifying equivalence classes. In the first case, the student knows how to distinguish equivalence classes when they are "obviously" different or when a certain test applies, but they do not have a general algorithm to distinguish equivalence classes. In the second case, they have a sound theoretical understanding of what differentiates equivalence classes, but lack the tools to test identity of classes as quickly as possible. In the third case, the student actually understands the theory and has the ability to apply other theoretical concepts for more effective calculations and deeper understanding.
In my experience, for most students, it is quite hard to get to that final stage and often takes a long time to get there, if they make it there at all. If they cannot solve this simple problem above at least at the second level (and ideally at the third level), then asking them to be able to manipulate cosets, which involve algebraic objects that are much less familiar to them than fractions, often becomes an exercise in futility.
One of the most famous mathematics education theoretical constructs, namely APOS, offers an answer for this question, in fact, the exact same question. A quick search leads you to some papers and even one or two books. Here is a very short summary.
APOS stands for Action, Process, Object, Schema. In simple terms, the difficulty relies on coming from forming (as an action/process) a coset to conceptualizing it as an object (O in the APOS). You can observe, more or less the same difficulty in early number learning where children need to move from counting say 5 items to conceptualize 5 as an object.
This is not an answer to your question of what makes cosets difficult, but whatever it is, visualization may help. For example, Nathan Carter's Visualizing Group Theory presentation (Download .ppt) illustrates cosets in $A_4$ over several slides. Here is one slide:
I think that one unfortunate aspect about the presentation of cosets in many elementary books is that they don't show any "geometric" examples. Thus the students try to treat the cosets entirely formally, which is much more challenging for them, particularly because most undergraduates have not mastered formal reasoning about sets before they reach group theory. Without intuition or a visual picture to guide them, they find it hard to understand what anything means.
To combat this, I refer to cosets early on as "shifted subgroups", and work through examples such as the cosets of lines through the origin viewed as subgroups of the group $\mathbb{R}^2$ under addition. These cosets are just lines parallel to the original subgroup, easy to see. This gives the students something to visualize geometrically, which is harder to see with cosets inside finite groups. They can also see the fact that "$aH = bH$ if and only if $a-b \in H$" in this special case first, where I can draw a diagram with vectors $a$, $b$, and $a-b$.
The problem I generally see with cosets is that people generally are shown the group axioms, then they're shown a couple simple examples of groups (cyclic groups, dihedral groups, symmetric/alternating groups, and maybe the quasidihedral group or something), and then they're asked to think about cosets and quotients. In particular, they don't have any idea what a group "is" or have any intuition for them.
It's not surprising to me that students don't understand, because it seems like we've made a completely arbitrary definition and asked them to compute with it. In particular, groups are often presented in such a way that students don't understand that group actions are fundamental and a group is just something with exactly the right amount of structure to act on something.
I usually find it useful to give concrete examples of the philosophy that cosets are what's left when one wants to neglect the elements of the considered subgroup.
The most basic example is the following: turn the light of the room on and off repeatedly and fast a number of time, and then ask the students: "how many time did I hit the button?" You want to have done it in a way that they won't be able to answer. Then ask: "did I hit the button an even number of time or an odd number of time?", and then the answer only depends on the state of the light before and after the operation.
The point is that the light does not care exactly how many time you hit the button, it only cares if this number is even or odd, or in other words the light does exactly as if 2 where equal to 0. The state of the light is described by a quotient, whose elements are cosets (all the numbers of it inducing a given state of the light).
Lights you tap to cycle through dim light/bright light/off give a similar example.
One possibility for orbits with a non-abelian group is to find several standard and a non-standard dices (most dices have the numbers arranged in the same patterns, but not all of them). Ask your student if two dices are the same : they will try to rotate them so that they match. The point is that not any position will do, and if they succeded it means they applied an element of the group of displacements of the cube (aka $S_4$, but that is another story of course) to a labeling of the faces by $1,\dots, 6$, to obtain another labeling. If they (provably) can't, then it means that the two dices belong to different orbits. The point again is that how to try answer the question should be obvious to the students. Then one can frame their methods in the mathematical framework and hope that they make the connection between what they did and the object of the course, and then the later may start making sense.
The problem with this example is that the set of labelings has no natural group structure, so it does not fit your question well. I guess there are other possible examples which may do.
Jessica,
Norman Wildberger has a few interesting approaches to thinking about quotient groups, leading up to him talking about homology in his intro Algebraic Topology course. I cannot remember which video he begins quotients, but the first of the (I think 4) videos is here:
https://www.youtube.com/watch?v=Kcyitn0LHSM&list=PL6763F57A61FE6FE8&index=14
I propose that it's not cosets themselves that are hard to understand.
Instead, the problem is that they often arise not because they are a thing of interest but instead as a technical device — a trick for systematically converting problems about congruence relations into problems about equations.
Although the memories are hazy, I recall that I personally didn't really gain any facility in working with cosets until I did the reverse — ignore the coset structure and instead treat them as funny notation for working with a congruence relation.
And honestly, I think that better reflects actual real-world practice for doing computations in many cases, such as modular arithmetic. Which version of Fermat's little theorem would you expect to see and use?
(in fact, I've since become enamored with the idea of working with setoids instead of sets)
I opine that using cosets for the purpose of this technical trick would be easier to swallow if students first had experience actually working with a congruence relation, and then emphasis was placed on the mechanics of how working with congruence classes can be used as a substitute for working with the congruence relation, and that the congruence classes are cosets of the class of zero.
Of course, cosets as being objects of interest their own object of interest needs to be introduced somewhere along the line too.
In fact, seeing this question again, I think that there is one aspect which may explain a big chunk of why student struggle with cosets and seems not to have been mentioned: coset might be hard to understand because with them, we try to change the very notion of "mathematical object" in the students' mind.
Similar problems arise in many occasions when a similar cause can be blamed:
students struggle with functions, which turn into object the operations that where applied to their only objects thus far, numbers;
they struggle with propositional and basic logic, which turns assertions into objects that can be manipulated just like numbers, although with their own set of operations;
they struggle with basic set theory, which turns sets into objects to be manipulated while they where mere containers, or symbols only meant to give the type of a number;
they struggle with cosets, which are subsets, meant to precise the type of an element of a set, and (after a previous struggle) could be subject to some operations such as intersection and union, but are now meant to be subject to algebraic operations, as if they where numbers.
I wonder if some geometric examples might be useful. Then the cosets would be something that students could draw.
The most basic would be to take the Lie group $U(1)$, thought of as the unit circle in the complex plane. For any $n$, there is an isomorphic copy of $\mathbb{Z}/n\mathbb{Z}$ inside, given by the multiplicative group generated by $e^{2\pi i / n}$. What are the cosets of these $\mathbb{Z}/n\mathbb{Z}$s?: Each coset is the vertices of a regular $n$-gon. You can even talk about what does the space of cosets looks like. (Topologically an $S^1$ and isomorphic to $U(1)$, but we should probably think it as having $1/n$ times the linear measure of the $U(1)$ we started with.)
Another geometric example would be to look at a copy of $U(1)$ inside of $SU(2)$. (It's geometric since $SU(2)$ is diffeomorphic to the 3-sphere.) The copy of $U(1)$ in $SU(2)$ I am thinking about is the one which contains matrices of the form $$ \left[\begin{matrix} e^{i\theta} & 0 \\ 0 & e^{-i\theta} \end{matrix}\right]$$ This is a quotient (the coset space is a 2-sphere) which shows up naturally in physics: https://en.wikipedia.org/wiki/Bloch_sphere
Since each of the cosets is an unknotted curve in $S^3$ and cosets link each other once, it also shows up naturally in knot theory: https://en.wikipedia.org/wiki/Hopf_fibration
