I think the situation is clearer in $\mathbb{R}^2$ than in $\mathbb{R}^3$. Draw an open curve and ask students how much area it enclosed. Presumably they will recognize that the question has no real answer -- there is no way to tell what is "inside" and "outside" an open curve.
(Perhaps one way to dramatize this is using a Paint-type program... They typically have a "fill" tool that lets you fill an enclosed region with a solid color. If you draw a curve that is not closed and try to "fill" it the paint pours out and fills the entire drawing area.)
As to why they have this problem, I suspect it is because the axes themselves, together with horizontal and vertical gridlines, create visual "noise" that is hard to distinguish from the actual curve or surface. For example, when the graph of $y = \sqrt{1-x^2} + 2$ is drawn it looks like an open upper semicircle, but when the graph of $y = \sqrt{1-x^2}$ is drawn looks like a closed semicircle because the x-axis appears to be part of the picture. I imagine something analogous happens in 3-dimensions; I note that in the example you gave, students had difficulty recognizing that the plane $z=0$ was not already part of the figure, and I suspect this may have had something to do with how the surface was sketched or plotted (did the illustration include some kind of shading to show where the xy-plane was?).
Some thing that might help is to use more examples where the "opening" to the surface does not easily align with an axis or other visual element that could masquerade as part of the surface. Of course, this technique could make the computations more complicated, as we typically prefer examples that have precisely that kind of arrangement because they are easier to work with!
Edit: It occurs to me that another reason students have trouble with the notion that open surfaces do not enclose a definite volume is because that idea runs utterly counter to everyday, normal experience. Consider measuring cups, cereal bowls, 2-gallon paint pails. Our very notion of "volume" is, in the world of real life, based on filling open containers like graduated cylinders or beakers. Even in calculus classes we give them questions where they are supposed to find the dimensions of an open-top box or cylinder containing a maximal volume under certain constraints. So of course they think that a surface that is open encloses a volume.
What makes this notion so stubborn? Perhaps it stems in part from the fact that when we do consider an open surface it typically has a boundary curve that lies in a plane, and therefore there is a "natural" way to cap it. That, after all, is how real-life containers work.
I think the best way to deal with issues like this -- where there is a technical usage that differs significantly from the way a word or concept is used in everyday experience -- is to confront it head-on: before discussing integration over surfaces or volumes, it is essential to have a discussion about the difference between open and closed surfaces, and to be explicit about the precise way the phrase "enclosed volume" or "volume of the interior" is used in this context. And in every example you demonstrate you should begin by asking: Is this surface open or closed? If open, how can we close it off to create a well-defined interior? etc.