The problem is that the question was written using the Blackboard graphical formula editor, which cannot recognize the square brackets as variable definitions. For that you need to put the square-bracket variables in normal Blackboard text, not the formula editor. (Note the UI directions on the top of the screen: "Add question text that includes variables in square brackets [x] corresponding to variables in the formula that will be used to calculate the answer to this question.").
Having just tested it in Blackboard: The answer formula is fine. If I write the question using a Blackboard formula, then I get the same error as you. But if I instead write the question using basic text (like "Compute ||<[x], [y], [z]> x <[u], [v], [w]>||"), then that is accepted without the error. Obviously that text version isn't perfectly formatted, so you might need to play around with fonts or special characters to get it looking right.
Alternatively, you could write the question in natural language, as I think that feature is intended, i.e., "Find the norm when multiplying vectors <[x], [y], [z]> and <[u], [v], [w]>".