I know the question in the title is very broad so I will try to explain it as succinctly as I can. Half my time is spent on as a researcher on didactics, while the other half is devoted to teaching. The course I discuss below is an introductory course to mathematics for those who only need some mathematics in their further studies, some brands of chemistry students, biology, CS students and so forth (whether they only need a bit is up for discussion). The curriculumn is a litle bit of set theory, algebra, functions, graphs, limits, derivation, integration and the most simple differential equations.
We have done some surveys amongst the students where they have been given a series of simple problems within algebra. Factorize this fraction, simplify this expression, alongside some more involved problems such as evaluating 1 # (2 # 3) when they are given a # b = a/b + b/a. This test is conducted in a closed environment, individually with no aids other than a pen and pencil. According to what we would expect university students to score, the results are abysmal.
However, I also give them big hand ins during the semester, about 3 of them give or take. Each project takes about a month for them to solve these problems in a group, and they are fairly involved compared to the survey we did. These projects or exercises uses "real life" problems to motivate the students.
We also did a post-test (not representative due to covid)
According to what we would expect university students to score, the results are abysmal in the pre and post-test.
On the other hand just looking at the what the students handed in, the results where fantastic.
Possible solutions and explanations
It was clear that several groups did use online CAS solutions to perform the algebra part of the projects. Note that the algebra part in these projects was just a small part of the problem. This was clear by the replies from the teaching assistants and a very small percentage of students even screenshoted WolframAlpha step by step in their hand ins!
The projects were new as of this year, so they could not have obtained solutions from older students. In addition the numbers on every groups project varied, so they could not copy directly another groups solution.
The projects counted indirectly count towards their grade as the plan was to have a individual pass / no pass grading system with a group oral exam at the end of the semester. The oral exam would be based upon the projects, and start by asking how they went about solving some of the problems. With some follow up questions to see if they understood what they did. Again not possible this semester.
I realize that the pre and post-test are conducted in a very unnatural environment to the students. Yet, based on the results from these tests I would not have expected any of the students to be able to solve the complex projects.
If I asked a student at point blank today if he would be able to find the derivative of (1-x)/(1+x) I doubt he or she could do it. However, seemingly in a group, with every resource available and enough time, they are. I think I am just a tad confused what it means to understand algebra.
Do my students know algebra / how do I know whether I have taught them anything?
My gut feeling is that algebra is more than symbolic manipulation, but I do not have sources to back me up on this.