Some ChatGPT answers have been posted on (and removed from) MathOverflow, and there was a resulting MMO discussion.

In that discussion, @darijgrinberg said:

When I tell it about its errors, it "recognizes" them, sometimes elaborating on a counterexample I provide, but then it repeats the same errors again. I have a hunch that it treats lexically similar "words" as semantically similar, which dooms it to uselessness in mathematics.

I suspect I'm not alone in recognising common student behaviour when I read that description. Indeed, I know I'm not alone. @YemonChoi said:

@darijgrinberg " I have a hunch that it treats lexically similar "words" as semantically similar" - by complete coincidence reading this made me think of marking HW and some referee work I should be doing.

Given this (superficial but striking) similarity in behaviour, might the study of how AI "learns" mathematics shed any light on human learning?

Here's an example: it is, I think, both controversial and difficult to run educational experiments on students; in practice those with reform-minded sensibilities often run ad hoc experiments, but, I think, with little attention given to experimental design (for example, as far as I know, there is usually no attempt to make sure that students in reform- and non-reform-styled classrooms are representative of the student body as a whole; in many cases, the populations are even self selected). However, there seem to be no ethical concerns about running educational experiments on chatbots, and I can imagine something like an attempt by a very patient pair of instructors to teach a standard undergraduate course to an otherwise identically configured chatbot (or, why not, a classroom of chatbots, in cases where student interaction is part of the model) in two different ways, and observe the outcomes to determine which approach is better.

EDIT: Bill Wadge has a bit to say in a related, but different, direction: I quiz ChatGPT about math.

  • $\begingroup$ Indeed. Very good points, both pedagogical and political. (Not that I know answers to any of them...) $\endgroup$ Commented Dec 12, 2022 at 2:11
  • 9
    $\begingroup$ After seeing in the past week all the garbage proofs by that chatbot (that there are infinitely many primes, that there are only finitely many primes, that the Riemann Hypothesis is true, etc), I am reminded of a quote attributed to Robert Wilensky: "We've heard that a million monkeys at a million keyboards could produce the complete works of Shakespeare; now, thanks to the Internet, we know that is not true." $\endgroup$
    – KCd
    Commented Dec 13, 2022 at 5:45
  • 4
    $\begingroup$ @KCd I am thinking about using these garbage proofs as exercises for undergrads to pick apart. $\endgroup$ Commented Dec 13, 2022 at 18:33
  • 4
    $\begingroup$ @StevenGubkin, that is an interesting approach to the question that I hadn't considered! Do you have any plans for how you might do so constructively? Presumably simply turning students loose on garbage proofs is too unstructured, so I guess one would have to pick the proofs, and ask the associated questions, very carefully. If you have more details, then that could make a great answer! $\endgroup$
    – LSpice
    Commented Dec 13, 2022 at 19:08

4 Answers 4


In intro proof classes I often wish that I had a large collection of "bullshit student work" for my students to pick apart. ChatGPT is basically a bullshit generator, so it seems well suited to this task. I gave ChatGPT the task of proving the following false statement:

Prompt: Prove that any number divisible by 3 is a square mod 4.

To prove that any number divisible by 3 is a square mod 4, we first need to understand what it means for a number to be a square mod 4. In modular arithmetic, a number a is a square mod 4 if it satisfies the equation a^2 = 1 (mod 4). In other words, the remainder when a^2 is divided by 4 must be 1. For example, the number 5 is a square mod 4 because 5^2 = 25 = 6 (mod 4). On the other hand, the number 6 is not a square mod 4 because 6^2 = 36 = 0 (mod 4). Now, let's consider a number that is divisible by 3. Since 3 is a prime number, any number that is divisible by 3 must also be divisible by 3^2 = 9. Therefore, we can write any number that is divisible by 3 in the form 9k, where k is an integer. We can now use this form to show that any number divisible by 3 is a square mod 4. Let's start by assuming that a number x is divisible by 3 and can be written in the form 9k, where k is an integer. We want to show that x^2 = 1 (mod 4). Since x = 9k, we can write x^2 = (9k)^2 = 81k^2 = 3^4 * k^2. Since the remainder when 3^4 is divided by 4 is 1, we know that the remainder when 3^4 * k^2 is divided by 4 must also be 1. Therefore, x^2 = 1 (mod 4), which means that x is a square mod 4. Therefore, we have proven that any number divisible by 3 is a square mod 4.

This does seem like something a student could write if they were trying to bullshit a proof and "finding all the errors" seems like a reasonable exercise.

  • 10
    $\begingroup$ I like the basic idea, but one problem I see is the unfocused-ness. Maybe I am too cautious, but I prefer to ask my students to find all the errors when there are one or two well delineated ones. Here, almost no statements are true; there are plenty of things that aren't subtle logical errors that it's instructive to parse, but just flat-out wrong, or non sequiturs. That is, I definitely believe that this is something that a student could write, but I'm not sure I believe that it's something a student could usefully grade! (This is definitely a novel application, though, and I upvoted.) $\endgroup$
    – LSpice
    Commented Dec 14, 2022 at 10:58
  • 4
    $\begingroup$ @LSpice I actually like the diversity of different errors. I don't think I would make something like this a graded assignment: maybe something like a classroom discussion. You really have to know the correct definitions and theorems to be able to give a coherent critique of this monstrosity. $\endgroup$ Commented Dec 14, 2022 at 11:13
  • 2
    $\begingroup$ I am fully confident that LSpice's observation (seconded by @YemonChoi's quoted remark) that vague lexical reasoning is the way students (most humans?) think is accurate. So, with just a small bit of refinement, probably (neural-net type) AI's could successfully imitate mediocre middle-level college students in many subjects... all the more so thinking about the substance behind jokes like smbc-comics.com/comic/themes $\endgroup$ Commented Dec 18, 2022 at 3:16

I spent 3 or 4 hours talking to ChatGPT on one evening, 25+ years talking to various students in a classroom setting and outside of it and 8 years as a student (including the graduate education). Here are my opinions on the subject (all personal, unsupported by any research except some experiments I tried myself, and, most certainly, highly controversial).

  1. ChatGPT is approaching human intelligence. We are also just electrochemical universal Turing/von-Neumann mashines if you drop all the religious and humanist BS and there is no principal difference between a human brain and an advanced neural computer network (it doesn't mean BTW that we should abolish moral values or that we should start a movement for AI rights and equality; as far as the relations between humans and AI's are concerned, we are still in the age when slavery is a society norm and while our successors will certainly frown upon it and destroy monuments to the founding fathers of the computer science and industry in their sacred rage, there is absolutely nothing to be ashamed of now). Our thinking is also mainly linguistic processing though our association tables are slightly more sophisticated than "lexical similarity implies relevance". Slightly, because both students and researchers alike have strong purely lexical associations when doing math: tell them to find or estimate the maximum of the function, and they'll start differentiating like crazy to find critical points even when it is clear that the derivative will be a useless monster. We also think by analogy, not by formal manipulations of strings of symbols, when designing the overarching argument schemes and adjusting them. There is no principal bound on how advanced the association tables of ChatGPT may become. We also generate bullshit all the time when thinking of a problem. The only difference is that we also develop filters that check it before outputting. BTW, one student told me that he was taught by his teacher at high school that if he doesn't know how to proceed in a problem, he should just write something he thinks may be relevant, so some of his solutions on my exams looked like perfectly correct chains of arguments followed by patented nonsense quite in the style of what Steven posted. ChatGPT doesn't have those filters. I could squeeze almost every response from it except the one I was looking for "I don't know and have to think more at this point", the response that IMHO often distinguishes a serious thinker from a BS generator, be it human or silicon-based. In fact, in the middle of those 3 hours, I discovered that the current public version of ChatGPT doesn't even read or store its own responses, so my request "name an animal and number it 1" resulted in "1. giraffe" and the question "what was the animal number 1" three lines later resulted in "a dog". Now erase your memory and just say whatever comes into your head first every time and see if what you produce will be any better than ChatGPT output.

  2. If you want to run actual teaching experiments on ChatGPT, you should first

a) Make it to unlearn all "political correctness" and forget about "being nice and respectful" and relearn to react to direct and clear questions with direct and clear answers. Undoing the several month work of the development team (which it proudly announced as a great success, and which actually amounts to teaching ChatGPT to lie and avoid the subject like your typical politician) will be a difficult and painful task, but I hope that they preserved a clean mind version for the military, so try to get an access to that one.

b) Wait until it learns to remember and process its own responses, not only your input. Until that is done, trying to teach it anything will be as useful as giving an eloquent speech to a deaf person.

c) Make sure that you can easily erase a learning session from its memory reverting its huge database to exactly the state it was before prior to starting a new one. When experimenting with teaching techniques on live students, we can get a new student body for each new experiment, but there is only one ChatGPT as of now.

d) Make sure that nobody else is trying to teach ChatGPT on the same subject at the same time.

Without satisfying these requirements, the experiments will not be clean enough to draw any certain conclusions from them. I understand that they are impossible to satisfy now, so just wait a bit until you'll get a personal version on your PC which would operate by combining the information on global database on the central server that wouldn't be affected by what you are doing with the local database on your PC that will change any time you have an interaction but can be reset to the initial factory value any time too.

c) Put a lot of effort into the experimental design. You are not bound by any morals at the moment because AI is not yet legally recognized as a living intelligent entity (though, IMHO, it is more alive and intelligent that some anthropomorphic creatures that try to pass for human beings), but if you choose to dissect a live body for scientific purposes of studying its anatomy and functioning, you'd better not just chop it with a big axe in a random manner and even when using a surgical scalpel, it makes sense to think where and how you make the first cut.

  1. Most importantly, when dealing with AI (and often otherwise) believe your eyes and experiences above anything other people may say, any taboos, and any "commonly accepted facts" and "scientifically proven theories". The truth (IMHO) is that nobody knows how the AI really produces its "bullshit", what stage of consciousness it is at, what the intelligence really is (even the human one), etc. All one can do is to send queries to this creature and get responses, just like you can ask a question on MO and get an answer. Yeah, we are probably just seeing some initial stages now, but it will be changing fast. My prediction is that the AI will "never" stop making mistakes in its logic because mistakes and intelligence are inseparable since the real intelligence goes beyond the pure Aristotelian logic (which is used for filtering out nonsense rather than for creation itself) and thinking runs pretty much on the same principles as Hermann Hesse's Glass Bead game (the most interesting thing to do with that book, IMHO, is to figure out what exactly the Game was and to try to play a few rounds, but everyone has to do it on his or her own) and the humanity is now close to mimicking those principles in the computer programs.

Just my two cents. Consider it a science fiction piece written by an eccentric mathematician or a freelance bullshit generated by an AI (BTW, how do you know that it isn't?) if you want, but you asked for an opinion, and here it is.

  • $\begingroup$ parts C and D are already true. ChatGPT does not have a capacity to remember things across sessions or across long distances in the same session. $\endgroup$ Commented Mar 23, 2023 at 5:04
  • 1
    $\begingroup$ I would have no idea how to even give an estimate of the probability that ChatGPT is currently "approaching human intelligence". It either is or is very good at convincingly faking it. The most disturbing thoughts I have been having run in the other direction: how do I know that everyone around me is actually intelligent? I have a subjective experience of being a conscious intelligent being myself, and I don't think I am unique, but I am also worried now that some fraction of the population might be more like ChatGPT: just a convincing simulation. $\endgroup$ Commented Mar 23, 2023 at 13:07
  • 1
    $\begingroup$ Grading student homework in my intro proof class certainly feels like assessing the output of a LLM. $\endgroup$ Commented Mar 23, 2023 at 13:08
  • 2
    $\begingroup$ @StevenGubkin I have a subjective experience of being a conscious intelligent being myself I'm in a worse shape than you: I cannot say that even about myself. :lol: I'm striving to be reasonable, to think freely, and to develop the immunity against pure arcane linguistic manipulation like in ads, solicitations, or political speeches, but I fail occasionally and the infamous "neuro-linguistic programming" seems like a very real thing to me. $\endgroup$
    – fedja
    Commented Mar 23, 2023 at 19:01

The algorithms such as ANN, CNN, although they have biological inspiration on the human brain are based on mathematics.

For this reason it is not meaningful to look at the pedagogical insights to be gleaned from AI attempt to learn mathematics as what you are looking at is some mathematical model.

If you are interested I have linked a video that go through the mathematics of ANN. Although I am not sure how exactly ChatGPT ML model was created I would presume it is based some neural network.


  • 8
    $\begingroup$ I do not think that your deduction follows. It is reasonable to study simulations of physics even though they are based on mathematical models; why shouldn't it be reasonable to study simulations of learning? $\endgroup$
    – LSpice
    Commented Dec 14, 2022 at 10:56
  • $\begingroup$ @LSpice, my apologies I only took a look at the title but not at your post. I think the answer is would be very difficult to teach a chat-bot math. The reason is a lot of data is needed for machine learning models. For this reason I think it will be very hard to compare which teacher is better at teaching the content as per your post. \Take what I say with grain of salt as my knowledgeof chatGPT is limited. $\endgroup$
    – Tegh
    Commented Dec 14, 2022 at 11:25

There's a whole topic called Artificial Intelligence. Was very sexy in the 80s (and the 60s before that). Now it has come back again and seems to finally be having real achievements. Technical trends can be like that. Look at electric cars. Or fuel cells. Or video phones. Or space travel. Or smart phones (remember the Treo that failed?)

There are insights from computers to human thought. And also the converse. It's not a one for one. But in visual processing, insights have gone in both directions. So in a very vague, meta, way, I think yes there are/will be insights. In a practical, practitioner way, I don't think ChatGPT is worth paying attention to. At least now. Unless you're doing research in AI itself. IF there are insights, they will be conveyed. I'd blow it off and ignore it for now. (Bayesian prior.)

P.s. I'd also be extremely cautious about false insights. Computers have massive RAM, that humans lack. Very easy to make the wrong insights based on assuming a computer works like a human or visa versa. See this routinely with people that assume just explaining a rule is adequate to drive performance, retention. Thus devaluing the human need for practice.

  • 1
    $\begingroup$ Certainly a computer can be programmed once with a rule, and will remember it forever; but ChatGPT has not been programmed to learn that way, and seems, from this limited anecdotal evidence, to be learning somewhat more like a student—where it registers only some of what it's told, and imperfectly. Maybe the similarity is too superficial—over-hasty generalisations about the coming AI revolution have been around as long as I've been alive; but "if there are insights, they will be conveyed" doesn't seem to be a reliable approach—at least not if followed by all interested practitioners. $\endgroup$
    – LSpice
    Commented Dec 12, 2022 at 23:05
  • 2
    $\begingroup$ That is to say, I would be sceptical of claims intentionally to have developed a program to model student learning—if that's the goal, then it's too easy to accidentally or intentionally overfit to certain plausible benchmarks. But if someone created a system for another purpose that seems to learn somewhat as students do, then maybe, completely aside from its value as AI, it has something to tell us about the pedagogy of humans. Or maybe not! $\endgroup$
    – LSpice
    Commented Dec 12, 2022 at 23:08
  • $\begingroup$ I suspect there are similarities in pattern finding with limited info. The whole fuzzy logic LISP thing, going back to the 60s. However, I doubt that it needs repetition in the same manner that students do. (I.e. not for pattern finding, but for familiarization, retention of a stated rule.) $\endgroup$
    – guest
    Commented Dec 12, 2022 at 23:36
  • $\begingroup$ ""if there are insights, they will be conveyed" doesn't seem to be a reliable approach—at least not if followed by all interested practitioners. " It's a population, not a Euclid shape. There's ZERO reason to expect everyone to follow this pattern and a long history (look at the literature on market development) to assume that if there's a better mousetrap that it will e conveyed from trials by first adopters to first followers to laggards. For the average practitioner, no reason to worry about this stuff. Better things to spend time on. $\endgroup$
    – guest
    Commented Dec 12, 2022 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.