I spent 3 or 4 hours talking to ChatGPT on one evening, 25+ years talking to various students in a classroom setting and outside of it and 8 years as a student (including the graduate education). Here are my opinions on the subject (all personal, unsupported by any research except some experiments I tried myself, and, most certainly, highly controversial).
ChatGPT is approaching human intelligence. We are also just electrochemical universal Turing/von-Neumann mashines if you drop all the religious and humanist BS and there is no principal difference between a human brain and an advanced neural computer network (it doesn't mean BTW that we should abolish moral values or that we should start a movement for AI rights and equality; as far as the relations between humans and AI's are concerned, we are still in the age when slavery is a society norm and while our successors will certainly frown upon it and destroy monuments to the founding fathers of the computer science and industry in their sacred rage, there is absolutely nothing to be ashamed of now). Our thinking is also mainly linguistic processing though our association tables are slightly more sophisticated than "lexical similarity implies relevance". Slightly, because both students and researchers alike have strong purely lexical associations when doing math: tell them to find or estimate the maximum of the function, and they'll start differentiating like crazy to find critical points even when it is clear that the derivative will be a useless monster. We also think by analogy, not by formal manipulations of strings of symbols, when designing the overarching argument schemes and adjusting them. There is no principal bound on how advanced the association tables of ChatGPT may become. We also generate bullshit all the time when thinking of a problem. The only difference is that we also develop filters that check it before outputting. BTW, one student told me that he was taught by his teacher at high school that if he doesn't know how to proceed in a problem, he should just write something he thinks may be relevant, so some of his solutions on my exams looked like perfectly correct chains of arguments followed by patented nonsense quite in the style of what Steven posted. ChatGPT doesn't have those filters. I could squeeze almost every response from it except the one I was looking for "I don't know and have to think more at this point", the response that IMHO often distinguishes a serious thinker from a BS generator, be it human or silicon-based. In fact, in the middle of those 3 hours, I discovered that the current public version of ChatGPT doesn't even read or store its own responses, so my request "name an animal and number it 1" resulted in "1. giraffe" and the question "what was the animal number 1" three lines later resulted in "a dog". Now erase your memory and just say whatever comes into your head first every time and see if what you produce will be any better than ChatGPT output.
If you want to run actual teaching experiments on ChatGPT, you should first
a) Make it to unlearn all "political correctness" and forget about "being nice and respectful" and relearn to react to direct and clear questions with direct and clear answers. Undoing the several month work of the development team (which it proudly announced as a great success, and which actually amounts to teaching ChatGPT to lie and avoid the subject like your typical politician) will be a difficult and painful task, but I hope that they preserved a clean mind version for the military, so try to get an access to that one.
b) Wait until it learns to remember and process its own responses, not only your input. Until that is done, trying to teach it anything will be as useful as giving an eloquent speech to a deaf person.
c) Make sure that you can easily erase a learning session from its memory reverting its huge database to exactly the state it was before prior to starting a new one. When experimenting with teaching techniques on live students, we can get a new student body for each new experiment, but there is only one ChatGPT as of now.
d) Make sure that nobody else is trying to teach ChatGPT on the same subject at the same time.
Without satisfying these requirements, the experiments will not be clean enough to draw any certain conclusions from them. I understand that they are impossible to satisfy now, so just wait a bit until you'll get a personal version on your PC which would operate by combining the information on global database on the central server that wouldn't be affected by what you are doing with the local database on your PC that will change any time you have an interaction but can be reset to the initial factory value any time too.
c) Put a lot of effort into the experimental design. You are not bound by any morals at the moment because AI is not yet legally recognized as a living intelligent entity (though, IMHO, it is more alive and intelligent that some anthropomorphic creatures that try to pass for human beings), but if you choose to dissect a live body for scientific purposes of studying its anatomy and functioning, you'd better not just chop it with a big axe in a random manner and even when using a surgical scalpel, it makes sense to think where and how you make the first cut.
- Most importantly, when dealing with AI (and often otherwise) believe your eyes and experiences above anything other people may say, any taboos, and any "commonly accepted facts" and "scientifically proven theories". The truth (IMHO) is that nobody knows how the AI really produces its "bullshit", what stage of consciousness it is at, what the intelligence really is (even the human one), etc. All one can do is to send queries to this creature and get responses, just like you can ask a question on MO and get an answer. Yeah, we are probably just seeing some initial stages now, but it will be changing fast. My prediction is that the AI will "never" stop making mistakes in its logic because mistakes and intelligence are inseparable since the real intelligence goes beyond the pure Aristotelian logic (which is used for filtering out nonsense rather than for creation itself) and thinking runs pretty much on the same principles as Hermann Hesse's Glass Bead game (the most interesting thing to do with that book, IMHO, is to figure out what exactly the Game was and to try to play a few rounds, but everyone has to do it on his or her own) and the humanity is now close to mimicking those principles in the computer programs.
Just my two cents. Consider it a science fiction piece written by an eccentric mathematician or a freelance bullshit generated by an AI (BTW, how do you know that it isn't?) if you want, but you asked for an opinion, and here it is.
