# How to test student's skills in programming or using software?

I am going to teach an undergraduate statistics course next year. My plan is to spend less time on theory and a more time on teaching students to doing some statistics on computer. However, this raises the questions of how to evaluate students fairly and discourage cheating. If you have taught courses involving some programming, how did you test students? What are the pitfalls to avoid?

Here are some ideas which comes to my mind, each with some drawbacks --

## Randomized assignments

Gave each student a slightly different assignment with some randomized parameters or data.

The drawback of this is that it is going to make grading more difficult.

## Group project

Divide students in groups and ask them to propose a project. Each group then chooses a project proposed by other students, collect and analyze the data, and then present their result to the class.

From my own experience as a student, usually the stronger students in a group do most of the work, and the weak ones almost do nothing.

## In-class closed-book exam

Give some students some problems to solve on their own laptops within limited time, in class, and without Internet access.

The problem with this approach is that it is difficult to enforce no Internet access (we don't have computer labs). And there is a bit danger that if a student's suddenly have problem with his coding environment, he/she won't be able to do the exam.

• I wonder if there's a way to partially automate grading of the randomized assignments? May 14, 2021 at 18:34
• Automating grading is quite a lot of work. For a small class, it usually does not worth the effort. There are many many ways an automatic grading system can fail.
– user11702
May 14, 2021 at 19:12
• How much will you have to teach programming? "Less time on theory" ... is that OK with the follow-up courses? May 14, 2021 at 23:44
• May 15, 2021 at 9:54
• If I signed up for a statistics course and found it to be a programming intro course, I would be surprised and not in a good way. May 16, 2021 at 16:43

I teach intro to Comp Sci and my main methodology is to flip the classroom, give the lecture in a video to be watched before class, make class a lab, give them homework-like assignments that they work on in front of me to practice and break and fail and ask questions, and then homework is a project that's either the same thing with a slight variation, or a combination of several things practised during the lab. I don't give tests because I'd rather them spend time on projects than study to an exam.

I generally don't worry about randomizing their projects too much, but I do try to randomize the INPUT. I design them so I can write a test harnass to run all their programs with some test cases, check the output, and print if the output was correct. So to that end, it can be randomized, just to make sure they're not hardcoding output.

Example 1: labs one day include reading 1 filename on the command line, opening the file, and counting lines and characters. Homework: also count words, and have it work for N files on the command line. (With ample additional lecture notes on what goes into counting words, and leeway if they're just close, as there are easy and hard algos for doing this.)

Example 2: labs include printing out looped data in html, including tables. homework is to take a previous homework that printed matrices (and their cross product) just in text, and to add all the tags inbetween so that it formats properly.

I can feed in my own files to test their output for example 1. And for the other example, their matrices already have randomized data in them, so I wrote an output reader that reads their output and checks to see if the math was correct, also guaranteeing that their output isn't hardcoded.

Beyond that, I just hand check to see there's no direct copying going on.

• Thanks! Do have exams? How does it work?
– user11702
May 14, 2021 at 20:30
• no exams, i prefer them to spend time actually programming. studying for an exam eats into that time. May 14, 2021 at 20:34
• Do you divide students to groups during lab sessions? Or do they work by themselves?
– user11702
May 21, 2021 at 13:54
• i haven't found a good dividing metric. i have them all in a single room. if anyone has a question, i have them share their work and everyone can watch. only when they're completely done the labs and working on their homework do i not have them share with everyone any more. instead, i take them one at a time into a private room to share screens and work with them, and then return the main room. May 24, 2021 at 21:27

Among other courses, I teach Introduction to CompSci via Programming. I use group projects because every post-grad job will involve programming in groups.

• I've had success with elementary 2D graphics, allowing the students considerable artistic latitude: spirographs, fish swimming in an acquarium, etc. Various requirements, i.e., use of classes. Difficult to grade, but cheating not an issue.
• Less innovative but still effectively engaging: Design and implement a simple game, some variety of breakout or PacMan. Again with only the most primitive graphics.
• Among my most successful projects was to capture a sampled stream of all Twitter tweets within a specific time period (quite a firehose), and ask the students to parse and then analyze the tweets for, say, emotion, negativity, political leaning, English vs. Spanish, etc. I spent ~$$8$$ hrs figuring out how to capture a Twitter stream, which I then made available to them. Again, difficult to grade but cheating not an issue. They felt they were doing something "real" (including running out of memory), and could potentially make sociological discoveries.

If you have access to a Moodle server, you could look at CodeRunner (https://coderunner.org.nz/). This is a Moodle question type that can be used for automated assessment of programming exercises in a variety of languages. The documentation says the following about supported languages:

CodeRunner currently natively upports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, Pascal, JavaScript (NodeJS), Octave and Matlab. However, other languages are easily supported without altering the source code of either CodeRunner just by scripting the execution of the new language within a Python-based question.

(Personally I have set up a server with CodeRunner that one of my colleagues is using for teaching Python. He seems happy with it, but I have not had much involvement since the initial setup.)