7
$\begingroup$

I am analyzing some data on ALEKS for my home institution (ALEKS, an acronym that stands for "Assessment and Learning in Knowledge Spaces", is an online tutoring and assessment program that includes course material in mathematics and many of the sciences). ALEKS has an adaptive assessment program that is used by many institutions of higher learning for placement purposes. I am currently in the process of analyzing the assessment scores from ALEKS for placement into our Calculus 1 class. The way ALEKS was administered was that the students were allowed to take the placement test multiple times and given the chance to improve their background (i.e. remediate) using ALEKS in between test attempts. My data set consists of the ALEKS placement test scores and the total time spent in remediation with ALEKS (for each student).

I am trying to find a ballpark estimate for the expected rate of improvement (in terms of number of new topics learned or % increase in score) a student should see from remediation. I would like to have some benchmark with which to compare the rate of improvement from my data set. In other words, are my students doing better, worse , or about the same as expected using the remediation available via ALEKS?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Not a study, but I have a colleague who implemented ALEKS at a different institution as a replacement for remedial precalculus. The result was that it did no worse than instructors in classrooms. Other folks I've talked to say gamification is a problem. Also, some institutions have instructors who teach to the test I guess, this can also reduce the verity of ALEKS results. I think similar complaints can be waged w.r.t. "emporiums" for which studies "show" effectiveness without regard to any proper control group. $\endgroup$
    – James S. Cook
    Commented Dec 27, 2018 at 3:15
  • $\begingroup$ My main concern is with the "gamification" aspect. If students take a placement test unproctored, based on anecdotal evidence and reports that I have come across, I am pretty confident that a fair percentage of the students will in some shape, way, or form take the test inappropriately. This becomes a challenge for math departments to figure out how to set ALEKS placement test cut off scores with a data set that is only partially valid. $\endgroup$
    – Matt Brenneman
    Commented Dec 27, 2018 at 13:07

1 Answer 1

Reset to default
3
$\begingroup$

I don't think you are going to find data yielding an "expected rate of improvement." Much depends on the circumstances of how ALEKS is deployed.

However, one can find academic papers on the effectiveness of ALEKS, even if they do not answer your question directly. For example, (1) says that "ALEKS had a significant effect on students‘ mathematics achievement":

(1) Nwaogu, Eze. "The effect of ALEKS on students' mathematics achievement in an online learning environment and the cognitive complexity of the initial and final assessments." Dissertation, Georgia State University, (2012). GSU PDF download.

Another study (2) says that "students randomly assigned to the ALEKS condition significantly out-performed students assigned to the teacher condition on a state assessment test (TCAP). However, this was only if the students received sufficient exposure to the program":

(2) Craig, Scotty D., Celia Anderson, Anna Bargagloitti, Arthur C. Graesser, Theresa Okwumabua, Allan Sterbinsky, and Xiangen Hu. "Learning with ALEKS: the impact of students’ attendance in a mathematics after-school program." In International Conference on Artificial Intelligence in Education, pp. 435-437. Springer, Berlin, Heidelberg, 2011. Conference link.

We have used ALEKS at my institution, with as-yet inconclusive results.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I appreciate the response. I had studied the first reference (but it pertained to a lower level math class and the quality of the study was so poor, the validity of the conclusions are at best, tenuous). The other reference is directly relevant, but does give a number of references that look promising. $\endgroup$
    – Matt Brenneman
    Commented Dec 27, 2018 at 16:21
  • 3
    $\begingroup$ I look forward to going through them because speaking as a statistician, most of what I have seen is generally sloppy research that seems to be adopted by other educators as "evidence" that ALEKS is better than traditional methods. Like you pointed out, assessment is a complicated business, but you would expect that educators, knowing this, would work towards developing a standards of best practice (since ALEKS didn't apparently do this research on their own before marketing their product). Anyhow, my forte isn't educational research, so this has been an interesting experience for me. $\endgroup$
    – Matt Brenneman
    Commented Dec 27, 2018 at 16:34
  • $\begingroup$ @MattBrenneman: IMO educational research is generally pretty terrible/unreliable. (The only thing that I feel confident about from the research is that online educational for remedial students is a disaster.) $\endgroup$
    – Daniel R. Collins
    Commented Dec 30, 2018 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.