12
$\begingroup$

First off I apologize if this question comes out to be off-topic. I wasn't sure whether to post it on MSE or here.

Lately I gave an exam named SSC-CGL. This exam is conducted every year in India for the selection of grade B officers. The exam is of 200 questions out of which 50 questions are asked from mathematics. The exam is conducted in four shifts. Each shift is allocated different question-set, but the syllabus is exactly same. When I compared my paper with other shift's paper I found their difficulty level was very low and questions from my paper were comparatively lengthy. As an example consider these two questions.

My paper's question:

If $x^2+x=5$ then find the value of $\dfrac{1}{(x+3)^3} + (x+3)^3$

Their paper's question:

If the measure of three angles of a triangle are in ratio 2:3:5 then find the angles of the triangle

First question requires the trick of substitution $x+3=t$ but the other question can be solved by a school student because an 8th grade student knows that the sum of the angles is 180 degrees.

Similarly I found every question in my paper relatively difficult than theirs, because my questions required tricks but their question required direct use of the concept. This is my paper and here is the other one, math questions start from 101st question in both the papers.

The problem is that last year too there was a difference in the difficulty of different shifts but we could not give a solid argument in court to justify the fact. That is why now I'm asking here about whether there is some standard procedure in mathematics which could compare the difficulty level of questions. E.g. I think that number of steps or length of the solution can be valid parameter. Length might not signal the difficulty but in competitive exams time per question is an important parameter.

Some say that difficulty is a subjective matter and hence we can't say some question is more difficult than others. But we very well know that questions asked in Mathematics Olympiad are more difficult than those asked in school exams. Whenever teachers set an exam they ask questions based upon the pre-defined difficulty of the exams. No teacher would ask an Olympiad type question in a school exam and vice-versa.

Tl;dr: Is some standard procedure in mathematics which could compare the difficulty level of questions asked from same topics(syllabus)?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Direct use of a concept may turn out to be more difficult than somewhat tricky calculations. $\endgroup$
    – Tommi
    Commented Aug 19, 2015 at 10:29
  • $\begingroup$ @TommiBrander But if all the candidates are given a prescribed syllabus, that is a particular set of concepts. Every candidate knows that question will be asked from only these concepts and in the exam they can easily recall some particular concept, but using an altogether afresh trick and doing relatively lengthy calculations is far more difficult and time consuming. $\endgroup$
    – user31782
    Commented Aug 19, 2015 at 11:30
  • $\begingroup$ There's the scale used by Knuth in "The Art of Computer Programming". But even if I've occasionally seen it used elsewhere (to my shame I cannot think of any examples though) it would hardly be considered standard. $\endgroup$
    – tilo.wiklund
    Commented Aug 17, 2016 at 18:21

5 Answers 5

Reset to default
9
$\begingroup$

I don't know that there are any standardized measures that work by analyzing the questions themselves (I doubt it) but there are a number of statistical approaches that work by analyzing the test-takers' performance. For example, depending on the size of the shifts one could look to see if there are statistically significant differences in mean score or mean time to complete. (You did not indicate whether the shifts are all taught by the same instructors; if not, then it will be hard to argue that the differences in outcomes should be ascribed to the test rather than to the teacher.)

You could also pre-evaluate the different versions by gathering a sample of people who successfully took the exam the previous year, randomly assign each them 1 of the 4 different versions of the exam, and look to see if there are statistically significant differences in time to complete or score. If it is possible to design the sample so that everybody in it took the same version of the exam the previous year, you could also correct based on past performance.

You may want to look at Wikipedia's articles on on Psychometrics and Item Response Theory.

Improve this answer
$\endgroup$
1
  • $\begingroup$ There are no teachers. It is a competitive exam and the top scoring candidates get Grade B govt job in India. There is a predefined syllabus. There are 4 shifts. Each shift gets a different paper but in merit list they consider all candidates equivalent. I'm dead sure that the mean score of easy-exam-shift will be quite high as compared to others. $\endgroup$
    – user31782
    Commented Aug 19, 2015 at 15:01
5
$\begingroup$

Good topic.

Let me say something about it. This year I've been working in my thesis which involves in part an answer for your question. In my work, some approach are posed, some of them are related to recognize a good problem. Although, some scholars say this topics is complex and controversial, in Mathematics Education there are some frameworks which propose theories about how a good problem could be understood.

For instance, you might read these papers:

Crespo, S., & Sinclair, N. (2008). What makes a problem mathematically interesting? Inviting prospective teachers to pose better problems. Journal of Mathematics Teacher Education, 11(5), 395-415. (Springer link.)

Kontorovich, I. (2012). What makes an interesting mathematical problem? A perception analysis of 22 adult participants of the competition movement. In B. Roesken & M. Casper (Eds.), Proceedings of the 17th MAVI (Mathematical Views) Conference (pp. 129-139). Bochum: Germany. (PDF download.)

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Although these are both good references, the question was about how difficult a question is, not how interesting it is. $\endgroup$
    – mweiss
    Commented Aug 28, 2015 at 1:04
  • $\begingroup$ Could you give links to those papers? $\endgroup$
    – user31782
    Commented Aug 28, 2015 at 16:30
  • $\begingroup$ If I could get any way to prove difficulty, then that will be very helpful for lacks of people who appear in that exam. $\endgroup$
    – user31782
    Commented Aug 28, 2015 at 16:35
3
$\begingroup$

Again, not a "standard method" but one that is gaining traction is comparative judgement, as explained on the No More Marking website. https://www.nomoremarking.com/aboutcj

This has been used in studies by the regulators for UK exams (Ofqual) to determine factors such as if difficulty has increased over time and if different exam boards are producing examinations of comparable standards of difficulty. See for example https://ofqual.blog.gov.uk/2015/05/21/gcse-maths-final-decisions/

My understanding is that each individual question would be uploaded to the site. These are then allocated randomly in pairs for 'experts' to make a series of judgements on the basis of one focussed question. (For example, "Which of these questions is the more difficult to answer?".) Once enough judgments have been submitted by enough experts, a ranking of the questions is produced. However, you would then still need to decide how to apply that ranking to compare the overall difficulty of each shift's examination.

Improve this answer
$\endgroup$
2
$\begingroup$

In If you are feeling difficulty in SSC then SSC exam there is no method till now made by officials, they do not use normalisation method. But if in future SSC decides to use normalisation method to take into consideration the difficulty level variations in the question papers across different sessions.

The normalization process is based on a simple formula and based on a few parameters as discussed below, the Exam organizing committee arrives at the following formula for calculating the normalized marks for the multi session papers.

Once the answers are checked and assessed, the normalized marks of a candidate will be calculated corresponding to the actual marks secured by the candidate in the examination and the Score card will be devised based on the normalized marks.

Improve this answer
$\endgroup$
1
$\begingroup$

I have a practical suggestion which is to not have one person write a test for one shift and the other write it for the other. Instead pool your questions and just do a random sample of them for the group. Still some danger that luck will drive a difference but probably much less than the way you are doing it now. This will also ensure coverage is more even (e.g. algebra versus geometry).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.