Introduction to Item Response Theory
IRT is an adaptive testing framework that can give graded scores on tests and reduce the time it takes to administer tests.
At the base IRT defines a model that associates some latent ability of the student to probability of answering correct on problems. where is answering correctly on problem given the student's ability and the intrinsic parameters of the problem collected in the vector .
We often want to study the problem intrinsic parameters () as a function of latent ability (). We call this the characteristic equation of a problem which is just a plot.
In standard models, is unbounded that is , and a sigmoid function () is usually used as a base for the characteristic equation. This is convenient since sigmoid is defined , and has a range of which makes it a natural probability density function.
Even though is unbounded, in practice, is typically in the range of .
There are some formal assumptions for how should behave. They're mostly common sense, but it's worth noting them regardless.
- Monotonicity: The probability of answering correctly, , should monotonically increase as increases.
- Unidimensionality: Basic models assume that there is only one latent trait that encodes student ability, , but this assumption can be relaxed in more advanced models.
- Local Independence: Responses to different problems are independent from each other.
- Invariance: The intrinsic parameters of the problems are stable across different students.
1 Parameter (Rasch model)
The simplest model uses only one intrinsic parameter: the difficulty of the problem. We write the
where is the difficulty parameter. It is analogous to in its range. The neutral value is .
2 Parameter model
We can extend this model by introducing a discrimination parameter that varies the slope of the characteristic equation.
where is the discrimination parameter.
should be in the range where if means that skill has no effect. Everyone has the same probability of answering correctly given by , and in the 4 parameter model the average of . If approaches then the discrimination becomes a perfect step function. The neutral value is .
3 Parameter model
The 3 parameter model takes account for the probability of guessing correctly by raising the lower asymptote
where is the guessing probability parameter.
should be in the range where means that it's impossible to guess the solution, and means that it is impossible to fail the question. The neutral value is .
4 Parameter model
The 4 parameter model is used less often, but is still worthwhile to discuss It introduces a slip factor: The probability of someone who knows the concept makes a mistake.
where is the upper bound probability parameter.
should be bigger than and is in the range . It encodes the probability of someone who knows the answer actually answering correctly. The neutral value is .
Slip factor should be larger than guess factor.
Guess factor (0):
Slip factor (1):
Fitting the model
We fit the models using maximum likelihood estimation. This is the same whether or not you know the intrinsic problem parameters. However, if you already know the problem difficulties, then you can choose to iteratively estimate student abilities using only a subset of the problem dataset.
By choosing the problems based on which one gives the maximum information: The maximum of , you can quickly estimate the student's ability to a high degree of confidence. The highest information problem is the problem where you guess the student has the closest to chance of answering correctly.