Signal detection theory, confidence ratings

Old/new recognition on a rating scale: K − 1 thresholds on one evidence axis give K response categories

The decision: K − 1 thresholds split the evidence into K ratings

Evidence on noise (new) and signal (old) trials, and the ratings between the thresholds. Drag a threshold or use the arrow keys.

Task settings

Parameters dist

Shape of the noise and signal distributions: normal; logistic, symmetric with heavier tails; gumbel_min or gumbel_max, skewed mirror images whose location 0 is the peak. Each enters with scale 1, so under the last three d is in units of that scale, not of the SD. In bmm: the dist argument. At the start, logistic instead of normal raises the share of noise trials rated 6 from 0.02 to 0.11.

Sensitivity: how far the signal distribution sits from the noise distribution, in units of √((1 + sdratio²)/2), the root mean square of the two scales. At sdratio = 1 it is d′. At the start, d = 1, rating 6 takes 0.13 of the signal trials and 0.02 of the noise trials.

sdratio

Scale of the signal distribution over the scale of the noise distribution. Above 1 the signal distribution is wider, and the ROC is no longer symmetric. At sdratio = 2 the signal trials spread to both ends: rating 1 takes 0.09 of them instead of 0.02, rating 6 takes 0.40 instead of 0.13. bmm estimates it on the log scale.

Thresholds t1 < t2 < … split the evidence axis into the ratings: evidence below t1 gives rating 1, between t1 and t2 rating 2, and so on. They are measured from the noise location in units of the noise scale. bmm builds them from a criterion and further parameters, set by threshold_type; the code below the dashboard uses log_distance, where delta1, delta2, … are the log widths of the gaps. The start thresholds −1.11, −0.19, 0.5, 1.19 and 2.11 are criterion = 0.5 and delta1 to delta4 = −0.083, −0.371, −0.371, −0.083.

t2 t3 t4 t5 t6 t7

Predicted ratings simulate

Draws N trials, half noise and half signal, marks each rating's share in the sample on its bar (black lines) and the sample's cumulative rates as points on the ROC. Below the panel: the expected mean rating on noise and on signal trials and, in brackets, the sample's. The sample keeps its random numbers while the sliders move, so it changes with the parameters; New sample draws other ones. In bmm: rsdt_rating(), in the code under In bmm.

Probability of each rating on noise and striped signal trials. Dashed: bmm's prior means. Black: the sample.

ROC from the cumulative rates

Circles: P(rating ≥ j), noise vs. signal. Solid: all criteria. Dashed: prior means. Black dots, with simulate on: the sample. Grey: chance.

What the model says

A confidence rating is a yes/no decision with several criteria at once (Green & Swets, 1966). The evidence axis and the two distributions are those of the yes/no model: noise trials (new items) draw their evidence from a distribution at 0 with scale 1 (for the normal distribution, SD 1), signal trials (old items) from the same distribution moved to \(\delta\) and stretched by \(s\) (sdratio). d is the sensitivity: \(\delta = d\,\sqrt{(1 + s^2)/2}\), which equals d when \(s = 1\). dist names the shape, normal, logistic or one of the two skewed Gumbel distributions. \(K - 1\) ordered thresholds \(t_1 < \dots < t_{K-1}\) split the axis into \(K\) ratings, from 1 (sure new) to \(K\) (sure old). The probability of rating \(j\) is the area under the density between thresholds \(j - 1\) and \(j\). With \(F\) the cumulative distribution function of the distribution at location 0 and scale 1, \(t_0 = -\infty\) and \(t_K = \infty\):

\[ P(j \mid \text{noise}) = F(t_j) - F(t_{j-1}), \qquad P(j \mid \text{signal}) = F\!\left(\frac{t_j - \delta}{s}\right) - F\!\left(\frac{t_{j-1} - \delta}{s}\right). \]

The thresholds are positions measured from the noise location in units of the noise scale, and sdratio does not rescale them. A row of data holds the \(K\) counts of one stimulus class, which are multinomial with these probabilities. bmm does not estimate the thresholds themselves; its argument threshold_type decides how they are built from a criterion and further parameters. parsimonious and equidistant use fewer parameters than there are thresholds; log_distance uses as many. Every type places a criterion at the same point, the middle threshold when \(K\) is even and the centre of the middle rating when \(K\) is odd, and builds the other thresholds around it. The default, parsimonious, uses one more parameter, spacing, and puts threshold \(k\) at \(\text{criterion} + e^{\text{spacing}} \log\big(k / (K - k)\big)\) (Selker et al., 2019). log_distance instead estimates the log widths of the \(K - 2\) gaps between neighbouring thresholds, delta1, delta2, and so on, which can place any ordered set. This page moves the thresholds one by one, as log_distance can, and the code below gives the matching criterion and delta values.

The task

One trial presents one item, a studied or a new one, and the participant rates it on a \(K\)-point scale from “sure new” to “sure old”. bmm fits counts: each row of the data holds one count per rating, in columns ordered from “sure new” to “sure old”, and stimulus coded 0 for new (noise) and 1 for old (signal) items. The number of ratings \(K\) is the one task setting the likelihood knows: it sets the number of thresholds.

Explore

The top panel draws both densities with the \(K - 1\) thresholds; the numbers between them are the ratings. Drag a threshold or move its slider (t1 is the leftmost). A threshold that runs into its neighbour pushes it along, so the order holds; a typed value out of order is flagged instead. The bars give the probability of each rating on noise and on signal trials.

Each threshold also turns the ratings into a yes/no decision: count every rating at or above \(j\) as “old”. The rate of such ratings on noise trials is then a false-alarm rate, on signal trials a hit rate. The ROC (receiver operating characteristic) panel plots these pairs, one circle per threshold: the circle labelled ≥ \(j\) comes from threshold \(t_{j-1}\). They lie on the solid curve, which gives the hit rate against the false-alarm rate for every possible threshold at the current d and sdratio. The grey diagonal is chance.

d and sdratio have the ranges of the yes/no page: d from −2 to 4 (bmm’s prior normal(1, 1), mean 1 and SD 1, ± 3 SD), sdratio from 0.41 to 2.46 on the log scale (normal(0, 0.3)). The threshold sliders span the prior of the criterion, normal(0.5, 1.5), from −4 to 5. Changing \(K\) places the thresholds where parsimonious puts them at its prior means, criterion 0.5 and spacing 0, rounded to two decimals: for \(K = 6\), \(0.5 + \log(1/5) = -1.11\) up to \(0.5 + \log 5 = 2.11\). The dashed bars and the dashed ROC are these thresholds with d = 1 and sdratio = 1, so at the start values they lie under the solid ones.

Across task settings

\(K\) is the only design variable, and it does not change what a threshold predicts: each threshold is the criterion of a yes/no decision, and its point lies on the same ROC whatever \(K\) is. What \(K\) changes is how many points one condition gives. A yes/no condition gives a single point, and many curves pass through one point, which is why sdt_yn() cannot tell sdratio from d within one condition. \(K - 1\) points on one curve fix its shape, so sdt_rating() identifies sdratio from one condition. Under normal noise this is easy to see on z-scores, the rates transformed by the inverse of the normal distribution function: the ROC becomes a straight line with slope 1/sdratio. Counting what a condition offers: each stimulus class has \(K\) proportions that sum to 1, so \(2(K - 1)\) are free, and the model with sdratio estimated has \(K + 1\) parameters (d, sdratio and \(K - 1\) thresholds). At \(K = 3\) the fit uses every degree of freedom, and from \(K = 4\) the shape of the ROC can be tested.

In bmm

The code updates with the controls. sdt_rating() names the count columns, the stimulus column, the threshold parametrisation and the evidence distribution; dsdt_rating() returns the log likelihood of the counts; and bmm() estimates the parameters from data. my_data stands for your own data, and fitting runs Stan through brms. dsdt_rating() takes the thresholds as positions and sdratio as the ratio itself, while the model estimates the log_distance parameters and the log of sdratio. See the signal detection article and the reference pages for sdt_rating() and dsdt_rating().