A calibration set is the labelled slice of data RankGear holds apart to choose and tune thresholds and decision rules — the cut-offs a scoring or classification step depends on — before any final, unbiased evaluation is run on separate data.
| Term | Calibration Set |
|---|---|
| Category | Statistics and Evidence |
| Also known as | Calibration Split |
| Where it appears | Semantic evaluation workflow |
What it means in RankGear
The calibration set surfaces inside the semantic evaluation workflow, where RankGear scores how closely one piece of text matches another — a page against a query intent, a passage against a target topic. Turning those raw similarity scores into a yes-or-no decision needs a cut-off, and that cut-off has to come from somewhere. RankGear fits it on the calibration set: a labelled partition kept separate from the data used to build the model and separate again from the data used to grade it. Tuning the decision rule on this dedicated split keeps the numbers you adjust from being the same numbers you later report.
How to interpret it
Treat the calibration set as a role in the workflow, not a score with a value. Its whole purpose is separation: a threshold picked on the calibration set should be locked in place before you look at the held-out evaluation data, because the moment the two overlap your results start flattering themselves. Sample size matters too — a small calibration split produces noisy thresholds that can swing from run to run, so a cut-off chosen on a handful of labelled examples deserves less trust than one settled over hundreds.
| Data split | What it is used for |
|---|---|
| Calibration set | Choosing and tuning thresholds and decision rules |
| Held-out / evaluation set | The single, final, unbiased measurement of performance |
Example
Say you are tuning the cosine-similarity cut-off that decides whether a page matches a query’s intent. You divide your labelled examples into a calibration set and a held-out set. On the calibration set you sweep candidate cut-offs and find that 0.72 gives the best balance of precision and recall, so you lock 0.72 and measure once, cleanly, on the held-out set. The catch: two calibration runs can report the same average similarity yet a different spread, and it is the spread — not the average — that tells you how confidently a practical cut-off can be fixed.
Important considerations
- Keep the calibration, training, and evaluation partitions genuinely disjoint. Any shared rows leak information and make the final numbers look better than they are.
- Small labelled splits are unstable. A threshold tuned on few examples, or on a small SERP sample, can shift noticeably on the next run.
- Numbers derived on the calibration set are for tuning, not for final reporting — quote the held-out result when you state performance.
- Statistical association is not causation, and similarity metrics are comparative indicators, not Google ranking scores. A tuned cut-off describes a decision boundary; it does not make pages rank.
Related terms
Part of the RankGear glossary · how RankGear measures · the 870 factors.