A corpus statistic that gives more weight to terms appearing in few documents and less weight to terms that appear almost everywhere — so distinctive words count for more than common ones.
| Term | Inverse Document Frequency |
|---|---|
| Category | Statistics and Evidence |
| Also known as | IDF |
| Where it appears | BM25 and BM25F math |
What it means in RankGear
Inverse Document Frequency is one of the two ingredients inside RankGear’s BM25 and BM25F scoring. Where term frequency counts how often a word occurs on a single page, IDF looks across the whole set of pages RankGear pulled for a keyword and asks how rare that word is. A word that shows up in only a handful of the analyzed documents carries a high IDF; a word that appears in nearly every one carries a low IDF. That weighting is what keeps filler words from dominating a relevance score and lets the terms that actually distinguish one page from another do the work.
How to interpret it
Read IDF as a rarity weight, not a quality signal on its own. Its scale is comparative and tied to the corpus RankGear assembled for that keyword — the same word can carry a different IDF for a different query because the document set changed. A high value marks a term as distinctive within this set; a low value marks it as common. Because IDF is a log-scaled function of how many documents contain the term, the gap between a word in 2 pages versus 5 is far larger than the gap between one in 200 versus 205. Confirm the corpus is a reasonable size before leaning on any single term’s weight.
| Value | What it tells you |
|---|---|
| High IDF | Term appears in few of the analyzed pages; it is distinctive and contributes strongly to the BM25 score. |
| Low IDF | Term appears in most of the pages; it is common and adds little to the score. |
| Near zero | Term is present in essentially every document; it barely moves relevance either way. |
Example
For a keyword like “waterproof hiking boots,” a word such as “the” appears in every page RankGear pulls, so its IDF sits near zero and it barely shifts the score. “Waterproof” might appear in about a third of the pages, earning a moderate weight, while a specific term like “gore-tex” that surfaces in only a few pages carries a high IDF. A page using that rarer term gains more BM25 credit for it than for a common word, even when both appear the same number of times.
Important considerations
- IDF is comparative and corpus-dependent: it describes rarity within the specific set of pages RankGear analyzed for one keyword, not an absolute property of the word.
- It is one input to BM25 and BM25F, not a ranking factor. A high-IDF term does not make a page rank — the score is a comparative indicator, not a Google score.
- Statistical association is not causation, and small SERP samples can make any single term’s weight unstable.
- Log scaling means differences in document count matter most among rare terms; treat very small corpora with caution.
Related terms
Part of the RankGear glossary · how RankGear measures · the 870 factors.