How WikiSignals turns expert ratings, vetted lists, and a few telltale signals into one credibility score per website.
The short version
Every website in WikiSignals can carry a WikiSignals score from 0 (avoid) to 1 (rock solid). It estimates how credible the site is as a source of information, at the level of the whole domain, not individual articles. Right now 20,968 of our 73,339 domains have one.
A site gets its score in one of three ways:
- Professionally rated. If Media Bias/Fact Check (MBFC) has rated the site, we use that rating. If not, we fall back on Domain Quality (DQ). That covers 13,608 sites.
- Vouched for by a vetted list. Press associations and news indexes don’t let just anyone in. When a site is on one of our 7 vetted lists but nobody has rated it, we estimate its score from how its rated list-mates do, then nudge it up a little for a few good signs. That covers 6,951 more sites.
- Flagged by an unreliable list. If an unrated site turns up on a list of fake-news, spam, or deprecated sites, it gets that list’s low score, no matter what else it has going for it. That covers 409 sites.
A few things worth knowing up front:
- Birds of a feather. Members of vetted press lists are overwhelmingly rated high by the experts: about 9 in 10 rate High or Very High, and fewer than 1 in 100 rate Low.
- The more scored domains in a list, the higher our confidence in the list’s average. So lists with fewer rated members get a bigger safety discount.
- We err low on purpose. A score is a kind of vouching, and others reuse our data. Over-scoring a dubious site is worse than under-scoring a good one, so our estimates for unrated sites land below what the experts typically give comparable sites (around 0.66 vs. 0.86).
- Estimates are estimates. Unrated list members are smaller and less known than rated ones, and they may not share every trait of their rated list-mates. Some have gone offline or changed hands. We explain below how we tested for this and what we still can’t rule out.
If you use our data for research, each score comes with its basis (rated, list, or unreliable) so you can keep professional ratings separate from our estimates.
In detail
The rest of this page is for the curious, the skeptical, and the peer reviewers. It gets drier and more numeric as it goes.
1. Ground truth: professional ratings
Our reference point is expert ratings. We use MBFC whenever it has rated a domain (8,863 domains) and Domain Quality otherwise (4,745 more). DQ aggregates several expert rating sets, but it has not been updated since 2023, and it tends to rate fake-local (“pink slime”) news sites higher than MBFC does, so MBFC comes first.
Why trust expert ratings at all? A study of six rating sets found that experts “generally agree on the relative quality of news domains” (Lin et al., PNAS Nexus, 2023). Ratings still differ in the details, so we treat them as the best available reference, not as infallible.
MBFC publishes words, not numbers, so we convert its Factual Reporting and Credibility ratings into a 0–1 score, weighting factual reporting more heavily:
| Factual Reporting (value) | Low credibility (0) | Medium credibility (0.5) | High credibility (1) |
|---|---|---|---|
| Very High (1.0) | 0.70 | 0.85 | 1.00 |
| High (0.8) | 0.56 | 0.71 | 0.86 |
| Mostly Factual (0.6) | 0.42 | 0.57 | 0.72 |
| Mixed (0.4) | 0.28 | 0.43 | 0.58 |
| Low (0.2) | 0.14 | 0.29 | 0.44 |
| Very Low (0.0) | 0.00 | 0.15 | 0.30 |
2. Which lists count
We track dozens of lists: press-association member directories, news indexes, and lists of unreliable sites. Each is tagged as a Reliable candidate, an Unreliable candidate, or excluded. A candidate only counts toward scores if its rated members pass a strict test:
- Reliable lists: we must be 95% confident that no more than 3% of the list’s members are rated Low or Very Low (0.4 or under).
- Unreliable lists: we must be 95% confident that no more than 25% of the list’s members are rated High or Very High (above 0.6).
“95% confident” comes from the Wilson score interval, a standard way to put error bars on a percentage. Its upper bound is the worst case we can’t rule out. It automatically accounts for sample size: a list with 0 lows among 37 rated members still has an upper bound of about 9%, so it waits until more of its members are rated.
Why 3%? It roughly matches how often the experts themselves disagree this badly. When DQ rates a site High, MBFC rates it Low about 3% of the time (leaving out pink-slime sites, where DQ is known to be off); the reverse happens about 0.2% of the time. Why the looser 25% for unreliable lists? Wrongly scoring a good site low is the less harmful mistake, and these lists only ever lower a score.
| List | Members | Rated | Share rated | Average rating | Upper bound (95%) | Selected | Baseline score |
|---|---|---|---|---|---|---|---|
| US Newspaper Links | 4,304 | 1,407 | 33% | 0.80 | 0.6% | Yes | 0.665 |
| State Press Associations | 4,020 | 1,092 | 27% | 0.81 | 0.8% | Yes | 0.659 |
| JournalList: trust.txt | 2,220 | 679 | 31% | 0.79 | 1.9% | Yes | 0.651 |
| Alliance for Audited Media | 816 | 396 | 49% | 0.80 | 2.6% | Yes | 0.697 |
| National Newspaper Association | 1,200 | 270 | 22% | 0.80 | 2.1% | Yes | 0.641 |
| Project Oasis | 1,440 | 226 | 16% | 0.77 | 2.5% | Yes | 0.593 |
| Institute for Nonprofit News | 385 | 144 | 37% | 0.83 | 2.6% | Yes | 0.703 |
| Wikipedia (en) Perennial Sources: Generally reliable | 196 | 118 | 60% | 0.80 | 3.2% | Not yet | — |
| Local News Initiative: Digital | 513 | 89 | 17% | 0.82 | 4.1% | Not yet | — |
| Local Independent Online News Publishers | 493 | 80 | 16% | 0.80 | 4.6% | Not yet | — |
| CoLab News | 163 | 64 | 39% | 0.80 | 5.7% | Not yet | — |
| National Newspaper Publishers Association | 130 | 39 | 30% | 0.62 | 9.0% | Not yet | — |
| Association of Alternative Newsmedia | 77 | 37 | 48% | 0.77 | 9.4% | Not yet | — |
| Impress | 173 | 23 | 13% | 0.69 | 14.3% | Not yet | — |
| Corporation for Public Broadcasting | 307 | 11 | 4% | 0.86 | 25.9% | Not yet | — |
| Global Forum for Media Development | 185 | 7 | 4% | 0.80 | 35.4% | Not yet | — |
| List | Members | Rated | Share rated | Average rating | Upper bound (95%) | Selected | Baseline score |
|---|---|---|---|---|---|---|---|
| Pink Slime (fake-local news) | 1,316 | 1,126 | 86% | 0.33 | 8.5% | Yes | 0.305 |
| Wikipedia (en): List of fake news websites | 333 | 311 | 93% | 0.17 | 1.8% | Yes | 0.161 |
| MediaWiki: SpamBlacklist | 177 | 85 | 48% | 0.23 | 18.9% | Yes | 0.119 |
| Wikipedia (en) Perennial Sources: Deprecated | 148 | 43 | 29% | 0.26 | 15.5% | Yes | 0.118 |
| Wikipedia (en): List of satirical news websites | 32 | 29 | 91% | 0.39 | 11.7% | Yes | 0.370 |
| Wikipedia (en) Perennial Sources: Blacklisted | 43 | 19 | 44% | 0.21 | 24.6% | Yes | 0.100 |
| Wikipedia (en) Perennial Sources: Generally unreliable | 182 | 59 | 32% | 0.55 | 43.1% | Not yet | — |
Some lists are deliberately excluded. Google News indexes 9,225 rated domains, but 22% of them rate Low or Very Low, so it is used only as a small add-on signal (below), not as a list. The MBFC index itself is excluded because it is the ground truth. The Iffy.news Index of Unreliable Sources draws on WikiSignals data, so using it here would be circular. And the OpenSources list was a classroom project, never meant as a vetted list.
3. The baseline: list average minus a safety discount
Each selected list gets a baseline score for its unrated members:
baseline = average rating of rated members − 0.2042 × (share of members not rated)
In words: start with how the list’s rated members score, then subtract more the less of the list has been rated. A list where nearly everyone is rated barely gets discounted; a list where most members are unrated gets a larger cut. We use the share not rated, rather than just the number rated, because the experts don’t pick sites at random. They rate prominent outlets first, so the less of a list is rated, the more its unrated members may differ from the rated ones.
The 0.2042 weight was not chosen by hand. It is the value that best fits the data under our “err low” rule (section 6), with a 95% bootstrap interval of 0.164 to 0.233. Baselines for the 7 Reliable lists range from 0.593 to 0.703 (Table 1). For a reliable list’s average we leave out members who are also on a selected Unreliable list, since those sites never get the list’s baseline anyway.
4. Add-ons: a few good signs
An unrated list member can earn up to about 0.12 on top of its baseline:
| Part | Value used | 95% interval | Notes |
|---|---|---|---|
| Penalty weight | 0.2042 | 0.164 – 0.233 | Multiplies the unrated share of the list |
| Wikipedia links add-on | up to +0.0476 | -0.009 – 0.161 | 0 at 200 or fewer links, full at 1,000+, log scale in between |
| Google News add-on | +0.0463 | -0.000 – 0.139 | Domain is indexed by Google News |
| Second vetted list | +0.0303 | -0.019 – 0.089 | Domain is on two or more of the 7 vetted lists |
Each signal was picked because it tracks credibility among rated sites, and each value was estimated from the data rather than set by hand. They are fitted together, so overlapping signals (well-known sites tend to have all three) aren’t double-counted. Each add-on’s interval is wide, and taken one at a time none is a sure thing. Together they make estimates noticeably more accurate (section 7). We also tested the domain’s registration year: once the other signals were included it added nothing, so it isn’t used.
One caution on Wikipedia links: a site can collect links for reasons that have nothing to do with credibility (for example, a podcast linked from every guest’s biography). That is one reason the add-on is capped and only applies to members of vetted lists.
When a site is on more than one selected Reliable list, it starts from the lowest of their baselines.
5. The unreliable-list override
If an unrated site is on any selected Unreliable list, it gets the lowest baseline among all its selected lists, and no add-ons. That holds even if it is also on a vetted press list. 405 of the 409 sites scored this way appear only on Unreliable lists. For a check, we scored the 8 rated sites that sit on both a vetted list and an Unreliable list as if they were unrated: the override gave them an average of 0.18, against an actual average rating of 0.34, and none was scored more than 0.1 above its rating.
6. How the numbers were fitted
Three choices shape every number above:
- Erring low, made precise. We fit the penalty and add-ons with quantile regression at the 25th percentile. That treats overshooting a site’s rating as three times as costly as undershooting it by the same amount.
- Making rated sites stand in for unrated ones. Rated list members are much better known than unrated ones (median 137 vs. 5 Wikipedia links when we checked). So we modelled which members get rated (from Wikipedia links, Google News, domain age, Wayback Machine captures, traffic rank, and number of lists; the model separates rated from unrated well, AUC 0.88) and gave extra weight to rated sites that look like unrated ones (inverse-probability weighting). After weighting, the fit rests on the equivalent of about 420 sites, so we report intervals throughout.
- Out-of-sample testing. Every accuracy figure below comes from 5-fold cross-validation, repeated with 10 different random splits: list averages and fitted values are recalculated without the sites being scored.
With the current ratings, refitting the add-ons from scratch did not improve out-of-sample accuracy. MBFC gives most small newspapers exactly 0.86, and the refit chased that step. So we refit only the penalty weight and kept the add-on values, which sit inside the new intervals.
7. Does it work?
The big assumption is that unrated list members resemble rated ones. We can’t test that directly, since they’re unrated, but we can test its weakest point: do the least-known rated members still rate high?
They do. The least-known tenth averages 0.77, against about 0.80–0.82 for the best-known, and Low ratings stay near zero throughout. The spread does widen for less-known sites, which is exactly why the safety discount exists.
Scoring rated members as if they were unrated, the formula overshoots their real rating by more than 0.1 for 10% of them, and for 6% of the little-known ones. On average it lands 0.07 below the real rating (0.09 below for little-known sites). Using a list’s plain average would overshoot 21% and 36% of the time. We also tested a full regression model: about as accurate overall, but it overshot little-known sites twice as often, so we kept the simpler, more cautious formula.
A human spot-check. A WikiSignals editor visited 100 randomly chosen unrated list members (spread evenly across score levels), without seeing their scores, and rated each one. 69% of our estimates fell within 0.1 of the editor’s rating, and none fell more than 0.1 below it. The estimates averaged 0.08 above the editor’s ratings, but the editor scored a “normal site, nothing special” at 0.6, while MBFC typically rates such outlets 0.86, so most of that gap reflects scale, not over-scoring. The spot-check also turned up our biggest blind spot: 27 of the 100 sites were offline, redirected, for sale, or broken, and 3 had been taken over by unrelated (and unsavory) operators. A list membership doesn’t expire when a domain does.
| Measure | Value | What it means |
|---|---|---|
| Rated High or Very High | 90.4% of members vs 47.2% of all rated domains | Members cluster at the top |
| Rated Low or Very Low | 0.6% of members vs 33.2% of all rated domains | Members almost never land at the bottom |
| Relative risk (bottom half) | 20.5 | Non-members are this many times as likely to rate in the bottom half |
| Odds ratio (bottom half) | 40.9 | The same comparison, as odds |
| Cramér’s V | 0.38 | Overall association, 0 (none) to 1 (perfect); diluted because non-membership says little |
8. What we can’t rule out
- Unrated sites may differ from rated ones. Our weighting corrects for the differences we can measure (links, indexing, age, traffic), not for ones we can’t. Most unrated list members are less known than nearly every rated member, so their scores rest partly on extrapolation.
- Some low-quality sites will slip through. Of the 12 rated members of our vetted lists that rate Low or Very Low, only 5 are also on an Unreliable list. Applying the same rate to unrated members, we expect roughly 25 to 81 of the 6,951 list-scored sites (0.4% to 1.2%) to be low-quality sites with a middling score. As Unreliable lists grow, that number should shrink.
- Dead, moved, and hijacked domains keep their scores. That is deliberate: links to them are still out there, and the score describes the outlet the domain belonged to. It does mean a score can describe a site that no longer exists, or a domain now run by someone else. Better tracking of site status is on our to-do list.
- The experts can be wrong, or out of date. MBFC is updated regularly; DQ, used for sites MBFC hasn’t rated, was last updated in 2023.
- Lists change. Members join and leave, and directories go stale (one of our lists survives only as a 2021 web archive). We re-check lists annually.
- Scores are about domains, not articles. A credible outlet can publish a bad story, and vice versa.
9. Keeping it current
- Monthly: after each MBFC update, list baselines are recalculated and every score is refreshed.
- When lists are added or updated: the list is re-tested against the thresholds above and, if it qualifies, gets a new baseline.
- Annually: the penalty weight is refit and the method version is updated.
Each score records its method version (currently 2026.11-B2) alongside its basis.
10. Using the scores
Scores are available through the WikiSignals API (example: abqjournal.com) as ws_score, with ws_score_basis (rated, list, or unreliable) and ws_score_method. For research that needs professional ratings only, filter to rated; for broader coverage, include list and unreliable, and consider reporting results both ways. We do not recommend treating list-derived scores as ground truth for evaluating other lists or signals: that would be circular.
Glossary
- Rated: has an MBFC or Domain Quality rating.
- Vetted (Reliable) list: a press association or news index whose rated members pass the 3% test.
- Baseline: the starting score for a list’s unrated members.
- Upper bound (95%): the largest share of “wrong-side” members we can’t rule out, given how many are rated.
- Quantile regression (25th percentile): a fitting method that aims for a value that real ratings exceed about three times out of four.
- Cross-validation: testing on sites left out of the fitting.
Documentation collaboratively composed with Claude Science.