Data Reliability in Chess Reporting: Ratings, Engine Evaluations and the Challenge of Fact-Checking

The biggest problem in modern chess journalism is not a shortage of information. It is that the information available is frequently presented without the parameters that make it meaningful. Within hours of any major tournament, dozens of reports appear, and very few of them state which rating list they are quoting, what date it carries, or which format category it belongs to. In a game like chess this gap is especially serious, because almost every claim can be attached to a number: a rating, an engine evaluation, or the success rate of an opening move. The presence of a number does not make a claim true. What makes it true is the full statistical and institutional context behind it. This report is not a review of any single tournament result. It is focused on the four categories into which nearly every chess story falls: ratings and lists, engine evaluations, opening statistics, and rules and formats. All four share one feature: each requires a parameter whose mention is routinely dropped. When that parameter disappears, even a correct number produces a wrong conclusion. The first category is the Elo rating. The founding principle of the system is that it measures relative difference between players, not absolute strength. The practical consequence is that 2700 today does not represent the same level it did two decades ago, and comparing numbers across different time controls is methodologically invalid. Classical, rapid and blitz maintain separate lists, and a single player can hold notably different figures in each. The most common error in reporting is exactly this: a figure from one format is presented in the context of another. The K-factor matters as well. It determines how much a single game's result will move a rating, and it varies according to a player's age, rating bracket and number of games previously played. An experienced player's win and a newcomer's win are therefore not identical in rating arithmetic. A sentence such as 'the win caused a large jump in their rating' is only meaningful if it states which list, which date, and which K-factor applied. Without those three details it becomes a misleading simplification. Ratings also carry a long-running debate about inflation and deflation. When comparing figures across eras, both mathematicians and historians caution that the size of the rating pool, the entry rate and the number of listed players keep changing. A claim that today's top players are stronger than earlier ones cannot be made without era adjustment. A responsible report should make clear which list, which month and which category it is citing. The second category is engine evaluation. Engines such as Stockfish and Leela Chess Zero have become the standard of analysis, but a number they produce — say, half a point ahead — is not an absolute truth. It is a conclusion reached at a particular search depth, on a particular network version, on particular hardware. Change the depth and the evaluation may change; change the network version and it may change again; two different engines may disagree on the same position. This is not an anomaly. It is normal. The journalistic implication is direct: the sentence 'the engine showed a clear advantage' is incomplete on its own. It needs at least four things to be complete — the engine's name, its version, the search depth or time limit, and the hardware. Without these, no reader can re-check the claim, and a claim that cannot be re-checked is not journalism; it is only a tone of authority. There is a further subtlety: evaluation and winning probability are not the same thing. An engine's number describes the quality of a position, but converting it into a win depends on human factors — time pressure, fatigue, morale and the opponent's response. Many reports treat the engine's number as the final outcome, and then describe the result as a shock when it reverses. It was not a shock. The report's model was simply incomplete. The third category is opening statistics. Opening browsers offered by online platforms and databases show which moves were played how often, and how those games ended. The central confusion here is the assumption that the most-played moves and the best moves are the same thing. They are not. A database records what people played, not what should have been played. Three methodological problems attach to this. First, sample size: a percentage based on a few dozen games in a rare line is statistical noise, not a finding. Second, mixing player strength: if a database includes players of every level, the win rate of a move played mostly by weaker players distorts its true quality. Third, selection bias: a database of top-level games and a database of all games will sometimes yield opposite conclusions about the same line. Any report built on opening statistics must therefore include three things: the sample size, the time window and the rating filter. Presenting a percentage and omitting the rest is the easiest way to mislead a reader. The fourth category is rules and formats. This is the least time-sensitive category, but the one where version numbers matter most. The FIDE handbook is revised periodically, and amendments are often issued as appendices. Time controls, restrictions on electronic devices, and tie-break regulations in team events all belong here, and a report written against an older version can be demonstrably wrong today. The practical rule is simple: any rules-based claim should state which version it relies on. Writing 'according to the rules' is not enough, because the rules are not static. Team-event tie-breaks are the clearest illustration — several different criteria apply in a fixed order, and changing the order can change the winner. The fifth point is editorial rather than technical: the weighting of sources. A player's remark immediately after a game, the same player's analysis days later, a coach's comment, a commentator's assessment and a platform's promotional statement do not carry equal weight. When an engine's conclusion and a player's intuition conflict, a good report presents both and lets the reader decide; a weak report selects one and hides the other. The remedy for all of this is not clever. It is the old method: build a fact sheet. For every claim, record three things — source, date and confidence level. Then sort them into four boxes: facts, opinions, rules and data. For facts, official announcements come first, platform data second, social media last. For opinions, the speaker's identity is mandatory. For rules, the version number is mandatory. For data, the parameters are mandatory. The chess world is exceptionally active at the moment — players such as Magnus Carlsen, Fabiano Caruana, Hikaru Nakamura, Ian Nepomniachtchi, Ding Liren, Gukesh Dommaraju, Rameshbabu Praggnanandhaa and Alireza Firouzja compete in one format or another nearly every week, and every result becomes a data point immediately. In that environment, reader trust accrues to the reports that explain where their numbers came from. A report that shows only the number and not its context may be fast, but it is not reliable. In chess, the right move is not the one that appears fastest, but the one chosen with the whole position in view. The same rule applies to journalism.