Inspector General

The Open Generalization Framework

Measuring the evidentiary credibility of the humanities.

Inspector General offers a systematic view of the relationship between research claims and evidence across leading humanities journals.

Explore the findings Open the repository

Introduction

One evidentiary standard across fields

This project is an extension of the book Can We Be Wrong? The Problem of Textual Evidence in a Time of Data (Cambridge 2020). It asks how we can make cultural and historical interpretation more credible and align humanistic methods with cross-disciplinary frameworks of evidentiary evaluation. Rather than consider two different evaluation frameworks for the humanities and the sciences — or the humanities and computational humanities — the book argues that both should be subject to the same criteria, which I call the Open Generalization Framework.

Framework

The Open Generalization Framework

The framework is built around three core principles:

Methods

From journal issues to comparative analysis

  1. 01

    Select

    Start with leading journals grouped into nine humanities disciplines and select each journal’s final 2025 issue.

  2. 02

    Interpret

    Uses GPT-5.6 to label the article's central claims, evidence, and transparency surrounding the evidence and its relationship to the claims made.

  3. 03

    Analyze

    Classify the reports and use a documented R workflow to compare journals, disciplines, evidence counts, and generalization scores.

Measures

Two measures anchor the comparison

Check out the full workflow for further measures of the types of research claims (classes such as concepts, behavior, technology, etc.) and scales of research claims (both spatial and temporal, e.g. decades, centuries, cities, nations, the world).

0–15

Open Generalization Score

The sum of three 0–5 assessments: evidence transparency, analytical transparency, and limitations transparency.

N

Evidence count

The non-duplicated count of explicit or identifiable evidence units used to support the central generalizable claim.

Selected results

Generalization and evidence across the corpus

The figures below are generated by the repository’s documented R workflow. Click any figure to inspect it at full resolution.

Dot-and-range chart of median Open Generalization Score by journal, colored by discipline.
Figure 1
Median Open Generalization Score by journal. Points show journal medians and horizontal bars show the interquartile range. Colors identify disciplines. Computational humanities journals occupy the upper end of the observed range.
Dot-and-range chart of median evidence count N for 41 journals on a pseudo-logarithmic scale.
Figure 2
Median evidence count by journal. A pseudo-log scale preserves journals with low or zero counts while making the long right tail visible. Points are medians. Bars mark the interquartile range.
Two-panel box-and-jitter chart comparing generalization scores and evidence counts across nine disciplines.
Figure 3
Score and N by discipline. Article-level distributions reveal both disciplinary differences and substantial within-field variation. Boxes mark medians and interquartile ranges. Points represent articles.
Scatterplot of article evidence count N and Open Generalization Score, colored by discipline, with a positive fitted relationship.
Figure 12
Generalization score and evidence count. Across 370 articles, evidence counts and scores have Spearman ρ = 0.34.

Open methods

Inspect the workflow, data structures, and R analysis.

The repository contains the article-harvesting workflow, structured diagnostic schema, classification step, discipline crosswalk, statistical models, and figure-generating R code.

Explore Inspector General on GitHub