Six errors that were the same error

The Integrative Corpus gathers eighteen sources into 3,788,294 forms from 4,735 doculects across 225 macrosystems, with row-level provenance and a computed layer of 3,091,596 consonantal skeletons that no source supplies.

The base layer is aggregation: every form comes from somewhere else, and those somewheres are good. What is contributed is not “more data”. It is a union auditable back to the source of every figure, a phonological representation computed with the same rules for all 225 macrosystems, and the gaps declared with their cost measured.

But the most transferable part of the paper is something else, and it takes up its longest section.

The problem that does not look like a problem

A corpus integrating eighteen sources accumulates errors that do not present as errors. There is no exception, no missing row, no failing query.

There is a figure slightly different from what it should be, and nothing flags it.

The paper publishes, with the corpus, the apparatus built to make that class of failure visible: 60 invariants that fail rather than warn — plus the five that only warn, enumerated one by one — a schema contract declaring what each column means and which one it is confused with, and meta-checks whose object is not the data but the reach of the apparatus itself.

And it publishes its justification: six real errors from this project. Five of the six passed every check that existed at the time.

The six

1. The null that measured our own draw. Three papers published enrichment figures with confidence intervals. The null was estimated by shuffling eighty thousand times, and the interval was presented as though it measured the uncertainty of the data. It measured the uncertainty of the draw. It was found by removing six Turkic forms and watching the null move by 14 % — impossible if the interval described the population. Replaced by the closed form, the figures changed: 225× → 218×, 77× → 78×, 39× → 37×.

What caught it: nothing. The interval was wide, and a wide interval looks like prudence.

2. The appendix spliced three times, with 8/8 checks green. Six papers carried their appendix pasted in three times. All six passed the eight checks. The check verified that the appendix existed. It did. Three times.

The difference between ≥1 and == 1 is the whole error.

3. Five of six scripts giving a different result on every run. The query loading a family’s data had no ORDER BY. PostgreSQL promises no order without it, and gave none. Downstream, Counter.most_common(1) broke ties by insertion order. Published result: the substitution table read b→p at 0.5× or b→n at 1.9× depending on the run.

What caught it: nothing. The figures were plausible both times. Nobody runs the same script twice to compare.

4. Two columns with the same name and different meanings. form.source_id is the source of the form; lect.source_id, the source of the language record. The code loaded the second into a field called source. Published consequence: a whole section of the Indo-European paper said one source loses 65.6 % of its forms when it loses 100 %.

The error was born from a name.

5. The licence invariant reporting zero with 2,140 rows inside. When non-redistributable sources were withdrawn, the sweep deleted 5,971 rows and the invariant reported zero in quarantine. There were 2,140: one table stored provenance in a column named source, with no suffix, and neither the sweep nor the invariant looked at it.

What caught it was another check — the cognate-network meta-check, which complained about a network with no declared invariant.

6. The meta-check reporting 120 % coverage. I computed the denominator myself, duplicating the analyser’s regular expressions rather than asking it for its own. The two versions diverged and the ratio went above one.

The lesson, in four words: ask the parser, don’t reimplement it.

The common shape, which is the thesis

All six are the same error.

A check that covered less than its name promised, and reported conformity exactly when there was a failure.

The appendix check promised “the appendix is fine” and verified that it existed. The licence invariant promised “nothing is quarantined” and looked at one of the two columns where quarantine can live. The meta-check promised “coverage” and divided by a number it invented.

In all three cases, green was more dangerous than no check at all, because a green check closes the question.

Five rules, for anyone publishing data

  1. Write down what each check does NOT cover, next to what it does. If it cannot be written, the check is not understood.
  2. Make them fail, not warn. A warning in a hundred-line log does not exist.
  3. Discover from the catalogue, not from a hand-maintained list.
  4. Prove they fire. Introduce the failure, check that the check sees it, revert. A check that has never fired is not tested — it has merely not failed yet, which is not the same thing.
  5. Measure the apparatus’s own coverage, and make that fail too.

A note on coverage

The paper gives coverage by layer rather than as a single figure, because no single figure exists: two consumers of the same corpus — a query layer over a word’s history, and a catalogue of form–meaning correspondences — need different layers and arrive at different figures, both correct.

This section is not a confession. It is the only material that demonstrates the apparatus is good for anything: an invariant that has never fired is not tested, and an apparatus described without the failures that produced it is a list of good intentions.

The paper: Integrative Corpus (PDF) · Versión en español

The query layer: Meulemans — named in homage to C. S. Meulemans, who founded endolinguistics together with J. Á. Elias.