1 Preface: Someone Has to Judge It

Let’s be honest with each other. You said yes to refereeing a paper, and the paper is fifty pages long, and three of those pages are a computation you would have to redo yourself to believe.

Or a committee has asked what you think of work outside your area. Or a student wants to build on a result and you’d like to know, before they spend a year on it, whether the result is actually there.

The labor is the same in all three cases and it is mostly invisible. Read the paper closely enough to find what’s wrong with it. Check that the cited lemma says what the paper uses it for. Work out whether the gap in Section 4 is a gap or just a compression. Establish what was already known, so you can tell what is new here. And then — the part that costs the most and shows the least — convince yourself that each thing you are about to call an error really is one, because you are going to put your name on it.

write-critical-guide does that labor and hands you the result. It is working material for a person who has to assess a paper: the paper read closely, its context established, everything questionable found and located and verified, with repairs where repairs exist.

What it does not do is decide. The words accept, reject, major revision appear nowhere in what it produces, as judgments about the paper. That restraint is the point rather than a limitation. The decision carries your name, and material that arrives at a verdict invites you to ratify a judgment you did not make. Findings, ordered by consequence, and nothing that presumes the decision.

Two properties are worth knowing before you run it. It records what checked out, not only what failed — if it verified thirty citations and nine were wrong, it says thirty, and it says what it did not check, because that is exactly where your own work begins. And it tries to refute every serious finding before writing it down, by sending it to independent skeptics whose job is to show the paper is right. A defect reported that isn’t there costs the authors weeks and costs you your credibility; a missed comma costs a reader three seconds. The errors are not symmetric and the process doesn’t treat them as though they were.

It runs unattended. Point it at a paper and it goes to completion without asking you anything.

Four chapters follow. Chapter 1 is what the genre is and the three rules that govern it. Chapter 2 is how it reads a paper — the sweeps, run in parallel, and the refutation pass they feed. Chapter 3 is the ledger every artifact is generated from, which is the piece of engineering that keeps the prose and the annotations from drifting apart. Chapter 4 covers long works reviewed in chunks, and how to correct a guide after it’s finished. The appendix has the command lines and the schema.

2 Findings, Never a Verdict

A critical guide is the material a judgment gets written from, not the judgment.

Hold that distinction and everything else in this manual follows from it. Lose it and the output becomes a referee report with no name on it, which is the one thing it must not be.

2.1 The sibling genre

The quickest way to see what belongs in a critical guide is to put it beside the reading guide that write-paper-guide produces. Same family, same close reading underneath, same voice — opposite job.

A reading guide is expository. It takes the paper as given and helps someone through it. Where the paper compresses, it expands, so the reader can keep moving.

A critical guide is evaluative. It asks whether the paper holds. Where the paper compresses, it stops and says so: what is missing, how much it matters, and whether the gap can be closed.

So the test for any sentence is which document it belongs in. If it explains the paper to someone who wants to understand it, that’s the reading guide. If it weighs the paper for someone who has to judge it, it belongs here.

2.2 Three rules

2.2.1 Findings, never a disposition

Accept, accept with revision, major revision, minor revision, reject: none of these appears as a judgment about the paper, anywhere in the package. Nor does anything that writes your report for you — no I recommend, no this reviewer finds.

What appears instead: what is wrong, how consequential it is, and what would repair it. Ordering the issues by consequence is as close to a verdict as it comes, and that ordering is the most useful thing the summary does.

This isn’t modesty about the mathematics. It follows from what the document is. The disposition is yours to reach and your name goes on it.

2.2.2 Try to refute a finding before writing it down

Every finding that would be graded major goes to three independent skeptics before it enters the ledger. Each gets the paper and the finding and is told to refute it — to find the argument the finding claims is missing, the hypothesis it claims is undischarged, the reading under which the notation is consistent. They are told to default to refuted when genuinely uncertain, and none of them is told what the others concluded.

Two or more refute it and the finding is dropped, or demoted to minor if a weaker true version survives. The drop is recorded with its refutation rather than silently discarded, so you can see what was considered and rejected.

Claims about the literature get the same treatment whatever their grade, because nothing else in the pipeline checks them and they are the most quotable sentences in the package. ``The first result in arbitrary dimension”; ``X proved only the surface case”; ``the technique originates with Y.” Being confidently wrong about priority in front of the author whose priority it is costs you more than a missed lemma would.

The effort spent checking scales to what being wrong would cost, not to how likely the error seems. A finding whose consequence is that an author adds a sentence gets one pass. A finding whose consequence is that you accuse someone of taking another person’s result without credit gets as many as it takes.

2.2.3 Record what checked out, not only what failed

A list of nine problems reads as the complete result of an examination. It usually isn’t.

So the package says what was verified and came out clean — a table of citations checked, what each is used for, which held. And it says explicitly what was not checked, with reasons. If the long computation in Section 6 wasn’t re-derived, that’s stated.

You cannot tell a clean bill of health from an unopened box unless someone tells you which it was. Where the coverage ends is where your own work starts.

2.3 Grades, and what they measure

Three of them, and they grade consequence rather than length.

A misprinted subscript that makes two displays contradict each other is major. A paragraph of clumsy prose is trivial. The item that takes three words to state can be the one that breaks the theorem.

Alongside the grade sits a qualifier where a judgment call needs recording — trivial (but should be reconciled) is a different instruction to an author than trivial.

2.4 What it is not

Not a referee report. It carries no signature and reaches no disposition.

Not a reading guide — it assesses and itemizes rather than orienting a newcomer.

Not a rewrite. Where the authors’ prose is defensible and merely not yours, it is left alone, and a preference is never reported as an error.

Not a survey. Context appears in service of assessing this paper.

3 How It Reads a Paper

Point it at a paper and walk away. This chapter is what happens while you’re gone.

3.1 Starting it

/write-critical-guide ~/refereeing/descent.pdf
/write-critical-guide arXiv:0000.00000
/write-critical-guide ~/refereeing/descent/

Given a folder, it looks for a single PDF or LaTeX source and uses that. Given nothing, it looks in the working directory. If there are several candidates and nothing distinguishes them, it says which it found and stops — that is the only thing that halts a run. Everything else has a default, and it takes the default and keeps going.

3.2 Which text is under examination

First it settles what the findings are about, and this matters more than it sounds.

Where a PDF exists, the PDF is the artifact. Source — the authors’ LaTeX, an arXiv tarball, a repository — is an aid: for exact quotation, for line numbers, for placing annotations, for corroborating what the compiled document shows.

The test is whether a thing appears in the compiled paper. An author’s marker that a macro renders, a cross-reference set as ??, a sentence that stops mid-clause, a statement with no proof: all of these print, so all are in scope. Commented-out source does not print, and is out of scope entirely — not reported, not reasoned from, not referred to.

That last rule is stricter than it first appears, and it’s right. A passage may be commented out because it was wrong, because it was superseded, or because it was never meant to survive, and none of that is knowable from outside. Findings drawn from working notes are findings about a person’s process rather than their work, and they’re unsound in a way that is hard to see afterwards. A statement whose proof is commented out is a statement with no proof — that is the finding, and it is a finding about the printed paper.

3.3 Three source situations

Where the annotations get inserted is a separate question, and there are three cases. Which one you’re in is recorded in the package README.

3.4 The reading pass

You cannot assess a paper you have not read, and the reading is where the findings come from. So the first real phase commissions a reading guide: write-paper-guide, run unattended, at proof depth, calibrated to a reader who is refereeing — an active researcher in an adjacent field, comfortable with the machinery, without the local folklore.

Alongside that guide it keeps a running ledger of friction. Every time the reading has to supply a step the paper leaves out, reconstruct an argument to follow it, guess which of two things a symbol means, or look up whether a cited result says what it’s used for, an entry goes in: where it happened, quoted verbatim; what the paper says; what had to be supplied; and whether the gap looks fillable.

That ledger is the reason for the reading. A guide written to proof depth has to touch every place the paper is thin, and writing down the friction as it happens is cheaper and more honest than trying to recall it afterwards.

If a guide already exists for this paper, it’s read and used rather than rebuilt.

3.5 Four sweeps, in parallel

Then independent agents sweep the paper, one per category, each writing its own file. None of them sees another’s output. Uncorrelated readings are the point; a shared draft would collapse them into one.

Those four suit most pure-mathematics papers. They’re a starting set, not a schema: a paper with substantial numerics or released code earns a computational sweep, a long computation may be worth re-deriving independently, and two sweeps that would return nearly the same thing get merged. The sweeps that run determine the categories in the ledger, not the other way round.

Every reported item must carry a verbatim quotation from the source, long enough to locate the passage uniquely. That quotation becomes the annotation anchor, and an item without one cannot be placed. This is why retyping from a rendered PDF is a mistake — the quotation has to match the file.

3.5.1 What the mathematical sweep produces besides findings

A dependency map: which results rest on which, and in particular which findings are independent defects and which are consequences of an earlier one.

A theorem whose proof fails only because a lemma it cites fails is one error, not two. Presenting it as two makes the paper look worse than it is, and a reader working from that list will grade the paper on a count that isn’t real.

3.5.2 What the reference sweep reports

Three things, all of them. The citations with problems. A table of the citations that came out clean, saying what each is used for. And an explicit list of what could not be verified, with the reason.

Anything central cited to a work that is unpublished, ``in preparation,” or otherwise unavailable to a reader gets flagged, because that’s a question you will have to take a position on whether or not you want to.

3.6 Refutation, then promotion

Every candidate major finding then goes through the adversarial pass described in Chapter 1: three skeptics, told to refute it, defaulting to refuted when uncertain.

What survives enters the ledger carrying its confidence basis, and the distinction is kept because the two claims are not the same:

Then the promotion check runs the other way: any finding graded minor that the dependency map shows a main theorem resting on is re-examined, and promoted if the dependency is real.

3.7 When repair has more than one route

Sometimes a confirmed major issue admits several plausible repairs — add a hypothesis here, or restrict the generality there, and the theorem survives either way in different amended forms. When that happens a separate dossier gets written: one agent per route, each saying what the hypothesis is, what it costs in generality, exactly which of the paper’s results it restores and in what form, and whether it is sharp or merely sufficient. A synthesis follows, recommending an order of preference.

That recommendation is about mathematics, not about publication, so it doesn’t touch the first rule.

When there’s one obvious fix, none of this fires — the fix goes in the issue’s own repair field and the dossier is skipped. Branches are never manufactured to fill it.

Many independent branch points is itself a finding: it says the repair is underdetermined, and the guide says so plainly instead of presenting a menu.

4 The Ledger and the Package

Everything converges on one file. issues.yaml holds every finding, and both the point-by-point issue list and the annotated copies of the paper are generated from it.

That direction is the whole design. Maintaining the prose and the annotation anchors side by side is what lets them drift, and a note pointing at the wrong passage is worse than no note at all. Here an anchor that no longer matches its source is a hard error rather than a silent mismatch.

4.1 Categories

Nothing in the tooling knows what a category is. They come from the ledger, and the tags, colours, output files, and annotated copies all follow from what you declare.

The default three fit most pure-mathematics papers, and they work because they split by what the author has to do rather than by subject matter:

An author works through those in different sittings and often in different moods, which is why the split earns its keep.

Depart from them when the paper warrants it. Substantial numerics or released code may deserve a computational category; a paper whose real problems are organizational may want exposition kept separate from typography; a survey may need attribution. Two constraints, and they’re the only ones. A category that would hold one or two items is not a category — fold it into a neighbour, since every category costs the reader another PDF to open. And categories partition: every issue belongs to exactly one, filed where the action lies, with a cross-reference from the other.

Colours are any xcolor name, used as a light fill and a darker border. Keep the hues far apart so a reader flipping between PDFs can tell them apart at a glance, and keep each category’s colour stable across revisions. After red and green and blue, reach for orange, violet, teal, brown. Yellow and lime are invisible at that tint.

4.2 Four attributes, kept independent

Collapsing any two of these loses information the author needs. In particular, a dependent item is still listed — the author needs to know which downstream statements are affected — but it is never presented as a separate problem.

4.3 Anchors

One rule decides whether the package builds at all:

Copy it out of the .tex, macros and all. Retyping from the rendered PDF produces an anchor that never matches, because what you see set on the page is not what’s in the file.

Not found: lengthen or correct it against the file. Matching several lines: lengthen it until unique, or ask for a particular occurrence when the repetition is real. An anchor on a line that ends in the middle of inline math is fine — the insertion point advances past any straddling math or display, which is what keeps a note from landing inside an equation and producing a Missing $ inserted error pointing at a line nobody wrote.

Check anchors before any prose gets written. Fixing an anchor is cheap; fixing it after four documents quote the passage is not.

python3 annotate_tex.py check -v

4.4 The generator

annotate_tex.py builds the generated half of the package from the ledger.

# anchors only; writes nothing
python3 annotate_tex.py check -v

# one annotated .tex per category
python3 annotate_tex.py annotate -o .

# the point-by-point Markdown issue list
python3 annotate_tex.py issues-md -o .

# compile each one, then verify the result
python3 annotate_tex.py build -o .

# all four, after clearing stale aux files
python3 annotate_tex.py all --clean-aux -o .

build runs pdflatex, bibtex, pdflatex, pdflatex and then checks the result rather than trusting the exit status: undefined references, LaTeX errors, the note count against the ledger, and whether the PDF is actually newer than its source. It exists because one pdflatex pass leaves every cross-reference reading ?? and make calls that a success.

Where a paper’s numbering is self-referential, run build twice and read what it reports. A fixed number of passes is not a guarantee.

4.5 Writing notes

Note bodies are Markdown, because the issue list is Markdown, and the generator translates them for the LaTeX path. Quotation marks become directional quotes, emphasis becomes \emph, backtick spans become \texttt, and the characters that would otherwise break LaTeX are escaped. Math between dollar signs passes through untouched, so a quotation containing mathematics still pairs correctly.

Do not hand-write LaTeX quoting in a note — it reaches the Markdown literally.

Two forms for the entries themselves. Bulk copy-editing goes in a table: thirty one-line usage corrections as thirty paragraphs is far worse to read and act on than one three-column table of location, what the paper writes, and what to write instead. Prose entries are reserved for items that need an argument. And when one item inside a table section carries real consequence — the garbled sentence that happens to be the one completing the proof — it gets lifted out of the table into a full entry, keeping its tag.

Adjacent one-line nits in the same paragraph can be bundled into a single annotation box rather than stacking three boxes each a line tall. They stay separate entries in the list.

4.6 What is written by hand

Three documents. The rest is generated.

The summary is the part you read first and lean on hardest, since it is where the paper is characterized rather than itemized. Four parts, in order: what the paper does, with the main theorem stated in full and the shape of the proof in as many steps as it actually has; the context, with the precise trade against the nearest prior work; the significance, with reasons, and set against it what qualifies the assessment — how much is assembly, what the hypotheses cost, what the result does not give; and the findings, ordered by consequence and cross-referenced to their tags, together with what checked out and what was never checked.

It ends there. No fifth section, no recommendation.

The repairs document covers issues with a clear route, where the route is short enough to state inline. It opens by saying what it does not claim — that every downstream calculation has been rechecked under each replacement hypothesis.

The README says what each file is, how to rebuild, and which of the three source cases you’re in.

4.7 The finished package

critical-guide/
  00-guide.pdf         summary + issue list
  01-summary.md        hand-written prose
  02-issues.md         generated, point by point
  03-repairs.md        repairs with a clear route
  repairs/             the multi-route dossier
  issues.yaml          the ledger
  annotated-<cat>.tex  one per category
  annotated-<cat>.pdf  compiled, colour-coded
  annotate_tex.py      copied in
  Makefile
  README.md
  guide-preamble.tex
  guide-meta.yaml

review-notes/ sits beside it and is not part of the handover. It is the audit trail — the sweeps, the refutations, the findings that were dropped and why — kept so that a finding you doubt can be traced back to how it was reached.

4.8 The gates

The build is verified, never assumed. Five things are checked before the package is handed over:

1. make exits zero from a clean tree. Not make guide, not the tool run by hand. It is the documented rebuild path and the one thing you will actually run.

2. Every annotated document reports ok from the generator’s own build check.

3. The guide PDF passes the same mechanical gate, not a lesser one. It is the document you actually read and the one that fails quietly — an undefined citation warns rather than errors, and unresolved citations can survive a visual spot-check.

4. The issue list’s tables break across pages with their headers repeating. Nothing checks this automatically; a page is rendered and looked at.

5. In Case B, every annotated passage was confirmed present in the submitted PDF.

Then the hand-back: where the package is, what it contains, the count of findings by category and grade, what the refutation pass dropped, and what was not verified. Every quantity in that report was measured from the files — not carried over from an earlier run’s figures.

5 Long Work, and Corrections

Two situations the ordinary flow doesn’t cover: a work too long to examine in one pass, and a guide that turns out to need fixing after it’s finished.

5.1 When one pass isn’t enough

The question to ask before anything else — because it changes everything after — is whether a reader would take this work in one sitting and form one judgment.

A paper, however long, yes. A book, a thesis, a memoir in parts, no. Those get examined a chunk at a time, in separate sessions, with you reading each chunk’s output before declaring the next. Forty pages is not chunked. Three hundred is. In between, the sitting-and-judgment test decides.

The pause between chunks is the point. It keeps you in control of a long examination rather than handing you one unreadable delivery at the end.

Chunking also settles a tooling question. The generator annotates one source file, so a main.tex full of \include lines would fail on every anchor, since the anchors live in the chapter files. Under chunking each chunk’s source is its chapter file and the problem doesn’t arise.

5.1.1 Reconnaissance first

Before any chapter is examined there’s a chunk 0: front matter, table of contents, introduction, bibliography, and a skim for structure. No deep reading.

It produces the carry file with the work’s structure and the decisions filled in, and nothing else. It’s cheap, and it stops chunk 1 from doing double duty as both a chapter examination and the establishment of the frame that every later chunk inherits. Categories, colours, reader profile, and voice are fixed here and do not drift afterwards.

5.1.2 The carry file

review-state.yaml lives beside the work, and it is the whole design. Nothing survives between sessions except what is written into it — a fresh agent starting chunk 5 knows only what that file says.

Four of those carry most of the weight.

inherited is what earns the design. Before anything is graded, the chunk checks whether the result it’s examining rests on an open defect from an earlier chunk. If it does, the finding is provisional rather than independent, and says so — the same distinction the ledger’s dependency field makes within a chunk, carried across chunks.

decisions is frozen. If a later chunk finds the category scheme inadequate it says so in the hand-back and lets you decide; changing it mid-examination makes the packages mutually inconsistent.

precedents is what keeps ten chapters examined by ten agents from disagreeing about what major means. Not every grade goes in it — only the ones where a reasonable reader could have gone the other way.

revisions is appended to, never acted on. A chunk-5 agent that discovers a chapter-1 problem records it and stops. Whether to re-run chapter 1 is your call, and re-running is expensive.

5.1.3 What changes per chunk

The pipeline runs as written, scoped to the chunk, with four differences.

The carry file is read in full, before anything else.

The context sweep does not run. Where the work sits, what’s new, how it compares to prior work: none of that is knowable from chapter 2, and a chunk that attempts it produces confident nonsense. It belongs to the final synthesis.

The reference sweep consults the cache first. A book cites the same works many times; only what’s new gets verified, and what’s checked gets added.

Tags are namespaced by chunk — M1-7 is the seventh mathematical issue in chunk 1. No collisions, no high-water mark to track, and a tag says where it lives.

At the end the carry file is updated and the run stops. It does not begin the next chunk.

5.1.4 The synthesis

After the last chunk, one pass over the accumulated state produces what no chunk could: the summary with context and significance, the global assessment, the findings that cross chapters, and a consolidated issue list.

5.2 Correcting a finished guide

A finished guide is not frozen. A claim turns out to be overstated, someone disputes a finding, a cited preprint appears in print, the authors post a new version and the anchors stop matching.

When that happens, do not start a new run. The guide is the product, and the job is to correct it in place so that what you’re holding is right. Three phases, and the order is what keeps a correction from becoming a rewrite.

5.2.1 Verify first, changing nothing

Enumerate the claims in scope, quote each verbatim with its location, and attack them one at a time — opening the cited work rather than relying on its abstract, on what the paper under examination says about it, or on the field’s reputation of it. A verdict and its evidence are committed for every claim before any prose is touched.

Negative and priority claims take a different standard. ``No prior occurrence,” ``the first in arbitrary dimension,” ``only the surface case was known”: none of these can be re-verified by failing to find a counterexample again. What search would surface one is stated, run, and reported. A thin original basis means the claim gets weakened, not re-affirmed.

5.2.2 Then apply, narrowly

Each failed claim is corrected in the summary, in the guide’s existing voice — and corrected in the ledger too, wherever it also appears there. A literature claim frequently lives in both, and fixing only the prose leaves the guide contradicting its own annotated PDFs.

An honest hedge beats a deletion. A claim narrowed to what is actually true is more useful than a gap where it used to be. Nothing that wasn’t audited gets touched, and no recommendation gets added.

5.2.3 Then regenerate, and report the changelog

python3 annotate_tex.py all --clean-aux

Then the guide PDF. A correction that leaves the ledger and the built PDFs out of step has not been made — it has been half made, which is worse, because the package now disagrees with itself.

Finally: what changed, from what to what, on what evidence. The detail stays in the audit trail.

The same three phases apply to any correction, not only to a literature claim — a withdrawn finding, a regraded issue, a repaired anchor after a new version appears.

6 Quick Reference

6.1 Invocation

/write-critical-guide <pdf path>
/write-critical-guide arXiv:<id>
/write-critical-guide <folder with the paper>

It runs unattended and takes every default. The only thing that halts it is being unable to identify the paper.

6.2 The generator

# anchors only; writes nothing
python3 annotate_tex.py check -v

# annotated .tex, one per category
python3 annotate_tex.py annotate -o .

# the Markdown issue list
python3 annotate_tex.py issues-md -o .

# compile, then verify the result
python3 annotate_tex.py build -o .

# all four, aux files cleared first
python3 annotate_tex.py all --clean-aux -o .

# the same gate, for 00-guide.pdf
python3 check_guide.py

6.3 Package layout

6.4 Ledger fields

Required per issue: id, category, anchor, note. Everything else is optional, and grade defaults to minor.

Top-level blocks of the ledger:

paper            the work; what locations mean
categories       tag, title, colour, output
issues           the findings themselves
verified_clean   what came out clean
not_verified     what was not checked, and why
scope_caveat     what is not claimed certified
issue_list_name  the issue list's filename
caption_symbols  ASCII spellings for captions

6.5 Where fields appear

The annotated source omits the quotation on purpose: the note sits at the passage, so quoting it back is noise. The issue list includes it, since it’s read away from the paper.

6.6 Source cases

6.7 LaTeX hazards, already handled

Recorded so that placing a note by hand, or debugging a build, doesn’t mean rediscovering them.

6.8 Building the guide PDF

Bare LaTeX macros quoted from the paper must sit in backticks in the Markdown, or they reach an engine that has never heard of them. This is the commonest way the build breaks, because quoting the paper’s macros is exactly what an issue list does.

Use pdflatex, not lualatex — an order of magnitude faster here, and it handles every character a critical guide actually uses. Long tables need longtable with repeating headers and ragged-right cells, or they are silently truncated. And the title lives in the metadata file; repeating it as a heading prints it twice.

6.9 Chunked examinations

6.10 Words the house does not use

Disposition vocabulary as a judgment on this paper: accept, reject, revision, publishable. Anything that writes your report for you: I recommend, this reviewer finds. Load-bearing — say what rests on what. The key definition, the crucial step — there is rarely exactly one; name the thing and say what it does. Negative framing where positive carries the same content. Executive-summary register: headline, takeaway, at a glance. Reading advice to the author. And the usual filler: utilize, leverage, seamless, robust, powerful, elegant, simple, easy, straightforward, clean.