agentleFS
jpoindexterjpoindexter/tufte-skills/skills/tufte-annotation-as-thinking/SKILL.md

tufte-annotation-as-thinking

Skill1 starsChanged 4 months agoMITView on GitHub

Use when an image, chart, scan, screenshot, diagram, or dashboard is about to ship without explanation on it; when writing captions, callouts, keys, figure legends, or marginalia; when reviewers cannot tell what they are looking at; when a display must carry a dispute, rebuttal, or provenance beside the evidence; or when designing two-reader formats, commentary layers, or annotated reference editions.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do / Don't: never assume the words explain the picture.

    line 112
  2. Distinctive:

    Design consequence: if a display is going to accumulate commentary over time, design the *positions* first.

    line 150

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Annotation as Thinking

Overview

Chapter 5 of Seeing With Fresh Eyes (printed pp. 121–130) makes a claim stronger than "label your figures." Tufte's chapter heading states it as a question — whether thinking simply is annotating the world — and pairs it with a warning: annotations are hard to unsee (p. 121). Annotation is not a service layer added after the evidence is finished. It is the act of reasoning about evidence, performed in the same visual field as the evidence, on content-responsive local grids rather than on a page-wide production grid.

The chapter's operating principle is stated plainly:

"Good annotation is like a knowledgeable expert/teacher at the viewer's side pointing and saying 'Now see how this works with that, how this might explain that . . .'" — Tufte, Seeing With Fresh Eyes, p. 121

The corresponding failure is equally plain: evidence that carries no annotation is not evidence, it is decoration. And the warning matters as much as the principle — once a reading has been pinned onto an image, viewers cannot un-see it. Annotation directs attention permanently, so it must be used with the same care as any other irreversible act.

Related skills: tufte-typography-for-data (annotations-on-annotations, letter-coded captions, direct labels), tufte-micro-macro-readings (the Talmud page as a micro/macro object), tufte-mapped-pictures, tufte-layering-and-separation.


§1. What Annotation Actually Does — Four Simultaneous Jobs

A good annotation is doing more than naming a part. Tufte's formulation on p. 121 makes annotation do two things at once: it calls out and explains the information, and it teaches the viewer how to read the display. Unpacked, that is four jobs.

JobWhat it producesTest
IdentifyThe thing has a name at its locationCan a stranger name any element without a round trip to a key?
ExplainThe mechanism, cause, or consequence is statedDoes any annotation contain a verb?
Instruct the readingThe viewer learns the display's own grammarDoes the display say how to read itself, or assume fluency?
Record provenanceSource, date, scale, units, and who made itCould a skeptic trace the display to primary measurement?

Do / Don't:

PracticeVerdictWhy
Caption below the frame naming the figureWeakIdentifies only; does not explain and does not teach the reading
Words placed at the feature they describe, containing a verbStrongIdentify + explain in one act, no decode round trip
A key that maps A, B, C to phrases underneathContent-hostileForces the exact decode round trip a legend forces (see tufte-typography-for-data; SWFE p. 55)
Second-color annotations correcting first-color annotations in placeStrongThe display holds a dispute rather than a conclusion

§2. Annotation Across Notational Systems — Kircher, 1650

In Athanasius Kircher's Musurgia universalis (1650), five birdsongs annotate five bird images (p. 121). The songs run as musical notation on staves that curve and blow across the plate; the birds are engraved illustrations; the sounds themselves are written as non-lexical vocables — syllables with no word meaning, the gucu gucu of a cuckoo. Tufte's point is that images, staves, Latin labels, and nonsense syllables all play together at once, the way an opera score does. The parrot says hello in ancient Greek.

The transferable move: annotation is not restricted to the annotator's home notation. Any mode that carries the content — words, numbers, musical notation, sketches, arrows, color, a photograph — is admissible in the same field, and the mix is a feature rather than a mess.

Named failure mode — Mode Segregation. Confining explanation to the notation the tool defaults to (prose in the caption slot, numbers in a table, image in a frame), so no single field holds the reasoning. Fix: put whatever it takes onto the evidence itself.


§3. Annotating a Work of Art in a Newspaper — Reinhardt, 1947

Ad Reinhardt's "How to look at a mural" (P.M. daily newspaper, Brooklyn, January 5, 1947) is reproduced across pp. 122–123. Reinhardt reproduces Picasso's Guernica — a 12 × 26 foot mural that represented the Spanish Loyalist Government at the 1937 Paris World's Fair, later toured London and America's seven largest cities and was seen by over a million people — and pins about fifteen labeled readings directly onto its parts: the bull, the dying horse, the mother with the lifeless child, the electric-bulb sun as an all-seeing eye, the severed hand with its crossed life-lines, the decapitated statue, the clenched fist and broken sword.

Three design decisions worth stealing:

  • Cut-outs beside the labels. Each annotation sits next to a clipped fragment of the mural it describes, so the reader never has to hunt the full painting for the referent.
  • The whole and the parts on one spread. The complete mural runs as a band across the layout, with the fragment-plus-label pairs above and below it — a micro/macro reading in one eyespan.
  • The annotator declares his own limits. Reinhardt states up front that pointing out symbolic meanings will not explain the picture's art-meanings, and offers the labels for what they are worth. An annotation that names its own scope is more credible, not less.

§4. Macro-Annotations on Micro-Data — the ICU Billing Record

The strongest worked example in the chapter is Tufte's own (p. 124, revised from Envisioning Information, 1990, p. 56, redrawn from David Hellerstein, "The Slow, Costly Death of Mrs. K——," Harper's 268, March 1984, 84–89). A machine-printed hospital bill — dense columns of dates, procedure codes, and dollar amounts — runs down the center of the page. Flanking it in the left and right margins are prose annotations that turn the printout into a 26-day narrative of a life and death in an intensive care unit.

LayerContentWhere it sits
SequenceDates and times, in the bill's own orderCenter, machine-set
Accounting dataProcedure codes, charges, running totalsCenter, machine-set
CommentaryClinical meaning, comparative costs, the patient's deteriorationLeft and right margins, prose
Linking marksRed asterisks tying a paragraph to a specific line itemBetween the layers

Tufte's own description of the effect is that the design stays transparent to the disturbing data as a layered polyphony of voices — sequence, accounting data, and commentary weaving together to trace out days, hours, minutes, money. The annotations import macro-data the bill itself cannot show: the California ICU base rate against Mississippi's, the daily rental cost of a respirator against the roughly $15,000 price of buying the machine outright, what independent labs charge for a blood-chemistry panel against what hospitals charge, the clinical laboratory as a profit center, ICU psychosis and vest restraints. A single instruction makes the whole thing legible thirty-eight years later: the costs were incurred in 1982, so multiply by 7.5 to express them in 2020 dollars.

Do / Don't — annotating a machine-generated record:

PracticeVerdictWhy
Re-typeset the record to make it "clean"WrongThe record's own ugliness and density are part of the evidence
Leave the raw record and annotate its marginsRightThe reader sees both the artifact and its meaning, and can check one against the other
Summarize the record in prose and drop itWrongThe reader loses every micro-detail and must trust the summarizer
Attach a unit/inflation instruction to dated moneyRightWithout it, every figure silently decays into nonsense

§5. Two-Reader Formats — the Annotated Pride and Prejudice

In the annotated Austen (p. 125), left-hand pages carry the novel's text and the facing right-hand pages carry the notes, with the cover and first right-hand page introducing the format before the reader needs it. Tufte's assessment is that the layout serves the casual reader and the avid reader simultaneously — the casual reader can run down the left pages uninterrupted, while the avid reader reads across the spread.

Generalizable rule: when two audiences need different depths of the same content, give each a spatial channel rather than making one audience wade through the other's material. Footnote apparatus, API docs with an "advanced" tier, and analytical reports with a technical appendix all fail when the depths are interleaved in one column and succeed when they are adjacent in space.

The counter-example is the display calibrated to its least informed viewer, which throws away most of its value for everyone else (compare the National Weather Service data paragraph, p. 60, in tufte-typography-for-data).


§6. Reciprocal Annotation — Galileo's Sidereus Nuncius

Galileo's 1610 report of Jupiter's moons is a 21-page narrative in which nightly observations record the four moons' positions over time (p. 125). Tufte's summary of the mechanism is the point of the whole chapter: words annotate the data graphics, and the data graphics annotate the words. Neither is the subordinate layer.

Tufte then remodels the original, rearranging the first eleven days of Galileo's January 1610 report into a data matrix. Each line becomes a time series of three parts — date, image, words — and the eleven lines stack into a list. The result is 3-space data moving in time, annotated and stacked, 170 years before Playfair.

Original formRemodeled formWhat the remodel buys
Prose narrative with inline star diagramsOne line per night: date · star image · sentenceNights become directly comparable up and down
Observations separated by paragraphs of textObservations aligned on a common axisThe moons' motion becomes visible as a pattern, not a claim
Cloudy nights buried in proseCloudy nights present as a labeled gap in the stackMissing data stays visible instead of vanishing

Do / Don't: never assume the words explain the picture. Ask which one is doing the explaining at each point, and let it swap.


§7. Evidence as Substrate — Leonardo, Avicenna

Tufte's spread on Leonardo's anatomical hand studies (pp. 126–127) uses the drawing sheet itself as the grid. Leonardo's own mirror-written notes are translated and placed onto local grids around and between the drawings, in two visually distinct treatments — the translated Leonardo text in one color and editorial descriptions of each figure ("deep dissection of the palm of the hand," "action of the lumbrical and interosseous muscles") in another. Nothing is squared off; each block sits where its figure sits.

The facing argument (p. 128) is Avicenna's encyclopedia (Ibn Sīnā, 1024 CE), whose Arabic commentary was written by many different hands on many different local grids — annotation running diagonally, in the margins, between lines, wherever the page had room. Tufte's heading for it is that annotated annotations turn up anywhere and everywhere.

The principle: a sufficiently dense and relevant evidence layer becomes the coherent substrate for everything laid over it. You do not need to build a grid; you need to annotate the data layer you already have. (Same move as Menuhin's pencil markings on a printed Bach score, p. 27 — see tufte-visual-thinking §3.)

Nabokov's Metamorphosis — the chapter's own epigraph image (p. 120)

The plate opening the annotations chapter is Nabokov's teaching copy of Kafka's The Metamorphosis (Lloyd's 1946 English translation, NYPL Berg Collection). It does three things at once, and each is transferable:

  • Annotation as argument, not gloss. Pencil beetles run across the top of the opening page with a marginal note constraining the creature to no more than three feet long; interlinear strikeouts then argue with the translation against Kafka's German ("uneasy" struck for "troubled," "gigantic" for "monstrous"). This is not commentary on a text — it is a competing edition drafted on top of one.
  • The substrate rule stated outright. Tufte's caption pairs it with Menuhin: Kafka's text became the substrate of Nabokov's mark-up as Bach's score became the substrate of Menuhin's. A substrate earns the role by being dense, fixed, and authoritative enough to argue with — annotate the best available original, never a summary of it.
  • Two channels on one surface. The drawings answer what the prose cannot (what the insect looks like, and how big); the strikeouts answer what the drawings cannot (what the words actually say). Neither decorates the other, and both work because they sit on the same page at the point of need.

Placement note: p. 120 belongs to Ch. 5, not Ch. 4 — Tufte's own visual index (p. 170) and permissions (p. 173) both file it under "Annotations." It is the chapter's epigraph image, not a stray example from the data chapter.


§8. Annotation as an Accumulating Institution — the Talmud Page

The chapter's culminating object (p. 129) is a single Talmud page carrying 18 separate rabbinical mark-up traditions accumulated over 1,000 years. Editions run about 6,200 pages with more than 100,000 annotations. The central Mishnaic text is annotated by interpretations, which are themselves annotated by further considerations.

Layers visible on one page include the Mishnaic text; the Gemara (200–600 CE); the commentary of Rashi (1040–1105 CE); comments of Tosafot; comments of Rabbi Nissim Gaon (d. 1062 CE); a textual emendation by Rabbi Joel Sirkes (1561–1640); notes of Rabbi Akiba Eger (1761–1837); references to medieval codes of Jewish law; cross-references to other Talmud passages; a key to biblical quotations; and catchwords giving the first word of the next page.

Property of the Talmud pageWhy it worksModern analogue that fails
Every layer keeps its own typographic identityThe reader always knows which voice is speakingThreaded comments flattened into one visual style
All layers visible in one eyespanComparison and cross-reference cost nothingCollapsed/expandable annotations requiring clicks
Layers accumulated over centuries without redesignThe format absorbed new voices without breakingFormats that must be rebuilt whenever a layer is added
Position encodes relationshipInner margin, outer margin, and footer mean different thingsAnnotations appended in arrival order
Catchwords carry the reader forwardNavigation lives at the point of needPage numbers only

Design consequence: if a display is going to accumulate commentary over time, design the positions first. Annotation systems fail when new voices have nowhere to go except the end.


§9. Annotation Patterns — the Working Catalogue

PatternMechanismSWFE source
Cross-notation annotationWords, images, notation, and vocables in one fieldKircher, p. 121
Fragment-plus-labelA clipped detail sits beside its reading, with the whole nearbyReinhardt, pp. 122–123
Marginal macro-annotationProse margins import context a raw record cannot carryICU bill, p. 124
Facing-page apparatusText left, notes right; two reading depths in parallelAnnotated Austen, p. 125
Reciprocal annotationWords and graphics take turns explaining each otherGalileo, p. 125
Substrate annotationThe evidence layer itself serves as the gridLeonardo, pp. 126–127
Many-hands local gridsIndependent annotators use independent local gridsAvicenna, p. 128
Layered institutional apparatusFixed positions for each named commentary traditionTalmud, p. 129
Annotations on annotationsA second color rebuts the first in placeSWFE p. 55
Unit/inflation instructionOne line making dated quantities re-computableICU bill, p. 124

§10. Failure Modes

A1 — Decorative Evidence. An image, chart, or scan shipped with no annotation at all. It reads as illustration, not evidence, and the viewer has no way to check anything. Fix: annotate to the point where a skeptic could argue with the display.

A2 — The Decode Round Trip. Letters, numbers, or symbols on the figure, with their meanings parked in a caption or key below. Identical in cost to a legend. Fix: words at the thing; add a small glyph if a verdict must be carried (SWFE p. 55).

A3 — Mode Segregation. Explanation confined to whatever notation the tool defaults to, so the reasoning never appears in one field. Fix: mix modes freely on the evidence itself.

A4 — Verbless Annotation. Labels that name parts and state no mechanism, cause, or consequence — the annotation equivalent of an unlabeled arrow. Fix: every important annotation gets a verb (see tufte-link-differentiation).

A5 — Annotation Louder Than Evidence. Callout boxes, drop shadows, heavy leader lines, and oversized type that out-shout the thing being explained. Fix: annotation sits on a quieter visual layer than the data (see tufte-layering-and-separation).

A6 — Un-anchored Annotation. Commentary floating in whitespace with no positional or graphical link to its referent. Fix: place it adjacent, or run a hairline leader to the exact feature.

A7 — Summary Replacing Record. Prose that describes an artifact instead of annotating it, discarding every micro-detail the reader might want to check. Fix: keep the artifact, annotate its margins.

A8 — Silent Units. Dated money, instrument-specific measurements, or local scales presented without the instruction needed to convert them. Fix: one line stating the conversion, as with the 1982-dollars × 7.5 note.

A9 — Unseeable Annotation. Commentary hidden behind hovers, tooltips, expand-arrows, or a separate tab, so it is never in the same eyespan as the evidence. Fix: spatial adjacency beats temporal disclosure.

A10 — Premature Pinning. Forgetting that annotations are hard to unsee (p. 121). A confident early reading pinned onto ambiguous evidence forecloses every other reading for every later viewer. Fix: annotate what is observed before annotating what it means; state scope limits, as Reinhardt did.


§11. Checklist Before Shipping an Annotated Display

  • Could a stranger name every element without leaving the figure?
  • Does at least one annotation state a mechanism, not just a name?
  • Does the display teach its own reading, or assume fluency?
  • Are source, date, scale, and units on the display itself?
  • Are dated or instrument-specific quantities accompanied by a conversion instruction?
  • Is every annotation quieter than the evidence it explains?
  • Is each annotation positionally anchored to its referent?
  • If commentary will accumulate, do fixed positions exist for future layers?
  • Where the evidence is contested, does the display hold the dispute rather than a verdict?
  • Have you separated observation from interpretation, given that both are hard to unsee?

Source: Edward Tufte, Seeing With Fresh Eyes: Meaning, Space, Data, Truth (Graphics Press, 2020), Chapter 5, "Annotations," printed pp. 121–130; with p. 27 (Menuhin substrate), p. 55 (annotations on annotations), p. 60 (data paragraphs). Reinhardt's Guernica annotation is reproduced from P.M., January 5, 1947 (© Estate of Ad Reinhardt / ARS). The ICU billing record is revised from Tufte, Envisioning Information (1990), p. 56, itself redrawn from David Hellerstein, Harper's 268 (March 1984). All passages paraphrased; quoted lines limited to single attributed sentences with page citations.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.