On the money

Even before the widespread rise of LLMs, one of the earlier uses of Machine Learning models was classification – taking a stream of input data and deciding what out of a series of ‘categories’ it fell into. LLMs reduced barriers to entry – the adaptable nature of LLMs allow them to be used as an ‘out of the box’ classifier method, with no pre training required. However, an issue with this approach is cost – although you save on upfront training costs on the model, per-token billing can quickly add up, particularly if you’re processing a large number of texts – for example, my use case of classifying multiple textbooks and research papers over the course of my upcoming university studies. To create an agent able to answer questions from this corpus, I would need to spend a total of hundreds of millions of tokens on ingestion alone, which would lead to costs quickly adding up.

Of course, local solutions are available. However, at a rate of about 50 tokens per second on my device, these aren’t scalable solutions. When utilising a faster neural network-based solution such as docling, I found its’ ability to intuitively chunk text was limited, leading it to – on occasion – capture an entire page of list-style text as a single ‘table’, rather than break it down into sub-sections.

This ingestion error was intermittent – I speculate that it happened on the margins of the models’ capabilities, as some pages with similar layouts were ingested without issue whilst with others, the ingestion process continually failed.

This presented a choice between two imperfect approaches. I could either accept the inconsistent structure produced by a specialised local model, or send every page to a larger hosted model and accept a potentially significant recurring cost. Neither seemed particularly satisfying.

Instead, I separated extraction from interpretation. PyMuPDF became responsible for recovering the exact embedded text, coordinates and page images from the PDF. These were treated as immutable source evidence. GPT-5 nano was then asked to classify the blocks which already existed, rather than transcribe or regenerate the page.

This distinction reduced the scope of the model’s task. For each block, the model received an identifier, its existing label, its bounding box and a bounded excerpt of its text. It could propose that a generic text block was actually a heading, caption, footnote, formula or list item. It could also associate a caption with an existing picture, table or formula. It could not introduce an unknown block, change its text or silently move its bounding box.

The model also classified each page into a broad category, such as body text, contents, chapter opener, review, references or index. This was useful contextual information, although the current implementation does not yet make full use of it. The page category is stored alongside a block when that block is changed, rather than being persisted as an independent property of every page. Consequently, “broad semantic enrichment” currently describes block relabelling and relationship assignment more accurately than it describes complete document understanding.

This broader pass was not the first use of GPT-5 nano within the ingestion process. My initial implementation was considerably narrower. A deterministic classifier first identified likely contents pages by looking for headings such as “Contents”, “Brief Contents” and “Table of Contents”. Adjacent pages could be included when they resembled a continuation, for example by containing several lines ending in Arabic or Roman page numbers.

Only contents pages containing suspicious parser output were eligible for repair. A block labelled as code on a contents page was treated as suspicious, as was Docling’s document_index structure or any label which could not be mapped safely into the application’s canonical model. GPT-5 nano then received a high-detail image of that individual page and proposed a structured reconstruction of its contents entries.

Even here, the model did not have complete authority. Every returned title, number and page locator was checked against the text extracted from the PDF. Duplicate entries, invented text, invalid bounding boxes and entries in an impossible visual order caused the repair to be rejected. The original parser output was retained, and a deterministic fallback converted obviously unsuitable labels into safer ones.

This repair stage was limited to twelve pages per document. Later candidates remained available to the deterministic safeguards rather than triggering an unbounded series of paid requests. Every attempted, accepted or rejected repair was recorded in repair-report.json, including the model used, the reason the page was selected and the original parser references which had been replaced.

When I later broadened the use of GPT-5 nano, I did not remove these rules. I added a separate, lower-detail pass before the specialised contents repair. Every non-empty rendered PDF page could now be considered for semantic labelling, up to a default limit of 800 pages. Pages were processed in batches of four, with no more than three simultaneous requests and a total reported token ceiling of 750,000 tokens per document.

The two passes therefore served different purposes. The broad pass asked relatively constrained questions about every eligible page: what kind of page is this, are any of its existing block labels wrong, and should one supplied block be related to another? The specialised repair pass asked a much deeper question about a small number of suspicious contents pages: what are the visually distinct entries on this page, and where are they located?

This approach reduced the need to treat ingestion as a choice between a completely local parser and an unconstrained LLM. Local code remained responsible for file identity, text extraction, geometry, provenance, validation, persistence and budget enforcement. GPT-5 nano supplied semantic judgement where the deterministic system lacked enough context to make a reliable decision.

There is still considerable room to broaden this process. The first step would be to persist the page category independently and make controlled use of it. A blank page could be excluded from chunking, a chapter opener could provide evidence for a structural boundary, and references or index pages could be processed according to their actual purpose rather than treated as ordinary prose.

The block vocabulary could also become more specific to textbooks. At present, labels such as text, section_header and list_item remain fairly general. Additional categories could distinguish definitions, learning objectives, worked examples, exercises, answers, case studies, summaries and key terms. These distinctions would later allow retrieval to answer more precise requests, such as asking specifically for practice exercises rather than merely retrieving prose which discusses the same subject.

Reading order is another promising extension. A model may correctly understand a two-column page or recognise that a sidebar interrupts the main body, even when the local parser produces a plausible but incorrect sequence. GPT-5 nano could return an ordered list of the existing block identifiers. Local validation could then require every block to appear exactly once, rejecting any response which omitted, duplicated or invented an identifier.

The same principle could be applied to heading hierarchy and cross-page continuity. Rather than allowing the model to rewrite the document, it could propose that one existing heading belongs beneath another, that a paragraph continues onto the next page, or that a table is split across two pages. Small overlapping page windows would provide this context while still restricting changes to a single target page.

More complex visual extraction would require greater care. GPT-5 nano may understand that a region is a table more readily than a specialised parser, but semantic capability does not guarantee exact recovery of every row, cell, negative sign or formula symbol. A convincing reconstruction which silently omits one value can be more dangerous than an obviously malformed extraction.

For that reason, I would not route every difficult page to the same model or ask GPT-5 nano to regenerate every page wholesale. A better design would use the broad pass to produce structured diagnostic signals. It could flag missing visible text, merged columns, broken reading order, a table extracted as prose, a formula represented as meaningless text or a caption without an identifiable parent.

A deterministic router could then select the appropriate response. Semantic ambiguity could receive a high-detail GPT repair. Missing scanned text could be sent to OCR. A damaged table or formula could be handled by a specialised recognition model. Unsupported or uncertain cases could be preserved unchanged and surfaced for manual review.

This creates a layered ingestion process:

Local text, blocks and geometry
             ↓
Low-detail semantic inspection
             ↓
Validated labels, relationships and diagnostic signals
             ↓
Selective high-detail or specialist repair
             ↓
Canonical document normalisation
             ↓
Chunking, retrieval and citations

The aim is not simply to replace Docling with an LLM. Docling and GPT-5 nano are capable in different ways. A specialised layout model is designed to measure regions, recover tables and produce repeatable document objects. GPT-5 nano is better positioned to interpret why a region exists and how it relates to the surrounding textbook. One behaves more like a typesetter; the other behaves more like an editor.

Whether the broader approach is actually better must ultimately be established through evaluation. I would want a manually reviewed collection containing ordinary prose, contents pages, multi-column layouts, tables, formulas, exercises, sidebars, references and scanned pages. Page classification, block labels, reading order, hierarchy and exact text preservation would need to be measured separately.

The result I am working towards is therefore not an ingestion pipeline in which an LLM owns the document. It is one in which local extraction establishes the evidence, GPT-5 nano proposes an interpretation, deterministic code decides whether that interpretation is admissible, and every accepted change remains traceable to the original page.

This retains much of the adaptability which made LLM classification attractive in the first place, while placing explicit limits around cost, provenance and trust. For a corpus which may eventually contain hundreds of textbooks and research papers, those limits are not an incidental optimisation, rather, they are what make the approach practical.