Ambiguity is a signal your ingest pipeline should surface, not swallow.

The first ingest pipeline I built made me feel like a genius. Walk a folder tree, apply a regex to each name, extract an ID, hand it to the next stage. It worked on all 50 folders I'd tested against. Then I pointed it at the real corpus — hundreds of folders, years of accumulated convention drift — and it silently returned an empty set for something like a fifth of them.

Not an error. Just… nothing. The regex didn't match, so the file was skipped, so the file might as well not have existed.

That was the day I learned that in an ingest pipeline, silent failure is worse than a crash, and that the pipeline's most important output is the report of everything it couldn't confidently classify.

Conventions are aspirations, not contracts

Every enterprise folder tree is a museum of what the naming convention used to be. Someone at some point decided Proposal/ should always be the subfolder. Ten years later, half the older folders have Proposal Files/, three have [RFP] Files/, one has PROPOSAL_FINAL/, and two of those don't exist at all because someone deleted them “to clean up.”

A pipeline that only handles the current convention isn't robust. It's brittle in a way that will only reveal itself when the interesting data — the historical stuff, the outliers, the exceptions — is exactly the data you needed.

The move: assume the convention will be mostly right, write the parser for the mostly-right case, and route everything the parser can't confidently handle into a diagnostics report.

Silent skip vs loud drop vs surfaced ambiguity

Three ways to handle a folder your parser can't classify:

1. Silent skip. The default. It's what happens if you continue on a non-match. It is by far the worst option, because there is no signal anywhere that data is missing. You will find out at deadline.
2. Loud drop — crash the pipeline. Not much better. It stops the healthy 80% from being ingested because 20% is ambiguous.
3. Surface the ambiguity. Ingest what you can confidently classify. Write everything else into a structured report — folder path, guess at what went wrong, closest matching known pattern — and hand that report back as first-class output.

Option 3 is more code. It's the only one that scales.

Once the ambiguity report exists as a real deliverable, an interesting thing happens: the people who own the data start caring about it. A pipeline output that says “here are 47 folders I couldn't classify, here's why, here's what I'd guess” is a to-do list for the data owner. A pipeline that silently drops those 47 folders is just… wrong, and no one knows.

Two triggers, not one

The other lesson: an ingest pipeline needs both a timer (run every N hours to catch drift) and an HTTP endpoint (let an operator kick a run when they know something changed). Timers alone create latency between change and ingest. On-demand alone means nothing runs when no one is watching.

The same pipeline function, wrapped in two triggers. The timer covers steady-state. The endpoint covers the case where someone just uploaded something and needs it indexed now.

Extract with the dumbest tool that works

I over-engineered text extraction on my first pass. Custom heuristics per file type, careful handling of tables, layout awareness. It was slower than the “dumb” approach and produced worse output.

The lesson: a battle-tested document-intelligence API run on default settings almost always beats a bespoke extractor. Save cleverness for the parts of the pipeline that are specific to your domain — the naming conventions, the ID extraction, the ambiguity report — and let commodity infra do the commodity work.

What I'd tell past-me

• Never continue on a parse failure without also writing to a diagnostics stream.
• Design the ambiguity report before you design the happy path.
• Two triggers: a timer for steady-state, an HTTP kick for “now.”
• Use the boring, battle-tested extractor. Reserve custom logic for domain-specific parsing.
• Health and readiness endpoints matter more than you think. When something breaks at 2am, the ability to remotely confirm the pipeline is alive is the difference between a five-minute fix and a two-hour panic.

An ingest pipeline's job isn't to succeed. It's to be honest about what it did.