What is a data pipeline?

The four things that break

1. The source changes shape. A column is renamed upstream and the pipeline keeps running, writing blanks. This is the most common failure and the hardest to notice.
2. Duplicates on re-run. A job that appends rather than upserts turns one retry into a double-counted day.
3. Time zones and late data. Records that arrive after the window closed either vanish or land in the wrong day.
4. Silence. A pipeline that fails loudly is a nuisance. A pipeline that fails quietly is a quarter of wrong reports.

Notice that three of the four produce plausible wrong numbers rather than errors. That is why the useful investment is not more pipeline, it is a check on the output — row counts against expectation, a total against a known figure — that runs every time.

The smallest pipeline that counts

A pipeline does not require a warehouse or an orchestrator. A scheduled fetch from an API that appends to a table and recalculates the columns built on it is a pipeline. It has all the same failure modes, at a scale where one person can actually see them.

TableDI 2 stops one step short of this, on purpose. TableDI 2 is a desktop app for file work you redo every period: it keeps your sources, rules and delivery as a job, so next month you drop in the new files and run it again. Everything runs on your own machine. Nothing runs on a schedule: a person starts each period, the preflight checks the inputs, and the exceptions list is the check on the output described above.