Why Raw Data Isn't Enough: The Pipeline

7 min read

What you'll take away Describe what happens to vessel data between the sensor and the decision, and why each stage exists.

"We have the data" may be the most expensive sentence in fleet management. A vessel fitted with modern sensors produces torrents of measurements — and between those measurements and a decision worth money stands a series of unglamorous steps that determine whether the decision is sound or confidently wrong. That series of steps is the data pipeline, and the difference between having one and not having one is the difference between an asset and a liability.

What the pipeline actually does

Acquisition sounds trivial and is not. Signals arrive from different systems at different rates with different clocks; a pipeline aligns them onto one timeline, because a power reading is only meaningful next to the speed reading from the same moment.

Validation is the immune system. Sensors drift, freeze, and fail — and a failed sensor rarely announces itself; it just keeps transmitting plausible numbers. Validation cross-checks signals against each other (speed through water against GPS speed and known current; fuel flow against power) and flags what does not cohere. Without it, a speed log over-reading by half a knot poisons every analysis for months, silently.

Cleaning and filtering removes what cannot be used: transients during manoeuvring, records during gear changes, moments where key signals are missing. The goal is not less data — it is data that means what it appears to mean.

Normalisation is where physics enters. Two speed–power observations — one in calm water, one in Beaufort 6 — are not comparable until the weather's contribution is accounted for. Normalisation corrects observations to reference conditions using exactly the resistance physics of the first course. Skip it, and every comparison quietly mixes hull condition with weather.

Only after all four stages does data deserve the name information.

Engineered, not improvised

Every fleet does some of this somewhere — often in a spreadsheet, maintained by one person, differently each month. The word engineered means the pipeline is built like the infrastructure it is: documented, so the method can be explained to a verifier or a counterparty; repeatable, so the same input always produces the same output; monitored, so the pipeline itself raises a hand when something upstream breaks.

The distinction stops being philosophical the day the numbers are challenged — by a charterer disputing a performance claim, a verifier auditing an emissions report, or your own management asking whether the hull cleaning paid back. "The analyst cleaned it by hand" is not an answer that survives those conversations. The next lesson is about why, increasingly, it is not even an answer regulators accept.

Related reading: Why Raw Data Isn't Enough: Engineered Data Pipelines.

Check yourself

1. A vessel's speed log has been over-reading by 0.4 knots for three months. What does an engineered pipeline do that a raw data feed does not?
2. Why must vessel data be normalised before performance comparison?
3. The reason to prefer a documented, repeatable pipeline over ad-hoc spreadsheet cleaning is:
4. What must the acquisition stage do before any two signals can be compared?
5. What does the lesson give as the goal of cleaning and filtering?
6. Only after which point does the lesson say data deserves the name information?
7. A failed sensor normally announces itself, so validation is mainly a matter of spotting gaps where data is missing.
8. Which properties does the lesson say make a pipeline engineered rather than improvised?

Select all that apply.