Impossible, Improbable, Interesting

There is a version of data cleaning that consists of applying a three-sigma rule and moving on. It is fast, it is what most tools default to, and it quietly destroys the most valuable records in the dataset. This lesson sets up the alternative: a taxonomy that decides treatment, and a pipeline discipline that makes every decision visible.

Three kinds of suspect record

The instinct to treat "outlier" as a single category is the source of most of the damage. Suspect records fall into three groups, and each calls for a different action.

Impossible records violate what the ship can do. A bulk carrier at 28 knots, a negative fuel flow, a main engine well above its overload limit, a draught outside the hull's geometry. These are facts about the instrument or the pipeline, never about the vessel. They are excluded from analysis and retained as evidence, because next quarter's question, "when did that sensor start misbehaving?", has no other answer.

Improbable records are inside physics but unusual: a power spike during a hard turn, a reading a few per cent above MCR (the maker permits about 110 per cent of it for an hour in every twelve, so MCR is a rating, not a ceiling), a rapid draught change from ballast operations, a consumption figure well above the trend. They are kept and examined. Some will turn out to be the third category.

Interesting records are the improbable ones with an explanation: heavy weather, an engine transient, a fouling step change after a cleaning that did not take. These are why the data were collected at all.

The storm problem

The single most damaging habit in performance work is calibrating a filter on quiet data and running it over everything. Extreme values are where the interesting physics lives: added resistance in a seaway, machinery under stress, the operational decisions that actually differ between good and bad voyages.

A filter tuned to reject the unusual will conclude that the fleet never meets heavy weather. That conclusion is flattering, useless, and entirely a product of the filter. It is also almost impossible to detect afterwards, because the evidence was thrown away by the step that created the error.

The defence is the taxonomy above, applied in that order: physics first, statistics second. Physical-range checks come from the ship's own limits and would flag the same record whatever the rest of the dataset looked like. Statistical tests come from the sample and are therefore only as trustworthy as the sample's composition. The rest of this course is about the statistical layer, and everything in it should sit downstream of the physical one.

Flag, never delete

Two rules carry the whole discipline, and both are about keeping options open.

Raw data are immutable. Corrections are applied downstream, by scripts, never by editing source records. The moment raw data become editable, no analysis built on them can prove what it was built on, and a disputed figure cannot be reconstructed.

Records are flagged, not deleted. A flag says which rule fired and why, so that the analyst works from a clean subset while the evidence trail survives intact. Deletion is not a stronger version of flagging; it is flagging with the reason thrown away.

Both rules are cheap to follow at the start of a project and expensive to retrofit, which is the usual shape of a discipline worth having.

Report what you removed

A performance figure computed on 88 per cent of the available records is a different figure from one computed on 99 per cent, and the reader cannot judge which they are holding unless you say.

Three numbers belong beside every cleaned result: how many records the period should have produced, how many survived cleaning, and the share removed by each rule. The third is the one that catches problems. A filter that removes 2 per cent every month and 19 per cent in one month has found something, and whether that something is a failing sensor or the worst weather of the year is exactly the question the report should provoke.

Here is that discipline exercised on two real ships: sister VLCCs of about 317,000 DWT, in the same trade, each logging one record a minute, put through the same pipeline. The bars carry the third of those numbers in the form the pipeline itself publishes it: the share of each ship's logged records still standing once each group of rules has fired, so that each rule's cost is the step down to the bar below.

Read the two chains rather than the two end points. The stage where the ships separate is the first one, the stuck-sensor filters, which look for a channel repeating a single value while the ship's state moves around it. VLCC1's speed log is flagged stuck in 11.0 per cent of all her records and her power channel in 8.1 per cent, and 23.91 per cent of her records carry at least one stuck sensor, against 5.56 per cent on her sister. Every one of those shares is of everything the ship logged, not of what survived. The largest single fall on both ships then comes at the next stage, the performance filters, and a stage that decides that much deserves to be the best documented rule in the chain rather than the least examined one.

The end points are what a report quotes, so they are worth one more sentence. VLCC1 finishes with 19.48 per cent of her logged records usable and VLCC2 with 37.31 per cent, while the two ships delivered 97.9 and 97.5 per cent of the minutes the calendar asked for. Coverage is not quality. A fleet dashboard counting records received has said nothing yet about how many of them anyone can use.

Monday morning

Write down the cleaning rules currently applied to your fleet's data, in the order they run. Most organisations discover that this list has never been written down, that some rules exist only inside a spreadsheet formula, and that nobody can say what share of records each removes. That inventory is the prerequisite for everything else in this course, because a threshold you cannot find is a threshold you cannot defend.