What a Ship's Data Actually Looks Like

Before a single filter, correlation or model, there is a step that decides how much of the rest will be worth reading: looking at what you actually have. This lesson is the tour. What kind of question can this dataset answer, what kinds of variables are in it, and how large is the sample really, as opposed to how large the row count says it is.

Four questions, in order of ambition

Data analysis is usually sold as prediction, but prediction is the third of four things a dataset can be asked to do, and the first two have to be finished before it is entitled to attempt the third.

Most commercial disappointment with performance analytics comes from selling rung four on the strength of rung one. A model that recommends a trim is implicitly claiming that the fuel differences it learned from were caused by trim rather than by the weather that happened to accompany it, and that claim is diagnostic work nobody did.

What actually arrives

A modern autologger samples continuously and reports on a fixed interval. At one record every ten minutes, that is 144 records a day and 52,560 a year per channel, and a well-instrumented ship carries dozens of channels. Noon reporting, by contrast, produces 365 records a year, each of them an average over a day that may contain a storm, a port call and a speed change.

The gap between those two numbers is the whole argument for continuous monitoring, and it is smaller than it looks. What matters is not how many records you hold but how many of them can be compared with each other.

The variables, and why their type matters

A shipboard dataset mixes kinds of variable that behave differently under analysis, and the arithmetic that is legal on one is nonsense on another.

• Quantitative continuous. Speed through water, shaft power, fuel consumption, draught, wind speed. These are measured on a numeric scale where differences and ratios mean something: 12 knots really is twice 6 knots.
• Quantitative discrete. Counts: number of generators running, number of records in a period.
• Ordinal. Categories with a genuine order but unequal steps. Beaufort number is the classic trap: force 8 is worse than force 4, but it is not twice as bad, and averaging Beaufort numbers produces a quantity with no physical meaning.
• Categorical. Labels with no order at all: port, voyage leg, laden or ballast, fuel grade.

The rule that follows is unglamorous and saves a great deal of embarrassment: before computing anything, establish which of these you are holding. A mean Beaufort of 4.7 has appeared in more performance reports than anyone would like to admit.

Why a year of data is a small sample

Here is the fact that separates marine performance data from the tidy datasets in statistics textbooks. Suppose you want the relationship between speed and fuel consumption for one bulk carrier, and you have a full year at ten-minute resolution: 52,560 records, which sounds like more than enough.

Now impose the conditions under which those records are actually comparable. Weather and sea state must be similar, because added resistance in waves and wind moves consumption more than most of the effects being studied. Draught and trim must be similar, because the hull is a different hull at a different displacement. Hull and propeller condition must be similar, and this one is guaranteed to drift: fouling accumulates over the very interval that makes the dataset long.

What survives all three filters is not 52,560 records. Depending on the trade, it can be a few hundred. The dataset is long in rows and short in comparable rows, and the second number is the one that governs what can be concluded. This is why the physics matters even in a data-driven analysis, and it is the theme the last course of this specialisation is built on.

There is a second reason, and the third course of this specialisation puts a number on it: consecutive records are not independent. A ship making 12 knots in one record is usually still making about 12 knots in the next, so each new record adds far less than one record's worth of evidence, and a long run of them carries the information of far fewer separate observations than its row count.

Sample, population, and the question underneath

Statistics describes a sample in order to say something about a population. In a laboratory this distinction is obvious. On a ship it is slippery, and getting it wrong is the commonest analytical error in the field.

The sample is the set of records collected. The population is almost never "all ships" and almost never "all records ever transmitted". It is usually this ship's future operation: the thing you are implicitly predicting when you quote a figure to a charterer or a superintendent. If the sample came from a laden Atlantic winter and the claim is about ballast passages in the Arabian Sea, no amount of statistical machinery repairs the mismatch.

First contact with a real dataset

Throughout this course we work with signals from two real vessels, sister VLCCs of about 317,000 DWT whose performance recorders log one reading a minute. This lesson uses the least filtered under-way cut of the first ship's record: a pair of channels, speed through water and propeller shaft power, taken over every minute she reported herself under way, with no quality filter applied at all. That is 562,788 records. Before any analysis, the description alone is informative: mean speed 10.07 knots, mean shaft power 7,208 kW, a minimum shaft power of exactly 0 kW, a minimum speed of minus 4.93 knots and a maximum shaft power of 181,528 kW.

Three of those numbers are questions rather than facts. The zero asks whether the ship was at anchor with her main engine stopped or whether the sensor dropped out; the statistics cannot tell you, and the answer changes what every other record in the file means. The negative speed is the speed log reporting a longitudinal speed below zero at and around rest, which is what that instrument does there: a true record of what the sensor said, not a ship making sternway. And the maximum is not a reading of anything. These ships' main engines are rated 27,440 kW, so no shaft on board ever carried 181,528 kW; the figure is a fault in the power channel, and the maximum is where a fault of that kind shows first.

Notice what the first of those does to the cut itself. Every one of these records carries the ship's own under-way flag, so a shaft power of zero here is a question about the flag rather than an answer about the ship. Average that column, zeros and 181,528 kW and all, and you get a mean that describes the file rather than the machine, exactly as the generator's average at the top of this course did. From the next lesson onwards we work with the cleaned cut of the same ship instead, and say so each time.

Monday morning

Take the last dataset somebody sent you and answer three questions before computing anything. Which rung of the ladder is the request on, and has the rung below it been done? Which of the columns are ordinal or categorical and therefore off-limits to averaging? And how many of the rows are genuinely comparable with one another, as opposed to merely present? The next lesson starts with the measures that summarise what survives those questions, and with the shipboard signal that defeats all of them.