What Correlation Is, and What It Is Not

A correlation matrix is the first thing most analysts produce from a cleaned dataset and the last thing most of them interrogate. This lesson is about what the numbers in it actually claim, which is both less and stranger than it appears.

Three properties, not one

A correlation has three separable properties, and confusing them is the source of most misreadings.

Kind is whether the relationship is linear. Two variables can be tightly related through a curve and produce a linear correlation near zero. Power against speed is close to cubic, and over a wide enough speed range a linear coefficient understates how tightly they are bound.

Direction is the sign. Positive means they rise together, negative that one rises as the other falls.

Strength is how closely the points follow the relationship. This is the only one of the three the coefficient's magnitude reports, and it reports it only for the kind of relationship the coefficient can see.

One convenient property: the coefficient is dimensionless and unchanged by any positive rescaling. Consumption in tonnes per day or kilograms per hour gives the same number, which is what makes a correlation matrix comparable across channels at all.

The thing it does not contain

A correlation coefficient contains no information about causation, and the standard illustration is worth keeping because it is so hard to argue with.

Ice cream sales and shark attacks, tallied over the same months, correlate positively and strongly. Neither causes the other. Both respond to a third variable, summer, which raises ice cream consumption and simultaneously puts far more people in the sea.

Two mechanisms generate correlations without causation, and they need different responses.

Common cause. A third variable drives both, as above. This is the case that matters most in performance work, because ships are full of them: weather drives speed, power and motion simultaneously; loading condition drives draught, displacement and consumption together. Partial correlation, in lesson five, is the tool for it.

Coincidence. With enough variables, some pairs will track each other for no reason at all. Published collections of spurious correlations pair the popularity of a first name with wind power generation, or planetary distances with degrees awarded in a particular subject, at coefficients above 0.9. The mechanism here is arithmetic: test a thousand variable pairs at the 5 per cent level and about fifty will come back significant from chance alone.

That second mechanism is a growing hazard rather than a curiosity. A well-instrumented ship offers dozens of channels, and an automated search over their pairwise correlations will always find something.

What a coefficient is good for

The tone above is sceptical, not dismissive. Correlation earns its place in three ways, and all three are descriptive.

Screening. Before building a model, a correlation matrix says which candidate inputs carry any linear information about the target. It is a way of shortening a list, not of choosing from it.

Redundancy detection. Two channels correlating above about 0.95 are carrying nearly the same information. Sometimes that is data lineage, one channel computed from the other somewhere upstream; sometimes it is physics holding two quantities almost rigidly together. Either way it is worth knowing, and either way putting both into a model will make its coefficients unstable. The two sister VLCCs behind this course supply the measured case: on their cleaned records, main engine fuel consumption against shaft power gives 0.998 on one ship and 0.997 on the other. On these ships the consumption channel comes from a fuel meter, not from a calculation on power, so the near-perfect agreement can be physics: an engine burns fuel almost in proportion to the power it makes. A channel calculated from power would have produced a coefficient just as high, and nothing in the number tells the two apart. Only the channel's source can, so find out where each channel comes from before treating the pair as two measurements.

Prompting a physical question. A coefficient that is not what the physics leads you to expect, or the absence of one that is, is a good reason to go and look. The 0.868 between speed and power that opens this course is not itself that prompt: a cube law should show as a strong positive association, and it does. The prompt arrives a step later. Control for propeller revolutions and fuel consumption on the same records and the pair reads −0.135, and a coefficient that changes sign when you hold something still is a question about the mechanism rather than an answer about it. Lesson five is where that question gets asked.

The rule for the rest of the course

Every method in the next seven lessons measures association, and none of them measures causation. What they differ in is which kinds of association they can see: linear, monotone, arbitrary but smooth, or any dependence at all. Choosing among them is a question about the shape you expect, and the physics is what tells you which shape to expect.

Monday morning

Take your last correlation matrix and mark each entry above 0.9. For each, answer one question: is this two independent measurements of a real relationship, or one channel derived from another? The coefficient will not tell you; the source of each channel will. Most fleets find at least one pair in the second category, and it has usually been feeding a model as though it were evidence.