100% Coverage is a Crock of Clam Juice

Marking, a kindred spirit from a faraway place high above Point Defiance near Old Town Tacoma, was in the Twin Cities for a visit last night. One of Marking’s quirks is that he’ll passionately engage you with various topics from non-violent conflict resolution to single-payer health care, but if you ask him about his profession, he instantly becomes a clam with lockjaw.

But last night, while dodging rotten tomatoes at the Artists' Quarter poetry slam, I cracked the calcified exoskeleton of Marking's professional ennui.

Like shucking clams with a catcher's mitt, I tried my best to explain what I learned about Black Swan theory from Nassim Taleb's influential book, The Black Swan: The Impact of the Highly Improbable.

Marking is a radiologist. When Marking is not marking time day-dreaming about his next day off, he uses imaging technology, like x-ray and radiation, to evaluate the likelihood of disease.

Marking grinned like a giddy mollusk when I laid out Nassim Taleb's Round-Trip fallacy like this

You can tell your patient there is No Evidence of Disease (NED), but cannot tell him that there is Evidence of No Disease (END)

Sadly, END is probably how the patient prefers to hear the diagnosis, but NED is what the report says.

Marking intimates that his profession is a crock of clam juice because the reports radiologists produce, after examining a chest film that falls within a normal range, have marginal clinical value. Many of his colleagues will order an ongoing series of baseless tests to confirm what they know for fear of litigation (e.g., Taleb calls this the Confirmation Fallacy) .

Marking's colleagues remind me of software professionals like us who make Zero Defect Pledges, or who shoot for 100% Code Coverage. We mindlessly pad our tests with meaningless test assertion after meaningless test assertion to confirm the obvious.

Coverage numbers give false security. In The Fallacy of 100% Code Coverage, Andrew Binstock says the raison d'être of unit testing is two-fold:

1. validate the operation of code; and
2. create sensors that detect when code operation has changed; thereby identifying unanticipated effects of code changes.

Beyond that, please ask yourself, am I full of clam juice?

The Fallacy of 100% Code Coverage

The fallacy of 100% Code Coverage is this

Just because you have attained 100% code coverage does not make it significantly more certain you haven't forgotten to test some critical chunk of code.

No Evidence of Defects (NED) is not the same as Evidence of No Defects (END). Sometimes a meaningful conclusion from test coverage is downright Rumsfeldian:

I would not say that the future is necessarily less predictable than the past. I think the past was not predictable when it started. --Donald Rumsfeld

The (END).