Beware the algorithm

One of my doc students studied Aviso‘s predictive analytic system on his campus, and compared their predictions to actual academic outcomes for students: he found almost no relationship. I suspect most of the early-alert systems being peddled in higher ed also have low predictive ability. But they sound cool!

Law enforcement’s eagerness to use an immature technology underscores a worrisome trend you may have noticed elsewhere: Humans have a habit of trusting the output of an algorithm without troubling themselves to think about the consequences. Take the errors we blame on spell check, or the tales of people who follow their GPS over a cliff. We assume that the facial-recognition booths at passport control must be accurate simply because they’re installed at our borders.

In my years of working as a mathematician with data and algorithms, I’ve come to believe that analyzing how an algorithm works is the only way to objectively judge whether it is trustworthy. Algorithms are a lot like magical illusions. At first they appear to be nothing short of wizardry, but as soon as you know how the trick is done, the mystery evaporates. There’s often something laughably simple (or reckless) hiding behind the facade.


Our reluctance to question the power of a machine has handed junk algorithms the power to make life-changing decisions, and unleashed a modern snake-oil salesman willing to trade on myths and profit from gullibility.

This is pretty funny:

The helpfulness of algorithms varies drastically. So how can you tell the bad from the good? There’s a quick trick I use to weed out suspicious examples. I call it the Magic Test. Whenever you see a story about an algorithm, replace buzzwords like “machine learning,” “artificial intelligence” and “neural network” with the word “magic.” Does everything still make grammatical sense? Is any of the meaning lost? If not, I’d be worried that something smells like bull—. Because I’m afraid—long into the foreseeable future—we’re not going to “solve world hunger with magic” or “use magic to write the perfect screenplay” any more than we are with AI.