Halloween Special Post: The Horror of Certificate-Related Outages

As you may remember, my inaugural post on the dangers of certificate-related outages likened them to earthquakes because they are hard to predict, potentially catastrophic and too often ignored until another one happens.
 

But in using my earthquake metaphor, maybe you didn’t grasp how scary outages in themselves can be. So, a short quiz:
 

1. In Bill Friedkin’s The Exorcist, what caused Regan (Linda Blair) to rotate her head 360°, projectile vomit and spew obscenities?
    
2. In Alfred Hitchcock’s The Birds, what caused all those crows to attack and kill Annie (Suzanne Pleshette)?
    
3. In Fritz Kiersch’s Children of the Corn, what brought the demon “He Who Walks Behind the Rows” to the cornfields?
    
4. In Stanley Kubrick’s The Shining, what is REDRUM?
    

If you answered, “certificate-related outage” for each of the four questions, you would be wrong. That’s because REDRUM, in fact, is an expired wildcard certificate that had been cloned on an untold number of virtual machines. Apologies for the trick question.

A disclaimer: As a professional writer, I have the unfortunate tendency to use metaphor and analogy to make my points about anything, including certificate-related outages. It’s especially galling this Halloween season, given that real-world outage horror stories abound. Just invite my colleagues Mark Miller or John Muirhead-Gould to your next campfire or slumber party if you really want to be scared skinless.
 

In the meantime, let me whet your bloodlust for outage horror stories by recounting two recent ones we’ve encountered. Neither one involved the supernatural—well, as far as I know ...


 

The Financial Services Bloodbath

One October afternoon, a Fortune 500 financial services firm—the sort that manages institutional investments for other Fortune 500 companies—was orchestrating its usual late-day heavy trading of stocks, bonds and assorted futures (pork bellies aren’t quite spine-tingling as human brains, but apparently, zombies aren’t choosy when it comes to innards). Less than an hour before the markets closed, CRASH! their entire trading system went down. Its customers, many of which were hedging their bets to sneak one of those close-of-business trades, could no longer access the firm’s system.

Scythe-wielding children emerged from corn futures! Pigs’ blood drenched the auditorium! Moses parted the Red Sea! And ...
 

All right, none of those things happened, but the firm did lose millions of dollars along with several dozen customers, a FinServ horror story if I’ve ever heard one. And all because of an expired certificate.

 

The 13 Missing Organs

Okay, this horror story needs no embellishment. So, a long time ago there was this “healthcare provider,” whose primary-care hospitals were responsible for organ transplants. I don’t know the specifics of their life-saving work, but I know they kept a donor list for kidneys, livers and other giblets. Well, hospitals use a lot of certificates to manage the security of everything from servers to implantable devices, and because they lacked visibility into their certificate inventory, they experienced almost 100 TLS certificate-related outages over a one-year period. And 13 of those outages—all caused by expired certificates—were critical.
 

At least one of the 13 expired certificates caused an outage on the server hosting their organ-donor list, and it scrambled the list, which actually kept that organ from coming to the hospital. No word on how seriously the patient was impacted. But seriously, we’re talking about outages being life and death here.

Stop Screaming! Venafi Can Help.

Get a drink. Breathe deeply. Wipe your brow. I know you’re terrified—with good reason! No one wants to be the protagonist of a horror story, whether it’s because of demons or rabid birds or expired certificates. While Venafi can’t help with the former two (maybe Mark Miller can but no promises), we can help you handle the latter. Our new offering, No Outages Guarantee VIA Venafi, does exactly what it says, so you can sleep nightmare-free.

Want to learn more about our No Outages Guarantee VIA Venafi? Contact us directly, either through ESP or our contact form in the link.

Until then, want to share any certificate-related outage scares? Let us know in the comments or on Twitter! We promise not to reveal your identity (REDRUM).

Related posts

• Outages Are Like Earthquakes—Both Are Catastrophic and Hard to Predict
• Think Machine Identity Management Costs More than an Outage? Think Again.
• 3 Hack Attacks that Are Scarier than the Lame Halloween Costume You’re Wearing to Work Next Week!