The Backup That Saved a Billion-Dollar Franchise 🎬

There are two kinds of developers in this world. Those who’ve done something stupid and put the whole project at risk, and those who lie.

I’ve worked on projects where we didn’t protect the main branch, and although nothing stupid happened I certainly went close.

So if you’ve a story where you’ve run rm -rf / by accident I’m more than willing to believe it. Pixar are not immune from such shenanigans either, because they almost lost Toy Story 2 in just such an incident.

Lights, Camera… Catastrophe!

In 1998, Pixar was working on Toy Story 2, a sequel that had everyone excited. The original had made waves in the animated movies industry and people love a sequel of a successful IP.

So when somewhere inside their production servers, a misfired Unix command (reportedly rm -r -f *) was executed in the project directory…things didn’t go well. For those unfamiliar, that’s like pointing a death ray at your hard drive and saying, “Go nuts”.

In minutes, 90% of Toy Story 2 was deleted. Models, assets, animation. Not just a few lost scenes. Not just some render previews. The movie. Was. Gone.

🪦 RIP Backups

You’d think Pixar would have bulletproof backups, right? They did.

Sort of.

Turns out their backups hadn’t actually been working for weeks. Nobody noticed. Why would they? That’s the thing about backup systems. It’s like smoke alarms, you don’t check them until there actually is a fire.

You could call it Schrodinger’s backup. It only works if you don’t check it.

I’m not even great at commiting my work at the right time to avoid losing a day’s work. Judge me all you like. 

Enter the WFH Supermom

Here’s where the story turns from tragic to cinematic. Galyn Susman, a technical director working on the film, had recently had a baby and was working from home. As part of her remote setup, she had a working copy of the project files on her home machine.

Yes, in a twist worthy of Toy Story 3, Pixar recovered the film from a copy stored on a mom’s Linux box in a spare bedroom, nestled between diapers and Cheerios.

They carefully transported the machine (blanketed and seatbelted like royalty), plugged it in, and… the files were there. Like Buzz Lightyear returning from deep space.

There’s More

You’d think the story ends there, but nah. Pixar execs took one look at what they’d salvaged and said, “This looks terrible.” So they threw most of it away anyway and rebuilt it. In nine months.

Imagine surviving a disaster, restoring the lost work, then saying “eh, let’s start over”. That’s commitment. Or insanity. Or both.

The Takeaway

This story is passed around developer circles like a cautionary bedtime tale. But here’s what it really says:

  • Never trust your backups. Test them instead.

  • The most important asset in your company may be someone’s personal machine in a suburban living room.

  • Just because something can be undone, doesn’t mean you shouldn’t start over anyway.

and perhaps it’s time to stop scapegoating WFH employees, because here is one that saved an entire movie.

About The Author

Professional Software Developer “The Secret Developer” can be found on Twitter @TheSDeveloper.

The Secret Developer trusts nothing except Ctrl+Z and blind panic.

Previous
Previous

When Leaders Say They Wish They Were Senior Engineers Again

Next
Next

This Is Why We Test