Ed Catmull tells a story in his book, Creativity Inc., about the time Toy Story 2 started to disappear from Pixar’s servers one day… The book is a fantastic read for anyone who loves Pixar movies, and especially for process improvement geeks who hope to glean insight into Pixar’s culture of problem solving and leadership.
I love to tell this story when teaching the continuous improvement principle, “Blame process, not people.”
To understand this first event, you need to know that we rely on Unix and Linux machines to store the thousands of computer files that comprise all the shots of any given film. And on those machines, there is a command—/ bin/ rm -r -f *— that removes everything on the file system as fast as it can. Hearing that, you can probably anticipate what’s coming: Somehow, by accident, someone used this command on the drives where the Toy Story 2 files were kept. Not just some of the files, either. All of the data that made up the pictures, from objects to backgrounds, from lighting to shading, was dumped out of the system. First, Woody’s hat disappeared. Then his boots. Then he disappeared entirely. One by one, the other characters began to vanish, too: Buzz, Mr. Potato Head, Hamm, Rex. Whole sequences— poof!— were deleted from the drive.
Oren Jacobs, one of the lead technical directors on the movie, remembers watching this occur in real time. At first, he couldn’t believe what he was seeing. Then, he was frantically dialing the phone to reach systems. “Pull out the plug on the Toy Story 2 master machine!” he screamed. When the guy on the other end asked, sensibly, why, Oren screamed louder: “Please, God, just pull it out as fast as you can!” The systems guy moved quickly, but still, two years of work— 90 percent of the film— had been erased in a matter of seconds.
An hour later, Oren and his boss, Galyn Susman, were in my office, trying to figure out what we would do next. “Don’t worry,” we all reassured each other. “We’ll restore the data from the backup system tonight. We’ll only lose half a day of work.” But then came random event number two: The backup system, we discovered, hadn’t been working correctly. The mechanism we had in place specifically to help us recover from data failures had itself failed. Toy Story 2 was gone and, at this point, the urge to panic was quite real. To reassemble the film would have taken thirty people a solid year.
I remember the meeting when, as this devastating reality began to sink in, the company’s leaders gathered in a conference room to discuss our options— of which there seemed to be none. Then, about an hour into our discussion, Galyn Susman, the movie’s supervising technical director, remembered something: “Wait,” she said. “I might have a backup on my home computer.” About six months before, Galyn had had her second baby, which required that she spend more of her time working from home. To make that process more convenient, she’d set up a system that copied the entire film database to her home computer, automatically, once a week. This— our third random event— would be our salvation.
Within a minute of her epiphany, Galyn and Oren were in her Volvo, speeding to her home in San Anselmo. They got her computer, wrapped it in blankets, and placed it carefully in the backseat. Then they drove in the slow lane all the way back to the office, where the machine was, as Oren describes it, “carried into Pixar like an Egyptian pharaoh.” Thanks to Galyn’s files, Woody was back— along with the rest of the movie.
Here, in rapid succession, we’d had two failures and one success, all of them random, all of them unforeseen. The real lesson of the event, though, was in how we dealt with its aftermath. In short, we didn’t waste time playing the blame game. After the loss of the film, our list of priorities, in order, were: (1) Restore the film; (2) Fix our backup systems; (3) Install precautionary restrictions to make it much more difficult to access the deletion command directly.
Notably, one item was not on our list: Find the person responsible who typed the wrong command and punish him or her.
And so we’ll never learn who launched the delete code, but we do learn of a bright spot in Pixar’s culture, one that fosters improvement by focusing on the process that enabled the mistake… not pointing fingers. (The one exception to this rule is Stinky Pete. He’s just the worst.)