Who Deleted Toy Story 2? A Pixar Lesson in Continuous Improvement

Ed Catmull tells a story in his book, Creativity Inc., about the time Toy Story 2 started to disappear from Pixar’s servers one day… The book is a fantastic read for anyone who loves Pixar movies, and especially for process improvement geeks who hope to glean insight into Pixar’s culture of problem solving and leadership.

I love to tell this story when teaching the continuous improvement principle, “Blame process, not people.

To understand this first event, you need to know that we rely on Unix and Linux machines to store the thousands of computer files that comprise all the shots of any given film. And on those machines, there is a command—/ bin/ rm -r -f *— that removes everything on the file system as fast as it can. Hearing that, you can probably anticipate what’s coming: Somehow, by accident, someone used this command on the drives where the Toy Story 2 files were kept. Not just some of the files, either. All of the data that made up the pictures, from objects to backgrounds, from lighting to shading, was dumped out of the system. First, Woody’s hat disappeared. Then his boots. Then he disappeared entirely. One by one, the other characters began to vanish, too: Buzz, Mr. Potato Head, Hamm, Rex. Whole sequences— poof!— were deleted from the drive.

Oren Jacobs, one of the lead technical directors on the movie, remembers watching this occur in real time. At first, he couldn’t believe what he was seeing. Then, he was frantically dialing the phone to reach systems. “Pull out the plug on the Toy Story 2 master machine!” he screamed. When the guy on the other end asked, sensibly, why, Oren screamed louder: “Please, God, just pull it out as fast as you can!” The systems guy moved quickly, but still, two years of work— 90 percent of the film— had been erased in a matter of seconds.

An hour later, Oren and his boss, Galyn Susman, were in my office, trying to figure out what we would do next. “Don’t worry,” we all reassured each other. “We’ll restore the data from the backup system tonight. We’ll only lose half a day of work.” But then came random event number two: The backup system, we discovered, hadn’t been working correctly. The mechanism we had in place specifically to help us recover from data failures had itself failed. Toy Story 2 was gone and, at this point, the urge to panic was quite real. To reassemble the film would have taken thirty people a solid year.

I remember the meeting when, as this devastating reality began to sink in, the company’s leaders gathered in a conference room to discuss our options— of which there seemed to be none. Then, about an hour into our discussion, Galyn Susman, the movie’s supervising technical director, remembered something: “Wait,” she said. “I might have a backup on my home computer.” About six months before, Galyn had had her second baby, which required that she spend more of her time working from home. To make that process more convenient, she’d set up a system that copied the entire film database to her home computer, automatically, once a week. This— our third random event— would be our salvation.

Within a minute of her epiphany, Galyn and Oren were in her Volvo, speeding to her home in San Anselmo. They got her computer, wrapped it in blankets, and placed it carefully in the backseat. Then they drove in the slow lane all the way back to the office, where the machine was, as Oren describes it, “carried into Pixar like an Egyptian pharaoh.” Thanks to Galyn’s files, Woody was back— along with the rest of the movie.

Here, in rapid succession, we’d had two failures and one success, all of them random, all of them unforeseen. The real lesson of the event, though, was in how we dealt with its aftermath. In short, we didn’t waste time playing the blame game. After the loss of the film, our list of priorities, in order, were: (1) Restore the film; (2) Fix our backup systems; (3) Install precautionary restrictions to make it much more difficult to access the deletion command directly.

Notably, one item was not on our list: Find the person responsible who typed the wrong command and punish him or her.

And so we’ll never learn who launched the delete code, but we do learn of a bright spot in Pixar’s culture, one that fosters improvement by focusing on the process that enabled the mistake… not pointing fingers. (The one exception to this rule is Stinky Pete. He’s just the worst.)



Happiness, Delivered

This post was originally published in January, 2011

delivering-happinessToday I finished listening to Tony Hsieh’s book Delivering Happiness. It was a fantastic audio book that taught me why Zappos is so Zappos!

There are so many things the company is doing to keep their culture alive and growing. One thing that really stood out throughout the book was Tony’s humility when it comes to talking about Zappos. He said they didn’t pioneer anything new with the Zappos culture, they just applied what they learned from others, such as reading Good to Great and Tribal Leadership. They put the research that is available to all of us, to work at Zappos.

It is also evident that Tony’s previous experience, the mother of all learning, played a large role in building the Zappos brand.

One solid takeaway from Delivering Happiness — for all the continuous improvement buffs out there: In chapter 6 Tony talks about Continuous Incremental Improvement (which by the way was #25 on Zappos’ initial list of 37 core values).

“So the challenge to everyone is this: make at least one improvement every week that makes Zappos better reflect our core values. The improvements don’t have to be dramatic. It can be as simple as adding an extra sentence or two to a form to make it more fun, for example. But if every employee made just one small improvement every week, to better reflect our core values, then by the end of this year we will have over 50,000 small changes that collectively will be a very dramatic improvement compared to where we are today.”

Continuous improvement is about the small stuff: consistent improvements from everyone. Matt Wrye at Beyond Lean finds the Lean parallels in the Zappos core values.

One last thought from the book. Forget what you learned about networking. Tony says to stop trying to network in the traditional sense. Build lasting relationships instead. Then in 2-3 years something might become of the relationship. The Huffington Post posted a snippet about Tony’s networking philosophy.

So there you have it. Improve continually and build relationships. If only we did these two things we would be on the path to greatness. But more importantly, we would surely be happy.