Ratchet throws a (monkey) wrench
It’s 2008 and I’m working on the PS2 port of Ratchet & Clank: Size Matters I was a part of team that shipped the original PSP version that had released the year before and so my days were filled with making sure that this new version was as close to (or better than) how things played on the PSP.
The first level begins on a beautiful tropical island – waves lap up against the shore of a sandy beach and Giant Enemy Crabs wander around on a large hill that twists around the corner into the unknown. It’s a pretty great opening shot.
It’s getting very the end of the project and things are looking great. Sony was sure the game would pass submission (the last step before a game is sent off for mass production.) Their super-strict internal QA team has given us the green light. This is Gold Code. Immutable. Complete. Until… a high-ranking Sony executive had played it and was unhappy about something.
The first thing he does is walk right up to that damn hill and, upon seeing one of those giant blue crabs, presses the ‘Wrench Throw Button’.
The wrench flew forward towards the crab and then, instead of hitting it, the wrench bounced off the base of that hill and ricocheted right back into Ratchet’s hand. The crab lived – it was utter failure! That wrench needed to slide up the hill. It needed to kill that Giant Blue Crab.
This was clearly unacceptable. It was, in fact, so unacceptable that shipping the game needed to wait. This wasn’t the only “must fix” bug that came from the meeting, but the Wrench Throw Bug was particularly special for me: when I was hired at High Impact Games the second thing I wrote was the Wrench Throw Code.
This is code that I hadn’t seen in two years. Plus I wrote it as a complete noob in the games industry. Let’s just say it was far from elegant.
Regardless, I need to fix this code. But because this was ‘Gold Code’ I added a few new rules to my work flow:
- If I changed more than ten lines of code, I had to start over.
- If I changed more than one file, I had to start over.
- No changing anything relating to global systems (collision, for example) – my change had to be local to the wrench or Ratchet.
I spent an entire weekend working on the bug. During those two days I reverted and retried fixing the bug any way I could think of. With each try I would learn some more about what I had originally done, what might work and what certainly did not, and each time I got closer to solving this ridiculous riddle. Of all bug-related “brain teasers” that I’ve dealt with as a programmer, this bug stands out as one of the most entertaining. It was honestly quite thrilling – I knew I had to be perfect since based on the timeline the code would certainly not be getting its fair time with QA.
In the end I checked in a fix that changed one const value and one line of code. It was a very satisfying feeling to finally check a file in that I had reverted so many times before over those two days. The game still went out on schedule. It’s crazy that that single check in with one file in it still feels so gratifying to this day. Of all my war stories it’s the most peaceful and the most personal.
And hey, PS2 owners of the game… you guys can throw the wrench up the first hill in the game and kill a blue crab.