As we near the end of this decade, and more importantly the end of the hell that was 2020, I realised two things. First, I survived catching Covid-19, and secondly that I had a good story about the history of Silicon Valley that I’d never written down. I’m still functional, so here, for perpetuity, is my tale.
Back in 2013 I was working in the Ads Interfaces organisation at Facebook, building mostly front end products (other people did the AI, database etc). We had an application called Power Editor which was the kitchen sink of products, and 25% of all Facebook revenue depended on it working. Every single thing you could do with ads on Facebook was supported in Power Editor (we called it P.E. for short). This made it huge, hugely complex and pretty user hostile. However our big spending customers were forced to use it as it was the only way they could efficiently scale – PE had a lot of cool tools for duplication, permuting your ads and working in batches of thousands of changes.
Clik here to view.
By 2013, PE was creaking under its own weight (about 150k lines of front end JavaScript), and no one wanted to work on it. We had half of one (awesome) engineer supporting it, and she could just about keep it working. I was looking for an opportunity to become a manager, and my manager Brian and I decided that building a team to properly support PE would be a good idea.
I looked into the code base, and was shocked to find that the entire application depended on a technology called WebSQL. It only ran on Chrome, and Google had deprecated WebSQL over a year earlier. I kind of flopped back in my chair, dragged my manager to a room, and told him that Google could shut off 25% of Facebook’s revenue, and lose us all our large accounts, by turning off WebSQL in Chrome, and it could happen any time.
This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. They had already deprecated WebSQL, and other browsers had removed it. They would have been well within their rights to just flip a feature flag in Chrome and do the same.
There was a whispered joke among those who knew about it that Google never had to bother building Google +, they could shut us down by changing one boolean in a database.
We quickly put in place a team of 5 engineers and one PM to work on it, with me managing it and coding probably 75% of the time. The plan was fairly complex. There was no easy way to get us off of WebSQL without a full rewrite of 150k lines of JavaScript. We couldn’t just build a new application from the ground up, that would take years to support all the features, and Google could turn off WebSQL at any time. Also, PE was falling apart – it was blocking the entire company from shipping any new ads products at all.
So, we decided to first make it not fall apart with a lot of performance and reliability work. Next, we had a long running project called PE Live, where we took each subset of code and made it read from the live API rather than locally from the WebSQL database. To unblock the other teams we would rewrite the whole thing in ReactJS, whereas it was then built using two frameworks that we had sunset called UkiJS and BoltJS.
Clik here to view.
This whole process took over three years. By the time it was complete in 2016, we had improved Power Editor so much, with better features, more stability, speed and ease of development for partner teams, that over 50% of all Facebook’s revenue was spent through it. The team grew to 13 engineers, with lots of help from dozens more across the Ads organisation building new APIs and infrastructure to support our work.
Google could have killed it at any time, and there were no complete alternatives for our customers – some third party applications existed, but they were even buggier than PE, were always late with new features (we didn’t have to wait for a new API to be public, they did), and often each specialised in a subset of the features. 50% of our revenue disappearing over night could have happened. That it didn’t and that we moved mountains of code doing the horrible, inglorious work of rewriting hundreds of thousands of lines of spaghetti code in production while people used the product, while also building an infinitely better product, is the most satisfying period of professional work I’ve ever been fortunate enough to experience.
A huge thanks to all the amazing people I worked with on that crazy project, you’re the best team I ever worked with, and I’d hop into the trenches with you again any time!
Clik here to view.
[Edit: So this ended up on the front page of Hacker News, and there’s much more conversation about it over at https://news.ycombinator.com/item?id=26086056 ]