An engineer pushes a small change and OkCupid goes up in flames.
This is the story of an OKCupid developer who pushed a rather small change and crashed the entire network.
“That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down — sometimes bring it down to the point where we couldn’t recover it for hours.” When I saw him in the office the morning after I’d broken the site, totally mortified, he consoled me by saying that site outages were just the cost of relying on such a small engineering team.Logic Magazine
I’m not sure many outside the development community realize the level of complexity with which we currently deal. Even those of us who swim in the (in my opinion) less-volatile LAMP waters are still often Composer‘ing in tens of thousands of lines of other people’s code that has to live beside legacy code and the new stuff we’re adding.
OkCupid was a complex site. Had we tried to make it perfect, it might not have come to exist in the first place.Logic Magazine
But software is built on top of other software. You’re working not just with your own code but with code from your coworkers and from third-party software libraries. If these dependencies are buggy or complicated or behave in non-intuitive ways, errors may seep into the software that relies on them.
Great article that I just had to share.