One of the things I do every day as a site reliability engineer is deal with failure. Building and running a reliable system requires understanding all the ways that the system can fail, planning for those failures, and resolving them gracefully when they happen. If we know anything about complex systems, it's this: they can fail, they will fail, they will fail often, and they will fail in every way that is mathematically possible.
Most of the time, failure has real, negative consequences - systems go down, companies lose money, and sometimes one internal system failure can bring down an entire company - but it doesn't have to. You can build a system that is prepared for failures, experiences those failures, and handles them gracefully. The best systems are those that can recover from failures without any negative consequences, and emerge stronger, more stable, and more reliable. We call these kinds of resilient systems "fault tolerant" - a label reserved for the best of the best of our complex systems.
To build fault-tolerant systems, there's this wonderful thing we do in software engineering called "resiliency testing". Resiliency testing has many aspects, the most important of which is actively pushing the system to fail in real-time. When the system is running perfectly, we can make all of the failures we can predict happen in real-time: we can kill some of the servers it's running on, and see if it survives; we can take away other systems that it depends on, and see if it can recover. If the system completely fails when we are running these tests, we step back, figure out how to re-architect it so that it can recover without any negative consequences, and it emerges a stronger, more resilient, and fault-tolerant system.
The principles of building a fault-tolerant system are applicable to every complex system. They apply to the Google search engine, they apply to the vast systems that make up the Uber application, and, perhaps surprisingly, I've found that they apply to our own lives.
We are, both collectively and individually, terrified of and embarrassed by failure. We hide our failures so deeply within our hearts, fearing the repercussions if they were to be revealed to others, and scared of how we might view ourselves if we were to allow them to define or shape our sense of self-worth, our sense of who we are.
For many of us, failure is devastating. At times, it is devastating enough that people who experience failure in life (failing to get a promotion at their job, being fired from their job, failing a course, realizing they can't cut it in their dream career, failure in their relationships, etc.) experience severe depression, even sometimes to the point of taking their own lives. For most of us, failures are life-altering catastrophes - but they don't have to be.
I have failed so many times within my own life, and I am sure that I will fail over and over again until I die. I failed to become a physicist. I failed to become a philosopher. I have failed classes. I have failed to meet the expectations of some of the most brilliant people on this earth, who believed in me and expected great things. I failed to make it as a professional violinist. I failed to become a star student at one of the best universities in the world when I was given the opportunity. I have failed miserably, over and over and over again.
Several years ago, when I was reeling from one of these failures, I was completely devastated. I couldn't sleep, I couldn't eat. At times, I was unable to do anything except curl up on my couch and stare at the wall for hours and hours and hours on end. I didn't know how I could recover, didn't know what to do next. By some strange, wonderful twist of fate, I randomly picked up a book by Parker J. Palmer called Let Your Life Speak about finding your calling in life. In it, he kept saying the same thing, over and over, in as many ways as he could: you are your failures as well as your successes. Still reeling from the devastation of my failure, I grabbed all my failures, and I allowed them to define me. I am my shadow (my failures), I said to myself, as well as my light (my successes). I was who I was not in spite of my failures, but because of them. By doing this, I began to grow resilient. I picked myself back up, and I tried again.
You see, something really extraordinary happens when we allow ourselves to fail. After each failure, we learn to pick ourselves back up and try again. With each try, with each failure, we learn to approach the problem at hand with a little more grit, a little more wisdom, a little more elegance, and a greater deal of humility. We learn how to fail gracefully, we allow ourselves to fail, we push ourselves to fail, and we emerge stronger and more resilient. We become fault-tolerant.
The best complex systems are fault-tolerant. Each of our lives, the deepest sense of self, the deepest part of who we really are, is a complex system. If we know anything about complex systems, we know that they fail, and we know that the only way to make them better is to identify the ways they can fail, plan for their failures, actively push them to fail, and re-architect them so that they can handle failures gracefully, so that they can emerge stronger, more resilient, more stable complex systems.
We can learn something about how to become the best versions of ourselves from how we engineer the best complex systems in the world of software engineering. If we truly want to be the best versions of our incredibly complex selves, we need to embrace failure. We need to strive for fault-tolerance. We need to realize that we will fail, identify where we can and will fail, plan for our failures, push ourselves to try as hard as we can at the things most important to us, learn from our mistakes, and recover gracefully when those failures do happen.
Because those failures will happen. We are complex systems. Each of us will fail at something, and many of us will fail again and again and again. If you haven't failed, you haven't tried to become the best you can truly be, and you probably haven't even come close to realizing your full potential.
I hope that I keep failing. I hope that I fail again and again and again and never stop trying to become the best possible person that I can be, and I hope the same for all of you.