Words
I mentioned in my last post that work had been piling on. It had. But in a strange way that left me drained and lost and it had relatively little to do with the usual things that make work awful and a lot to do with words.
For 15 years I've been responsible for Linfield College's website. Like an engineer on a ship, I nudged the site along through the years and the changes. It's been the sole task of my professional life and it's been an honorable one. After a decade and a half, I've come to understand that the web has always been about words. Yes, cat videos too; and scheduling your toilet bowl cleaner refill deliveries on Amazon as well. If you wind it back to the early 90's though, a hyperlink was the revolutionary connection of a word to another document filled with words with similar connections. That was the web. The pictures and videos and eventually the Apps and the Cloud and all that we understand as the digital world all followed the words and really, it still does. For better or worse, I got the words out there and millions saw them each year.
For all that time, we've served the site from on-campus on our own servers. On October 6th, our webserver stopped talking to the outside world. It worked fine on-campus, but if your computer wasn't in the IP range of Linfield's network, you couldn't see the site. You just got a blank page with some text saying the server wasn't responding. It actually took a day to even notice there was a problem. It was the sports fans who let us know. Most of us regard a page load failure as part of life on the web. Linfield has one of the best D3 football teams in the country. You may not see many Linfield flags in the sea of Beaver and Ducks banners flapping off suvs driving between Corvallis and Eugene, but They're out there and just as passionate; and if they can't get to their team's website when a game is on, the world is broken and someone better fix it.
We couldn't fix it. That was a first. We've had outages. We've been hacked so badly we had to throw away the physical webserver and buy a new box. But we've always fixed it. Not this time.
While I was waiting for others in the IT department to update me, I'd keep checking the site from off-campus. Parts of the site would come up occasionally then it would go down again. Most of the time I just stared at a blank browser window. It was so strange seeing a hole where my career used to be. It was... silent. It was silent against the complete chaos that was happening in the physical world around me. I'd joke with people about how we should have had children, or I should have volunteered more, or done something meaningful with my life because at the moment, I didn't have much to show for 15 years of work. That also happened to be how I felt.
Our new Chief Technology Officer wanted to move the server off-site and we were about a week and a half into testing on a new server from an off-site provider when things went bad. When word came back that we couldn't get the site to stay up, I made the call to cutover to the new webserver ready or not.
We weren't ready. The server wasn't ready. It was a mess. The site began to come back but the application firewall on the new server looked like it was blocking all kinds of page requests. It took almost a week to sort out how to fix it. Most of that week was spent staring at thousands of lines of error logs repeating "Access Denied" over and over. Worse, every time we tested a page, we'd get "Access Forbidden" messages instead of a page. "Access Denied" , "Access Forbidden". White text on a black screen. Thousands of times.
While I was waiting for others in the IT department to update me, I'd keep checking the site from off-campus. Parts of the site would come up occasionally then it would go down again. Most of the time I just stared at a blank browser window. It was so strange seeing a hole where my career used to be. It was... silent. It was silent against the complete chaos that was happening in the physical world around me. I'd joke with people about how we should have had children, or I should have volunteered more, or done something meaningful with my life because at the moment, I didn't have much to show for 15 years of work. That also happened to be how I felt.
Our new Chief Technology Officer wanted to move the server off-site and we were about a week and a half into testing on a new server from an off-site provider when things went bad. When word came back that we couldn't get the site to stay up, I made the call to cutover to the new webserver ready or not.
We weren't ready. The server wasn't ready. It was a mess. The site began to come back but the application firewall on the new server looked like it was blocking all kinds of page requests. It took almost a week to sort out how to fix it. Most of that week was spent staring at thousands of lines of error logs repeating "Access Denied" over and over. Worse, every time we tested a page, we'd get "Access Forbidden" messages instead of a page. "Access Denied" , "Access Forbidden". White text on a black screen. Thousands of times.
In spite of what you might think, computers don't hate us but they're infinitely patient and that can feel like hate. Those error messages were coded by people in a hurry to get that part of their programming work done. They were fatal errors; the end of the road, so little effort was invested in thinking about how an end user would feel seeing them. Most would be hackers anyway and we want to say as little to them as possible. In this situation, the software was blocking average people looking for information. And it was blocking me. And those words hurt. This was excluding language. It prohibits and in prohibiting, implies inferiority. By the end of the week I couldn't sleep, my stomach hurt, and worst of all, I was losing my ability to be civil to the living world, who were figuring out in droves that the site was seriously fucked up. I was frantic and angry and my self esteem was just gone.
As is often the case in IT, the solution presented itself during a sleepless morning. Most of the "Access Forbidden" pages people we were seeing were a result of stuck browser caches. How could we tell them to clear their caches? We took control of the language by replacing the "Access forbidden" page with one of our own design. We used friendly language. We provided cache clearing instructions and contact numbers for people who were still stuck. Things immediately improved.
Once the poisoned browser caches were cleared, we were left with people who were actually being shunned by the application firewall. I worked with a few users I knew well who would contact me with the exact time when they couldn't get a page to come up. I could filter through the thousands of error messages and find their page and error. We isolated about 5 rules that we switched off and everything came back online.
Nobody died. The autumn leaves continued to change color. Carolyn's side yard is spectacular right now. The football team is still undefeated. The rain came back. I'm amazed at what a wreck I was over something that in hindsight, wasn't that important. I marvel at how a written word, with no intent behind it, could make me physically ill.
I think about the words we level at each other with great intent. I think a great deal about the power they have.
Once the poisoned browser caches were cleared, we were left with people who were actually being shunned by the application firewall. I worked with a few users I knew well who would contact me with the exact time when they couldn't get a page to come up. I could filter through the thousands of error messages and find their page and error. We isolated about 5 rules that we switched off and everything came back online.
Nobody died. The autumn leaves continued to change color. Carolyn's side yard is spectacular right now. The football team is still undefeated. The rain came back. I'm amazed at what a wreck I was over something that in hindsight, wasn't that important. I marvel at how a written word, with no intent behind it, could make me physically ill.
I think about the words we level at each other with great intent. I think a great deal about the power they have.
Comments