I have a personal connection to this natural disaster. My wife has a cousin who lives in Tonga (if you’re wondering about how her cousin ended up there, she was there on [I think it was] a Peace Corps mission and met a guy there with whom she fell in love and eventually married). The last I heard, my wife’s cousin and her family are okay, but they are cut off from the rest of the world.
I don’t think there’s an experienced web developer or DBA who isn’t familiar with the classic “Bobby Tables” XKCD cartoon above. Just about any time you mention “Bobby Tables” to most experienced IT people, (s)he will immediately know to whom you are referring. Most experienced web developers and DBAs are aware of SQL injection and will take steps to ensure that it’s addressed. Grant Fritchey has a presentation about SQL injection (you can view and download his slide deck here) in which he’s not shy about his desire to “kill Bobby Tables.” I’ve seen him present it at SQL Saturday, and I highly recommend it.
Of course, the keyword here is “experienced.” For people who don’t have that experience, and who build websites that connect to databases, I think it should be lesson #1. Today, I had an experience that reminded me of that.
Earlier today, my sister texted me, asking for help with editing SQL code. She asked me what I use to edit SQL. I told her I generally use SSMS, although you can edit SQL code with a straight-up text editor, if necessary (she is not a DBA, so I felt somewhat comfortable telling her this). She told me she had to clean up spam comments in her data.
That last comment immediately grabbed my attention. I then asked her, how are your security settings, and do you have data backups.
She told me: that IS her data backup.
If her earlier comment had gotten my attention, this one immediately set off alarm klaxons in my head.
I started thinking about what could have corrupted her data to this extent. I started asking questions about her admin setup (I should’ve asked her to make sure she wasn’t using “sa” or “admin” as her admin login — Sis, if you’re reading this, make sure you check this!), including her passwords. Her admin password was pretty secure (thankfully).
She then mentioned her website. I asked if her website was accessing her data. She said yes.
I asked her about Bobby Tables (admittedly, in my advancing age, the term “SQL injection” didn’t immediately come to my mind). Her response: “who?”
At this point, I was convinced that I had my answer. Her database had been corrupted through SQL injection attacks. I told her to make sure you address your SQL injection issue before you even think about your data backups. Worrying about your data backups before addressing your SQL injection issue is like trying to rebuild your house before you’ve put out the fire.
I’ve been talking about SQL injection all throughout this article. For a brand-new web or database developer who has no idea what SQL injection is, here’s a quick primer: it’s a data security attack in which a hacker breaches your database by sending SQL commands through your web interface. I won’t get too much into how it works; instead, here are a few links that explain what it is.
So consider this a warning to any fledgling developers who are interested in web or data development: data security issues, such as SQL injection (and there are many others) are a big deal and need to be considered when building your setup; it’s not as simple as just setting up your website and connecting it to a database. By not considering this when you first assemble your system, you might be setting yourself up for major issues down the road.
I’ve always had a morbid fascination for air disasters. (Don’t ask me why; I have no idea.) I’m fascinated by shows such as Air Disasters, Why Planes Crash, and Mayday: Air Disasters. Whenever I hear about a plane going down, I’ll start thinking about what happened, clues, what might have caused it, and so on. There are times when I think I should have gotten a job with the NTSB.
Greg Moore has some publications in which he talks about lessons learned from aircraft accidents; his book partially discusses these lessons. He also has an excellent SQL Saturday presentation titled “Who’s Flying The Plane?” which talks about lessons learned from air disasters and how they can apply to IT. Go check it out if you have a chance; Greg gives a great presentation!
For the purposes of this article, however, I want to concentrate on a particular topic: how communication — or, the lack of — either contributed to or was the root cause of a disaster.
Last night, I watched an episode of Air Disasters that talked about the plane crash that took the life of professional golfer Payne Stewart. The plane went down after the cabin depressurized (the cause of which was never determined), the crew became incapacitated, and the plane ran out of fuel. What made it interesting to me was that bad documentation might have been one of the contributing factors to the accident. After the cabin lost pressure, the crew likely consulted a checklist, as is standard procedure for nearly any cockpit activity or incident. The checklist was poorly written and unclear. What should have been the very first instruction was, “put on your oxygen mask.” It was not. By the time the crew got to that instruction, it was too late; they were overcome by hypoxia.
It reminded me of a tenet that I preach in my documentation presentation: if, in a step-by-step instruction, an instruction cannot be understood within a few seconds, it has failed.
I also remember another Air Disasters episode that focused on Avianca Flight 52. In January of 1990, the plane, a Boeing 707 carrying 158 people, crashed on approach to Kennedy Airport in New York after running out of fuel, killing 73 people. There were numerous communication issues during the flight. Had any one of them been addressed, chances are the disaster never would have occurred.
How often have you been involved in some kind of activity where things were miscommunicated? How well did those activities go? I’m guessing that they didn’t go well. How often have they happened when deadlines were approaching? What was the mood of your organization? I’ll guess that it was likely one of high stress and low morale. And during that time, how smoothly did things go? Probably not very. I’ll bet that plenty of mistakes were made.
I’m painting this picture within a business environment. Imagine what these conditions are like when people’s lives are at stake.
The number of disasters that have occurred from poor communication are countless; entire studies have been dedicated to the subject. Indeed, numerous solutions and subcategories related to miscommunication have been devised. The airline industry developed the process of crew resource management. Extensive research has been done on the phenomenon known as groupthink. Even simple measures such as checklists have been studied and implemented.
The moral of the story: good communication, including documentation, is critical. The consequences of it can have adverse effects. At best, bad communication can disrupt your business. At worst, it can cost lives.
At the end of my presentation, one attendee started an argument with me. He kept saying that paper was dead, everything was online, and there was no reason to keep hardcopy documents. I argued, what if you can’t get to your online documentation?
Not surprisingly, he gave me a poor evaluation.
The bottom line is this: even documentation needs a backup. Other than, say, getting lost in a fire, paper documents can’t break. At a minimum, have hardcopy documents that instruct how to get minimal services back up and running, and back up other recovery documentation so you can recover it later.
Disaster recovery is one of the core tasks that many DBAs think about on a regular basis. Ensuring that we can get our data back online, available, accessible, and intact is important. More than a few DBAs that haven’t been able to recover systems, find themselves seeking new employment.
That’s not to say that most DBAs perform perfectly under pressure. Plenty make mistakes, and there may be times when they can’t recover all data. There does seem to be a correlation between how often DBAs practice recovery skills and how well they perform in an actual emergency. I know that at a few companies, we scheduled regular disaster tests, though often with simulated recovery of a systems that didn’t expect to actually take over a workload. Arguably not a good test, but better than nothing.
A scary topic and one attack that is apparently more common than I suspected. Before you go further, if you haven’t restored a database backup in the last month, stop and go verify your DR plan works. That’s one of the overconfident issues facing lots of government and businesses. While this might not help your entire organization, at least you’ll have some confidence in your process and that you can recover a database.
This is a great article from Ars Technica and worth reading: A take of two cities: Why ransomware will just get worse. I’d recommend you read it and think about a few things. First, do you have insurance because things (or substitute your own word here) happen? Second, have you really tested a DR plan for some sort of software issue like this? You might think about a way to restore systems in an air-gapped…
For those of you who are not regular readers of my ‘blog, SQL Saturday is a daylong conference centered mostly (but not entirely) around data topics related to SQL Server. It’s also a great networking event, and an opportunity to hook up with a number of data professionals! Check out the schedule to see what sessions interest you!
Additionally, there are three pre-con sessions on Friday, July 19. Unlike SQL Saturday, these sessions are not free, but they provide quality daylong training for specific topics at a decent price. Information about these pre-cons can also be found on the web site!
For more information and to register for the event, visit our website! Upstate New York is a great place to visit during the summertime! Hope to see you there!
As is the case with most SQL Saturday events, I had a chance to network and connect with a number of people. Most notably, I had the opportunity to meet Andy Leonard, who has written a number of books, writes frequently for SQLServerCentral, and is considered a rock star among SQL circles. I told him about my writing exploits, and he hooked me up with the editor at Apress publishing. Once I’ve had a chance to get everything settled and back into the normal routine after nearly a week away, I’ll have a conversation with him. Could a book be in my future? I’ve always dreamed about having my name on a book. We’ll see!
My own presentation went well, although I still think it could be better. This was the first time I’d given this presentation since I revamped my slides. I’ll have to see what feedback I received and use it to make the presentation even better.