Communication lessons from air disasters

I’ve always had a morbid fascination for air disasters.  (Don’t ask me why; I have no idea.)  I’m fascinated by shows such as Air Disasters, Why Planes Crash,  and Mayday: Air Disasters.  Whenever I hear about a plane going down, I’ll start thinking about what happened, clues, what might have caused it, and so on.  There are times when I think I should have gotten a job with the NTSB.

Greg Moore has some publications in which he talks about lessons learned from aircraft accidents; his book partially discusses these lessons.  He also has an excellent SQL Saturday presentation titled “Who’s Flying The Plane?” which talks about lessons learned from air disasters and how they can apply to IT.  Go check it out if you have a chance; Greg gives a great presentation!

For the purposes of this article, however, I want to concentrate on a particular topic: how communication — or, the lack of — either contributed to or was the root cause of a disaster.

Last night, I watched an episode of Air Disasters that talked about the plane crash that took the life of professional golfer Payne Stewart. The plane went down after the cabin depressurized (the cause of which was never determined), the crew became incapacitated, and the plane ran out of fuel. What made it interesting to me was that bad documentation might have been one of the contributing factors to the accident. After the cabin lost pressure, the crew likely consulted a checklist, as is standard procedure for nearly any cockpit activity or incident. The checklist was poorly written and unclear. What should have been the very first instruction was, “put on your oxygen mask.” It was not. By the time the crew got to that instruction, it was too late; they were overcome by hypoxia.

It reminded me of a tenet that I preach in my documentation presentation: if, in a step-by-step instruction, an instruction cannot be understood within a few seconds, it has failed.

I also remember another Air Disasters episode that focused on Avianca Flight 52.  In January of 1990, the plane, a Boeing 707 carrying 158 people, crashed on approach to Kennedy Airport in New York after running out of fuel, killing 73 people.  There were numerous communication issues during the flight.  Had any one of them been addressed, chances are the disaster never would have occurred.

How often have you been involved in some kind of activity where things were miscommunicated?  How well did those activities go?  I’m guessing that they didn’t go well.  How often have they happened when deadlines were approaching?  What was the mood of your organization?  I’ll guess that it was likely one of high stress and low morale.  And during that time, how smoothly did things go?  Probably not very.  I’ll bet that plenty of mistakes were made.

I’m painting this picture within a business environment.  Imagine what these conditions are like when people’s lives are at stake.

The number of disasters that have occurred from poor communication are countless; entire studies have been dedicated to the subject.  Indeed, numerous solutions and subcategories related to miscommunication have been devised.  The airline industry developed the process of crew resource management.  Extensive research has been done on the phenomenon known as groupthinkEven simple measures such as checklists have been studied and implemented.

The moral of the story: good communication, including documentation, is critical. The consequences of it can have adverse effects. At best, bad communication can disrupt your business. At worst, it can cost lives.

Expect the Unexpected with DiRT

Steve‘s article reminded me about the first time I gave my Disaster Documents presentation at a SQL Saturday.

At the end of my presentation, one attendee started an argument with me. He kept saying that paper was dead, everything was online, and there was no reason to keep hardcopy documents. I argued, what if you can’t get to your online documentation?

Not surprisingly, he gave me a poor evaluation.

The bottom line is this: even documentation needs a backup. Other than, say, getting lost in a fire, paper documents can’t break. At a minimum, have hardcopy documents that instruct how to get minimal services back up and running, and back up other recovery documentation so you can recover it later.

Voice of the DBA

Disaster recovery is one of the core tasks that many DBAs think about on a regular basis. Ensuring that we can get our data back online, available, accessible, and intact is important. More than a few DBAs that haven’t been able to recover systems, find themselves seeking new employment.

That’s not to say that most DBAs perform perfectly under pressure. Plenty make mistakes, and there may be times when they can’t recover all data. There does seem to be a correlation between how often DBAs practice recovery skills and how well they perform in an actual emergency. I know that at a few companies, we scheduled regular disaster tests, though often with simulated recovery of a systems that didn’t expect to actually take over a workload. Arguably not a good test, but better than nothing.

Google takes things a step further. They have annual, company wide, multi-day DiRT (Disaster Recovery…

View original post 359 more words

Ransomware and DevOps

Another post by Steve Jones that I think is really important…

Voice of the DBA

Ransomware.

A scary topic and one attack that is apparently more common than I suspected. Before you go further, if you haven’t restored a database backup in the last month, stop and go verify your DR plan works. That’s one of the overconfident issues facing lots of government and businesses. While this might not help your entire organization, at least you’ll have some confidence in your process and that you can recover a database.

This is a great article from Ars Technica and worth reading: A take of two cities: Why ransomware will just get worse. I’d recommend you read it and think about a few things. First, do you have insurance because things (or substitute your own word here) happen? Second, have you really tested a DR plan for some sort of software issue like this? You might think about a way to restore systems in an air-gapped…

View original post 356 more words

July 20 — SQL Saturday, Albany, NY

On Saturday, July 20 (one week from tomorrow), the Capital Area SQL Server User Group (CASSUG) will host SQL Saturday for the sixth time in Albany, NY!

For those of you who are not regular readers of my ‘blog, SQL Saturday is a daylong conference centered mostly (but not entirely) around data topics related to SQL Server. It’s also a great networking event, and an opportunity to hook up with a number of data professionals! Check out the schedule to see what sessions interest you!

And yes, I am presenting, too! I will do a brand-new presentation about ‘blogging, as well as a lightning talk about business cards! I always look forward to doing presentations in my own backyard!

Additionally, there are three pre-con sessions on Friday, July 19. Unlike SQL Saturday, these sessions are not free, but they provide quality daylong training for specific topics at a decent price. Information about these pre-cons can also be found on the web site!

For more information and to register for the event, visit our website! Upstate New York is a great place to visit during the summertime! Hope to see you there!

SQL Saturday Virginia Beach — the debrief

My wife and I arrived home in Troy, NY around 10 pm last night after driving all day from Norfolk, VA. Despite the lengthy drive home from Virginia, it was, nevertheless, a fun and fruitful trip!

First, Greg Moore, who also attended SQL Saturday #839, summed up his assessment of the event in his article here. Go ahead and read his article. I don’t think I could’ve summed it up much better than he did.

As is the case with most SQL Saturday events, I had a chance to network and connect with a number of people. Most notably, I had the opportunity to meet Andy Leonard, who has written a number of books, writes frequently for SQLServerCentral, and is considered a rock star among SQL circles. I told him about my writing exploits, and he hooked me up with the editor at Apress publishing. Once I’ve had a chance to get everything settled and back into the normal routine after nearly a week away, I’ll have a conversation with him. Could a book be in my future? I’ve always dreamed about having my name on a book. We’ll see!

My own presentation went well, although I still think it could be better. This was the first time I’d given this presentation since I revamped my slides. I’ll have to see what feedback I received and use it to make the presentation even better.

I met a fellow fraternity brother who recognized the letters on the hat I was wearing!

I also had another textbook example about how your clothing can initiate a networking opportunity! On Friday, my wife and I were enjoying the Norfolk Harborfest. While walking through the park, a guy recognized the letters on the baseball cap I was wearing, and came over to say hi! He was a fellow fraternity brother from the chapter at Norfolk State University. We took the photo you see above. Always a pleasure to meet another fraternity brother!

The rest of the trip was a great time! I knew that there was a plethora of activities around Norfolk/Virginia Beach, and suggested to my wife that we make a vacation out of it. We hit Colonial Williamsburg, the Chesapeake Bay Bridge-Tunnel, and the beach while we were there.

Although it was a lengthy trip, it was a good one! I look forward to doing it again at some point!

I’m speaking at SQL Saturday Virginia Beach — June 8

Saturday, June 8 is coming soon (a week from this Saturday). How will you be spending it?

For me, I will be speaking at SQL Saturday #839, Virginia Beach that day. I will be doing my presentation entitled: “Disaster Documents: The role of documentation in disaster recovery.” In my presentation, I tell the story of my company and my workgroup — which had an office in the World Trade Center on September 11, 2001. I had written several documents for our organization, and they ended up being critical to our recovery.

Come on out and check out my presentation. I can always use a good audience. Hope to see you there!

SQL Saturday Virginia Beach — I’m speaking June 8!

I just found out today that I will be speaking at SQL Saturday #839, Virginia Beach on June 8!

I will be doing my presentation about the role of documentation in disaster recovery!

I’ll post about this event again as we get closer to the date. See you there in early June!

Is Your DR Plan Complete?

Here’s another article reblog, this time from my friend, Andy Levy. Disaster recovery is a big deal, and you need to make sure that you’re prepared.

Don’t think a disaster can’t happen to you? Well, it happened to me.

The Rest is Just Code

Kevin Hill (b|t) posted a thought-provoking item on his last week about Disaster Recovery Plans. While I am in the 10% who perform DR tests for basic functionality on a regular basis, there’s a lot more to being prepared for disaster than just making sure you can get the databases back online.

You really need to have a full-company business continuity plan (BCP), which your DR plan is an integral portion of. Here come the Boy Scouts chanting “Be Prepared!”

When disaster strikes:

  • How will you communicate it to your customers, including regular status updates?
  • How will you communicate within the company?
  • Do you have your systems prioritized so that you know what order things have to be brought online? Which systems can lag by a day or two while you get the most critical things online?
  • Do you have contingency plans for all of…

View original post 543 more words