On IT & Common Sense

I guess you could say I grew up in IT and it was a very accidental meeting. I'll thank my good buddy, Brad Pierce for that-- definitely a story unto itself, but for now lets just say that ever since the tender age of 18, I have been slogging away at something IT related and usually totally loving it. From troubleshooting broken this's and that's over the phone back in my Gateway days, to setting up rich folk, local companies and schools with "The Internet" waaaaaay back before anyone even know what the hell to do once they got "on" it during my Pionet/MCI days (hysterical really: "well, here you are! the internet!" "ok, now what do I do?" "uh, connect to your favorite BBS?" "huh?" "yeah") to some fun and some heartbreaking consulting in the late 90's, to my dream team in the latest of the 90s/early 2000's, to teaching technical classes all through the 2000's-- I could go on and on because, well-- I just could-- but let me tell you...there was one job that was not fun but it definitely taught me a lot. I used to always tell this story to my students and urged them to listen and not be the dumb yet somehow smart techno-weenie that hides in the corner doing more harm for his business than good.

It is Day Three and I am at a new job. I was hired as an architect to oversee a migration of 500 hospitals to a new point of sale application which also required a major OS rev.  The current and very old POS was limping along and did not have anywhere near the functionality they needed to process financials, offer new benefits, etc. These hospitals were franchised and the vets paid IT fees and they were peeved to be on such an archaic POS- it's not what they bought into.

It is important to note that they had already tried to upgrade the POS and failed two years before. Many heads rolled, all the way up to CIO and Sr. VP's and those who were left behind were deeply traumatized and many of them were angry at IT. The business felt it was an IT failure and they hated IT.

My new management and technical team assured me they had it dialed as we discussed the project in my interviews and in the first few days on the job. Something just felt off and I couldn't put my finger on it. One afternoon I was sitting in the Executive Steering Committee meeting for the project and they were discussing migration plans as the first migration of one test (yet 100% live) hospital was in two weeks. The VP on the business POS side, now a good friend, started off with her questions to my counterparts:

So, walk me through it. When does the migration begin?   Ok, well, immediately after the hospital closes, a team will come onsite--

A team? What? We don't have budget for a team to go to 500 sites! We've discussed this!

Oh, we don't? Um, ok. Well, we can connect remotely via (insert technobabble here) and uh, do the upgrades and uh-- Ok, how long will that take?  Uhhh....  You don't know?

The conversation continued and I was shocked. The business clenched up in an angry fist of questions and IT dodged and red-faced their way through it. Abysmal! The infrastructure group supporting server & desktop engineering had been working on this project for well over a year in earnest and they did not know the answers to any of these very cogent- very important questions. They didn't even know when the hospitals closed for the night and were unaware of the finance interfaces that had to run for ~thirty minutes updating the Mother Ship with its daily financials. Lordy. What a mess I was in.

I went home and immediately poured a glass of wine and thus dubbed a new and now well used saying: "It's a wine with shoes on day!" hence me clacking through the house direct upon my arrival with a mission only to find the wine bottle and pour a glass the size of my head. Tomorrow was a new day.

The first thing I did was pull them all into a room. I'd been there three whole days, mind you. I start out nice, asking questions and get poo poo'd by everyone in the place. It's fine...so-and-so is a dramatist...we've got a good solid plan. 

Ok. Show me.

I had them set up a test lab identical to a remote hospital. I brought in a stopwatch and had them conduct an end to end POS migration. 7 hours. Impossible! Actually, inconceivable! If the hospital closed at 10pm and had an hour set aside for financial processing, there just aren't enough hours in a day. What the hell where these guys thinking? Well, I guess they'll just have to close early one day. Yeah right. These hospitals were franchised and there was no way that was gonna fly especially with such little notice.

I was baffled. I couldn't understand why it would take so long to deploy the operating system + POS. A new server would be shipped to every hospital and the client side POS on the hard drive. The transfer would be local. Why the heck would it take so long? I then asked what would be a game changer of a question. "Are all of these hospitals on 10MB switches?" Yeah, they are. Why? Ok, well, we need to configure them to 100MB. What will that do for us? Um, carry the image ten times faster...AKA solve this problem. Sure enough, a 100MB switch took the entire transfer down to just over an hour. Sadly, my elation was soon shattered by the fact that none of the switches at the hospitals supported 100MB even under brute force config. They had bought super cheap. Barrier #1 discovered, averted and Barrier #2 now presented itself with six weeks to fix and a half million dollar price tag. It was 4 days into my new job and I had already booked my first meeting with the CIO and CFO to discuss this little "oversight".

I wish I could say it got easier from here. The switches were ordered and I now had to address a few other discoveries: none of the hardware at the hospitals was consistent. They had no inventory, didn't know what models were deployed and some of them were beyond ancient. They guessed at what was out there. I wrote a quick and dirty little inventory and threw it in the login script and within a day, started compiling a list of the crappiest hardware known to man. The upshot was this: the hardware itself would not support Windows XP and a new POS. They were screwed. Back to the CFO I went-- this time requesting another $1M for workstations and servers -- having discovered that the servers that were currently on purchase were the cheapest of the cheap and had ZERO redundancy. Sigh. Check in hand, literally, and another forest fire is being tended to and I am flying the helicopter and putting gas into it at the same time.

There is more- I could definitely go on and on with more examples but I won't. Every single issue that nearly prevented -- note I said nearly!-- this project from going live and being successful could have been avoided if this team had been better at planning, employed common sense and actually knew the business they worked for. They thought that just because they worked on the infrastructure side of the house that they didn't even need to understand the business. Man, they were so wrong and many of them lost their jobs as a result. It all boils down to common sense, folks. It is very rarely about being brilliant.

This company is not normal by far in that they are a huge growth story. Once a ma and pa veterinary company, they grew from 5 hospitals to 400 in a day after a serendipitous big-box retail partnership. Their problem was that IT did not scale along with the business. The business landscape had changed- big time- but IT was stuck in the past, never stopping to assess whether old processes fit the way the company was growing or how they needed to change to support what was now an enterprise. An example is buying the cheapest of the cheap of hardware thinking they would please their customer by not spending a lot of money when in fact all they were doing was hurting their customer.

This was not my last firefight at the vet behemoth. There would be many more. Eventually, the engineering team and much of IT reported to me and I was able to bring in a lot of new blood and build a really solid team. I was the first IT person to ever win the esteemed "Flying Pig" award for the greatest accomplishment of the year. A real pig with wings made of cast iron. Imagine my surprise ;-)

To this day, I enjoy a random happy hour with my old boss and still the CIO. Poor guy had only been on the job himself for about six months and went through a hell of a ride himself. So sometimes, we get a beer, laugh about the madness of that project and more...but we never say we miss those times.

One last word on this: I could not have survived this insane time without the help and aid of my friend and colleague at that time. Lorri Ely. Cheers to you!! We kicked some ass!