Lessons Learned - The Hard Way
I promised this blog wouldn’t be just about all the great things we do and all the knowledge we gained and can share with you. It’s going to be about our mistakes, too. So here’s one:
About a year ago, we started to work with a new customer, the CIO (let's call him 'Roy') of an organization that provides data repository services for regional hospitals and healthcare providers. Roy's project involved the creation of a global data replication system that included storage for a PACS system in the main data center, with a backup system for the imaging files in a disaster recovery site.
We began the sales process, like we always do, by understanding the environment, objectives, budget and growth requirements, along with many other factors associated with a well thought-out project planning process. We then designed a solution based on the latest software release of one of our storage vendors, which although in beta, happened to address all the requirements of this particular project. We communicated to Roy that our solution was a beta revision of software, and offered some financial incentives to him for agreeing to be a beta user. We articulated the benefits of the solution and explained the main principals of this architecture, the pros and cons, etc. Everything was according to plan.
Everything? Did I mention risk? We’ve been in business for quite sometime, seen many beta installations, and have come to have certain expectations of them. Typically, you may find some minor GUI glitches, perhaps some small bugs, and sometimes some other non-critical issues that can be resolved almost instantaneously by the vendor as soon as they are identified.
Well, this installation was a disaster! Nothing worked correctly, and we made numerous mistakes in trying to rectify the situation, perhaps even making matters worse. It took us about two months to stabilize the main data center site, and another ten (yes, 10!) months to get to the point where we were 100% confident that this ‘new’ software release worked as promised.
You would be correct in assuming that Roy did not want to take another risk until we had stabilized and tested this new software revision with numerous customers, so we held off on implementing the installation at his remote data replication site during this time. For nearly a year, Roy had only a single instance of the repository of his images, albeit on a RAID-5 array.
I’ve stayed in touch with Roy throughout this entire year, calling or emailing him almost every other week. I have kept him up to date on the vendor’s progress with the software and the status of other implementations using this feature. I also set up several conference calls with Roy and the vendor so we could all set new expectations, re-define the objectives and agree on our ‘next steps’, making sure all three parties were always in full agreement.
Needless to say, I’m not proud of this project! Roy – this is not the first time you’ve heard this from me, but I’m terribly, tremendously, dreadfully sorry for this experience! If I had ever suspected that this could be the outcome of our experience, I would have never put you through this.
SORRY!
Lessons Learned - The Hard Way:
1. Avoid being a beta customer, unless risks are low and financial gains are high.
2. Testing is important. When the stakes are high, do more testing upfront.
3. If possible, leave yourself a contractual ‘escape’ route to mitigate the risks of a newer technology.
4. Speak to references that are using the same technology for the same application.
5. Speak with references using the same people/resources. Validate that the human factor is trustworthy and knowledgeable.
6. If you happen to be the exception, make sure you re-write your ‘contract’. It is important to redefine objectives, timelines and financial gains/penalties throughout the experience.
Exceptions happen to everyone! What we at RADirect do with them and how we use them to improve our internal processes and decrease our bad incident rate, I’d like to think, sets us apart.