If Disaster Recovery were a Superhero?

 

The other day I was thinking back to when my son was about 3 or 4 years old and he started to mimic some of the Superheroes he saw in the movies and on television. He would put together all kinds of unique Superhero outfits, combined with various assorted make-believe weapons, but, over time he gravitated to one particular costume which always made us laugh.

Most mornings he would appear at his bedroom door with a red plastic helmet on his head, blue plastic swim goggles strapped tightly over his eyes, a long flowing black cape (think Darth Vader), various pajama shorts in assorted kid related patterns and knee-high black rubber boots. Of course, he was a very macho superhero and was adamant that a shirt would never be part of any respectable superhero’s ensemble, even when outside temperatures dipped below freezing.cool-goggle-28-image-girl-cool-goggle-boy-toddler-kid-anti-uv-kid-girl-boy-cool

I think our resident superhero began to gravitate to this particular outfit because he was one of those kids that always liked making people laugh and no matter how I tried, when he would appear in the above- mentioned Superhero garb, I would invariably laugh out loud or at minimum, his appearance always made me smile. The fact that I started calling him “Goggle Boy” only seemed to solidify his new Superhero identity. Both of our kids loved hearing bedtime stories and every once and awhile, I would regale them with the occasional Adventures of Goggle Boy stories.

This trip down Superhero memory lane got me thinking what kind of Superhero it takes to be a Backup and Disaster Recovery provider in today’s always-on credit union environment. Being a Disaster Recovery superhero to credit unions is not for the faint of heart and at times it requires nothing short of superhero intelligence, talent, grit and determination. Our twenty years of disaster recovery experience has taught us that no two recovery scenarios are the same and it takes tremendous teamwork to recover multiple IT systems and get them fully functional within the designated SLA’s, RTO’s and RPO’s. We think our employees are true Superheroes and one thing you can count on is that we will work non-stop and around the clock until all systems are up and fully functional… in true Superhero fashion!

If you’re in the teAoU_Iron_Man_Mk43_artchnology space and want to recruit some of the Marvel Mystery Superheroes, then Iron Man (Tony Stark) may be Team Disaster Recovery’s franchise player of choice.  After all, Tony Stark is an ordinary human who is highly skilled in the world of technology and science… actually, he’s one of the three smartest people on earth.  If Tony Stark can figure out how to fly, build an Arc Reactor on his chest and a Nano-tube armor suit, he should be pretty good at restoring servers, managing network infrastructures and keeping the whole IT environment safe and secure. Iron Man could even use his direct cyberpathic control over the entire telecom and satellite system to get the recovery process to where it’s really as easy as it looks to the technology novice.

The Black Panther (T’Challa) would be a welcome addition to Team Disaster Recovery, with his PhD in physics from Oxford University, and his extensive knowledge of all things technology, T’Challa and Tony Stark would make a potent technological team.  With Black Panther’s knowledge of advanced military technology, it would be easy to keep all that data safe and secure. Thanks to the Black Panther’s advanced psychic powers and shadow psychic weapons, any cyber-attacks would either be detected well in advance of the attack or quickly neutralized if the attack were to occur.

It’s common knowledge that the technology space isn’t exactly packed with members of the female persuasion, which means captain marvelwe should take look at Captain Marvel (Carol Danvers) and do our part to tip the scales that direction. Carol Danvers went directly from high school to the Air Force in order to pursue her love of flying and her interest in aircraft.  Because of her strong combat skills, stellar performance and natural intelligence she quickly rose to the rank of Major.  Shortly thereafter Carol was recruited by the CIA and later by N.A.S.A., as the Cape Canaveral Security Director. With her highly developed intellect and strong leadership skills, Captain Marvel may be just the team player to balance out all of that Superhero testosterone.

Speaking of testosterone, Thor is a Superhero that a lot of guys probably identify with.  After all he’s ruggedly handsome, possesses superhuman strength, is virtually immortal and gets to yield all the powers of Mjolnir’s Hammer.   Here’s the thing, Thor isn’t likely to make a great Disaster Recovery Superhero, since he has the ability to summon the destructive powers of the storm, wind, rain, thunder, and lightning.  Obviously, Thor’s super storm powers are what Disaster Recovery Superheroes are often working to mitigate.   Sorry Thor, you’re a cool dude, but Disaster Recovery and super storm powers… not a great match, but we’ll keep your resume on file, just in case we come up with a position that matches your unique skillsets.

While we’re in the process of elimination, it may be prudent to cross the Hulk off the candidate list.  In order to be a great disaster recovery Superhero, it helps to be cool as a cucumber with the ability to handle extreme pressure. Since the Hulk only appears when something totally ticks-off Bruce Banner, there is a good chance the Hulk would lose his cool under stress and end up tearing up the data center. Which means it’s probably best to keep Thor and Hulk on the sideline. With Thor and Hulk out of the picture, we’re left with Spiderman, Wolverine, Luke Cage, Black Widow, Daredevil, and Captain America.

Rather than making the remaining selections, perhaps you should decide who belongs on your Disaster Recovery Superhero Team? Here are the talents of the remaining candidates… who would you add to ensure that your data, IT systems and hardware receives Superhero recovery attention? The future of the planet may not be at stake, but the future of your credit union may hang in the balance… You decide!

Spiderman: Generally enhanced physiology, possible mystical connection, wall-crawling, superhuman strength, durability, healing factor, jumping, leaping, and speed, superhuman agility, reflexes and equilibrium, fighting style, Spider-sense.

Wolverine: Super-human senses and the power to heal from almost any wound, unbreakable skeleton and three retractable claws in each hand, exceptional hand-to-hand combatant, having mastered virtually every fighting style, trained expert in multiple types of weapons, vehicles, computer systems, explosives, and assassination techniques, fluent in many languages, including Japanese, Russian, Chinese, Cheyenne, Lakota, and Spanish; and some knowledge of French, Thai, and Vietnamese.

Captain America: Agility, strength, speed, endurance, and reaction time superior to any Olympic athlete who ever competed. The Super-Soldier formula that he has metabolized has enhanced all of his bodily functions to the peak of human efficiency. Notably, his body eliminates the excessive build-up of fatigue-producing poisons in his muscles, granting him phenomenal endurance. Master of the martial arts of American-style boxing and judo, and has combined these disciplines with his own unique hand-to-hand style of combat. Captain America is subject to all human vulnerabilities, although his immunity to diseases is extraordinary.  Captain America’s round shield was developed by Tony Stark (Iron Man) an is made of virtually indestructible Vibranium.

Luke Cage: Luke Cage possesses superhuman strength and stamina, and has extremely-dense skin and muscle tissue, which render him highly resistant to physical injury. Cage possesses these abilities as a result of a cellular regeneration experiment which fortified the various tissues of his body. His skin can resist high-caliber bullets, puncture wounds, corrosives, biological attacks, and extreme temperatures and pressures without sustaining damage. A second exposure to said experiments further enhanced his strength and durability.

Black Widow: World class athlete, gymnast, acrobat, aerialist capable of numerous complex maneuvers and feats, expert martial artist, marksman and weapons specialist as well as having extensive espionage training. Enhanced by biotechnology that makes her body resistant to aging and disease, as well as psychological conditioning that suppresses her memory of true events as opposed to implanted ones of the past without the aid of specially designed system suppressant drugs.

Daredevil: Daredevil is blind, and his remaining four senses function with superhuman accuracy and sensitivity, giving him abilities far beyond the limits of a sighted person. Daredevil developed a radar sense, which is similar to echolocation. Daredevil has no superhuman physical attributes beyond an enhanced sense of balance, but he is a master of martial art and hand-to-hand combat.

*Information was derived on various Marvel-centric websites including Marvel.com.


Is the Public Cloud Worthy of our Trust?

The Cloud as we all know it, has become such a massive reality in our daily lives that may seem a bit overwhelming at times.  For many people, the Cloud seems to hold a strange, almost magical mystique.  When discussions turn to the Cloud, there is sometimes a hushed reverence that permeates the conversation, something akin to prayer and worship.  For certain individuals, the Cloud evokes a nearly religious devotion, but is the Cloud worthy of such avid devotion or is the Cloud more of a flawed Deity, no less vulnerable than the humans who created it and continue to nurture it today?

Let’s take a quick look at the Cloud’s simple origins.   In its simplest form, the Cloud is merely a server or several servers, sitting in a data center somewhere and connected by intranet for private use or provided for public use via internet.  The Cloud Almighty has been in existence since January 1, 1983, when ARPANET adopted TCP/IP, which took on a more familiar form in 1990 when ARPANET was decommissioned and computer scientist Tim Berners-Lee was credited with inventing the World Wide Web.image - data center - cloud

A private cloud typically provides connectivity between two dedicated sites and is locked down for use by an organization.   Also known as an internal cloud, where all data is protected behind firewalls on the company’s intranet.  A private cloud is a common option for companies with more than one data center and all the hardware and components needed to create a cloud.  All maintenance and updating of infrastructure is the sole responsibility of the company.  Private clouds may offer an increased level of security and there is very little or no sharing of resources with other organizations.

The typical public cloud is a scenario where data is stored in a data center of a service provider and the provider is responsible for management and maintenance of the data center and all related functions.  More and more companies are moving toward the public cloud or a mixture of private/public options.  Some companies feel security may be lacking with the public cloud, however, breaches are rare and your data typically remains separate from others.

Smaller companies may tend to choose a public cloud in their effort to reduce maintenance costs, infrastructure expenses, OPEX and CAPEX.  Larger companies may be inclined to choose a private cloud to maintain greater control and an enhanced sense of security… whether real or perceived.

220px-Dictionnaire_Infernal_-_BehemothWhen it comes to Private or Public Clouds, there is still a preverbal elephant in the room.  This elephant looms large in the psyche of companies of any size, whether large or small.   Cloud Network Outages are huge lumbering Mammoths that represent a catastrophic event no company wants to experience.  Amazon Web Services (AWS), is another behemoth which is the dominant market player in the space.  The AWS idea was conceived as early as 2000, and while the AWS concept began to take shape and was publicly discussed in 2003, and the first customer facing launch took place in 2005. Those individuals religiously devoted to the Public Cloud often place AWS on a very tall pedestal and AWS enjoys an exalted position of respect and dominance in the public cloud arena, but not all is Roses and Tulips in the Kingdom of Cloud.  AWS continues to prick its fingers on the thorns of Network Outages.

The most recent AWS Network Outage occurred in the Northern Virginia region on the morning of February 28th, 2017, as the S3 Team was debugging an issue causing the S3 billing system to progress more slowly than expected.   An employee error took down a large swath of Amazon services for nearly 4 hours.  Another AWS Network Outage took place in Sydney, Australia in June 2016 as massive thunder storms caused AWS EC2 and EBS services to fail and a significant number of prime websites and other online presence were down for 10 hours over a weekend. Since AWS’s inception there have been 7 notable Network Outages.

What conclusions can be drawn about the Public Cloud from events like these?  Some might say that regardless of the problems that exist, there are few inventions that positively influence our lives so profoundly on a daily basis.  Others might say that events like these point to dangerous flaws in the systems that impact our lives and there is much to be concerned about.

Regardless of your perspective of all things Cloud and Internet, one thing is certain, both are here to stay and what the future holds may be significantly different than how it is imagined today.


Credit Unions Face Increasing Challenges in DIY Disaster Recovery

 

Credit unions face unique challenges with DIY disaster recovery operations, which are becoming increasingly complex due to the proliferation of numerous applications ancillary to the Core.   The growing list of ancillary and interdependent applications, disparate data stores, and third party services, are making the recovery process an increasingly daunting task.

Credit union members are becoming progressively more accustomed to instant data access and limited downtime.  Member expectations put increased pressure on IT departments to reduce downtime to near zero.  IT professionals understand the importance and impacts of downtime, yet a study by Forrester and Disaster Recovery Journal, in 2013 showed that median actual recovery times were 8 hours, up from 3 hours in 2010. 1  This trend can be directly correlated to the growing complexities of the disaster recovery process.

Many credit unions favor in-house DIY disaster recovery operations, but the decision to take on the burden of DIY disaster recovery often comes at a high cost. That burden is amplified by the cost of downtime, as noted in an Aberdeen survey of IT professionals in 2013 that found that the average cost of downtime per hour across companies of all sizes was a staggering $163,674. 2  In addition to the complexity and increasing financial burden, DIY disaster recovery results often fall short of desired outcomes.

One of the reasons DIY disaster recovery results fall short of expectations is the realization that credit union IT staff continually face long lists of projects, which demand a tremendous amount of time and attention.  When faced with continual project management, it’s not uncommon for IT to allow backup and recovery objectives to become a lower priority and the impact on recovery results are bound to be less than ideal.

Virtualization and the recovery of virtual machines has simplified certain aspects of recovery, however credit union staff may be lulled into a false sense of security. Advances in backup technologies and virtualization are also leading many credit unions to invest heavily in secondary sites or colocation.  However, the CAPEX required to implement secondary sites or colocation is not only significant, it requires refresh cycles that ultimately become cost prohibitive over time.

Recent enhancements in backup and recovery technologies appear to make recovery seem easy, when in reality, the opposite is true.  Technology offers great promise and certain aspects of recovery are more dependable than in the past, yet there are still problems:

  • Credit unions place a heavy emphasis on daily operations and the maintenance of vital functions, but most spend limited time testing recovery procedures and reviewing the logistical necessities of a viable recovery system. Daily maintenance of interdependent systems is an entirely different skill set than the expertise and knowledge required to facilitate a functional recovery of those systems. Nor do most take into consideration the expertise required to facilitate recovery of multiple systems with inter-dependencies distributed throughout disparate tiers within the data center.   In other words, emphasis and expenditures focus on daily operations and maintenance, and only a small percentage is dedicated to disaster recovery.  Fully 23% of organizations admit they never test. 3
  • The DIY approach to recovery processes can heighten the risk of improperly vetting the complex inter-dependencies between multiple, if not dozens of applications that run the credit union. Credit unions often operate with lean IT departments and lack the staffing depth necessary to properly manage the complexities of today’s backup and recovery functions.  The development and management of growth initiatives further exacerbate staffing pressures, which result in the neglect of backup and recovery priorities.
  • Typical DIY recovery environments often lack the tools and the mindset necessary to discover shadow or forgotten applications that reside within the IT infrastructure. Even applications and inter-dependencies residing in plain sight may be overlooked in disaster recovery implementation.

Why DIY Recovery Falls Short

  • The majority of business applications are interdependent upon one another. Facilitating full recovery of all business processes from end-to-end requires more than simply storing and accessing applications and data. All components, including proper OS selection, current application selection and most current data set must be incorporated in the recovery environment.  Even achieving recovery of proper applications and data may leave critical components out of the process. Overlooking a seemingly innocuous interdependent component will hinder or prevent end-to-end recovery.
  • Credit unions utilizing colocation or secondary sites face the very real challenge of ensuring that properly qualified IT staff will be able to get to the secondary site in a disaster scenario. Many credit unions run with lean IT departments and it can be impractical or extremely challenging to get properly trained people to the secondary site.  Simply getting staff to the secondary site doesn’t ensure recovery, staff members must have the necessary knowledge and skill sets to guarantee a successful end-to-end recovery.
  • For the most part, IT managers are proficient at managing integration of technology to ensure all systems work together smoothly and at top efficiency. To reliably recover your applications, data and interdependent systems, you must duplicate the precise mix of servers, storage, operating systems, hypervisors, networks and software.  You must also manage any and all changes taking place within that environment and change management must take place constantly. The unavailability of even one component or application can trickle down to impact a wide array of business functions.
  • DIY backup and recovery using colocation or a secondary site typically requires duplication of everything in use at the primary site. All servers and storage must be replicated or mirrored at the secondary location and both sites must have adequate bandwidth and networking infrastructures.  Licensing requirements will be duplicated, as will most security measures.  Staff members will be required to put in extra hours to manage the secondary location or additional staffing will be required.   Costs for the recovery infrastructure can be significant and should be expected to increase every 3 to 4 years in conjunction with hardware and software refresh cycles.
  • Misalignment of Recovery Point Objectives (RPO’s) and Recovery Time Objectives (RTO’s) with Recovery Methods (RM’s) is a common and costly occurrence. When it comes to RPO’s and RTO’s, the concept of “one size fits all” is a dangerous miscalculation.  Credit unions practicing this mindset will quickly discover that the incident that caused an initial system failure isn’t the only disaster they will encounter. Misalignment or miscalculations of RPO’s and RTO’s with an associated RM will quickly short-circuit or prevent end-to-end recovery capabilities.

Viable Alternatives to DIY Disaster Recovery

  • Credit unions are founded on the principles of a cooperative and shared business model. Backup and Recovery as a Service (BRaaS) is the model for cooperative sharing of technical knowledge, IT infrastructure, hardware, software, and resources. Managed Service Providers (MSP’s) provide the infrastructure, experience and knowledge for BRaaS at a fraction of the cost of DIY backup and recovery models.   BRaaS is also recognized as a readily sustainable business model.
  • A primary benefit of BRaaS is the deployment of highly qualified and efficient resources in a shared environment. Distribution of shared resources eliminates the significant outlay of CAPEX, while amortizing OPEX in a simple pay-as-you-grow model.  BRaaS leverages cutting edge infrastructure along with extensive knowledge and experience, to deliver true business resilience.
  • Core system vendors often focus recovery efforts on the Core system specifically and may be inclined to neglect critical inter-dependencies necessary to provide end-to-end recovery and resiliency. However, a select number of MSP’s are proficient at recovering the core and all ancillary servers with related dependencies. Third party vendor connections to ATM’s, mobile banking, internet banking and the FED are typically provided as well. Qualified MSP’s provide true end-to-end business recovery while eliminating the expense and frustrations of DIY endeavors.

What to look for in a BRaaS solution and vendor.

Ask if the service provider:

  • Performs Backup and Recovery of the Core and all Ancillary Servers
  • Provides Connectivity to Third Party Vendors, such as: ATM, Mobile and Internet Banking, FED, etc.
  • Performs Infrastructure and Network Discovery Analysis
  • Manages Backup and Recovery Resources and Procedures
  • Monitors Alerts for Potential Complications or Issues
  • Suggests Technical Improvements and Implementation Processes
  • Assists With Systems Maintenance to Resolve Backup and Recovery Challenges
  • Identifies and Tracks Inter-dependencies and Application Road-maps
  • Matches RTO’s and RPO’s With Appropriate Recovery Methods
  • Performs, Manages and Maintains Restore/Recovery Procedures
  • Initiates Pre-test Discovery, Meetings and Planning
  • Tracks, Monitors and Documents DR Test and Results
  • Conducts Post-Test Reviews, Remediation and Gap Analysis
  • Initiates Change Discovery and Management
  • Maintains and Stores Backup and Recovery Procedural Docs and Configurations
  • Provides Timely Reporting of all Vital Backup & Recovery Elements
  • Provides Relevant Case Studies and Use Examples
  • Provides Credible References and Testimonials.

 

1  DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014

2  Aberdeen Group, Downtime and Data Loss: How Much Can You Afford? August 2013

3  DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014