If Disaster Recovery were a Superhero?

 

The other day I was thinking back to when my son was about 3 or 4 years old and he started to mimic some of the Superheroes he saw in the movies and on television. He would put together all kinds of unique Superhero outfits, combined with various assorted make-believe weapons, but, over time he gravitated to one particular costume which always made us laugh.

Most mornings he would appear at his bedroom door with a red plastic helmet on his head, blue plastic swim goggles strapped tightly over his eyes, a long flowing black cape (think Darth Vader), various pajama shorts in assorted kid related patterns and knee-high black rubber boots. Of course, he was a very macho superhero and was adamant that a shirt would never be part of any respectable superhero’s ensemble, even when outside temperatures dipped below freezing.cool-goggle-28-image-girl-cool-goggle-boy-toddler-kid-anti-uv-kid-girl-boy-cool

I think our resident superhero began to gravitate to this particular outfit because he was one of those kids that always liked making people laugh and no matter how I tried, when he would appear in the above- mentioned Superhero garb, I would invariably laugh out loud or at minimum, his appearance always made me smile. The fact that I started calling him “Goggle Boy” only seemed to solidify his new Superhero identity. Both of our kids loved hearing bedtime stories and every once and awhile, I would regale them with the occasional Adventures of Goggle Boy stories.

This trip down Superhero memory lane got me thinking what kind of Superhero it takes to be a Backup and Disaster Recovery provider in today’s always-on credit union environment. Being a Disaster Recovery superhero to credit unions is not for the faint of heart and at times it requires nothing short of superhero intelligence, talent, grit and determination. Our twenty years of disaster recovery experience has taught us that no two recovery scenarios are the same and it takes tremendous teamwork to recover multiple IT systems and get them fully functional within the designated SLA’s, RTO’s and RPO’s. We think our employees are true Superheroes and one thing you can count on is that we will work non-stop and around the clock until all systems are up and fully functional… in true Superhero fashion!

If you’re in the teAoU_Iron_Man_Mk43_artchnology space and want to recruit some of the Marvel Mystery Superheroes, then Iron Man (Tony Stark) may be Team Disaster Recovery’s franchise player of choice.  After all, Tony Stark is an ordinary human who is highly skilled in the world of technology and science… actually, he’s one of the three smartest people on earth.  If Tony Stark can figure out how to fly, build an Arc Reactor on his chest and a Nano-tube armor suit, he should be pretty good at restoring servers, managing network infrastructures and keeping the whole IT environment safe and secure. Iron Man could even use his direct cyberpathic control over the entire telecom and satellite system to get the recovery process to where it’s really as easy as it looks to the technology novice.

The Black Panther (T’Challa) would be a welcome addition to Team Disaster Recovery, with his PhD in physics from Oxford University, and his extensive knowledge of all things technology, T’Challa and Tony Stark would make a potent technological team.  With Black Panther’s knowledge of advanced military technology, it would be easy to keep all that data safe and secure. Thanks to the Black Panther’s advanced psychic powers and shadow psychic weapons, any cyber-attacks would either be detected well in advance of the attack or quickly neutralized if the attack were to occur.

It’s common knowledge that the technology space isn’t exactly packed with members of the female persuasion, which means captain marvelwe should take look at Captain Marvel (Carol Danvers) and do our part to tip the scales that direction. Carol Danvers went directly from high school to the Air Force in order to pursue her love of flying and her interest in aircraft.  Because of her strong combat skills, stellar performance and natural intelligence she quickly rose to the rank of Major.  Shortly thereafter Carol was recruited by the CIA and later by N.A.S.A., as the Cape Canaveral Security Director. With her highly developed intellect and strong leadership skills, Captain Marvel may be just the team player to balance out all of that Superhero testosterone.

Speaking of testosterone, Thor is a Superhero that a lot of guys probably identify with.  After all he’s ruggedly handsome, possesses superhuman strength, is virtually immortal and gets to yield all the powers of Mjolnir’s Hammer.   Here’s the thing, Thor isn’t likely to make a great Disaster Recovery Superhero, since he has the ability to summon the destructive powers of the storm, wind, rain, thunder, and lightning.  Obviously, Thor’s super storm powers are what Disaster Recovery Superheroes are often working to mitigate.   Sorry Thor, you’re a cool dude, but Disaster Recovery and super storm powers… not a great match, but we’ll keep your resume on file, just in case we come up with a position that matches your unique skillsets.

While we’re in the process of elimination, it may be prudent to cross the Hulk off the candidate list.  In order to be a great disaster recovery Superhero, it helps to be cool as a cucumber with the ability to handle extreme pressure. Since the Hulk only appears when something totally ticks-off Bruce Banner, there is a good chance the Hulk would lose his cool under stress and end up tearing up the data center. Which means it’s probably best to keep Thor and Hulk on the sideline. With Thor and Hulk out of the picture, we’re left with Spiderman, Wolverine, Luke Cage, Black Widow, Daredevil, and Captain America.

Rather than making the remaining selections, perhaps you should decide who belongs on your Disaster Recovery Superhero Team? Here are the talents of the remaining candidates… who would you add to ensure that your data, IT systems and hardware receives Superhero recovery attention? The future of the planet may not be at stake, but the future of your credit union may hang in the balance… You decide!

Spiderman: Generally enhanced physiology, possible mystical connection, wall-crawling, superhuman strength, durability, healing factor, jumping, leaping, and speed, superhuman agility, reflexes and equilibrium, fighting style, Spider-sense.

Wolverine: Super-human senses and the power to heal from almost any wound, unbreakable skeleton and three retractable claws in each hand, exceptional hand-to-hand combatant, having mastered virtually every fighting style, trained expert in multiple types of weapons, vehicles, computer systems, explosives, and assassination techniques, fluent in many languages, including Japanese, Russian, Chinese, Cheyenne, Lakota, and Spanish; and some knowledge of French, Thai, and Vietnamese.

Captain America: Agility, strength, speed, endurance, and reaction time superior to any Olympic athlete who ever competed. The Super-Soldier formula that he has metabolized has enhanced all of his bodily functions to the peak of human efficiency. Notably, his body eliminates the excessive build-up of fatigue-producing poisons in his muscles, granting him phenomenal endurance. Master of the martial arts of American-style boxing and judo, and has combined these disciplines with his own unique hand-to-hand style of combat. Captain America is subject to all human vulnerabilities, although his immunity to diseases is extraordinary.  Captain America’s round shield was developed by Tony Stark (Iron Man) an is made of virtually indestructible Vibranium.

Luke Cage: Luke Cage possesses superhuman strength and stamina, and has extremely-dense skin and muscle tissue, which render him highly resistant to physical injury. Cage possesses these abilities as a result of a cellular regeneration experiment which fortified the various tissues of his body. His skin can resist high-caliber bullets, puncture wounds, corrosives, biological attacks, and extreme temperatures and pressures without sustaining damage. A second exposure to said experiments further enhanced his strength and durability.

Black Widow: World class athlete, gymnast, acrobat, aerialist capable of numerous complex maneuvers and feats, expert martial artist, marksman and weapons specialist as well as having extensive espionage training. Enhanced by biotechnology that makes her body resistant to aging and disease, as well as psychological conditioning that suppresses her memory of true events as opposed to implanted ones of the past without the aid of specially designed system suppressant drugs.

Daredevil: Daredevil is blind, and his remaining four senses function with superhuman accuracy and sensitivity, giving him abilities far beyond the limits of a sighted person. Daredevil developed a radar sense, which is similar to echolocation. Daredevil has no superhuman physical attributes beyond an enhanced sense of balance, but he is a master of martial art and hand-to-hand combat.

*Information was derived on various Marvel-centric websites including Marvel.com.


Is the Public Cloud Worthy of our Trust?

The Cloud as we all know it, has become such a massive reality in our daily lives that may seem a bit overwhelming at times.  For many people, the Cloud seems to hold a strange, almost magical mystique.  When discussions turn to the Cloud, there is sometimes a hushed reverence that permeates the conversation, something akin to prayer and worship.  For certain individuals, the Cloud evokes a nearly religious devotion, but is the Cloud worthy of such avid devotion or is the Cloud more of a flawed Deity, no less vulnerable than the humans who created it and continue to nurture it today?

Let’s take a quick look at the Cloud’s simple origins.   In its simplest form, the Cloud is merely a server or several servers, sitting in a data center somewhere and connected by intranet for private use or provided for public use via internet.  The Cloud Almighty has been in existence since January 1, 1983, when ARPANET adopted TCP/IP, which took on a more familiar form in 1990 when ARPANET was decommissioned and computer scientist Tim Berners-Lee was credited with inventing the World Wide Web.image - data center - cloud

A private cloud typically provides connectivity between two dedicated sites and is locked down for use by an organization.   Also known as an internal cloud, where all data is protected behind firewalls on the company’s intranet.  A private cloud is a common option for companies with more than one data center and all the hardware and components needed to create a cloud.  All maintenance and updating of infrastructure is the sole responsibility of the company.  Private clouds may offer an increased level of security and there is very little or no sharing of resources with other organizations.

The typical public cloud is a scenario where data is stored in a data center of a service provider and the provider is responsible for management and maintenance of the data center and all related functions.  More and more companies are moving toward the public cloud or a mixture of private/public options.  Some companies feel security may be lacking with the public cloud, however, breaches are rare and your data typically remains separate from others.

Smaller companies may tend to choose a public cloud in their effort to reduce maintenance costs, infrastructure expenses, OPEX and CAPEX.  Larger companies may be inclined to choose a private cloud to maintain greater control and an enhanced sense of security… whether real or perceived.

220px-Dictionnaire_Infernal_-_BehemothWhen it comes to Private or Public Clouds, there is still a preverbal elephant in the room.  This elephant looms large in the psyche of companies of any size, whether large or small.   Cloud Network Outages are huge lumbering Mammoths that represent a catastrophic event no company wants to experience.  Amazon Web Services (AWS), is another behemoth which is the dominant market player in the space.  The AWS idea was conceived as early as 2000, and while the AWS concept began to take shape and was publicly discussed in 2003, and the first customer facing launch took place in 2005. Those individuals religiously devoted to the Public Cloud often place AWS on a very tall pedestal and AWS enjoys an exalted position of respect and dominance in the public cloud arena, but not all is Roses and Tulips in the Kingdom of Cloud.  AWS continues to prick its fingers on the thorns of Network Outages.

The most recent AWS Network Outage occurred in the Northern Virginia region on the morning of February 28th, 2017, as the S3 Team was debugging an issue causing the S3 billing system to progress more slowly than expected.   An employee error took down a large swath of Amazon services for nearly 4 hours.  Another AWS Network Outage took place in Sydney, Australia in June 2016 as massive thunder storms caused AWS EC2 and EBS services to fail and a significant number of prime websites and other online presence were down for 10 hours over a weekend. Since AWS’s inception there have been 7 notable Network Outages.

What conclusions can be drawn about the Public Cloud from events like these?  Some might say that regardless of the problems that exist, there are few inventions that positively influence our lives so profoundly on a daily basis.  Others might say that events like these point to dangerous flaws in the systems that impact our lives and there is much to be concerned about.

Regardless of your perspective of all things Cloud and Internet, one thing is certain, both are here to stay and what the future holds may be significantly different than how it is imagined today.


All Howl-ows…Tide?

 

IMG_0438 copy 2Every October, a large segment of our population is simply enthralled with having the living spook scared out of them and with Halloween rapidly approaching, we thought it would be ghoulishly appropriate to share some frightening fun facts about our fascination with All Hallows’ Eve.

Halloween is believed to have originated in Ireland with the ancient Celtic Festival known as Samhain (pronounced säwėn), which is celebrated on November 1st. However, the night before Samhain, (October 31) the Celtic people believed that the dead returned as Ghosts to roam the countryside. Villagers left food and wine on their doorsteps to keep the Ghosts at bay, and when the villagers left their homes, they wore masks so the dead would mistake them for fellow Ghosts.

In the 8th Century, the Christian Church turned Samhain into All Saints Day. October 31, or All Saints Eve had evolved into Halloween or Hallowe’en, also known as Allhalloween or All Hollows’ Eve. Observances encompass All Saints’ Eve (Halloween), All Saints Day (All Hallows) and All Souls’ Day which last from October 31 to November 2 annually. Each of these observances stem from Allhallowtide, which is a time to remember the dead, including martyrs, saints and all faithful departed Christians.

In Medieval Britain, the tradition of “Souling” began on All Souls Day (November 2nd) in which the needy would beg for pastry know as soul cakes and in return they would pray for people’s dead relatives. As time passed, the practice of “Souling” evolved into “Guising” where young people would dress up in costume and accept food, wine, money, and other offerings in exchange for singing, reciting poetry or telling stories or jokes. In the 19th Century, Irish immigrants instituted the tradition of dressing up in costume in America. In the 1950’s the tradition of Trick or Treating went mainstream with a whole new generation.

According to the National Retail Federation, Halloween is the second highest grossing holiday after Christmas and Nielson Research reports that nearly 600-million pounds of candy is purchased each Halloween. Halloween spending also extends to costume purchases of nearly $2.6 billion… adult costume purchases rack up to nearly $1.22 billion, kids costumes $1.04 billion, and millions are spent each year on pet costumes. Let’s not forget all the life-size skeletons, blow-up monsters, fake cob webs, mantle pieces and other scary decorations, which average around $1.96 billion annually. We spend approximately $360 million on Halloween related greeting cards and there is an annual spike in alcohol purchases in the days preceding Halloween.

Want to have a little spooky fun? Try these Halloween related activities:

  • Halloween Name Generators:

http://en.vonvon.me/quiz/3684?utm_viral=2

https://fun.namerobot.com/name/halloween

http://witch.namegeneratorfun.com/

  • Not Too Scary Stories for Kids:

http://www.sheknows.com/parenting/articles/1016713/scary-halloween-stories-for-kids

  • Best Horror Movies of 2017

http://www.esquire.com/entertainment/movies/a56573/best-horror-movies-2017/

  • Best Horror Podcasts

https://www.thrillist.com/entertainment/nation/best-scary-podcasts-horror

  • Pinterest Best Halloween Pranks

https://www.pinterest.com/explore/halloween-pranks/?lp=true


No Easy Button For Disaster Recovery: CU Edition

We know that people would rather not think about bad things happening, much less the gazillion details required to ensure that credit unions stay open for business during a crisis, but sadly… it is a matter of when, not if.

At IMS, our corporate motto is “We Know Credit Unions” and when it comes to disaster recovery (DR), most CU’s place data protection high on their priority list.  However, nineteen years as a Managed Service Provider (MSP) for financial institutions of all shapes and sizes, has taught us that while CU’s place a high value on DR, making headway toward reliable disaster recovery initiatives often ends up on the back burner.

Another commonality among CU’s is that many operate with lean IT staffing.  When you consider all the projects and day-to-day maintenance involved in keeping a credit union running smoothly, it’s no mystery that disaster recovery tends to slide to the bottom of the priority list.

Statistically, it is not the massive natural disaster that poses the greatest risk to credit unions, it is simple human error or hardware failure that can cause daily operations to come to a screeching  halt.

Human error is the single most frequent cause of business and IT disruptions, with the most significant economic impact.  –  Ponemon Institute – Cost of Data Center Outages, Jan 2016

Credit unions are wholly unique in the fact that many are smaller institutions with limited resources to allocate to IT related demands, and yet, they deploy systems that are more diverse and complicated than those of a typical business.  The addition of the mandatory compliance requirements of financial institutions and the fact that all are prime targets of cyber criminals, creates additional burdens for IT staff.  These unique factors cause a significant gap between what is required to properly maintain and secure the IT environment and the resources available to do the job effectively.

The reality for many CU’s is they may not have the resources available to provide the protections required to ensure that the business remains fully operational when disruptions occur.  Situations where IT staffing is stretched to capacity, merely increases the likelihood of human error, thereby increasing the odds of data disruptions they work so diligently to avoid.

Another unique factor is the use of a “Core” system that requires IT staff to have extensive knowledge specific to that core. For many institutions, this may mean that there is only one individual on a team who possesses the skills set needed to fully manage and maintain the core. Since a single source of knowledge represents the proverbial weak-link scenario, situations of this nature often complicate IT disruptions.

Research indicates that many assumed cost savings of DIY backup and recovery components tend to evaporate and, when it comes to protecting critical IT data, there is genuine value in considering Disaster Recovery as a Service (DRaaS). For credit unions, the temptation to cut corners on backup and DR tends to be greatest in areas where scrimping on expenditures puts the institution at even greater risk of IT disruptions or outright failures.

The DRaaS market size is estimated to grow from USD 1.68 Billion in 2016 to USD 11.11 Billion by 2021, at an estimated CAGR of 45.9% from 2016 to 2021.

The key forces driving the DRaaS market are its features of faster recovery, cost-effectiveness, enhanced flexibility, and simple testing.  Also, DR services provide automation capabilities that lead to limited utilization of resources and low up-front cost. With the increase in the adoption rate of DR services among Small and Medium Enterprises (SMEs), DRaaS market is expected to gain major traction during the forecast period. 

– Markets and Markets – Disaster Recovery as a Service Markets – 2016

Next time you’re tempted to reach for the Easy Button, remember that disaster recovery is a complex and multi-faceted endeavor that requires constant vigilance and attention to detail.   It also requires an IT staff that is proficient and up-to-date on all vital aspects of the Backup and Recovery domain.

IMS is a unique MSP in the DRaaS arena and is widely recognized for their intimate knowledge of credit union core systems, operations and extensive experience providing managed services in virtualization, networking, security and IaaS.


Why Business Birthdays Matter: IMS Celebrates 19 Years

 

Most of us enjoy birthdays and the celebrations that accompany them.  After all, birthdays acknowledge and revere our grand journey of life and the triumphs, trials and tribulations along the way. This August, Information Management Solutions (IMS) will celebrate 19 years of serving our credit union customers and jubilation is on the menu.

So why are business birthdays important and what’s the importance behind celebrating them?  Being in business for 19 years may see like a rather mundane achievement considering the average life expectancy of the typical American (human) is approximately 79 years. However, according to the Harvard Business Review, “over the last 50 years, the average lifespan of S&P 500 companies has shrunk from around 60 years to closer to 18 years”. Many start-ups never make it to the age of 5.

With all the critical business decisions that must be made to ensure that the business thrives, customers are happy, employees are productive, and financial stewardship is sound, the volume of decisions made within a business are exponential to those made by even the most prodigious of individuals.

When Devon Wilson founded IMS, he had one goal in mind… to provide a level of service that exceeded every expectation and to forge CU customers into wildly enthusiastic fans. By focusing on this goal, it wasn’t long before the phones rang incessantly and Devon quickly began to realize that running a successful business would require more than just an ardent clientele.

Companies are often viewed in two different ways:

  • As a machine for making money
  • As a living being or entity

Either of these perspectives illuminate an entire host of core assumptions about management and the organizational philosophy that propels the business. A Living Company is described as “an organization that is viewed as a ‘Community’ of human beings that is in business – any business – to stay alive.” Arie de Geus, The Living Company.

IMS was founded with the view that a community of people bearing a like-minded vision of natural evolution and sense of identity will grow, adapt, and change through its own capacity for autonomous action and regeneration.

The community dedicated to IMS’s longevity understand that a key tenant of sustaining that life is founded on the simple principle of Trust. The very core of our services provided (data backup, disaster recovery and hosting) require a tremendous amount of trust. After all, what could be more important than the care and keeping of the IT systems that are the central nervous system with which credit unions rely upon to serve their members and stay in business?

In his book “The Trust Edge”, David Hosager defines trust as: The confident belief in someone or something.  It is the confident belief in an entity:

  • To do what is right
  • Deliver what is promised
  • To be the same every time, whatever the circumstances

Moreover, trust carries the implication of being reliable, dependable, and capable.  Building and maintaining trust is a tall order and one that requires constant effort and vigilance. It can take years to build and can be destroyed in a moment, and once lost, it can be virtually impossible to regain.

Trust also plays a vital role in seeing a company as a living body that can evolve as an entity, much like a professional sports team looks at losses and victories as ways to adapt to changing environments and circumstances. The three tenants of trust are essential to a business being able to learn as a unified team. Without trust, the entity lacks the foundational values necessary to learn and grow.

During our 19-year history, IMS has been in a constant state of adjustment.  We continually monitor the internal and external forces that could impact our status as a living entity.  We look for events or circumstances that could jeopardize our stability and we seek opportunities to better serve our customers that may provide greater growth for IMS as an entity.  No one can say for certain what will happen in the future, so we ask, “if something happens, what will we do?”

Striving to look outside ourselves (and typical industry standards) allows us to create scenarios that facilitate our ability to adapt to change. And with practice, to become better prepared for the unexpected. We celebrate our business birthdays, not as an opportunity to review and reminisce about our past, but to reinforce our unity as a living entity and to safeguard the future as a living community.

Our business birthdays remind us that life… even the life of a business, is fragile and past successes do not ensure that the living entity will survive and renew. It is only through optimizing of our resources to create a cohesive identity that continued life is viable.

Cheers to another 19.

 


Credit Unions Step up in Crisis

 

Like so many others, I start my day with a bowl of cereal and the morning news. It seems that most of the stories focus on tragedies throughout our country and throughout the world. I always look forward to the occasional positive story. Stories where people step up and go out of their way to help others.

This morning there was a story about a man that rolled his SUV off of the side of the road and down a slope. The police were able to pull him from the burning vehicle but the slope was too steep to pull him up. Several citizens pulled over, got out of their vehicle and along with the police, formed a human chain in order to save this man. As he was pulled to the top, his vehicle exploded in flames.

Next up, was a story on Hurricane Mathew and the devastating flooding. A lady was interviewed and talked about how everything was going to be alright because they are all pulling together as a community. This got me thinking about many other disasters in the past, including Sandy, Ike, and Katrina.  There were wild fires in California, Tornado’s in Alabama, blizzards in the northeast and just a few months ago, floods in Louisiana.

Having worked in the credit union space for over 25 years, I am consistently impressed how credit unions step up in crisis to help others.

For most of us that are not directly impacted by these regional disasters, it’s hard to imagine the challenges that survivors and communities face. For many, it often takes months and even years to recover. It is estimated that over 146,000 homes were damaged in the Louisiana floods which was characterized as the worst US natural disaster since Hurricane Sandy.

According to an estimate from Goldman Sachs, Hurricane Mathew may have inflicted as much as $10 billion in damage. Current figures would make Mathew the 22nd-worst storm since World War II.

These disasters are affecting communities throughout our country and in many cases, insurance is only covering a portion of the damage which leaves families and businesses to fend for themselves.

During these tragic times, credit unions are usually the first to show their community spirit. Credit unions throughout the country, whose communities have not been affected, frequently organize fundraising efforts and provide donations in order to help out others. Credit unions in the communities affected by these disasters freely donate money, time and labor to help rebuild their communities. Some credit unions will even offer much needed low interest loans for individuals affected by disasters.

Having worked both for and with credit unions, I can’t imagine being associated with a better group of people. As the southeast continues to recover from this most recent event, I have no doubt, credit unions will play a key role in the rebuilding of their communities.

 


Credit Unions Face Increasing Challenges in DIY Disaster Recovery

 

Credit unions face unique challenges with DIY disaster recovery operations, which are becoming increasingly complex due to the proliferation of numerous applications ancillary to the Core.   The growing list of ancillary and interdependent applications, disparate data stores, and third party services, are making the recovery process an increasingly daunting task.

Credit union members are becoming progressively more accustomed to instant data access and limited downtime.  Member expectations put increased pressure on IT departments to reduce downtime to near zero.  IT professionals understand the importance and impacts of downtime, yet a study by Forrester and Disaster Recovery Journal, in 2013 showed that median actual recovery times were 8 hours, up from 3 hours in 2010. 1  This trend can be directly correlated to the growing complexities of the disaster recovery process.

Many credit unions favor in-house DIY disaster recovery operations, but the decision to take on the burden of DIY disaster recovery often comes at a high cost. That burden is amplified by the cost of downtime, as noted in an Aberdeen survey of IT professionals in 2013 that found that the average cost of downtime per hour across companies of all sizes was a staggering $163,674. 2  In addition to the complexity and increasing financial burden, DIY disaster recovery results often fall short of desired outcomes.

One of the reasons DIY disaster recovery results fall short of expectations is the realization that credit union IT staff continually face long lists of projects, which demand a tremendous amount of time and attention.  When faced with continual project management, it’s not uncommon for IT to allow backup and recovery objectives to become a lower priority and the impact on recovery results are bound to be less than ideal.

Virtualization and the recovery of virtual machines has simplified certain aspects of recovery, however credit union staff may be lulled into a false sense of security. Advances in backup technologies and virtualization are also leading many credit unions to invest heavily in secondary sites or colocation.  However, the CAPEX required to implement secondary sites or colocation is not only significant, it requires refresh cycles that ultimately become cost prohibitive over time.

Recent enhancements in backup and recovery technologies appear to make recovery seem easy, when in reality, the opposite is true.  Technology offers great promise and certain aspects of recovery are more dependable than in the past, yet there are still problems:

  • Credit unions place a heavy emphasis on daily operations and the maintenance of vital functions, but most spend limited time testing recovery procedures and reviewing the logistical necessities of a viable recovery system. Daily maintenance of interdependent systems is an entirely different skill set than the expertise and knowledge required to facilitate a functional recovery of those systems. Nor do most take into consideration the expertise required to facilitate recovery of multiple systems with inter-dependencies distributed throughout disparate tiers within the data center.   In other words, emphasis and expenditures focus on daily operations and maintenance, and only a small percentage is dedicated to disaster recovery.  Fully 23% of organizations admit they never test. 3
  • The DIY approach to recovery processes can heighten the risk of improperly vetting the complex inter-dependencies between multiple, if not dozens of applications that run the credit union. Credit unions often operate with lean IT departments and lack the staffing depth necessary to properly manage the complexities of today’s backup and recovery functions.  The development and management of growth initiatives further exacerbate staffing pressures, which result in the neglect of backup and recovery priorities.
  • Typical DIY recovery environments often lack the tools and the mindset necessary to discover shadow or forgotten applications that reside within the IT infrastructure. Even applications and inter-dependencies residing in plain sight may be overlooked in disaster recovery implementation.

Why DIY Recovery Falls Short

  • The majority of business applications are interdependent upon one another. Facilitating full recovery of all business processes from end-to-end requires more than simply storing and accessing applications and data. All components, including proper OS selection, current application selection and most current data set must be incorporated in the recovery environment.  Even achieving recovery of proper applications and data may leave critical components out of the process. Overlooking a seemingly innocuous interdependent component will hinder or prevent end-to-end recovery.
  • Credit unions utilizing colocation or secondary sites face the very real challenge of ensuring that properly qualified IT staff will be able to get to the secondary site in a disaster scenario. Many credit unions run with lean IT departments and it can be impractical or extremely challenging to get properly trained people to the secondary site.  Simply getting staff to the secondary site doesn’t ensure recovery, staff members must have the necessary knowledge and skill sets to guarantee a successful end-to-end recovery.
  • For the most part, IT managers are proficient at managing integration of technology to ensure all systems work together smoothly and at top efficiency. To reliably recover your applications, data and interdependent systems, you must duplicate the precise mix of servers, storage, operating systems, hypervisors, networks and software.  You must also manage any and all changes taking place within that environment and change management must take place constantly. The unavailability of even one component or application can trickle down to impact a wide array of business functions.
  • DIY backup and recovery using colocation or a secondary site typically requires duplication of everything in use at the primary site. All servers and storage must be replicated or mirrored at the secondary location and both sites must have adequate bandwidth and networking infrastructures.  Licensing requirements will be duplicated, as will most security measures.  Staff members will be required to put in extra hours to manage the secondary location or additional staffing will be required.   Costs for the recovery infrastructure can be significant and should be expected to increase every 3 to 4 years in conjunction with hardware and software refresh cycles.
  • Misalignment of Recovery Point Objectives (RPO’s) and Recovery Time Objectives (RTO’s) with Recovery Methods (RM’s) is a common and costly occurrence. When it comes to RPO’s and RTO’s, the concept of “one size fits all” is a dangerous miscalculation.  Credit unions practicing this mindset will quickly discover that the incident that caused an initial system failure isn’t the only disaster they will encounter. Misalignment or miscalculations of RPO’s and RTO’s with an associated RM will quickly short-circuit or prevent end-to-end recovery capabilities.

Viable Alternatives to DIY Disaster Recovery

  • Credit unions are founded on the principles of a cooperative and shared business model. Backup and Recovery as a Service (BRaaS) is the model for cooperative sharing of technical knowledge, IT infrastructure, hardware, software, and resources. Managed Service Providers (MSP’s) provide the infrastructure, experience and knowledge for BRaaS at a fraction of the cost of DIY backup and recovery models.   BRaaS is also recognized as a readily sustainable business model.
  • A primary benefit of BRaaS is the deployment of highly qualified and efficient resources in a shared environment. Distribution of shared resources eliminates the significant outlay of CAPEX, while amortizing OPEX in a simple pay-as-you-grow model.  BRaaS leverages cutting edge infrastructure along with extensive knowledge and experience, to deliver true business resilience.
  • Core system vendors often focus recovery efforts on the Core system specifically and may be inclined to neglect critical inter-dependencies necessary to provide end-to-end recovery and resiliency. However, a select number of MSP’s are proficient at recovering the core and all ancillary servers with related dependencies. Third party vendor connections to ATM’s, mobile banking, internet banking and the FED are typically provided as well. Qualified MSP’s provide true end-to-end business recovery while eliminating the expense and frustrations of DIY endeavors.

What to look for in a BRaaS solution and vendor.

Ask if the service provider:

  • Performs Backup and Recovery of the Core and all Ancillary Servers
  • Provides Connectivity to Third Party Vendors, such as: ATM, Mobile and Internet Banking, FED, etc.
  • Performs Infrastructure and Network Discovery Analysis
  • Manages Backup and Recovery Resources and Procedures
  • Monitors Alerts for Potential Complications or Issues
  • Suggests Technical Improvements and Implementation Processes
  • Assists With Systems Maintenance to Resolve Backup and Recovery Challenges
  • Identifies and Tracks Inter-dependencies and Application Road-maps
  • Matches RTO’s and RPO’s With Appropriate Recovery Methods
  • Performs, Manages and Maintains Restore/Recovery Procedures
  • Initiates Pre-test Discovery, Meetings and Planning
  • Tracks, Monitors and Documents DR Test and Results
  • Conducts Post-Test Reviews, Remediation and Gap Analysis
  • Initiates Change Discovery and Management
  • Maintains and Stores Backup and Recovery Procedural Docs and Configurations
  • Provides Timely Reporting of all Vital Backup & Recovery Elements
  • Provides Relevant Case Studies and Use Examples
  • Provides Credible References and Testimonials.

 

1  DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014

2  Aberdeen Group, Downtime and Data Loss: How Much Can You Afford? August 2013

3  DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014


Windows Active Directory in the Age of Virtualization

 

As the IT department for many small businesses over the last 25 years, I’ve had to configure and manage a variety of user and security management solutions. With Windows NT server, Microsoft introduced Active Directory. At the time, server hardware was barely better than a desktop. IT departments spent more money per system for larger disks, bigger CPU and maybe additional network cards, but the fundamental components and their function were that of a desktop. Because of this unreliability, Microsoft and IT technicians recommended having a Primary Domain Controller (PDC) and a Backup Domain Controller (BDC) along with Read Only Domain Controllers (RODC). This insured that in the case of hardware failure by the PDC, you still had DC’s to authenticate to, rebuild your Active Directory hive, etc.

Even as hardware, and Windows Server improved, this practice of dividing up the load and replicating to multiple servers stuck. Let’s face it, it was still a good idea even with hard drives in a raid 5 array and frequent backups. You could still have hardware or software failures from a variety of possible causes. Therefore, the model remained the best way to protect your AD.

However, for some reason many stuck to this when virtualization came along. As a caveat to what I’m about to say, know that I’m referring to a virtualization environment where there are three or more hosts, clustered, with fail-over. Virtualization removes the need to spread the AD roles around, and there are five of them now: PDC (a legacy from NT), Naming Master, Infrastructure Master, Schema Master and RID Master. One would think after all this time these roles could be clustered, meaning you could assign the same role to multiple DC’s and they would stay mirrored, but the sad truth is that each role can only be on one DC. The good news is that at this point, from a hardware failure perspective, an IT person can be just as secure having all the roles on one DC as they can dividing them up. If a host has a hardware failure, the server restarts, as it was, on another host. This is true in a cluster of any major virtualization technology.

At this point you might be wondering why you should care. Virtualization lets you create multiple servers quickly and easily, so why not keep the roles divided. There is one huge area where this comes into consideration, and that’s backup. Even if you backup every DC that has at least one of the five roles, restoring multiple servers takes more time. If the environment for restoration is during a disaster, hardware, IP address and other changes can make even a “whole” Active Directory environment take a while to come up while the hive reconfigures itself. If you don’t have access to a Domain Admin login that also is in the Schema Admin group (which Domain/Enterprise Admins are NOT in by default) and you didn’t back up your Schema Master, you will probably need Microsoft’s help to have any chance of getting your AD to work in the new environment.

All this being said, here is my recommendation in a virtual environment for your domain controllers. Build one server with one virtual CPU and 4 GB of RAM and put all the primary DC roles on it. Install NOTHING unrelated to Active Directory on the DC. Don’t let WSUS or file services or any other functions put it at risk of software corruption. Build a second server in the environment and make it a DC, global catalog only. Make sure you have administrator login(s) in the Schema Admins group in the case you need to seize all the roles on the second DC. As a global catalog server it has all the information it needs, but the roles do not come up without seizing in the case of software failure on the primary DC. Then make sure that both servers do all the things related to AD that do get mirrored, like DNS or group policy management for instance, but, again, stick to only Active Directory related activities on the servers.

Then if you have remote sites, with physical or virtual servers, make them global catalog domain controllers only. A GC DC provides everything users need to login, authenticate, etc. You can even run DNS and DHCP on these servers and, unlike the AD roles, they will stay replicated and mirrored. As long as you have Schema Admin users, you can even seize the roles on these servers in a pinch. I have witnessed that without making sure all your roles are backed up, and that you have access to Schema Admin, an AD environment can be completely unable to come up. My recommendations are by no means mandatory, but for backing up and restoring your Active Directory environment, they will both simplify your life and cause you the least headache in the case of recovery, especially during a disaster.


What Every Credit Union IT Manager Needs to Know About Backup

 

With technology rapidly changing, it is difficult for credit union IT managers to keep up. The entire landscape has changed over the past 10 years with virtualization becoming so widely adopted. Additionally, with the rapid growth of data and the demand for up-time due to internet and mobile banking, backup windows are shrinking.

Many credit unions are still backing up to tape. However, tape is unreliable, inefficient and there are serious concerns as it relates to compliance. If you are one of many credit unions still using tape or just struggling with the aforementioned, this article might help you better navigate the complexities of backup technology.

In the world of tape backups, you copy all files and databases to tape. To backup more efficiently, you might perform an incremental backup. An incremental compares files from the prior backup and only copies the ones that have changed. If there is a database on the server and even a single record is written, an incremental will need to backup the entire database. If a single word is added to an existing Word document, an incremental will need to backup the entire file. Should you need to recover, you will need to restore the original full backup and the subsequent incremental backups.

Deduplication

In order to understand deduplication, you need to forget everything you know about tape. Enterprise data is highly redundant with identical files or data stored within and across systems. Traditional backup methods magnify this by storing all of this redundant data over and over again. Deduplication is the process of analyzing files and databases at the block level and only storing the unique blocks of data eliminating redundancy. Sounds easy, right? Well, not so fast. First, you need to know, not all deduplication is the same. There are two types of deduplication, inline and post-process. Inline deduplication identifies duplicate blocks as they are written to disk. Post-process deduplication deduplicates data after it has been written to disk.

Inline deduplication is considered more efficient in terms of overall storage requirements because non-unique or duplicate blocks are eliminated before they’re written to disk. Because duplicate blocks are eliminated, you don’t need to allocate enough storage to write the entire data set for later deduplication. However, inline deduplication requires more processing power because it happens “on the fly”; this can potentially affect storage performance, which is a very important consideration when implementing deduplication on primary storage. On the other hand, post-process deduplication doesn’t have an immediate impact on storage performance because deduplication can be scheduled to take place after the data is written. However, unlike inline deduplication, post-process deduplication requires the allocation of sufficient data storage to hold an entire data set before it’s reduced via deduplication.

In order to remain competitive, many tape based backup software providers have stepped into the deduplication arena. Most write to disk just as they do to tape and then run post-process deduplication to minimize the disk footprint.

A primary concern with both inline and post-process deduplication is they require streaming the data across the LAN or WAN to disk (target) which consumes a considerable amount of bandwidth. As deduplication has evolved, rather than only target based deduplication, a few vendors are now offering source based deduplication. This is the process of deduplicating at the client (source server) and then streaming only the unique blocks of data to the target (backup server). Taking it a step further, once the data hits the target, it can perform global deduplication (inline deduplication) where it compares the blocks with the blocks that have already been written to disk on the target, then only write the unique blocks of data. Rather than performing inline deduplication on 100% of the data, it may only need to compare 1% or less, eliminating the concern for processing power. As you can imagine, streaming and writing only the unique blocks of data significantly reduces the required daily network bandwidth and storage.

Virtual Environments

VMware changes your server and application IT environment. Server utilization has commonly run as low as 5 percent to 20 percent. Because virtualization can make a single physical server act like multiple logical servers, it can improve server utilization by combining numerous computing resources on a single server. VMware allows users to run 10 or more virtual machines on a single server, increasing server utilization to 70 percent or more.

Virtual server backups can be accomplished using a traditional approach with conventional backup software. The backup software is simply installed and configured on each virtual machine, and backups will run normally to any conventional backup target, including tape drives, virtual tape libraries, or disk storage. However, applying traditional backup tactics to virtual server backups does have drawbacks. The most significant challenge is resource contention. Backups demand significant processing power, and the added resources needed to execute a backup may compromise the performance of that virtual machine and all virtual machines running on the system—constraining the VMware host server’s CPU, memory, disk, and network components—and often making it impossible to back up within available windows.

Backup processes have evolved to deliver greater efficiencies in your highly consolidated environment. How is it this possible with larger workloads and shared resources?

The key to making VMware infrastructure backup as efficient as possible is source-based global deduplication.

Backing up at the source can quickly and efficiently protect virtual machines by sending only the changed segments of data on a daily basis, providing up to 500 times daily reduction in network resource consumption compared to traditional full backups. Source based deduplication also reduces the traditional backup load—from up to 200 percent weekly to as little as 2 percent weekly—dramatically reducing backup times.

Some of the more sophisticated backup solutions can back up at the guest level—an individual virtual machine—or at a VMware Consolidated Backup server. In addition, disk based deduplication software negates the need for transporting tapes to offsite repositories for disaster-recovery or compliance purposes by providing remote backup immediately offsite via the cloud.

Second, source based deduplication is the optimal granularity to find changes anywhere within a virtual machine disk format (VMDK), and this is where target based deduplication alone fails to deliver.

Recovery

As I have stated in prior blogs, backups are the means to recovery. Once data has been deduplicated, in order to perform a recovery, it has to go through what is called a re-hydration process. This is a process of putting all of the pieces back together again and as you can imagine, some software performs this process much more efficiently than others.

Some target based solutions will store multiple revisions before deduplicating so that, in the event of a recovery, it does not have to rehydrate since the re-hydration process can take so long. If you are considering backing up to the cloud, you have to remember that once your initial backup has been seeded (fully written to disk) and your daily backups are running reasonably fast, should you have to recover, you have to rehydrate the entire backup and pull it across the internet. This can add hours or even days to your recovery depending on various factors; re-hydration time, bandwidth etc. If it is your core system, waiting several hours or even days to recover is not an option.

For this very reason, many vendors are now offering a hybrid approach. A hybrid approach requires placing a backup appliance local (at the credit union) to allow for much faster recovery. Additionally, the backup appliance will replicate off-site to the cloud provider.

Backing up to The Cloud

Credit Unions have been slower than most to adopt The Cloud. No surprise since Credit Unions by nature are very conservative. However, we have passed the tipping point and more and more Credit Unions are moving services that direction and backups are no exception. When selecting a backup provider, it is important to understand how the majority of cloud providers price their service. Since deduplication creates a much smaller footprint, pricing is typically based on the amount of data stored in the cloud. The issue with this is nobody truly knows exactly what that number will represent until you have backed up all of your data. This is where it gets complex.

There are two types of data, structured and unstructured. Unstructured data is typical file system files, Word and Excel documents etc. Structured data is primarily databases; Exchange, domain controller, SQL, Oracle etc. On Average, roughly 70% of data at most businesses is unstructured. Unstructured data will deduplicate much more efficiently than structured. In order to estimate your deduplication footprint, it requires the service provider gathering the details on your data to calculate the percentage of structured and unstructured data.

Additionally, retention is a key factor since once the seed is calculated, you have to factor in the average daily change rate and multiply it times your defined retention policies. You also need to factor in average annual data growth. As you can see, this becomes highly complex. If not accurately calculated, you can sign on expecting to pay one amount and end up paying another. Additionally, some software deduplicates much more efficiently than others. Although one vendor may have a lower price per GB or TB than another, they may end up storing two to three times more data, essentially costing you more. It is very important to demo the software before making a long-term commitment and ideally, choosing a vendor that understands credit unions.

Disaster Recovery

One common challenge credit unions encounter after selecting a cloud backup provider is how to transport their data to their disaster recovery service provider in a timely manner. Also, will the DR provider know what to do with it once it arrives?

More and more disaster recovery service providers are offering backup solutions. It just makes sense to have your data stored at the site where the recovery will be performed, avoiding a logistics nightmare. Not to mention, the last thing you want is to have them fumbling around trying to figure out how to use someone else’s software. They need to be experts on the tools they will be using to perform the recovery. The key is to ensure they are capable of meeting all of your recovery needs, they are security conscious, and they perform a regular SSAE examination.

As you can see, technology is rapidly changing and backup software is evolving to keep up with the pace. If you are still using tape, struggling with up-time or just unsatisfied overall with your current backup, I hope this article helps guide you in the right direction.

 

 

 

 

 


SEVEN Components of a Successful Disaster: Component Seven

Component Seven: Test, and Test Often

Once you’ve implemented your disaster recovery solution you are on the right track towards positioning your credit union for success in the event of a disaster. However, IT is not static, it is ever changing. Therefore, it is important you implement the processes to adapt as circumstances change. Whether it’s an upgrade to existing servers, additional servers, changes to third-party vendors, or adding or removing branches, you need to have procedures in place that streamline the DR process.

On many occasions, during tests and even in actual disasters, we’ve encountered customers who have made changes and neglected to notify us. This oversight can be the difference in experiencing a successful disaster, and as I have already mentioned, you may only get one shot. By putting the proper procedures in place, you can avoid this by ensuring when you upgrade or add equipment and services, disaster recovery is always included on your check-off list.

It has become so commonplace that we implemented a process ourselves to send out a notification to our customers throughout the year, reminding them to review their DR services to ensure there have not been any changes that would affect their recovery. Additionally, we do a pre-DR test meeting to discuss and review both expectations for the upcoming test and their existing services to ensure nothing has been overlooked. Nobody wants to wing-it during a disaster. Preparation is key.

Now that you have clearly defined RTO’s and RPO’s you have a quantifiable way of measuring success during a test. Whether you are going with an in-house solution or outsourcing, you must ensure you keep track of your recovery times. This isn’t to say that by not meeting your RTO’s your test has failed. It simply allows you to determine if expectations are set too high and need adjusted, or if a different recovery method is required to meet the RTO.

As you proceed through a test, it’s critical you thoroughly document each individual process being recovered. Having detailed documentation allows for a post-DR test review to evaluate and determine if the recovery procedures were effective or if adjustments are needed.

Since a disaster is unpredictable, it’s important that you vary your disaster test scenarios. Many times, credit unions have a standard script they run through when performing their test. To be most effective, and increase your chances of success, you must enter each test simulating scenarios that are most likely to occur. For example, if you are located in the south east, you may want to simulate the effects of a hurricane or tornado. If you are in the Midwest, maybe a tornado or flood. And if in the west, maybe an earthquake or flood. However, don’t overlook a Murphy moment, a random scenario such as the failure of a single server or an irate employee wreaking havoc in the computer room.

Also, in large regional disasters, you may not have access to your entire IT staff. Even during hurricane Katrina many police and firemen did not show up to work because they were taking care of their family. This is a likely scenario, so you need to be prepared to function with minimal staff. If outsourcing, selecting a vendor that understands your business is a very real consideration and if you choose to go in-house, having adequate staff, cross-trained and working outside of the region will be required.

As you look at the various types of disasters it is clear to see there is a significant difference between a disaster where the data center is completely out of commission versus a single server recovery where routing of inter-dependent servers and third-party communications is needed.

With so many natural disasters making the news, disaster recovery has moved to the forefront for many auditors. However, this really isn’t as much about passing an audit as it is keeping your credit union in business and ensuring your members are able to access the funds they entrusted your credit union to protect. Without thorough testing on a re-occurring basis, you are playing a guessing game on whether or not you will be able to successfully recover from a bad situation.

So, there you have it! Disaster recovery is very complex. However, with the proper components in place, should the unforeseen occur, your credit union will be well prepared to experience a successful disaster.