The best disaster recovery solution may be one that helps avoid disasters altogether. As software-defined data center solutions come into the mainstream of IT, many organizations are building out data centers that deliver redundancy, high availability and real-time failover in the event of an outage.

This was the type of solution Health New England was striving for in 2017 when it began exploring what its next-generation, VMware-based data center infrastructure might look like. It wasn’t looking to eliminate disaster recovery, just to reimagine it. The IT team, which was already running a VMware environment on Hewlett Packard Enterprise blade servers at its Springfield, Mass., data center, was focused on storage resiliency and high availability as it planned for the future.

“We had traditional Fibre Channel storage to the VMware cluster, and while it was highly available in itself, it was bound to the single site,” says Brian Gay, LAN administrator for HNE. “We did a little 3PAR storage replication to a DR site for some virtual machines, but it wasn’t an active-active solution.” In an active-active architecture, an organization supports at least two data centers that can service applications at any time. Each data center operates as an active application site.

After sketching out several solutions for the new data center infrastructure, HNE’s small IT team made the case for an active-active architecture with VMware Cloud Foundation and vSAN Stretched Cluster with NSX. Together, along with the SDDC Manager software lifecycle management tool, these solutions built the foundation of a software-defined data center that was easier to maintain, monitor and manage. NSX, VMware’s network virtualization platform, was an important part of the project, as it enabled HNE to retain the IP addresses of several legacy systems. This allowed the team to seamlessly migrate these systems to the new environment without having to reconfigure applications or create other IT disruptions.

“We could have kept putting money into business as usual and gained a couple benefits, or we could take our budget and focus on this one project,” says HNE Lead Cloud Services Engineer Dave Lewis. “You can build high availability into applications like SQL and others, or you can build it into everything by stretching the cluster between data centers.”

The Health New England team (standing, from left, Mike Chabot, Dave Lewis and, seated, Brian Gay) implemented stretched clusters with help from CDW.

Health New England, a subsidiary of Baystate Health, is an HMO serving roughly 200,000 members in and around western Massachusetts. It operates several lines of business, including commercial health insurance and Medicare Advantage. The company has a significant remote workforce of about 400 associates, and up to 85 percent work remotely at any one time.

“Once, when we were half the current staff count, it was different. Everybody had a seat here,” Gay says. “You could work from home if you wanted to, but now there are actually more folks on the employee roster than there are seats in the primary location.”

HNE has traditionally supported its remote workers using a VMware virtual desktop infrastructure. The new active-active, stretched cluster architecture, which went live in July 2019, will eventually enable a highly available VDI, scheduled to roll out next year.

Before HNE implemented Cloud Foundation and the vSAN Stretched Cluster, the company’s sole data center was in a high-rise building that houses its Springfield headquarters. As the company continued to grow and introduce new business lines, traditional DR proved to have some limitations.

“We’ve been here long enough that we've seen a lot of things happen in this high-rise,” says Lewis. “We’ve had generators that have kicked on and then kicked right off, and we've gone black. We’ve had water issues. We had a tornado blow by here. There are so many scenarios where we could become dark.”

The active-active data center infrastructure was designed to overcome such challenges. In addition to the primary data center in Springfield, HNE built a second data center at a nearby Baystate Health facility and connected the two via 40-gigabit-per-second dark fiber.

HNE implemented its software-defined data center using VMware Cloud Foundation 3.8.1. Working directly with VMware, HPE and CDW, the company was among the first to build an active-active vSAN stretched cluster.

Stretched clusters basically build on a prior vSAN capability called fault domains, which could distribute virtual machines across hosts in multiple racks. “VMware’s vSAN Version 6.6 offers similar failover capabilities, but among multiple data centers, delivering the fault tolerance we needed to confidently run active-active across two data centers,” says Network Administrator Mike Chabot. “It also meets our needs for encryption, a regulatory requirement under HIPAA.”

The project’s leaders took their time to make sure they got the deployment right. 

“We were patient,” Lewis says. “We’d heard about Cloud Foundation and waited to be able to get the stretched cluster capability with it. We were part of VMware’s design partner program and worked tightly with them to understand where the solution was going and what we would need as far as networking to build it out.”

Dave Lewis, Lead Cloud Services Engineer, Health New England

George Kotsiovos, a CDW executive account manager, connected HNE with engineers from VMware, HPE and Cisco, who helped the IT teams plan and implement the project. “George was instrumental in the success of the whole project,” says Jack Costa, HNE’s network manager. “He orchestrated the resources we needed for planning, pricing and delivery.”

Carlo Bonura, CDW solution architect team lead, and other CDW engineers provided necessary support to the project, bringing valuable resources to the effort. “The stretched cluster, especially with Cloud Foundation, is a newer concept,” he says. “Customers are looking at stretched clusters more now that infrastructure can support it. Infrastructure is the biggest challenge when it comes to any of these technologies. As infrastructures mature, organizations have the ability to do more.”

At HNE, that meant upgrades to its data center architecture. Gay says the HNE team worked closely with CDW, VMware and Cisco to understand what storage and networking options were coming out that would best support its high-availability solution. “They introduced us to technology partners out in the field and within CDW’s organization that can help us with scoping,” Gay says. 

For example, HNE knew it wanted to use HPE servers and was able to acquire the latest generation of HPE ProLiant DL380 Gen10 servers as soon as they came out. In addition, because of its relationship with CDW and HPE, the HNE team got word that alternative storage technology could be a better option and ultimately adopted high-speed, solid-state drives that support Non-Volatile Memory Express for its stretched vSAN.

“It wasn’t just Cloud Foundation we were building here,” Lewis says. “If we were going to build high availability between two data centers, we had to upgrade our existing, single-core switch. The networking was complicated, and we worked closely with CDW to build two Cisco cores at one site and two cores at the new site.”

HNE enjoys the resiliency and high availability it sought by implementing VMware Cloud Foundation and vSAN stretched clusters across its two data centers. “Basically, we get out of that one-basket-of-eggs scenario, so that if this site does go down, with the vSAN Stretched Cluster, VMs will fail over to the other site and come back up,” says Gay.

And the further maturation of HNE’s software-defined data center stands to yield fresh benefits. Moving to VMware Cloud Foundation enabled the company to simplify management of its VMware environment. “We’re running vSphere, vSAN and NSX all in a bundle. We’ve really got a single pane of glass for SDDC management,” Gay says.

CDW’s Bonura says HNE’s move from a traditional three-tier environment with conventional backup to the current Cloud Foundation with vSAN stretched clusters benefits IT and the company’s users. “Cloud Foundation is the basis of data center modernization,” he says. “It’s a process to simplify your environment, and it gives you the ability to be more proactive than reactive. It allows you to be more granular when rolling out infrastructure and provide users a cloudlike offering.

“It doesn’t eliminate backup and recovery, but the stretched cluster enables that active-active data center, giving HNE a more resilient environment.”