SOFTWARE DEFINED BLOG

The High Availability feature for HCX Network Extension appliances was introduced in the version 4.3. This was a big deal, because if someone needed HA for their appliances that extend L2 networks (as this is a basic requirement for the resiliency) to the cloud and wanted to use VMware’s stack in particular, they had to deploy a pair of NSX-T standalone Edges on-prem leveraging NSX-T on the cloud side and setup a L2VPN.

VMware documentation mentions the following important prerequisites to consider before deploying HCX NE HA:

• Network Extension HA requires the HCX Enterprise license.
• Network Extension High Availability protects against one Network Extension appliance failure in a HA group.
• Network Extension HA operates without preemption, with no automatic failback of an appliance pair to the Active role.
• Network Extension HA Standby appliances are assigned IP addresses from the Network Profile IP pool.
• The Network Extension appliances selected for HA activation must have no networks extended over them.

Also an interesting thing about NE in HA mode is the upgrade process:

In-Service upgrade is not available for Network Extension High Availability (HA) groups. HA groups use the failover process to complete the upgrade. In this case, the Standby pair is upgraded first. After the Standby upgrade finishes, a switchover occurs and the Standby pair takes on the Active role. At that point, the previously Active pair is upgraded and takes on the Standby role.

Let’s take a look at this new feature. In the HCX UI 4.3+, in the Interconnect -> Service Mesh -> View Appliances view, there is a new option called ACTIVATE HIGH AVAILABILITY.

First, you need to have a pair of deployed NE appliances, the option won’t work when there is no eligible partner for HA.

Also, system checks, if there are extended networks on an appliance that you select for HA.

It can be a challenge to enable HA in an environment, where networks are already extended. In most cases this would require a downtime because we would have to unextend existing networks on NE appliances for the time of the HA configuration.

The system also checks if there are eligible NE appliances to activate HA feature and we get a button that is a shortcut to edit a Service Mesh and add more appliances. It can also be a challenge, if we don’t have a sufficient number of free IPs in our Network Profile’s IP pool. For 2 additional NE appliances on-prem and at the cloud side, we need one management IP and one uplink IP for each of them.

Once appliances are deployed, you only need to press a button Activate HA. Everything is configured for us, like in vSphere HA. New HA appliances create a HA group with a specific uuid.

When HA is enabled, we can monitor its health in the HA Management tab. In the example below, we have 2 pairs: us-east-NE-I2 (on-prem) with us-east-NE-R2 (the cloud side) and us-east-NE-I3 (on-prem) with us-east-NE-R3 (the cloud side). Right after creation, the first pair is ACTIVE and the second is STANDBY.

Also in the Appliances view we can quickly check with NE appliance is ACTIVE and which one is STANDBY.

NE appliances with HA enabled can coexist with other NE appliances. This means we can still use single NE appliances for less critical workloads and HA groups only for selected networks.

During a HA group creation, there is a VM/Host Rule created in a vCenter to make sure NE appliances in the same HA group won’t run on a same host.

After HA group is deployed, we can simulate a failure of one NE.

In this example I am using two sites, Sindey represents on-prem and Ashburn is my cloud side.

The network test 192.268.99.1/24 is extended on this new HA Group from on-prem (NE-I2 and NEI-3) to cloud (NE-R2 and NE-R3).

There is a vm test_aga_vm_2 (192.168.99.107) on-prem connected to the test subnet and on a cloud side, there is a vm test_aga_vm (192.168.99.103) on the extended network L2E_test. As VMs are at the opposite sides, we can have a ping running between them to test the connectivity during the NE failover.

If you look closely at NE appliances, there is nothing specific to HA there, no specific HA subnet, only management, uplink and extended networks.

On the on-prem side, there is a test network attached to NE-I2, same for NE-I3.

On the cloud side, there is a L2_test network attached to NE-R2, same for NE-R3.

Now, to force a failover, I am shutting down an appliance on the on-prem: NE-I2.

What happens next is there is a failover for the first pair of appliances: NE-I3 and NE-R3 become active. This is always a pair of appliances that changes the state, so in case NE-I2 is down, NE-R2 also fails over to NE-R3.

There is a small (as expected) packet loss observed and also it takes some time in the HA view in HCX UI to show the updated status of the appliances. The communication path between NE on-prem and NE on GCVE was restored and pings successful but the UI still showed a DEGREADED state for HA appliances for some time.

I lost 5 pings only during the failover, it was pretty quick.

After few refreshes the UI shows a new state: HEALTHY with NE-I2 and NE-R2 being in the STANDBY and NE-I3 and NE-R3 in the ACTIVE state.

There is also an interesting option in the UI to view HA activity timeline. We can quickly check the HA history and states of the appliances.

If we don’t want to power off the appliances, we can use “manual failover” option to test HA.

This one seems to be more graceful as I lost only 1 ping during this test.

I bet many administrators have been surprised by this new ESXi appearing on a hosts list in vCenter UI after a HCX service mesh had been created for the first time. And for every HCX service mesh we create, there is one ESXi “host” being deployed.

Of course it is not a “real” host, it looks a little bit like its a nested one and its name seems to always be its HCX Interconnect (IX) appliance’s management IP. I think of it as a IX appliance’s alter ego 😉

It does have its own “VMware Mobility Agent Basic” license for 2 CPUs.

But it not a typical nested ESXi, it’s a dedicated VMware Mobility Platform.

It has its own VMFS local datastore called ma-ds with a total “capacity” of 500 TB and 1 TB of RAM 😉 Fortunately, nothing is really taken from our physical resources.

In my case HCX service mesh Interconnect Appliance IX has 172.16.4.2 as management address, hence the host was named “172.16.4.2”.

IX appliance is a VM with many network interfaces, among the most important ones are HCX Management Interface, HCX vMotion Interface and HCX Uplink Interface. Uplink Interface is in fact the only one that is used to communicate with a target side, Management and vMotion Interfaces are used locally. The CIDR for those interfaces (and many other settings) are set during Network Profiles creation in HCX UI.

So what does this new dummy Host do?

Its job looks like a proxy for vMotion tasks between two paired HCX sides and allows for long distance cross vCenter vMotion. It is configured when the following service option is enabled: vMotion Migration service.

“The vMotion Migration service provides zero-downtime, bi-directional Virtual Machine mobility. The service is deployed as an embedded function on the HCX-WAN-IX virtual appliance.“

For HCX vMotion, we don’t need direct connectivity between source and target vCenters and their vMotion networks. A source IX appliance task is to trick a source ESXi to believe the destination ESXi for the vMotion task is local. Source ESXi “thinks” the target ESXi for the vMotion task is its local Mobility Agent host. A target IX appliance task is to trick a target ESXi to believe the vMotion task is local. Target ESXi “thinks” the source ESXi for the vMotion task is its local Mobility Agent host.

What is really going on is transparent for the both source and target side. IX appliance on the source side acting as a receiving end for the vMotion task transfers the VM data via HCX Uplink Interface (IPsec tunnel) to the target IX that acts as the initiator of the vMotion task at the destination side.

This explains why there is a requirement for IX appliance to be able to communicate with ESXi over their vMotion networks.