Guidelines on Deploying NSX ALB as a Load Balancer for vSphere with Tanzu

Photo by Ray Harrington on Unsplash

Introduction

Using NSX ALB as a platform load balancer is becoming a ubiquitous choice as this does not require some heavy lifting with a fully functional NSX deployment. Many customers want to leverage the Kubernetes dial tone of vSphere with Tanzu on a standard VLAN-backed VDS networking. Tanzu for Kubernetes Operations provides NSX ALB as the L4 load balancer for the underlying platform. This article will discuss some common architectural patterns of deploying vSphere with Tanzu, with NSX ALB as the platform load balancer.

This article will discuss the solution based on vSPhere with Tanzu 8.0u1 and NSX ALB version 22.1.3.

Concepts

Before discussing some architectural patterns, let us introduce some concepts and terminologies.

In the simplest form, the NSX ALB controller(s) is a set of VMs running on the vSphere platform. AKO is a controller that runs within the Supervisor Cluster and interacts with the NSX ALB controllers. Based on the platform's requirements and as requested by AKO, the NSX ALBs deploy service engines that host the load balancer services.

A typical vCenter environment could have multiple Supervisor Clusters. As a refresher, Supervisor clusters are traditional vSphere Cluster with a few configuration changes and can now be accessed using the standard Kubernetes APIs.

• Cluster ID. Within a vSphere environment, each Cluster has a unique ID. It consists of two parts, a unique value for the Cluster and a UUID of the vCenter. The simplest way to grab the Cluster ID (relevant to this article) is to select the Cluster within the vCenter's Main Menu -> Invenrory -> Host and Cluster view. The red highlighted value in the figure below is the relevant Cluster ID. In this example, domain-c67 is the unique ID of the Cluster in question, and 2332c5ab-29ec-4e32–9545-bd2c6c480a69 is the UUID derived from the vCenter. Hence the cluster-ID is domain-c67:2332c5ab-29ec-4e32–9545-bd2c6c480a69. There are ways to extract this information programmatically.

• Markers. Within an NSX ALB environment, markers are identical to tags that can be assigned to various objects. While these markers are not exposed to the UI, they can be easily accessed (created and modified) using the NSX ALB CLI.
• Cloud. One of the core concepts of NSX ALB is the Cloud. More details on the concept of clouds can be found here. An NSX ALB deployment has one Default Cloud. A cloud of type VMware vCenter when configured, is associated with a unique vCenter and a Datacenter within that vCenter. All resources accessible to the vCenter/Datacenter combination, such as networks, vrfs, and IPAMs, are accessible to such a cloud.
• Service engine group. Each Cloud has a default service engine group called Default-Group. Additional service engine groups can be created per Cloud. More details on service engine groups and their settings can be found here.

Scenarios

The table below lists the possible configurations that could be leveraged while configuring the NSX ALB controller and the Supervisor clusters. Depending on the number of unique vCenter and Datacenters combinations where the Supervisor may be enabled, and the number of clusters within a Datacenter where the Supervisor may be enabled, the NSX ALB configurations and the corresponding ako configurations would need to be modified. VMware fully supports some scenarios, while one of the configurations is currently unsupported and may require a workaround.

Please scroll right on the table to see additional details.

While Default Cloud, Default-Group for Service Engines Groups, and single VIP network are documented in the official documentation, we will discuss the other scenarios next in the article, mainly,

• Leveraging Custom Groups for Service Engine
• Leveraging unique VIP Networks per Supervisor
• Leveraging Custom Cloud(s) and using them with the Supervisor(s) — currently unsupported.

Custom Service Engine Groups

Before enabling the Superivsor on a given vSphere Cluster, create a Custom Service Engine Group within the Cloud of choice (see table above). Since it is recommended that the service engines (data plane for the load balancer) reside close to the infrastructure it is servicing, the placement and configuration of the service engines can be tweaked to meet this and other recommendations. While creating the custom group, do make sure to select Cluster (and optionally hosts and storage) based on the infrastructure requirements.

In the example above, the Scope section allows you to define the placement of the service engines. Selecting the correct Cluster (and host and data stores) is the key to correctly placing the service engines.

Next, the appropriate Cluster ID marker (see above) must be added to this newly created object. This can be quickly done by the NSX ALB CLI (see above).

[admin:192–168–100–58]: > configure serviceenginegroup Test-Group
[admin:192–168–100–58]: serviceenginegroup> markers
New object being created
[admin:192–168–100–58]: serviceenginegroup:markers> key clustername values domain-c67:2332c5ab-29ec-4e32–9545-bd2c6c480a69
[admin:192–168–100–58]: serviceenginegroup:markers> save
[admin:192–168–100–58]: serviceenginegroup> save

The above example adds and saves a tag clustername with a value of domain-c67:2332c5ab-29ec-4e32–9545-bd2c6c480a69 to the object serviceenginergroup called Test-Group .

Once completed, enable the Supervisor on the Cluster (within the vCenter interface).

Unique VIP Networks per Supervisor

There may be a requirement that each Supervisor within a Datacenter has a unique VIP network. This is common where a Datacenter may have clusters within multiple DMZ and Internal network zones. Security may mandate ingress traffic to each Supervisor and its namespaces via its dedicated VIP network.

Before enabling the Superivsor on a given vSphere Cluster, within the appropriate Cloud, configure the required networks with their Subnet prefixes and IP address ranges (see picture below for reference)

Log in to the NSX ALB CLI and create the markers to associate the Network to the relevant Supervisor cluster. This is very similar to the steps performed above, and an example is provided below —

[admin:192-168-100-58]: > configure network Workload1-VDS-PG
[admin:192-168-100-58]: network> markers
New object being created
[admin:192-168-100-58]: network:markers> key clustername values domain-c67:2332c5ab-29ec-4e32-9545-bd2c6c480a69
[admin:192-168-100-58]: network:markers> save
[admin:192-168-100-58]: network> save

In the example, the Workload1-VDS-PG Network has been tagged (using a marker clustername) to be used by the Cluster with Cluster ID (see above) domain-c67:2332c5ab-29ec-4e32–9545-bd2c6c480a69 , when the Supervisor will be enabled on it.

Note: Remember to add this Network to the IPAM Profile of the relevant Cloud the NSX ALB uses. You may also need to adjust the VRF Context to adjust the routing.

Once completed, enable the Supervisor on the Cluster (within the vCenter interface).

Custom Cloud

Note: This is currently not an officially supported configuration.

As the table above highlights, this scenario may arise with the same NSX ALB controller servicing multiple vCenters and/or multiple Datacenters. Supervisors will be enabled on Clusters within such a distributed setup.

Before enabling the Superivsor on a given vSphere Cluster, follow the official documentation to create a new Custom Cloud of type VMware vCenter for each vCenter/Datacenter combination. Configure the Service Engine Groups, Networks with their IP Pools and subnet prefixes, VRF Contexts, and IPAM profiles. Based on the scenarios discussed above, you may need to configure custom groups for Service Engines and Unique VIP networks for each Supervisor. If so, complete the steps outlined above.

Enable Supervisor on the Clusters(s) as per the official documentation. During the enablement process, log in to one of the Supervisor Control Plane VMs. See KB 90194 for instructions. Once logged in, edit the ako deployment on the Supervisor.

$ kubectl edit deployment -n vmware-system-ako vmware-system-ako-ako-controller-manager

The Deployment .spec.template.spec.container section looks something like this —

      containers:
      - command:
        - /manager
        env:
        - name: CTRL_IPADDRESS
          valueFrom:
            configMapKeyRef:
              key: controllerIP
              name: avi-k8s-config
        - name: ADVANCED_L4
          valueFrom:
            configMapKeyRef:
              key: advancedL4
              name: avi-k8s-config
        - name: CLOUD_NAME
          valueFrom:
            configMapKeyRef:
              key: cloudName
              name: avi-k8s-config
        - name: CLUSTER_ID
          valueFrom:
            configMapKeyRef:
              key: clusterID
              name: avi-k8s-config
        - name: AKO_API_PORT
          value: "9999"

Modify the CLOUD_NAME value to something like this —

        - name: CLOUD_NAME
          value: My_Custom_Cloud

where My_Custom_Cloud is the new Cloud you created in the NSX ALB, within which the Supervisor will reside. Once the Supervisor is fully enabled, the ako should be interacting with the NSX ALB controller and leveraging the new Cloud and its associated artifacts to configure the platform load balancers.