Deploy Nested VCF to NSX-T Overlay

I have used VLC to deploy nested VCF for a long time and I am quite happy with how it works. VLC is usually deployed to a VLAN Trunk Port Group. This requires the VLANs used in the nested VCF to be configured on the physical switches in the environment. This does not scale well, and it is hard to automate. By following the steps below we are able to deploy VLC to an NSX-T Overlay Segment which allows each VFC instance to be isolated on their own layer 2 network. NSX-T Overlay Segments can be deployed automatically and they don’t require any changes to the physical network. This also allows us to use overlapping IP addressing between them. I have not yet tested to connect my Segment to a Tier-1 Gateway so that the nested VCF can connect to any external networks, but I plan to do this soon and update this post.

NSX-T Configuration

The following configuration needs to be done on the hosting NSX-T environment.

IP Discovery Profile

Namevcf-nested-ip-profile
Duplicate IP DetectionDisabled
ARP SnoopingEnabled
ARP Binding Limit256
ND SnoopingDisabled
ND Snooping Limit3
ARP ND Binding Limit Timeout10
Trust on First UseEnabled
DHCP SnoopingDisabled
DHCP Snooping – IPv6Disabled
VMware ToolsEnabled
VMware Tools – IPv6Disabled

MAC Discovery Profile

Namevcf-nested-mac-profile
MAC ChangeEnabled
MAC Learning Aging Time600
MAC LearningEnabled
MAC Limit4096
MAC Limit PolicyAllow
Unknown Unicast FloodingEnabled

Segment Security Profile

Namevcf-nested-security-profile
BPDU FilterDisabled
BPDU Filter Allow ListNot Set
Server BlockDisabled
Server Block – IPv6Disabled
Non-IP Traffic BlockDisabled
Rate LimitsDisabled
Receive Broadcast
0
Receive Multicast0
Client BlockDisabled
Client Block – IPv6Disabled
RA GuardEnabled
Transmit Broadcast0
Transmit Multicast0

Segments

Namevcf-nested-trunk-01
Transport Zoneoverlay-tz
Connected GatewayNone
SubnetNone
Profilesvcf-nested-ip-profile
vcf-nested-mac-profile
vcf-nested-security-profile
VLAN0-4094

Jump Host Configuration

To deploy VLC we need a jump host with two network adapters, one connected to your management network so that we can access it with RDP, and one connected to the nested environment so that we can connect to the nested appliances there. More details on this can be found in the VLC installation guide.

Jump Host 01

Namejumpy-01
OSWindows Server 2019
NIC1  Portgroup: pg-management
Driver: VMXNET3
NIC2Portgroup: vcf-nested-trunk-01
GW: None
VLAN: 10 (Tagged in Guest OS)
Driver: VMXNET3
SoftwarePowershell 5.1+
PowerCLI 12.1+
OVFTool 4.4+
.Net Framework
Static RoutesFor example: route ADD 10.50.0.0 MASK 255.255.255.0 10.0.0.221
Windows FirewallOn
Powershell PolicySet-ExecutionPolicy Unrestricted

VLC Configuration

LocationC:\VLC
Bringup ConfigurationC:\VLC\NOLIC-44-TMM-vcf-ems-public.json
ESXi Host ConfigurationC:\VLC\conf\default_mgmt_hosthw.json
LicensesMust be added to NOLIC-44-TMM-vcf-ems-public.json
Cloud BuilderC:\VLC\VMware-Cloud-Builder-4.4.0.0-19312029_OVF10.ova
MTU1700 (can probably be increased to 8800 or more)

VLCGui.ps1 Configuration

The following changes need to be done to the default VLCGui.ps1 to make it work with NSX-T.

Changed the following to be able to select NSX-T Segments:

From
If ($isSecSet.AllowPromiscuous.Value -and $isSecSet.ForgedTransmits.Value -and $isSecSet.MacChanges.Value){   
To
If(-not ($isSecSet.AllowPromiscuous.Value -and $isSecSet.ForgedTransmits.Value -and $isSecSet.MacChanges.Value)){  

Changed the following to get 1500 bytes MTU on vSwitch0:

From
$kscfg+="esxcli network vswitch standard set -v vSwitch0 -m 9000`n"
To
$kscfg+="esxcli network vswitch standard set -v vSwitch0 -m 1500`n"

Added the following to recreate vmk0 so that it gets a unique MAC address:

$kscfg+="esxcfg-vmknic --del vmk0 -p `"Management Network`"`n"
$kscfg+="esxcfg-vmknic --add vmk0 --portgroup `"Management Network`" --ip `${IPADDR} --netmask `${SUBNET} --mtu 1500`n"
$kscfg+="esxcfg-route -a default `${IPGW}`n"

Change the MAC Address of NSX-T Virtual Distributed Router

You must change the default MAC address of the NSX-T virtual distributed router in the nested VCF deployment so that it does not use the same MAC address that is used by the hosting NSX-T virtual distributed router.

Change the MAC Address of NSX-T Virtual Distributed Router

An alternative is to configure the hosting NSX-T environment’s Overlay Transport Zone with the nested_nsx property set to true, but this has to be done when creating the Transport Zone.

Thanks to Ben Sier for helping me getting this to work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s