When using nested ESXi we have to enable either Promiscuous mode or MAC Learning on the VDS on the physical host running the nested environment. Forged transmits is also required to be enabled. I changed to MAC Learning long ago since I knew Promiscuous mode had a performance impact. I have had great results running a nested VCF lab but keep hearing about others having performance issues, and sometimes it comes down to slow storage, low memory or weak processors, but not always. I was wondering what kind of performance impact Promiscuous mode vs MAC Learning could have in my lab so I tested it using iperf3 and here is the result.
Note that if you are using vSphere Standard Switches (VSS) instead of vSphere Distributed Switches (VDS), you are stuck with using Promiscuous mode. I would recommend deploying a vCenter and set up a VDS if you want to use nested ESXi, especially if you want to run nested VCF.
Promiscuous mode
[root@esxi-2:~] /usr/lib/vmware/vsan/bin/iperf3.copy -i 1 -t 10 -c 10.0.0.101
Connecting to host 10.0.0.101, port 5201
[ 5] local 10.0.0.102 port 30109 connected to 10.0.0.101 port 5201
iperf3: getsockopt - Function not implemented
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 35.2 MBytes 296 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 1.00-2.00 sec 14.5 MBytes 122 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 2.00-3.00 sec 24.2 MBytes 203 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 3.00-4.00 sec 16.1 MBytes 135 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 4.00-5.00 sec 22.1 MBytes 186 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 5.00-6.00 sec 18.6 MBytes 156 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 6.00-7.00 sec 21.0 MBytes 176 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 7.00-8.00 sec 19.2 MBytes 161 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 8.00-9.00 sec 19.5 MBytes 164 Mbits/sec 0 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 9.00-10.00 sec 20.5 MBytes 172 Mbits/sec 0 0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 211 MBytes 177 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 211 MBytes 177 Mbits/sec receiverMAC Learning
root@esxi-2:~] /usr/lib/vmware/vsan/bin/iperf3.copy -i 1 -t 10 -c 10.0.0.101
Connecting to host 10.0.0.101, port 5201
[ 5] local 10.0.0.102 port 60767 connected to 10.0.0.101 port 5201
iperf3: getsockopt - Function not implemented
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1019 MBytes 8.54 Gbits/sec 469059936 0.00 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 1.00-2.00 sec 1011 MBytes 8.48 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 2.00-3.00 sec 987 MBytes 8.28 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 3.00-4.00 sec 1000 MBytes 8.38 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 4.00-5.00 sec 1.01 GBytes 8.68 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 5.00-6.00 sec 1.03 GBytes 8.81 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 6.00-7.00 sec 1.01 GBytes 8.68 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 7.00-8.00 sec 995 MBytes 8.35 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 8.00-9.00 sec 1004 MBytes 8.42 Gbits/sec 0 215 Bytes
iperf3: getsockopt - Function not implemented
[ 5] 9.00-10.00 sec 1.00 GBytes 8.59 Gbits/sec 3825907360 0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.92 GBytes 8.52 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 9.92 GBytes 8.52 Gbits/sec receiver
As you can see, the difference is huge so please make sure to always use Mac Learning over Promiscuous mode in your nested lab.
Below you can see the result from the vSAN Network Performance Test.
Promiscuous mode
MAC Learning
If you have to use Promiscuous mode on a VSS it seems that using a single Active NIC can minimize the performance impact. Take a look at Daniel Kriegers blog for more details: MAC Learning is your friend
CyberNils.net 