I received was asked what the maximum number of VMkernel interfaces per ESXi host was. Off the top of my head I was not aware of one so I consulted the vSphere 5.1 Configuration Maximums guide. The guide did list any VMkernel maximums.
So is the answer unlimited?
Well, to create a vmknic you must create a portgroup. For example, in ESXi 5.1 you can run the following commands:
esxcli network vswitch standard portgroup add -p <pgName> -v <vSwitchName> esxcli network ip interface add -i vmk<0-255> -p <pgName>
Per the vSphere 5.1 Configuration Maximums guide you can have:
- A maximum of 256 portgroups per standard switch. This means you should be able to create up to 256 vmknics per standard switch.
- A maximum of 256 ephemeral port groups per vCenter (not recommended per http://kb.vmware.com/kb/1022312). This means you should be able to create up to 256 vmknics per host.
- A maximum of 10,000 static portgroups per distributed switch. However, per esxcli it appears you are only allowed up to 256 vmknics (see comments after -i flag below) per host.
~ # uname -a VMkernel esx01.local 5.1.0 #1 SMP Release build-799733 Aug 1 2012 20:03:00 x86_64 GNU/Linux ~ # esxcli network ip interface add -h Error: Invalid option -h Usage: esxcli network ip interface add [cmd options] Description: add Add a new VMkernel network interface. Cmd options: -P|--dvport-id=<str> DVPort ID of the connection point. This requires --dvs-name to be given in the same command -s|--dvs-name=<str> DVSwitch name of the connection point. This requires --dvport-id to be given in the same command -i|--interface-name=<str> The name of the VMkernel network interface to create. This name must be in the form vmkX, where X is a number 0-255 -M|--mac-address=<str> Set the MAC address for the newly created VMkernel network interface. -m|--mtu=<long> Set the MTU setting for a given VMkernel network interface on creation -p|--portgroup-name=<str> The name of the vswitch port group to add this VMkernel network interface to.
I tested on an ESXi 5.1 host using the standard switch and I was only able to make it to 125 vmknics before running into an “Out of resources” errors:
~ # for n in $(seq 256); do echo $n; esxcli network vswitch standard portgroup add -p vmk$n -v vSwitch0; esxcli network ip interface add -i vmk$n -p vmk$n; done 1 2 3 ... 124 125 Error performing operation: Sysinfo error on operation returned status : Out of resources. Please see the VMkernel log for detailed error information 126 Error performing operation: Sysinfo error on operation returned status : Out of resources. Please see the VMkernel log for detailed error information ...
Looking at /var/log/vmkwarning.log I saw:
~ # tail -n 2 /var/log/vmkwarning.log 2013-03-05T15:05:44.637Z cpu6:8660)WARNING: Tcpip_Vmk: 851: Could not open a port to vmk126: Out of resources 2013-03-05T15:05:44.637Z cpu6:8660)WARNING: Tcpip_Vmk: 788: Unable to attach vmk125 to vmk127 [error = 0xbad0017]
The error code returns:
~ # vmkerrcode -l | grep 0017 VMK_NO_RESOURCES 195887127 0xbad0017 ENOMEM Out of resources
Next, I ran the same test on the distributed switch and made it to over 256 vmknics without issue. Does this mean that the distributed switch can have more VMkernel interfaces than the standard switch? Not so fast! I was almost thrown off by this as well until I remembered the default number of ports configured on a vSwitch is 128 (http://kb.vmware.com/kb/1008040). Upon increasing the number of ports on the standard switch and rebooting the host, repeating the test resulted in the same results as the distributed switch.
So back to the original question, what is the maximum number of VMkernel interfaces per ESXi host? The answer from a VMkernel perspective is unlimited. The real limits are those imposed in the vSphere 5.1 Configuration Maximums guide being:
- 256 port groups per standard switch (if applicable)
- 1050 active ports per host (VDS and VSS)
In both cases, I believe the limits specified are “soft” support maximums and not “hard” enforced maximums. At the end of the day, I cannot think of any reason why you would need more than a dozen VMkernel interfaces so all you need to know is that you should not hit any known limit.
© 2013, Steve Flanders. All rights reserved.
