For one of the group projects I am currently working on, we are in the process of ordering hardware for a VMware deployment. The other day I was asked by one of the hardware engineers for a standard BIOS configuration to be applied to all newly ordered systems. The question was directed at me because I am the VMware Subject Matter Expert (SME). This reminded me how difficult it is to determine the best practices on configuring a BIOS for a VMware environment. Searching the Internet, I found multiple attempts over the years to put together a VMware optimized list, but the list was never maintained. In addition, I found that most BIOS Optimization Guides (BOGs) were vendor specific. Finally, I was unable to find a BOG specific to ESX(i) and as many of you know a properly configured BIOS can make all the difference.
As such, I decided it would be a good idea to put together a VMware BIOS Optimization Guide or vBOG. While every BIOS is different, I hope to put together a comprehensive guide over time that will be continually maintained. My goal is not to explain every BIOS option, but to highlight all the VMware specific and important ones. I expect this topic to be controversial and I welcome any feedback. I will attempt to provide supporting documentation on all recommendations I provide. Please be advised that I am in no way a hardware expert. These recommendations come primarily from online research and general practice. In the future, I hope to add actually benchmarks into the guide to reforce the recommendations made.
General Comments (based primarily on the VMware Performance Best Practices for vSphere 4.0)
- Make sure you are running the latest version of the BIOS available for your system. There are constant bug fixes and additions to the BIOS and I am a big fan of preventative maintenance so I would highly recommend this as well. It is important to note, once you update your BIOS be sure check your settings again as some updates change default settings.
- Turn on all hardware virtualization assistances. Hardware always has less overhead than software so if your hardware can take some of the load off of VMware why would you not do it?
- Turn off any power saving features if performance is your primary concern. The downside to this approach is that operating expenses will increase for the servers.
Recommended Settings (grouped into categories)
Boot Order
While not technically VMware specific / important, I feel it is necessary to discuss the implications of the configuration chosen. I recommend performing all ESX(i) installations in a scripted fashion to prevent human error and to establish a standard deployment model. As such, I recommend the following settings:
- NIC Finction Support: [PXE]
- NIC1 Option ROM: [Enabled]
- NIC2 Option ROM: [Disabled]
- PXE Boot Only: [Disabled]
- Force PXE Boot First: [Enabled]
- 1st Boot Device: [Network: IBA GE Slot 1]
- 2nd Boot Device: [Network: IBA GE Slot 2]
- 3rd Boot Device: [HDD:P0]
- 4nd Boot Device: [CD/DVD]
- 5th Boot Device: [Disabled]
Hardware
- Active Processor Cores: [All] – VMware Performance Best Practices for vSphere 4.0 states that all populated sockets and all cores in each socket should be enabled.
- All Serial ports: [Disabled] – VMware Performance Best Practices for vSphere 4.0 states that you should disabled all unneeded devices. Unless serial console connectivity is used this should be disabled.
- All USB ports: [Disabled] – VMware Performance Best Practices for vSphere 4.0 states that you should disabled all unneeded devices. Unless needed or no CD/DVD drive exists this should be disabled.
- BIOS EHCI (EHCI = Enhanced Host Controller Interface) Hand-Off: [Disabled] – Used for USB 2.0. VMware Performance Best Practices for vSphere 4.0 states that you should disabled all unneeded devices. If USB support is not required, disable this setting.
- Hard Disk Write Protect: [Disabled] – If SATA Configuration is AHCI this should be disabled.
- SATA Configuration: [AHCI] – AHCI appears to give the best performance
- Wait For ‘F1’ If Error – While VMware CIM should pick up any hardware problems, it is a good idea to enable this option to prevent a system in a bad state from coming online.
Performance
- Adjacent Cache Line Prefetcher: [Disabled] – For performance reasons it typically makes sense to disable this. For example, Java applications can significantly benefit from this being disabled (see: http://blogs.vmware.com/performance/2008/08/esx-runs-java-v.html)
- Channel Interleaving: [Disabled] – VMware Performance Best Practices for vSphere 4.0 states that some NUMA-capable systems provide an option to disable NUMA by enabling node interleaving. VMware states that in most cases you will get the best performance by disabling node interleaving. If you do for some reason decide to enable this please see http://kb.vmware.com/kb/1021454 for directions on how to enable interleaving NUMA nodes on ESX(i) hosts.
- Hyper-Threading Technology(HT): [Enabled] – VMware Performance Best Practices for vSphere 4.0 states that hyper-threading should be enabled in the BIOS. The only comment I would make here is that if FT is to be used HT should be disabled.
- NUMA Support: [Enabled] – ESX is NUMA-aware and benchmarks have shown that NUMA performs better than interleaving (e.g. http://blogs.vmware.com/performance/2007/02/studying_numa_w.html).
- TurboMode (Intel): [Enabled] – VMware Performance Best Practices for vSphere 4.0 states that any processor with a “Turbo Mode” should have that feature enabled.
- Rank Interleaving: [Enabled] – VMware Performance Best Practices for vSphere 4.0 only talks about node interleaving and NUMA support. Since it does not mention rank interleaving, I feel this option should be enabled unless performance tests show otherwise.
Power
- C1E Support (Intel): [Disabled] – VMware Performance Best Practices for vSphere 4.0 states that this should be disabled. For the best VMware performance, I agree this feature should be disabled. Be advised that disabling this feature will result in increased processor power consumption and as such increased operating cost.
- Restore on AC Power Loss: [Last State] – While not technically VMware specific / important this can prevent unwanted servers from coming back on after a power outage and ensure that servers that were powered on automatically come back on after a power outage.
- tRankIdle Optimized: [Disabled] – Since this is a power saving feature, it should be disabled for the maximum VMware performance.
Security
- No execute (NX – Intel) / Execute disable (XD – AMD): [Enabled] – Must be enabled to support VMware EVC mode (see http://kb.vmware.com/kb/1003212). Even if EVC is not enabled this option should be enabled for security purposes and in case EVC is ever enabled in the future. There are no known performance implications for this decision.
Virtualization – All available options should be enabled!
- Intel VT-c: [Enabled] – Please note enabling this feature is not sufficient. Once the feature is enabled, you must completely power off and then power on your server (i.e. cold reboot) for the changes to take place. For more information, please see http://communities.vmware.com/docs/DOC-8978.
- Intel VT-d: [Enabled] – Please note enabling this feature is not sufficient. Once the feature is enabled, you must completely power off and then power on your server (i.e. cold reboot) for the changes to take place. For more information, please see http://communities.vmware.com/docs/DOC-8978.
- Intel VT-x: [Enabled] – Please note enabling this feature is not sufficient. Once the feature is enabled, you must completely power off and then power on your server (i.e. cold reboot) for the changes to take place. For more information, please see http://communities.vmware.com/docs/DOC-8978.
- Virtualization Tech (VT – Intel): [Enabled] – Amongst other things, this allows you to run 64-bit guest VMs. Please note enabling this feature is not sufficient. Once the feature is enabled, you must completely power off and then power on your server (i.e. cold reboot) for the changes to take place. For more information, please see http://communities.vmware.com/docs/DOC-8978.
© 2010, Steve Flanders. All rights reserved.
