While I have seen people discuss this error message and solution, I figured it would be a good idea to discuss in terms of specific configurations such as on Cisco hardware and VMware virtualization. I feel this is important to understand the implications of the error message and to express the importance of BIOS configurations.

First, the issue: Cisco UCS B230-M2 blades (dual 10-core = 20 'processors') running ESXi were throwing processor halted log messages. While this in itself may or may not be an issue, under little load via VMware clone operations ESXi hosts were disconnecting from vCenter Server (vCS) and becoming unresponsive for several minutes. Further digging uncovered that when the ESXi host disconnected from vCS the logs shows that all processors on the host were halting at exactly the same time.

I having been working on a syslog architecture and one key component to the architect was leveraging log rotate for all log files. One section of my log rotate file looked like the following:

/path/to/logs/*
{
   daily
   dateext
   missingok
   copytruncate
   compress
   compresscmd /bin/bzip2
   compressext .bz2
   sharedscripts
   lastaction
      /usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
   endscript
}

The problem was, I noticed that after the cron for logrotate ran the system started to become slow. Looking at top I noticed several things: the load average continued going up, the logrotate process continued to run with the process consuming around 50% of memory, and the syslog process never restarted.

What was causing the problem?

The VMware Configuration Maximums document is something I reference quite often. One configuration maximum that became relevant for me this week was under ESXi Host Maximums - Storage Maximums - Fibre Channel: LUN ID. According to the document the maximum LUN ID is 255, but what does that mean? Does it mean that you can have a maximum of 255 LUN IDs or the maximum LUN ID number allowed is 255?

For those who know the answer, let me explain where my confusion came from:

1. Two items above LUN ID in the Configuration Maximums document is 'LUNs per host'. The maximum for 'LUNs per host' is 256. Like most numbering in Linux (e.g. arrays) LUN IDs start at 0. This means LUN IDs 0 to 255 are valid and would total 256, the maximum number of 'LUNs per host'.
2. Looking at the storage side, a very important piece of information would be the maximum number of LUNs per storage system. For an EMC VNX7500, the maximum number of LUNs (including private LUNs) is 8192. Since every LUN has to have a unique LUN ID this means on a VNX7500 at a minimum the LUN IDs 0 to 8191 must be valid.

So why was I looking at this maximum in the first place?

The latest half-width Cisco UCS B-series servers are beautiful pieces of machinery. In such a small footprint it is possible to get two processors each with 10 cores and 20 threads. In addition, the blade supports an amazing 512GB of memory and features the ability to support up to two hot-swappable SSD drives.

Unfortunately, while working with this bleeding edge technology, I have run into several limitations. I would like to share some of the limitations I have experienced in the workarounds available today.