Blog

WARNING: CpuSched: XXXX: processor apparently halted for XXXX ms

steve@sflanders.net (Steve Flanders) — Fri, 27 Jan 2012 04:29:07 GMT

While I have seen people discuss this error message and solution, I figured it would be a good idea to discuss in terms of specific configurations such as on Cisco hardware and VMware virtualization. I feel this is important to understand the implications of the error message and to express the importance of BIOS configurations.

First, the issue: Cisco UCS B230-M2 blades (dual 10-core = 20 'processors') running ESXi were throwing processor halted log messages. While this in itself may or may not be an issue, under little load via VMware clone operations ESXi hosts were disconnecting from vCenter Server (vCS) and becoming unresponsive for several minutes. Further digging uncovered that when the ESXi host disconnected from vCS the logs shows that all processors on the host were halting at exactly the same time.

Logrotate Limitations

steve@sflanders.net (Steve Flanders) — Sun, 16 Oct 2011 22:59:09 GMT

I having been working on a syslog architecture and one key component to the architect was leveraging log rotate for all log files. One section of my log rotate file looked like the following:

{codecitation class="brush:bash"}/path/to/logs/*
{
daily
dateext
missingok
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
sharedscripts
lastaction
/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
endscript
}{/codecitation}

The problem was, I noticed that after the cron for logrotate ran the system started to become slow. Looking at top I noticed several things: the load average continued going up, the logrotate process continued to run with the process consuming around 50% of memory, and the syslog process never restarted.

What was causing the problem?

ESXi LUN ID Maximum

steve@sflanders.net (Steve Flanders) — Sun, 02 Oct 2011 02:05:25 GMT

The VMware Configuration Maximums document is something I reference quite often. One configuration maximum that became relevant for me this week was under ESXi Host Maximums - Storage Maximums - Fibre Channel: LUN ID. According to the document the maximum LUN ID is 255, but what does that mean? Does it mean that you can have a maximum of 255 LUN IDs or the maximum LUN ID number allowed is 255?

For those who know the answer, let me explain where my confusion came from:

Two items above LUN ID in the Configuration Maximums document is 'LUNs per host'. The maximum for 'LUNs per host' is 256. Like most numbering in Linux (e.g. arrays) LUN IDs start at 0. This means LUN IDs 0 to 255 are valid and would total 256, the maximum number of 'LUNs per host'.
Looking at the storage side, a very important piece of information would be the maximum number of LUNs per storage system. For an EMC VNX7500, the maximum number of LUNs (including private LUNs) is 8192. Since every LUN has to have a unique LUN ID this means on a VNX7500 at a minimum the LUN IDs 0 to 8191 must be valid.

So why was I looking at this maximum in the first place?

Cisco UCS B230 M2 Caveats

steve@sflanders.net (Steve Flanders) — Mon, 12 Sep 2011 04:47:57 GMT

The latest half-width Cisco UCS B-series servers are beautiful pieces of machinery. In such a small footprint it is possible to get two processors each with 10 cores and 20 threads. In addition, the blade supports an amazing 512GB of memory and features the ability to support up to two hot-swappable SSD drives.

Unfortunately, while working with this bleeding edge technology, I have run into several limitations. I would like to share some of the limitations I have experienced in the workarounds available today.

Site Down 8/13 and 8/14

steve@sflanders.net (Steve Flanders) — Sat, 13 Aug 2011 04:31:34 GMT

I know it is short notice, but I wanted to let everyone know that my site will be down this weekend. More details to come!

Frequent Flyer Part 5: Car Rental

steve@sflanders.net (Steve Flanders) — Thu, 28 Jul 2011 02:39:28 GMT

I do not really have much to say about car rental companies other than to find one and stick with them. Most are identical in amenities and service. All car rental companies offer some kind of rewards program and whatever car rental company you choose you should ensure you sign up for the rewards program. Many companies have preferred car rental companies and some will even pay for you to join a rewards program (you read that right, car rental reward programs cost money). Be sure to check with your company to see what options are available to one.

Though most care rental companies are available at any destination you can think of, it is still a good idea to see what companies are available at your most common destinations when choosing a company.

The one comment I would make is that as reward status increases that you be sure to read over the amenities offered to you and take advantage of them whenever you can.

Frequent Flyer Part 4: Airlines

steve@sflanders.net (Steve Flanders) — Sun, 17 Jul 2011 04:01:01 GMT

Your choice of airline really matters and because you are going to marry one be sure to do your homework to prevent a bitter divorce. The most important decision when choosing an airline is to try and find one with non-stop service to your most common destinations. While you may not mind lay-overs and can benefit from the rewards of multiple segments especially for travel of short distances, they significantly increase the time of trips and increase the chances of being delayed.

Once you have narrowed down the choices, you should look to see what the remaining airlines offer in terms of amenities and incentives. You can search online for individual airlines, but I have included my thoughts on some of the airlines I have traveled. You will notice a lot of airlines are not listed as I do not have enough experiences with them to comment. From what I have heard from others, JetBlue is one of the best airlines. I plan to travel with them soon and will comment when I do.

vSphere 5.0 and Cloud Infrastructure Suite Announced!

steve@sflanders.net (Steve Flanders) — Thu, 14 Jul 2011 04:34:44 GMT

As I am sure you have heard by now, VMware announced yesterday the next step for the company in the cloud space. I would like to quickly highlight what this means for VMware users and some comments and questions that came to mind.

Frequent Flyer Part 3: Marriage

steve@sflanders.net (Steve Flanders) — Sun, 10 Jul 2011 15:39:12 GMT

Now that you have a basic understanding of your travel requirements and how to pack, it is time to pop the question and commit yourself. You need to commit to an airline or airline alliance, a hotel or hotel chain, and a car rental company. This is important in order to gain status and all the perks that come with it. While this may not sound important initially, it you will be, especially if you expect to be traveling more than 25% of the time.

Frequent Flyer Part 2: Luggage

steve@sflanders.net (Steve Flanders) — Tue, 05 Jul 2011 23:27:17 GMT

Packing and luggage are possibly the most important decisions you need to make when traveling. To get you started, you should answer the following questions:

What do you need to take with you? (make an exact list)
How much luggage do you plan to bring? (list the exact items)
What type of luggage to you have or plan to purchase? (list sizes)