Blog

VMworld Call for Papers - Submitted!

steve@sflanders.net (Steve Flanders) — Fri, 18 May 2012 20:14:34 GMT

I am happy to announce that I have submitted my first session for VMworld:

Scale-Out Cloud Infrastructure - A Look into the Cloud Foundry PaaS. In this session vSpecialist Tarik Dwiek and I will discuss how the infrastructure for Cloud Foundry has been architected to ensure scalability. In addition, we will cover specific testing done between VMware and EMC in order to fully understand performance, reliability, and replication during typical Cloud Foundry operational functions.

Now it is a waiting game to see if the session is accepted. I am really excited about the topic and hope to have the opportunity to present at VMworld 2012!

Logrotate Limitations Revisited

steve@sflanders.net (Steve Flanders) — Sat, 12 May 2012 16:20:48 GMT

The more I use logrotate the less I like it. If you recall from my previous post on logrotate, I choose to leverage the copytruncate option. While this configuration seemed to work well when I tested it, I have now experienced some significant limitations that are not documented in the man page:

After rotation, high volume logs files remained the same size and continued to grow
Pre and post rotated high volume log files contained NUL characters
Large sized log files lost messages during logrotate operation

So what caused the issues, what was the impact, and how can you rotate logs messages and not experience these issues?

Starting Vyatta router: configure failed!

steve@sflanders.net (Steve Flanders) — Sun, 06 May 2012 22:52:50 GMT

Since I have experienced this issue several times before and I can never remember the commands needed to figure out what is wrong, I figured I would write a quick post. The issue is upon reboot of a Vyatta firewall you receive a 'configure failed!' error message. The exact error message depends on which features you have enabled on your device, but one example is:

Starting Vyatta router: migrate r1-system firewall configure failed!

So how do you check what changed / what is causing the issue?

vCloud Director and Wildcard SSL Certificates

steve@sflanders.net (Steve Flanders) — Sun, 22 Apr 2012 22:02:16 GMT

I was recently asked to assist in configuring a wildcard SSL certificate on a pair of vCloud Director (vCD) cells. While the certificate had been installed on the cells, some browsers were displaying SSL errors such as the following:

While other browsers appeared to work:	Until you drilled down a little further:

In addition, while uploading/downloading VMs SSL errors like the following were displayed:

So what was going on and how can it be fixed?

Cisco Bug: Show Commands Cause (Dual) Fabric Reboot(s)

steve@sflanders.net (Steve Flanders) — Sat, 10 Mar 2012 17:56:45 GMT

Over the last two weeks I have been hit by the same UCS bug, though by different means, twice and as such I would like to educate others about it. The issue initially came up after running a 'show tech' command on a UCS Fabric Interconnect (FI). Shortly after the process started my session to the FI dropped. Since I have experienced random disconnects from an FI in the past I tried to reconnect. To my surprise the FI was unresponsive. Not knowing what was going on I tried the second FI and it also was not responding. A ping check confirmed my fear, both FIs were down.

For those who have never experienced a dual fabric reboot on an active/production environment before, the ten minutes that follow will be the longest of your life (even if you do have access to the console port - locally or remotely). After about ten minutes the FIs started to respond again. As if a dual fabric reboot was not enough, the problem did not end there. About 5-10 minutes after the FIs came back online they went down again! This cycle continued until manual intervention stopped it.

So what was the problem; what was the impact; how can you fix it; and how can you prevent it?

vWYNTK: vSphere Storage Appliance 1.0

steve@sflanders.net (Steve Flanders) — Tue, 21 Feb 2012 01:42:12 GMT

With the announcement of vSphere 5.0 VMware also announced the first edition of the vSphere Storage Appliance (VSA). This new product is targeted toward SMBs who may not have the budget or want to deal with the complexity of building a storage infrastructure, but who would like to leverage more advanced features of ESXi/vCS like DRS and HA.

Since its announcement, I have been asked several times if/when this product should be implemented. While there are some very good use-cases for this product, it is important to understand the current target audience and the limitations of this release. Below are the details one should keep in mind.

Introducing vStuff!

steve@sflanders.net (Steve Flanders) — Sat, 11 Feb 2012 11:12:22 GMT

I have added a new menu item called vStuff. Over the next several months, I plan on posting VMware specific information under this header for reference. To get the section started, I have posted the beta version of the vSphere BIOS Optimization Guide (vBOG). Expect many more exciting additions soon!

WARNING: CpuSched: XXXX: processor apparently halted for XXXX ms

steve@sflanders.net (Steve Flanders) — Fri, 27 Jan 2012 04:29:07 GMT

While I have seen people discuss this error message and solution, I figured it would be a good idea to discuss in terms of specific configurations such as on Cisco hardware and VMware virtualization. I feel this is important to understand the implications of the error message and to express the importance of BIOS configurations.

First, the issue: Cisco UCS B230-M2 blades (dual 10-core = 20 'processors') running ESXi were throwing processor halted log messages. While this in itself may or may not be an issue, under little load via VMware clone operations ESXi hosts were disconnecting from vCenter Server (vCS) and becoming unresponsive for several minutes. Further digging uncovered that when the ESXi host disconnected from vCS the logs shows that all processors on the host were halting at exactly the same time.

Logrotate Limitations

steve@sflanders.net (Steve Flanders) — Sun, 16 Oct 2011 22:59:09 GMT

I having been working on a syslog architecture and one key component to the architect was leveraging log rotate for all log files. One section of my log rotate file looked like the following:

{codecitation class="brush:bash"}/path/to/logs/*
{
daily
dateext
missingok
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
sharedscripts
lastaction
/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
endscript
}{/codecitation}

The problem was, I noticed that after the cron for logrotate ran the system started to become slow. Looking at top I noticed several things: the load average continued going up, the logrotate process continued to run with the process consuming around 50% of memory, and the syslog process never restarted.

What was causing the problem?

ESXi LUN ID Maximum

steve@sflanders.net (Steve Flanders) — Sun, 02 Oct 2011 02:05:25 GMT

The VMware Configuration Maximums document is something I reference quite often. One configuration maximum that became relevant for me this week was under ESXi Host Maximums - Storage Maximums - Fibre Channel: LUN ID. According to the document the maximum LUN ID is 255, but what does that mean? Does it mean that you can have a maximum of 255 LUN IDs or the maximum LUN ID number allowed is 255?

For those who know the answer, let me explain where my confusion came from:

Two items above LUN ID in the Configuration Maximums document is 'LUNs per host'. The maximum for 'LUNs per host' is 256. Like most numbering in Linux (e.g. arrays) LUN IDs start at 0. This means LUN IDs 0 to 255 are valid and would total 256, the maximum number of 'LUNs per host'.
Looking at the storage side, a very important piece of information would be the maximum number of LUNs per storage system. For an EMC VNX7500, the maximum number of LUNs (including private LUNs) is 8192. Since every LUN has to have a unique LUN ID this means on a VNX7500 at a minimum the LUN IDs 0 to 8191 must be valid.

So why was I looking at this maximum in the first place?