I was called into an interesting issue over the past week. I was told that a chassis worth of UCS blades had powered off without any apparent reason bringing down part of production. Initial troubleshooting of the issue showed no real culprits. UCSM was clean of errors except for an IOM post error. A show-tech command was initiated and a sev1 was opened with Cisco TAC. The technician on-call attempted to power on the servers by selecting them all in UCSM, right-clicking on them, and selecting reset. The blades powered on and came back online without issue.
So what caused the blades to power off unexpectedly?
TAC combed the logs and asked if we either:
- Hit the power button on the front of the blades
- Selected multiple blades in UCSM, right-clicked, and select reset
The answer to number 2 was yes. This is when we were presented with bug id CSCty26754, which states:
Symptom:
Blades power off unexpectedly, and stay in an off state.
Conditions:
A shallow discovery has happened which puts the blade into its desired power state. Some examples of actions that can trigger a shallow discovery are:
- Loss of any link between the FI and the IOM
- Reset of an IOM
- Killing a process with debug plugin
- Re-acknowledge of a chassis
Blades that have been powered on with the following methods will be left in this inconsistent power state:
- Pressing the physical power button on the front
- Clicking the reset button on the server in the equipment tab
- Right clicking the server in the list of servers on the equipment tab and selecting reset.
Workaround:
When a blade is powered off, only use the Power On button on the General tab to turn on the blade.
If the service-profile has a desired power state of Off, but the blade is actually On: Click the Set Desired Power State button that will appear on the General tab of the service-profile and change the desired power state to On. The Set Desired Power State button will disappear when the desired and actual power states match. Under the Status Details drop down, the Desired Power State will be changed too Up.
Turns out UCS has the notion of an actual power state and a desired power state (more on this in a future blog post). Certain operations can result in the two power states getting out of sync. While a disconnect in power states in itself is not an issue, a regression in UCS firmware versions 2.0-2.0(2l) can lead to blades powering off unexpectedly. In our case, the IOM post error resulted in the shallow discovery that powered off the blades.
So how can you check if your systems are vulnerable to this issue? First, you would need to check your UCS version and confirm you are running a version between 2.0 and 2.0(2l). Second, you would need to check the desired power state of each blade in your environment. To manually check the desired power state in each blade you can:
- Log into the UCSM GUI and under either the Equipment or Servers tab select each blade / service profile as appropriate. If once selected you notice a ‘Set Desired Power State’ option under the Actions section then the power states are not in sync. If you do not see the aforementioned option then the power states are in sync. (I have requested a bug/enhancement be created for this as well – I believe if the power states are in sync then the option should still exist under Actions either grayed out or with the ability to changed the desired power state.)
Example of power states being in sync – no action required
Example of power states in mismatch – correction required - Log into the UCSM CLI – not sure the exact commands here, but must be possible
As you can see, manually checking each blade is time consuming and depending on the size of your infrastructure may not be feasible. In addition, while running a code version between 2.0 and 2.0(2l) you should frequently check to ensure power states do not get out of sync. As such, I worked with an awesome Cisco programmer by the name of Eric Williams and we created a PowerShell script, shown below, capable of detecting as well as fixing a power state mismatch. I was able to get this script tied into our monitoring to ensure the power states stay in sync while an upgrade change request is processed and approved.
NOTE: This is not an officially supported cmdlet. Your milage with this script may vary. Do not run this script in any production environment without first fulling understanding this script and of course testing it. Eric and I are not responsible for any problems you experience.
Function Get-UcsPowerStates {
<#
.SYNOPSIS
Check for actual power state and desired power state mismatch.
.DESCRIPTION
The Get-UcsPowerStates function uses the UCS Power Tools to retrieve the actual power state versus the desired power state.
Each UCS domains is checked sequentially, not in parallel.
.PARAMETER Domains
Array of UCS domains (IPs or FQDNs).
.PARAMETER Username
The username for the UCS domains (assumes all domains have the same username).
.PARAMETER Password
The password for the UCS domains (assumes all domains have the same password).
.PARAMETER Sync
Tells function to fix any issues experienced.
This is an optional parameter; if it is not included, issues will be reported, but not fixed.
.PARAMETER Info
Tells function to output additional information and then display a return value.
This is an optional parameter; if it is not included, only a return value will be displayed.
.EXAMPLE
UCS-CheckPowerStates -domains @('10.20.30.6', '10.20.30.9') -username 'psuser' -password 'pspasswd'
Connect to multiple UCS domains and check power states.
.EXAMPLE
UCS-CheckPowerStates -domains @('10.20.30.6') -username 'psuser' -password 'pspasswd' -sync
Connect to a UCS domain, check power state, and sync the desired power state if necessary.
.NOTES
Authors: Eric Williams and Steve Flanders ([email protected])
Date: June 6, 2012
#>
Param (
[Parameter(Mandatory=$true, Position=0, HelpMessage="Array of UCS domains (either IP or FQDN)")]
[array] $Domains,
[Parameter(Mandatory=$true, Position=1, HelpMessage="Username for UCS domains (assumes all domains have the same username)")]
[string] $Username,
[Parameter(Mandatory=$true, Position=2, HelpMessage="Password for UCS domains (assumes all domains have the same password)")]
[string] $Password,
[Parameter(Mandatory=$false, Position=3, HelpMessage="Tells function to fix any issues experienced.")]
[switch] $Sync,
[Parameter(Mandatory=$false, Position=4, HelpMessage="Tells function to output additional information.")]
[switch] $Info
)
Set-UcsPowerToolConfiguration -SupportMultipleDefaultUcs $true
$mismatch = 0
Try {
$ucspasswd = ConvertTo-SecureString $password -AsPlainText -Force
$ucscreds = New-Object System.Management.Automation.PSCredential ($username, $ucsPasswd)
foreach ($singleucs in $domains) {
if ($Info.isPresent) { Write-Host "INFO: Logging into UCS Domain: $singleucs" }
$ucslogin = Connect-Ucs -Credential $ucscreds $singleucs
}
if ($Info.isPresent) { Write-Host "INFO: Getting list of unassociated SPs" }
$assocSps = Get-UcsServiceProfile -Filter 'AssocState -ine "unassociated" -and Type -inotlike "*template"'
foreach ($sp in $assocSps) {
if ($Info.isPresent) { Write-Host "INFO: Checking Domain: $($sp.ucs) Service Profile: $($sp.name) for power state mismatch" }
$spDesiredPowerState = $sp | Get-UcsServerPower -Ucs $sp.Ucs
$spServer = Get-UcsManagedObject -Dn $sp.PnDn -Ucs $sp.Ucs
if ( $spDesiredPowerState.State -ieq "up" ) {
if ( $($spServer).OperPower -ine "on") {
if ($Info.isPresent) { Write-Host "CRIT: Service Profile: $($sp.name) desired power state 'up' mismatch with physical server: $($spServer.Dn) power state: $($spServer.OperPower)" }
if ($Sync.isPresent) {
if ($Info.isPresent) { Write-Host "INFO: Powering on Domain: $($sp.ucs) Service Profile: $($sp.name)" }
$setsppower = $sp | Set-ucsserverpower -ucs $sp.Ucs -State "admin-up" -Force
}
++$mismatch
} else {
if ($Info.isPresent) { Write-Host "INFO: Domain: $($sp.ucs) Service Profile: $($sp.name) desired power state and physical server: $($spServer.Dn) power state matches" }
}
} elseif ( $spDesiredPowerState.State -ieq "down" ) {
if ( $($spServer).OperPower -ine "off" ) {
if ($Info.isPresent) { Write-Host "CRIT: Domain: $($sp.ucs) Service Profile: $($sp.name) desired power state 'down' mismatch with physical server: $($spServer.Dn) power state: $($spServer.OperPower)" }
if ($Sync.isPresent) {
if ($Info.isPresent) { Write-Host "INFO: Setting Domain: $($sp.ucs) Service Profile: $($sp.name) desired power to 'up'" }
$setsppower = $sp | Set-UcsServerPower -ucs $sp.Ucs -State "up" -force
}
++$mismatch
} else {
if ($Info.isPresent) { Write-Host "INFO: Domain: $($sp.ucs) Service Profile: $($sp.name) desired power state and physical server: $($spServer.Dn) power state matches" }
}
}
}
Disconnect-Ucs
}
Catch {
Write-Host "CRIT: Error occurred in script"
Write-Host $($Error)
Disconnect-Ucs
Break
}
if ($Info.isPresent) { Write-Host "`n======================================================" }
if ($mismatch -eq 0) {
if ($Info.isPresent) { Write-Host "INFO: No issues found" }
$return = 0
} else {
if ($Sync.isPresent) {
if ($Info.isPresent) { Write-Host "INFO: $mismatch issues found and fixed" }
$return = 2
} else {
if ($Info.isPresent) { Write-Host "CRIT: $mismatch issues found! Run script with 'fix' parameter." }
$return = 1
}
}
if ($Info.isPresent) { Write-Host "======================================================`n" }
Return $return
}
More information on how to properly change blade power states can be found here.
© 2012, Steve Flanders. All rights reserved.
