Operation Timed Out

Another interested VMware problem today: I attempted to put a host into maintenance mode, but after making it to 60% the operation failed. The error message displayed was: Operation timed out. Trying again, I ran into the same issue. What was going on?

I started by checking the ESX host. First, I wanted to see if any VMs were running on the host:

[root@esx04 log]# ps aux www | grep volumes
root 29403 0.0 0.1 3684 672 pts/0 S 15:09 0:00 grep volumes

Good, no VMs. Next, I checked the most common logs:

• vmkwarning – looked normal
• vmksummary – looked normal
• vmkernel – looked normal
• hostd.log – found the timeout

The hostd.log had some interesting information:

[2010-04-06 14:56:48.279 ‘TaskManager’ 3076432800 info] Task Created : haTask-ha-host-vim.host.FirewallSystem.disableRuleset-156874510
[2010-04-06 14:56:48.485 ‘TaskManager’ 3076432800 info] Task Completed : haTask-ha-host-vim.host.FirewallSystem.disableRuleset-156874510
[2010-04-06 14:56:48.486 ‘Vmomi’ 3076432800 info] Activation [N5Vmomi10ActivationE:0x98a8858] : Invoke done [disableRuleset] on [vim.host.FirewallSystem:firewallSystem]
[2010-04-06 14:56:48.486 ‘Vmomi’ 3076432800 info] Throw vmodl.fault.InvalidRequest
[2010-04-06 14:56:48.486 ‘Vmomi’ 3076432800 info] Result:
(vmodl.fault.InvalidRequest) {dynamicType = <unset>,
msg = “”
}
[2010-04-06 14:56:50.921 ‘TaskManager’ 23395248 info] Task Created : haTask-ha-host-vim.HostSystem.enterMaintenanceMode-156874512
[2010-04-06 14:56:50.921 ‘ha-host’ 23395248 info] EnterMaintenanceMode, timeout = 1
[2010-04-06 14:56:50.921 ‘ha-host’ 23395248 info] ModeMgr::Enter: next = maintenance, current = normal, count = 1, timeout = 1
[2010-04-06 14:56:51.928 ‘ha-host’ 45681584 info] ModeTaskTimedOut
[2010-04-06 14:56:51.928 ‘TaskManager’ 45681584 info] Task Completed : haTask-ha-host-vim.HostSystem.enterMaintenanceMode-156874512

It appeared a firewall rule could not be disabled. Checking the firewall I noticed that aam was disabled, which is expected since I was trying to put the host back into maintenance mode. I re-enable aam and tried to enter maintenance mode again:

[2010-04-06 15:04:29.990 ‘TaskManager’ 22719408 info] Task Created : haTask-ha-host-vim.host.FirewallSystem.disableRuleset-156874566
[2010-04-06 15:04:30.501 ‘TaskManager’ 22719408 info] Task Completed : haTask-ha-host-vim.host.FirewallSystem.disableRuleset-156874566
[2010-04-06 15:04:32.728 ‘TaskManager’ 50289584 info] Task Created : haTask-ha-host-vim.HostSystem.enterMaintenanceMode-156874568
[2010-04-06 15:04:32.728 ‘ha-host’ 50289584 info] EnterMaintenanceMode, timeout = 1
[2010-04-06 15:04:32.728 ‘ha-host’ 50289584 info] ModeMgr::Enter: next = maintenance, current = normal, count = 1, timeout = 1
[2010-04-06 15:04:33.739 ‘ha-host’ 22719408 info] ModeTaskTimedOut

No more firewall issue, but still the operation timed out.
Any guesses on how to fix the issue? If you guessed restart the management services you would be correct! Upon doing this I was successfully able to enter maintenance mode.

© 2010, Steve Flanders. All rights reserved.