VC Upgrade went haywire
Yesterday was a fun day. What looked like a simple, clean upgrade became something more interesting. After making a before snapshot I started the VirtualCenter installation wizard. Then the fun started. Nothing happened.
Let me rephrase that. NOTHING happened. Machine hung. Clicking the reboot button in the VI client caused the machine to reboot. At least, so it seemed.
The machine didn’t get back to life. VirtualCenter thought that it was in limbo, not started and not shutdown. So I restarted the management services on the ESX host, you know,
service mgmt-vmware restart and I even restarted the VirtualCenter client by hand like
service vpxa restart
It looked like the problem got bigger. VirtualCenter now saw the machine as invalid. Logging on the ESX host with the VI client it looked very very strange.
After trying the usual stuff like
vmware-cmd /vmfs/volumes/Production1/SRV01/SRV01.vmx stop hard
I went for the more rigorous approach. I tried unregistering and registering the VM with the host like:
vmware-cmd -s unregister /vmfs/volumes/Production1/SRV01/SRV01.vmx
vmware-cmd -s register /vmfs/volumes/Production1/SRV01/SRV01.vmx
Even this didn’t work. FUBAR as they used to say in the telco business. Since I suspected a host issue I tried unregistering the machine from this host and registered it on another ESX host.
Everything looked great again. The machine was visible on the ESX host. I carefully started to breath again.. Too soon.. Dang! “Could not power on VM: No swap file. Failed to power on VM”.
Knowing the swap file was present I suspected the files were still in use by the original VM. While googling I came across the site from Rene Helweg , with whom I worked in the past for a client in The Netherlands.
ps aux | grep
Strangely enough ps didn’t give anything. No rogue processes. Hmm.. gonna be a long day.
After consulting some pals (TAZZ) I tried the vm-support approach Ifound on the blog from Sanjat Kabi.
vm-support -x gave a list of all worlds. My hung machine was in the list.
vm-support -X it does some really nifty things. Not sure what it does exactly but it said: “This might crash your VM’.. well, it is broken, who cares. After that it was ‘This will crash your VM’. Fine 😀
It ran for a while and.. whadayaknow.. It worked!! VM crashed so I started it. It was fine! Still in the mode of updating VirtualCenter I started the installer again… Wheeeeee!!!! Running. Finally the machine was upgraded to Update 3.
Update Just got the client on AIM. Their exact words “The customer was happy .. yippee :-P”
- Untangle the mess! Network issues with VMware VMs by Sander Martijn
- VMware Fling – PowerActions for vSphere Web Client by Erik Scholten
- PowerShell Friday: Creating Virtual Machines by Sander Martijn
- Removing VMware Data Security after vShield to NSX upgrade by Martijn Smit
- VMware Fling – ESXi Mac Learning dvFilter by Anne Jan Elsinga