Monday, January 19, 2009

VirtualCenter Physical to Virtual

This post describes a process I executed to replace a physical VMware VirtualCenter box with a virtual equivalent running in its own cluster. There were no VM outages of machines running on the cluster – each ESX kept merrily running the virtuals until everything was sorted out with the VC layer.

Note that this was done in a lab instance of ESX 3.5 and VirtualCenter 2.5, had this been production I probably would have taken a little more care.

I don’t think there is any compelling reason why you wouldn’t run a virtual VC box, it would be hypocritical of VMware to suggest virtualizing your application servers, except VirtualCenter which should be physical. Having said that, this process could also be used in reverse, taking a VM VirtualCenter instance physical should the need arise. This would also be useful for a VirtualCenter disaster recovery scenario.

The configuration before completing these steps

  1. Physical VirtualCenter 2.5 running on Windows Server 2003, called vc01, static IP
  2. Virtual Windows Server 2003 computer running in the cluster, called vc02, dynamic IP \
  3. SQL 2005 database for VC, running on a separate SQL server.
  4. ESX 3.5 VC 2.5, single network, iSCSI shared storage


  1. The IP address of the VirtualCenter server
  2. The ESX host of the VM becoming the new VC server
  3. The path to your VMware FlexLM license file (assuming you’re using a license server)
  4. The logon details for the SQL connection between vpxd and the database

Steps Taken

On the physical VirtualCenter box that is going to be decommissioned:

  1. Take note of the physical ESX host running the VM becoming the new VC server
  2. Stop the vpxd service and change the startup to manual (sc stop %service%, sc config %service% start= demand)
  3. Stop the vmountVpx, vmware-ufad-vci, vmware-converter and webAccess services change the startup to manual
  4. Stop the flexlm instance and set the startup to manual
  5. Take a backup of your .lic license file
  6. Change the IP address to dynamic (assuming it is static)
  7. Power off the physical machine
  8. Delete the computer account for vc01 from the domain
  9. On the SQL server hosting the VirtualCenter database, backup the VC database and log file with the following command executed through management studio:

    TO DISK = N'c:\temp\VC_PreMoveToVM.bak'
    DESCRIPTION = N'VirtualCenter Pre-move to VM backup'
    , INIT
    , NAME =
    N'VC Pre move to VM'

    N'c:\temp\VC_PreMoveToVM.bak '
    WITH FILE = 1
  10. In the service console of one of the ESX hosts, backup the current certificates (just in case):
    1. mkdir /tmp/cert_backup
    2. cp /etc/vmware/ssl/* /tmp/cert_backup
  11. On the VM becoming the new VirtualCenter server running on a host in the cluster: Rename the server from vc02 to vc01, with the same static IP as the previous vc01
  12. Restart the virtual machine
  13. Install the SQL Native Client - required for VC 2.5 SQL connectivity on a non-SQL server
  14. Install VirtualCenter including FlexLM, connecting to the existing database and using the license file copied off the physical server
  15. In VirtualCenter, disconnect the first ESX host in the cluster - maintenance mode is not possible at this stage – login errors occur because of the incorrect certificate
  16. Copy the new VC certificates - I use pscp here, but whatever you normally use to copy files to ESX would be fine:
    1. cd "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\ssl"
    2. pscp rui.crt vcadmin@esx01:/etc/vmware/ssl/
    3. pscp rui.key vcadmin@esx01:/etc/vmware/ssl/"
  17. In the service console of the ESX node you’ve just updated the certificates on, restart the management interface, the ESX host did not seem to pick up the new certificate dynamically (maybe it would on a schedule without a restart):
    1. service mgmt-vmware restart
  18. Connect the ESX host to VC through the VI Client interface
  19. Repeat steps 15-18 for each other ESX host in the cluster
  20. In VirtualCenter, run 'Reconfigure for HA' on each ESX node


  21. Ensure the vpxd.log file reports no problems with host connectivity or certificates (also check the VC/ESX logs)
  22. Ensure each ESX host is receiving licenses from the 'new' license server, either through the VI client or the FlexLM admin tool.
  23. Ensure you can perform simple tasks such as powering on a virtual machine
  24. Ensure VMotion/HA/DRS is working

The (untested) rollback plan if something goes wrong:

  1. Shutdown the new VirtualCenter VM
  2. Restore the database from the backup taken
  3. Power on the physical VC box, change the IP to the static IP
  4. Restart the vpxd and VMware License Server services
  5. For each ESX host in the cluster, disconnect the host, restore the old certificates, restart the management service and connect the ESX host to the old VirtualCenter instance

Additional notes:

  • Even though user login errors were returned when the vpxd service tried to form the cluster - which points to the vpxuser account used by VC to manage ESX hosts - this was misleading as this username and password is stored in the VC database – which had not been modified (in the vpx_host table). The next logical step was certificates, which lead to certificate update process used above.
  • Manually copying the certificates may not be strictly required, as when I went part-way to reconnecting a host without updating the certificates, I was prompted that another VC instance was managing these servers, would I like to continue. Presumably it would have automatically updated the certificates as required.

No comments:

All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 20 years, and I know just about enough to realise that I don’t know very much.