vSphere Storage vMotion traffic

This post provides information on storage vMotion in VMware vSphere 4 with ESXi 4.0 U2 hosts. While using storage vMotion to move data between unconnected storage, I conducted some simple testing to identify the networks and ports used for vMotion.

I’ve never seen an explanation of how storage vMotion works, and while it seems kind of obvious in retrospect, I thought some people might find this interesting.


Summary

1. The scope of a storage vMotion operation is datacenter wide, not limited to clusters or which host can see which storage.
2. In the testing conducted, storage vMotion relies upon the management network to transfer the disk data between the two ESXi hosts
3. If this method was used to import a large number of VMs from foreign storage then the limitation would typically be the 1GB vmnic used for the management network.
4. To allow the migration between datastores not visible by a single host, the VM would also be migrated between hosts.

Scenario

Two hosts managed by single vCenter instance, vc01:

1. esx01 part of the CLUS1 cluster
2. esx02 not in a cluster
3. esx01, connected only to the iSCSI SAN 1
4. esx02, connected only to the iSCSI SAN 2

Management network on vmk0 using two GbE interfaces on both hosts. Vmk0 on the management network with load balancing based on the virtual port ID.

Test

The test conducted was using storage vmotion to move a virtual machine through VirtualCenter on vc01 from a datastore visible only to esx02 to a datastore visible only to esx01.

1. The source datastore was LUN02 (visible only to esx02) and the destination datastore was LUN01 (visible only to esx01).
2. vMotion was used to automatically migrate the VM from esx02 to esx01 (as the VM had moved storage)

Traffic and network usage observed

1. On esx02 unsupported console, vmk0 source 192.168.0.12.60504 (esx02), destination 192.168.0.11.902 (esx01).
2. Command: /bin/tcpdump-uw -I vmk0 -nn

esx02

1. MbRx 190Mb/sec on software iSCSI NIC – Reading the disk information from iSCSI SAN 2 for the VM
2. MbTx 136Mb/sec – Transmit the information across the management network to esx01.
3. Command: resxtop --server esx02 (‘n’ for network information)

esx01

1. MbRx 120Mb/sec – receiving the disk data from esx02 across the management network
2. MbTx 200Mb/sec on software iSCSI NIC – writing the data to the SAN across the iSCSI network
3. Command: resxtop --server esx01 (‘n’ for network information)

The following diagram shows the traffic flow of storage vMotion on an iSCSI network:

Read more!

Resizing a VM's Windows system OS with Set-HardDisk

With vSphere PowerCLI Update 1 there is now an automated method of increasing the system disk of a virtual machine. This essentially performs the same thing as I’ve been doing manually for a while now - the disk is resized and mounted to a ‘helper vm’ and diskpart extends the disk, the helper VM is then shutdown and the risk removed, and when the resized VM starts up the OS prompts for a final reboot after detecting the disk change.

Note that both the target and helper VM must be powered off, and credentials must be passed or the helpervm argument is ignored. The guest credentials are used on the Windows helper VM to authenticate using the VMtools mechanism to invoke a command inside a VM.

$guetsusername = "domain\username"
$guestpass = Read-Host "Guest password (administrator of the server)?" -assecurestring
$hostusername = "root"
$hostpass = Read-Host "Host Password (VI SDK access to the ESX host)?" -assecurestring
$hostcredential = new-object System.Management.Automation.PSCredential($hostusername, $hostpass)
$guestcredential = new-object System.Management.Automation.PSCredential($guetsusername, $guestpass)
$sizeGB = 30
$sizeKB = $sizeGB *1024*1024
$harddisk = Get-HardDisk -vm vm01
Set-HardDisk -HardDisk $harddisk[0] -capacityKB $sizeKB -hostcredential $hostcredential -guestcredential $guestcredential -helpervm $vm

An error occurred when trying to use domain credentials to pass to the vCenter SDK – host credentials are required, not credentials to vCenter (which seems odd).

Set-HardDisk -HardDisk $harddisk[0] -capacityKB 31457280 -hostcredential $hostcredential -guestcredential $guestcredential -helpervm helpervm
Set-HardDisk : 4/02/2010 1:25:04 PM    Set-HardDisk    919835D1-AD13-41DF-ABDE-D630E96FE722    While performing operation 'Connect to host service 'https://esx01/sdk' at port 902' the following error occured: 'Insufficient permissions in host operating system'
At line:1 char:13
+ Set-HardDisk  <<<< -HardDisk $harddisk[0] -capacityKB 31457280 -hostcredential $hostcredential -guestcredential $guestcredential -helpervm helpervm

Another error occurred when trying to resize a VM, the script used failed.  Note that I tried this manually and it also failed (must have been something particular to that VM).  It was interesting to see that from this it looks like the cmdlet uses the invoke-vmscript to run a remote script through the VMtools interface that calls the diskpart commands.
Set-HardDisk : 4/02/2010 11:22:11 AM    Set-HardDisk    919835D1-AD13-41DF-ABDE-D630E96FE722    Execution of the  script in guest OS on VM 'helpervm' failed: '
Microsoft DiskPart version 5.2.3790.3959
Copyright (C) 1999-2001 Microsoft Corporation.
On computer: HELPERVM
Disk 1 is now the selected disk.
Volume 2 is the selected volume.
The volume you have selected may not be extended.
Please select another volume and try again.
'
At line:1 char:13
+ Set-HardDisk  <<<< -HardDisk $harddisk[0] -capacityKB 31457280 -hostcredential $hostcredential -guestcredential $guestcredential -helpervm $vm

Wayne's World of IT (WWoIT), Copyright 2010 Wayne Martin. 

Read more!

PowerCLI: Copy-VMGuestFile

In VMware PowerCLI Update 1, a new cmdlet has been added to copy files to/from guest OS’s using the VMTools. Under normal circumstances this wouldn’t be all that useful, but if a machine is not accessible on the network – either a DMZ or a test-lab environment for example – there is now a method to easily copy files to and from the guest machines.

For example:

copy-vmguestfile -source c:\windows\system32\imageres.dll -destination c:\temp\ -vm vm01 -guesttolocal -hostuser root -hostpassword * -guestuser administrator -guestpassword *

Note that you need to specify the passwords, or not specify credentials at all for a prompt, or you can use a pscredential object.

Unfortunately wildcards are currently unsupported, but it would be easy to loop through local files within PowerShell to upload based on wildcards.

I ran this with the measure-command cmdlet to see how fast it was – about 0.5MB/sec– not too quick, but a lot better than nothing.

measure-command -expression {copy-vmguestfile -source c:\windows\system32\imageres.dll -destination c:\temp\ -vm vm01 -guesttolocal -hostuser root -hostpassword * -guestuser administrator -guestpassword *}

Seconds : 26
Milliseconds : 539
Ticks : 265390633
TotalDays : 0.000307165084490741
TotalHours : 0.00737196202777778
TotalMinutes : 0.442317721666667
TotalSeconds : 26.5390633
TotalMilliseconds : 26539.0633

Wayne's World of IT (WWoIT), Copyright 2010 Wayne Martin. 

Read more!

vCenter file copy with virtual floppy disk

n an ESXi or vCenter environment, without network connectivity to a virtual machine from your administrative workstation it's quite hard to bi-directionally transfer files.

This post discusses a simple method I use for small quick file transfer from a secure virtual machine to my administrative workstation, using virtual floppy drives managed from the command-line.

This process uses vfd.exe, but presumably any virtual floppy software on your workstation could be used. The virtual machine also needs to have a floppy drive...

To bi-directionally transfer files from a non-network connected VM to my administrative workstation:

1. Create a virtual floppy disk on my workstation


2. Mount that floppy disk file as a virtual floppy drive on my workstation as A:


3. Through VI client, connect the VM to the A: drive


4. On the VM, copy files to/from A: drive and disconnect from the floppy


5. On my workstation, copy files to/from the virtual A:


6. Close the virtual floppy

Using vfd.exe:

1. vfd install


2. vfd open c:\temp\new.flp


3. Use VI Client to connect to A:


4. Copy files, then disconnect the VM virtual device


5. copy a:\*.*


6. vfd close

The obvious limitation of this is that the size is limited to 1.44MB. I tried with a 2.44MB floppy and the virtual machine didn't recognise this disk.

This doesn’t work with a virtual CD-ROM, as it’s mounted as read-only in the VM, so bi-directional copies aren’t available. You can still use a similar process for creating an ISO to copy files into a VM though. For this I use oscdimg.exe (Microsoft utility) to create my ISO from the command-line.

Read more!

VMware PowerCLI commands

The VMware PowerCLI PowerShell interface provided for managing vSphere systems is a fantastic tool that should be useful for all VMware admins.

I've gathered these commands while implementing and managing ESXi 4.0 clusters, use with caution on any production system.

This is an extension of a previous post:
VMware vSphere PowerCLI commands

Join a cluster by moving an ESX host from one location to the cluster
Move-Inventory -Item (Get-VMHost -Name esxHost) -Destination (Get-Cluster -Name clusterName)

Get the VMware.Vim.ClusterComputeResource MO from the PowerCLI cluster object
$clusterview = get-view $cluster.Id

Reconfigure a host for VMware HA (high availability)
$vmhost = get-vmhost -name esxHost; $hostMO = Get-View -ID $vmhost.ID; $hostMO.ReconfigureHostForDAS()

Find migration events for the last day
$events = Get-VIEvent -Start (Get-Date).AddDays(-1) | where {$_.fullFormattedMessage -match "Migrating.*"}

Find events other than CPU Alarms or user login/logout for the last day
$events = Get-VIEvent -Start (Get-Date).AddDays(-1) | where {$_.fullFormattedMessage -notmatch "Alarm.*CPU.*|User.*logged.*"}

Find events for degraded MPIO path redundancy 
$events = Get-VIEvent -Start (Get-Date).AddDays(-1) | where {$_.fullFormattedMessage -match "Path redundancy to storage.*degraded"}

Report the date, host and description for MPIO path redundancy errors
foreach ($event in $events) {write-output ($event.createdTime.ToString() + "," + $event.host.get_name() + "," + $event.fullFormattedMessage)}

List a table of VI events with only the date and message
$events | format-table -wrap -autosize -property createdTime,fullFormattedMessage

List the physical networks adapters and the current link speed (ESX 4.0)
$hostSystem = get-view -ViewType HostSystem; $hostConfigManager = $hostSystem.get_ConfigManager(); $hostNetworkSystem = $hostConfigManager.get_NetworkSystem(); $netSystem = Get-View $hostNetworkSystem; $netSystem.NetworkConfig.pnic; foreach ($pnic in  $netSystem.NetworkConfig.pnic) {Write-Output ($pnic.Device + "," + $pnic.spec.linkspeed.SpeedMB)}

List the vSwitches and the uplinks currently attached
$hostSystem = get-view -ViewType HostSystem; $hostConfigManager = $hostSystem.get_ConfigManager(); $hostNetworkSystem = $hostConfigManager.get_NetworkSystem(); $netSystem = Get-View $hostNetworkSystem; foreach ($vswitch in  $netSystem.NetworkConfig.vSwitch) {Write-Output ($vSwitch.Name + "," + $vswitch.spec.policy.NicTeaming.NicOrder.ActiveNic)}

Remove snapshots from a group of machines
$VMs = Get-VM -Location (get-folder -name "vmFolder"); foreach ($vm in $vms) {remove-snapshot -snapshot (Get-Snapshot -vm $vm) -confirm:$false}

Take snapshots of a group of machines
$VMs = Get-VM -Location (get-folder -name "vmFolder"); foreach ($vm in $VMs) {New-Snapshot -Name "snapshot 01" -description "Snapshot description" -vm $vm -Quiesce:$false}

Find VM name, description and primary disk datastore
$VMs = get-vm; foreach ($vm in $VMs) {write-output ($vm.Name + ",""" + $vm.Description + """," + $vm.harddisks[0].FileName.Replace(" ", ",")) | out-file -append -filepath c:\temp\VM_Datastores.txt}

Bring a host out of maintenance most
Set-VMHost -VMHost esxHost -State Connected

Generate diagnostic support bundles for all hosts
get-log -vmhost (get-vmhost) -bundle -destinationpath c:\temp\bundles

Find the network adapter type for each VM
$vms = get-vm ; foreach ($vm in $vms) {write-host $vm.Name "-"  $vm.networkadapters[0].type}

Find physical NICs and whether they're set to autonegotiate or hardcoded
foreach ($pnic in $hostNetwork.pnic) {if($pnic.linkSpeed -eq $null) {$ls = "Auto"} else {$ls= $pnic.linkSpeed.speedMB.toString() + ":" + $pnic.linkSpeed.duplex} ;write-output ($pnic.Device + "," + $ls)}

Find host sytem build information
$hostSystems = get-view -ViewType HostSystem; foreach ($hostSystem in $hostSystems) {Write-Output ($hostSystem.Name + "," + $hostSystem.config.product.Fullname)}

Find VMs and whether the VMtools is configured to synchronising time 
$vmSet = Get-VM ; foreach ($vm in $vmSet) { $view = get-view $vm.ID ;$config = $view.config; $tools = $config.tools; Write-Output ($vm.Name + "," + $tools.SyncTimeWithHost) }

Revert to a snapshot
set-vm -vm vmName -snapshot (get-snapshot -vm vmName) -confirm:$false

Remove a virtual machine from inventory and delete from disk
remove-vm -DeleteFromDisk:$true -RunAsync:$true -vm vmName

Shutdown one or more Virtual Machine guests
shutdown-vmguest -vm $vms -confirm:$false

Start one or more Virtual Machine guests
start-vm -vm $vms -confirm:$false

Forcefully power off one or more Virtual Machines
stop-vm $vms -confirm:$false

Get a virtual switch from the specified ESX host
get-virtualswitch -name vSwitch1 -vmhost esxHost

Create a new port group on the specified vSwitch
New-VirtualPortGroup -Name "pgName" -VirtualSwitch $vs

Find ESX memory balloon averages for the last five days
get-stat -entity $hosts -start (Get-Date).AddDays(-5) -finish (Get-Date) -stat mem.vmmemctl.average

Export a list of VMs
$vms | select-object -prop Name | out-file -filepath c:\temp\vms.txt

Export a list of VM guest hostnames 
$vms = get-vm; foreach ($vm in $vms) { write-output $vm.guest.get_HostName()}

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

VMware Command-Line Interface commands

The Windows command-line interface provided for managing ESX/ESXi systems is an invaluable tool for managing ESX infrastructure at the command-line. When using ESXi without a service console the CLI becomes even more useful.

I've gathered these commands while implementing and managing ESXi 4.0 clusters, use with caution on any production system.

ESXi 4.0 RCLI:

List the NTP servers used by the host
vicfg-ntp.pl --server esx01 --list

Add a Software iSCSI NIC named vmk2
esxcli --server=esx01 swiscsi nic add -n vmk2 -d vmhba33

List the Software iSCSI NICs
esxcli --server=esx01 swiscsi nic list -d vmhba33

List the software iSCSI status on a host
vicfg-iscsi.pl --server esx01 --swiscsi --list

Enable software iSCSI on an ESX host
vicfg-iscsi.pl --server esx01 --swiscsi --enable

List the adapters bound to software iSCSI
esxcli --server=esx01 swiscsi nic list -d vmhba33

List the VMKernel NICs
vicfg-vmknic.pl --server esx01 --list

List the software iSCSI adapters
vicfg-iscsi.pl --server esx01 --adapter --list

Set the iSCSI alias for the specified adapter
vicfg-iscsi.pl --server esx01 --iscsiname --alias esx01 vmhba33

Bind a VMK software iSCSI NIC for MPIO PSA
esxcli --server=esx01 swiscsi nic add -n vmk2 -d vmhba33

Rescan a storage adapter bus
vicfg-rescan.pl --server esx01 vmhba33

Add a dynamic iSCSI discovery target
vicfg-iscsi.pl --server esx01 --discovery --add --ip 10.1.1.10:3260 vmhba33

Add CHAP authentication to an iSCSI discovery target
vicfg-iscsi.pl --server esx01 --authentication --level chapRequired --method CHAP --auth_username esxchap --auth_password chapacc3ss80 --ip 10.2.128.33:3260 vmhba33

Find the current ScratchConfig scratch location
vicfg-advcfg.pl --server esx01 -g ScratchConfig.ConfiguredScratchLocation

Set the scratch location for ESXi
vicfg-advcfg.pl --server esx01 -s "/vmfs/volumes/esxds01/Scratch/esx01" ScratchConfig.ConfiguredScratchLocation

Check if CIM OEM providers are enabled (such as Dell OM)
vicfg-advcfg.pl --server esx01 -g UserVars.CIMOEMProvidersEnabled

Enable CIM OEM Providers (such as Dell OM)
vicfg-advcfg.pl --server esx01 -s "1" UserVars.CIMOEMProvidersEnabled

Query the patches/updates/bulletins/VIBs installed on ESXi
vihostupdate.pl --server esx01 -q

Set the SNMP community for the ESXi host
vicfg-snmp.pl --server esx01 -c public

Enable the SNMP agent on an ESXi host
vicfg-snmp.pl --server esx01 -E

List the iSCSI node name
vicfg-iscsi.pl --server esx01 --list --iscsiname --adapter vmhba36

List the preferred nativte multipathing (NMP) path for a device
 esxcli --server esx01 nmp fixed getpreferred -d naa.6090332880cfdc44fda634b1ca2457b8

Check whether round robin path selection is used for a device
esxcli --server esx01 nmp roundrobin getconfig -d naa.6090a02833cfdc7ffd4434b1ca5457b8 

List the disk NAA/UUIDs known to a host
esxcli --server esx01 nmp device list

List the MPIO path to device mapping
esxcfg-mpath.pl --server esx01 -m

List the SCSI devices known to a host
vicfg-scsidevs.pl --server esx01 --list   

List the available datastores on a host
vifs.pl --server esx01 --listds

List the contents of a datastore
 vifs.pl --server esx01 --dir [datastore]

Upload a local file to a datastore through vifs
vifs.pl --server esx01 --put c:\temp\file.txt dir/file.txt?dsName=datastoreName

List virtual switches, port groups, uplinks and MTU
vicfg-vswitch.pl -l --server esx01

Browse the datastores or local host through ssl
https://esx01/folder or https://esx01/host

Find the vmnic configuration, including driver, current speed
vicfg-nics.pl --server esx01 -l 

Set a vNIC to auto-negotiate
vicfg-nics.pl --server esx01 --vihost esx01 -a vmnic0

List the host-based files on an ESXi client
vifs.pl --server esx01 --dir /host  

A few other useful VMware tips:

Recreate the rui.pfx file for VirtualCenter
openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx

Forcefully power off a suspended VM
Delete the vmss file, and then power on the VM (state is lost)

ESXi 4.0, access unsupported ssh console through the dcui
Press Alt+F1, type unsupported, then the root password

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

VMware PowerCLI - Reconfigure cluster options

The following script contains a simple example of how to modify the configuration of a cluster – in this case to add two advanced options to the HA settings. This uses the new ClusterConfigSpecEx object, which can be used with the more generic function to reconfigure a resource – ReconfigureComputeResource().

The options set in this example are the das.allowNetworkX options, configuring HA to set specific management networks/service consoles that are used for HA communication and heartbeats.

Consider the following scenario for a reason why you might want to use these options:

1. Software iSCSI with ESX 3.x – using the service console for authentication, meaning you need a service console on your iSCSI network (or routes between your corporate network and iSCSI). The standard result is that HA could be used across this interface.
2. You are progressing with an upgrade to ESXi 4.0, still using software iSCSI, which is improved to remove the need for a management interface on your iSCSI network
3. You put an ESX 3.5 and ESXi 4.0 host in the same cluster, HA no longer works because your iSCSI service console on ESX 3.5 can’t talk to your ESXi 4.0 host.
4. Adding the two options above (with names changed appropriately) would force HA to use the service console on ESX 3.5 and the equivalent management network on ESXi 4.0, ignoring the iSCSI service console on 3.5 - resolving HA issues.

# http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.ComputeResource.html#reconfigureEx
# http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.cluster.DasConfigInfo.html#field_detail

param (
    $vcServerName = ""
)

$viServer = Connect-VIServer -server $vcServerName

$optionValue = New-Object Vmware.Vim.OptionValue
$optionValue.Key = "das.allowNetwork0"
$optionValue.Value = "Service Console"

$optionValue2 = New-Object Vmware.Vim.OptionValue
$optionValue2.Key = "das.allowNetwork1"
$optionValue2.Value = "Management Network"

[Vmware.Vim.OptionValue[]]$optionValues = $optionvalue, $optionValue2  # Create the array of the two new option values

$cluster = get-cluster -Name 'Cluster01'     # Get the cluster
$clusterview = get-view $cluster.Id      # Get the SDK object from the PowerCli object MO

$spec = New-Object Vmware.Vim.ClusterConfigSpecEx
$spec.dasConfig = New-Object Vmware.Vim.ClusterDasConfigInfo   # New VMware HA config

$spec.dasConfig.option = $optionValues      # Add the array of optionValues
$clusterview.ReconfigureComputeResource($spec, $true)    # Modify the configuration. When configuring clusters, can be a ClusterConfigSpecEx object


# Check the settings were saved

$cluster = get-cluster -Name 'Cluster01'     # Get the cluster
$clusterview = get-view $cluster.Id      # Get the SDK object from the PowerCli object MO

$clusterview.Configuration.dasConfig.Option | format-list   # Retrieve the advanced options

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

VMware PowerCLI - Backup ESXi 4.0 firmware

The following script contains a simple function to use the VMware vSphere PowerCLI to connect to a vCenter and/or ESXi 4.0 host and backup the 'firmware' - the configuration of the ESX host. I don’t think this functionality is exposed directly through the PowerCLI, so a call to the very useful Get-View function is used to get the VI SDK object to call the appropriate method.

From what I can tell all this command does is generate a config dump on the server and return the HTTP URL to access the file download, which I’m then using the .Net web client object to download the file and store as the filename returned with a unique date suffix.

This certainly isn’t original, all I did was look at the vicfg-backup.pl perl RCLI script and (badly) translate to PowerCLI from there.

#
# Description:
#  Backup the firmware configuration on an ESXi 4.0 host
#
# Limitations:
#  -
#
# Assumptions, this script works on the assumption that:
#  The caller provides credentials with permissions to connect to the specified host
#
# Arguments:
#  esxServer, the ESX host to connect to, eg. esx01
#  vcServerName, The vCenter Server to connect to, eg vc01
#  outputDir, The directory to write the backup file to, defaults to %temp%
#  username, The username to use when connecting to the vCenter server (if specified), defaults to %username%
#  password, The password to use for the connection to vCenter, secure string prompt by default
#
# Usage:
#  PowerShell . .\BackupFirmware.ps1 -esxServer 'esx01'
#  PowerShell . .\BackupFirmware.ps1 -esxServer 'esx01' -vcServer vc01 -u domain\username
#
# References:
#  http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.host.FirmwareSystem.html
#
# Changes:
#  04/07/2009, Wayne Martin, initial version

#$ErrorActionPreference = "Continue"


param (
    $esxServer = "",
    $vcServerName = "",

    $outputDir = $env:Temp,

    $username = $env:username,
    $password = ""
)

if ($password -eq "" -and !($pass)) {
    write-output "No password specified from the command-line"
    $pass = Read-Host "Password?" -assecurestring
    $credential = new-object System.Management.Automation.PSCredential($username,$pass)    
}

if ($esxServer -eq "" -OR ($vcServerName -eq "" -and $esxServer -eq "")) {
    Write-Output "Please specify either a standalone host, or a host and a cluster"
    Exit 2
}

$viServer = $null


function BackupConfiguration([string] $vcServerName, [string] $esxServer, [string] $outputDir){
    $hostSystem = get-view -ViewType HostSystem -Filter @{"Name" = $esxServer}    # Find the .Net view of the specified host

    $hostConfigManager = $hostSystem.get_ConfigManager()      # Get the config manager
    $hostfirmwareSystem = $hostConfigManager.get_firmwareSystem()     # Find the MOR of the host firmware system

    $hostfirmware = Get-View $hostfirmwareSystem        # Get the VMware.Vim.HostFirmwareSystem object from the MOR

    $backupDownload = $hostfirmware.BackupFirmwareConfiguration()     # Call the backup method to generate the config bundle

    $backupDownload = $backupDownload.Replace("*", $esxServer)      # Replace '*' with the server name

    Write-Output "Backup saved to $backupDownload on the ESX host"
    $fileName = $backupDownload.SubString($backupDownload.LastIndexOf("/")+1)    # Extract the filename to reuse
    $fileType = $fileName.SubString($fileName.LastIndexOf("."))      # Find the extension (.tgz in this case)

    $Now = [DateTime]::Now.ToString("yyyyMMddTHHmmss")          # Unique identifier for the filename
    $file = $fileName.SubString(0, $fileName.Length - $fileType.Length)     # File name without extension
    $outputFile = $outputDir + "\" + $file + "_" + $Now + $fileType     # Construct the full filename path\bundle_date.tgz

    $wc = new-object system.net.WebClient        # use the .Net web client
    $wc.DownloadFile($backupDownload, $outputFile)       # Download the file from the URL returned

    if (test-path -path $outputFile) {         # Does the output file exist?
        Write-Output "$outputFile downloaded"
    } else {
        Write-Output "Error: $outputFile was not downloaded from $backupDownload"
    }

}

if ($vcServerName -ne "") {
    $viServer = Connect-VIServer -server $vcServerName -Credential $credential
} elseif ($esxServer -ne "") {
    $esxServer = Connect-VIServer -server $esxServer       # connect to VC
}


$results = ""
$results = BackupConfiguration $vcServerName $esxServer $outputDir

$results

if ($vcServerName -ne "") {
    Disconnect-VIServer -confirm:$false
}

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

VMware vSphere PowerCLI commands

The commands below are PowerCLI commands used to automate VMware ESX and VirtualCenter at the command prompt. Most of these commands were built on 3.5 with vSphere PowerCLI, and the majority have been tested against ESXi 4.0 and vCenter 4.0 infrastructure.

Each command-line can be copied and pasted at a PowerCLI command prompt, and most commands assume you already have a connection to the target, be it vCenter or ESX.

Note that most of these commands use OBN - Object By Name - instead of using the get* command to get the object. This is supported with all but a few of the commands I've come across.

List the vSphere PowerCLI commands
Get-VICommand

Connect to a ESX or VirtualCenter instance
connect-viserver -server %server%

List the currently available datastores
Get-Datastore | sort

List the currently available datastores filtered and sorted
Get-Datastore | where {$_.Name -like '*pr*'} | sort

Find the VMs attached to one or more datastores
foreach ($prodDatastore in $prodDatastores) { write-output $prodDatastore.Name; get-vm -datastore $proddatastore; write-output ''}

Get a Virtual Machine
$vm = get-vm -name '%vm%'

Get the virtual harddisk for the specified VMs
Get-HardDisk -vm $vm

Move a virtual machine to another container
Move-VM -Destination $prodApps -VM $vm

Update the VM description for a list of CSV entries
foreach ($virtualServer in $virtualservers) {$arr = $virtualServer.split(","); $desc = $arr[1]; $vmName = $arr[0]; write-output $vmName; $desc; $vm = get-vm -name $vmName; Set-VM -VM $vm -description $desc}

Query for a list of VMs and output in ANSI format
get-vm | sort-object | format-table -property Name | out-file -encoding ASCII -filepath c:\temp\vms_20090625.txt

Find VMware machine performance statistics
 get-stat -entity $vm -disk -start 01/01/2009 -finish ([DateTime]::Now.ToString("dd/MM/yyyy"))

For a group of VMs, report performance statistics and save to file
foreach ($vm in $devVMs) {get-stat -entity $vm -disk -start 01/01/2009 -finish ([DateTime]::Now.ToString("dd/MM/yyyy")) | out-file -filepath ("c:\temp\" + $vm.Name + "DiskPerformance.txt")}

Find VM datastore disk usage
$devVMs = get-vm -name '*dv*'; foreach ($vm in $devvms) {$vm.harddisks}

Find VM datastore disk usage
$testVMs = Get-VM -Location (get-folder -name "Test") ;foreach ($vm in $testVMs) {$vm.harddisks | format-table -hideTableHeaders -wrap -autosize | findstr /i /c:per}

Find SCSI devices attached to an ESX server
get-scsilun -vmhost (Get-VMHost -Location "cluster")[0]

Rescan HBAs on an ESX server
get-VMHostStorage -VMHost (Get-VMHost -Location "cluster")[0] -RescanAllHba

Storage vMotion a virtual machine to a new datastore
Move-VM -vm "vmName" -datastore "NewDatastore"

Storage vMotion a group of machines from a CSV input file
$servers = get-content -path inputfile.txt; foreach ($server in $servers) {move-vm -vm $server.split(",")[0] -datastore $server.split(",")[1]}

Remove a snapshot and child snapshots, reporting how long the operation took
measure-command -expression {remove-snapshot -snapshot $snapshots[0] -removechildren}

Find datastore space, usage and number of VMs per datastore
$datastores = get-datastore | sort-object; write-output "Name,Size,Used,Free,% Used,#VMs"; foreach ($datastore in $datastores) { write-output ($datastore.Name + "," + [math]::round($datastore.CapacityMB/1024) + "," + [math]::round(($datastore.CapacityMB/1024)-($datastore.FreeSpaceMB/1024)) + "," + [math]::round($datastore.FreeSpaceMB/1024) + "," + [math]::round(((($datastore.CapacityMB/1024) - ($datastore.FreeSpaceMB/1024)) / ($datastore.CapacityMB/1024)) * 100) + "," + (get-vm -datastore $datastore).count)}

From a set of VMs, find which have snapshots
foreach ($testvm in $testvms) {if (get-snapshot -vm $testvm){write-output $testvm.Name}}

Find the size of the first hard disk in each VM
foreach ($vm in $vms) {$vm.harddisks[0] | format-table -hideTableHeaders -wrap -autosize | findstr /i /c:per }

Find disk information for VMs in the specified datastore
 $VMs = Get-VM ;foreach ($vm in $VMs) {$vm.harddisks | where {$_.FileName -like '*clusterpr*'} | format-table -hideTableHeaders -wrap -autosize | findstr /i /c:per}

Find VMs in the specified datastore
$VMs = Get-VM | where {$_.harddisks[0].FileName -like '*clusterpr*'}

Get VM guest information, including virtual OS
get-vm | get-vmguest | format-table -wrap -autosize

Find virtual machines and their description/notes
$vms = get-vm ; $vms | format-table -wrap -autosize -property Name,Description

Create an associative array containing VM names and descriptions
$vmdesc = @{}; foreach ($vm in $vms) {$vmdesc.add($vm.Name, $vm.Description)}

Migrate a virtual machine to another host in a VMware ESX cluster
move-vm -vm %vmName% -destination %hostname%

Find the host a VM is currently located on
get-vmhost -vm %vnName%

Add a new harddisk to a virtual machine
New-HardDisk -vm %vmName% -CapacityKB 20971520

Retrieve details on the resource pools from the currently connected datacenter
Get-ResourcePool | format-table -wrap -autosize -property Name,Id,CpuExpandableReservation,CpuLimitMHz,CpuReservationMHz,CpuSharesLevel,CustomFields,MemExpandableReservation,MemLimitMB,MemReservationMB,MemSharesLevel,Name,NumCpuShares,NumMemShares

Find virtual machines and if they have a CD-ROM
get-vm | format-table -wrap -autosize -property Name,CDDrives

Find the last 100 events that aren't alarm related
$events = Get-VIEvent -MaxSamples 100 | where {$_.fullFormattedMessage -notmatch "Alarm*"}

Find all events for machine deployments from templates
$events = Get-VIEvent | where {$_.fullFormattedMessage -match "Deploying (.*) on host (.*) in (.*) from template (.*)"}

Create a resource pool with high CPU and memory shares
New-ResourcePool -location (get-cluster -name 'cluster') -Name ResPool1 -CpuSharesLevel [VMware.VimAutomation.Types.SharesLevel]::High -MemSharesLevel [VMware.VimAutomation.Types.SharesLevel]::High

Create a folder from the root of the tree
New-Folder -Name Workstations -location (get-folder -name 'vm')

Move one or more VMs to a resource pool (or other destination)
$vms = get-vm -name vmNames*; move-vm -vm $vms -destination (Get-ResourcePool -name 'ResPool1')

Get an OS customization specification, and list the properties in wide format
Get-OSCustomizationSpec -name "SpecName" | format-list

Take a snapshot of a virtual machine
New-Snapshot -Name "Snapshot 01" -description "Snapshot description" -vm vmName -Quiesce:$true

Convert a virtual machine to a template
$vmView = get-vm -name vm01 | Get-View; $vmView.MarkAsTemplate()

Find Datastore usage (custom written function)
get-datastoreusage

Get an ESX log bundle using PowerCLI
Get-Log -VMHost esxhostname -Bundle -DestinationPath c:\temp

Query for snapshots
Get-VM | Get-Snapshot | export-csv -path c:\temp\VMsnapshots.csv

Query for snapshot information
Get-VM | Get-Snapshot | foreach-object {$out=  $_.VM.Name + "," + $_.Name + "," + $_.Description + "," + $_.PowerState; $out}

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

PowerCli Relocate VM storage with VI35 / VI4

This post provides a script to help with storage vMotion – migrating a VMware virtual machine from one VMFS datastore to another with no downtime. It’s a great tool, but in VI35 at least it’s not exposed through the GUI or VC scheduled tasks, so I’ve written a script to perform a few checks and allow for very simplistic 'scheduling' (delay).

I’ve only tested this with VI3.5 and PowerCLI 1.0, but I have no reason to think this wouldn’t with with vSphere vCenter and VI 4.0 hosts.

There are a few caveats with storage vMotion (in VC 2.5/ESX 3.5) at least, you can’t:
- Move the storage of a VM that has a snapshot if the VM is powered on
- Move the storage of a VM that has a snapshot (in any power state) if the VM has disks in a different location than the config file.

Based on these caveats, if instructed the script will suspend a VM with snapshots in order to move the storage, then power the VM back on. Use this with caution, as this may cause an outage of your VMs.

#
# Description:
#   Relocate a virtual machine from one datastore to another.
#
# Limitations:
#  -
#
# Assumptions, this script works on the assumption that:
#   The caller provides credentials with permissions to perform the operations
#
# Arguments:
#  vmName, lowecase short name for the virtual machine, eg pibutepr03
#  dataStoreName, the new datastore to migrate the VM to
#  suspend, Whether or not to suspend a VM and move if the VM has snapshots (which prevent live storage vMotion vi VI35)
#  username, The username to connect with, default to the current username environment variable
#  password, The Password to use for the connection, not specifying a password will result in a prompt to enter a secure string
#  delay, The optional number of seconds to delay before starting the operation
#
#
# Usage:
#   PowerShell . .\RelocateVM.ps1 -vmName "vm01" -datastore 'ds02'
#   PowerShell . .\RelocateVM.ps1 -vmName "vm01" -datastore 'ds02' -suspend yes -username domain\user
#
# Changes:
#  07/04/2009, Wayne Martin, Initial version
#

param (

    $vmName = "",
    $dataStoreName = "",
    $suspendIfRequired = $false,
    $username = $env:username,
    $password = "",
    $delay = 0
)


$ErrorActionPreference = "Continue"

 $invalidArgs = $false
if ($dataStoreName -eq "") { write-output "Please specify the datastore target for the VM, eg. ds02"; $invalidArgs = $true}
if ($vmName -eq "") { write-output "Please specify the virtual machine to move, eg vm01"; $invalidArgs = $true}
if ($account -eq "") { write-output "Please specify a user account to connect with, eg domain\user"; $invalidArgs = $true}

if ($invalidArgs) { write-output "Invalid Arguments, terminating"; exit}


Write-Output "Moving VM '$vmName' to the '$dataStoreName' datastore"

if ($delay -gt 0) {
    $hours = $delay / 60 /60
    Write-Output "Delaying $delay seconds before beginning ($hours hours)"
    Sleep -seconds $delay
}


if ($suspendIfRequired) {
    Write-Output "The virtual machine will be suspended if snapshots are preventing storage vMotion"
} else {
    Write-Output "If the virtual machine has snapshots and is powered on the storage vMotion will not work"
}

if ($password -eq "" -and !($pass)) {
    write-output "No password specified from the command-line"
    $pass = Read-Host "Password?" -assecurestring
}
$credential = new-object System.Management.Automation.PSCredential($username,$pass)    


$viServer = $null
$suspend = $false
$poweredOn = ""
$snapshot = $null
$hasSnapshot = $null

$viServer = Connect-VIServer -server $vcServerName -Credential $credential


if ($viServer) {
    Write-Output (Get-Date -format "dd/MM/yyyy HH:mm:ss")
    write-output ("Connected to server " + $viServer.Name + " on port " + $viServer.Port)

    $vm = get-vm -name $vmName
    if ($vm -and $vm -isnot [object[]]) {
        Write-Output ("Found " + $vm.Name)

        $hardDisks = $vm.hardDisks
        $vmSize = 0
        $vmSizeMB = 0
        foreach ($harddisk in $vm.hardDisks) {
            $vmSize += $hardDisk.CapacityKB
        }
        $vmSizeMB = $vmSize /1024

        $datastore = get-datastore -name $dataStoreName
        if ($datastore) {
            $freeSpace = $datastore.FreeSpaceMB
        
            if ($freeSpace -gt $vmSizeMB)
            {
                Write-Output "The datastore $datastoreName has $freeSpace MB available, the VM has disks totalling $vmSizeMB MB"
            
                switch ($vm.PowerState)
                {
                    ([VMware.VimAutomation.Types.PowerState]::PoweredOn)
                    {
                        Write-Output "The virtual machine is currently powered on"
                        $poweredOn = $true
                    }

                    ([VMware.VimAutomation.Types.PowerState]::PoweredOff)
                    {
                        Write-Output "The virtual machine is currently powered off"
                    }

                    ([VMware.VimAutomation.Types.PowerState]::Suspended)
                    {
                        Write-Output "The virtual machine is currently suspended"
                    }
            
                    default
                    {
                        write-output "Virtual machine power state unknown"
                    }
                }

                [object[]]$snapshot = get-snapshot -vm $vm
                if ($snapshot) 
                {
                    $hasSnapshot = $true
                    $numSnapshots = $snapshot.Count
                    Write-Output "$vmName currently has $numSnapshots snapshot(s)"
                    if ($poweredOn) {
                        if ($suspendIfRequired) {
                            Write-Output "$vmName is powered on and has a snapshot, and will be suspended during the move"
                            $suspend = $true
                        } else {
                            Write-Output "Error: $vmName is powered on and has a snapshot, but will not be suspended, process aborted"
                            $suspend = $false
                            exit 2
                        }
                    }
                } else {
                    $suspend = $false
                    $hasSnapshot = False
                    Write-Output "No snapshots currently found for $vmName"

                }

                if ($suspend) {
                    Write-Output "Suspending $vmName"
                    Suspend-VM -vm $vm -confirm:$false
                }

                Write-Output "Moving $vmName to $datastoreName"
                Move-VM -vm $vmName -datastore $datastoreName

                if ($suspend) {
                    Write-Output "Bringing $vmName out of suspension"
                    Start-VM -vm $vm -confirm:$false
                }
                Write-Output "$vmName migrated to $datastoreName"

            } else {
                Write-Output "The datastore only has $freeSpace MB available, but the VM has disks totalling $vmSizeMB MB"
            }
        } else {
            Write-Output "Error: datastore $datastoreName not found"
        }
    } else {
        if ($vm -is [object[]])
        {
            Write-Output "Multiple objects returned for $vmname, please specify a single VM"
        } else {
            write-output "VM Not found - $vmName"
        }
    }
} else {
    write-output "ERROR: VI server not found - $viServer"
}

Write-Output (Get-Date -format "dd/MM/yyyy HH:mm:ss")

Disconnect-VIServer -confirm:$false
exit 0

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

PowerShell - vSphere VMKernel NIC MTU

I came across a problem with VI4 (vSphere) on ESXi, where I was unable to create a vmkernel NIC for iSCSI with the vicfg-vmknic.pl script to set a custom MTU of 9000. This post provides a method using the GUI/perl script, and a PowerShell vSphere PowerCLI script to modify an existing VMKernel NIC.

It seems there is a bug in the perl script doesn't allow you to create a VMKernel NIC, however it does work if you create one through the GUI, and then use the script to delete and re-create the virtual NIC.

This was using a standalone ESXi server with a normal vSwitch - it wasn't managed by VC and it wasn't a DVS.

The GUI/PL script method:

1. Create a VMKernel NIC in an existing vSwitch through the VI Client, with a port group called iSCSI1 in this example.
2. Using the CLI, run the following commands, the first to delete the existing NIC, and then another to re-create the same NIC, but with an increased MTU:
   vicfg-vmknic.pl -d iSCSI1
   vicfg-vmknic.pl -a -i x.x.x.x -n x.x.x.x -m 9000 iSCSI1
3. Run 'vicfg-vmknic.pl –l' to list the VMK NICs, which should show the adjusted MTU

I wasn't particularly happy with this method, so I looked at using the vSphere PowerCLI to set the MTU. For whatever reason, VMware chose not to expose the MTU property, which leaves only the SDK.

Luckily the PowerCLI makes it much easier to use the .Net SDK than previously possible with the Get-View cmdlet, so the script below is a combination of standard PowerCLI with a hook into the vNIC collection and the UpdateVirtualNic method to change a virtual NIC spec.

param (

    $MTU = 9000, 
    $nicName = vmk1, 
    $vcServer = "vcServer", 
    $vmServerName = "esx01"
)

#
#
# Description:
#  Update the MTU of a vmkernel NIC used for iSCSI
#
# Limitations:
#  DVS untested
#
# Assumptions, this script works on the assumption that:
#  The caller provides credentials with permissions to change the specified NIC
#
# Arguments:
#  MTU, The new MTU size, eg. 9000
#  nicName, The vmkernel NIC, eg. vmk1.  Note that this differs from the port group (eg. iSCSI1, with a NIC name of vmk1)
#  vcServer, the VC instance to connect to, eg. vcServer
#  vmServerName, The server name controlled by the VC instance (or direct), eg. esx01
#
# Usage:
#  PowerShell . .\UpdateVMKMTU.ps1 -MTU 9000 -nicName 'vmk1' -vcServer 'vcServer' -vmServerName 'esx01'
#
# References:
#  http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.host.VirtualNic.Specification.html
#
# Changes:
#  26/06/2009, Wayne Martin, initial version

#$ErrorActionPreference = "Continue"


Connect-VIServer -server $vcServer        # connect to VC
$hostSystem = get-view -ViewType HostSystem -Filter @{"Name" = $vmServerName}   # Find the .Net view of the specified host
$hostConfigManager = $hostSystem.get_ConfigManager()      # Get the config manager
$hostNetworkSystem = $hostConfigManager.get_NetworkSystem()     # Find the MOR of the host network system
$netSystem = Get-View $hostNetworkSystem       # Get the object from the reference for the update method later

$hostconfig = $hostSystem.Config        # Get the current host config
$hostNetwork = $hostconfig.Network        # Get the current network host config
$hostvNIC = $hostNetwork.vNic         # Get the virtual NICs

$nicBeingUpdated = $null
foreach ($hostVirtualNIC in $hostvNIC) {       # For each virtual NIC
    if ($hostVirtualNIC.Device -eq $nicName) {        # Is this the device specified?
        $nicBeingUpdated = $hostVirtualNIC       # Yes, copy the object
    }
}

if ($nicBeingUpdated) {          # Was something found?
    $nicSpec = $nicBeingUpdated.Spec        # Yes, get the current spec
    $currentMTU = $nicSpec.MTU         # Get the current MTU from the spec

    if ($currentMTU -ne $MTU) {         # Is the MTU different?
        $nicSpec.set_Mtu($MTU)         # Yes, update the MTU on the copy of the spec of the existing NIC
        $netSystem.UpdateVirtualNic($nicName, $nicSpec)             # Call the update method from the netsystem, to update the NIC with the modified device spec

        write-output "MTU for $nicName updated from $currentMTU to $MTU"
        (get-vmhostnetwork).VirtualNic        # Output updated information
    } else {
        write-output "MTU for $nicName already set to $MTU, no changes made"
    }

} else {

    write-output "NIC $nicName not found, no changes made"

}

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

Calling CPUID from PowerShell for Intel VT

I was trying to work out whether the Intel VT (VMX) extensions had been enabled on a server and whether the server support the SSE4.1 and SSE4.2 processor instructions - without having to reboot the server to check the BIOS. I ended up 'writing' a PowerShell script that compiles some C# to call a DLL written in assembler which has run the CPUID instruction and returned the results. Note that I didn't really write any of this, the PowerShell script compiling and running the C# is from the reference below, as is the actual C$ and asm dll.

Does this seem like a lot of effort? Sure does, but unfortunately the simple method - using Win32_Processor.ProcessorID was useless in this example, as for some reason MS chose to only return the EAX and EDX registers, whereas the bits I was after were returned in ECX. There may also be other ways to achieve this through PowerShell/.Net framework, rather than compiling c# to call unmanaged code.

Below is the PowerShell script, which contains the inline C# that is compiled and executed, which in turn requires the dll, using the EFLAGS register to confirm support and the CPUID instruction with eax=1 (if further supported) to query feature information from the CPU.

The original c# and asm dll: (see below for the hex text stream)
http://devpinoy.org/blogs/cvega/archive/2006/04/07/2658.aspx

The compile-csharp function: (adding the -unsafe parameter to allow pointer use)
http://monadblog.blogspot.com/2005/12/calling-win32-api-functions-through.html

#
# Description:
#  Call the CPUID function from an external DLL through C# inline code.
#  Report whether Intel VT (VMX) is supported, bit 5 of the ECX register returned from CPUID eax=1 call.
#  Note that WMI Win32_Processor:ProcessID contains the results of a call to CPUID - except that it only contains EAX and EDX, Intel VT (what I was after) is in ECX
#
# Author:
#  Wayne Martin, note that I didn't write any of this code, the inline C# was from the monad reference below, and the CPUID DLL and C# was from the first reference

#References:
# cpuid.dll and the namespace from: http://devpinoy.org/blogs/cvega/archive/2006/04/07/2658.aspx
# Intel cpuid: http://www.intel.com/Assets/PDF/appnote/241618.pdf
# http://monadblog.blogspot.com/2005/12/calling-win32-api-functions-through.html
# http://msdn.microsoft.com/en-us/library/aa394373.aspx


function Compile-Csharp ([string] $code, $FrameworkVersion="v2.0.50727",
[Array]$References)
{
    $cp = new-object Microsoft.CSharp.CSharpCodeProvider
    $cpar = New-Object System.CodeDom.Compiler.CompilerParameters
    $cpar.CompilerOptions = "-unsafe"      # unsafe to compile the code which uses pointers in the DLL call
    $cpar.GenerateInMemory = $true
    $cpar.GenerateExecutable = $false
    $cpar.OutputAssembly = "custom"
    # $cpar.ReferencedAssemblies.AddRange($refs)
    $cr = $cp.CompileAssemblyFromSource($cpar, $code)

    if ( $cr.Errors.Count)
    {
        $codeLines = $code.Split("`n");
        foreach ($ce in $cr.Errors)
        {
            write-host "Error: $($codeLines[$($ce.Line - 1)])"
            write-host $ce
           # $ce out-default
        }
        Throw "INVALID DATA: Errors encountered while compiling code"
    }
}



$code = @'
namespace CPUID
{
 using System;
 using System.Runtime.InteropServices;
 using System.Text;

 public class cpuid
 {
  private cpuid()
  {
  }

  [DllImport("cpuid.dll")]
     public static extern bool CPUIDIsSupported();

  [DllImport("cpuid.dll")]
     private unsafe static extern bool __cpuid
            (uint function, 
             int* eax, 
             int* ebx, 
             int* ecx, 
             int* edx);

  // Invoke __cpuid function
  public unsafe static bool Invoke
      (uint level, 
       out int eax, 
       out int ebx, 
       out int ecx, 
       out int edx)
  {
   int __eax = 0;
   int __ebx = 0;
   int __ecx = 0;
   int __edx = 0;

   if (__cpuid(level, &__eax, &__ebx, &__ecx, &__edx))
   {
    eax = __eax;
    ebx = __ebx;
    ecx = __ecx;
    edx = __edx;

    return true;
   }
   else
   {
    eax = 0;
    ebx = 0;
    ecx = 0;
    edx = 0;

    return false;
   }
  }
 }
}


'@


compile-CSharp $code
$eax = ""
$ebx = ""
$ecx = ""
$edx = ""
[CPUID.CPUID]::Invoke(1, [ref]$eax, [ref]$ebx, [ref]$ecx, [ref]$edx) 

$INTEL_CPUID_VT_FLAG = 0x0020                                                           # bit 5 of ECX returned from CPUID EAX=1
$INTEL_CPUID_SSE41 = 0x80000                                                            # bit 19 of ECX returned from CPUID EAX=1
$INTEL_CPUID_SSE42 = 0x100000                                                           # bit 20 of ECX returned from CPUID EAX=1
$INTEL_CPUID_XD = 0x00100000

write-output ("Intel VT: " + ($ecx -band $INTEL_CPUID_VT_FLAG) )                        # Does this processor support Intel VT / VMX?
write-output ("SSE 4.1: " + ($ecx -band $INTEL_CPUID_SSE41) )                           # Does this processor support SSE 4.1?
write-output ("SSE 4.2: " + ($ecx -band $INTEL_CPUID_SSE42) )                           # Does this processor support SSE 4.2?
write-output ("eXecute Disable: " + ($edx -band $INTEL_CPUID_XD) )                      # Does this processor support XD?

Note that in a previous post, I have written two powershell scripts, one to convert binary to a hex string, and vice versa. The following stream is the cpuid.dll file, if you write it to cpuid.txt and run the following command it will create the dll:

Powershell . .\HexStringToBinary.ps1 cpuid.txt cpuid.dll

4d5a90000300000004000000ffff0000b800000000000000400000000000000000000000000000000000000000000000000000000000000000000000c00000000e1fba0e00b409cd21b8014ccd21546869732070726f6772616d2063616e6e6f742062652072756e20696e20444f53206d6f64652e0d0d0a240000000000000071d4f7db35b5998835b5998835b59988c9958b8834b59988bbaa8a8834b599885269636835b59988000000000000000000000000000000000000000000000000504500004c0103006a793a440000000000000000e0000e210b01050c00020000000400000000000000100000001000000020000000000010001000000002000004000000040000000400000000000000004000000004000070fa000002000000000010000010000000001000001000000000000010000000002000005f0000000000000000000000000000000000000000000000000000000000000000000000003000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002e746578740000006c000000001000000002000000040000000000000000000000000000200000602e726461746100005f000000002000000002000000060000000000000000000000000000400000402e72656c6f6300000c000000003000000002000000080000000000000000000000000000400000420000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000558becb801000000c9c20c005352b8000000009c588bc83500002000509d9c5b33c325000020007507b801000000eb0233c0519d5a5bc3558bec5357e8cbffffff48740433c0eb1e8b45080fa28b7d0c89078b7d14890f8b7d1889178b7d10891fb8010000005f5bc9c214000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a793a44000000003c2000000100000002000000020000002820000030200000382000000c1000003710000046200000572000000000010063707569642e646c6c0043505549444973537570706f72746564005f5f6370756964000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

Extending a server 2003 VM system/boot disk

Whether using basic or dynamic disks, Windows Server 2003 doesn’t provide a method to extend the size of the system or boot disk. In VI35, I have used the following process to extend the system/boot disk of 2003 server virtual machines. Note that this is for the system/boot disk only, data volumes can be grown and extended live.

This process requires the VM to be shutdown, then a small outage while the disk is extended, then a reboot once the OS is first started.

The process I used in VI 3.5 U4, with a 2003 SP2 virtual machine:

1. Turn the VM off - VM1 in this example
2. Use VC to increase the size of the boot/system disk of VM1 by xGB (6 in my test)
3. Use VC to attach the disk inside another running VM that can be shutdown, eg. VM2
4. On VM2, load diskmgmt.msc, rescan disks (or run diskpart rescan)
5. On VM2, start diskpart:
   
   1. List vol (get a list of volumes)
      
   2. Select vol x (where x is the number of the newly added disk)
      
   3. Extend
6. Shutdown VM2 and detach the disk (don’t delete it!)
7. Start VM1
8. On my test, after logon, a setupapi message ‘Windows has finished installing new devices. Do you want to restart your computer now?’.
9. Said yes and the server restarted, diskmgmt.msc shows correctly sized disk.
10. Ran chkdsk c: to verify no corruption

Step 8 has a matching event ID 271 from PlugPlayManager and setupapi.log entry:

The Plug and Play operation cannot be completed because a device driver is
preventing the device from stopping. The name of the device driver is listed as
the vetoing service name below.
Vetoed device:
STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00
Vetoing device:
STORAGE\Volume\1&30a96598&0&Signature6AF3AEFAOffset7E00Length53FD06C00
Vetoing service name: FileSystem\Ntfs
Veto type 6: PNP_VetoDevice
When Windows attempts to install, upgrade, remove, or reconfigure a device,
it queries the driver responsible for that device to confirm that the operation
can be performed. If any of these drivers denies permission (query-removal
veto), then the computer must be restarted in order to complete the operation.
User Action
Restart your computer.

Setupapi.log driver logging showing the first vetoed attempt at installation and then the shadow copy volume snapshot install:
[2009/05/06 08:45:46 404.3 Driver Install]
#-019 Searching for hardware ID(s): storage\volume
#-198 Command line processed: C:\WINDOWS\system32\services.exe
#I393 Modified INF cache "C:\WINDOWS\inf\INFCACHE.1".
#W383 "volume.PNF" migrate: PNF Language = 0409, Thread = 0c09.
#I022 Found "STORAGE\Volume" in C:\WINDOWS\inf\volume.inf; Device: "Generic volume"; Driver: "Generic volume"; Provider: "Microsoft"; Mfg: "Microsoft"; Section name: "volume_install".
#I023 Actual install section: [volume_install.NTx86]. Rank: 0x00000000. Driver date: 10/01/2002. Version: 5.2.3790.3959.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [volume_install] in "c:\windows\inf\volume.inf".
#I320 Class GUID of device remains: {71A27CDD-812A-11D0-BEC7-08002BE2092F}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [volume_install.NTx86.Interfaces] from "c:\windows\inf\volume.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00".
#W100 Query-removal during install of "STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00" was vetoed by "STORAGE\Volume\1&30a96598&0&Signature6AF3AEFAOffset7E00Length53FD06C00" (veto type 6: PNP_VetoDevice).
#W104 Device "STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00" required reboot: Query remove failed (install) CfgMgr32 returned: 0x17: CR_REMOVE_VETOED.
#I121 Device install of "STORAGE\VOLUME\1&30A96598&0&SIGNATURE6AF3AEFAOFFSET7E00LENGTH53FD06C00" finished successfully.
[2009/05/06 08:55:18 400.3 Driver Install]
#-019 Searching for hardware ID(s): storage\volumesnapshot
#-198 Command line processed: C:\WINDOWS\system32\services.exe
#W383 "volsnap.PNF" migrate: PNF Language = 0409, Thread = 0c09.
#I022 Found "STORAGE\VolumeSnapshot" in C:\WINDOWS\inf\volsnap.inf; Device: "Generic volume shadow copy"; Driver: "Generic volume shadow copy"; Provider: "Microsoft"; Mfg: "Microsoft"; Section name: "volume_snapshot_install".
#I023 Actual install section: [volume_snapshot_install.NTx86]. Rank: 0x00000000. Driver date: 10/01/2002. Version: 5.2.3790.3959.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [volume_snapshot_install] in "c:\windows\inf\volsnap.inf".
#I320 Class GUID of device remains: {533C5B84-EC70-11D2-9505-00C04F79DEAF}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "STORAGE\VOLUMESNAPSHOT\HARDDISKVOLUMESNAPSHOT1".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [volume_snapshot_install.NTx86.Interfaces] from "c:\windows\inf\volsnap.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "STORAGE\VOLUMESNAPSHOT\HARDDISKVOLUMESNAPSHOT1".
#I121 Device install of "STORAGE\VOLUMESNAPSHOT\HARDDISKVOLUMESNAPSHOT1" finished successfully.

References

How to extend a data volume in Windows Server 2003, in Windows XP, in Windows 2000, and in Windows Server 2008
http://support.microsoft.com/kb/325590

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin.

Read more!

Generating 100% CPU with calc or PowerShell

Occasionally when testing something I want to generate 100% CPU load on a Windows computer. There are several utilities out there to do this, but that implies you have the utility on hand and are comfortable running it on the server. A colleague of mine (thanks Mark S.) showed me this nifty trick of using calc.exe to generate 100% CPU. The best thing about this is that every standard Windows OS installation has calc.exe.

This post provides two methods of generating 100% CPU load, the original calc.exe method, and a simple one line PowerShell command to do the same thing from the command-line (locally or on a remote server with PS v1 and psexec). Note that there may be a better method with PowerShell, I simply scripted the same operation calc was performing.

On dual-CPU/core computers this uses 100% of one CPU/core. To use more than that, calc or the PowerShell command can be run more than once, and Windows by default will run the new process on another less-busy CPU/core.

One practical application of this is to load-test a VMware VI3 cluster, generating 100% CPU on one or more VMs to see how ESX and DRS/VC handles the load. I have also used this in the past when testing multi-threaded applications and processor affinity to see how Windows allocates a processor.

calc.exe

Use calc to calculate the factorial of a number - the product of all integers from 1 up to and including the number specified, eg 5! = 1x2x3x4x5

1. Run calc.exe and switch to scientific mode
2. Type a large number (eg. 12345678901234567890), press the 'n!' button.
3. Calc will ask to confirm after warning this will take a very long time
4. 100% CPU utilisation will now occur (essentially forever)
   

PowerShell
 

Using the largest int32 positive integer, calculate the factorial to generate 100% CPU utilisation
$result = 1; foreach ($number in 1..2147483647) {$result = $result * $number};

Depending on how fast the CPU is, this could finish, so a loop to run the command above 2 billion times:
foreach ($loopnumber in 1..2147483647) {$result=1;foreach ($number in 1..2147483647) {$result = $result * $number};$result}

If you want to see how long the command takes to run:
Measure-Command {$result = 1; foreach ($number in 1..2147483647) {$result = $result * $number}}

Using the command-line then provides the ability to run the command remotely. To use psexec to remotely execute powershell v1 factorial to generate 100% CPU:
psexec \\%computername% /s cmd /c "echo. | powershell $result = 1; foreach ($number in 1..2147483647) {$result = $result * $number}"

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin. 

Read more!

Converting VHD to VMDK SCSI for ESX

I had problems converting a 2003 server VHD to a vmdk that I could import into a VM running on ESX. I used WinImage to convert eh VHD->VMDK, but it seems WinImage creates the VMDK as an IDE device, which is unsupported by ESX. I'm sure there are better ways to do this, such as satisfying whatever the pre-requisites are to getting VMware converter to automatically inject the drivers, but it was interesting doing it manually.

Below is information on the process that I thought would have worked automatically, followed by the manual steps I took to make it work.

To convert the vmdk, the following process was first tried to get VMware converter to convert the IDE disk to something ESX would recognise:

1. 1. Use WinImage to create the vmdk from the vhd
2. 2. Use a VMware workstation VMX, including the disk as an IDE device. (a modified vmx is fine you don’t actually need VMware workstation)
3. 3. Use VMware converter to import the workstation VMX into VC

I thought this would have been enough, but the VMware Converter process failed at 95% saying that it couldn’t find symmpi.sys. Symmpi.sys is the LSI Logic SCSI driver for the virtual SCSI adapter. I’m guessing that running VMware converter should automatically inject the drivers into the vmdk but couldn’t in this scenario (maybe because my local PC didn’t have the driver, or maybe because the driver cache cab files containing symmpi.sys weren’t on the target vmdk).

Powering on the machine resulted in a stop 0x7b inaccessible disk error. To manually fix the problem, I then:

1. Added the disk to another VM. When starting the VM it warned that the new disk was created for LSI not buslogic. I said yes to convert to buslogic (which this VM was using as opposed to LSI).

The drive was then accessible through the VM, and I added the drivers (file and registry):

1. Copied the driver file to the drivers directory: copy "\\%workingVM%\c$\WINDOWS\system32\drivers\symmpi.sys" "%mountedDrive%:\WINDOWS\system32\drivers"
2. Copied the driver cache files to the machine from a working 2003: copy "\\%workingVM%\c$\WINDOWS\Driver Cache\i386\*.*" "%mountedDrive%:\WINDOWS\Driver Cache\i386"
3. Exported the HKLM\SYSTEM\CurrentControlSet\Services\symmpiregistry and HKLM\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase\pci#ven_1000&dev_0030 entries from a working server as regedit4 files (not Unicode).
4. Loaded the %mountedDrive%:\windows\system32\config\system registry hive on the disk to HKLM\VM: reg load HKLM\VM %mountedDrive%:\windows\system32\config\system
5. Modified the reg files to match the path the hive was loaded to (eg HKLM\VM\controlset001 instead of HKLM\system\currentcontrolset). The modified reg files are included below.
6. Imported the registry files which modified the loaded system hive on the disk
7. Disconnected the hive, shutdown the VM and disconnected the disk from the VM and reattached to the server created during the VMware Converter process
8. Turned on the server and was prompted to convert the disk type back to LSI (which I did).
9. The server started normally

Note that the conversion between buslogic and LSI logic in both directions were only required because the virtual machine that I mounted the disk on had a different adapter type.

---

REGEDIT4
[HKEY_LOCAL_MACHINE\VM\ControlSet001\Services\symmpi]
"ErrorControl"=dword:00000001
"Group"="SCSI miniport"
"Start"=dword:00000000
"Type"=dword:00000001
"ImagePath"=hex(2):73,79,73,74,65,6d,33,32,5c,44,52,49,56,45,52,53,5c,73,79,6d,\
6d,70,69,2e,73,79,73,00
"Tag"=dword:00000021

[HKEY_LOCAL_MACHINE\VM\ControlSet001\Services\symmpi\Parameters]
"BusType"=dword:00000001

[HKEY_LOCAL_MACHINE\VM\ControlSet001\Services\symmpi\Parameters\PnpInterface]
"5"=dword:00000001

[HKEY_LOCAL_MACHINE\VM\ControlSet001\Services\symmpi\Enum]
"0"="PCI\\VEN_1000&DEV_0030&SUBSYS_00000000&REV_01\\3&61aaa01&0&80"
"Count"=dword:00000001
"NextInstance"=dword:00000001

REGEDIT4
[HKEY_LOCAL_MACHINE\VM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1000&dev_0030]
"Service"="symmpi"
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"

References:
Injecting SCSI controller device drivers into Windows
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005208

Troubleshooting a virtual machine that fails to boot with STOP 0x7B error
http://kb.vmware.com/selfservice/viewContent.do?externalId=1006295&sliceId=1

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin.

Read more!

VMware VI3 iSCSI with multiple non-HBA NICs

While researching iSCSI on VI3, I came across some interesting information when using the ESX iSCSI software initiator that would be applicable to many installations, highlighting a potential bottleneck.

The short version is that if you’re using the iSCSI software initiator connecting to a single iSCSI target, multiple uplinks in an ESX network team for the VMKernel iSCSI port would not be used for load balancing.

This can be easily proven by connecting to the service console and running esxtop (n) to view the traffic for individual network adapters. Assuming your storage is in use, one or more physical uplinks for the vSwitch handling iSCSI should be showing traffic. You can also use resxtop through the RCLI on ESXi.

Why this happens

My understanding is that current ESX software initiated iSCSI connections have a 1:1 relationship between NIC and iSCSI targets. An iSCSI target in this sense is a connection to the IP-based SAN storage, not LUN targets. This limitation applies when the SAN presents a single IP address for connectivity.

VI3 software initiated iSCSI doesn’t support multipathing, which within ESX leaves only load balancing the physical uplinks in a team. Unfortunately, that leaves load balancing up to the vSwitch load balancing policy exceptions. I don’t believe any of the three choices fit most scenarios when connectivity to the iSCSI is through a single MAC/IP:

• Route based on the originating virtual switch port ID, based on virtual port ID, of which there is only one VMKernel iSCSI port
• Route based on source MAC hash, based on source MAC, of which there is only one
• Route based on IP hash, based on layer 3 source-destination IP pair, of which there is only one (VMKernel -> iSCSI virtual address). I don’t think this is a generally recommended load balancing approach anyway

Link aggregation

The VI3 SAN Deploy guide does state that one connection is established to each target. This seems to indicate one connection per LUN target, but the paragraph starts with software iSCSI and switches half way through to discuss iSCSI HBA’s.

I’m still unsure of whether software iSCSI has multiple TCP sessions, one per target (I don’t believe this is the case). The blog referenced below also talks about 802.3 link aggregation which states the ESX 3.x software initiator does not support multiple TCP sessions.

However, if multiple TCP sessions were being established for the iSCSI software initiator to a single target IP address, this opens the possibility of link aggregation at the physical switch. When using 802.3ad LACP in this IP-IP scenario, the switches would have to distribute connections based on the hash of TCP source/destination ports, rather than just IP/MAC.

The following excerpt from the SAN deploy guide:

Software iSCSI initiators establish only one connection to each target.

Therefore, storage systems with a single target that contains multiple LUNs have all LUN traffic routed through that one connection. In a system that has two targets, with one LUN each, two connections are established between the ESX host and the two available volumes. For example, when aggregating storage traffic from multiple connections on an ESX host equipped with multiple iSCSI HBAs, traffic for one target can be set to a specific HBA, while traffic for another target uses a different HBA. For more information, see the “Multipathing” section of the iSCSI SAN Configuration Guide. Currently, VMware ESX provides active/passive multipath capability. NIC teaming paths do not appear as multiple paths to storage in ESX host configuration displays, however. NIC teaming is handled entirely by the network layer and must be configured and monitored separately from ESX SCSI storage multipath configuration.

VI4/vSphere

Excerpts from the following blog, indicate that changes in vSphere for software iSCSI to support multiple iSCSI sessions, allowing multipathing or link aggregation, which would allow separate iSCSI TCP sessions to be spread across more than one NICs (depending on how many iSCSI sessions).

http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html

The current experience discussed above (all traffic across one NIC per ESX host):

VMware can’t be accused of being unclear about this. Directly in the iSCSI SAN Configuration Guide: ESX Server‐based iSCSI initiators establish only one connection to each target. This means storage systems with a single target containing multiple LUNs have all LUN traffic on that one connection, but in general, in my experience, this is relatively unknown.

This usually means that customers find that for a single iSCSI target (and however many LUNs that may be behind that target – 1 or more), they can’t drive more than 120-160MBps. This shouldn’t make anyone conclude that iSCSI is not a good choice or that 160MBps is a show-stopper. For perspective I was with a VERY big customer recently (more than 4000 VMs on Thursday and Friday two weeks ago) and their comment was that for their case (admittedly light I/O use from each VM) this was working well. Requirements differ for every customer.

The changes in vSphere:

Now, this behavior will be changing in the next major VMware release. Among other improvements, the iSCSI initiator will be able to use multiple iSCSI sessions (hence multiple TCP connections). Looking at our diagram, this corresponds with “multiple purple pipes”for a single target. It won’t support MC/S or “multiple orange pipes per each purple pipe” – but in general this is not a big deal (large scale use of MC/S has shown a marginal higher efficiency than MPIO at very high end 10GbE configurations) .

Multiple iSCSI sessions will mean multiple “on-ramps” for MPIO (and multiple “conversations” for Link Aggregation). The next version also brings core multipathing improvements in the vStorage initiative (improving all block storage): NMP round robin, ALUA support, and EMC PowerPath for VMware which integrates into the MPIO framework and further improves multipathing. In the spirit of this post, EMC is working to make PowerPath for VMware as heterogeneous as we can.

Together – multiple iSCSI sessions per iSCSI target and improved multipathing means aggregate throughput for a single iSCSI target above that 160MBps mark in the next VMware release, as people are playing with now. Obviously we’ll do a follow up post.

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin.

Read more!

VirtualCenter Physical to Virtual

This post describes a process I executed to replace a physical VMware VirtualCenter box with a virtual equivalent running in its own cluster. There were no VM outages of machines running on the cluster – each ESX kept merrily running the virtuals until everything was sorted out with the VC layer.

Note that this was done in a lab instance of ESX 3.5 and VirtualCenter 2.5, had this been production I probably would have taken a little more care.

I don’t think there is any compelling reason why you wouldn’t run a virtual VC box, it would be hypocritical of VMware to suggest virtualizing your application servers, except VirtualCenter which should be physical. Having said that, this process could also be used in reverse, taking a VM VirtualCenter instance physical should the need arise. This would also be useful for a VirtualCenter disaster recovery scenario.

The configuration before completing these steps

1. Physical VirtualCenter 2.5 running on Windows Server 2003, called vc01, static IP
2. Virtual Windows Server 2003 computer running in the cluster, called vc02, dynamic IP \
3. SQL 2005 database for VC, running on a separate SQL server.
4. ESX 3.5 VC 2.5, single network, iSCSI shared storage

Prerequisites

1. The IP address of the VirtualCenter server
2. The ESX host of the VM becoming the new VC server
3. The path to your VMware FlexLM license file (assuming you’re using a license server)
4. The logon details for the SQL connection between vpxd and the database

Steps Taken

On the physical VirtualCenter box that is going to be decommissioned:

1. Take note of the physical ESX host running the VM becoming the new VC server
2. Stop the vpxd service and change the startup to manual (sc stop %service%, sc config %service% start= demand)
3. Stop the vmountVpx, vmware-ufad-vci, vmware-converter and webAccess services change the startup to manual
4. Stop the flexlm instance and set the startup to manual
5. Take a backup of your .lic license file
6. Change the IP address to dynamic (assuming it is static)
7. Power off the physical machine
8. Delete the computer account for vc01 from the domain
9. On the SQL server hosting the VirtualCenter database, backup the VC database and log file with the following command executed through management studio:
   
   

   BACKUP DATABASE [VMVC]
   TO DISK = N'c:\temp\VC_PreMoveToVM.bak'
   WITH
   DESCRIPTION = N'VirtualCenter Pre-move to VM backup'
   , INIT
   , NAME =
   N'VC Pre move to VM'
   GO
   
   RESTORE VERIFYONLY
   FROM DISK =
   N'c:\temp\VC_PreMoveToVM.bak '
   WITH FILE = 1
   GO
   

10. In the service console of one of the ESX hosts, backup the current certificates (just in case):
1. mkdir /tmp/cert_backup
2. cp /etc/vmware/ssl/* /tmp/cert_backup
11. On the VM becoming the new VirtualCenter server running on a host in the cluster: Rename the server from vc02 to vc01, with the same static IP as the previous vc01
12. Restart the virtual machine
13. Install the SQL Native Client - required for VC 2.5 SQL connectivity on a non-SQL server
14. Install VirtualCenter including FlexLM, connecting to the existing database and using the license file copied off the physical server
15. In VirtualCenter, disconnect the first ESX host in the cluster - maintenance mode is not possible at this stage – login errors occur because of the incorrect certificate
16. Copy the new VC certificates - I use pscp here, but whatever you normally use to copy files to ESX would be fine:
    
    1. cd "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\ssl"
    2. pscp rui.crt vcadmin@esx01:/etc/vmware/ssl/
    3. pscp rui.key vcadmin@esx01:/etc/vmware/ssl/"
17. In the service console of the ESX node you’ve just updated the certificates on, restart the management interface, the ESX host did not seem to pick up the new certificate dynamically (maybe it would on a schedule without a restart):
    1. service mgmt-vmware restart
18. Connect the ESX host to VC through the VI Client interface
19. Repeat steps 15-18 for each other ESX host in the cluster
20. In VirtualCenter, run 'Reconfigure for HA' on each ESX node
    
    Testing
    
    
21. Ensure the vpxd.log file reports no problems with host connectivity or certificates (also check the VC/ESX logs)
22. Ensure each ESX host is receiving licenses from the 'new' license server, either through the VI client or the FlexLM admin tool.
23. Ensure you can perform simple tasks such as powering on a virtual machine
24. Ensure VMotion/HA/DRS is working

The (untested) rollback plan if something goes wrong:

1. Shutdown the new VirtualCenter VM
2. Restore the database from the backup taken
3. Power on the physical VC box, change the IP to the static IP
4. Restart the vpxd and VMware License Server services
5. For each ESX host in the cluster, disconnect the host, restore the old certificates, restart the management service and connect the ESX host to the old VirtualCenter instance

Additional notes:

• Even though user login errors were returned when the vpxd service tried to form the cluster - which points to the vpxuser account used by VC to manage ESX hosts - this was misleading as this username and password is stored in the VC database – which had not been modified (in the vpx_host table). The next logical step was certificates, which lead to certificate update process used above.
• Manually copying the certificates may not be strictly required, as when I went part-way to reconnecting a host without updating the certificates, I was prompted that another VC instance was managing these servers, would I like to continue. Presumably it would have automatically updated the certificates as required.

Read more!

Virtual 2003 MSCS Cluster in ESX VI3

This post shares a method I've used to create test-lab instances of standard 2003 file and print Microsoft Cluster Services (MSCS) clusters in a VMware ESX VI3 virtual environment. The resultant solution is not supported and definitely not production-ready, but if you want a real multi-node MSCS cluster in an ESX lab environment, this process might be helpful with a minimum set of requirements.

With my usual theme of repeatable command-line execution, most of these operations can be completed via the command-line, either in the ESX service console or a command-prompt from the virtual MSCS nodes.

I followed bits and pieces of the VMware supported method - which is very specific and quite restrictive. Note that I’m a little dubious that this cluster would be particularly stable – the SCSI reservations MSCS uses to lock disks are in no way supported when using a shared VMDK through a shared SCSI adapter (I think RDM is the only supported method), but it does work and at least provided me with a test environment.

The shared nothing model of 2003 MSCS clustering dictates that only one node accesses the partition at any one time, but the disk still needs to be visible to both nodes. A limitation of this solution is that both MSCS nodes need to be hosted on one ESX server – a requirement you could satisfy with a DRS rule to keep the two nodes together. However, if DRS decided to migrate both VMs, the cluster would almost certainly break during the failover (and possibly after).

If you follow the steps below, you should end up with two virtual x64 2003 enterprise servers, both members of a single MSCS cluster. In the cluster there will three shared disks (VMDKs), one for the quorum and one each for file and print – with a virtual server and relevant cluster resources. A test file share is created, along with drivers and a test printer. You'll need to modify the commands that reference the public adapter and IP addresses

Steps involved:

1. Create an area for storage of the shared disk on your datastore:
   1. mkdir /vmfs/volumes/%datastore%/cluster01
2. Create a 5GB quorum disk:
   
   1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-Quorum.vmdk
3. Create a 5GB disk for shared data:
1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-disk01.vmdk
4. Create two 2003 x64 enterprise virtual machines, either through cloning, deployment with templates or whatever your standard build process may be
5. If cloning was used, run sysprep on both nodes to give a unique SID and join your lab domain
6. Shutdown the first node and add the shared disk
1. Add the quorum disk, mounted under scsi 1:0 (which adds a new SCSI adapter)
2. Set the newly created SCSI Adapter to SCSI bus sharing virtual
3. Add disk01, attached as scsi 1:1
7. In the first VM, use disk administrator (or diskpart) to initialise the quorum and disk01 disks, partitioned with basic. Record the signature of the disk and the drive letter used (although this is the disk volume when the disk is owned by the OS, not the cluster).
8. Add a service account for the cluster service:
1. dsadd user "CN=clustersvc,CN=Users,DC=test,DC=local" -pwdneverexpires yes -pwd password -disabled no -desc "MSCS VM cluster service account"
2. Ensure the service account is an administrator of each virtual 2003 node
9. Use Cluster Administrator to install the cluster on the first node, with your chosen cluster name, using the created quorum disk and service account
10. Verify correct operation of the single-node cluster, and then add the second VM node to the cluster.
11. Create a new port group to allow a second private adapter on each ESX server:
1. esxcfg-vswitch -A MSCS-Private Private
2. Add a second interface to each VM cluster node, allocated separate address space
3. Verify connectivity (ping) and configuration following cluster best practices (no gateway, no DNS etc)
4. Mark as a private heartbeat connection for the cluster, prioritised above the LAN connection.
12. Create a virtual resource group, creating IP, network name and disk resources in the group, the following commands will create a group called v01, in the lab01 cluster. For these steps, you’ll need the drive letter to use (M: below), the disk signature, the public network name, IP Address and subnet mask of the virtual server being created:
1. cluster /cluster:lab01 group "v01" /create
2. cluster /cluster:lab01 res "v01 Disk01" /create /group:"v01" /type:"physical disk"
3. cluster /cluster:lab01 res "v01 Disk01" /priv Drive="M:"
4. cluster /cluster:lab01 res "v01 Disk01" /priv signature=0x%disksignature%
5. cluster /cluster:lab01 res "v01 Disk01" /prop Description="M: disk01"
6. cluster /cluster:lab01 res "v01 Disk01" /On
7. cluster /cluster:lab01 res "v01 IP" /create /group:"v01" /type:"IP Address"
8. cluster /cluster:lab01 res "v01 IP" /priv Network="%publicNetwork%"
9. cluster /cluster:lab01 res "v01 IP" /priv Address=192.168.10.10
10. cluster /cluster:lab01 res "v01 IP" /priv SubnetMask=255.255.255.0
11. cluster /cluster:lab01 res "v01 IP" /priv EnableNetBIOS=1
12. cluster /cluster:lab01 res "v01 IP" /priv OverrideAddressMatch=0
13. cluster /cluster:lab01 res "v01 IP" /AddDep:"v01 Disk01"
14. cluster /cluster:lab01 res "v01 IP" /On
15. cluster /cluster:lab01 res "v01" /create /group:"v01" /type:"Network Name"
16. cluster /cluster:lab01 res "v01" /priv RequireKerberos=1
17. cluster /cluster:lab01 res "v01" /AddDep:"v01 IP"
18. cluster /cluster:lab01 res "v01" /priv Name="v01"
19. cluster /cluster:lab01 res "v01" /On
13. Install ABEUIamd64.msi on each node if Access Based Enumeration is required
14. To create a test directory, share and ABE resource on the new virtual server on the cluster (v01):
1. md \\v01\m$\Dir01
2. cluster /cluster:lab01 res "v01 Dir01 Share" /create /group:"v01" /type:"File Share"
3. cluster /cluster:lab01 res "v01 Dir01 Share" /priv path="M:\Dir01"
4. cluster /cluster:lab01 res "v01 Dir01 Share" /priv Sharename=Dir01
5. cluster /cluster:lab01 res "v01 Dir01 Share" /priv Remark="Dir01 File Share"
6. cluster /cluster:lab01 res "v01 Dir01 Share" /prop Description="Dir01 File Share"
7. cluster /cluster:lab01 res "v01 Dir01 Share" /priv security=Everyone,grant,F:security
8. cluster /cluster:lab01 res "v01 Dir01 Share" /AddDep:"v01"
9. cluster /cluster:lab01 res "v01 Dir01 Share" /AddDep:"v01 Disk01"
10. cluster /cluster:lab01 res "v01 Dir01 Share" /On
11. cluster /cluster:lab01 res "v01 Dir01 ABE" /create /group:"v01" /type:"Generic Application"
12. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv CommandLine="cmd.exe /k abecmd.exe /enable Dir01"
13. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv CurrentDirectory="%SystemRoot%"
14. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv InteractWithDesktop=0
15. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv UseNetworkName=0
16. cluster /cluster:lab01 res "v01 Dir01 ABE" /prop SeparateMonitor=1
17. cluster /cluster:lab01 res "v01 Dir01 ABE" /prop Description="Access Based Enumeration for Dir01 File Share"
18. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01"
19. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01 Disk01"
20. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01 Dir01 Share"
21. cluster /cluster:lab01 res "v01 Dir01 ABE" /On
15. Additional shared cluster disks can be created as required, eg:
1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-disk02.vmdk
2. Add the disks to one node, (scsi 1:2 in this example). Initialise and allocate in the cluster (as in step 7 above)
16. To create a virtual print server (assuming you’ve mounted disk02 from step 15 for use in the cluster):
1. cluster /cluster:lab01 group "v02" /create
2. cluster /cluster:lab01 res "v02 Disk02" /create /group:"v02" /type:"physical disk"
3. cluster /cluster:lab01 res "v02 Disk02" /priv Drive="P:"
4. cluster /cluster:lab01 res "v02 Disk02" /priv signature=0x%disksignature%
5. cluster /cluster:lab01 res "v02 Disk02" /prop Description="P: print01"
6. cluster /cluster:lab01 res "v02 Disk02" /On
7. cluster /cluster:lab01 res "v02 IP" /create /group:"v02" /type:"IP Address"
8. cluster /cluster:lab01 res "v01 IP" /priv Network="%publicNetwork%"
9. cluster /cluster:lab01 res "v01 IP" /priv Address=192.168.10.11
10. cluster /cluster:lab01 res "v01 IP" /priv SubnetMask=255.255.255.0
11. cluster /cluster:lab01 res "v02 IP" /priv EnableNetBIOS=1
12. cluster /cluster:lab01 res "v02 IP" /priv OverrideAddressMatch=0
13. cluster /cluster:lab01 res "v02 IP" /AddDep:"v02 Disk02"
14. cluster /cluster:lab01 res "v02 IP" /On
15. cluster /cluster:lab01 res "v02" /create /group:"v02" /type:"Network Name"
16. cluster /cluster:lab01 res "v02" /priv RequireKerberos=1
17. cluster /cluster:lab01 res "v02" /AddDep:"v02 IP"
18. cluster /cluster:lab01 res "v02" /priv Name="v02"
19. cluster /cluster:lab01 res "v02" /On
17. Create v02 print spooler:
1. cluster /cluster:lab01 res "v02 Spooler" /create /group:"v02" /type:"print spooler"
2. cluster /cluster:lab01 res "v02 Spooler" /priv DefaultSpoolDirectory="P:\Spool"
3. cluster /cluster:lab01 res "v02 Spooler" /prop Description="v02 Print Spooler"
4. cluster /cluster:lab01 res "v02 Spooler" /AddDep:"v02 Disk02"
5. cluster /cluster:lab01 res "v02 Spooler" /AddDep:"v02"
6. cluster /cluster:lab01 res "v02 Spooler" /On
18. On v02, add a standard Laserjet 4000 retail driver for x64 and x86, run from a cluster node:
1. rundll32 printui.dll,PrintUIEntry /ia /c \\v02 /m "HP LaserJet 4000 Series PCL6" /h "x64" /v "Windows XP and Windows Server 2003"
2. rundll32 printui.dll,PrintUIEntry /ia /c \\v02 /m "HP LaserJet 4000 Series PCL6" /h "x86" /v "Windows 2000, Windows XP and Windows Server 2003"
19. Create a test printer on v02 called printer01 using the LJ 4000 driver, with a record in DNS, published in AD, set to duplex by default, with customised permissions using the standard winprint processor:
1. dnscmd %DNSserver% /recordadd %zone% printer01 A 192.168.10.100
2. cscript //nologo portmgr.vbs -a -c \\v02 -p printer01 -h 192.168.10.100 -t LPR -q printer01
3. cscript //nologo prnmgr.vbs -a -c \\v02 -b printer01 -m "HP LaserJet 4000 Series PCL6" -r printer01
4. cscript //nologo prncfg.vbs -s -b \\v02\printer01 -h printer01 -l "%Location%" +published
5. setprinter.exe \\v02\printer01 8 "pDevMode=dmDuplex=2,dmCollate=1,dmFields=duplex collate"
6. subinacl /printer \\v02\printer01 /grant=%domain%\%group%=F
7. setprinter \\v02\printer01 2 pPrintProcessor="WinPrint"

References

VMware Support method of running MSCS clusters:
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_mscs.pdf

Implementing an MSCS 2003 server cluster Cluster
http://waynes-world-it.blogspot.com/2008/03/implementing-mscs-2003-server-cluster.html

subinacl 5.2.3790.1180:
http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B

Windows Server 2003 Resource Kit Tools:
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&DisplayLang=en

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin.

Read more!