vSphere Storage vMotion traffic

This post provides information on storage vMotion in VMware vSphere 4 with ESXi 4.0 U2 hosts. While using storage vMotion to move data between unconnected storage, I conducted some simple testing to identify the networks and ports used for vMotion.

I’ve never seen an explanation of how storage vMotion works, and while it seems kind of obvious in retrospect, I thought some people might find this interesting.


Summary

1. The scope of a storage vMotion operation is datacenter wide, not limited to clusters or which host can see which storage.
2. In the testing conducted, storage vMotion relies upon the management network to transfer the disk data between the two ESXi hosts
3. If this method was used to import a large number of VMs from foreign storage then the limitation would typically be the 1GB vmnic used for the management network.
4. To allow the migration between datastores not visible by a single host, the VM would also be migrated between hosts.

Scenario

Two hosts managed by single vCenter instance, vc01:

1. esx01 part of the CLUS1 cluster
2. esx02 not in a cluster
3. esx01, connected only to the iSCSI SAN 1
4. esx02, connected only to the iSCSI SAN 2

Management network on vmk0 using two GbE interfaces on both hosts. Vmk0 on the management network with load balancing based on the virtual port ID.

Test

The test conducted was using storage vmotion to move a virtual machine through VirtualCenter on vc01 from a datastore visible only to esx02 to a datastore visible only to esx01.

1. The source datastore was LUN02 (visible only to esx02) and the destination datastore was LUN01 (visible only to esx01).
2. vMotion was used to automatically migrate the VM from esx02 to esx01 (as the VM had moved storage)

Traffic and network usage observed

1. On esx02 unsupported console, vmk0 source 192.168.0.12.60504 (esx02), destination 192.168.0.11.902 (esx01).
2. Command: /bin/tcpdump-uw -I vmk0 -nn

esx02

1. MbRx 190Mb/sec on software iSCSI NIC – Reading the disk information from iSCSI SAN 2 for the VM
2. MbTx 136Mb/sec – Transmit the information across the management network to esx01.
3. Command: resxtop --server esx02 (‘n’ for network information)

esx01

1. MbRx 120Mb/sec – receiving the disk data from esx02 across the management network
2. MbTx 200Mb/sec on software iSCSI NIC – writing the data to the SAN across the iSCSI network
3. Command: resxtop --server esx01 (‘n’ for network information)

The following diagram shows the traffic flow of storage vMotion on an iSCSI network: