OpsMgr 2007 performance script - VMware datastores

This post provides a method of collecting VirtualCenter datastore information for ESX hosts in one or more clusters as performance data in Operations Manager 2007. You can then trend datastore usage for longer term analysis, and alert on the data for proactive problem management. This is my first attempt at using script to gather performance data in OpsMgr 2007, and my lack of OpsMgr knowledge combined with the limited documentation/examples made this difficult and possibly harder than it needs to be.

Background

In OpsMgr 2007, I thought it would be useful to report on LUN disk space on the ESX side of things in a similar fashion to agent-based Windows disk performance counters, and while this sounds quite easy, I couldn’t find a simple method because:

1. ESX doesn’t seem to support an SNMP trap to report on VMFS volume capacity/free space. You could write a shell script and add a cron job on the Linux ESX service console, but this would be targeted at each ESX server, and the results would then be duplicated (and wouldn't work with ESX 3i)
2. VirtualCenter doesn’t appear to support a custom alert based on storage usage, only storage transfer rates.
3. The ESX servers are only in OpsMgr as network devices, not true agents, so we can’t use WBEM or any other method to query directly. If the upcoming OpsMgr 2007 cross platform extensions work on ESX, this may be easier, although as with option 1 this would still return duplicate datastore information for each ESX server in a cluster and wouldn't work with ESX 3i)

This left a few options to gather the data in OpsMgr:

1. Query the VMFS storage through the VI snap-in and a PowerShell Script.
2. Query the VC database directly for this information from a VBS or PowerShell script.
3. Create a DTS job on the VC SQL server to extract the data to a suitable format.

Options two and three above would not be supported by VMware and may change as the database structure changes between VC versions.

Solution

Querying the VMFS storage from PowerShell and collect the information with VBScript, which includes:

1. The PowerShell script using the VI snap-in to extract the datastore information
2. The VBScript to gather the information as performance data in Operations Manager 2007
3. A trigger to execute the powershell script and put the data in a file to be gathered.
4. Rules in OpsMgr to gather the data using the type 'Script (Performance)'.

Steps taken:

1. Install the VMware VI PowerShell snap-in to your Virtual Centre box (you may need to install PowerShell first). VMware-Vim4PS-1.0.0-113525.exe is the binary I used.
2. Create an AD group and user for rights to VC. Add the user to the group.
3. Set rights in VirtualCenter to allow the group read-only rights at the root. This should be more granular if possible.
4. Add a recurring trigger on your VC box to run the PowerShell script to extract the information. Eg, this could be a scheduled task that runs a batch file or directly executes:
   
   1. powershell . ".\PerfGatherVMwareDSSpace.ps1"
5. Create two rules in Operations Manager, Script (Performance) to gather the daily log file and store in the database as performance data. One rule is for the free space, another is for the free space as a percentage of the capacity. See the performance mapper notes below.
   
   1. The rules were created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server (where the powershell script runs). You may want to target the rule differently, depending on your environment.
6. Create a performance view, showing performance data collected by the two rules above.
7. Create a monitor, using the LogicalDisk ‘% Disk Free’ object and counter to monitor on a threshold of 10%. Similarly, the monitor could be created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server. The alert contains the following information:
1. Object: $Data/Context/ObjectName$, Counter: $Data/Context/CounterName$, Instance: $Data/Context/InstanceName$, Value: $Data/Context/Value$

Note that when using the script performance data provider, the ‘Performance Mapper’ tab is used to map the data returned by the script to the database. To make this generic, both the instance name and the value are used from each element:

• Object: LogicalDisk
• Counter: Free Megabytes
• Instance: $Data/Property/@Name$
• Value: $Data/Property[@Name]$

Note that the VBScript creates three typed ‘property bags’ and returns all. This results in a single XML that contains three dataitem elements, one for each instance. Creating a single property bag and adding the three instances results in only the first being processed.

This should now gather free space in MB and as a percentage of the size for all VMFS datastores your VC server knows about, visible through the performance view, and alerted on when free space is less than 10% of the capacity for each volume.

SQL Queries

Some SQL queries against the OperationsManagerDW database I used along the way to query from the datawarehouse that the data was being gathered. Note that where I've specified 'datastore%' as a filter, you'll need to change this to the prefix of your datastores, eg ds01, ds02 would be ds%.

/*Return the raw performance data collected for the datastore* instances: */

select DateAdd(Hour, 10, DateTime) as DateTime, InstanceName, SampleValue from perf.vperfRaw
inner join vPerformanceRuleInstance on perf.vperfRaw.PerformanceRuleInstanceRowID = vPerformanceRuleInstance.PerformanceRuleInstanceRowID
where InstanceName like 'datastore%'
order by datetime desc

/* Find new rules: */
select top 10 * from dbo.vPerformanceRuleInstance
where instancename like 'datastore%'
order by performanceruleinstancerowid desc

/* Find new raw performance data: */

select top 100 DateTime, SampleValue, ObjectName, CounterName, FullName, Path, Name, DisplayName, ManagedEntityDefaultName from perf.vperfraw
inner join vPerformanceRule on perf.vperfraw.PerformanceRuleInstanceRowID = vPerformanceRule.RulerowID
inner join vManagedEntity on perf.vperfraw.ManagedEntityRowID = vManagedEntity.ManagedEntityRowID
order by datetime desc

/* Find new rule instances that have been created: */

select top 10 * from dbo.vPerformanceRuleInstance
order by performanceruleinstancerowid desc
--

#PowerShell Script - PerfGatherVMwareDSSpace.ps1
# Note that this includes a context to connect to VC, with hardcoded username and password.  You could avoid this by running the scheduled task under the security context you created with inherent rights to VC, and then remove the explicit -credential argument to Connect-VIServer.

$ErrorActionPreference = "SilentlyContinue"
add-pssnapin VMware.VimAutomation.Core

$ADMINLOG = "c:\admin\logs"
$outputFile = ""
$today = [DateTime]::Now.ToString("yyyyMMdd")

$scriptName = $MyInvocation.MyCommand.Name
$scriptName = $scriptname.substring(0, $scriptname.LastIndexOf("."))

if ($env:adminlog -ne $null) {        # Was there an adminlog environment variable?
    $outputFile = $env:adminlog
} else {
   $outputFile = $ADMINLOG        # No, use the constant default  
}

$outputFile += "\" + $scriptname + "_" + $today + ".csv"    # Construct the full path to the output file 

$server = "vc01"          # VC instance
$username = "domain\user"        # Hardcoding is bad, but at least this is a RO account.
$password = "password"

$pwd = convertto-securestring $password -asplaintext -force

$cred = New-Object Management.Automation.PSCredential ($username, $pwd)   # Create the credentials to use with the VC connection

$vmserver = & { trap {continue}; Connect-VIServer -server $server -Credential $cred } # Connect to VC, trapping the error

if ($vmserver -is [System.Object]) {       # Do we have a connection?
    Get-Datastore -server $vmserver | export-csv -noTypeInformation -path $outputFile # Yes, get the datastore details and store as CSV
    if (test-path -path $outputFile) {
        write-output "Datastore details exported to $outputFile"
    } else {
        write-output "Error: Datastore details were not exported to $outputFile"
    }
    $vmserver = $null
} else {
    write-output "Could not connect to $server"
}


#------



' VBScript


' Parse a CSV file containing the VMware volume information, and return the specified field from the data file or the calculated percent free

Const TOKEN_DATE = "%date%"
Dim SOURCE_FILE : SOURCE_FILE = "c:\admin\logs\PerfGatherVMwareDSSpace_" & TOKEN_DATE & ".csv"  ' Today's log file exported from the PowerShell VI-snapin script

Const DELIMITER = ","           ' CSV data file
Const ForReading = 1
Const DISKTYPE_VMFS = "VMFS"          ' We're interested only in lines with VMFS volumes

Const QUERY_SIZE = "Size"
Const QUERY_FREE = "Free"
QUERY_PERCENT = "Percent"

Dim QUERY_DEFAULT : QUERY_DEFAULT = QUERY_FREE 

Const FIELD_SIZE = 1           ' The field in the CSV that contains the capacity of the VMFS volume
Const FIELD_FREE = 0           ' The field in the CSV that contains the free disk space on the VMFS volume
Const FIELD_PERCENT = 2

Const PerfDataType  = 2

Set oFSO = CreateObject("Scripting.FileSystemObject")

Main()

wscript.quit(0)

Sub Main()

 If WScript.Arguments.UnNamed.Count >= 1 Then
  strQueryType = WScript.Arguments.UnNamed(0)
  wscript.echo "Command-line argument passed, querying for " & strQueryType
 Else
  strQueryType = QUERY_DEFAULT
  wscript.echo "No command-line argument passed, querying for the default of " & strQueryType
 End if

 Select Case strQueryType
  Case QUERY_SIZE  strField = FIELD_SIZE 
  Case QUERY_FREE  strField = FIELD_FREE
  Case QUERY_PERCENT strField = QUERY_PERCENT
  Case Else  strField = FIELD_FREE
 End Select

 dtmDate = Now 
 strToday = DatePart("yyyy", dtmDate)         ' YYYY
 strToday = strToday & String(2 - Len(DatePart("m", dtmDate)),"0") & DatePart("m", dtmDate) ' MM
 strToday = strToday & String(2 - Len(DatePart("d", dtmDate)),"0") & DatePart("d", dtmDate) ' DD

 strDataFile = Replace(SOURCE_FILE, TOKEN_DATE, strToday)     ' Build the path to the data file
 wscript.echo "Looking for " & strDataFile

 Dim oAPI, oBag
 Set oAPI = CreateObject("MOM.ScriptAPI")

 If oFSO.FileExists(strDataFile) Then        ' Does the data file exist for today?
  WScript.Echo "Log file found, continuing"
  Call ReadFile(strDataFile, strBuffer)       ' Yes, read the contents of the file into the buffer

  For Each strLine in Split(strBuffer, vbCRLF)      ' For each line
   arrEntry = Split(strLine, DELIMITER)      ' Split the line into an array on comma
   If UBound(arrEntry) = 5 Then       ' Did this line have a valid number of fields?
    If strComp(arrEntry(3), DISKTYPE_VMFS) = 0 Then    ' Yes, is it a VMFS volume? (excludes the header)
    wscript.echo "Processing " & strLine
     If IsNumeric(arrEntry(FIELD_FREE)) AND IsNumeric(arrEntry(FIELD_SIZE)) Then     ' Yes, is the field we're after a numeric?
      Set oBag = oAPI.CreateTypedPropertyBag(PerfDataType)  Create a typed name/value pair property bag

      If (strField = FIELD_SIZE) OR (strField = FIELD_FREE ) Then
       Call oBag.AddValue(arrEntry(5),CLng(arrEntry(strField))) ' Yes, convert to a long and add to the bag
      ElseIf strField = QUERY_PERCENT Then
       dblFreePercent = CDbl(arrEntry(FIELD_FREE) / arrEntry(FIELD_SIZE)*100)
       Call oBag.AddValue(arrEntry(5), Round(dblFreePercent, 2))
      End If
      oAPI.AddItem(oBag)     ' Add the item to the bag (doing it here results in three datatime elements)

     End If  
    End If
   End If
  Next

 Else
  WScript.Echo "Error: Daily data file not found - " & strDataFile 

 End If

 Call oAPI.ReturnItems 

End Sub    


'********************************************************
' Purpose: Read a file and store the contents in the buffer
' Assumptions: oFSO exists
'              strLogFile contains the path/filename of the file to operate on
' Effects: strBuffer is by reference
' Inputs: strFileName, Path and filename of the source file
'   strBuffer, the buffer used to store the contents of the file
' Returns: None
'
'********************************************************
Sub ReadFile (ByVal strFileName, ByRef strBuffer)   
 On Error Resume Next
 Dim objTextStream

 If Not oFSO.FileExists(strFileName) Then
  WScript.Echo "Error: " & strFileName & " file not found."
  Exit Sub
 End If
    Set objTextStream = oFSO.OpenTextFile(strFileName, ForReading)
 strBuffer = objTextStream.ReadAll
End Sub

'------

References:

How to Create a Probe-Based Performance Collection Rule in Operations Manager 2007
http://technet.microsoft.com/en-us/library/bb381406.aspx

MOMScriptAPI.CreatePropertyBag Method
http://msdn.microsoft.com/en-us/library/bb437556.aspx

XPath Examples:
http://msdn.microsoft.com/en-us/library/ms256086(VS.85).aspx

http://blogs.msdn.com/mariussutara/archive/2008/01/24/momscriptapi-createtypedpropertybag-method.aspx

http://blogs.technet.com/kevinholman/archive/2008/07/02/collecting-and-monitoring-information-from-wmi-as-performance-data.aspx

http://www.afinn.net/2008/07/collecting-performance-data-in-operations-manager-2007-and-publishing-to-sharepoint-part-1/

http://blogs.msdn.com/mariussutara/archive/2007/11/13/alert-description-and-parameter-replacement.aspx

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.