Sunday, November 2, 2008

OpsMgr 2007 performance script - VMware datastores

This post provides a method of collecting VirtualCenter datastore information for ESX hosts in one or more clusters as performance data in Operations Manager 2007. You can then trend datastore usage for longer term analysis, and alert on the data for proactive problem management. This is my first attempt at using script to gather performance data in OpsMgr 2007, and my lack of OpsMgr knowledge combined with the limited documentation/examples made this difficult and possibly harder than it needs to be.


In OpsMgr 2007, I thought it would be useful to report on LUN disk space on the ESX side of things in a similar fashion to agent-based Windows disk performance counters, and while this sounds quite easy, I couldn’t find a simple method because:

  1. ESX doesn’t seem to support an SNMP trap to report on VMFS volume capacity/free space. You could write a shell script and add a cron job on the Linux ESX service console, but this would be targeted at each ESX server, and the results would then be duplicated (and wouldn't work with ESX 3i)
  2. VirtualCenter doesn’t appear to support a custom alert based on storage usage, only storage transfer rates.
  3. The ESX servers are only in OpsMgr as network devices, not true agents, so we can’t use WBEM or any other method to query directly. If the upcoming OpsMgr 2007 cross platform extensions work on ESX, this may be easier, although as with option 1 this would still return duplicate datastore information for each ESX server in a cluster and wouldn't work with ESX 3i)

This left a few options to gather the data in OpsMgr:

  1. Query the VMFS storage through the VI snap-in and a PowerShell Script.
  2. Query the VC database directly for this information from a VBS or PowerShell script.
  3. Create a DTS job on the VC SQL server to extract the data to a suitable format.

Options two and three above would not be supported by VMware and may change as the database structure changes between VC versions.


Querying the VMFS storage from PowerShell and collect the information with VBScript, which includes:

  1. The PowerShell script using the VI snap-in to extract the datastore information
  2. The VBScript to gather the information as performance data in Operations Manager 2007
  3. A trigger to execute the powershell script and put the data in a file to be gathered.
  4. Rules in OpsMgr to gather the data using the type 'Script (Performance)'.

Steps taken:

  1. Install the VMware VI PowerShell snap-in to your Virtual Centre box (you may need to install PowerShell first). VMware-Vim4PS-1.0.0-113525.exe is the binary I used.
  2. Create an AD group and user for rights to VC. Add the user to the group.
  3. Set rights in VirtualCenter to allow the group read-only rights at the root. This should be more granular if possible.
  4. Add a recurring trigger on your VC box to run the PowerShell script to extract the information. Eg, this could be a scheduled task that runs a batch file or directly executes:
    1. powershell . ".\PerfGatherVMwareDSSpace.ps1"
  5. Create two rules in Operations Manager, Script (Performance) to gather the daily log file and store in the database as performance data. One rule is for the free space, another is for the free space as a percentage of the capacity. See the performance mapper notes below.
    1. The rules were created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server (where the powershell script runs). You may want to target the rule differently, depending on your environment.
  6. Create a performance view, showing performance data collected by the two rules above.
  7. Create a monitor, using the LogicalDisk ‘% Disk Free’ object and counter to monitor on a threshold of 10%. Similarly, the monitor could be created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server. The alert contains the following information:
    1. Object: $Data/Context/ObjectName$, Counter: $Data/Context/CounterName$, Instance: $Data/Context/InstanceName$, Value: $Data/Context/Value$

Note that when using the script performance data provider, the ‘Performance Mapper’ tab is used to map the data returned by the script to the database. To make this generic, both the instance name and the value are used from each element:

  • Object: LogicalDisk
  • Counter: Free Megabytes
  • Instance: $Data/Property/@Name$
  • Value: $Data/Property[@Name]$

Note that the VBScript creates three typed ‘property bags’ and returns all. This results in a single XML that contains three dataitem elements, one for each instance. Creating a single property bag and adding the three instances results in only the first being processed.

This should now gather free space in MB and as a percentage of the size for all VMFS datastores your VC server knows about, visible through the performance view, and alerted on when free space is less than 10% of the capacity for each volume.

SQL Queries

Some SQL queries against the OperationsManagerDW database I used along the way to query from the datawarehouse that the data was being gathered. Note that where I've specified 'datastore%' as a filter, you'll need to change this to the prefix of your datastores, eg ds01, ds02 would be ds%.

/*Return the raw performance data collected for the datastore* instances: */

select DateAdd(Hour, 10, DateTime) as DateTime, InstanceName, SampleValue from perf.vperfRaw
inner join vPerformanceRuleInstance on perf.vperfRaw.PerformanceRuleInstanceRowID = vPerformanceRuleInstance.PerformanceRuleInstanceRowID
where InstanceName like 'datastore%'
order by datetime desc

/* Find new rules: */
select top 10 * from dbo.vPerformanceRuleInstance
where instancename like 'datastore%'
order by performanceruleinstancerowid desc

/* Find new raw performance data: */

select top 100 DateTime, SampleValue, ObjectName, CounterName, FullName, Path, Name, DisplayName, ManagedEntityDefaultName from perf.vperfraw
inner join vPerformanceRule on perf.vperfraw.PerformanceRuleInstanceRowID = vPerformanceRule.RulerowID
inner join vManagedEntity on perf.vperfraw.ManagedEntityRowID = vManagedEntity.ManagedEntityRowID
order by datetime desc

/* Find new rule instances that have been created: */

select top 10 * from dbo.vPerformanceRuleInstance
order by performanceruleinstancerowid desc

#PowerShell Script - PerfGatherVMwareDSSpace.ps1
# Note that this includes a context to connect to VC, with hardcoded username and password.  You could avoid this by running the scheduled task under the security context you created with inherent rights to VC, and then remove the explicit -credential argument to Connect-VIServer.

$ErrorActionPreference = "SilentlyContinue"
add-pssnapin VMware.VimAutomation.Core

$ADMINLOG = "c:\admin\logs"
$outputFile = ""
$today = [DateTime]::Now.ToString("yyyyMMdd")

$scriptName = $MyInvocation.MyCommand.Name
$scriptName = $scriptname.substring(0, $scriptname.LastIndexOf("."))

if ($env:adminlog -ne $null) {        # Was there an adminlog environment variable?
    $outputFile = $env:adminlog
} else {
   $outputFile = $ADMINLOG        # No, use the constant default  

$outputFile += "\" + $scriptname + "_" + $today + ".csv"    # Construct the full path to the output file 

$server = "vc01"          # VC instance
$username = "domain\user"        # Hardcoding is bad, but at least this is a RO account.
$password = "password"

$pwd = convertto-securestring $password -asplaintext -force

$cred = New-Object Management.Automation.PSCredential ($username, $pwd)   # Create the credentials to use with the VC connection

$vmserver = & { trap {continue}; Connect-VIServer -server $server -Credential $cred } # Connect to VC, trapping the error

if ($vmserver -is [System.Object]) {       # Do we have a connection?
    Get-Datastore -server $vmserver | export-csv -noTypeInformation -path $outputFile # Yes, get the datastore details and store as CSV
    if (test-path -path $outputFile) {
        write-output "Datastore details exported to $outputFile"
    } else {
        write-output "Error: Datastore details were not exported to $outputFile"
    $vmserver = $null
} else {
    write-output "Could not connect to $server"


' VBScript

' Parse a CSV file containing the VMware volume information, and return the specified field from the data file or the calculated percent free

Const TOKEN_DATE = "%date%"
Dim SOURCE_FILE : SOURCE_FILE = "c:\admin\logs\PerfGatherVMwareDSSpace_" & TOKEN_DATE & ".csv"  ' Today's log file exported from the PowerShell VI-snapin script

Const DELIMITER = ","           ' CSV data file
Const ForReading = 1
Const DISKTYPE_VMFS = "VMFS"          ' We're interested only in lines with VMFS volumes

Const QUERY_SIZE = "Size"
Const QUERY_FREE = "Free"


Const FIELD_SIZE = 1           ' The field in the CSV that contains the capacity of the VMFS volume
Const FIELD_FREE = 0           ' The field in the CSV that contains the free disk space on the VMFS volume

Const PerfDataType  = 2

Set oFSO = CreateObject("Scripting.FileSystemObject")



Sub Main()

 If WScript.Arguments.UnNamed.Count >= 1 Then
  strQueryType = WScript.Arguments.UnNamed(0)
  wscript.echo "Command-line argument passed, querying for " & strQueryType
  strQueryType = QUERY_DEFAULT
  wscript.echo "No command-line argument passed, querying for the default of " & strQueryType
 End if

 Select Case strQueryType
  Case QUERY_SIZE  strField = FIELD_SIZE 
  Case QUERY_FREE  strField = FIELD_FREE
  Case Else  strField = FIELD_FREE
 End Select

 dtmDate = Now 
 strToday = DatePart("yyyy", dtmDate)         ' YYYY
 strToday = strToday & String(2 - Len(DatePart("m", dtmDate)),"0") & DatePart("m", dtmDate) ' MM
 strToday = strToday & String(2 - Len(DatePart("d", dtmDate)),"0") & DatePart("d", dtmDate) ' DD

 strDataFile = Replace(SOURCE_FILE, TOKEN_DATE, strToday)     ' Build the path to the data file
 wscript.echo "Looking for " & strDataFile

 Dim oAPI, oBag
 Set oAPI = CreateObject("MOM.ScriptAPI")

 If oFSO.FileExists(strDataFile) Then        ' Does the data file exist for today?
  WScript.Echo "Log file found, continuing"
  Call ReadFile(strDataFile, strBuffer)       ' Yes, read the contents of the file into the buffer

  For Each strLine in Split(strBuffer, vbCRLF)      ' For each line
   arrEntry = Split(strLine, DELIMITER)      ' Split the line into an array on comma
   If UBound(arrEntry) = 5 Then       ' Did this line have a valid number of fields?
    If strComp(arrEntry(3), DISKTYPE_VMFS) = 0 Then    ' Yes, is it a VMFS volume? (excludes the header)
    wscript.echo "Processing " & strLine
     If IsNumeric(arrEntry(FIELD_FREE)) AND IsNumeric(arrEntry(FIELD_SIZE)) Then     ' Yes, is the field we're after a numeric?
      Set oBag = oAPI.CreateTypedPropertyBag(PerfDataType)  Create a typed name/value pair property bag

      If (strField = FIELD_SIZE) OR (strField = FIELD_FREE ) Then
       Call oBag.AddValue(arrEntry(5),CLng(arrEntry(strField))) ' Yes, convert to a long and add to the bag
      ElseIf strField = QUERY_PERCENT Then
       dblFreePercent = CDbl(arrEntry(FIELD_FREE) / arrEntry(FIELD_SIZE)*100)
       Call oBag.AddValue(arrEntry(5), Round(dblFreePercent, 2))
      End If
      oAPI.AddItem(oBag)     ' Add the item to the bag (doing it here results in three datatime elements)

     End If  
    End If
   End If

  WScript.Echo "Error: Daily data file not found - " & strDataFile 

 End If

 Call oAPI.ReturnItems 

End Sub    

' Purpose: Read a file and store the contents in the buffer
' Assumptions: oFSO exists
'              strLogFile contains the path/filename of the file to operate on
' Effects: strBuffer is by reference
' Inputs: strFileName, Path and filename of the source file
'   strBuffer, the buffer used to store the contents of the file
' Returns: None
Sub ReadFile (ByVal strFileName, ByRef strBuffer)   
 On Error Resume Next
 Dim objTextStream

 If Not oFSO.FileExists(strFileName) Then
  WScript.Echo "Error: " & strFileName & " file not found."
  Exit Sub
 End If
    Set objTextStream = oFSO.OpenTextFile(strFileName, ForReading)
 strBuffer = objTextStream.ReadAll
End Sub



How to Create a Probe-Based Performance Collection Rule in Operations Manager 2007

MOMScriptAPI.CreatePropertyBag Method

XPath Examples:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

No comments:

All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 13 years, and I know just about enough to realise that I don’t know very much.