Monday, December 29, 2008

Command-line automation – Echo and macros

This is the fifth and last in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the echo command and how to execute your commands using macros with doskey.


Echo is one of the most important commands to use when constructing repeatable commands – you should never run a command against multiple objects without first using echo. Some commands can be disastrous without appropriate input checking.

For example, using a wildcard incorrectly in a file spec, instead of putting *.txt, below I’ve put ‘* .txt’ – the wildcard selecting everything in the current directory (with the unfortunate del command for each of those files).
for %i in (* .txt) do echo del %i

Or more commonly – an incorrectly formed control file. The following command is one I’ve used to fix invalidly set UPNs on user accounts in the local directory. This command is quite powerful, and can be used to fix dozens or hundreds of accounts in seconds – but if the input file is not well formed, you could end up breaking each account:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix%

Incidentally, the command used to generate the ‘NoUPN.txt’ file was:
dsquery * %OU% -filter "&(objectclass=user)(objectcategory=person)(!(userprincipalname=*))" -s %server% -scope onelevel -attr name distinguishedname > NoUPN.txt

If you are untrusting of using the ‘for /f’ to execute operations against multiple objects, you can also use echo to construct a batch file that you can run at a later time – good practice rather than writing and executing the command in a single attempt. This may also useful when you’re modifying hundreds or thousands of objects and you don’t want to execute the commands immediately – for example if the modifications were in Active Directory and you wanted to prevent excessive replication during the day.

Running the command above with an appended redirect (>>) to a file, you could then pass this file to someone else for sanity checking before actual execution of the batch job:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix% >> %temp%\SetUPN.bat

Doskey macros

In the previous post there was a command to calculate a subnet at the command-line. If you had to type it every time - or even copy and paste - it’s not something you would use (okay, you’d probably never use it anyway), but doskey macros provide a simple method of allowing command re-use.

Creating a macro file

A text file can be created containing your macros, choose a shorthand name for the command, followed by the command you wish to execute. One slightly odd aspect with a macro file is that variables passed at the command-line for the macro use the $ prefix instead of %.

To make macros more useful, I often add two options – one without any parameters with a defined default, and another accepting a parameter to customise the command. For example, the following macro - called ‘AD2003’ - would by default query the local directory for all 2003 computer accounts. If a parameter is passed ($1), the dsquery command passes the parameter as the start node for the search (which could be a DN to limited the containers queried, or a different domain than domainroot).

AD2003=if _$1==_ (dsquery * domainroot -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0) else (dsquery * $1 -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0)

The find subnet command from the previous post could be shortened to execute the command with something as simple as ‘fs’, by adding the following to a macro file (and executing on a command processor with delayed expansion enabled):

FS=for /f "tokens=1-8 delims=.- " %i in ('echo $1 $2') do @set /a Octet1="%i & %m" >nul & @set /a Octet2="%j & %n" >nul & @set /a Octet3="%k & %o" >nul & @set /a Octet4="%l & %p" >nul & Echo !Octet1!.!Octet2!.!Octet3!.!Octet4!

Automatically enabling the macro file

It would also be unwieldy if you had to enable the doskey macros each time you ran a command shell. However, you can easily automate this with the autorun key of the shell, stored in either

Key - HKEY_LOCAL_MACHINE\software\microsoft\Command Processor
Value - AutoRun
Value Type - REG_SZ
Value Data - doskey /macrofile=c:\util\macros.txt (or wherever)

This concludes what hopefully was a useful description of how I use built-in shell commands to automate repetitive tasks and make my work life easier and more efficient. Some of the resultant commands are not particularly easy to follow, but each can be broken down into the component parts and digested separately.

For more real-world examples of how I use the information in the last five posts, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Saturday, December 27, 2008

Command-line automation – set

This is the fourth in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘set’ command, which I find very useful when automating repetitive tasks with ‘for’. I use set in four ways – string manipulation, simple arithmetic operations with ‘set /a’, gathering input with ‘set /p’ or concatenating output in a single for loop with delayed variable expansion.

String Manipulation

My most frequent use of the set command is simple string manipulation at the command prompt. I use this to replace one character with another or extract a substring.


Replacing one string with another is a core function of string manipulation. Even though this is included under help for ‘set’, this functionality works for other commands, including the echo command.

For example, say you have the fully qualified user DNS name – – and this conveniently is also the distinguished name of your AD domain, the following command replaces periods with ‘,DC=’ and prefixes an additional string to provide the distinguished name:
echo DC=%userdnsdomain:.=,DC=%

This would turn ‘’ into:

Wildcards can also be used, for example, to remove the first element, you could replace up to the first dot with nothing:
echo %userdnsdomain:*.=%

You could also remove spaces and print the output, for example, if the ‘test’ variable contained ‘A Sentence With Spaces’, running:
echo %test: =%

Would result in:


You can extract from existing variables (or inline if using delayed expansion), providing Left/Right/Mid functionality. This can be useful in many ways, but for most of the input I deal with, splitting with ‘for’ and delimiters (such as comma or space) makes for simpler processing with dynamic input. For examples sake, if you had a defined length of padded text, and you wanted the 10 characters starting at the 20th character (0-based), you could run:
echo %input:~19,10%

You could also extract the last 10 characters:
echo %input:~-10%

Or all but the last 10 characters:
echo %input:~0,-10%

Arithmetic operations

‘set /a’ supports a relatively simple but still powerful set of arithmetic, bitwise and shift operators, and while I’ve never found much use for them, upon occasion they have come in handy.

For example, this command will tell you what next year will be and set the NextYear environment variable, based on the current year + 1:
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1

Similarly, find the current month, increment by one, unless the current month is 12, then wrap around and start at 1 again:
for /f "tokens=2 delims=/" %i in ('echo %date%') do if %i==12 (Set NextMonth=1 & echo !NextMonth!) else (Set /a NextMonth=%i+1)

Note that the example above uses delayed environment variable expansion, see below for a description.

Incidentally, if you were going to use this output for something and if the month were a single digit you would lose the padding, which would then be incorrectly sorted (as compared to other padded output). The example below pads the month, then extracts the last two characters, effectively padding single-digit months (see string manipulation above):
set NextMonth=0%NextMonth%
echo %nextmonth:~-2%

An actual use I have found for the arithmetic operators is to calculate subnets from the command-line. I have made use of this when parsing netsh dhcp output (there were thousands of leases across several DHCP servers serving dozens of subnets, and I wanted to know from the leases which subnets were in use).

This can be done with a single command-line, but it’s a little complicated, the batch file below may be easier to demonstrate the ‘set /a’ functionality. Running this batch file with an IP and mask, eg ‘findsubnet.bat’ will return the subnet ( in this example) using bitwise AND of the two sets of octets:

@Echo Off
Set Input=%1
Set Mask=%2
If /i "%Input%"=="" Goto End

for /f "tokens=1-8 delims=.- " %%i in ('echo %Input% %Mask%') do Call :FindOctet %%i %%j %%k %%l %%m %%n %%o %%p
Goto End

::Echo %1 %2 %3 %4 %5 %6 %7 %8

set /a Octet1="%1 & %5" >nul
set /a Octet2="%2 & %6" >nul
set /a Octet3="%3 & %7" >nul
set /a Octet4="%4 & %8" >nul
Echo %1.%2.%3.%4,%Octet1%.%Octet2%.%Octet3%.%Octet4%,%5.%6.%7.%8
Set Octet1=
Set Octet2=
Set Octet3=
Set Octet4=
Goto End

Echo Invalid arguments, please provide an IP and longhand mask, eg
Goto End



Set can be used to gather input, making your commands or batch files interactive. For example, the following command assigns the input to the ‘test’ environment variable, after asking the question:
set /p test=Tell me something?

I don’t often use this, as it somewhat defeats the purpose of automation – I prefer to have all the required input in a control file, which separates the intended action from the actual operator.

Set with Delayed expansion

Delayed environment variable expansion provides a method of executing commands and storing the output for use by other commands within a loop – either in different iterations or nested commands. This is required when looping through ‘for’ commands or nesting commands, otherwise state between each command is not maintained.

For example, without delayed expansion, the following would not work:
set test=test variable & echo %test%

However, using delayed environment variable expansion, this problem can be easily overcome, using the !variable! syntax, instead of the normal %variable%. Note that you may need to enable delayed environment variable expansion, which can be done using ‘cmd /v:on’ when starting the command prompt, or running ‘setlocal ENABLEDELAYEDEXPANSION’ within an existing shell.

With delayed expansion, the first command would set the variable and the second echo the newly set variable:
set test=test variable & echo !test!

Or to re-use an example above, calculate next year and then do something with that information (echo in this case):
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1 1>nul & echo Next year is !NextYear!

Unless there is already a variable of the same name, running the commands above using the percent signed variables will result in the variable name being printed, as the environment hasn’t yet been changed by the first command.

Note that while testing, it seems the Vista command shell has been updated and as long as delayed expansion is enabled, this command would work:
set test=test variable & echo %test%

The following command is a one-line version of the batch above, using delayed variable expansion to calculate the variables as each command is executed:
for /f "tokens=1-8 delims=.- " %i in ('echo') do set /a Octet1="%i & %m" >nul & set /a Octet2="%j & %n" >nul & set /a Octet3="%k & %o" >nul & set /a Octet4="%l & %p" >nul & Echo %i.%j.%k.%l,!Octet1!.!Octet2!.!Octet3!.!Octet4!,%m.%n.%o.%p

An easier example to follow, set the environment variable called ‘number’ to a random number (using %random% - very useful), echo the newly set variable, and then use ‘set /a’ to increment the number by one:
set number=%random% & echo !number! & set /a number=!number!+1

If you are using nested ‘for’ commands, you can just reference variables from an earlier command, but in some circumstances delayed expansion is the only possibility. I have had to use this syntax when using ‘set /a’ and using the replace/substring functionality, as this requires a full variable (eg. %path%, not a temporary %i type variable.

This was a basic overview of ‘set’ and some of the ways in which I use this command when automating tasks at the command-line.

For more real-world examples of how I use ‘set’, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Thursday, December 18, 2008

Command-line automation – errorlevels and if

This is the third in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are errorlevels, and if/then/else statements, useful for branching in a single command-line to execute based on the error level set or some other condition.

Using errorlevels, concatenated commands and if/then/else statements provides enhanced functionality, and most complex operations would not be possible on a single command-line without these features.


In the context of command-line automation, I find error levels most useful when trying to cater for both sets of outcomes when executing a command – either what you wanted happened, or it didn’t.

In the previous examples, I’ve used a control file to ping one or more machines and then echoed the IP address of those that were contactable. This is only part of the picture – what about those machines in the list that didn’t respond to a ping? They are simply dropped from the output. This may be what you are after, but normally I would prefer to know the positive and the negative.

To work around this problem, the following command could be used:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do echo %i

This will ping ‘computer’, echo the IP Address if found, otherwise echo NotFound. This occurs because the find command sets the errorlevel according to whether the string was found, and then we are concatenating the command with another to check if an errorlevel occurred indicating the string was not found, and echoing an alternate output to be picked up by the body of the for loop.

A simpler way of looking at this may be without the ‘for’ loop:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)

The command is similar to the example above, it will ping ‘computer’, and if the word reply was found in the output, it will print ‘replied’, otherwise it will print ‘no reply’.

Command concatenation and if/then/else

Commands can be concatenated – when running natively or inside a ‘for’ loop – providing a simple method to group commands. For example, looping through a control file, you might want to nslookup and then ping each device:
for /f %i in (c:\temp\control.txt) do nslookup %i & ping %i

This is a simple chain of commands, execute one and then the other. However, provided each command returns a relevant errorlevel, the if/then/else syntax can also be used to execute the second only if the first succeeded.

Unfortunately nslookup does not provide an errorlevel, but the find command does:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)
ping -n 1 computer | find /i "reply" >nul && echo Replied || echo No Reply

The above commands return the same result – they each print a statement depending on whether ‘reply’ was found in the ping output, the second using the shorthand ‘if’ (&&) and ‘else’ (||) syntax.

‘If’ can be used to check the errorlevel (as in the above example), check whether one string equals another, and check whether a file/directory exists.

If ErrorLevel checking

The following examples send an invalid service a stop control, and perform various checks on the errorlevel returned. Note that sc.exe on XP does not return error codes, this was Vista which does at least return 1060 when the specified service does not exist.

Check whether errorlevel (a special variable that can either be enclosed in % or not) was exactly 1060:
sc stop invalidservice >nul & if errorlevel==1060 echo Service not found

Check whether errorlevel was exactly 1060, if so echo ‘service not found’, otherwise echo the errorlevel returned:
sc stop invalidservice >nul & if errorlevel==1060 (echo Service not found) else (echo Error code %errorlevel% returned)

Check whether the errorlevel was 1060 or greater, and not 1061 (therefore only 1060).
sc stop invalidservice >nul & if errorlevel 1060 if not errorlevel 1061 echo Service not found

Check if anything above errorsuccess (0) was returned, echoes an error occurred and then uses ‘net helpmsg’ to return the error description for the specified error code:
sc stop invalidservice >nul & if errorlevel 1 echo An error occurred: %errorlevel% & net helpmsg %errorlevel%

If string checking

If can be used to conditionally check two strings to determine the action taken. Note that if the string could be blank, you’ll need to either append/prepend another character or enclose the string in quotes.

Using the above example of pinging a computer, check the response and branch accordingly (different echoes in this case):
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do if "%i"=="reply" (echo Ping success) else (echo Ping failed)

Note that ‘if /i’ can be used for a case insensitive comparison.

Additionally, if command extensions are enabled, and the strings in comparison are numeric, both are converted to numbers and additional operators are available for the comparison, for example:
If 9 equ 9 echo Equals
If 5 lss 14 echo Less than

If exist

To determine whether a file or directory exists, the ‘if exist’ syntax can be used. This can be useful when checking flag files or control files exist to avoid encountering unexpected errors.

If exist %windir% echo %windir% exists
If not exist c:\temp\control1.txt (echo Control File does not exist, terminating & break) else (echo continuing)

Some other examples of concatenating commands which help to demonstrate the concepts:

Use ‘sc.exe’ to send a service a stop control, if the service was running (exists, security allows etc) and was sent a stop, sc will return errorsuccess (0), pause for 5 seconds and then restart the service, otherwise echo that the service was not running in the first place:
sc stop "service" && (echo error %errorlevel% & sleep 5 & sc start "service") || echo Service was not running, error %errorlevel%

nslookup, find the second address (the first in nslookup is the DNS server address, and echo that address:
for /f "skip=1 tokens=2" %i in ('"nslookup 2>nul | find /i "address""') do echo %i

nslookup, skip the first DNS server address (the second line of the nslookup output – find /n), return the other address if found, otherwise return ‘Not-Found’ in the %i variable – the second token of the string:
for /f "tokens=2" %i in ('"nslookup 2>nul | find /i /n "address" | find /i /v "[2]" & if errorlevel 1 echo Address Not-Found"') do echo %i

Create a file, if the file is found type the contents, and then delete the file if the type succeeded, otherwise report that the ‘type’ command failed:
echo test > %temp%\test.txt & if exist %temp%\test.txt type %temp%\test.txt && del %temp%\test.txt || Echo could not type %temp%\test.txt

This was a basic overview of the errorlevels and if/then/else branching, future posts will build on this with other useful commands and nested ‘for’ loops.

For more real-world examples of how I use errorlevels and if/then/else, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Wednesday, December 17, 2008

Command-line automation – find and findstr

This is the second in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are the ‘find’ and ‘findstr’ commands - unless you want to spend all your time manually creating or filtering control files you will often use these commands when constructing repeatable command-lines.

These two commands are crucial in efficient ‘for’ loop processing, invaluable when filtering a control file or the results of an in-line command. Help on each is available from the command prompt with the ‘/?’ argument.


I usually use find because more often than not I want to filter a string literal – a specific word or combination of characters – from an input string. As often as not, I also use ‘find /v’, returning the lines that don’t match the specified string.

For example, if you have a control file listing servers in several geographic locations, but you are only interested in processing servers in ‘bne’ (case insensitive with ‘/i’), you could run:
find /i "bne" c:\temp\control.txt

Combined with a ‘for’ command, this could be:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do echo %i

Note that this line uses the following syntax explained in the previous post on ‘for’:

  1. ‘Skip=2’ tells the loop to skip the first two lines – the result header of the ‘find’ command
  2. Inside the brackets I’ve specified to run a command and parse the results, wrapping the command in single-quotes.

The results are that for each server containing the word ‘bne’, the body of the ‘for’ statement will be executed – in this case simply to echo the first token.

You might also want to run the commands against all servers except those in bne, which is the above command repeated with the addition of the ‘/v’ switch to tell ‘find’ to return those lines not containing a match:
for /f "skip=1" %i in ('find /i /v "bne" c:\temp\control.txt') do echo %i

Filtering in-line output with find

Filtering output of a command executed within a ‘for’ loop is syntactically a little more complicated, but makes sense once you get the hang of it. For example, suppose you want to filter the output of a ping command, returning the IP address of those hosts that replied to a ping, you could run:
for /f "tokens=3" %i in ('"ping -n 1 computer find /i "reply""') do @echo %i

This will:

  • Execute the ping command, pinging the host named computer, filtering the output to find the word ‘reply’ (indicating a successful reply). Note that to use the pipe inside the brackets, the whole string needs to be enclosed in double-quotes – inside the single-quotes.
  • Using spaces/tabs, choose the third token to return in the first variable - %i. This will be the IP address in the standard ping output returned by XP and Vista.

Unfortunately the third token of the ping output includes a irrelevant colon, so to strip that, we could specify colons and spaces as delimiters in the for loop:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer find /i "reply""') do echo %i

This will be better explained in a future post, but you can also nest two ‘for’ commands and for each host in the control file (in bne in this example), ping and return the IP address if there is a reply:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do @for /f "tokens=3 delims=: " %m in ('"ping -n 1 %i find /i "reply""') do @echo %i,%m

The output would give you a comma-separated list of server names and IP address of those from the original control file that currently respond to a ping. Theoretically not all that useful with servers, as they’d usually be known IPs and always on, but if you’re wanting to check something on hundreds or thousands of workstations this would give you a point-in-time control file of online machines (assuming name resolution is accurate).

It’s also worth noting that ‘find’ commands can be cascaded – piping the output of one find command to another if you need to further refine the output. For example to find servers in ‘bne’ and then servers in ‘bne’ that also contain ‘vm’:
find /i /c "bne" c:\temp\control.txt find /i "vm"

The possibilities are nearly endless, I use ‘find’ to:

  • Filter in and out items that I want, ‘find /i’ and ‘find /i /v’ from generic control files that I use for multiple operations, either in-line in a for command, or to create a stand-alone filtered control file
  • Run commands and then filter out useless or useful information, often headers/footers or other extraneous detail, making it possible to list data, filter that data and run a command against each filtered item in a single command.
  • Use errorlevels to determine whether the find command was successful or not (explained in a future post). Often the only way of knowing whether a command returned what you were after is the presence or absence of a string – and ‘find’ will tell you this based on the errorlevel (an advantage over findstr which in XP at least does not set the errorlevel).

Note that find also has several other features that are useful in some scenarios:

  • By default, offline files will be skipped – if you use a file archiving product that sets the offline (‘O’) attribute, these files will not be searched unless you specify the /offline parameter
  • The ‘find /n’ option will return the line number the match was found, occasionally useful when parsing large or sequenced control files and you want to know where in a file the string was found.
  • The ‘find /c’ option will return a count of the matches found, instead of the actual matching lines. This can often be useful when determining how many records would be processed. For example, in the example above you could first run the following command to determine how many records would be processed with the for loop:
    find /i /c "bne" c:\temp\control.txt


These same principles can be used with findstr to filter output – findstr provides advanced searching, with the following benefits in my use of the commands, you can:

  • Search for more than one string in a single command. This simplifies searches, in the example above, you could search for ‘bne’ and ‘syd’ in a single findstr command.
  • Use basic regular expressions and/or string literals

In the example above with a control file containing a list of servers with geographic names (or descriptions somewhere on the line), you could run the following command to return servers in both syd and bne and then run the subsequent command against each server:
for /f %i in ('findstr /i /c:bne /c:syd c:\temp\control.txt') do echo %i

Regular expressions are used to group similar items, without expressly providing each possible combination. For example, parsing robocopy log output, to search for new directories that have been created with a regular expression, you could run:
findstr /i /n "^.*New.Dir" Robocopy.log"

You could just use ‘find’ and search for the string ‘new dir’, but this could easily return false positives if a file or directory has the string ‘new dir’ in it. Using the regular expression looking for the start of the line followed by whitespace followed by the ‘New Dir’ is much more precise.

This was a basic overview of the ‘find’ and ‘findstr’ commands, future posts will build on this foundation with other useful commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘find’ and ‘findstr’ commands, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Tuesday, December 16, 2008

Building blocks of command-line automation – FOR

This is the first in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used for data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘for’ command, the most important component in command-line automation. This command provides several methods of looping through a list, and running a command against each element in that list. Use the ‘for /?’ to get Microsoft help on this command.

Using the ‘for’ command with one of the syntaxes below provides many benefits, including:

  • Repeatability – If you save the command you run, it can be re-run several or hundreds of times
  • Change control and testing – It’s easy to record what you are planning and simple to run the same commands in a test-lab environment. The output of commands can also be redirected to file, making accountability much easier. Using control files also provides a straightforward method of recording the targets of various actions, knowing that you have one master list, and do not risk accidentally missing an entry.
  • Documentation – Implementation using a series of commands very easily translates to an as-built document – with the added benefit of providing a quicker DR process.
  • Efficiency – Even though designing the command for the first run may not be as quick as using the GUI, every time after that will usually be much quicker, and previous commands can often be quickly adapted to new tasks.

You can use the ‘for’ command to:

Process a set of filenames or string literals

The filenames or literals can either be directly named in a space-separated set, or you can use wildcards to process more than one file. For example:
for %i in (*.txt) do echo %i
for %i in (test1.txt test2.txt) do echo %i

For example, I would use this syntax if I’m trying to:

Quickly execute something against a group of machines, eg ping each machine:
for %i in (server1 server2 server3 server4) do ping -n 1 %i

Process a series of data files that I have created from another process, eg this uses the regview utility to export the registry entries modified by *.pol files into *.txt:
for %i in (*.pol) do regview %i > %i.txt

Quickly execute the same command with a different variable, eg use the setprinter utility to view all levels of configuration for the specified printer (you could also use for /l in this example):
for %i in (0 1 2 3 4 5 6 7 8 9) do setprinter -show \\server\printer %i

Process a set of directories

The directory names can either be directly named in a space-separated set, or you can use wildcards to process more than one directory. For example:
for /d %i in (%windir%\*) do echo %i
for /d %i in (c:\temp c:\windows) do echo %i

I would use this syntax if I’m trying to do something with each top-level directory, for example:

Report or set ACLs:
for /d %i in (%rootDir%\*) do icacls %i /save %~ni.txt

Rename all the top-level directories to start with a new prefix:
for /d %i in (%rootDir%\*) do ren %i New-%~ni

Process the contents of a text file, line by line

The contents of a file – which I usually refer to as a control file – can be read line-by-line and your command would be run once for each line, substituting tokens from the control file. This provides unlimited capability – construct a control file through any means available and you can then process the entries one-by-one and run a command against that entry.

Note that in Vista at least, just a LF is enough to separate the lines, rather than the Windows standard CR+LF.

For example, assuming you have a control file with a list of servers or workstations, you could:

Lookup the IP address of each machine:
for /f %i in (test.txt) do nslookup %i

Ping each machine:
for /f %i in (test.txt) do ping %i

Remote dir of each machine:
for /f %i in (test.txt) do dir \\%i\c$

I use this constantly to run a command against multiple AD objects, machines, printers, or other network devices, whether the command queries or checks something, or makes a change to each device.

Process the results of a command, line by line

The results of almost any command can be used as the looping mechanism for a ‘for /f’ command, providing an in-memory control file. For example, you could:

Find the local hostname and then nslookup the computer (you could also use %computername% for this):
for /f %i in ('hostname') do nslookup %i

Query the local Active Directory for a list of DCs (server records) and lookup the IP of each DC:
for /f %i in ('dsquery server -o rdn') do nslookup %i

Recursively enumerate a path

It’s possible to recursively enumerate files or directories from a specified starting location, passing each to the body of the for loop. This provides a rudimentary search and response facility, allowing you to traverse a tree looking for objects of a particular type – and then execute something for each found.

For example, you could search from the root of C: for *.txt files, and then report the filename and size (you would just use dir /s if all you wanted to do was echo)
for /r c:\ %i in (*.txt) do echo %i %~zi

Step through a sequence of numbers and execute for each

The ‘for /l’ option allows stepping through a sequence of numbers, passing the number as a parameter to the body of the ‘for’ loop.

I don’t use this method very often, but it would be another method to the setprinter command above:
for /l %i in (1,1,9) do setprinter -show \\server\printer %i

Variable Expansion

When using ‘for’, ‘for /f’ and ‘for /d’ variable references can also be modified to return substrings or additional information. Note that when using ‘for /f’, most of these only make sense if you are processing lists of files or directories, but if you did have a control file with files/paths variable expansion does work as expected.

This substitution can be very useful, particularly when constructing parameters to pass to the command in the body of the ‘for’ loop. For Example:

If you had a number of control files that you wanted to process, outputting the results to a same-named log file:
for %i in (c:\temp\*.txt) do echo Do something with %i > %~ni.log

If the output of a previous command wrapped the results in quotes, but you need to append/prepend something else you can easily remove surrounding quotes:
for /f %i in ('echo "c:\windows"') do echo %~i\temp

Given a list of files, echo those that are zero bytes in size:
for %i in (c:\temp\*.txt) do @if %~zi == 0 @echo %i

Given a full path, split into drive, path and filename:
for %i in (c:\windows\temp\test.txt) do echo %~di %~pi %~nxi

Tokens, delimiters and skipping lines

The simple functionality of the ‘for /f’ command can be extended very easily with three options:

  1. Tokens – By default only the first token is returned in the variable specified. You can change this behaviour to return one or more tokens, eg tokens=2,3,5 or tokens=1-3 would populate %i, %j and %k with the respective tokens
  2. Delimiters – Instead of the normal space and tab delimiters, one or more alternate characters can be specified. For example, you can specify a comma as a delimiter to process as CSV file
  3. Skipping lines – the skip command can be used to skip one or more lines from the start of a control file, useful when trying to skip a header line, or bypass logo information in a command result.

These options can be used individually or as a combination, for example:

Skip the first line of the control file:
for /f "skip=1" %i in (test.txt) do echo %i

Skip the first line, and use comma’s as the delimiter:
for /f "skip=1 delims=," %i in (test.txt) do echo %i

Skip the first two lines, use the second token, separated by comma and space:
for /f "skip=2 tokens=2 delims=, " %i in (test.txt) do echo %i

This was a basic overview of the ‘for’ command, future posts will build on this foundation with multiple commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘for’ command, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Thursday, December 11, 2008

Useful PowerShell command-line operations

The commands below are a small but growing list of powershell one-liners I've used (and when I remember to take note). Note that this may not be the best and almost certainly isn't the only way in most cases.

Each command-line can be copied and pasted at a PowerShell command prompt, or you can use the commands as part of a PS1 script file if you prefer.

Show help on commands such as 'if', regular expressions etc
help about*

Show help about about_Automatic_variables such as $_ $DebugPreference $OFS
help about_Automatic_variables

Run a command and pipe the results to a variable
cscript.exe //nologo script.wsf | Set-Variable -name scriptResults

Run a command and set a variable with the output
$scriptResults = cscript.exe //nologo script.wsf

Run a command and set a variable with the output using the call operator
$scriptResults = & cscript.exe //nologo script.wsf

Filter an object based on an array of strings
$array = "a", "b"; write-output a b c d | select-string -pattern $array -simpleMatch

Reformat today's date to YYYYMMDD
$today = [DateTime]::Now.ToString("yyyyMMdd")

Find local disks except C: using powershell and output to CSV
Get-WmiObject -Namespace root\cimv2 -ComputerName %server% -Query "SELECT * from Win32_LogicalDisk WHERE FileSystem='NTFS' AND Description = 'Local Fixed Disk' AND Name != 'C:'" | export-csv c:\disk.csv

Use the VI Toolkit Powershell snap-in to query for snapshots
Get-VM | Get-Snapshot | export-csv -path c:\temp\VMsnapshots.csv

Use the VI Toolkit Powershell snap-in to query for snapshot information
Get-VM | Get-Snapshot | foreach-object {$out= $_.VM.Name + "," + $_.Name + "," + $_.Description + "," + $_.PowerState; $out}

Start a remote process using Powershell/WMI
$computer = "."; ([WMICLASS]"\\$computer\root\CIMv2:win32_process").Create("notepad.exe")

Find remote processes and the command-line parameters with PowerShell
Get-WmiObject win32_process | Format-Table ExecutablePath,Caption,CommandLine,CreationDate,WorkingSetSize,ProcessId

Generate a password 10 characters in length, at least 5 punctuation marks
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Web") ; [System.Web.Security.Membership]::GeneratePassword(10,5)

Format a date to XML XSD:dateTime
$Now = [DateTime]::Now.ToString("yyyy-MM-ddTHH:mm:ss")

Check if the given variable is an object
if ($variable -is [object]) {}

Check which network connections (drive mappings) a 2003 computer has
get-WMIObject -computerName "$env:computername" -class win32_logicaldisk -property ProviderName, DeviceID -filter "DriveType=4" | Format-List -property ProviderName, DeviceID

See whether an IP can be resolved to a host name

See whether a host name can be resolve to an IP
$ip = & { trap {continue}; []::GetHostAddresses($computer) }

View the datastore information for the available storage
get-vc -server $server ; Get-Datastore

Check if a file or path exists or not
test-path -path $path ; if (!(test-path -path $path)) {write-output "$path does not exist"}

Use psexec to run a powershell command remotely and report to stdout (eg query 'remote' event logs in this example)
psexec \\machine /s cmd /c "echo. | powershell . get-eventlog -newest 5 -logname application | select message"

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

MSCS 2003 Cluster Virtual Server Components

This post provides my interpretation of a simple MSCS 2003 virtual server with a file share, including how the cluster interacts with the OS and network services to provide access to the share. This follows on from the last post on low-level detail of file access in an attempt to provide a clearer picture of these often taken-for-granted components.

Note that this is only my opinion, based on less-than complete knowledge and more than likely contains semantic errors if nothing else.

File & Print Cluster Native x64 (EM64T/AMD64)

  1. Cluster Service. Includes Checkpoint Manager, Database Manager, Event Log Replication Manager, Failover Manager, Global Update Manager, Log Manager, Membership Manager, Node Manager.
    1. Operating System Interaction with the LanManServer Service which advertised shares.
    2. NetBIOS registration of the virtual server name through existing network services
    3. DNS registration of the virtual server name through existing network services
    4. Kerberos SPNs registered against an AD computer account through Active Directory
  2. Resource Monitor. Spawned child process of the cluster service, separate resource monitors can exist for resource DLLs
  3. ClusRes.dll Physical Disk <-> IsAlive/LooksAlive SCSI reservation and directory access. LooksAlive issues a SCSI reservation every 3 seconds through ClusDisk.sys against all managed disks. IsAlive performs a ‘dir’ equivalent
  4. ClusRes.dll Network Name <-> IsAlive/LooksAlive check on NetBT/DNS registration. LooksAlive relies on MSCS NIC failure detection. IsAlive queries local TCP/IP stack for virtual IP and the NetBT driver if NetBIOS is enabled
  5. ClusRes.dll IP Address <-> IsAlive/LooksAlive check on cluster NIC. LooksAlive queries NetBT driver and every 24 hours issues a dynamic DNS host record registration. If ‘DNS is required’ resource will fail if DNS registration fails. Same test for IsAlive
  6. ClusRes.dll Resource DLL File Shares <-> IsAlive/LooksAlive check on file share visibility. LooksAlive queries lanmanserver service for the share name. IsAlive does the same, and if unlimited users, the first file on the share is copied
  7. 32-bit Resource Monitor WOW64, Eg. Enterprise Vault Cluster application. Third-party cluster resources, eg Enterprise Vault, which in this case notifies the local FSA placeholder service on each physical node of virtual server changes
  8. ABE enabled by generic cluster resource. Access based enumeration with a generic cluster application running abecmd.exe during virtual server/share creation. Uses 32-bit cluster resource monitor with WOW64, setting SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM (0x0800) flag set on the otherwise standard share.

Pretty picture view:


Server side processes for simple file access

Access Based Enumeration

SHARE_INFO_1005 Structure

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Friday, November 7, 2008

Server-side process for simple file access

This post provides my interpretation of what happens at a low level when a user on a workstation tries to access a file on a server - in this case a Windows server 2003 x64 MSCS cluster. I was trying to demonstrate the complexity of what seems like such a simple action, and in particular trying to incorporate the cluster network/disk elements and highlighting the WOW64 side of things when you are running x64 with x86 third-party software (such as archiving, quotas etc).

Note that I'm reasonably confident this isn't correct because my understanding if lacking, combined with lack of time/information, but even if it were accurate in content, the step-by-step / flowchart views aren't the best way to represent these multi-layered processes. But anyway, somebody else might also think it interesting (nerds!)

  1. Workstation redirector file access to a virtual node in the cluster. Includes DNS/NetBIOS calls to determine the cluster virtual server IP address
  2. LanManServer service. User-mode LAN Manager 2.x server service, providing file and print sharing, and with 2003 SP1, Access Based Enumeration
  3. NDIS layer. Network Driver Interface Specification, Hardware interrupts passing frames to the NIC driver, then passed to the bound transport driver. send and receive raw packets, includes LLC
  4. TDI Layer. Single interface for upper-level clients to access transport providers, such as TCP/IP
  5. ClusNet.sys Driver. Cluster specific driver interpreting and routing intracluster traffic and determining communication failure
  6. Srv.sys Server Service. SMB file server driver, kernel-mode companion to the LanManServer service
  7. I/O Manager. I/O support routines, I/O Request Packets (IRPs) and Fast I/O interfaces
  8. FSD Manager. File System Driver Manager, loads FSDs and legacy file system filters, interacts with the file system cache manager.
  9. Filter Manager. A file system filter that provides a standard interface for minifilters, managing altitudes and connectivity to the file system stack. Interacts with the file system cache manager, and in the case of x86 filters, an instance of WOW64 controls the runspace for the filters (on an x64 platform)
  10. File System Cache Manager. A file system filter, working with kernel memory-mapped and paging routines to provide file level caching
  11. ntfs.sys File System FSD. Windows NT File System driver, creates a Control Device Object (CDO) and Volume Device Object (VDO)
  12. ClusDisk.sys upper-level storage filter driver. Cluster specific storage filter driver maintaining exclusive access to cluster disks using SCSI reserve commands
  13. Volume Snapshot Volsnap.sys. Manages software snapshots through a standard storage filter
  14. Volume Manager ftDisk.sys. Presents volumes and manages I/O for basic and dynamic disk configurations
  15. Partition Manager Partmgr.sys. Filter driver that sits on top of the disk driver, creates partition devices, notifies volume manager and exposes IOCTLs
  16. Class Driver disk.sys. Presents a standard disk interface to the storage stack, creating a Function Device Object (FDO)
  17. Storage Port Driver - Storport. Assists with PnP and power functionality, providing a Physical Device Object (PDO) for the device->bus connection
  18. Miniport Driver. Interface to the storage adapter’s hardware, combining with the storport driver to create the storage stack
  19. SMB response to the client. SMB response to the redirector on the workstation requesting the file
Pretty picture view:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Sunday, November 2, 2008

OpsMgr 2007 performance script - VMware datastores

This post provides a method of collecting VirtualCenter datastore information for ESX hosts in one or more clusters as performance data in Operations Manager 2007. You can then trend datastore usage for longer term analysis, and alert on the data for proactive problem management. This is my first attempt at using script to gather performance data in OpsMgr 2007, and my lack of OpsMgr knowledge combined with the limited documentation/examples made this difficult and possibly harder than it needs to be.


In OpsMgr 2007, I thought it would be useful to report on LUN disk space on the ESX side of things in a similar fashion to agent-based Windows disk performance counters, and while this sounds quite easy, I couldn’t find a simple method because:

  1. ESX doesn’t seem to support an SNMP trap to report on VMFS volume capacity/free space. You could write a shell script and add a cron job on the Linux ESX service console, but this would be targeted at each ESX server, and the results would then be duplicated (and wouldn't work with ESX 3i)
  2. VirtualCenter doesn’t appear to support a custom alert based on storage usage, only storage transfer rates.
  3. The ESX servers are only in OpsMgr as network devices, not true agents, so we can’t use WBEM or any other method to query directly. If the upcoming OpsMgr 2007 cross platform extensions work on ESX, this may be easier, although as with option 1 this would still return duplicate datastore information for each ESX server in a cluster and wouldn't work with ESX 3i)

This left a few options to gather the data in OpsMgr:

  1. Query the VMFS storage through the VI snap-in and a PowerShell Script.
  2. Query the VC database directly for this information from a VBS or PowerShell script.
  3. Create a DTS job on the VC SQL server to extract the data to a suitable format.

Options two and three above would not be supported by VMware and may change as the database structure changes between VC versions.


Querying the VMFS storage from PowerShell and collect the information with VBScript, which includes:

  1. The PowerShell script using the VI snap-in to extract the datastore information
  2. The VBScript to gather the information as performance data in Operations Manager 2007
  3. A trigger to execute the powershell script and put the data in a file to be gathered.
  4. Rules in OpsMgr to gather the data using the type 'Script (Performance)'.

Steps taken:

  1. Install the VMware VI PowerShell snap-in to your Virtual Centre box (you may need to install PowerShell first). VMware-Vim4PS-1.0.0-113525.exe is the binary I used.
  2. Create an AD group and user for rights to VC. Add the user to the group.
  3. Set rights in VirtualCenter to allow the group read-only rights at the root. This should be more granular if possible.
  4. Add a recurring trigger on your VC box to run the PowerShell script to extract the information. Eg, this could be a scheduled task that runs a batch file or directly executes:
    1. powershell . ".\PerfGatherVMwareDSSpace.ps1"
  5. Create two rules in Operations Manager, Script (Performance) to gather the daily log file and store in the database as performance data. One rule is for the free space, another is for the free space as a percentage of the capacity. See the performance mapper notes below.
    1. The rules were created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server (where the powershell script runs). You may want to target the rule differently, depending on your environment.
  6. Create a performance view, showing performance data collected by the two rules above.
  7. Create a monitor, using the LogicalDisk ‘% Disk Free’ object and counter to monitor on a threshold of 10%. Similarly, the monitor could be created under ‘Windows Server 2003 Computer’, not enabled by default and overridden for the VC server. The alert contains the following information:
    1. Object: $Data/Context/ObjectName$, Counter: $Data/Context/CounterName$, Instance: $Data/Context/InstanceName$, Value: $Data/Context/Value$

Note that when using the script performance data provider, the ‘Performance Mapper’ tab is used to map the data returned by the script to the database. To make this generic, both the instance name and the value are used from each element:

  • Object: LogicalDisk
  • Counter: Free Megabytes
  • Instance: $Data/Property/@Name$
  • Value: $Data/Property[@Name]$

Note that the VBScript creates three typed ‘property bags’ and returns all. This results in a single XML that contains three dataitem elements, one for each instance. Creating a single property bag and adding the three instances results in only the first being processed.

This should now gather free space in MB and as a percentage of the size for all VMFS datastores your VC server knows about, visible through the performance view, and alerted on when free space is less than 10% of the capacity for each volume.

SQL Queries

Some SQL queries against the OperationsManagerDW database I used along the way to query from the datawarehouse that the data was being gathered. Note that where I've specified 'datastore%' as a filter, you'll need to change this to the prefix of your datastores, eg ds01, ds02 would be ds%.

/*Return the raw performance data collected for the datastore* instances: */

select DateAdd(Hour, 10, DateTime) as DateTime, InstanceName, SampleValue from perf.vperfRaw
inner join vPerformanceRuleInstance on perf.vperfRaw.PerformanceRuleInstanceRowID = vPerformanceRuleInstance.PerformanceRuleInstanceRowID
where InstanceName like 'datastore%'
order by datetime desc

/* Find new rules: */
select top 10 * from dbo.vPerformanceRuleInstance
where instancename like 'datastore%'
order by performanceruleinstancerowid desc

/* Find new raw performance data: */

select top 100 DateTime, SampleValue, ObjectName, CounterName, FullName, Path, Name, DisplayName, ManagedEntityDefaultName from perf.vperfraw
inner join vPerformanceRule on perf.vperfraw.PerformanceRuleInstanceRowID = vPerformanceRule.RulerowID
inner join vManagedEntity on perf.vperfraw.ManagedEntityRowID = vManagedEntity.ManagedEntityRowID
order by datetime desc

/* Find new rule instances that have been created: */

select top 10 * from dbo.vPerformanceRuleInstance
order by performanceruleinstancerowid desc

#PowerShell Script - PerfGatherVMwareDSSpace.ps1
# Note that this includes a context to connect to VC, with hardcoded username and password.  You could avoid this by running the scheduled task under the security context you created with inherent rights to VC, and then remove the explicit -credential argument to Connect-VIServer.

$ErrorActionPreference = "SilentlyContinue"
add-pssnapin VMware.VimAutomation.Core

$ADMINLOG = "c:\admin\logs"
$outputFile = ""
$today = [DateTime]::Now.ToString("yyyyMMdd")

$scriptName = $MyInvocation.MyCommand.Name
$scriptName = $scriptname.substring(0, $scriptname.LastIndexOf("."))

if ($env:adminlog -ne $null) {        # Was there an adminlog environment variable?
    $outputFile = $env:adminlog
} else {
   $outputFile = $ADMINLOG        # No, use the constant default  

$outputFile += "\" + $scriptname + "_" + $today + ".csv"    # Construct the full path to the output file 

$server = "vc01"          # VC instance
$username = "domain\user"        # Hardcoding is bad, but at least this is a RO account.
$password = "password"

$pwd = convertto-securestring $password -asplaintext -force

$cred = New-Object Management.Automation.PSCredential ($username, $pwd)   # Create the credentials to use with the VC connection

$vmserver = & { trap {continue}; Connect-VIServer -server $server -Credential $cred } # Connect to VC, trapping the error

if ($vmserver -is [System.Object]) {       # Do we have a connection?
    Get-Datastore -server $vmserver | export-csv -noTypeInformation -path $outputFile # Yes, get the datastore details and store as CSV
    if (test-path -path $outputFile) {
        write-output "Datastore details exported to $outputFile"
    } else {
        write-output "Error: Datastore details were not exported to $outputFile"
    $vmserver = $null
} else {
    write-output "Could not connect to $server"


' VBScript

' Parse a CSV file containing the VMware volume information, and return the specified field from the data file or the calculated percent free

Const TOKEN_DATE = "%date%"
Dim SOURCE_FILE : SOURCE_FILE = "c:\admin\logs\PerfGatherVMwareDSSpace_" & TOKEN_DATE & ".csv"  ' Today's log file exported from the PowerShell VI-snapin script

Const DELIMITER = ","           ' CSV data file
Const ForReading = 1
Const DISKTYPE_VMFS = "VMFS"          ' We're interested only in lines with VMFS volumes

Const QUERY_SIZE = "Size"
Const QUERY_FREE = "Free"


Const FIELD_SIZE = 1           ' The field in the CSV that contains the capacity of the VMFS volume
Const FIELD_FREE = 0           ' The field in the CSV that contains the free disk space on the VMFS volume

Const PerfDataType  = 2

Set oFSO = CreateObject("Scripting.FileSystemObject")



Sub Main()

 If WScript.Arguments.UnNamed.Count >= 1 Then
  strQueryType = WScript.Arguments.UnNamed(0)
  wscript.echo "Command-line argument passed, querying for " & strQueryType
  strQueryType = QUERY_DEFAULT
  wscript.echo "No command-line argument passed, querying for the default of " & strQueryType
 End if

 Select Case strQueryType
  Case QUERY_SIZE  strField = FIELD_SIZE 
  Case QUERY_FREE  strField = FIELD_FREE
  Case Else  strField = FIELD_FREE
 End Select

 dtmDate = Now 
 strToday = DatePart("yyyy", dtmDate)         ' YYYY
 strToday = strToday & String(2 - Len(DatePart("m", dtmDate)),"0") & DatePart("m", dtmDate) ' MM
 strToday = strToday & String(2 - Len(DatePart("d", dtmDate)),"0") & DatePart("d", dtmDate) ' DD

 strDataFile = Replace(SOURCE_FILE, TOKEN_DATE, strToday)     ' Build the path to the data file
 wscript.echo "Looking for " & strDataFile

 Dim oAPI, oBag
 Set oAPI = CreateObject("MOM.ScriptAPI")

 If oFSO.FileExists(strDataFile) Then        ' Does the data file exist for today?
  WScript.Echo "Log file found, continuing"
  Call ReadFile(strDataFile, strBuffer)       ' Yes, read the contents of the file into the buffer

  For Each strLine in Split(strBuffer, vbCRLF)      ' For each line
   arrEntry = Split(strLine, DELIMITER)      ' Split the line into an array on comma
   If UBound(arrEntry) = 5 Then       ' Did this line have a valid number of fields?
    If strComp(arrEntry(3), DISKTYPE_VMFS) = 0 Then    ' Yes, is it a VMFS volume? (excludes the header)
    wscript.echo "Processing " & strLine
     If IsNumeric(arrEntry(FIELD_FREE)) AND IsNumeric(arrEntry(FIELD_SIZE)) Then     ' Yes, is the field we're after a numeric?
      Set oBag = oAPI.CreateTypedPropertyBag(PerfDataType)  Create a typed name/value pair property bag

      If (strField = FIELD_SIZE) OR (strField = FIELD_FREE ) Then
       Call oBag.AddValue(arrEntry(5),CLng(arrEntry(strField))) ' Yes, convert to a long and add to the bag
      ElseIf strField = QUERY_PERCENT Then
       dblFreePercent = CDbl(arrEntry(FIELD_FREE) / arrEntry(FIELD_SIZE)*100)
       Call oBag.AddValue(arrEntry(5), Round(dblFreePercent, 2))
      End If
      oAPI.AddItem(oBag)     ' Add the item to the bag (doing it here results in three datatime elements)

     End If  
    End If
   End If

  WScript.Echo "Error: Daily data file not found - " & strDataFile 

 End If

 Call oAPI.ReturnItems 

End Sub    

' Purpose: Read a file and store the contents in the buffer
' Assumptions: oFSO exists
'              strLogFile contains the path/filename of the file to operate on
' Effects: strBuffer is by reference
' Inputs: strFileName, Path and filename of the source file
'   strBuffer, the buffer used to store the contents of the file
' Returns: None
Sub ReadFile (ByVal strFileName, ByRef strBuffer)   
 On Error Resume Next
 Dim objTextStream

 If Not oFSO.FileExists(strFileName) Then
  WScript.Echo "Error: " & strFileName & " file not found."
  Exit Sub
 End If
    Set objTextStream = oFSO.OpenTextFile(strFileName, ForReading)
 strBuffer = objTextStream.ReadAll
End Sub



How to Create a Probe-Based Performance Collection Rule in Operations Manager 2007

MOMScriptAPI.CreatePropertyBag Method

XPath Examples:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Saturday, November 1, 2008

Enumerating URLs in Internet Explorer

Have you ever wanted to get a list of the URLs you are currently browsing in Internet Explorer? Probably not, but occasionally I've got a large number of IE windows open and I realise I need to close them all, but I would like to know the pages I'm in the middle of browsing.

The PowerShell and VBScripts below do just that - enumerate the current iexplore.exe windows, and report the URL and name of any identified iexplore.exe windows.

This is also good for gathering references, if you've got many pages open and you'd like to record the URLs for later reference, I find this script easier than repeated copy/paste.

# PowerShell

$shell = new-object –com Shell.Application

$windows = $shell.Windows()

write-output ($windows.count.ToString() + " windows found")
foreach ($window in $windows) {
  if ($window.FullName -like "*iexplore*") {
    write-output ($window.LocationURL + ", " + $window.LocationName)

$shell = $null

' VBScript
' Find the URLs of the currently running Internext Explorer Windows

' References:

Const IE_EXE = "iexplore.exe"

Call FindCurrentURLs(strURLSet)
WScript.Echo strURLSet


Function FindCurrentURLs(ByRef strURLSet)
 Dim objShell, objWindowSet, objWindow
 Dim strwindowName, strURL, strFullName

 Set objShell = CreateObject("Shell.Application")    ' Create a Windows shell automation object
 Set objWindowSet = objShell.Windows      ' Get the collection of open windows belonging to the shell

 Wscript.Echo "Processing " & objWindowSet.Count & " windows"   ' Report how many instances were found

 For Each objWindow in objWindowSet      ' For each InternetExplorer object in the ShellWindows set
  strFullName = objWindow.FullName     ' Get the full path and executable of this window
  If InStr(1, strFullName, IE_EXE, 1) <> 0 Then    ' Is this an IE shell object?
   strURL = objWindow.LocationURL     ' Get the URL

   If strURL <> "" Then 
    strURLSet = strURLSet & vbCRLF & strURL   ' Append to the set of URLs
   End If
  Else         ' No, probably explorer.exe skip
   WScript.Echo "Skipped " & strFullName & " - not IE"
  End If

 If Len(strURLSet) >= Len(vbCRLF) Then strURLSet = Right(strURLSet, Len(strURLSet) - Len(vbCRLF)) ' Strip the leading vbCRLF
 Set objShell = Nothing
 Set objWindowSet = Nothing : Set objWindow = Nothing
End Function

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Sunday, October 26, 2008

NTLM Trusts between 2003 and NT4

This post includes a few notes and example on an NTLM trust between two domains in a scenario I was working on - creating a trust between a 2003 user domin and an NT4 resource domain.

I realise NT4 was a long time ago, but NT 4.0 was one end of a trust I set up earlier this year (2008), so other people might be in a similar situation. Note that to have a forest trust - even if it's only a one way trust - both the domain and the forest have to be in Windows 2003 functional mode, so whenever you're talking about trusts between external domains/forests, if one side is NT4 or 2000 then it's an NTLM trust.

The types of access through the trust that will need to be considered:

  • Cross-domain resource access, including pass-through authentication, Trusted Client->Trusting Server->Trusting DC->Trusted DC
  • Authenticating access requests for a cross-domain object, Trusting DC -> Trusted DC
  • Using the object picker and retrieving a list of trusted domains and trusted accounts also results in PDC->PDC communication between trusting and trusted.
  • Establishing the trust, the PDC of the trusting NT4 domain will try and resolve and then query the PDC of the trusted domain. TCP and NetBIOS sessions will be established, with named pipes, SMB and RPC. A combination of LSA RPC and Netlogon RPC
  • Sometimes there may be a need to control the DC Locator process on the NT4 DCs to ensure that only the single DC is contacted to authenticate cross-domain access requests. This will be a local lmhosts #PRE #DOM: entry, or a static WINS record, but this would only work when controlling authentication, some other operations require PDC emulator connectivity.

How pass-through authentication across domains between 2003 and NT4 typically works:

  1. Client from 2003 sends a request to the resource server in the NT4 domain
  2. NT4 resource domain server passes the request to a NT4 DC in the local trusting domain
  3. The NT4 DC checks the request is from a trusted domain, then establishes NetBIOS/SMB sessions with a DC in the trusted domain
  4. Netr netlogon RPCs to create a secure channel and authenticate the samlogon request

If you need network security between the trusts, I recommend this be completed in a lab first. Otherwise you will probably end up with the connection between here and there looking like swiss cheese, or at the very least having larger holes than really required. VMware is great for creating a test-lab for this type of exercise, you could reproduce your existing production configuration and monitor the traffic going between various locations - this also requires something of a test network though.

Note that this information can also be used in concert with the 'Replica Domain for Authentication' post to create a trust with a replica of your production domain, reducing the risk of exposing your domain.

Both the RPC and SMB will be over NetBIOS when coming from NT4:

nbname 137/UDP
nbname 137/TCP
nbsession 139/TCP

In the scenario I was working on, the applications in the resource domain were either accessed via windows authenticated HTTP or direct SQL connectivity. Two examples below show the conversation in authenticating over the trust, between an XP member of the 2003 domain to a member server in the NT4 domain.

Assuming there’s a 2003 member server running IIS using windows authentication in the NT 4.0 resource domain, and XP clients from the corporate forest are accessing the application directly:

  1. Internet Explorer from the workstation initiates a HTTP session with the web server
  2. HTTP 401 is returned by the web server, indicating that authentication is required, with the WWW-Authenticate response headers indicating Negotiate and NTLM are the available schemes.
  3. NTLM Negotiate (0x00000001) is then sent from the client to the server indicating supported NTLM options
  4. The web server responds with a challenge message (0x00000002) for the client to prove their identity
  5. The workstation responds with an authenticate message (0x00000003) – an encrypted challenge response based on the logged on users’ password hash
  6. A Netlogon RPC call is initiated from the web server to the NT 4.0 DC the server has its secure channel with to initiate the samlogon request, providing the username, domain name, challenge, and challenge-response.
  7. The NT 4.0 DC checks the information and determines that it has a trust with the named domain.
  8. The NT 4.0 DC establishes a secure channel with the trusted replica domain (or uses the existing channel)
  9. A second Netlogon RPC call is then passed through to the replica 2003 domain using the same information as the first call (wrapped in NetBIOS in this case)
  10. The replica DC retrieves the hash of the specified user’s password, the original challenge is then encrypted using this hash and compared to the challenge-response provided with the request.
  11. Assuming passwords are synchronised, the hashes match and the trusted netlogon call is successful
  12. The Netlogon call from the web server is also returned as successful
  13. The web page is served to the XP workstation

Connecting across domains to SQL in NT4 from an XP workstation in 2003:

  1. The user starts the local MDE (which has an embedded connection to remote SQL)
  2. The workstation initiates a connection over TCP 1433 to the SQL server
  3. The workstation tries to verify the name/IP of the SQL server, with a reverse lookup DNS query and a NetBIOS Adapter Status Request to the IP over UDP 137 NBNS.
  4. The local Active Directory KDS is queried to see if there is a matching SPN for the requested SQL service. This is unsuccessful and the authentication protocol falls back to NTLM.
  5. A TDS NTLMSSP challenge/response occurs between the workstation and the cross-domain SQL server, including:
    a. The negotiate response from the workstation contains the client name and the database (0x1)
    b. The NT4 challenge from the SQL server (0x2)
    c. The authenticate response messages, containing response to the challenge using the currently logged in credentials (of the XP workstation)
  6. A Netlogon RPC call is initiated from the SQL server to the NT4 DC the server has its secure channel with to initiate the samlogon request, providing the username, challenge, and challenge-response.
  7. The NT4 DC checks the information and determines that it has a trust with the named domain (the 2003 user domain)
  8. The NT4 DC establishes a secure channel with the trusted domain
  9. A second Netlogon RPC call is then passed through to a trusted DC using the same information as the first call
  10. The NT4 DC retrieves the hash of the specified user’s password, the original challenge is then encrypted using this hash and compared to the hash provided with the request.
  11. Assuming the password is correct, the hashes match and the trusted netlogon call is successful across the trust.
  12. The Netlogon call between workstation SQL server is also returned as successful


Replica Domain for Authentication

How Domain and Forest Trusts Work

Trust Technologies

Establishing Trust Relationships

NTLM user authentication in Windows

Direct hosting of SMB over TCP/IP

How to configure a firewall for domains and trusts

Windows NT 4.0 tries to resolve manually-typed names by contacting the PDC for the remote user's domain (UDP 138). If that communication fails, a Windows NT 4.0-based computer contacts its own PDC, and then asks for resolution of the name.

Describes some of the network traffic between 2003 trusts, including methods of setting the LSA and Network RPC ports used for various operations.

I've also used this document in the past, touching on some topics such as DNS, netlogon optimisation/site synchronisation

Forest trust information – the flow and types of traffic for cross-forest authentication, which shows the workstation from the account domain (trusted), contacting a KDC in the resource domain (trusting) for kerberos authentication, often overlooked - this means traffic goes directly from all workstation endpoints to DCs in the resource domain, as opposed to an NTLM trust where DCs normally talk to each other)

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Tuesday, October 21, 2008

2003 Servers with Hibernation enabled

How many of your 2003 servers have hibernation enabled and you never realised? If you're like me, possibly a few, and I can't think of a scenario where I would want to hibernate a server.

This post provides a simple method of detecting whether hibernation is enabled, and then a command to remotely disable hibernation on each server. It works based on the premise that if hibernation is enabled, you'll have a file called hiberfil.sys on the root of your system drive.

  1. Create a text file called servers.txt listing one server per-line and then run the following command
  2. for /f %i in (servers.txt) do @if exist \\%~i\c$\hiberfil.sys (echo %~i,Enabled) else (echo %~i,Disabled)
  3. Save the output filterd by Disabled (find /i "Disabled" results.txt) and run the following command to disable on each (assuming you have psexec)
  4. for /f %i in (servers_hibernate.txt) do psexec \\%i powercfg /HIBERNATE off

This should automatically delete the hibernation file, reclaiming the hard disk space equal to the RAM of the machine.

Note that this assumes your systemdrive is C: and that you have administrative shares enabled (C$). To accurately distinguish between servers that don't have C:/C$. you could also run a nested if command (which assumes an installation directory of Windows):
for /f %i in (servers.txt) do @if exist \\%~i\c$\windows (if exist \\%~i\c$\hiberfil.sys (echo %~i,Enabled) else (echo %~i,Disabled)) else echo %~i,Not Found

You can also manually run the command to disable hibernation against one server:
psexec \\%server% powercfg /HIBERNATE off

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Sunday, September 28, 2008

Reading Shortcuts with PowerShell and VBS

This post provides a simple method of enumerating shortcuts and retrieving their properties, either using PowerShell or VBScript. There is nothing particularly clever here, but I had a need a while back to use the enumeration, and I wrote the simple powershell equivalent to see how easy it would be. Note that I couldn't easily see a managed code .Net method to process shortcuts, so I fell back on the wscript COM object. More than likely there is a better method (get it? :) than this.

Run the scripts:

powershell . .\ReadShortcut.ps1 -p \\%server%\%share%\folder\path
powershell . .\ReadShortcut.ps1 -f \\%server%\%share%\shortcut.lnk
cscript //nologo "ReadShortcut.vbs" \\%server%\%share%\folder\path
cscript //nologo "ReadShortcut.vbs" \\%server%\%share%\shortcut.lnk

   [string] $path = "",
   [string] $file = ""

$WshShell = new-object -comobject "WScript.Shell"       # Instantiate the COM object

if ($path -ne "") {
    $shortcuts = get-childitem -path $path -filter "*.lnk" -rec    # Find all .lnk files, recursing in to subdirectories
    $shortcuts | foreach-object {$WshShell.CreateShortcut($_.FullName) }  # For each file, pass the fullname to the COM object to open the shortcut and enumerate the properties
} elseif ($file -ne "") {
    $shortcut = get-item -path $file       # Get the single file
    if ($shortcut -ne $null) { $WshShell.CreateShortcut($shortcut) }   # If exists, read the shortcut properties
} else {
    write-output "No arguments specified, please use either -p to specify a path or -f for a specific .lnk file"


' Read a shortcut or a top-level directory of shortcuts and write the properties to stdout
If WScript.Arguments.UnNamed.Count = 1 Then
 strShortcut = WScript.Arguments.UnNamed(0)
 WScript.Echo "Please supply the name of an lnk file or directory to read, eg c:\test.lnk or c:\shortcuts"
End If

Set objFSO = CreateObject("Scripting.FileSystemObject")

If objFSO.FolderExists(strShortCut) Then      ' Was a directory specified?
 Set objFolder = objFSO.getFolder(strShortcut)     ' Get the folder
 For Each objfile in objFolder.Files      ' For each file in the top-level directory
  If objfile.type = "Shortcut" Then     ' Is this file a shortcut?
   Call Readshortcut(objFile.Path, strProperties)   ' yes, read the properties
   dtmCreationDate = objFile.DateCreated
   WScript.Echo dtmCreationDate & "," & strProperties  ' output the results
  End If 
ElseIf objFSO.FileExists(strShortCut) Then      ' Was an individual file specified?
 Call Readshortcut(strShortcut, strProperties)     ' read the properties of the file
 WScript.Echo strProperties       ' output the results
Else           ' file-not-found
 WScript.Echo "Error: Could not read '" & strShortcut & "'"
End If
Set objFSO = Nothing

Function Readshortcut(ByRef strShortcut, ByRef strProperties)

 set objWshShell = WScript.CreateObject("WScript.Shell")    ' Create the shell object
 set objShellLink = objWshShell.CreateShortcut(strShortcut)   ' Execute the createshortcut method, which also retrieves an existing shortcut
 strProperties = strShortCut & "," & objShellLink.TargetPath & "," & objShellLink.WindowStyle & "," & objShellLink.Hotkey & "," & objShellLink.IconLocation & "," & objShellLink.Description & "," & objShellLink.Arguments & "," & objShellLink.FullName & "," & objShellLink.WorkingDirectory & """"

 ' This propertly can be set, but not read? - objShellLink.RelativePath
 Set objShellLink = Nothing
 Set objWshshell = Nothing
End Function

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 20 years, and I know just about enough to realise that I don’t know very much.