Command-line automation – Echo and macros

This is the fifth and last in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the echo command and how to execute your commands using macros with doskey.

Echo

Echo is one of the most important commands to use when constructing repeatable commands – you should never run a command against multiple objects without first using echo. Some commands can be disastrous without appropriate input checking.

For example, using a wildcard incorrectly in a file spec, instead of putting *.txt, below I’ve put ‘* .txt’ – the wildcard selecting everything in the current directory (with the unfortunate del command for each of those files).
for %i in (* .txt) do echo del %i

Or more commonly – an incorrectly formed control file. The following command is one I’ve used to fix invalidly set UPNs on user accounts in the local directory. This command is quite powerful, and can be used to fix dozens or hundreds of accounts in seconds – but if the input file is not well formed, you could end up breaking each account:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix%

Incidentally, the command used to generate the ‘NoUPN.txt’ file was:
dsquery * %OU% -filter "&(objectclass=user)(objectcategory=person)(!(userprincipalname=*))" -s %server% -scope onelevel -attr name distinguishedname > NoUPN.txt

If you are untrusting of using the ‘for /f’ to execute operations against multiple objects, you can also use echo to construct a batch file that you can run at a later time – good practice rather than writing and executing the command in a single attempt. This may also useful when you’re modifying hundreds or thousands of objects and you don’t want to execute the commands immediately – for example if the modifications were in Active Directory and you wanted to prevent excessive replication during the day.

Running the command above with an appended redirect (>>) to a file, you could then pass this file to someone else for sanity checking before actual execution of the batch job:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix% >> %temp%\SetUPN.bat

Doskey macros

In the previous post there was a command to calculate a subnet at the command-line. If you had to type it every time - or even copy and paste - it’s not something you would use (okay, you’d probably never use it anyway), but doskey macros provide a simple method of allowing command re-use.

Creating a macro file

A text file can be created containing your macros, choose a shorthand name for the command, followed by the command you wish to execute. One slightly odd aspect with a macro file is that variables passed at the command-line for the macro use the $ prefix instead of %.

To make macros more useful, I often add two options – one without any parameters with a defined default, and another accepting a parameter to customise the command. For example, the following macro - called ‘AD2003’ - would by default query the local directory for all 2003 computer accounts. If a parameter is passed ($1), the dsquery command passes the parameter as the start node for the search (which could be a DN to limited the containers queried, or a different domain than domainroot).

AD2003=if _$1==_ (dsquery * domainroot -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0) else (dsquery * $1 -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0)

The find subnet command from the previous post could be shortened to execute the command with something as simple as ‘fs 192.168.11.40 255.255.254.0’, by adding the following to a macro file (and executing on a command processor with delayed expansion enabled):

FS=for /f "tokens=1-8 delims=.- " %i in ('echo $1 $2') do @set /a Octet1="%i & %m" >nul & @set /a Octet2="%j & %n" >nul & @set /a Octet3="%k & %o" >nul & @set /a Octet4="%l & %p" >nul & Echo !Octet1!.!Octet2!.!Octet3!.!Octet4!

Automatically enabling the macro file

It would also be unwieldy if you had to enable the doskey macros each time you ran a command shell. However, you can easily automate this with the autorun key of the shell, stored in either

HKCU or HKLM:
Key - HKEY_LOCAL_MACHINE\software\microsoft\Command Processor
Value - AutoRun
Value Type - REG_SZ
Value Data - doskey /macrofile=c:\util\macros.txt (or wherever)

This concludes what hopefully was a useful description of how I use built-in shell commands to automate repetitive tasks and make my work life easier and more efficient. Some of the resultant commands are not particularly easy to follow, but each can be broken down into the component parts and digested separately.

For more real-world examples of how I use the information in the last five posts, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – set

This is the fourth in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘set’ command, which I find very useful when automating repetitive tasks with ‘for’. I use set in four ways – string manipulation, simple arithmetic operations with ‘set /a’, gathering input with ‘set /p’ or concatenating output in a single for loop with delayed variable expansion.

String Manipulation

My most frequent use of the set command is simple string manipulation at the command prompt. I use this to replace one character with another or extract a substring.

Replace

Replacing one string with another is a core function of string manipulation. Even though this is included under help for ‘set’, this functionality works for other commands, including the echo command.

For example, say you have the fully qualified user DNS name – my.domain.com – and this conveniently is also the distinguished name of your AD domain, the following command replaces periods with ‘,DC=’ and prefixes an additional string to provide the distinguished name:
echo DC=%userdnsdomain:.=,DC=%

This would turn ‘my.domain.com’ into:
DC=my,DC=domain,DC=com

Wildcards can also be used, for example, to remove the first element, you could replace up to the first dot with nothing:
echo %userdnsdomain:*.=%

You could also remove spaces and print the output, for example, if the ‘test’ variable contained ‘A Sentence With Spaces’, running:
echo %test: =%

Would result in:
ASentenceWithSpaces

Substrings

You can extract from existing variables (or inline if using delayed expansion), providing Left/Right/Mid functionality. This can be useful in many ways, but for most of the input I deal with, splitting with ‘for’ and delimiters (such as comma or space) makes for simpler processing with dynamic input. For examples sake, if you had a defined length of padded text, and you wanted the 10 characters starting at the 20th character (0-based), you could run:
echo %input:~19,10%

You could also extract the last 10 characters:
echo %input:~-10%

Or all but the last 10 characters:
echo %input:~0,-10%

Arithmetic operations

‘set /a’ supports a relatively simple but still powerful set of arithmetic, bitwise and shift operators, and while I’ve never found much use for them, upon occasion they have come in handy.

For example, this command will tell you what next year will be and set the NextYear environment variable, based on the current year + 1:
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1

Similarly, find the current month, increment by one, unless the current month is 12, then wrap around and start at 1 again:
for /f "tokens=2 delims=/" %i in ('echo %date%') do if %i==12 (Set NextMonth=1 & echo !NextMonth!) else (Set /a NextMonth=%i+1)

Note that the example above uses delayed environment variable expansion, see below for a description.

Incidentally, if you were going to use this output for something and if the month were a single digit you would lose the padding, which would then be incorrectly sorted (as compared to other padded output). The example below pads the month, then extracts the last two characters, effectively padding single-digit months (see string manipulation above):
set NextMonth=0%NextMonth%
echo %nextmonth:~-2%

An actual use I have found for the arithmetic operators is to calculate subnets from the command-line. I have made use of this when parsing netsh dhcp output (there were thousands of leases across several DHCP servers serving dozens of subnets, and I wanted to know from the leases which subnets were in use).

This can be done with a single command-line, but it’s a little complicated, the batch file below may be easier to demonstrate the ‘set /a’ functionality. Running this batch file with an IP and mask, eg ‘findsubnet.bat 192.168.11.40 255.255.254.0’ will return the subnet (192.168.10.0 in this example) using bitwise AND of the two sets of octets:

@Echo Off
Set Input=%1
Set Mask=%2
If /i "%Input%"=="" Goto End

for /f "tokens=1-8 delims=.- " %%i in ('echo %Input% %Mask%') do Call :FindOctet %%i %%j %%k %%l %%m %%n %%o %%p
Goto End

:FindOctet
::Echo %1 %2 %3 %4 %5 %6 %7 %8

set /a Octet1="%1 & %5" >nul
set /a Octet2="%2 & %6" >nul
set /a Octet3="%3 & %7" >nul
set /a Octet4="%4 & %8" >nul
Echo %1.%2.%3.%4,%Octet1%.%Octet2%.%Octet3%.%Octet4%,%5.%6.%7.%8
Set Octet1=
Set Octet2=
Set Octet3=
Set Octet4=
Goto End

:InvalidArgs
Echo Invalid arguments, please provide an IP and longhand mask, eg 192.168.11.40 255.255.254.0
Goto End

:End

Input

Set can be used to gather input, making your commands or batch files interactive. For example, the following command assigns the input to the ‘test’ environment variable, after asking the question:
set /p test=Tell me something?

I don’t often use this, as it somewhat defeats the purpose of automation – I prefer to have all the required input in a control file, which separates the intended action from the actual operator.

Set with Delayed expansion

Delayed environment variable expansion provides a method of executing commands and storing the output for use by other commands within a loop – either in different iterations or nested commands. This is required when looping through ‘for’ commands or nesting commands, otherwise state between each command is not maintained.

For example, without delayed expansion, the following would not work:
set test=test variable & echo %test%

However, using delayed environment variable expansion, this problem can be easily overcome, using the !variable! syntax, instead of the normal %variable%. Note that you may need to enable delayed environment variable expansion, which can be done using ‘cmd /v:on’ when starting the command prompt, or running ‘setlocal ENABLEDELAYEDEXPANSION’ within an existing shell.

With delayed expansion, the first command would set the variable and the second echo the newly set variable:
set test=test variable & echo !test!

Or to re-use an example above, calculate next year and then do something with that information (echo in this case):
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1 1>nul & echo Next year is !NextYear!

Unless there is already a variable of the same name, running the commands above using the percent signed variables will result in the variable name being printed, as the environment hasn’t yet been changed by the first command.

Note that while testing, it seems the Vista command shell has been updated and as long as delayed expansion is enabled, this command would work:
set test=test variable & echo %test%

The following command is a one-line version of the batch above, using delayed variable expansion to calculate the variables as each command is executed:
for /f "tokens=1-8 delims=.- " %i in ('echo 192.168.11.40 255.255.254.0') do set /a Octet1="%i & %m" >nul & set /a Octet2="%j & %n" >nul & set /a Octet3="%k & %o" >nul & set /a Octet4="%l & %p" >nul & Echo %i.%j.%k.%l,!Octet1!.!Octet2!.!Octet3!.!Octet4!,%m.%n.%o.%p

An easier example to follow, set the environment variable called ‘number’ to a random number (using %random% - very useful), echo the newly set variable, and then use ‘set /a’ to increment the number by one:
set number=%random% & echo !number! & set /a number=!number!+1

If you are using nested ‘for’ commands, you can just reference variables from an earlier command, but in some circumstances delayed expansion is the only possibility. I have had to use this syntax when using ‘set /a’ and using the replace/substring functionality, as this requires a full variable (eg. %path%, not a temporary %i type variable.

This was a basic overview of ‘set’ and some of the ways in which I use this command when automating tasks at the command-line.

For more real-world examples of how I use ‘set’, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – errorlevels and if

This is the third in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are errorlevels, and if/then/else statements, useful for branching in a single command-line to execute based on the error level set or some other condition.

Using errorlevels, concatenated commands and if/then/else statements provides enhanced functionality, and most complex operations would not be possible on a single command-line without these features.

errorlevels

In the context of command-line automation, I find error levels most useful when trying to cater for both sets of outcomes when executing a command – either what you wanted happened, or it didn’t.

In the previous examples, I’ve used a control file to ping one or more machines and then echoed the IP address of those that were contactable. This is only part of the picture – what about those machines in the list that didn’t respond to a ping? They are simply dropped from the output. This may be what you are after, but normally I would prefer to know the positive and the negative.

To work around this problem, the following command could be used:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do echo %i

This will ping ‘computer’, echo the IP Address if found, otherwise echo NotFound. This occurs because the find command sets the errorlevel according to whether the string was found, and then we are concatenating the command with another to check if an errorlevel occurred indicating the string was not found, and echoing an alternate output to be picked up by the body of the for loop.

A simpler way of looking at this may be without the ‘for’ loop:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)

The command is similar to the example above, it will ping ‘computer’, and if the word reply was found in the output, it will print ‘replied’, otherwise it will print ‘no reply’.

Command concatenation and if/then/else

Commands can be concatenated – when running natively or inside a ‘for’ loop – providing a simple method to group commands. For example, looping through a control file, you might want to nslookup and then ping each device:
for /f %i in (c:\temp\control.txt) do nslookup %i & ping %i

This is a simple chain of commands, execute one and then the other. However, provided each command returns a relevant errorlevel, the if/then/else syntax can also be used to execute the second only if the first succeeded.

Unfortunately nslookup does not provide an errorlevel, but the find command does:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)
ping -n 1 computer | find /i "reply" >nul && echo Replied || echo No Reply

The above commands return the same result – they each print a statement depending on whether ‘reply’ was found in the ping output, the second using the shorthand ‘if’ (&&) and ‘else’ (||) syntax.

‘If’ can be used to check the errorlevel (as in the above example), check whether one string equals another, and check whether a file/directory exists.

If ErrorLevel checking

The following examples send an invalid service a stop control, and perform various checks on the errorlevel returned. Note that sc.exe on XP does not return error codes, this was Vista which does at least return 1060 when the specified service does not exist.

Check whether errorlevel (a special variable that can either be enclosed in % or not) was exactly 1060:
sc stop invalidservice >nul & if errorlevel==1060 echo Service not found

Check whether errorlevel was exactly 1060, if so echo ‘service not found’, otherwise echo the errorlevel returned:
sc stop invalidservice >nul & if errorlevel==1060 (echo Service not found) else (echo Error code %errorlevel% returned)

Check whether the errorlevel was 1060 or greater, and not 1061 (therefore only 1060).
sc stop invalidservice >nul & if errorlevel 1060 if not errorlevel 1061 echo Service not found

Check if anything above errorsuccess (0) was returned, echoes an error occurred and then uses ‘net helpmsg’ to return the error description for the specified error code:
sc stop invalidservice >nul & if errorlevel 1 echo An error occurred: %errorlevel% & net helpmsg %errorlevel%

If string checking

If can be used to conditionally check two strings to determine the action taken. Note that if the string could be blank, you’ll need to either append/prepend another character or enclose the string in quotes.

Using the above example of pinging a computer, check the response and branch accordingly (different echoes in this case):
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do if "%i"=="reply" (echo Ping success) else (echo Ping failed)

Note that ‘if /i’ can be used for a case insensitive comparison.

Additionally, if command extensions are enabled, and the strings in comparison are numeric, both are converted to numbers and additional operators are available for the comparison, for example:
If 9 equ 9 echo Equals
If 5 lss 14 echo Less than

If exist

To determine whether a file or directory exists, the ‘if exist’ syntax can be used. This can be useful when checking flag files or control files exist to avoid encountering unexpected errors.

If exist %windir% echo %windir% exists
If not exist c:\temp\control1.txt (echo Control File does not exist, terminating & break) else (echo continuing)

Some other examples of concatenating commands which help to demonstrate the concepts:

Use ‘sc.exe’ to send a service a stop control, if the service was running (exists, security allows etc) and was sent a stop, sc will return errorsuccess (0), pause for 5 seconds and then restart the service, otherwise echo that the service was not running in the first place:
sc stop "service" && (echo error %errorlevel% & sleep 5 & sc start "service") || echo Service was not running, error %errorlevel%

nslookup www.microsoft.com, find the second address (the first in nslookup is the DNS server address, and echo that address:
for /f "skip=1 tokens=2" %i in ('"nslookup www.microsoft.com 2>nul | find /i "address""') do echo %i

nslookup www.microsoft.com, skip the first DNS server address (the second line of the nslookup output – find /n), return the other address if found, otherwise return ‘Not-Found’ in the %i variable – the second token of the string:
for /f "tokens=2" %i in ('"nslookup www.microsoft.com 2>nul | find /i /n "address" | find /i /v "[2]" & if errorlevel 1 echo Address Not-Found"') do echo %i

Create a file, if the file is found type the contents, and then delete the file if the type succeeded, otherwise report that the ‘type’ command failed:
echo test > %temp%\test.txt & if exist %temp%\test.txt type %temp%\test.txt && del %temp%\test.txt || Echo could not type %temp%\test.txt

This was a basic overview of the errorlevels and if/then/else branching, future posts will build on this with other useful commands and nested ‘for’ loops.


For more real-world examples of how I use errorlevels and if/then/else, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – find and findstr

This is the second in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are the ‘find’ and ‘findstr’ commands - unless you want to spend all your time manually creating or filtering control files you will often use these commands when constructing repeatable command-lines.

These two commands are crucial in efficient ‘for’ loop processing, invaluable when filtering a control file or the results of an in-line command. Help on each is available from the command prompt with the ‘/?’ argument.

Find

I usually use find because more often than not I want to filter a string literal – a specific word or combination of characters – from an input string. As often as not, I also use ‘find /v’, returning the lines that don’t match the specified string.

For example, if you have a control file listing servers in several geographic locations, but you are only interested in processing servers in ‘bne’ (case insensitive with ‘/i’), you could run:
find /i "bne" c:\temp\control.txt

Combined with a ‘for’ command, this could be:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do echo %i

Note that this line uses the following syntax explained in the previous post on ‘for’:

1. ‘Skip=2’ tells the loop to skip the first two lines – the result header of the ‘find’ command
2. Inside the brackets I’ve specified to run a command and parse the results, wrapping the command in single-quotes.

The results are that for each server containing the word ‘bne’, the body of the ‘for’ statement will be executed – in this case simply to echo the first token.

You might also want to run the commands against all servers except those in bne, which is the above command repeated with the addition of the ‘/v’ switch to tell ‘find’ to return those lines not containing a match:
for /f "skip=1" %i in ('find /i /v "bne" c:\temp\control.txt') do echo %i

Filtering in-line output with find

Filtering output of a command executed within a ‘for’ loop is syntactically a little more complicated, but makes sense once you get the hang of it. For example, suppose you want to filter the output of a ping command, returning the IP address of those hosts that replied to a ping, you could run:
for /f "tokens=3" %i in ('"ping -n 1 computer find /i "reply""') do @echo %i

This will:

• Execute the ping command, pinging the host named computer, filtering the output to find the word ‘reply’ (indicating a successful reply). Note that to use the pipe inside the brackets, the whole string needs to be enclosed in double-quotes – inside the single-quotes.
• Using spaces/tabs, choose the third token to return in the first variable - %i. This will be the IP address in the standard ping output returned by XP and Vista.

Unfortunately the third token of the ping output includes a irrelevant colon, so to strip that, we could specify colons and spaces as delimiters in the for loop:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer find /i "reply""') do echo %i

This will be better explained in a future post, but you can also nest two ‘for’ commands and for each host in the control file (in bne in this example), ping and return the IP address if there is a reply:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do @for /f "tokens=3 delims=: " %m in ('"ping -n 1 %i find /i "reply""') do @echo %i,%m

The output would give you a comma-separated list of server names and IP address of those from the original control file that currently respond to a ping. Theoretically not all that useful with servers, as they’d usually be known IPs and always on, but if you’re wanting to check something on hundreds or thousands of workstations this would give you a point-in-time control file of online machines (assuming name resolution is accurate).

It’s also worth noting that ‘find’ commands can be cascaded – piping the output of one find command to another if you need to further refine the output. For example to find servers in ‘bne’ and then servers in ‘bne’ that also contain ‘vm’:
find /i /c "bne" c:\temp\control.txt find /i "vm"

The possibilities are nearly endless, I use ‘find’ to:

• Filter in and out items that I want, ‘find /i’ and ‘find /i /v’ from generic control files that I use for multiple operations, either in-line in a for command, or to create a stand-alone filtered control file
• Run commands and then filter out useless or useful information, often headers/footers or other extraneous detail, making it possible to list data, filter that data and run a command against each filtered item in a single command.
• Use errorlevels to determine whether the find command was successful or not (explained in a future post). Often the only way of knowing whether a command returned what you were after is the presence or absence of a string – and ‘find’ will tell you this based on the errorlevel (an advantage over findstr which in XP at least does not set the errorlevel).

Note that find also has several other features that are useful in some scenarios:

• By default, offline files will be skipped – if you use a file archiving product that sets the offline (‘O’) attribute, these files will not be searched unless you specify the /offline parameter
• The ‘find /n’ option will return the line number the match was found, occasionally useful when parsing large or sequenced control files and you want to know where in a file the string was found.
• The ‘find /c’ option will return a count of the matches found, instead of the actual matching lines. This can often be useful when determining how many records would be processed. For example, in the example above you could first run the following command to determine how many records would be processed with the for loop:
  find /i /c "bne" c:\temp\control.txt

Findstr

These same principles can be used with findstr to filter output – findstr provides advanced searching, with the following benefits in my use of the commands, you can:

• Search for more than one string in a single command. This simplifies searches, in the example above, you could search for ‘bne’ and ‘syd’ in a single findstr command.
• Use basic regular expressions and/or string literals

In the example above with a control file containing a list of servers with geographic names (or descriptions somewhere on the line), you could run the following command to return servers in both syd and bne and then run the subsequent command against each server:
for /f %i in ('findstr /i /c:bne /c:syd c:\temp\control.txt') do echo %i

Regular expressions are used to group similar items, without expressly providing each possible combination. For example, parsing robocopy log output, to search for new directories that have been created with a regular expression, you could run:
findstr /i /n "^.*New.Dir" Robocopy.log"

You could just use ‘find’ and search for the string ‘new dir’, but this could easily return false positives if a file or directory has the string ‘new dir’ in it. Using the regular expression looking for the start of the line followed by whitespace followed by the ‘New Dir’ is much more precise.

This was a basic overview of the ‘find’ and ‘findstr’ commands, future posts will build on this foundation with other useful commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘find’ and ‘findstr’ commands, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Building blocks of command-line automation – FOR

This is the first in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used for data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘for’ command, the most important component in command-line automation. This command provides several methods of looping through a list, and running a command against each element in that list. Use the ‘for /?’ to get Microsoft help on this command.

Using the ‘for’ command with one of the syntaxes below provides many benefits, including:

• Repeatability – If you save the command you run, it can be re-run several or hundreds of times
• Change control and testing – It’s easy to record what you are planning and simple to run the same commands in a test-lab environment. The output of commands can also be redirected to file, making accountability much easier. Using control files also provides a straightforward method of recording the targets of various actions, knowing that you have one master list, and do not risk accidentally missing an entry.
• Documentation – Implementation using a series of commands very easily translates to an as-built document – with the added benefit of providing a quicker DR process.
• Efficiency – Even though designing the command for the first run may not be as quick as using the GUI, every time after that will usually be much quicker, and previous commands can often be quickly adapted to new tasks.

You can use the ‘for’ command to:

Process a set of filenames or string literals

The filenames or literals can either be directly named in a space-separated set, or you can use wildcards to process more than one file. For example:
for %i in (*.txt) do echo %i
for %i in (test1.txt test2.txt) do echo %i

For example, I would use this syntax if I’m trying to:

Quickly execute something against a group of machines, eg ping each machine:
for %i in (server1 server2 server3 server4) do ping -n 1 %i

Process a series of data files that I have created from another process, eg this uses the regview utility to export the registry entries modified by *.pol files into *.txt:
for %i in (*.pol) do regview %i > %i.txt

Quickly execute the same command with a different variable, eg use the setprinter utility to view all levels of configuration for the specified printer (you could also use for /l in this example):
for %i in (0 1 2 3 4 5 6 7 8 9) do setprinter -show \\server\printer %i

Process a set of directories

The directory names can either be directly named in a space-separated set, or you can use wildcards to process more than one directory. For example:
for /d %i in (%windir%\*) do echo %i
for /d %i in (c:\temp c:\windows) do echo %i

I would use this syntax if I’m trying to do something with each top-level directory, for example:

Report or set ACLs:
for /d %i in (%rootDir%\*) do icacls %i /save %~ni.txt

Rename all the top-level directories to start with a new prefix:
for /d %i in (%rootDir%\*) do ren %i New-%~ni

Process the contents of a text file, line by line

The contents of a file – which I usually refer to as a control file – can be read line-by-line and your command would be run once for each line, substituting tokens from the control file. This provides unlimited capability – construct a control file through any means available and you can then process the entries one-by-one and run a command against that entry.

Note that in Vista at least, just a LF is enough to separate the lines, rather than the Windows standard CR+LF.

For example, assuming you have a control file with a list of servers or workstations, you could:

Lookup the IP address of each machine:
for /f %i in (test.txt) do nslookup %i

Ping each machine:
for /f %i in (test.txt) do ping %i

Remote dir of each machine:
for /f %i in (test.txt) do dir \\%i\c$

I use this constantly to run a command against multiple AD objects, machines, printers, or other network devices, whether the command queries or checks something, or makes a change to each device.

Process the results of a command, line by line

The results of almost any command can be used as the looping mechanism for a ‘for /f’ command, providing an in-memory control file. For example, you could:

Find the local hostname and then nslookup the computer (you could also use %computername% for this):
for /f %i in ('hostname') do nslookup %i

Query the local Active Directory for a list of DCs (server records) and lookup the IP of each DC:
for /f %i in ('dsquery server -o rdn') do nslookup %i

Recursively enumerate a path

It’s possible to recursively enumerate files or directories from a specified starting location, passing each to the body of the for loop. This provides a rudimentary search and response facility, allowing you to traverse a tree looking for objects of a particular type – and then execute something for each found.

For example, you could search from the root of C: for *.txt files, and then report the filename and size (you would just use dir /s if all you wanted to do was echo)
for /r c:\ %i in (*.txt) do echo %i %~zi

Step through a sequence of numbers and execute for each

The ‘for /l’ option allows stepping through a sequence of numbers, passing the number as a parameter to the body of the ‘for’ loop.

I don’t use this method very often, but it would be another method to the setprinter command above:
for /l %i in (1,1,9) do setprinter -show \\server\printer %i

Variable Expansion

When using ‘for’, ‘for /f’ and ‘for /d’ variable references can also be modified to return substrings or additional information. Note that when using ‘for /f’, most of these only make sense if you are processing lists of files or directories, but if you did have a control file with files/paths variable expansion does work as expected.

This substitution can be very useful, particularly when constructing parameters to pass to the command in the body of the ‘for’ loop. For Example:

If you had a number of control files that you wanted to process, outputting the results to a same-named log file:
for %i in (c:\temp\*.txt) do echo Do something with %i > %~ni.log

If the output of a previous command wrapped the results in quotes, but you need to append/prepend something else you can easily remove surrounding quotes:
for /f %i in ('echo "c:\windows"') do echo %~i\temp

Given a list of files, echo those that are zero bytes in size:
for %i in (c:\temp\*.txt) do @if %~zi == 0 @echo %i

Given a full path, split into drive, path and filename:
for %i in (c:\windows\temp\test.txt) do echo %~di %~pi %~nxi

Tokens, delimiters and skipping lines

The simple functionality of the ‘for /f’ command can be extended very easily with three options:

1. Tokens – By default only the first token is returned in the variable specified. You can change this behaviour to return one or more tokens, eg tokens=2,3,5 or tokens=1-3 would populate %i, %j and %k with the respective tokens
2. Delimiters – Instead of the normal space and tab delimiters, one or more alternate characters can be specified. For example, you can specify a comma as a delimiter to process as CSV file
3. Skipping lines – the skip command can be used to skip one or more lines from the start of a control file, useful when trying to skip a header line, or bypass logo information in a command result.

These options can be used individually or as a combination, for example:

Skip the first line of the control file:
for /f "skip=1" %i in (test.txt) do echo %i

Skip the first line, and use comma’s as the delimiter:
for /f "skip=1 delims=," %i in (test.txt) do echo %i

Skip the first two lines, use the second token, separated by comma and space:
for /f "skip=2 tokens=2 delims=, " %i in (test.txt) do echo %i

This was a basic overview of the ‘for’ command, future posts will build on this foundation with multiple commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘for’ command, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Useful PowerShell command-line operations

The commands below are a small but growing list of powershell one-liners I've used (and when I remember to take note). Note that this may not be the best and almost certainly isn't the only way in most cases.

Each command-line can be copied and pasted at a PowerShell command prompt, or you can use the commands as part of a PS1 script file if you prefer.


Show help on commands such as 'if', regular expressions etc
help about*

Show help about about_Automatic_variables such as $_ $DebugPreference $OFS
help about_Automatic_variables

Run a command and pipe the results to a variable
cscript.exe //nologo script.wsf | Set-Variable -name scriptResults

Run a command and set a variable with the output
$scriptResults = cscript.exe //nologo script.wsf

Run a command and set a variable with the output using the call operator
$scriptResults = & cscript.exe //nologo script.wsf

Filter an object based on an array of strings
$array = "a", "b"; write-output a b c d | select-string -pattern $array -simpleMatch

Reformat today's date to YYYYMMDD
$today = [DateTime]::Now.ToString("yyyyMMdd")

Find local disks except C: using powershell and output to CSV
Get-WmiObject -Namespace root\cimv2 -ComputerName %server% -Query "SELECT * from Win32_LogicalDisk WHERE FileSystem='NTFS' AND Description = 'Local Fixed Disk' AND Name != 'C:'" | export-csv c:\disk.csv

Use the VI Toolkit Powershell snap-in to query for snapshots
Get-VM | Get-Snapshot | export-csv -path c:\temp\VMsnapshots.csv

Use the VI Toolkit Powershell snap-in to query for snapshot information
Get-VM | Get-Snapshot | foreach-object {$out= $_.VM.Name + "," + $_.Name + "," + $_.Description + "," + $_.PowerState; $out}

Start a remote process using Powershell/WMI
$computer = "."; ([WMICLASS]"\\$computer\root\CIMv2:win32_process").Create("notepad.exe")

Find remote processes and the command-line parameters with PowerShell
Get-WmiObject win32_process | Format-Table ExecutablePath,Caption,CommandLine,CreationDate,WorkingSetSize,ProcessId

Generate a password 10 characters in length, at least 5 punctuation marks
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Web") ; [System.Web.Security.Membership]::GeneratePassword(10,5)

Format a date to XML XSD:dateTime
$Now = [DateTime]::Now.ToString("yyyy-MM-ddTHH:mm:ss")

Check if the given variable is an object
if ($variable -is [object]) {}

Check which network connections (drive mappings) a 2003 computer has
get-WMIObject -computerName "$env:computername" -class win32_logicaldisk -property ProviderName, DeviceID -filter "DriveType=4" | Format-List -property ProviderName, DeviceID

See whether an IP can be resolved to a host name
[System.net.Dns]::GetHostEntry($IP)

See whether a host name can be resolve to an IP
$ip = & { trap {continue}; [System.net.Dns]::GetHostAddresses($computer) }

View the datastore information for the available storage
get-vc -server $server ; Get-Datastore

Check if a file or path exists or not
test-path -path $path ; if (!(test-path -path $path)) {write-output "$path does not exist"}

Use psexec to run a powershell command remotely and report to stdout (eg query 'remote' event logs in this example)
psexec \\machine /s cmd /c "echo. | powershell . get-eventlog -newest 5 -logname application | select message"



Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

MSCS 2003 Cluster Virtual Server Components

This post provides my interpretation of a simple MSCS 2003 virtual server with a file share, including how the cluster interacts with the OS and network services to provide access to the share. This follows on from the last post on low-level detail of file access in an attempt to provide a clearer picture of these often taken-for-granted components.

Note that this is only my opinion, based on less-than complete knowledge and more than likely contains semantic errors if nothing else.

File & Print Cluster Native x64 (EM64T/AMD64)

1. Cluster Service. Includes Checkpoint Manager, Database Manager, Event Log Replication Manager, Failover Manager, Global Update Manager, Log Manager, Membership Manager, Node Manager.
1. Operating System Interaction with the LanManServer Service which advertised shares.
2. NetBIOS registration of the virtual server name through existing network services
3. DNS registration of the virtual server name through existing network services
4. Kerberos SPNs registered against an AD computer account through Active Directory
2. Resource Monitor. Spawned child process of the cluster service, separate resource monitors can exist for resource DLLs
3. ClusRes.dll Physical Disk <-> IsAlive/LooksAlive SCSI reservation and directory access. LooksAlive issues a SCSI reservation every 3 seconds through ClusDisk.sys against all managed disks. IsAlive performs a ‘dir’ equivalent
4. ClusRes.dll Network Name <-> IsAlive/LooksAlive check on NetBT/DNS registration. LooksAlive relies on MSCS NIC failure detection. IsAlive queries local TCP/IP stack for virtual IP and the NetBT driver if NetBIOS is enabled
5. ClusRes.dll IP Address <-> IsAlive/LooksAlive check on cluster NIC. LooksAlive queries NetBT driver and every 24 hours issues a dynamic DNS host record registration. If ‘DNS is required’ resource will fail if DNS registration fails. Same test for IsAlive
6. ClusRes.dll Resource DLL File Shares <-> IsAlive/LooksAlive check on file share visibility. LooksAlive queries lanmanserver service for the share name. IsAlive does the same, and if unlimited users, the first file on the share is copied
7. 32-bit Resource Monitor WOW64, Eg. Enterprise Vault Cluster application. Third-party cluster resources, eg Enterprise Vault, which in this case notifies the local FSA placeholder service on each physical node of virtual server changes
8. ABE enabled by generic cluster resource. Access based enumeration with a generic cluster application running abecmd.exe during virtual server/share creation. Uses 32-bit cluster resource monitor with WOW64, setting SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM (0x0800) flag set on the otherwise standard share.

Pretty picture view:

References

Server side processes for simple file access
http://waynes-world-it.blogspot.com/2008/11/server-side-process-for-simple-file.html

Access Based Enumeration
http://technet.microsoft.com/en-us/library/cc784710.aspx

SHARE_INFO_1005 Structure
http://msdn.microsoft.com/en-us/library/bb525404.aspx

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!