Tuesday, December 16, 2008

Building blocks of command-line automation – FOR

This is the first in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used for data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘for’ command, the most important component in command-line automation. This command provides several methods of looping through a list, and running a command against each element in that list. Use the ‘for /?’ to get Microsoft help on this command.

Using the ‘for’ command with one of the syntaxes below provides many benefits, including:

  • Repeatability – If you save the command you run, it can be re-run several or hundreds of times
  • Change control and testing – It’s easy to record what you are planning and simple to run the same commands in a test-lab environment. The output of commands can also be redirected to file, making accountability much easier. Using control files also provides a straightforward method of recording the targets of various actions, knowing that you have one master list, and do not risk accidentally missing an entry.
  • Documentation – Implementation using a series of commands very easily translates to an as-built document – with the added benefit of providing a quicker DR process.
  • Efficiency – Even though designing the command for the first run may not be as quick as using the GUI, every time after that will usually be much quicker, and previous commands can often be quickly adapted to new tasks.

You can use the ‘for’ command to:

Process a set of filenames or string literals

The filenames or literals can either be directly named in a space-separated set, or you can use wildcards to process more than one file. For example:
for %i in (*.txt) do echo %i
for %i in (test1.txt test2.txt) do echo %i

For example, I would use this syntax if I’m trying to:

Quickly execute something against a group of machines, eg ping each machine:
for %i in (server1 server2 server3 server4) do ping -n 1 %i

Process a series of data files that I have created from another process, eg this uses the regview utility to export the registry entries modified by *.pol files into *.txt:
for %i in (*.pol) do regview %i > %i.txt

Quickly execute the same command with a different variable, eg use the setprinter utility to view all levels of configuration for the specified printer (you could also use for /l in this example):
for %i in (0 1 2 3 4 5 6 7 8 9) do setprinter -show \\server\printer %i

Process a set of directories

The directory names can either be directly named in a space-separated set, or you can use wildcards to process more than one directory. For example:
for /d %i in (%windir%\*) do echo %i
for /d %i in (c:\temp c:\windows) do echo %i

I would use this syntax if I’m trying to do something with each top-level directory, for example:

Report or set ACLs:
for /d %i in (%rootDir%\*) do icacls %i /save %~ni.txt

Rename all the top-level directories to start with a new prefix:
for /d %i in (%rootDir%\*) do ren %i New-%~ni

Process the contents of a text file, line by line

The contents of a file – which I usually refer to as a control file – can be read line-by-line and your command would be run once for each line, substituting tokens from the control file. This provides unlimited capability – construct a control file through any means available and you can then process the entries one-by-one and run a command against that entry.

Note that in Vista at least, just a LF is enough to separate the lines, rather than the Windows standard CR+LF.

For example, assuming you have a control file with a list of servers or workstations, you could:

Lookup the IP address of each machine:
for /f %i in (test.txt) do nslookup %i

Ping each machine:
for /f %i in (test.txt) do ping %i

Remote dir of each machine:
for /f %i in (test.txt) do dir \\%i\c$

I use this constantly to run a command against multiple AD objects, machines, printers, or other network devices, whether the command queries or checks something, or makes a change to each device.

Process the results of a command, line by line

The results of almost any command can be used as the looping mechanism for a ‘for /f’ command, providing an in-memory control file. For example, you could:

Find the local hostname and then nslookup the computer (you could also use %computername% for this):
for /f %i in ('hostname') do nslookup %i

Query the local Active Directory for a list of DCs (server records) and lookup the IP of each DC:
for /f %i in ('dsquery server -o rdn') do nslookup %i

Recursively enumerate a path

It’s possible to recursively enumerate files or directories from a specified starting location, passing each to the body of the for loop. This provides a rudimentary search and response facility, allowing you to traverse a tree looking for objects of a particular type – and then execute something for each found.

For example, you could search from the root of C: for *.txt files, and then report the filename and size (you would just use dir /s if all you wanted to do was echo)
for /r c:\ %i in (*.txt) do echo %i %~zi

Step through a sequence of numbers and execute for each

The ‘for /l’ option allows stepping through a sequence of numbers, passing the number as a parameter to the body of the ‘for’ loop.

I don’t use this method very often, but it would be another method to the setprinter command above:
for /l %i in (1,1,9) do setprinter -show \\server\printer %i

Variable Expansion

When using ‘for’, ‘for /f’ and ‘for /d’ variable references can also be modified to return substrings or additional information. Note that when using ‘for /f’, most of these only make sense if you are processing lists of files or directories, but if you did have a control file with files/paths variable expansion does work as expected.

This substitution can be very useful, particularly when constructing parameters to pass to the command in the body of the ‘for’ loop. For Example:

If you had a number of control files that you wanted to process, outputting the results to a same-named log file:
for %i in (c:\temp\*.txt) do echo Do something with %i > %~ni.log

If the output of a previous command wrapped the results in quotes, but you need to append/prepend something else you can easily remove surrounding quotes:
for /f %i in ('echo "c:\windows"') do echo %~i\temp

Given a list of files, echo those that are zero bytes in size:
for %i in (c:\temp\*.txt) do @if %~zi == 0 @echo %i

Given a full path, split into drive, path and filename:
for %i in (c:\windows\temp\test.txt) do echo %~di %~pi %~nxi

Tokens, delimiters and skipping lines

The simple functionality of the ‘for /f’ command can be extended very easily with three options:

  1. Tokens – By default only the first token is returned in the variable specified. You can change this behaviour to return one or more tokens, eg tokens=2,3,5 or tokens=1-3 would populate %i, %j and %k with the respective tokens
  2. Delimiters – Instead of the normal space and tab delimiters, one or more alternate characters can be specified. For example, you can specify a comma as a delimiter to process as CSV file
  3. Skipping lines – the skip command can be used to skip one or more lines from the start of a control file, useful when trying to skip a header line, or bypass logo information in a command result.

These options can be used individually or as a combination, for example:

Skip the first line of the control file:
for /f "skip=1" %i in (test.txt) do echo %i

Skip the first line, and use comma’s as the delimiter:
for /f "skip=1 delims=," %i in (test.txt) do echo %i

Skip the first two lines, use the second token, separated by comma and space:
for /f "skip=2 tokens=2 delims=, " %i in (test.txt) do echo %i

This was a basic overview of the ‘for’ command, future posts will build on this foundation with multiple commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘for’ command, see my command-line operations:

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

No comments:

All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 13 years, and I know just about enough to realise that I don’t know very much.