VirtualCenter Physical to Virtual

This post describes a process I executed to replace a physical VMware VirtualCenter box with a virtual equivalent running in its own cluster. There were no VM outages of machines running on the cluster – each ESX kept merrily running the virtuals until everything was sorted out with the VC layer.

Note that this was done in a lab instance of ESX 3.5 and VirtualCenter 2.5, had this been production I probably would have taken a little more care.

I don’t think there is any compelling reason why you wouldn’t run a virtual VC box, it would be hypocritical of VMware to suggest virtualizing your application servers, except VirtualCenter which should be physical. Having said that, this process could also be used in reverse, taking a VM VirtualCenter instance physical should the need arise. This would also be useful for a VirtualCenter disaster recovery scenario.

The configuration before completing these steps

1. Physical VirtualCenter 2.5 running on Windows Server 2003, called vc01, static IP
2. Virtual Windows Server 2003 computer running in the cluster, called vc02, dynamic IP \
3. SQL 2005 database for VC, running on a separate SQL server.
4. ESX 3.5 VC 2.5, single network, iSCSI shared storage

Prerequisites

1. The IP address of the VirtualCenter server
2. The ESX host of the VM becoming the new VC server
3. The path to your VMware FlexLM license file (assuming you’re using a license server)
4. The logon details for the SQL connection between vpxd and the database

Steps Taken

On the physical VirtualCenter box that is going to be decommissioned:

1. Take note of the physical ESX host running the VM becoming the new VC server
2. Stop the vpxd service and change the startup to manual (sc stop %service%, sc config %service% start= demand)
3. Stop the vmountVpx, vmware-ufad-vci, vmware-converter and webAccess services change the startup to manual
4. Stop the flexlm instance and set the startup to manual
5. Take a backup of your .lic license file
6. Change the IP address to dynamic (assuming it is static)
7. Power off the physical machine
8. Delete the computer account for vc01 from the domain
9. On the SQL server hosting the VirtualCenter database, backup the VC database and log file with the following command executed through management studio:
   
   

   BACKUP DATABASE [VMVC]
   TO DISK = N'c:\temp\VC_PreMoveToVM.bak'
   WITH
   DESCRIPTION = N'VirtualCenter Pre-move to VM backup'
   , INIT
   , NAME =
   N'VC Pre move to VM'
   GO
   
   RESTORE VERIFYONLY
   FROM DISK =
   N'c:\temp\VC_PreMoveToVM.bak '
   WITH FILE = 1
   GO
   

10. In the service console of one of the ESX hosts, backup the current certificates (just in case):
1. mkdir /tmp/cert_backup
2. cp /etc/vmware/ssl/* /tmp/cert_backup
11. On the VM becoming the new VirtualCenter server running on a host in the cluster: Rename the server from vc02 to vc01, with the same static IP as the previous vc01
12. Restart the virtual machine
13. Install the SQL Native Client - required for VC 2.5 SQL connectivity on a non-SQL server
14. Install VirtualCenter including FlexLM, connecting to the existing database and using the license file copied off the physical server
15. In VirtualCenter, disconnect the first ESX host in the cluster - maintenance mode is not possible at this stage – login errors occur because of the incorrect certificate
16. Copy the new VC certificates - I use pscp here, but whatever you normally use to copy files to ESX would be fine:
    
    1. cd "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\ssl"
    2. pscp rui.crt vcadmin@esx01:/etc/vmware/ssl/
    3. pscp rui.key vcadmin@esx01:/etc/vmware/ssl/"
17. In the service console of the ESX node you’ve just updated the certificates on, restart the management interface, the ESX host did not seem to pick up the new certificate dynamically (maybe it would on a schedule without a restart):
    1. service mgmt-vmware restart
18. Connect the ESX host to VC through the VI Client interface
19. Repeat steps 15-18 for each other ESX host in the cluster
20. In VirtualCenter, run 'Reconfigure for HA' on each ESX node
    
    Testing
    
    
21. Ensure the vpxd.log file reports no problems with host connectivity or certificates (also check the VC/ESX logs)
22. Ensure each ESX host is receiving licenses from the 'new' license server, either through the VI client or the FlexLM admin tool.
23. Ensure you can perform simple tasks such as powering on a virtual machine
24. Ensure VMotion/HA/DRS is working

The (untested) rollback plan if something goes wrong:

1. Shutdown the new VirtualCenter VM
2. Restore the database from the backup taken
3. Power on the physical VC box, change the IP to the static IP
4. Restart the vpxd and VMware License Server services
5. For each ESX host in the cluster, disconnect the host, restore the old certificates, restart the management service and connect the ESX host to the old VirtualCenter instance

Additional notes:

• Even though user login errors were returned when the vpxd service tried to form the cluster - which points to the vpxuser account used by VC to manage ESX hosts - this was misleading as this username and password is stored in the VC database – which had not been modified (in the vpx_host table). The next logical step was certificates, which lead to certificate update process used above.
• Manually copying the certificates may not be strictly required, as when I went part-way to reconnecting a host without updating the certificates, I was prompted that another VC instance was managing these servers, would I like to continue. Presumably it would have automatically updated the certificates as required.

Read more!

Virtual 2003 MSCS Cluster in ESX VI3

This post shares a method I've used to create test-lab instances of standard 2003 file and print Microsoft Cluster Services (MSCS) clusters in a VMware ESX VI3 virtual environment. The resultant solution is not supported and definitely not production-ready, but if you want a real multi-node MSCS cluster in an ESX lab environment, this process might be helpful with a minimum set of requirements.

With my usual theme of repeatable command-line execution, most of these operations can be completed via the command-line, either in the ESX service console or a command-prompt from the virtual MSCS nodes.

I followed bits and pieces of the VMware supported method - which is very specific and quite restrictive. Note that I’m a little dubious that this cluster would be particularly stable – the SCSI reservations MSCS uses to lock disks are in no way supported when using a shared VMDK through a shared SCSI adapter (I think RDM is the only supported method), but it does work and at least provided me with a test environment.

The shared nothing model of 2003 MSCS clustering dictates that only one node accesses the partition at any one time, but the disk still needs to be visible to both nodes. A limitation of this solution is that both MSCS nodes need to be hosted on one ESX server – a requirement you could satisfy with a DRS rule to keep the two nodes together. However, if DRS decided to migrate both VMs, the cluster would almost certainly break during the failover (and possibly after).

If you follow the steps below, you should end up with two virtual x64 2003 enterprise servers, both members of a single MSCS cluster. In the cluster there will three shared disks (VMDKs), one for the quorum and one each for file and print – with a virtual server and relevant cluster resources. A test file share is created, along with drivers and a test printer. You'll need to modify the commands that reference the public adapter and IP addresses

Steps involved:

1. Create an area for storage of the shared disk on your datastore:
   1. mkdir /vmfs/volumes/%datastore%/cluster01
2. Create a 5GB quorum disk:
   
   1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-Quorum.vmdk
3. Create a 5GB disk for shared data:
1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-disk01.vmdk
4. Create two 2003 x64 enterprise virtual machines, either through cloning, deployment with templates or whatever your standard build process may be
5. If cloning was used, run sysprep on both nodes to give a unique SID and join your lab domain
6. Shutdown the first node and add the shared disk
1. Add the quorum disk, mounted under scsi 1:0 (which adds a new SCSI adapter)
2. Set the newly created SCSI Adapter to SCSI bus sharing virtual
3. Add disk01, attached as scsi 1:1
7. In the first VM, use disk administrator (or diskpart) to initialise the quorum and disk01 disks, partitioned with basic. Record the signature of the disk and the drive letter used (although this is the disk volume when the disk is owned by the OS, not the cluster).
8. Add a service account for the cluster service:
1. dsadd user "CN=clustersvc,CN=Users,DC=test,DC=local" -pwdneverexpires yes -pwd password -disabled no -desc "MSCS VM cluster service account"
2. Ensure the service account is an administrator of each virtual 2003 node
9. Use Cluster Administrator to install the cluster on the first node, with your chosen cluster name, using the created quorum disk and service account
10. Verify correct operation of the single-node cluster, and then add the second VM node to the cluster.
11. Create a new port group to allow a second private adapter on each ESX server:
1. esxcfg-vswitch -A MSCS-Private Private
2. Add a second interface to each VM cluster node, allocated separate address space
3. Verify connectivity (ping) and configuration following cluster best practices (no gateway, no DNS etc)
4. Mark as a private heartbeat connection for the cluster, prioritised above the LAN connection.
12. Create a virtual resource group, creating IP, network name and disk resources in the group, the following commands will create a group called v01, in the lab01 cluster. For these steps, you’ll need the drive letter to use (M: below), the disk signature, the public network name, IP Address and subnet mask of the virtual server being created:
1. cluster /cluster:lab01 group "v01" /create
2. cluster /cluster:lab01 res "v01 Disk01" /create /group:"v01" /type:"physical disk"
3. cluster /cluster:lab01 res "v01 Disk01" /priv Drive="M:"
4. cluster /cluster:lab01 res "v01 Disk01" /priv signature=0x%disksignature%
5. cluster /cluster:lab01 res "v01 Disk01" /prop Description="M: disk01"
6. cluster /cluster:lab01 res "v01 Disk01" /On
7. cluster /cluster:lab01 res "v01 IP" /create /group:"v01" /type:"IP Address"
8. cluster /cluster:lab01 res "v01 IP" /priv Network="%publicNetwork%"
9. cluster /cluster:lab01 res "v01 IP" /priv Address=192.168.10.10
10. cluster /cluster:lab01 res "v01 IP" /priv SubnetMask=255.255.255.0
11. cluster /cluster:lab01 res "v01 IP" /priv EnableNetBIOS=1
12. cluster /cluster:lab01 res "v01 IP" /priv OverrideAddressMatch=0
13. cluster /cluster:lab01 res "v01 IP" /AddDep:"v01 Disk01"
14. cluster /cluster:lab01 res "v01 IP" /On
15. cluster /cluster:lab01 res "v01" /create /group:"v01" /type:"Network Name"
16. cluster /cluster:lab01 res "v01" /priv RequireKerberos=1
17. cluster /cluster:lab01 res "v01" /AddDep:"v01 IP"
18. cluster /cluster:lab01 res "v01" /priv Name="v01"
19. cluster /cluster:lab01 res "v01" /On
13. Install ABEUIamd64.msi on each node if Access Based Enumeration is required
14. To create a test directory, share and ABE resource on the new virtual server on the cluster (v01):
1. md \\v01\m$\Dir01
2. cluster /cluster:lab01 res "v01 Dir01 Share" /create /group:"v01" /type:"File Share"
3. cluster /cluster:lab01 res "v01 Dir01 Share" /priv path="M:\Dir01"
4. cluster /cluster:lab01 res "v01 Dir01 Share" /priv Sharename=Dir01
5. cluster /cluster:lab01 res "v01 Dir01 Share" /priv Remark="Dir01 File Share"
6. cluster /cluster:lab01 res "v01 Dir01 Share" /prop Description="Dir01 File Share"
7. cluster /cluster:lab01 res "v01 Dir01 Share" /priv security=Everyone,grant,F:security
8. cluster /cluster:lab01 res "v01 Dir01 Share" /AddDep:"v01"
9. cluster /cluster:lab01 res "v01 Dir01 Share" /AddDep:"v01 Disk01"
10. cluster /cluster:lab01 res "v01 Dir01 Share" /On
11. cluster /cluster:lab01 res "v01 Dir01 ABE" /create /group:"v01" /type:"Generic Application"
12. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv CommandLine="cmd.exe /k abecmd.exe /enable Dir01"
13. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv CurrentDirectory="%SystemRoot%"
14. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv InteractWithDesktop=0
15. cluster /cluster:lab01 res "v01 Dir01 ABE" /priv UseNetworkName=0
16. cluster /cluster:lab01 res "v01 Dir01 ABE" /prop SeparateMonitor=1
17. cluster /cluster:lab01 res "v01 Dir01 ABE" /prop Description="Access Based Enumeration for Dir01 File Share"
18. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01"
19. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01 Disk01"
20. cluster /cluster:lab01 res "v01 Dir01 ABE" /AddDep:"v01 Dir01 Share"
21. cluster /cluster:lab01 res "v01 Dir01 ABE" /On
15. Additional shared cluster disks can be created as required, eg:
1. vmkfstools -d thick -a lsilogic -c 5G /vmfs/volumes/%datastore%/cluster01/MSCS-disk02.vmdk
2. Add the disks to one node, (scsi 1:2 in this example). Initialise and allocate in the cluster (as in step 7 above)
16. To create a virtual print server (assuming you’ve mounted disk02 from step 15 for use in the cluster):
1. cluster /cluster:lab01 group "v02" /create
2. cluster /cluster:lab01 res "v02 Disk02" /create /group:"v02" /type:"physical disk"
3. cluster /cluster:lab01 res "v02 Disk02" /priv Drive="P:"
4. cluster /cluster:lab01 res "v02 Disk02" /priv signature=0x%disksignature%
5. cluster /cluster:lab01 res "v02 Disk02" /prop Description="P: print01"
6. cluster /cluster:lab01 res "v02 Disk02" /On
7. cluster /cluster:lab01 res "v02 IP" /create /group:"v02" /type:"IP Address"
8. cluster /cluster:lab01 res "v01 IP" /priv Network="%publicNetwork%"
9. cluster /cluster:lab01 res "v01 IP" /priv Address=192.168.10.11
10. cluster /cluster:lab01 res "v01 IP" /priv SubnetMask=255.255.255.0
11. cluster /cluster:lab01 res "v02 IP" /priv EnableNetBIOS=1
12. cluster /cluster:lab01 res "v02 IP" /priv OverrideAddressMatch=0
13. cluster /cluster:lab01 res "v02 IP" /AddDep:"v02 Disk02"
14. cluster /cluster:lab01 res "v02 IP" /On
15. cluster /cluster:lab01 res "v02" /create /group:"v02" /type:"Network Name"
16. cluster /cluster:lab01 res "v02" /priv RequireKerberos=1
17. cluster /cluster:lab01 res "v02" /AddDep:"v02 IP"
18. cluster /cluster:lab01 res "v02" /priv Name="v02"
19. cluster /cluster:lab01 res "v02" /On
17. Create v02 print spooler:
1. cluster /cluster:lab01 res "v02 Spooler" /create /group:"v02" /type:"print spooler"
2. cluster /cluster:lab01 res "v02 Spooler" /priv DefaultSpoolDirectory="P:\Spool"
3. cluster /cluster:lab01 res "v02 Spooler" /prop Description="v02 Print Spooler"
4. cluster /cluster:lab01 res "v02 Spooler" /AddDep:"v02 Disk02"
5. cluster /cluster:lab01 res "v02 Spooler" /AddDep:"v02"
6. cluster /cluster:lab01 res "v02 Spooler" /On
18. On v02, add a standard Laserjet 4000 retail driver for x64 and x86, run from a cluster node:
1. rundll32 printui.dll,PrintUIEntry /ia /c \\v02 /m "HP LaserJet 4000 Series PCL6" /h "x64" /v "Windows XP and Windows Server 2003"
2. rundll32 printui.dll,PrintUIEntry /ia /c \\v02 /m "HP LaserJet 4000 Series PCL6" /h "x86" /v "Windows 2000, Windows XP and Windows Server 2003"
19. Create a test printer on v02 called printer01 using the LJ 4000 driver, with a record in DNS, published in AD, set to duplex by default, with customised permissions using the standard winprint processor:
1. dnscmd %DNSserver% /recordadd %zone% printer01 A 192.168.10.100
2. cscript //nologo portmgr.vbs -a -c \\v02 -p printer01 -h 192.168.10.100 -t LPR -q printer01
3. cscript //nologo prnmgr.vbs -a -c \\v02 -b printer01 -m "HP LaserJet 4000 Series PCL6" -r printer01
4. cscript //nologo prncfg.vbs -s -b \\v02\printer01 -h printer01 -l "%Location%" +published
5. setprinter.exe \\v02\printer01 8 "pDevMode=dmDuplex=2,dmCollate=1,dmFields=duplex collate"
6. subinacl /printer \\v02\printer01 /grant=%domain%\%group%=F
7. setprinter \\v02\printer01 2 pPrintProcessor="WinPrint"

References

VMware Support method of running MSCS clusters:
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_mscs.pdf

Implementing an MSCS 2003 server cluster Cluster
http://waynes-world-it.blogspot.com/2008/03/implementing-mscs-2003-server-cluster.html

subinacl 5.2.3790.1180:
http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B

Windows Server 2003 Resource Kit Tools:
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&DisplayLang=en

Wayne's World of IT (WWoIT), Copyright 2009 Wayne Martin.

Read more!

Finding duplicate DNS records

Internal DNS in many organisations is usually a little messy and if tasked with cleaning it, my first step would be to identify duplicate records in DNS - either an IP address that is recorded against multiple host names or a host name with multiple host (A) records. Assuming you have a file containing one record per line, this is fairly simple to report on.

The following examples use dnscmd.exe, which assumes Windows 2000/2003 DNS, but you could use any input, as long as it contains a name and an IP.

The two commands:

1. dnscmd %dnsServer% /enumrecords test.local @ /additional /continue | find /i /v "Aging:" | find /i "192.168" > DNSRecords.txt
2. echo. > DuplicateIPs.txt & (for /f "tokens=1,4" %i in (DNSRecords.txt) do @if "%j" NEQ "" @find /i "%j" DuplicateIPs.txt >nul & if errorlevel 1 for /f "tokens=1" %m in ('"findstr /i "%j$" DNSRecords.txt | find /i /c "%j""') do @if %m GTR 1 (@echo %j,%m: & findstr /i "%j$" DNSRecords.txt & echo.) >> DuplicateIPs.txt) & type DuplicateIPs.txt

The first command uses dnscmd to enumerate the records from the root of the test.local zone and exclude aging (dynamic DNS) records. The last find command further filters the output by IP, which can be useful when targeting specific subnets/sites. You might also want to check aging dynamic records instead of static to see how well scavenging is working – if you do, just change the tokens to 1,5 instead of 1,4 (as the aging data is another token separating the name and IP).

The second command:

1. Creates a new file called DuplicateIPs.txt in the current working directory
2. Iterates through each line in the DNS record dump, extracting the first and fourth token (Name and IP)
3. The commands in the first FOR loop check there is a value, then check the value hasn't already been processed in the duplicate list (otherwise you'd have duplicates of each duplicate) and then counts the duplicates and appends them to the file.
4. The final command types the duplicate IP file created by the for loop iteration.

For example, supposing your DNS export contained the following records:

printer1 3600 A 192.168.10.100
printer2 3600 A 192.168.10.101
printer3 3600 A 192.168.10.100
printer4 3600 A 192.168.10.102
printer5 3600 A 192.168.10.103
printer6 3600 A 192.168.10.100
printer7 3600 A 192.168.10.102

After running the second command above, a file called DuplicateIPs.txt would be created and then typed to the prompt:

192.168.10.100,3:
printer1 3600 A 192.168.10.100
printer3 3600 A 192.168.10.100
printer6 3600 A 192.168.10.100

192.168.10.102,2:
printer4 3600 A 192.168.10.102
printer7 3600 A 192.168.10.102

The command below is modified to report duplicate names instead of IP addresses. This was done by using the first token (%i) instead of the second (%j), and modifying the findstr command to use a literal string search ending with a space rather than the regular expression EOL:

echo. > DuplicateIPs.txt & (for /f "tokens=1,4" %i in (DNSRecords.txt) do @if "%i" NEQ "" @find /i "%i" DuplicateIPs.txt >nul & if errorlevel 1 for /f "tokens=1" %m in ('"findstr /i /c:"%i " DNSRecords.txt | find /i /c "%i""') do @if %m GTR 1 (@echo %i,%m: & findstr /i /c:"%i " DNSRecords.txt & echo.) >> DuplicateIPs.txt) & type DuplicateIPs.txt

If you wanted a summary rather than the detail of each duplicate, you could also run the following command:

echo. > DuplicateIPSummary.txt & (for /f "tokens=1,4" %i in (DNSRecords.txt) do @if "%j" NEQ "" @find /i "%j" DuplicateIPSummary.txt >nul & if errorlevel 1 for /f "tokens=1" %m in ('"findstr /i "%j$" DNSRecords.txt | find /i /c "%j""') do @if %m GTR 1 (@echo %j,%m) >> DuplicateIPSummary.txt) & type DuplicateIPSummary.txt

In the example above, this would produce the following report:

192.168.10.100,3
192.168.10.102,2

I use this sort of command to generate reports on duplicates, in this case from DNS, but it could also be useful in DHCP, WINS, or any number of Active Directory objects/attributes. People (myself included) are often wary of automated processes that make changes, but this is an excellent example of how powerful read-only automated commands can be – you can take thousands of objects and produce a report in seconds to quickly identify inconsistencies in an environment.

Dnscmd Overview
http://technet.microsoft.com/en-us/library/cc778513.aspx

Read more!

Command-line automation – Echo and macros

This is the fifth and last in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the echo command and how to execute your commands using macros with doskey.

Echo

Echo is one of the most important commands to use when constructing repeatable commands – you should never run a command against multiple objects without first using echo. Some commands can be disastrous without appropriate input checking.

For example, using a wildcard incorrectly in a file spec, instead of putting *.txt, below I’ve put ‘* .txt’ – the wildcard selecting everything in the current directory (with the unfortunate del command for each of those files).
for %i in (* .txt) do echo del %i

Or more commonly – an incorrectly formed control file. The following command is one I’ve used to fix invalidly set UPNs on user accounts in the local directory. This command is quite powerful, and can be used to fix dozens or hundreds of accounts in seconds – but if the input file is not well formed, you could end up breaking each account:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix%

Incidentally, the command used to generate the ‘NoUPN.txt’ file was:
dsquery * %OU% -filter "&(objectclass=user)(objectcategory=person)(!(userprincipalname=*))" -s %server% -scope onelevel -attr name distinguishedname > NoUPN.txt

If you are untrusting of using the ‘for /f’ to execute operations against multiple objects, you can also use echo to construct a batch file that you can run at a later time – good practice rather than writing and executing the command in a single attempt. This may also useful when you’re modifying hundreds or thousands of objects and you don’t want to execute the commands immediately – for example if the modifications were in Active Directory and you wanted to prevent excessive replication during the day.

Running the command above with an appended redirect (>>) to a file, you could then pass this file to someone else for sanity checking before actual execution of the batch job:
for /f "skip=1 tokens=1,2-3" %i in (NoUPN.txt) do echo dsmod user "%j %k" -upn %i@%upnsuffix% >> %temp%\SetUPN.bat

Doskey macros

In the previous post there was a command to calculate a subnet at the command-line. If you had to type it every time - or even copy and paste - it’s not something you would use (okay, you’d probably never use it anyway), but doskey macros provide a simple method of allowing command re-use.

Creating a macro file

A text file can be created containing your macros, choose a shorthand name for the command, followed by the command you wish to execute. One slightly odd aspect with a macro file is that variables passed at the command-line for the macro use the $ prefix instead of %.

To make macros more useful, I often add two options – one without any parameters with a defined default, and another accepting a parameter to customise the command. For example, the following macro - called ‘AD2003’ - would by default query the local directory for all 2003 computer accounts. If a parameter is passed ($1), the dsquery command passes the parameter as the start node for the search (which could be a DN to limited the containers queried, or a different domain than domainroot).

AD2003=if _$1==_ (dsquery * domainroot -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0) else (dsquery * $1 -filter "(&(objectCategory=Computer)(objectClass=Computer)(operatingSystem=Windows Server 2003))" -limit 0)

The find subnet command from the previous post could be shortened to execute the command with something as simple as ‘fs 192.168.11.40 255.255.254.0’, by adding the following to a macro file (and executing on a command processor with delayed expansion enabled):

FS=for /f "tokens=1-8 delims=.- " %i in ('echo $1 $2') do @set /a Octet1="%i & %m" >nul & @set /a Octet2="%j & %n" >nul & @set /a Octet3="%k & %o" >nul & @set /a Octet4="%l & %p" >nul & Echo !Octet1!.!Octet2!.!Octet3!.!Octet4!

Automatically enabling the macro file

It would also be unwieldy if you had to enable the doskey macros each time you ran a command shell. However, you can easily automate this with the autorun key of the shell, stored in either

HKCU or HKLM:
Key - HKEY_LOCAL_MACHINE\software\microsoft\Command Processor
Value - AutoRun
Value Type - REG_SZ
Value Data - doskey /macrofile=c:\util\macros.txt (or wherever)

This concludes what hopefully was a useful description of how I use built-in shell commands to automate repetitive tasks and make my work life easier and more efficient. Some of the resultant commands are not particularly easy to follow, but each can be broken down into the component parts and digested separately.

For more real-world examples of how I use the information in the last five posts, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – set

This is the fourth in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘set’ command, which I find very useful when automating repetitive tasks with ‘for’. I use set in four ways – string manipulation, simple arithmetic operations with ‘set /a’, gathering input with ‘set /p’ or concatenating output in a single for loop with delayed variable expansion.

String Manipulation

My most frequent use of the set command is simple string manipulation at the command prompt. I use this to replace one character with another or extract a substring.

Replace

Replacing one string with another is a core function of string manipulation. Even though this is included under help for ‘set’, this functionality works for other commands, including the echo command.

For example, say you have the fully qualified user DNS name – my.domain.com – and this conveniently is also the distinguished name of your AD domain, the following command replaces periods with ‘,DC=’ and prefixes an additional string to provide the distinguished name:
echo DC=%userdnsdomain:.=,DC=%

This would turn ‘my.domain.com’ into:
DC=my,DC=domain,DC=com

Wildcards can also be used, for example, to remove the first element, you could replace up to the first dot with nothing:
echo %userdnsdomain:*.=%

You could also remove spaces and print the output, for example, if the ‘test’ variable contained ‘A Sentence With Spaces’, running:
echo %test: =%

Would result in:
ASentenceWithSpaces

Substrings

You can extract from existing variables (or inline if using delayed expansion), providing Left/Right/Mid functionality. This can be useful in many ways, but for most of the input I deal with, splitting with ‘for’ and delimiters (such as comma or space) makes for simpler processing with dynamic input. For examples sake, if you had a defined length of padded text, and you wanted the 10 characters starting at the 20th character (0-based), you could run:
echo %input:~19,10%

You could also extract the last 10 characters:
echo %input:~-10%

Or all but the last 10 characters:
echo %input:~0,-10%

Arithmetic operations

‘set /a’ supports a relatively simple but still powerful set of arithmetic, bitwise and shift operators, and while I’ve never found much use for them, upon occasion they have come in handy.

For example, this command will tell you what next year will be and set the NextYear environment variable, based on the current year + 1:
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1

Similarly, find the current month, increment by one, unless the current month is 12, then wrap around and start at 1 again:
for /f "tokens=2 delims=/" %i in ('echo %date%') do if %i==12 (Set NextMonth=1 & echo !NextMonth!) else (Set /a NextMonth=%i+1)

Note that the example above uses delayed environment variable expansion, see below for a description.

Incidentally, if you were going to use this output for something and if the month were a single digit you would lose the padding, which would then be incorrectly sorted (as compared to other padded output). The example below pads the month, then extracts the last two characters, effectively padding single-digit months (see string manipulation above):
set NextMonth=0%NextMonth%
echo %nextmonth:~-2%

An actual use I have found for the arithmetic operators is to calculate subnets from the command-line. I have made use of this when parsing netsh dhcp output (there were thousands of leases across several DHCP servers serving dozens of subnets, and I wanted to know from the leases which subnets were in use).

This can be done with a single command-line, but it’s a little complicated, the batch file below may be easier to demonstrate the ‘set /a’ functionality. Running this batch file with an IP and mask, eg ‘findsubnet.bat 192.168.11.40 255.255.254.0’ will return the subnet (192.168.10.0 in this example) using bitwise AND of the two sets of octets:

@Echo Off
Set Input=%1
Set Mask=%2
If /i "%Input%"=="" Goto End

for /f "tokens=1-8 delims=.- " %%i in ('echo %Input% %Mask%') do Call :FindOctet %%i %%j %%k %%l %%m %%n %%o %%p
Goto End

:FindOctet
::Echo %1 %2 %3 %4 %5 %6 %7 %8

set /a Octet1="%1 & %5" >nul
set /a Octet2="%2 & %6" >nul
set /a Octet3="%3 & %7" >nul
set /a Octet4="%4 & %8" >nul
Echo %1.%2.%3.%4,%Octet1%.%Octet2%.%Octet3%.%Octet4%,%5.%6.%7.%8
Set Octet1=
Set Octet2=
Set Octet3=
Set Octet4=
Goto End

:InvalidArgs
Echo Invalid arguments, please provide an IP and longhand mask, eg 192.168.11.40 255.255.254.0
Goto End

:End

Input

Set can be used to gather input, making your commands or batch files interactive. For example, the following command assigns the input to the ‘test’ environment variable, after asking the question:
set /p test=Tell me something?

I don’t often use this, as it somewhat defeats the purpose of automation – I prefer to have all the required input in a control file, which separates the intended action from the actual operator.

Set with Delayed expansion

Delayed environment variable expansion provides a method of executing commands and storing the output for use by other commands within a loop – either in different iterations or nested commands. This is required when looping through ‘for’ commands or nesting commands, otherwise state between each command is not maintained.

For example, without delayed expansion, the following would not work:
set test=test variable & echo %test%

However, using delayed environment variable expansion, this problem can be easily overcome, using the !variable! syntax, instead of the normal %variable%. Note that you may need to enable delayed environment variable expansion, which can be done using ‘cmd /v:on’ when starting the command prompt, or running ‘setlocal ENABLEDELAYEDEXPANSION’ within an existing shell.

With delayed expansion, the first command would set the variable and the second echo the newly set variable:
set test=test variable & echo !test!

Or to re-use an example above, calculate next year and then do something with that information (echo in this case):
for /f "tokens=3 delims=/" %i in ('echo %date%') do set /a NextYear=%i+1 1>nul & echo Next year is !NextYear!

Unless there is already a variable of the same name, running the commands above using the percent signed variables will result in the variable name being printed, as the environment hasn’t yet been changed by the first command.

Note that while testing, it seems the Vista command shell has been updated and as long as delayed expansion is enabled, this command would work:
set test=test variable & echo %test%

The following command is a one-line version of the batch above, using delayed variable expansion to calculate the variables as each command is executed:
for /f "tokens=1-8 delims=.- " %i in ('echo 192.168.11.40 255.255.254.0') do set /a Octet1="%i & %m" >nul & set /a Octet2="%j & %n" >nul & set /a Octet3="%k & %o" >nul & set /a Octet4="%l & %p" >nul & Echo %i.%j.%k.%l,!Octet1!.!Octet2!.!Octet3!.!Octet4!,%m.%n.%o.%p

An easier example to follow, set the environment variable called ‘number’ to a random number (using %random% - very useful), echo the newly set variable, and then use ‘set /a’ to increment the number by one:
set number=%random% & echo !number! & set /a number=!number!+1

If you are using nested ‘for’ commands, you can just reference variables from an earlier command, but in some circumstances delayed expansion is the only possibility. I have had to use this syntax when using ‘set /a’ and using the replace/substring functionality, as this requires a full variable (eg. %path%, not a temporary %i type variable.

This was a basic overview of ‘set’ and some of the ways in which I use this command when automating tasks at the command-line.

For more real-world examples of how I use ‘set’, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – errorlevels and if

This is the third in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are errorlevels, and if/then/else statements, useful for branching in a single command-line to execute based on the error level set or some other condition.

Using errorlevels, concatenated commands and if/then/else statements provides enhanced functionality, and most complex operations would not be possible on a single command-line without these features.

errorlevels

In the context of command-line automation, I find error levels most useful when trying to cater for both sets of outcomes when executing a command – either what you wanted happened, or it didn’t.

In the previous examples, I’ve used a control file to ping one or more machines and then echoed the IP address of those that were contactable. This is only part of the picture – what about those machines in the list that didn’t respond to a ping? They are simply dropped from the output. This may be what you are after, but normally I would prefer to know the positive and the negative.

To work around this problem, the following command could be used:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do echo %i

This will ping ‘computer’, echo the IP Address if found, otherwise echo NotFound. This occurs because the find command sets the errorlevel according to whether the string was found, and then we are concatenating the command with another to check if an errorlevel occurred indicating the string was not found, and echoing an alternate output to be picked up by the body of the for loop.

A simpler way of looking at this may be without the ‘for’ loop:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)

The command is similar to the example above, it will ping ‘computer’, and if the word reply was found in the output, it will print ‘replied’, otherwise it will print ‘no reply’.

Command concatenation and if/then/else

Commands can be concatenated – when running natively or inside a ‘for’ loop – providing a simple method to group commands. For example, looping through a control file, you might want to nslookup and then ping each device:
for /f %i in (c:\temp\control.txt) do nslookup %i & ping %i

This is a simple chain of commands, execute one and then the other. However, provided each command returns a relevant errorlevel, the if/then/else syntax can also be used to execute the second only if the first succeeded.

Unfortunately nslookup does not provide an errorlevel, but the find command does:
ping -n 1 computer | find /i "reply" >nul & if errorlevel 1 (echo No Reply) else (echo Replied)
ping -n 1 computer | find /i "reply" >nul && echo Replied || echo No Reply

The above commands return the same result – they each print a statement depending on whether ‘reply’ was found in the ping output, the second using the shorthand ‘if’ (&&) and ‘else’ (||) syntax.

‘If’ can be used to check the errorlevel (as in the above example), check whether one string equals another, and check whether a file/directory exists.

If ErrorLevel checking

The following examples send an invalid service a stop control, and perform various checks on the errorlevel returned. Note that sc.exe on XP does not return error codes, this was Vista which does at least return 1060 when the specified service does not exist.

Check whether errorlevel (a special variable that can either be enclosed in % or not) was exactly 1060:
sc stop invalidservice >nul & if errorlevel==1060 echo Service not found

Check whether errorlevel was exactly 1060, if so echo ‘service not found’, otherwise echo the errorlevel returned:
sc stop invalidservice >nul & if errorlevel==1060 (echo Service not found) else (echo Error code %errorlevel% returned)

Check whether the errorlevel was 1060 or greater, and not 1061 (therefore only 1060).
sc stop invalidservice >nul & if errorlevel 1060 if not errorlevel 1061 echo Service not found

Check if anything above errorsuccess (0) was returned, echoes an error occurred and then uses ‘net helpmsg’ to return the error description for the specified error code:
sc stop invalidservice >nul & if errorlevel 1 echo An error occurred: %errorlevel% & net helpmsg %errorlevel%

If string checking

If can be used to conditionally check two strings to determine the action taken. Note that if the string could be blank, you’ll need to either append/prepend another character or enclose the string in quotes.

Using the above example of pinging a computer, check the response and branch accordingly (different echoes in this case):
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer | find /i "reply" & if errorlevel 1 echo 0:1:Not-Found"') do if "%i"=="reply" (echo Ping success) else (echo Ping failed)

Note that ‘if /i’ can be used for a case insensitive comparison.

Additionally, if command extensions are enabled, and the strings in comparison are numeric, both are converted to numbers and additional operators are available for the comparison, for example:
If 9 equ 9 echo Equals
If 5 lss 14 echo Less than

If exist

To determine whether a file or directory exists, the ‘if exist’ syntax can be used. This can be useful when checking flag files or control files exist to avoid encountering unexpected errors.

If exist %windir% echo %windir% exists
If not exist c:\temp\control1.txt (echo Control File does not exist, terminating & break) else (echo continuing)

Some other examples of concatenating commands which help to demonstrate the concepts:

Use ‘sc.exe’ to send a service a stop control, if the service was running (exists, security allows etc) and was sent a stop, sc will return errorsuccess (0), pause for 5 seconds and then restart the service, otherwise echo that the service was not running in the first place:
sc stop "service" && (echo error %errorlevel% & sleep 5 & sc start "service") || echo Service was not running, error %errorlevel%

nslookup www.microsoft.com, find the second address (the first in nslookup is the DNS server address, and echo that address:
for /f "skip=1 tokens=2" %i in ('"nslookup www.microsoft.com 2>nul | find /i "address""') do echo %i

nslookup www.microsoft.com, skip the first DNS server address (the second line of the nslookup output – find /n), return the other address if found, otherwise return ‘Not-Found’ in the %i variable – the second token of the string:
for /f "tokens=2" %i in ('"nslookup www.microsoft.com 2>nul | find /i /n "address" | find /i /v "[2]" & if errorlevel 1 echo Address Not-Found"') do echo %i

Create a file, if the file is found type the contents, and then delete the file if the type succeeded, otherwise report that the ‘type’ command failed:
echo test > %temp%\test.txt & if exist %temp%\test.txt type %temp%\test.txt && del %temp%\test.txt || Echo could not type %temp%\test.txt

This was a basic overview of the errorlevels and if/then/else branching, future posts will build on this with other useful commands and nested ‘for’ loops.


For more real-world examples of how I use errorlevels and if/then/else, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Command-line automation – find and findstr

This is the second in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used to control data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post are the ‘find’ and ‘findstr’ commands - unless you want to spend all your time manually creating or filtering control files you will often use these commands when constructing repeatable command-lines.

These two commands are crucial in efficient ‘for’ loop processing, invaluable when filtering a control file or the results of an in-line command. Help on each is available from the command prompt with the ‘/?’ argument.

Find

I usually use find because more often than not I want to filter a string literal – a specific word or combination of characters – from an input string. As often as not, I also use ‘find /v’, returning the lines that don’t match the specified string.

For example, if you have a control file listing servers in several geographic locations, but you are only interested in processing servers in ‘bne’ (case insensitive with ‘/i’), you could run:
find /i "bne" c:\temp\control.txt

Combined with a ‘for’ command, this could be:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do echo %i

Note that this line uses the following syntax explained in the previous post on ‘for’:

1. ‘Skip=2’ tells the loop to skip the first two lines – the result header of the ‘find’ command
2. Inside the brackets I’ve specified to run a command and parse the results, wrapping the command in single-quotes.

The results are that for each server containing the word ‘bne’, the body of the ‘for’ statement will be executed – in this case simply to echo the first token.

You might also want to run the commands against all servers except those in bne, which is the above command repeated with the addition of the ‘/v’ switch to tell ‘find’ to return those lines not containing a match:
for /f "skip=1" %i in ('find /i /v "bne" c:\temp\control.txt') do echo %i

Filtering in-line output with find

Filtering output of a command executed within a ‘for’ loop is syntactically a little more complicated, but makes sense once you get the hang of it. For example, suppose you want to filter the output of a ping command, returning the IP address of those hosts that replied to a ping, you could run:
for /f "tokens=3" %i in ('"ping -n 1 computer find /i "reply""') do @echo %i

This will:

• Execute the ping command, pinging the host named computer, filtering the output to find the word ‘reply’ (indicating a successful reply). Note that to use the pipe inside the brackets, the whole string needs to be enclosed in double-quotes – inside the single-quotes.
• Using spaces/tabs, choose the third token to return in the first variable - %i. This will be the IP address in the standard ping output returned by XP and Vista.

Unfortunately the third token of the ping output includes a irrelevant colon, so to strip that, we could specify colons and spaces as delimiters in the for loop:
for /f "tokens=3 delims=: " %i in ('"ping -n 1 computer find /i "reply""') do echo %i

This will be better explained in a future post, but you can also nest two ‘for’ commands and for each host in the control file (in bne in this example), ping and return the IP address if there is a reply:
for /f "skip=2" %i in ('find /i "bne" c:\temp\control.txt') do @for /f "tokens=3 delims=: " %m in ('"ping -n 1 %i find /i "reply""') do @echo %i,%m

The output would give you a comma-separated list of server names and IP address of those from the original control file that currently respond to a ping. Theoretically not all that useful with servers, as they’d usually be known IPs and always on, but if you’re wanting to check something on hundreds or thousands of workstations this would give you a point-in-time control file of online machines (assuming name resolution is accurate).

It’s also worth noting that ‘find’ commands can be cascaded – piping the output of one find command to another if you need to further refine the output. For example to find servers in ‘bne’ and then servers in ‘bne’ that also contain ‘vm’:
find /i /c "bne" c:\temp\control.txt find /i "vm"

The possibilities are nearly endless, I use ‘find’ to:

• Filter in and out items that I want, ‘find /i’ and ‘find /i /v’ from generic control files that I use for multiple operations, either in-line in a for command, or to create a stand-alone filtered control file
• Run commands and then filter out useless or useful information, often headers/footers or other extraneous detail, making it possible to list data, filter that data and run a command against each filtered item in a single command.
• Use errorlevels to determine whether the find command was successful or not (explained in a future post). Often the only way of knowing whether a command returned what you were after is the presence or absence of a string – and ‘find’ will tell you this based on the errorlevel (an advantage over findstr which in XP at least does not set the errorlevel).

Note that find also has several other features that are useful in some scenarios:

• By default, offline files will be skipped – if you use a file archiving product that sets the offline (‘O’) attribute, these files will not be searched unless you specify the /offline parameter
• The ‘find /n’ option will return the line number the match was found, occasionally useful when parsing large or sequenced control files and you want to know where in a file the string was found.
• The ‘find /c’ option will return a count of the matches found, instead of the actual matching lines. This can often be useful when determining how many records would be processed. For example, in the example above you could first run the following command to determine how many records would be processed with the for loop:
  find /i /c "bne" c:\temp\control.txt

Findstr

These same principles can be used with findstr to filter output – findstr provides advanced searching, with the following benefits in my use of the commands, you can:

• Search for more than one string in a single command. This simplifies searches, in the example above, you could search for ‘bne’ and ‘syd’ in a single findstr command.
• Use basic regular expressions and/or string literals

In the example above with a control file containing a list of servers with geographic names (or descriptions somewhere on the line), you could run the following command to return servers in both syd and bne and then run the subsequent command against each server:
for /f %i in ('findstr /i /c:bne /c:syd c:\temp\control.txt') do echo %i

Regular expressions are used to group similar items, without expressly providing each possible combination. For example, parsing robocopy log output, to search for new directories that have been created with a regular expression, you could run:
findstr /i /n "^.*New.Dir" Robocopy.log"

You could just use ‘find’ and search for the string ‘new dir’, but this could easily return false positives if a file or directory has the string ‘new dir’ in it. Using the regular expression looking for the start of the line followed by whitespace followed by the ‘New Dir’ is much more precise.

This was a basic overview of the ‘find’ and ‘findstr’ commands, future posts will build on this foundation with other useful commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘find’ and ‘findstr’ commands, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Building blocks of command-line automation – FOR

This is the first in a series of posts containing information on what I consider the building blocks to automate repetitive tasks at the Windows command-line. These components are the for, find, findstr, set, if and echo commands, control files used for data input, combined with errorlevels, command concatenation, nested loops and if/then/else constructs.

Described in this post is the ‘for’ command, the most important component in command-line automation. This command provides several methods of looping through a list, and running a command against each element in that list. Use the ‘for /?’ to get Microsoft help on this command.

Using the ‘for’ command with one of the syntaxes below provides many benefits, including:

• Repeatability – If you save the command you run, it can be re-run several or hundreds of times
• Change control and testing – It’s easy to record what you are planning and simple to run the same commands in a test-lab environment. The output of commands can also be redirected to file, making accountability much easier. Using control files also provides a straightforward method of recording the targets of various actions, knowing that you have one master list, and do not risk accidentally missing an entry.
• Documentation – Implementation using a series of commands very easily translates to an as-built document – with the added benefit of providing a quicker DR process.
• Efficiency – Even though designing the command for the first run may not be as quick as using the GUI, every time after that will usually be much quicker, and previous commands can often be quickly adapted to new tasks.

You can use the ‘for’ command to:

Process a set of filenames or string literals

The filenames or literals can either be directly named in a space-separated set, or you can use wildcards to process more than one file. For example:
for %i in (*.txt) do echo %i
for %i in (test1.txt test2.txt) do echo %i

For example, I would use this syntax if I’m trying to:

Quickly execute something against a group of machines, eg ping each machine:
for %i in (server1 server2 server3 server4) do ping -n 1 %i

Process a series of data files that I have created from another process, eg this uses the regview utility to export the registry entries modified by *.pol files into *.txt:
for %i in (*.pol) do regview %i > %i.txt

Quickly execute the same command with a different variable, eg use the setprinter utility to view all levels of configuration for the specified printer (you could also use for /l in this example):
for %i in (0 1 2 3 4 5 6 7 8 9) do setprinter -show \\server\printer %i

Process a set of directories

The directory names can either be directly named in a space-separated set, or you can use wildcards to process more than one directory. For example:
for /d %i in (%windir%\*) do echo %i
for /d %i in (c:\temp c:\windows) do echo %i

I would use this syntax if I’m trying to do something with each top-level directory, for example:

Report or set ACLs:
for /d %i in (%rootDir%\*) do icacls %i /save %~ni.txt

Rename all the top-level directories to start with a new prefix:
for /d %i in (%rootDir%\*) do ren %i New-%~ni

Process the contents of a text file, line by line

The contents of a file – which I usually refer to as a control file – can be read line-by-line and your command would be run once for each line, substituting tokens from the control file. This provides unlimited capability – construct a control file through any means available and you can then process the entries one-by-one and run a command against that entry.

Note that in Vista at least, just a LF is enough to separate the lines, rather than the Windows standard CR+LF.

For example, assuming you have a control file with a list of servers or workstations, you could:

Lookup the IP address of each machine:
for /f %i in (test.txt) do nslookup %i

Ping each machine:
for /f %i in (test.txt) do ping %i

Remote dir of each machine:
for /f %i in (test.txt) do dir \\%i\c$

I use this constantly to run a command against multiple AD objects, machines, printers, or other network devices, whether the command queries or checks something, or makes a change to each device.

Process the results of a command, line by line

The results of almost any command can be used as the looping mechanism for a ‘for /f’ command, providing an in-memory control file. For example, you could:

Find the local hostname and then nslookup the computer (you could also use %computername% for this):
for /f %i in ('hostname') do nslookup %i

Query the local Active Directory for a list of DCs (server records) and lookup the IP of each DC:
for /f %i in ('dsquery server -o rdn') do nslookup %i

Recursively enumerate a path

It’s possible to recursively enumerate files or directories from a specified starting location, passing each to the body of the for loop. This provides a rudimentary search and response facility, allowing you to traverse a tree looking for objects of a particular type – and then execute something for each found.

For example, you could search from the root of C: for *.txt files, and then report the filename and size (you would just use dir /s if all you wanted to do was echo)
for /r c:\ %i in (*.txt) do echo %i %~zi

Step through a sequence of numbers and execute for each

The ‘for /l’ option allows stepping through a sequence of numbers, passing the number as a parameter to the body of the ‘for’ loop.

I don’t use this method very often, but it would be another method to the setprinter command above:
for /l %i in (1,1,9) do setprinter -show \\server\printer %i

Variable Expansion

When using ‘for’, ‘for /f’ and ‘for /d’ variable references can also be modified to return substrings or additional information. Note that when using ‘for /f’, most of these only make sense if you are processing lists of files or directories, but if you did have a control file with files/paths variable expansion does work as expected.

This substitution can be very useful, particularly when constructing parameters to pass to the command in the body of the ‘for’ loop. For Example:

If you had a number of control files that you wanted to process, outputting the results to a same-named log file:
for %i in (c:\temp\*.txt) do echo Do something with %i > %~ni.log

If the output of a previous command wrapped the results in quotes, but you need to append/prepend something else you can easily remove surrounding quotes:
for /f %i in ('echo "c:\windows"') do echo %~i\temp

Given a list of files, echo those that are zero bytes in size:
for %i in (c:\temp\*.txt) do @if %~zi == 0 @echo %i

Given a full path, split into drive, path and filename:
for %i in (c:\windows\temp\test.txt) do echo %~di %~pi %~nxi

Tokens, delimiters and skipping lines

The simple functionality of the ‘for /f’ command can be extended very easily with three options:

1. Tokens – By default only the first token is returned in the variable specified. You can change this behaviour to return one or more tokens, eg tokens=2,3,5 or tokens=1-3 would populate %i, %j and %k with the respective tokens
2. Delimiters – Instead of the normal space and tab delimiters, one or more alternate characters can be specified. For example, you can specify a comma as a delimiter to process as CSV file
3. Skipping lines – the skip command can be used to skip one or more lines from the start of a control file, useful when trying to skip a header line, or bypass logo information in a command result.

These options can be used individually or as a combination, for example:

Skip the first line of the control file:
for /f "skip=1" %i in (test.txt) do echo %i

Skip the first line, and use comma’s as the delimiter:
for /f "skip=1 delims=," %i in (test.txt) do echo %i

Skip the first two lines, use the second token, separated by comma and space:
for /f "skip=2 tokens=2 delims=, " %i in (test.txt) do echo %i

This was a basic overview of the ‘for’ command, future posts will build on this foundation with multiple commands, error levels, if/then/else statements and nested ‘for’ loops.

For more real-world examples of how I use the ‘for’ command, see my command-line operations:
http://waynes-world-it.blogspot.com/2008/09/useful-ntfs-and-security-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-printer-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-windows-mscs-cluster-command.html
http://waynes-world-it.blogspot.com/2008/09/useful-vmware-esx-and-vc-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-general-command-line-operations.html
http://waynes-world-it.blogspot.com/2008/09/useful-dns-dhcp-and-wins-command-line.html
http://waynes-world-it.blogspot.com/2008/09/useful-active-directory-command-line.html

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

Useful PowerShell command-line operations

The commands below are a small but growing list of powershell one-liners I've used (and when I remember to take note). Note that this may not be the best and almost certainly isn't the only way in most cases.

Each command-line can be copied and pasted at a PowerShell command prompt, or you can use the commands as part of a PS1 script file if you prefer.


Show help on commands such as 'if', regular expressions etc
help about*

Show help about about_Automatic_variables such as $_ $DebugPreference $OFS
help about_Automatic_variables

Run a command and pipe the results to a variable
cscript.exe //nologo script.wsf | Set-Variable -name scriptResults

Run a command and set a variable with the output
$scriptResults = cscript.exe //nologo script.wsf

Run a command and set a variable with the output using the call operator
$scriptResults = & cscript.exe //nologo script.wsf

Filter an object based on an array of strings
$array = "a", "b"; write-output a b c d | select-string -pattern $array -simpleMatch

Reformat today's date to YYYYMMDD
$today = [DateTime]::Now.ToString("yyyyMMdd")

Find local disks except C: using powershell and output to CSV
Get-WmiObject -Namespace root\cimv2 -ComputerName %server% -Query "SELECT * from Win32_LogicalDisk WHERE FileSystem='NTFS' AND Description = 'Local Fixed Disk' AND Name != 'C:'" | export-csv c:\disk.csv

Use the VI Toolkit Powershell snap-in to query for snapshots
Get-VM | Get-Snapshot | export-csv -path c:\temp\VMsnapshots.csv

Use the VI Toolkit Powershell snap-in to query for snapshot information
Get-VM | Get-Snapshot | foreach-object {$out= $_.VM.Name + "," + $_.Name + "," + $_.Description + "," + $_.PowerState; $out}

Start a remote process using Powershell/WMI
$computer = "."; ([WMICLASS]"\\$computer\root\CIMv2:win32_process").Create("notepad.exe")

Find remote processes and the command-line parameters with PowerShell
Get-WmiObject win32_process | Format-Table ExecutablePath,Caption,CommandLine,CreationDate,WorkingSetSize,ProcessId

Generate a password 10 characters in length, at least 5 punctuation marks
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Web") ; [System.Web.Security.Membership]::GeneratePassword(10,5)

Format a date to XML XSD:dateTime
$Now = [DateTime]::Now.ToString("yyyy-MM-ddTHH:mm:ss")

Check if the given variable is an object
if ($variable -is [object]) {}

Check which network connections (drive mappings) a 2003 computer has
get-WMIObject -computerName "$env:computername" -class win32_logicaldisk -property ProviderName, DeviceID -filter "DriveType=4" | Format-List -property ProviderName, DeviceID

See whether an IP can be resolved to a host name
[System.net.Dns]::GetHostEntry($IP)

See whether a host name can be resolve to an IP
$ip = & { trap {continue}; [System.net.Dns]::GetHostAddresses($computer) }

View the datastore information for the available storage
get-vc -server $server ; Get-Datastore

Check if a file or path exists or not
test-path -path $path ; if (!(test-path -path $path)) {write-output "$path does not exist"}

Use psexec to run a powershell command remotely and report to stdout (eg query 'remote' event logs in this example)
psexec \\machine /s cmd /c "echo. | powershell . get-eventlog -newest 5 -logname application | select message"



Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!

MSCS 2003 Cluster Virtual Server Components

This post provides my interpretation of a simple MSCS 2003 virtual server with a file share, including how the cluster interacts with the OS and network services to provide access to the share. This follows on from the last post on low-level detail of file access in an attempt to provide a clearer picture of these often taken-for-granted components.

Note that this is only my opinion, based on less-than complete knowledge and more than likely contains semantic errors if nothing else.

File & Print Cluster Native x64 (EM64T/AMD64)

1. Cluster Service. Includes Checkpoint Manager, Database Manager, Event Log Replication Manager, Failover Manager, Global Update Manager, Log Manager, Membership Manager, Node Manager.
1. Operating System Interaction with the LanManServer Service which advertised shares.
2. NetBIOS registration of the virtual server name through existing network services
3. DNS registration of the virtual server name through existing network services
4. Kerberos SPNs registered against an AD computer account through Active Directory
2. Resource Monitor. Spawned child process of the cluster service, separate resource monitors can exist for resource DLLs
3. ClusRes.dll Physical Disk <-> IsAlive/LooksAlive SCSI reservation and directory access. LooksAlive issues a SCSI reservation every 3 seconds through ClusDisk.sys against all managed disks. IsAlive performs a ‘dir’ equivalent
4. ClusRes.dll Network Name <-> IsAlive/LooksAlive check on NetBT/DNS registration. LooksAlive relies on MSCS NIC failure detection. IsAlive queries local TCP/IP stack for virtual IP and the NetBT driver if NetBIOS is enabled
5. ClusRes.dll IP Address <-> IsAlive/LooksAlive check on cluster NIC. LooksAlive queries NetBT driver and every 24 hours issues a dynamic DNS host record registration. If ‘DNS is required’ resource will fail if DNS registration fails. Same test for IsAlive
6. ClusRes.dll Resource DLL File Shares <-> IsAlive/LooksAlive check on file share visibility. LooksAlive queries lanmanserver service for the share name. IsAlive does the same, and if unlimited users, the first file on the share is copied
7. 32-bit Resource Monitor WOW64, Eg. Enterprise Vault Cluster application. Third-party cluster resources, eg Enterprise Vault, which in this case notifies the local FSA placeholder service on each physical node of virtual server changes
8. ABE enabled by generic cluster resource. Access based enumeration with a generic cluster application running abecmd.exe during virtual server/share creation. Uses 32-bit cluster resource monitor with WOW64, setting SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM (0x0800) flag set on the otherwise standard share.

Pretty picture view:

References

Server side processes for simple file access
http://waynes-world-it.blogspot.com/2008/11/server-side-process-for-simple-file.html

Access Based Enumeration
http://technet.microsoft.com/en-us/library/cc784710.aspx

SHARE_INFO_1005 Structure
http://msdn.microsoft.com/en-us/library/bb525404.aspx

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

Read more!