Reading web content with PowerShell

The following PowerShell code shows 6 different methods of using the .Net framework to read HTML/ASP responses, useful for scraping web pages through script. If you don't have a proxy server, just comment out the $wc.proxy lines.

$user = $env:username
$webproxy = "http://proxy:8080"
$pwd = Read-Host "Password?" -assecurestring

$proxy = new-object System.Net.WebProxy
$proxy.Address = $webproxy
$account = new-object System.Net.NetworkCredential($user,[Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pwd)), "")
$proxy.credentials = $account


# -- Method 1 - DownloadData -- #

$url = "http://www.sitemeter.com/?a=stats&s=s451qaz2WSX&r=0"
$wc = new-object system.net.WebClient
$wc.proxy = $proxy
$webpage = $wc.DownloadData($url)
$string = [System.Text.Encoding]::ASCII.GetString($webpage)

# -- Method 2 - DownloadData (QueryString) -- #

$url = "http://www.sitemeter.com"
$col = new-object System.Collections.Specialized.NameValueCollection
$col.Add("a","stats")
$col.Add("s","s451qaz2WSX")

$wc = new-object system.net.WebClient
$wc.proxy = $proxy
$wc.QueryString = $col
$webpage = $wc.DownloadData($url)
$string = [System.Text.Encoding]::ASCII.GetString($webpage)

# -- Method 3 - UploadData (POST) -- #

$url = "http://www.sitemeter.com"
$wc = new-object system.net.WebClient
$postData = "a=stats&s=s451qaz2wsx"
$wc.Headers.Add("Content-Type","application/x-www-form-urlencoded")
$wc.proxy = $proxy
[byte[]]$byteArray = [System.Text.Encoding]::ASCII.GetBytes($postData)
$webpage = $wc.UploadData($url,"POST",$byteArray);
$string = [System.Text.Encoding]::ASCII.GetString($webpage)

# -- Method 4 - GetResponse Stream -- #

$url = "http://www.sitemeter.com/?a=stats&s=s451qaz2wsx"
$wr = [System.Net.WebRequest]::Create($url)
$wr.Proxy = $proxy
$wresponse=$wr.GetResponse()

$requestStream = $wresponse.GetResponseStream()
$readStream = new-object System.IO.StreamReader $requestStream
$wrs = $readStream.ReadToEnd()
$readStream.Close()
$wresponse.Close()

# -- Method 5 - UploadValues -- #

$url = "http://www.sitemeter.com/default.asp"
$wc = new-object system.net.WebClient
$wc.proxy = $proxy
$col = new-object System.Collections.Specialized.NameValueCollection
$col.Add("a","stats")
$col.Add("s","s451qaz2WSX")
$wc.QueryString = $col
$webpage = $wc.UploadValues($url, "POST", $col)
$string = [System.Text.Encoding]::ASCII.GetString($webpage)

# -- Method 6 - UploadString - this doesn't seem to post properly -- #

$url = "http://www.sitemeter.com/default.asp"
$wc = new-object system.net.WebClient
$wc.proxy = $proxy
$postData = "/?a=stats&s=s451qaz2wsx"
$webpage = $wc.UploadString($url, $postData)
$webpage


Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.