Making Fast Successive Web Requests in PowerShell

powershell

I’ve been working with Fail2Ban, playing with it’s configuration files in an attempt to get it to recognise when someone is making lots of fast successive web requests. Much like buses, though, when you want them, they just don’t come so I needed a quick way to generate my own requests on-demand.

For this, I used PowerShell since my main machine is a Windows 10 laptop, with these as my requirements:

  1. To be able to make lots of requests, quickly. Fairly obvious!
  2. To keep the output of the responses - I need to know what type of web page I get in return and,
  3. To ensure caching isn’t getting jiggy and interfering with my aims.

So here’s what I knocked up.

The Script

1
2
3
4
5
6
7
8
9
10
11
12
13
$pages = @("https://bbc.co.uk/", 
    "https://dailymail.co.uk",
    "https://sky.com")

For ($i = 0; $i -lt 2; $i++) {
    $count = 0;
    $pages | ForEach-Object {
        Write-Host $_
        $html = Invoke-WebRequest -Uri $_ -Headers @{"Cache-Control"="no-cache"}
        Set-Content -Path "$count.txt" -Value $html -Force
        $count++
    }
}

Type the above into a file named something like: quick-requests.ps1 and save it.

How Does it Work?

Firstly, we create an array of sites named $pages. Because I don’t want to show my own local site, I have instead replaced them with some heavily trafficked sites that won’t be hurt by a few requests. Change these into your own. On line 5 we have our main loop; in this example, I am going to make a requests to each site, 2 times. Change that number to something that suits your own goals. Line 6 is a variable used to name the files which are created that contain the web request responses. So, for 5 sites, we would have 5 files named 0.txt through to 4.txt. Line 7 loops through the sites we created in our array and line 8 outputs it’s name each time so that you know what is happening. 9 actually makes the request (turning off caching) and stores the response in a variable named $html. Line 10 is responsible for writing that response to the file (forcibly mind you!). Lastly, line 11 increments the $count variable, ready for the next site. As you can guess, on each pass, it writes to the same file over and over so that when the script finishes, each numbered file contains the response from the last request.

Running the Script

This is a simple command as you would expect.

1
> ./quick-requests.ps1

Please be responsible with this - only run it on your own sites and when you know what you are doing and what you can expect.

Hi! Did you find this useful or interesting? I have an email list coming soon, but in the meantime, if you ready anything you fancy chatting about, I would love to hear from you. You can contact me here or at stephen ‘at’ logicalmoon.com