Posted on Tuesday, June 3, 2008 - 10:32 am:

---

quote:

From John Howard:

WEB SITE MIRRORING (also called offline browsing) -- There are numerous products out there. One is called HTtrack and, with Windows, you use the windows version called WinHTTrack. It's a free open-source product and can be found at http://www.httrack.com/page/2/en/index.html

It can be configured to copy an entire web-site (definitely NOT a good idea), it can also be configured to copy a set of specific links contained in an HTML file, which is why I create the state by state "Compendium" link pages in advance - that way I can get the results pages from multiple jurisdictions, without having to spend a lot of time entering and re-entering the links I want captured. (my fall-2006 file captured just over 1200 selected precincts and counties in 11 states plus a couple of news sites, in each of about 20 readings throughout the evening)

Pre-requisites for gathering election night results:

- A reliable fast internet connection (NOT dial up)
- LOTS and LOTS and LOTS of disk space
- Someone to babysit the process and start it every so often. (I like starting every 15 minutes)

There are options - I exclude any graphics, limit the capture to 5 simultaneous connections, do not cache the files, and set a maximum external link depth of 1. This prevents the program from finding additional links and trying to copy them too.

Usually a couple of dry runs are needed to make sure that you're getting the files you want without too much extra 'stuff'. During a night of result gathering, I also update my source file with new links as they become available. If available, using a second computer to gather and update your source file is worthwhile.

Do a bit of experimenting with the program first to familiarize yourself with it.

---

---

quote:

1.0: first release of the WebSnapshot tool. Version 1.0: http://www.bbvforums.org/docs/tools/WebSnapshot-1.0.jar
(18 KB)
2.0: Added CNN support http://www.bbvforums.org/docs/tools/WebSnapshot-2.0.jar
(27 KB)
2.1: Added CNN exit poll support http://www.bbvforums.org/docs/tools/WebSnapshot-2.1.zip
(49 KB)
2.2: Added data collection for South Dakota counties http://www.bbvforums.org/docs/tools/WebSnapshot-2.2.zip
(52 KB)
2.2.1: Add a time out when downloading URLs
since the CERS URLs sometimes hang
http://www.bbvforums.org/docs/tools/WebSnapshot-2.2.1.zip(53 KB)

VERSION 1.0 NOTES, GENERAL USE
http://www.bbvforums.org/docs/tools/WebSnapshot-1.0.jar
Usage: java -jar WebSnapshot.jar <urls> [<refresh> <count>]
Where <urls> is the path to a file containing URLs, one on each line. The <refresh> and <count> arguments are optional. The <count> is the number of snapshots to create, and the <refresh> is the number of seconds to wait between each snapshot. Here's an example URLs file:
http://www.cnn.com/
http://www.yahoo.com/
MV "BARACK OBAMA" "HILLARY CLINTON" http://www.lakecountyin.org/cms-host/voters/microvote/2008PrimaryElectionResults /LakePrecSumm561.htm
The last line in the file is a special notation for downloading all the precinct summaries for an entire county. The notation for this is:
MV "CANDIDATE 1" "CANDIDATE 2" "..." <url>
Where <url> is the path to the last precinct in the county. The example above downloads all 561 precinct summaries and produces a csv file with results per precinct.
Example usage:
java -jar WebSnapshot.jar urls.txt 60 5
This would create a snapshot folder based on current date/time and then capture each of the URLs contained in urls.txt. It would then wait for 60 seconds and repeat the process, taking a total of 5 snapshots.

You will need Java 5.0+ installed. You need to run the tool from the command line (Start -> Run... "cmd" on Windows XP). Next I plan to add support for generating spreadsheets (with results per county) from CNN pages.

WEB SNAPSHOT 2.0 w/ CNN support
http://www.bbvforums.org/docs/tools/WebSnapshot-2.0.jar
Here is the updated WebSnapshot.jar and an example input file, adds a special notation for capturing CNN results per county in a csv file. Take a look at the example input file to see how this is done.

Installation instructions
---------------------------------------
1) Create a Directory C:\WebSnapshot
2) Download WebSnapshot.jar and urls.txt to this directory
http://www.bbvforums.org/docs/tools/urls-74129.txt
3) Open up a command prompt (Start -> Run... 'cmd')
4) Switch to directory (cd C:\WebSnapshot)
5) Run the java application from the command line (java -jar WebSnapshot.jar urls.txt)

Capture periodic snapshots by using the optional command line arguments (read the first post).
Press Ctrl+C to terminate the application early.
Java is completely free. Here is the instructions for downloading and installing Java 5.0.
1) http://java.sun.com/javase/downloads/index_jdk5.jsp
2) Select Java Runtime Environment (JRE) 5.0 Update 15
3) Choose your platform (e.g. Windows)
4) Download the Windows Offline Installation
5) Run the Installer (jre-1_5_0_15-windows-i586-p.exe)

Here's a guide to download and rename in Internet Explorer. This expands Step 2 in the installation instructions

a) Right click on WebSnapshot.jar link and select 'Save Target As...'
b) Select 'All Files' under Save as type
c) Enter 'WebSnapshot.jar' under File name.
d) Select the dropdown 'Save in' and choose 'Local Disk'
e) Double-click the WebSnapshot folder and click 'Save'

WEB SNAPSHOT 2.1: Added support for CNN exit polls. Extract the attached zip file to your C:\WebSnapshot directory.
http://www.bbvforums.org/docs/tools/WebSnapshot-2.1.zip
See the included urls.txt input file for an example. Next, I plan to add a snapshot summary feature to create spreadsheets of vote counts by snapshot time. This will help create graphs of the incoming voting data, and allow for a visual inspection of the results.

MERGE JOHN HOWARD'S SOUTH DAKOTA COMPENDIUM:
http://www.bbvforums.org/docs/tools/SouthDakotaCounties.html
(From John Howard): Here is the South Dakota Compendium of results sites. Basically there is only one - the SoS, with the exception of Pennington County which is also included here. I've included the addresses of the counties that have web sites, although they are not clickable links. Having visited each one today, I doubt that we'll see results posted on any of them. Please note that these are all results by Precinct for each county, with one file each for Dem and one for Rep voters. The SoS web site is SLOW to produce these results so I wouldn't advise running too many concurrent connections. Since all the SoS results will be of a consistent format, perhaps this can feed into Donovan Levinson's SNAPSHOT tool.

SOUTH DAKOTA UPDATE: WebSnapshot 2.2 including South Dakota http://www.bbvforums.org/docs/tools/WebSnapshot-2.2.zip

Finished adding support for South Dakota precinct results URLs (CERS). Also included is snapshot history report generation. These files end in "-history.csv" and contain all the vote counts by snapshot time. There is an issue where sometimes more than 256 columns are generated (#columns = #precincts * #choices + 1). Excel will only display the first 256 columns. I will add a workaround for this by saving across multiple files in a future release.
Extract the attached zip to your C:\WebSnapshot directory. See included urls.txt input file for an example.

2.2.1: Optimized to deal with URL hangs
http://www.bbvforums.org/docs/tools/WebSnapshot-2.2.1.zip

(Donovan): I hope to get some useful feedback from users during this last day of primaries. I have made one minor change, and that is to add a time out when downloading URLs, since the CERS URLs sometimes hang for a very long time. The timeout is set to 5 minutes. I will expose this variable and others in a properties file soon.

Also, I have attached two South Dakota input files for tracking the election results:

http://www.bbvforums.org/docs/tools/sd-dem.txt
http://www.bbvforums.org/docs/tools/sd-rep.txt

---