Scraping 2020 US Election Data

I thought this code could be useful for researchers. I wrote it a while ago, then never used it. Too busy 😦

elec.sh:

# Zoe Phin
# 2020/11/11

require() {
   sudo apt-get install curl jq gnuplot
}

download() {
   curl -O 'https://static01.nyt.com/elections-assets/2020/data/api/2020-11-03/state-page/{alabama,alaska,arizona,arkansas,california,colorado,connecticut,delaware,florida,georgia,hawaii,idaho,illinois,indiana,iowa,kansas,kentucky,louisiana,maine,maryland,massachusetts,michigan,minnesota,mississippi,missouri,montana,nebraska,nevada,new-hampshire,new-jersey,new-mexico,new-york,north-carolina,north-dakota,ohio,oklahoma,oregon,pennsylvania,rhode-island,south-carolina,south-dakota,tennessee,texas,utah,vermont,virginia,washington,west-virginia,wisconsin,wyoming}.json'
}

timelines() {
   for state in `ls *.json | sed s/.json//`; do
      jq -Mc '.data.races[0].timeseries[]|[.timestamp,.votes,.votes*.vote_shares.trumpd,.votes*.vote_shares.bidenj]' $state.json | tr -d '["]' |\
      awk -F, '{"TZ=US/Eastern date +%d%H%M%S -d "$1 | getline t; printf "%s %8d %8d %8d\n",t,$2-vlast,$3-tlast,$4-blast; vlast=$2; tlast=$3; blast=$4}' > $state.csv
   done
}

merged() {
   sort -n *.csv | awk '{vtotal+=$2; ttotal+=$3; btotal+=$4; printf "%s %9d %9d %9d\n",$1,vtotal,ttotal,btotal}' > elec.dat
}

plot() {
   echo 'set term png size 740,740
   set key top left 
   set grid xtics ytics
   set ylabel "Million Votes"
   set timefmt "%d%H%M%S"
   set xdata time
   set xtics format "11/%d\n%H:%M"
   set ytics format "%.0f"
   set xrange ["03190000":"04040000"]
   plot "elec.dat" u 1:($3/1e6) t "Trump" w lines lc rgb "red" lw 1,\
                "" u 1:($4/1e6) t "Biden" w lines lc rgb "blue" lw 1
   ' | gnuplot > elec.png
}

Run it:

$ source elec.sh
$ require && download
$ timelines && merged && plot

Result: elec.png

Enjoy 🙂 -Zoe

Update 2020/12/06

County level data can be obtained with this code snippet:

counties() {
   echo " County              |    Votes |    Votes    Votes    Votes |      %          %         %   "
   echo " Name                |    Total |    Biden    Trump    Jorg. |    Biden      Trump     Jorg."
   for state in `ls *.json | sed s/.json//`; do
      echo -e "\n--- $state ---\n"
      jq -Mc '.data.races[0].counties[]|[.name,.votes,.results.bidenj,.results.trumpd,.results.jorgensenj]' $state.json | tr -d '["]' |\
      awk -F, -vs=$state '$2!=0{ v+=$2; b+=$3; t+=$4; j+=$5;
         printf "%-20s | %8s | %8s %8s %8s | %9.3f %9.3f %9.3f\n",$1,$2,$3,$4,$5,$3/$2*100,$4/$2*100,$5/$2*100} END {
         printf "\nTotal                | %8s | %8s %8s %8s | %9.3f %9.3f %9.3f\n",v,b,t,j,b/v*100,t/v*100,j/v*100
      }'
   done
}

Run it:

$ . elec.sh; counties > counties.txt

Contents of counties.txt is archived here.

Published by Zoe Phin

https://phzoe.com

9 thoughts on “Scraping 2020 US Election Data

  1. So at 11.30pm, there’s a jump. But you’ve converted all times to Eastern? So does this mean they synchronised the ballot stuffing and some did it earlier? I’m at a loss why they’d do that. Also I have read that much of the fraud happened in the small hours. Any thoughts on all this?

    Liked by 1 person

    1. The 11:30 thing is just California going “online”. Would you like me to show that?

      The stuffing occured at various times throughout the country. This chart doesn’t show that.

      It’s hard to find a smoking gun in this data alone. One needs county level timelines and voter data.

      Like

  2. I’ve been trying to find election returns for Jo Jorgensen and other libertarian candidates. Using the raw link I can now make out Jorgensen votes, for which I only had a county in Texas and the State of Nevada. Eventually I expect to make headway, but this coding is way over my head. Good show graphing the results; impressive as always.

    Liked by 1 person

  3. Must assuredly, in the sense that I can hire talent. I flunked Diff Eq back when Pascal and Basic were things. Instead of “wrong decade” error messages I’m now starting to get “wrong century”. Just this morning I set aside a Dell i7 book to load with Freaks… ooops… I mean Linux. In any case your terse, to-the-point posts on subjects of interest are much appreciated.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

<span>%d</span> bloggers like this: