Jay Caines-Gooby

Back in 5 mins

Realtime Election Tweets

Tuesday May 04, 2010 @ 03:58 AM (BST)

Real-time UK General Election Tweets with node.js & websockets from jaygooby on Vimeo.

Inspired by Makato’s X-Factor real-time Twitter experiments and with just a couple of days to go until the country goes to the polls, I wondered if we could glean an outcome from the Twittersphere, especially as we get nearer to the actual count and results.

Implementation

My real-time election site, monitors the twitter streaming API for mentions of the four three main British political parties, plus the Greens (hey, I live in Brighton Pavillion and they look like they might get their first seat) and their leaders, and tallies a total score and current ratio of tweets.

I’ve since updated it and it monitors all the parties, plus various independents and niche (read 0 current seats in parliament) parties and also tallies phrases like “I voted for”, “I’m voting” etc to count actual votes as well as mentions.

Using node.js and client-side HTML5 websockets (with a fallback websocket implementation in Flash for older browsers), the site is entirely implemented in Javascript. The server-side JS listens for Tweets and emits the scores as json via the client’s websocket. Some client-side JS handles the screen updates.

Server set-up

The static portions of the site are served by nginx (which is itself an asynchronous evented server like node.js). Initially I’d tried to serve the whole thing via node.js using the paperboy.js static file serving module but I’d need both paperboy and the node.ws.js websocket server to share port 80, which would mean some re-engineering, and given that the election is in two days, wanted to get something up quickly!

So my architecture is static files served out of docroot handled by nginx, and the websocket running on a high port for use by the browser clients.

This high-port usage is itself a problem, as I’d imagine that many corporate firewalls block all bar 80 and maybe 8080, which is why the combined server running entirely out of node would be a good eventual goal.

To try and solve this, I thought I’d try and proxy the high port via nginx. You can turn the proxy buffer off in nginx, making it ideal for this, but I hit another hurdle with the Flash implementation of the websocket that will be used by most browsers (only Chrome is currently able to use native HTML5 websockets).

Flash and the crossdomain policy file

Flash requests its cross-domain policy file in an HTTP call to us (aka our Socket Policy Server) thus:

<policy-file-request/>\x00

Note there’s no GET verb, just the xml invocation and a NULL byte. Nice. Quite rightly nginx chokes on it, and issues an HTTP 400.

There’s an nginx hack I could have used to get the socket policy served and perhaps correctly tunnel the websocket stream itself, but I just wanted to launch, so I’ve left this as a to-do for now.

Comments

Lookin good Jay. The alignment of the text at the top is slightly out in Chrome 5.0.3 on OSX

Gravatar icon
Mat
Tuesday May 04, 2010 @ 08:31 AM (BST)

Thanks Mat – I’ve just pushed up a new version with better alignment, but also iPhone/iPad compatbility (no Flash nor HTML websockets in these two, so no real-time updates) and much better updating.

Gravatar icon
Wednesday May 05, 2010 @ 01:25 AM (BST)

nice idea but the Greens a main party? even in Scotland where we have PR they only get 2 seats, meanwhile the SNP are in Government as are Plaid Cymru in Wales and the DUP in Northern Ireland. I think this is a fascinating site but your approach is skewed by the inclusion of the Greens and the exclusionof other bigger parties.

Gravatar icon
domhnall
Wednesday May 05, 2010 @ 12:20 PM (BST)

Hey Dom, skewed by all the publicity the Greens have been getting in Brighton Pavillion I guess :)

I suppose I ought to add UKIP to the list too :(

Gravatar icon
Wednesday May 05, 2010 @ 12:25 PM (BST)

where’s the source? I want to see/learn. Can I see?

Gravatar icon
Alex
Thursday June 17, 2010 @ 05:50 AM (BST)

Is the source code avaliable? would love to look at it.

Gravatar icon
Tuesday August 10, 2010 @ 04:03 PM (BST)

yes performance stats during the election tweets are good for understanding how the hardware actually works. no doubt some redundancy is in order; how much was there? if there were 1/2 million tweets how many could be done? 8 mil JSON updates but could 16 mill be done? 4 json/sec for 98 clients but what if 4x that etc. nice work btw

Gravatar icon
Alex Beston
Monday August 30, 2010 @ 10:25 AM (BST)
New comment

required, won't be displayed

optional

Don't type anything here unless you're an evil robot:


And especially don't type anything here:

Basic XHTML (including links) is allowed, just don't try anything fishy. Your comment will be auto-formatted unless you use your own <p> tags for formatting. You're also welcome to use Textile.

Copyright © 2011 Jay Caines-Gooby. All rights reserved.
Powered by Thoth.