Real-time UK General Election Tweets with node.js & websockets from jaygooby on Vimeo.

Inspired by Makato’s X-Factor real-time Twitter experiments and with just a couple of days to go until the country goes to the polls, I wondered if we could glean an outcome from the Twittersphere, especially as we get nearer to the actual count and results.

Implementation

My real-time election site, monitors the twitter streaming API for mentions of the four three main British political parties, plus the Greens (hey, I live in Brighton Pavillion and they look like they might get their first seat) and their leaders, and tallies a total score and current ratio of tweets.

I’ve since updated it and it monitors all the parties, plus various independents and niche (read 0 current seats in parliament) parties and also tallies phrases like “I voted for”, “I’m voting” etc to count actual votes as well as mentions.

Using node.js and client-side HTML5 websockets (with a fallback websocket implementation in Flash for older browsers), the site is entirely implemented in Javascript. The server-side JS listens for Tweets and emits the scores as json via the client’s websocket. Some client-side JS handles the screen updates.

Server set-up

The static portions of the site are served by nginx (which is itself an asynchronous evented server like node.js). Initially I’d tried to serve the whole thing via node.js using the paperboy.js static file serving module but I’d need both paperboy and the node.ws.js websocket server to share port 80, which would mean some re-engineering, and given that the election is in two days, wanted to get something up quickly!

So my architecture is static files served out of docroot handled by nginx, and the websocket running on a high port for use by the browser clients.

This high-port usage is itself a problem, as I’d imagine that many corporate firewalls block all bar 80 and maybe 8080, which is why the combined server running entirely out of node would be a good eventual goal.

To try and solve this, I thought I’d try and proxy the high port via nginx. You can turn the proxy buffer off in nginx, making it ideal for this, but I hit another hurdle with the Flash implementation of the websocket that will be used by most browsers (only Chrome is currently able to use native HTML5 websockets).

Flash and the crossdomain policy file

Flash requests its cross-domain policy file in an HTTP call to us (aka our Socket Policy Server) thus:

<policy-file-request/>\x00

Note there’s no GET verb, just the xml invocation and a NULL byte. Nice. Quite rightly nginx chokes on it, and issues an HTTP 400.

There’s an nginx hack I could have used to get the socket policy served and perhaps correctly tunnel the websocket stream itself, but I just wanted to launch, so I’ve left this as a to-do for now.