http://jay.gooby.org/tag/javascript Posts tagged with "javascript" - Jay Caines-Gooby 2010-05-04T03:58:44+01:00 Jay Caines-Gooby jay@gooby.org http://jay.gooby.org/ http://jay.gooby.org/post/realtime-election-tweets Realtime Election Tweets 2010-05-04T03:58:44+01:00 2010-05-17T09:56:34+01:00 <p><object width="576" height="400"><param name="allowfullscreen" value="true" /><param name="allowscriptaccess" value="always" /><param name="movie" value="http://vimeo.com/moogaloop.swf?clip_id=11753010&amp;server=vimeo.com&amp;show_title=1&amp;show_byline=1&amp;show_portrait=0&amp;color=&amp;fullscreen=1" /><embed src="http://vimeo.com/moogaloop.swf?clip_id=11753010&amp;server=vimeo.com&amp;show_title=1&amp;show_byline=1&amp;show_portrait=0&amp;color=&amp;fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="576" height="400"></embed></object><p><a href="http://vimeo.com/11753010">Real-time UK General Election Tweets with node.js &amp; websockets</a> from <a href="http://vimeo.com/user3820082">jaygooby</a> on <a href="http://vimeo.com">Vimeo</a>.</p></p> <p>Inspired by <a href="http://blog.new-bamboo.co.uk/2009/12/14/pushing-the-boundary-of-real-time-web-with-twitter-and-xfactor">Makato&#8217;s X-Factor real-time Twitter experiments</a> and with just a couple of days to go until the country goes to the polls, I wondered if we could glean an outcome from the Twittersphere, especially as we get nearer to the actual count and results.</p> <h2>Implementation</h2> <p>My <a href="http://election2010.gooby.org/">real-time election site</a>, monitors the <a href="http://apiwiki.twitter.com/Streaming-API-Documentation">twitter streaming <span class="caps">API</span></a> for mentions of the <strike><strike>four</strike> three main British political parties, plus the Greens (hey, I live in Brighton Pavillion and they look like they might get their first seat) and their leaders, and tallies a total score and current ratio of tweets</strike>.</p> <p>I&#8217;ve since updated it and it monitors all the parties, plus various independents and niche (read 0 current seats in parliament) parties and also tallies phrases like &#8220;I voted for&#8221;, &#8220;I&#8217;m voting&#8221; etc to count actual votes as well as mentions.</p> <p>Using <a href="http://nodejs.org">node.js</a> and client-side <a href="http://dev.w3.org/html5/websockets/">HTML5 websockets</a> (with a <a href="http://github.com/gimite/web-socket-js">fallback websocket implementation in Flash</a> for older browsers), the site is entirely implemented in Javascript. The server-side JS listens for Tweets and emits the scores as json via the client&#8217;s websocket. Some client-side JS handles the screen updates.</p> <h2>Server set-up</h2> <p>The static portions of the site are served by <a href="http://wiki.nginx.org/Main">nginx</a> (which is itself an asynchronous evented server like node.js). Initially I&#8217;d tried to serve the whole thing via node.js using the <a href="http://github.com/felixge/node-paperboy">paperboy.js static file serving module</a> but I&#8217;d need both paperboy and the <a href="http://github.com/ncr/node.ws.js">node.ws.js websocket server</a> to share port 80, which would mean some re-engineering, and given that the election is in two days, wanted to get something up quickly!</p> <p>So my architecture is static files served out of docroot handled by nginx, and the websocket running on a high port for use by the browser clients.</p> <p>This high-port usage is itself a problem, as I&#8217;d imagine that many corporate firewalls block all bar 80 and maybe 8080, which is why the combined server running entirely out of node would be a good eventual goal.</p> <p>To try and solve this, I thought I&#8217;d try and proxy the high port via nginx. You can turn the <a href="http://wiki.nginx.org/NginxHttpProxyModule#proxy_buffering">proxy buffer off</a> in nginx, making it ideal for this, but I hit another hurdle with the Flash implementation of the websocket that will be used by most browsers (only Chrome is currently able to use native HTML5 websockets).</p> <h3>Flash and the crossdomain policy file</h3> <p>Flash requests its cross-domain policy file in an <span class="caps">HTTP</span> call to us (aka our Socket Policy Server) thus:</p> <p><code> &lt;policy-file-request/&gt;\x00 </code></p> <p>Note there&#8217;s no <span class="caps">GET</span> verb, just the xml invocation and a <span class="caps">NULL</span> byte. Nice. Quite rightly nginx chokes on it, and issues an <span class="caps">HTTP</span> 400.</p> <p>There&#8217;s <a href="http://blog.vokle.com/index.php/2009/06/10/dealing-with-adobe-and-serving-socket-policy-servers-via-nginx-and-10-lines-of-code/">an nginx hack</a> I could have used to get the socket policy served and perhaps correctly tunnel the websocket stream itself, but I just wanted to launch, so I&#8217;ve left this as a to-do for now.</p>