Christmas Twitter Tree Lights

I recently bought a string of 50 individually addressable RGB LEDs from Embedded Adventures and so had to decide what to do with them. I decided it would be festive if they could reflect Twitter’s current feeling about Christmas.

So I started Googling sentiment analysis and quickly discovered that Stanford University’s Natural Language Processing Group has released the source and data for their sentiment analyser. When fed a sentence it outputs one of “Very negative”, “Negative”, “Neutral”, “Positive” or “Very positive”. It’s pretty good but it’s trained on movie reviews so it’s not quite so accurate on tweets about Christmas. For example “Christmas time is defo the best time of the year !!” is classed as very negative.

The sentiment analysis is fairly slow. Much slower than the rate that the tweets come in, so I’m using beanstalkd for queueing. The tweets come in from the Twitter streaming API (filtering for “christmas” and “xmas”), get cleaned up, @usernames removed, hashes removed from hashtags, URLs deleted, then put in a queue. My desktop computer is the fastest in the house so it runs four instances of the sentiment analysis, pulling tweets out of the queue, processing them, pre-pending one of N, n, -, p or P and sticking the result in a second queue.

The lights themselves are controlled by a Raspberry Pi. They are connected to the SPI pins via a level shifter from Sparkfun. The pins on the Pi all work at 3.3V while the lights need 5V hence the shifter is needed. I’m using a fairly recent Raspian. To enable the SPI device /dev/spidev0.0 I removed the “blacklist spi_bcm2708” from /etc/modprobe.d/raspi-blacklist.conf. Then I just need to write a byte each for red, green and blue for each LED to the device file.

So the code on the Pi reads from the second queue, and shifts an LED on to the chain with a colour based on the initial sentiment letter: red for negative, green for positive, white for neutral, bright cyan for very positive and bright orange for very negative. The sentiment analysis processing can’t keep up with the rate of incoming tweets so I’ve had it skip every other tweet.

The result is quite a pretty light display, though it’s hard to say whether or not it’s successful in reflecting the opinion on Twitter of Christmas.

The Code

This is the code that streams tweets from Twitter, cleans them up and puts them in a queue. It uses the tweetstream and beaneater gems.

#!/usr/bin/ruby

require 'rubygems'
require 'tweetstream'
require 'beaneater'

TweetStream.configure do |config|
        config.consumer_key       = "SECRET"
        config.consumer_secret    = "SECRET"
        config.oauth_token        = "SECRET"
        config.oauth_token_secret = "SECRET"
        config.auth_method        = :oauth
end

beanstalk = Beaneater::Pool.new(['localhost:11300'])
tube = beanstalk.tubes["tweets-in"]

count = 0

TweetStream::Client.new.track('xmas', 'christmas') do |status|
        tweet = status.text
        tweet.gsub!(/@.+?($|\s)/, "")
        tweet.gsub!(/^RT /, "")
        tweet.gsub!(/https?:.+?($|\s)/, "")
        tweet.gsub!(/[^\w\d \(\),\.\?\!\-\'\:\$\&\;\+]/, "")
        tweet.gsub!(/&amp;/, "and")
        tweet.gsub!(/&.+?;/, "")
        if count % 2 == 0 then
                tube.put(tweet)
        end
        count += 1
end

beanstalk.close

This is the code that pulls tweets out of one queue, feeds them to the sentiment analyser and puts them in another queue with the result.

#!/usr/bin/ruby

require 'rubygems'
require 'beaneater'

beanstalk = Beaneater::Pool.new(['192.168.1.4:11300'])
tube_in = beanstalk.tubes["tweets-in"]
tube_out = beanstalk.tubes["tweets-out"]

IO.popen("./run_sentiment", "r+") do |sent_io|
        # Wait until the sentiment analyser has printed its startup text
        while not sent_io.readline.match(/EOF/)
        end

        while true do
                tweet_job = tube_in.reserve
                tweet = tweet_job.body
                sent_io.puts tweet
                sentiment = sent_io.readline.chomp
                case sentiment
                when "  Very positive"
                        sentiment = "P"
                when "  Positive"
                        sentiment = "p"
                when "  Neutral"
                        sentiment = "-"
                when "  Negative"
                        sentiment = "n"
                when "  Very negative"
                        sentiment = "N"
                end
                puts "#{sentiment} #{tweet}"
                tube_out.put("#{sentiment} #{tweet}")
                tweet_job.delete
        end

end

This starts the sentiment analyser telling it to accept input on STDIN. This is referenced as run_sentiment above.

#!/bin/bash

cd stanford-corenlp-full-2013-11-12
java -cp "*" -mx5g edu.stanford.nlp.sentiment.SentimentPipeline -stdin 2>&1

This code runs on the Pi, controlling the lights.

#!/usr/bin/ruby

require 'beaneater'

class Led

        attr_accessor :r, :g, :b

        def initialize(r = 0, g = 0, b = 0)
                @r = r
                @g = g
                @b = b
        end

        def out
                return "#{@r.chr}#{@g.chr}#{@b.chr}"
        end

        def to_s
                return "%02X%02X%02X" % [ @r, @g, @b ]
        end

end

LED_V_POS = Led.new(0,128,255)
LED_POS = Led.new(0,64,0)
LED_NEUTRAL = Led.new(8,8,8)
LED_NEG = Led.new(64,0,0)
LED_V_NEG = Led.new(255,128,0)

beanstalk = Beaneater::Pool.new(['192.168.1.4:11300'])
tube = beanstalk.tubes['tweets-out']

leds = []
50.times do
        leds.push Led.new
end

File.open("/dev/spidev0.0", "w") do |spi|

        while true do

                job = tube.reserve
                s = job.body[0]
                case s
                when 'P'
                        leds.unshift LED_V_POS
                when 'p'
                        leds.unshift LED_POS
                when '-'
                        leds.unshift LED_NEUTRAL
                when 'n'
                        leds.unshift LED_NEG
                when 'N'
                        leds.unshift LED_V_NEG
                else
                        leds.unshift Led.new
                end
                job.delete
                leds.pop
                #p leds

                50.times do |k|
                        spi.write leds[k].out
                end
                spi.flush

        end

end