Christmas Twitter Tree Lights 2

Posted by barnoid Sun, 22 Dec 2013 16:05:00 GMT

I recently bought a string of 50 individually addressable RGB LEDs from Embedded Adventures and so had to decide what to do with them. I decided it would be festive if they could reflect Twitter's current feeling about Christmas.

So I started Googling sentiment analysis and quickly discovered that Stanford University's Natural Language Processing Group has released the source and data for their sentiment analyser. When fed a sentence it outputs one of "Very negative", "Negative", "Neutral", "Positive" or "Very positive". It's pretty good but it's trained on movie reviews so it's not quite so accurate on tweets about Christmas. For example "Christmas time is defo the best time of the year !!" is classed as very negative.

The sentiment analysis is fairly slow. Much slower than the rate that the tweets come in, so I'm using beanstalkd for queueing. The tweets come in from the Twitter streaming API (filtering for "christmas" and "xmas"), get cleaned up, @usernames removed, hashes removed from hashtags, URLs deleted, then put in a queue. My desktop computer is the fastest in the house so it runs four instances of the sentiment analysis, pulling tweets out of the queue, processing them, pre-pending one of N, n, -, p or P and sticking the result in a second queue.

Raspberry Pi controlling the lights.
Raspberry Pi controlling the lights.

The lights themselves are controlled by a Raspberry Pi. They are connected to the SPI pins via a level shifter from Sparkfun. The pins on the Pi all work at 3.3V while the lights need 5V hence the shifter is needed. I'm using a fairly recent Raspian. To enable the SPI device /dev/spidev0.0 I removed the "blacklist spi_bcm2708" from /etc/modprobe.d/raspi-blacklist.conf. Then I just need to write a byte each for red, green and blue for each LED to the device file.

So the code on the Pi reads from the second queue, and shifts an LED on to the chain with a colour based on the initial sentiment letter: red for negative, green for positive, white for neutral, bright cyan for very positive and bright orange for very negative. The sentiment analysis processing can't keep up with the rate of incoming tweets so I've had it skip every other tweet.

The result is quite a pretty light display, though it's hard to say whether or not it's successful in reflecting the opinion on Twitter of Christmas.

The Code

This is the code that streams tweets from Twitter, cleans them up and puts them in a queue. It uses the tweetstream and beaneater gems.


require 'rubygems'
require 'tweetstream'
require 'beaneater'

TweetStream.configure do |config|
        config.consumer_key       = "SECRET"
        config.consumer_secret    = "SECRET"
        config.oauth_token        = "SECRET"
        config.oauth_token_secret = "SECRET"
        config.auth_method        = :oauth

beanstalk =['localhost:11300'])
tube = beanstalk.tubes["tweets-in"]

count = 0'xmas', 'christmas') do |status|
        tweet = status.text
        tweet.gsub!(/@.+?($|\s)/, "")
        tweet.gsub!(/^RT /, "")
        tweet.gsub!(/https?:.+?($|\s)/, "")
        tweet.gsub!(/[^\w\d \(\),\.\?\!\-\'\:\$\&\;\+]/, "")
        tweet.gsub!(/&/, "and")
        tweet.gsub!(/&.+?;/, "")
        if count % 2 == 0 then
        count += 1


This is the code that pulls tweets out of one queue, feeds them to the sentiment analyser and puts them in another queue with the result.


require 'rubygems'
require 'beaneater'

beanstalk =[''])
tube_in = beanstalk.tubes["tweets-in"]
tube_out = beanstalk.tubes["tweets-out"]

IO.popen("./run_sentiment", "r+") do |sent_io|
        # Wait until the sentiment analyser has printed its startup text
        while not sent_io.readline.match(/EOF/)

        while true do
                tweet_job = tube_in.reserve
                tweet = tweet_job.body
                sent_io.puts tweet
                sentiment = sent_io.readline.chomp
                case sentiment
                when "  Very positive"
                        sentiment = "P"
                when "  Positive"
                        sentiment = "p"
                when "  Neutral"
                        sentiment = "-"
                when "  Negative"
                        sentiment = "n"
                when "  Very negative"
                        sentiment = "N"
                puts "#{sentiment} #{tweet}"
                tube_out.put("#{sentiment} #{tweet}")


This starts the sentiment analyser telling it to accept input on STDIN. This is referenced as run_sentiment above.


cd stanford-corenlp-full-2013-11-12
java -cp "*" -mx5g edu.stanford.nlp.sentiment.SentimentPipeline -stdin 2>&1

This code runs on the Pi, controlling the lights.


require 'beaneater'

class Led

        attr_accessor :r, :g, :b

        def initialize(r = 0, g = 0, b = 0)
                @r = r
                @g = g
                @b = b

        def out
                return "#{@r.chr}#{@g.chr}#{@b.chr}"

        def to_s
                return "%02X%02X%02X" % [ @r, @g, @b ]


LED_V_POS =,128,255)
LED_POS =,64,0)
LED_NEG =,0,0)
LED_V_NEG =,128,0)

beanstalk =[''])
tube = beanstalk.tubes['tweets-out']

leds = []
50.times do
end"/dev/spidev0.0", "w") do |spi|

        while true do

                job = tube.reserve
                s = job.body[0]
                case s
                when 'P'
                        leds.unshift LED_V_POS
                when 'p'
                        leds.unshift LED_POS
                when '-'
                        leds.unshift LED_NEUTRAL
                when 'n'
                        leds.unshift LED_NEG
                when 'N'
                        leds.unshift LED_V_NEG
                #p leds

                50.times do |k|
                        spi.write leds[k].out



Leave a comment »

  1. james about 17 hours later:

    If you try the example “”Christmas time is defo the best time of the year !” on the live demo site, you’ll notice that it correctly analyses the sentence to have “very positive” sentiment. The reason why the result is different on the code that uses CoreNLP 3.3.0 + its model is explained in GitHub issue #7 ( The key reason probably is that sentence contains word “defo” that the system does not recognize and 3.3.0’s faulty sentiment model then categorizes the sentiment wrong.

    If you want to fix the issue you should implement the proposed solution in GitHub issue, you would need to compile the CoreNLP from GitHub sources and use different (“current”) model.

  2. Andrew Caines 13 days later:

    nice set up! thanks for sharing, will try this next year