BigData.be project n°2 – Real-time analytics with Storm – workshop 2

Tonight, in Gent [0], we have been invited by Wim, Daan, Kenny (and certainly many others) to tackle several challenges regarding the real-time analytics using tools enabling Big Data analysis:

How the stock market works – credits: unknown

Running Storm [6] in distributed mode
Create a web service for sentiment analysis [7]
Define business logic for positive/negative market movements

For non-IT people, this means that we want:

Read tweets published on Internet
Filter the tweets containing the names of publicly traded companies like Neuhaus (Chocolate), AB Inbev (Biers) (French Fries), Umicore (most sustainable company in the world) to name a few Belgian companies
Evaluate the content of the tweet based on 3 criteria:
1. a sentiment based on the words used inside the tweet
2. the amount of retweets of that initial tweet
3. the amount of followers of the person who retweets the initial tweet
Set a global score of our analysis that might be used to act upon such information i.e. call the police, inform by phone the parents as their children reached their destination, buy or sell a stock …

If you are interested by the early results check the Source code repository : https://github.com/bigdatabe
The repository will grow until we deliver an open source, working implementation of the current functional analysis (I may dream here)

Here is the result of a bit more than an hour of functional analysis…

The Functional analysis

Twitter stream is gathered (i.e. “NSA & FBI have tapped into servers of tech firms to gain intelligence data; firms include $GOOG, $YHOO, $MSFT, $AAPL, $FB. – @WashingtonPost“)
The system determines which tickers are stored in the tweet (i.e. $AAPL is APPLE, $GOOG is Google)
Determine the language of the tweet (i.e. US)
Filter the tweet based on our list of supported languages; we reject all the tweets that can’t be understood (i.e. only NL, FR and US language will be analysed)
in parallel we perform the following tasks in order to come up with a final scoring of the tweet
1. (A) split the tweet by language
2. split the tweet into words
3. Value the sentiment of each word using the corpus [3]. Note that Google provides the largest English corpus at 155 billion words. Also has corpora for other languages [5]
  1. Tweet id
  2. Company id (i.e. AAPL)
  3. Word id: sequence of word into the tweet (i.e. 1, 2, 3 …)
  4. Value: the value can be positive or negative.
4. Calculate tweet’ score ; i.e. you take the total tweet’ score and divide it by a predefined value (i.e. the total amount of relevant words like display retina, chromebook pixel …)
  1. Tweet id
  2. Company id
  3. Score type : [Sentiment score]
  4. Sentiment value (i.e. floating point, positive or negative: 0.8)
5. (B) Calculate the retweet’ score
  1. Tweet id
  2. Company id
  3. Score type : [Retweet score]
  4. Sentiment value (i.e. floating point, positive or negative: -0.2)
6. (C) Calculate follower’ tweet score
  1. Tweet id
  2. Company id
  3. Score type : [follower’ score]
  4. Sentiment value (i.e. floating point, positive or negative: 0.6)
We combine the Sentiment, Retweet and the Follower’ score into a final score using a predefined computation model (i.e. an average of the scores multiplied by a weight per score type: (0.8 * 2) + (-0.2 * 0.3) + (0.6 * 1) = 2.14
1. The outcome this process will be transmitted to another component:
  1. Company id (i.e. AAPL)
  2. Score: (2.14 in our case)
Now the analyst can decide and act upon the score i.e. call the police, inform by phone the parents as their children reached their destination, buy or sell a stock …

During the analysis we didn’t consider…

the linkage between words semantic (i.e. retina display) and the fact that those words relate to a certain company (i.e. Apple)
The absolute and relative value we should consider to act on the information

The early mistakes we made …

Insert too much complexity from the beginning (what if… what if .. what if…)
Include technical requirements into the functional requirements (IT guys never change)