Query Twitter with Talend to see what people think about …

Hi all,

Almost on holidays after a very hard working year. I have some beach time.

This morning, I tried to query Twitter and to process the data. My purpose is to quickly build a data set showing what people are talking about … let’s say, Obama. Easy, Twitter is providing an interface to run queries and retrieve the results back with json format.

Well, my proposition is to implement a basic word frequency analysis. My tool is Talend, but this process is also easy to set up using java, php, python, shell, Kettle …

Here are some details :

• fileInputJSON : just read the json from Twitter search engine. The syntax is : http://search.twitter.com/search.json?&q=YOUR_QUERY_HERE&rpp=10000, where YOUR_QUERY_HERE = the key word you want to search for. Don’t forget to distribute the answer into a string column :
  • answer ====> “$..text” : where “$..text” stands for the tweet message you want to read.
• tmap : just to transform the text into lower case. I tried to do it on the fly in the json step, but it does not work.
• tnormalize : transform all words from anwer into rows (separator is space).
• tfilter : filter your data in order to avoid “|” (pipes) and other special chars + get rid of words having less than 4 letters.
• taggregate : aggregate on the words and add a new column named nb that will store the distinct count of all words.
• tsortrow : now you have words and counts, just sort the data from higher counts to lower (desc).
• tsamplerow : as we just want to read the first top 20 words, we create a sample based on the sorted rows with range “1..20”.

Finally, I print everything into the console by using a tlogrow + “:” separator.

Here are the data for the keyword “obama”, the query was done on July 26 at 16h00.

Finally, I will use Jonathan Feinberg, wordle, to create a nice and sexy word map with the above results.

 

Setting up the process and testing it only took 20 minutes. Of course, this can be improved a lot by adding string cleaning, custom data filtering… or better :

• Capture who created the tweet and who received it : you can create relationship networks with, for instance, networkx.
• Send the data to a powerfull text analytics framework like NLTK, for better analytics.
• Create the word map picture into Talend. I have some source code for this, will have to work on it …

I ll see how to add one of these features soon.