Simon Willison’s Weblog

Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python?

21st December 2011

My answer to Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python? on Quora

It depends on how much data you intend to collect, and how you intend to then share that data.

SQLite should be fine if you are collecting thousands of tweets—if you are collecting millions it will probably be OK but you might be better off with something else.

If you plan to put a web interface in front of stage app which other people get to use you should go for a database designed for concurrent access (MySQL, MongoDB, PostgreSQL etc) instead of SQLite. From your question it sounds like SQLite will be fine though.

I haven’t used tweepy myself, but if it works for you then stick with it. The Twitter HTTP APIs are pretty painless to talk to directly though provided you have a good OAuth library (I use python-oauth2) and that might give you more flexibility, but don’t bother with that unless the library you are already using starts to get in your way.