Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python?
21st December 2011
My answer to Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python? on Quora
It depends on how much data you intend to collect, and how you intend to then share that data.
SQLite should be fine if you are collecting thousands of tweets—if you are collecting millions it will probably be OK but you might be better off with something else.
If you plan to put a web interface in front of stage app which other people get to use you should go for a database designed for concurrent access (MySQL, MongoDB, PostgreSQL etc) instead of SQLite. From your question it sounds like SQLite will be fine though.
I haven’t used tweepy myself, but if it works for you then stick with it. The Twitter HTTP APIs are pretty painless to talk to directly though provided you have a good OAuth library (I use python-oauth2) and that might give you more flexibility, but don’t bother with that unless the library you are already using starts to get in your way.
More recent articles
- Datasette Enrichments: a new plugin framework for augmenting your data - 1st December 2023
- llamafile is the new best way to run a LLM on your own computer - 29th November 2023
- Prompt injection explained, November 2023 edition - 27th November 2023
- I'm on the Newsroom Robots podcast, with thoughts on the OpenAI board - 25th November 2023
- Weeknotes: DevDay, GitHub Universe, OpenAI chaos - 22nd November 2023
- Deciphering clues in a news article to understand how it was reported - 22nd November 2023
- Exploring GPTs: ChatGPT in a trench coat? - 15th November 2023
- Financial sustainability for open source projects at GitHub Universe - 10th November 2023
- ospeak: a CLI tool for speaking text in the terminal via OpenAI - 7th November 2023
- DALL-E 3, GPT4All, PMTiles, sqlite-migrate, datasette-edit-schema - 30th October 2023