Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python?
My answer to Twitter API: What is the best data storage mechanism and client library for analysing tweets using Python? on Quora
It depends on how much data you intend to collect, and how you intend to then share that data.
SQLite should be fine if you are collecting thousands of tweets—if you are collecting millions it will probably be OK but you might be better off with something else.
If you plan to put a web interface in front of stage app which other people get to use you should go for a database designed for concurrent access (MySQL, MongoDB, PostgreSQL etc) instead of SQLite. From your question it sounds like SQLite will be fine though.
I haven’t used tweepy myself, but if it works for you then stick with it. The Twitter HTTP APIs are pretty painless to talk to directly though provided you have a good OAuth library (I use python-oauth2) and that might give you more flexibility, but don’t bother with that unless the library you are already using starts to get in your way.
More recent articles
- ChatGPT should include inline tips - 30th May 2023
- Lawyer cites fake cases invented by ChatGPT, judge is not amused - 27th May 2023
- llm, ttok and strip-tags - CLI tools for working with ChatGPT and other LLMs - 18th May 2023
- Delimiters won't save you from prompt injection - 11th May 2023
- Weeknotes: sqlite-utils 3.31, download-esm, Python in a sandbox - 10th May 2023
- Leaked Google document: "We Have No Moat, And Neither Does OpenAI" - 4th May 2023
- Midjourney 5.1 - 4th May 2023
- Prompt injection explained, with video, slides, and a transcript - 2nd May 2023
- download-esm: a tool for downloading ECMAScript modules - 2nd May 2023
- Let's be bear or bunny - 1st May 2023