hnarchive downloads all HN items (threads and comments) into an SQLite database. At this time, my database is 23.18 GiB with just over 25,000,000 items. I'd be happy to share it.
pip install requests and
pip install voussoirkit.
According to the HN API docs there is no enforced ratelimit, so just use a
threads count that seems polite.
To get started, just run
python hnarchive.py update and it will start from 1. In the future, you can run
update on a cronjob or use
livestream to get new items forever.
updatealways starts from the highest ID in the database. If you use
getto get a range of IDs that is ahead of your update schedule, your next
updatewill miss the skipped IDs.
update_itemswill overwrite previously fetched data with the new properties. Please know that HN moderators occasionally migrate comments between threads, adjust thread titles, etc. HN has a tight window in which authors can edit their own posts so you can expect actual item texts to remain pretty static outside of moderator action.
The exception is if an item is deleted and comes back as
Nonefrom the server, then hnarchive keeps the old data.
Here are all of the subcommands:
get: Get items between two IDs, inclusive. flags: --lower id: Lower bound item ID. --upper id: Upper bound item ID. --threads X: Use X threads to download items. Default = 1 thread. --commit_period X: Commit the database after every X insertions. Default = 200. livestream: Watch for new items in an infinite loop. flags: --commit_period X: Commit the database after every X insertions. Default = 200. update: Get new items, from the highest ID in the database to the present. flags: --threads X: Use X threads to download items. Default = 1 thread. --commit_period X: Commit the database after every X insertions. Default = 200. update_items: Redownload items to update their scores, descendant counts, etc. flags: --days X: Update items where the retrieval date is less than X days ahead of the submission date. Stories are only open for comments for 14 days, so the `descendants` count of any story younger than 14 days should be considered volatile. It seems the upvote button does not disappear at any age, though I don't know whether votes on old submissions will actually count. Regardless, votes and comments tend to solidify within a day or two after submission so a small number should be sufficient. --threads X: Use X threads to download items. Default = 1 thread. --only_mature: If True, only update items where the submission date is more than 14 days ago. Without this, you will be updating items which are very close to the present time, an effort which you may find wasteful. --commit_period X: Commit the database after every X insertions. Default = 200.