voussoir
/
timesearch
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Increase size of chunks going into db to reduce commit frequency.

master
Ethan Dalool 2020-09-06 21:43:48 -07:00
parent 5adc0943aa
commit eba495e3b7
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
timesearch_modules/get_comments.py
View File

@ -78,7 +78,8 @@ def get_comments(
if do_supplement: if do_supplement:
comments = pushshift.supplement_reddit_data(comments, chunk_size=100) comments = pushshift.supplement_reddit_data(comments, chunk_size=100)
comments = common.generator_chunker(comments, 200)
comments = common.generator_chunker(comments, 500)
for chunk in comments: for chunk in comments:
step = database.insert(chunk) step = database.insert(chunk)
message = form.format( message = form.format(