voussoir/timesearch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Increase size of chunks going into db to reduce commit frequency.

Browse source
This commit is contained in:
Ethan Dalool 2020-09-06 21:43:48 -07:00
parent 5adc0943aa
commit eba495e3b7
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
timesearch_modules/get_comments.py
View file

@ -78,7 +78,8 @@ def get_comments(
if do_supplement:
comments = pushshift.supplement_reddit_data(comments, chunk_size=100)
comments = common.generator_chunker(comments, 200)
comments = common.generator_chunker(comments, 500)
for chunk in comments:
step = database.insert(chunk)
message = form.format(