ed1b7dc4eb
Add module ingest_jsonfile.
2023-06-25 13:24:39 -07:00
01f2cdbf4a
Render wiki markdown while we're writing it.
2023-04-02 12:25:26 -07:00
82bc922cca
Transform comment parent_id from pushshift api.
...
Re #19
2023-02-22 17:50:44 -08:00
2c0ae7291a
Hardcode pushshift ratelimit due to removal of /meta.
2023-02-22 17:17:45 -08:00
c8c160e00e
Take crosspost parent as url when available.
2022-09-10 11:30:58 -07:00
806429d3f5
Fix column name of edited text.
...
Why didn't this error before? I shudder to think.
2022-09-03 06:25:38 -07:00
d3154e7b51
Update use of sqlhelpers.insert_filler.
2022-08-13 13:07:12 -07:00
6867d32798
Use separate loggers per module instead of common.log.
...
The separate names assist in debugging.
2021-11-14 13:48:51 -08:00
baded8e5fb
Remove --verbose since vlogging handles that.
2021-11-14 13:37:20 -08:00
925d6d587f
Use old.reddit.com.
2021-11-04 16:37:42 -07:00
e8107d9678
Remove separate permalink element, just link the timestamp.
2021-11-04 16:37:37 -07:00
e938a100a0
Don't call tsdb.for_submission if the subreddit is given.
2021-11-04 16:35:36 -07:00
50f61d792c
Print 404 error separately from general stacktrace.
2020-11-27 15:41:08 -08:00
Ethan Dalool
2750017a17
Use requests.Session.
2020-11-10 20:20:55 -08:00
Ethan Dalool
e2d1f2b9ef
Some small whitespace and log tweaks.
2020-10-08 14:17:19 -07:00
Ethan Dalool
55d9c064bd
Move thank you message into the pushshift module itself.
2020-10-08 13:47:02 -07:00
Ethan Dalool
0aa1ad1596
Use named logger 'timesearch'.
2020-10-08 13:15:58 -07:00
Ethan Dalool
3869e5a99a
Use the same print format for get_submissions and get_comments.
2020-10-08 13:15:05 -07:00
Ethan Dalool
b412b5fd98
Move contents of ts_modules/__init__.py to timesearch.py.
...
I realized that this two-layer launch was unnecessary.
2020-10-08 13:12:57 -07:00
Ethan Dalool
741f47d0b9
Use betterhelp.subparser_main instead of decorator.
...
To get the parser off the global.
2020-10-07 00:32:44 -07:00
Ethan Dalool
c27f8038c7
Use pathclass.Path.makedirs.
2020-09-24 14:19:10 -07:00
Ethan Dalool
3373e36f64
Use pathclass.Path.open.
2020-09-20 18:30:13 -07:00
Ethan Dalool
0bd52eb08b
Bump version to 2020.09.06.0.
2020-09-06 21:44:07 -07:00
Ethan Dalool
eba495e3b7
Increase size of chunks going into db to reduce commit frequency.
2020-09-06 21:43:48 -07:00
Ethan Dalool
5adc0943aa
Use pushshift's /meta endpoint to discover ratelimit.
2020-09-06 21:42:30 -07:00
Ethan Dalool
e136aeea4e
Log the size of received batches from pushshift.
2020-09-06 21:41:44 -07:00
Ethan Dalool
83d5a8e70e
Fix retry loop not breaking after successful try!
2020-09-06 21:31:46 -07:00
Ethan Dalool
4a61d820b3
Add skewer-case aliases for snake_case arguments.
2020-08-12 10:24:31 -07:00
Ethan Dalool
1132c0aee1
Remove unnecessary import requests from merge_db.
2020-05-21 22:30:12 -07:00
Ethan Dalool
10699893ea
Allow get_comments to create the database.
...
This was false because prior to Pushshift, commentaugment worked by
reading the comments tree on every submission, and thus couldn't be
done first. Now, I think there's nothing wrong with getting comments
before submissions.
2020-05-12 18:19:46 -07:00
Ethan Dalool
b98f096fd4
Replace "> timesearch.py" with "python timesearch.py"
...
The bracket was meant to look like the command prompt, but for
linux users they would be more used to $ or #, and for people who
do not use the command line often they may be confused by it
altogether. This change should help make it a little more clear as
to what's going on.
2020-04-22 23:25:42 -07:00
Ethan Dalool
45f6e36fad
Add some whitespace between these parts of the help summary.
2020-02-08 21:10:48 -08:00
Ethan Dalool
d027cce562
main should return args.func(args) value.
2020-02-08 16:20:19 -08:00
Ethan Dalool
263232e0a0
I keep forgetting I have a version number.
2020-02-08 15:51:59 -08:00
Ethan Dalool
dd38e0c8da
Print a helpful message when bot.py or bot4.py isn't found.
2020-02-08 15:51:45 -08:00
Ethan Dalool
5345e8f4fc
Use betterhelp to manage docstrings, aliases, and helptexts.
2020-01-30 23:25:24 -08:00
Ethan Dalool
31296000f9
Alphabetize & organize helptexts with new names.
2020-01-27 20:13:01 -08:00
Ethan Dalool
02b2dc166e
Fix docstring incorrectly referring to constants.
2020-01-27 20:09:02 -08:00
Ethan Dalool
24f155a40d
Update OutOfDate message to show actual argument.
2020-01-27 20:00:08 -08:00
Ethan Dalool
d12e66f19c
DB v2. Rename redmash folder to index.
2020-01-27 19:53:27 -08:00
Ethan Dalool
0ef423d780
Create _first_time_setup, _load_pragmas, and _check_version.
2020-01-27 19:51:23 -08:00
Ethan Dalool
ecfb96820c
Correct index helptext to say index instead of mash.
2020-01-27 19:21:41 -08:00
Ethan Dalool
1fb281bf7f
Rename mergedb.py -> merge_db.py.
2020-01-27 18:57:06 -08:00
Ethan Dalool
d2220a3ced
Rename getwiki.py -> get_wiki.py.
2020-01-27 18:54:04 -08:00
Ethan Dalool
6b0dd55431
Rename getstyles.py -> get_styles.py.
2020-01-27 18:53:19 -08:00
Ethan Dalool
2022db2b0c
Rename redmash.py -> index.py.
2020-01-27 18:45:36 -08:00
Ethan Dalool
744dadd3ee
Rename commentaugment.py -> get_comments.py.
2020-01-27 18:41:58 -08:00
Ethan Dalool
75648343e2
Rename timesearch.py -> get_submissions.py.
2020-01-27 18:39:54 -08:00
Ethan Dalool
b3db322a78
Rename timesearch directory -> timesearch_modules.
...
This marks the first in a long series of rename commits.
Hold on to your databases!
2020-01-27 18:32:05 -08:00
