Commit graph

126 commits

Author SHA1 Message Date
e4b769336b Set ytrss log level during flask launch. 2020-11-15 17:57:07 -08:00
81bd7b9ca2 Move playlist paginator out to get a clean looking generator. 2020-11-10 17:54:21 -08:00
294ccb77d9 More logging updates. 2020-11-10 17:53:52 -08:00
648d2af560 Add rss_assisted parameter to Channel.refresh too. 2020-11-10 17:15:19 -08:00
11414b4088 Add log_level param to constructor, like Etq. 2020-11-09 20:22:54 -08:00
13e6f7e1f0 Use vlogging and a distinct logger name instead of __name__. 2020-11-09 20:20:45 -08:00
927b5a559e Add parameter rss_assisted in case you want to turn it off. 2020-11-09 16:48:17 -08:00
b7ce46bef2 Make these keyword arguments kw-only. 2020-11-09 16:32:44 -08:00
aabb6691f0 Use voussoirkit.vlogging for loud. 2020-11-09 16:30:54 -08:00
a3f69573a4 Use requests.Session to greatly speed up RSS requests. 2020-11-09 16:30:25 -08:00
1263123a03 Dedent by performing loop outside of this function. 2020-11-09 16:29:59 -08:00
c2d10da6b6 Add log line when RSS assist fails, fallback to traditional. 2020-11-09 16:23:18 -08:00
c075133bdb Fix pascal_to_loudsnakes to handle repeated caps as single term.
Previous code made "R_S_S_ASSIST_FAILED", now "RSS_ASSIST_FAILED".
2020-11-09 16:19:37 -08:00
afe82b9c98 Add a dash more logging. 2020-10-25 20:16:43 -07:00
6270665737 Add repr for channel and video. 2020-10-25 20:16:24 -07:00
08290d7c6d Fix incorrect call to log. 2020-10-25 20:16:00 -07:00
b8ab1503c3 Don't use RSS when force=True. 2020-10-23 07:44:09 -07:00
6cc5fed930 Improve error handling surrounding most_recent_video. 2020-10-02 12:31:39 -07:00
b2115f88be Add better indexes and delete redundant ones. 2020-10-02 11:22:47 -07:00
7910e12bc8 Add RSS-assisted refresh to single channel refresh method. 2020-10-02 11:13:02 -07:00
38c7226ec2 Use pathclass.Path.makedirs. 2020-09-24 14:17:44 -07:00
eec72db4fd Add log.debug when RSS doesn't contain ID, and rename variables. 2020-09-23 19:02:01 -07:00
4f6080859a Synchronize Etiquette and YCDL. 2020-09-22 02:59:17 -07:00
248152c24a Use pathclass.Path.open. 2020-09-20 18:32:19 -07:00
741cddc2e4 Add set_queuefile_extension method, endpoint, and ui. 2020-09-16 10:23:38 -07:00
2f9444542f Fix OutOfDate message incorrectly referring to nested directory. 2020-09-04 15:56:06 -07:00
e5910c55f2 Rename database column download to state. 2020-09-04 15:55:48 -07:00
1eb9941e27 Fix call to get_video should have been get_videos. 2020-08-20 13:17:07 -07:00
720db06d4c Set cache_discovery=False to get rid of that console warning. 2020-08-20 13:16:55 -07:00
8eaae26e55 Add experimental RSS-based refresh_all_channels. 2020-08-12 12:32:01 -07:00
2da4568fb5 Create separate get_video/get_videos, plural looks better.
Not a big difference, but I didn't like the way
calling get_video([...]) looked.
2020-08-12 11:20:43 -07:00
299ea6b579 Simplify this useless two-step assignment. 2020-08-12 11:17:30 -07:00
3f3eda58b1 Move logging basicconfig effort to frontmost application layer. 2020-08-12 11:17:10 -07:00
7a7b1455ef Fix queuefile extension selection when NoSuchChannel. 2020-08-12 10:26:57 -07:00
323b2ead54 Use isodate instead of strptime to parse published date. 2020-08-11 23:20:26 -07:00
dfa5bb2390 Move some heavy lifting into new ingest_video. 2020-08-11 23:19:14 -07:00
ce028e4ebb Rearrange some lines for clarity, comment on known_ids. 2020-08-11 23:14:34 -07:00
b1e35f6604 Rewrite chunk_sequence to work with generators.
At the cost of always returning lists instead of the input's type,
which was never relevant anyway.
2020-08-11 23:12:22 -07:00
59eab3f997 Rename directory -> download_directory; add queuefile_extension. 2020-08-10 05:46:18 -07:00
0cf422977b If upload_playlist is not set during refresh, try to get it.
This is possible if a channel is temporarily suspended during the
time you tried to fetch the playlist id originally.
2020-07-26 18:47:34 -07:00
795aacb044 Add ChannelNotFound exception. 2020-07-20 19:03:53 -07:00
4911cee910 Use generators instead of lists for loading / searching videos.
Previously, creating the list meant that all database rows would be
loaded and objected even if you had a limit in place on your page
query, which was a huge waste.
2020-07-01 17:54:36 -07:00
de5992c906 Alphabetize methods. 2020-07-01 16:10:48 -07:00
4fe4ed9cb5 Database version 5. Store uploads_playlist id in database.
I am trying to cut down the API cost of refreshing all channels.
I hoped to find and endpoint that would give me videos from
multiple playlists at once, but Youtube doesn't seem to support
that. Next best option is to at least store the ID of the uploads
playlist, saving 2 api points each.
2020-07-01 16:09:11 -07:00
5d000f87c3 Fix DatabaseOutOfDate exception using wrong variable names. 2020-07-01 15:59:49 -07:00
215a46591e Remove support for username argument in get_user_videos.
Just get their id first with get_user_id.
2020-07-01 15:56:44 -07:00
e1eb53a114 Add method get_user_uploads_playlist. 2020-07-01 15:54:40 -07:00
3ff5de67bc Remove redundant _iter_playlist, just use get_playlist_videos. 2020-07-01 15:52:59 -07:00
8d28484240 Add multicolumn index on download, published.
This helps generate the "sub box" page more quickly, which
was previously using temp b-tree for sorting.
2020-06-24 22:28:57 -07:00
226bc03bff Add methods Channel.delete and Video.delete. 2020-06-18 09:01:05 -07:00
3f9565b874 Add video id to this print statement. 2020-06-17 15:55:30 -07:00
e3abf5f39d Fix automark:download by marking state after calling download.
The previous order of operations was wrongly triggering the
"video already downloaded" check in ycdldb.
2020-06-17 15:54:59 -07:00
b3d3e4ae6c Use .get instead of [] to handle missing some attributes.
Some of these are ones I've experienced first-hand, others are
defensive because I realized I don't know what kind of weird
things might be out there. Can do more if necessary.
2020-06-17 08:40:18 -07:00
8fcba8f05b Add more attributes to ytapi.Video from API data.
Not actually used by the application at the moment, but in the
interest of making this particular file a better representation
of the API, this felt like an easy change that I can take advantage
of later now that I have it.
2020-05-26 22:05:08 -07:00
dd99ddc10f Remove youtube_dl_function argument, but add configurable extension.
In order to improve the configurability of the queuefile creation,
which I can't really do when relegating that to a possibly third-party
function with only the video id as argument, I've decided I want to
go all in on the queuefile as the output of ycdl. Actually downloading
the video is best left to another tool designed for the task.
Any third-party downloading function would always introduce the
possibility of network errors and crashes, ruining the call stack
of ycdldb.download_video for no good reason.
2020-05-24 21:07:14 -07:00
5c4b2e9d21 Add jsonify.py. 2020-05-21 20:28:52 -07:00
57d87560a4 Add interface for setting automark state. 2020-05-21 17:28:34 -07:00
34f24d9940 Let default state of automark be pending, not None. 2020-05-21 17:25:06 -07:00
e165b16edc Rstrip the "Z" off of timestamps.
This only started occuring recently.
2020-05-06 21:23:24 -07:00
cd7a59cbfa Sort channel names case-insensitive. 2020-05-06 21:23:02 -07:00
0562c51d6c Use mark_state instead of doing the sql update here. 2020-04-15 16:34:33 -07:00
2f5ec40b89 Big: Object-oriented channels & videos, match Etiquette structure.
- Channels and videos are now objects instead of just dictionaries.
- Copied Etiquette's use of cachemanager mixin to cache those objects.
- Copied Etiquette's use of sql_ methods.
- Copied Etiquette's use of namespaced javascript.
- Copied Etiquette's use of config file.
- Redid video_card css to use grid, better on mobile.
- Improved usage of URL parameters with class=merge_class.
- Wrote some actual content on readme.
2020-04-05 18:59:16 -07:00
1bc3df5efe Split DB_INIT into pragmas and full init, like Etiquette. 2020-04-03 11:23:51 -07:00
11a147fbcd Use insert_filler for add_channel. 2020-04-03 11:12:56 -07:00
718399806a Add channel_name to videos before returning them, show on listings.
Previously, when viewing a /videos listing, there was a link
called (Chan) to bring you to the channel page, but since videos
only carried author_id and not author_name it was always kind of
ugly. This will inject that attribute on the way out.
I know, this would be more properly written as an SQL join in the
first place, but my row-dict conversion isn't set up for that and
I'm planning on converting this all to object-based returns instead
of dicts soon.
2020-04-03 08:49:35 -07:00
d0bfbaadfe Add method insert_playlist. 2020-03-17 20:53:18 -07:00
1a7662b6bd Add method get_playlist_videos. 2020-03-17 20:53:10 -07:00
b5f892d595 Add ?orderby url parameter. 2020-03-11 18:02:09 -07:00
b48c2fc37c Rename ycdl.py -> ycdldb.py; add exceptions.py. 2020-03-11 14:27:27 -07:00
6691c49481 Add parameter skip_failures to refresh_all_channels. 2020-01-19 10:53:49 -08:00
78ce6a6f41 Add views column and some other database changes.
And various other changes which have been sitting here for too long.
2020-01-06 22:07:25 -08:00
420a14bb88 Add a logger to ytapi. 2020-01-06 22:03:18 -08:00
74607b7e60 Index (author, download) so that listing all channels is faster. 2019-01-31 17:43:01 -08:00
c667ebd872 Batch requests & filterbox & update metadata
- Download / Ignore on multiple videos is batched into a single
  request, like it should have been all along.
- Text box at the top of the page lets you search for terms live
  instead of loading various ?q urls.
- Doing a force refresh will actually update the metadata of old
  videos instead of skipping them.
2019-01-23 21:22:09 -08:00
f1f12423b1 Changes
- Add thumbnails to the card (dynamically loaded)
- Add button for embedding the video
- Store video duration in db (use update script)
- Separate links for filter views with and without query.
2018-12-17 19:17:53 -08:00
ba1961349c It's been a while 2017-10-08 21:39:07 -07:00