else/OpenDirDL/README.md

86 lines
5.2 KiB
Markdown
Raw Permalink Normal View History

2016-01-24 20:48:39 +00:00
Open Dir DL
===========
2016-08-18 01:24:38 +00:00
The open directory downloader.
2016-07-29 20:39:04 +00:00
2017-01-16 06:04:20 +00:00
## Installing requirements
2016-11-18 06:24:52 +00:00
2017-01-16 06:04:20 +00:00
pip install -r requirements.txt
2016-11-18 06:24:52 +00:00
## Usage
2016-07-29 20:39:04 +00:00
See inside opendirdl.py for usage instructions.
2016-11-18 06:24:52 +00:00
 
### Changelog
2016-10-22 03:47:08 +00:00
- **[addition]** A new feature was added.
- **[bugfix]** Incorrect behavior was fixed.
2017-01-16 06:04:20 +00:00
- **[change]** An existing feature was modified or parameters were renamed.
2016-10-22 03:47:08 +00:00
- **[cleanup]** Code was improved, comments were added, or other changes with minor impact on the interface.
- **[removal]** An old feature was removed.
 
2017-01-16 06:04:20 +00:00
- 2017 01 06
- **[cleanup]** Much of the file tree builder has been moved to a new file in the voussoirkit, `pathtree.py`. The goal is to generalize the concept of a pathtree so I can start using it in other local-disk applications instead of having it locked into opendirdl's URL database. They were entangled very badly so it's still messy until I get the division of labor sorted out. Running the `tree` command on the commandline still works exactly the same as before.
2016-11-11 23:55:22 +00:00
- 2016 11 11
2016-11-11 23:52:25 +00:00
- **[addition]** You can now call opendirdl using the database filename as the first argument, and the subcommand as the second. Previously, the subcommand always had to come first, but now they are interchangeable when the system detects that argv[0] is a file. This makes it much easier to do multiple operations on a single database because you can just backspace the previous command rather than having to hop over the database name to get to it.
- **[addition]** `measure` now takes an argument `--threads x` to use `x` threads during the head requests.
- **[addition]** New subcommand `list_urls` to just dump the urls.
2016-10-04 03:29:24 +00:00
- 2016 10 03
- **[bugfix]** Fix KeyError caused by the 'root' -> 'domain' rename.
2016-10-04 02:20:58 +00:00
- 2016 10 01
- **[bugfix]** Fixed the download function so it actually passes `headers` into downloady.
- **[change]** `url_split` key 'root' has been renamed to 'domain'.
2016-10-22 03:47:08 +00:00
- **[change]** Improved some variable names, including `walkurl -> root_url`.
2016-10-04 02:20:58 +00:00
- **[cleanup]** Removed import for Ratelimiter since downloady handles all of that now.
2016-08-18 01:24:38 +00:00
- 2016 08 16
- **[cleanup]** Now that Downloady uses temp files for incomplete downloads, that logic can be removed from opendirdl.
2016-08-13 00:26:12 +00:00
- 2016 08 10
2016-08-18 01:24:38 +00:00
- **[addition]** Added clickable links to each directory on HTML tree pages.
- **[bugfix]** Fixed bug in smart_insert caused by 404's being considered falsey, triggering the 'one and only one' exception.
2016-11-18 06:24:52 +00:00
- **[bugfix]** Fixed bug in smart_insert where 404'd URLs were not being deleted from the database.
2016-08-13 00:26:12 +00:00
2016-08-03 01:44:32 +00:00
- 2016 08 02
2016-08-18 01:24:38 +00:00
- **[cleanup]** Removed the need for div IDs on the Tree pages by making the collapse button use `this.nextSibling`.
- **[cleanup]** Rewrote `build_file_tree` with a way simpler algorithm.
- **[removal]** Removed the ability to set a Node's parent during `__init__` because it wasn't fully fleshed out and doesn't need to be used anyway.
2016-08-03 01:44:32 +00:00
2016-08-01 23:42:03 +00:00
- 2016 08 01
2016-08-18 01:24:38 +00:00
- **[addition]** Made the digest work even if you forget the http://
2016-08-01 23:42:03 +00:00
2016-07-29 20:39:04 +00:00
- 2016 07 29
2016-08-18 01:24:38 +00:00
- **[change]** Moved some nested function definitions out to the top level, and made the construction of the file tree its own function. These functions really don't need to be used on their own, but they were cluttering the logic of the `tree` command.
- **[change]** Renamed `Tree.listnodes` to `Tree.list_children`, and the `customsort` now expects to operate on Node objects rather than `(identifier, Node)` tuples. Nodes already have their identifier so the tuple was unecessary.
- **[change]** Replaced local `download_file` function with a call to `downloady.download_file`. It supports download continuation and removes duplicate work.
- **[cleanup]** Replaced all `safeprint` calls with `write` because it provides access to safeprint as well as file writing if needed.
- **[removal]** Removed `Tree.sorted_children` since it was basically a duplicate of `Tree.listnodes` and I don't know why I had both.
2016-07-29 20:39:04 +00:00
2016-07-28 03:41:13 +00:00
- 2016 07 25
2016-08-18 01:24:38 +00:00
- **[change]** Bytespersecond is now parsed by `bytestring.parsebytes` rather than `eval`, so you can write "100k" as opposed to "100 * 1024" etc.
2016-11-18 06:24:52 +00:00
- **[removal]** Removed the `Downloader` class and replaced it with a function, because it only had one real method anyway.
2016-07-28 03:41:13 +00:00
2016-07-20 03:31:47 +00:00
- 2016 07 19
2016-08-18 01:24:38 +00:00
- **[addition]** Gave the HTML tree divs a very gentle shadow and alternating colors to help with depth perception.
- **[bugfix]** Fixed the allowed characters parameter of `filepath_sanitize`, which was not written correctly but worked out of luck.
- **[cleanup]** Rearranged the big blocks to be in a logical order rather than alphabetical order. Walker > Downloader > other classes
- **[cleanup]** Renamed the `keep_pattern` and `remove_pattern` functions to `keep_pattern_argparse` etc to be consistent with the other functions used by the argparser. *Does not affect the commandline usage!*
- **[cleanup]** Fixed some mismatched code vs comments
2016-07-20 03:31:47 +00:00
2016-07-10 04:38:49 +00:00
- 2016 07 08
2016-08-18 01:24:38 +00:00
- **[bugfix]** Fixed bug in which trees wouldn't generate on server:port urls.
2016-07-10 04:38:49 +00:00
2016-07-05 07:24:08 +00:00
- 2016 07 04
2016-08-18 01:24:38 +00:00
- **[addition]** Added new argparse command "tree"
2016-07-05 07:24:08 +00:00
- 2016 02 08
2016-08-18 01:24:38 +00:00
- **[bugfix]** Fixed bug where server:port urls did not create db files because of the colon. It's been replaced by a hash.
- **[change]** Moved db commits to only happen at the end of a digest.