Commit graph

451 commits

Author SHA1 Message Date
4c4fae62cf
Delete clipext.py. 2021-08-17 14:01:00 -07:00
8ce994a54b
Replace use of clipext with pipeable, split_lines=False. 2021-08-17 14:00:29 -07:00
cb5228fe0c
Add argument split_lines for when you just want the whole text.
Calling '\n'.join(pipeable.input()) is not great because in many
situations, you're splitting text and joining it back together again
for no reason. With this option, we can let the code do the right
thing, and use pipeable as a general-purpose inputter in
more situations.
2021-08-17 14:00:11 -07:00
db8c6e42e3
Add TypeError if arg is not a string. 2021-08-17 13:48:57 -07:00
f5e5db1908
Let hms use pipeable.input_many instead of go. 2021-08-17 13:46:47 -07:00
93605e2577
Split long useragent string. 2021-08-13 23:40:29 -07:00
6442e8e40c
Fix ctrlc_return1 to return the wrapped return value. 2021-08-13 23:40:29 -07:00
14dcbda9fd
Add networktools, though some ideas are still not implemented. 2021-08-13 23:40:28 -07:00
1d6f9eee10
Add niceprints.py. 2021-08-13 23:40:28 -07:00
72fdcb2b04
Clarify pipeable.input_many docstring. 2021-08-11 01:01:14 -07:00
cdf551f28a
Let glob_filenames, glob_directories be singular strings. 2021-08-11 01:01:02 -07:00
eddd7b3818
Remove set_level_by_argv. 2021-08-10 11:54:04 -07:00
61db9296c8
Improve some docstrings. 2021-08-10 11:53:55 -07:00
16d9b85e62
Various lint appeasements. 2021-08-10 11:37:01 -07:00
63251b979b
Describe Path parameters in docstring. 2021-08-09 16:27:33 -07:00
81782383a9
Use regular path.join when constructing glob patterns.
I don't like the idea of having an intermediate Path object that isn't
a real path.
2021-08-09 08:36:20 -07:00
41dd3eb0db
Clarify docstring reference to LogHandlerContext. 2021-08-09 08:35:12 -07:00
47e3a9d7d4
Add imagetools.pad_to_square. 2021-08-09 08:34:58 -07:00
c65743ec02
Use random.choices instead of random.choice in a loop. 2021-08-09 08:33:02 -07:00
13bf8e2e47
Improve Windows reserved name check by adding to pathclass. 2021-07-24 12:24:24 -07:00
a12bc59b17
Remove some old unnecessary globals. 2021-07-24 12:21:51 -07:00
7545433253
Improve some docstrings. 2021-07-04 15:45:00 -07:00
e9d450029d
Log the program's final return when using @main_decorator. 2021-07-04 15:44:50 -07:00
f18f0fd2df
Remove option log_exception, always log it under our logger.
This unifies the thought process so that everything is done via the
log lines, not these separate config variables. Caller can mute this
module if they want to. Also, it was previously appending directly
to the message but now we can have our logger's name on it so it's
more clear who's reporting what.
2021-07-04 15:42:57 -07:00
2971d7e4d6
Improve operatornotify module docstring. 2021-07-02 02:59:34 -07:00
26040d4372
Use vlogging.main_decorator. 2021-07-02 02:59:14 -07:00
4aed87b522
Fix downloady.is_special_file. 2021-07-01 15:50:57 -07:00
c709a28d08
Add operatornotify.main_decorator to reduce application boilerplate. 2021-06-21 22:02:43 -07:00
81d09ad3bc
Add vlogging.main_decorator to reduce application boilerplate. 2021-06-21 21:54:42 -07:00
9cd7713a6b
Replace call to basicConfig with own handler. 2021-06-21 21:53:36 -07:00
689b8f5718
Let rotate_by_exif return new exif with orientation=1. 2021-06-06 20:29:29 -07:00
663e65795a
Add flasktools.make_json_response. 2021-06-04 21:48:04 -07:00
ee8c4b7fd8
Add ensure_response_type decorator.
So that all further decorators or response-processors know they
have a Response object.
2021-06-04 21:26:23 -07:00
45da148493
Add globals REQUEST_TYPES, RESPONSE_TYPES. 2021-06-04 20:58:31 -07:00
68d12e4900
Pass _case_correct=True on trusted values to speed up walk. 2021-06-04 13:07:59 -07:00
b3e0b310c2
Include starting path in blacklist check but not whitelist. 2021-06-04 13:06:10 -07:00
1d555d20dc
Switch to image.getexif, official as of pillow 6. 2021-06-03 21:11:19 -07:00
a36c5ca478
Add stringtools.natural_sorter. 2021-05-30 21:35:21 -07:00
96a0e177e6
Move ORIENTATION_KEY to global. 2021-05-30 21:34:29 -07:00
db60b84ede
Don't bother with 1-element list, just do it right away. 2021-05-25 01:45:34 -07:00
4741946eda
Do None check before isinstance check. 2021-05-18 23:26:44 -07:00
2d8d51a4f1
Docstring updates for many functions. 2021-05-18 17:48:57 -07:00
375b00bfd5
Add dynamic chunk sizing to keep consistent progress bar pace. 2021-05-18 17:48:35 -07:00
e7798574b3
Rename argument fpobj -> path. 2021-05-18 17:45:25 -07:00
0d4349e942
Migrate scripts to main_level_by_argv. 2021-05-16 21:02:18 -07:00
03bcded66a
Add main_level_by_argv, the next iteration of set_level_by_argv.
I have realized that when passing log levels by argv, it's better
for them to go to the root logger instead of the module logger,
since that would not affect the logs that come from imported modules.
2021-05-16 21:02:04 -07:00
cb4cef9339
Allow --operatornotify-level to act as opt in without both flags. 2021-05-16 20:43:28 -07:00
bfb06d46c9
Add parameters force_hours, force_minutes. 2021-05-16 20:31:35 -07:00
a45c5908f0
Explain main_log_context *args. 2021-05-14 09:53:19 -07:00
c744135b21
Only reset the buffer if notify did not raise an exception. 2021-05-14 09:53:02 -07:00
f6ded0a879
Add sqlhelpers.executescript. 2021-05-08 18:26:43 -07:00
ea284c9199
Add hash_class passthrough argument. 2021-05-08 18:23:28 -07:00
313070754f
Add callback_post_file. 2021-05-08 09:39:48 -07:00
133225f77c
Add parens to dotdict repr. 2021-05-08 09:39:22 -07:00
65f766928c
Add 'written' to copy_file results dotdict. 2021-05-08 09:39:13 -07:00
9bdee227b3
Replace dotdict init by dict with init by kwargs. 2021-05-08 09:26:10 -07:00
450f42bca6
Add source to spinal returned dotdict.
Just in case you store the results and process them later, out of
order, etc.
2021-05-08 09:02:22 -07:00
14c9cb5212
Add SILENT as a module constant. 2021-05-08 09:00:48 -07:00
c34f875014
Add operatornotify.LogHandler, LogHandlerContext. 2021-05-07 20:03:56 -07:00
b871a083b1
Add vlogging.get_level_by_name. 2021-05-06 19:29:02 -07:00
9e0509ae17
Improve expressionmatch docstrings. 2021-05-05 19:18:37 -07:00
af5ec0b52e
Add argparse to eternalseptember. 2021-05-03 19:03:28 -07:00
fd296fdbb0
Replace call to pipeable.output with pipeable.stdout. 2021-05-03 18:57:04 -07:00
44271a56c4
Add argparse, betterhelp to bytestring. 2021-05-03 18:55:12 -07:00
12a0c81819
Add docstrings to Backoff class's methods. 2021-05-03 18:32:18 -07:00
60abcbc8e3
Move assignment of backoff.x to the base class. 2021-05-03 18:31:58 -07:00
7bfe0a48ef
Use regular __doc__ instead of own DOCSTRING variable.
So that help(operatornotify) works as expected.
2021-05-03 18:24:37 -07:00
d03c653b46
Add operatornotify.py. 2021-04-28 19:36:01 -07:00
6de9fb3eaf
Fix is_special_file handling of pathclass. 2021-04-28 19:29:18 -07:00
269fea9cb7
Fix special filenames being abspathed. 2021-04-25 20:20:43 -07:00
57e12faca9
Add flasktools.py. 2021-04-02 18:24:40 -07:00
6d6ae3bd5c
Add vlogging to downloady. 2021-03-10 16:37:07 -08:00
c1703357e8
Add main_fallback name to vlogging.getLogger. 2021-01-29 12:12:50 -08:00
80d66c4117
Alphabetize imports. 2021-01-29 12:12:50 -08:00
0cb90875bd
Use safeprint instead of always encoding to ascii. 2021-01-29 12:12:49 -08:00
d9e1e6bf7c
Add bytes_per_second to hash_file. 2021-01-29 12:12:49 -08:00
1d90086d96
Improve spinal.walk with glob_ arguments.
This saves the caller from having to filter the results, and saves
us from instantiating Path objects that we don't want. This improves
spinal's position as the one-stop-shop for file tree iteration.
2021-01-29 12:12:49 -08:00
c893a21e14
Improve consistency of callback_permission_denied, hash_progress. 2021-01-29 12:12:49 -08:00
2bb71c6ac2
Remove callback_exclusion. 2021-01-29 12:12:48 -08:00
0b7eda1e36
Use path.size instead of os.path.getsize. 2021-01-29 12:12:48 -08:00
262580ab41
Use vlogging instead of logging. 2021-01-29 12:12:48 -08:00
9070c5df82
Let hash_class be mandatory argument, return actual hasher object. 2021-01-29 12:12:42 -08:00
b41597ec71
Call callback_progress before the byte limiter sleeps. 2021-01-25 14:18:44 -08:00
51bbe54f2d
Improve winglob docstring.
Clarify that "right results" really means consistent with windows
behavior.
2021-01-25 02:45:07 -08:00
f7e81d4f7f
Add module docstring to pipeable. 2021-01-19 17:04:13 -08:00
32d8137201
Return dotdict instead of list from copy_file, copy_dir. 2021-01-18 12:12:27 -08:00
5c0482032d
Rename callback_v1 -> callback_progress_v1. 2021-01-18 12:04:43 -08:00
aa68744107
Replace duplicated docstring remarks with simple hints. 2021-01-18 01:34:27 -08:00
826c2f5e5d
Simplify this destination_file yielder. 2021-01-18 01:18:55 -08:00
acab8e0c78
Don't remove spaces from input string.
This was allowing inputs like "5 5" to equal 55.
2021-01-18 01:08:35 -08:00
8a0ed3d131
Add BytestringException, ParseError.
Allows for more granular catching on the caller side.
2021-01-18 01:07:08 -08:00
7be7c2c908
Improve clarity of limiter_or_none logic. 2021-01-18 00:54:30 -08:00
e0771afa77 Add input_many. 2021-01-14 02:47:44 -08:00
6666430ed8 Add small comment about stop_event. 2021-01-14 02:47:44 -08:00
ac6149094d Add a few docstrings, remove getpermission.py. 2021-01-13 16:37:28 -08:00
3ef4b31c46 Print log warning when subparser has no docstring. 2021-01-13 16:11:45 -08:00
c4dd1605a5 Add passwordy.random_hex in addition to urandom_hex. 2021-01-05 20:41:22 -08:00
cde65ca8f9 Add imagetools.rotate_by_exif. 2021-01-05 20:41:03 -08:00
9f1ce3238e Rename variable to day_of_month. 2021-01-04 02:40:48 -08:00
166ed18004 Add expiry seconds option to cacheclass. 2021-01-01 16:24:22 -08:00
593dfe5cb0 Add optional args_postprocessor. 2020-12-30 12:31:21 -08:00
f72c68bab2 Improve subproctools.quote. 2020-12-15 13:10:13 -08:00
f2456faf62 Convert line to string before sending it to stdout/stderr. 2020-12-08 23:42:13 -08:00
7969e38872 Remove trailing newlines from lines read from files.
The other forms of pipeable.input don't include trailing newlines,
so this makes read_files more consistent with the rest.
2020-12-07 20:23:14 -08:00
62b44be0d5 Print betterhelp to stderr, not stdout. 2020-12-07 20:09:36 -08:00
6873beb823 Rename output -> stdout, add stderr. 2020-12-07 20:08:39 -08:00
0a1a3a545e Add interactive.py to supercede getpermission with more prompts.
Will delete getpermission at a later date.
2020-12-06 15:48:57 -08:00
60b6278876 Add subproctools.py. 2020-12-06 15:48:57 -08:00
3e1a24a2a6 Move touch.py out of voussoirkit, into voussoir/cmd.
Within voussoirkit, the solution for touching a single file is
pathclass.Path.touch.
2020-12-06 15:48:57 -08:00
4b1e396d9f Add docstring to pipeable.input. 2020-12-06 15:48:57 -08:00
9340e172df Remove unused builtin_input. 2020-12-06 15:48:57 -08:00
86f2186dd6 Move not in_pipe->list(lines) so it is inherent to multiline input. 2020-12-06 15:48:57 -08:00
051b783964 Remove support for input(arg=None).
If the caller doesn't want to specify an arg, they should be using go().
2020-12-06 15:48:57 -08:00
a0c7bd0258 Add docstring to pipeable.go. 2020-12-06 15:48:56 -08:00
8fe13e0f83 Rewrite touch.py with argparse, pipeable. 2020-12-06 15:48:46 -08:00
8ff596a95e Add function is_glob. 2020-12-05 20:04:51 -08:00
bf2865077a Fix to_base for 0.
Further proof that nobody is using this code.
2020-12-05 20:00:27 -08:00
f8c0d46c5e Alphabetize these functions. 2020-12-05 15:07:04 -08:00
244691713d Add constant WINDOWS_BADCHARS for use in other programs. 2020-12-05 15:06:27 -08:00
ebe009f2a2 Add assert_not_file, assert_not_directory to round it out. 2020-12-05 14:55:02 -08:00
44d9bd9913 Add assert_is_link, assert_not_link. 2020-12-05 14:51:51 -08:00
946f329e8c Add helpful decorator ctrlc_return1. 2020-11-30 22:01:37 -08:00
10a447cd07 Add argument read_files so that arg may be a filepath. 2020-11-30 22:01:23 -08:00
1cf2209910 Add function excise. 2020-11-26 18:18:47 -08:00
2cb443cf6d Rename variable s -> text. 2020-11-26 18:18:26 -08:00
adc4247efd Add Backoff.current to get value without stepping forward. 2020-11-20 13:29:33 -08:00
5d83ed655d Add module docstring. 2020-11-18 03:18:01 -08:00
29541d2570 Add stringtools.py, gentools.py. 2020-11-18 03:17:54 -08:00
Ethan Dalool
2cbb93386e Improve binding of loud. 2020-11-09 16:31:19 -08:00
Ethan Dalool
03e5830133 Add some docstrings. 2020-11-09 11:04:40 -08:00
Ethan Dalool
63689e02c0 Fix result_generator hanging when the job generator produces nothing. 2020-11-09 11:03:08 -08:00
Ethan Dalool
cbf546ee69 Add level_by_argv level --silent. 2020-11-08 20:12:19 -08:00
Ethan Dalool
e3d9336383 Pull code out to separate function get_level_by_argv. 2020-11-08 20:12:05 -08:00
Ethan Dalool
1468f318f8 Handle sys.stdin, stdout are None under pythonw. 2020-11-05 22:00:36 -08:00
Ethan Dalool
2ec77f287b Add vlogging.py. 2020-11-02 23:18:53 -08:00
Ethan Dalool
d29501bae7 Having removed self.spawn(other), add back Path(other). 2020-11-02 23:18:19 -08:00
Ethan Dalool
426da9b52b Let listdir spawn children with _correct_case already set. 2020-10-28 14:56:52 -07:00
Ethan Dalool
adf7353ae6 Cache _case_correct property to cut down on fs calls. 2020-10-28 14:55:49 -07:00
Ethan Dalool
1364d66264 Let kbps return all three values always.
Instead of just the missing value.
2020-10-23 07:47:10 -07:00
Ethan Dalool
3347962438 Fix normalization of paths during blacklist check. 2020-10-23 07:46:29 -07:00
Ethan Dalool
5232ca81fc Fix dotdict being subscripted instead of dot. 2020-10-23 07:46:06 -07:00
Ethan Dalool
e3cc64c19c Remove useless const from argparse arg. 2020-10-23 07:45:31 -07:00
Ethan Dalool
c0e9870b91 Rewrite threadpool with persistent threads that pick up jobs. 2020-10-12 16:34:26 -07:00
Ethan Dalool
4e9b43be8b Big threadpool update. 2020-10-09 13:10:24 -07:00
Ethan Dalool
12175c23a7 Add lazychain.py. 2020-10-09 13:09:51 -07:00
Ethan Dalool
d56eea85e6 Fix punctuation of these statements. 2020-10-08 09:39:41 -07:00
Ethan Dalool
5147d7b988 Move nested function do_it out to _run. 2020-10-06 23:09:06 -07:00
Ethan Dalool
0202aeaca0 Add backoff.py. 2020-10-06 11:18:25 -07:00
Ethan Dalool
d37dbe40f8 Add base exception class DownloadyException. 2020-09-25 16:09:42 -07:00
Ethan Dalool
0613fd10e4 Use DotDict for the download plans. 2020-09-25 15:56:24 -07:00
Ethan Dalool
0948139883 Rename variable localname -> temp_localname within download_plan. 2020-09-25 15:50:59 -07:00
Ethan Dalool
3deb6270ba Add support for more special filenames like Windows's "con". 2020-09-25 15:43:36 -07:00
Ethan Dalool
677d8a251f Don't worry with .seek if we're doing a fulldownload plan. 2020-09-25 15:42:39 -07:00
Ethan Dalool
ade7d63231 Use voussoirkit safeprint instead of duplicate function. 2020-09-25 15:30:49 -07:00
Ethan Dalool
6b546b39df Split safeprint into separate functions to remove deep indents. 2020-09-25 15:30:00 -07:00
Ethan Dalool
025c8d51a3 Use pathclass.Path.makedirs. 2020-09-24 14:17:14 -07:00
Ethan Dalool
a05d34ea9e Add Path.makedirs. 2020-09-24 14:16:51 -07:00
Ethan Dalool
ba88731ae2 Fix cwd. 2020-09-22 01:55:30 -07:00
Ethan Dalool
342062dd99 Use pathclass.Path.open. 2020-09-20 18:25:12 -07:00
Ethan Dalool
c6d7450028 Add Path.open. 2020-09-20 17:31:28 -07:00
Ethan Dalool
dcf84dedf8 Escape single quotes by doubling them up. 2020-09-15 14:03:41 -07:00
Ethan Dalool
a2337346fb Add helpful pathclass.cwd function. 2020-09-12 10:15:14 -07:00
Ethan Dalool
5510d94054 Add docstrings to all functions. 2020-09-09 14:40:37 -07:00
Ethan Dalool
331de6eaec Replace chunk_size=None with CHUNK_SIZE.
The idea was that you could overwrite the module-level CHUNK_SIZE so
that all future quickids would use it as the default, but I don't like
the code clutter and anybody who wants to change the default should
write their own partial function if it's that important.
2020-09-09 14:39:30 -07:00
Ethan Dalool
d95d1c9533 Add matches_handle, matches_file. 2020-09-09 14:34:50 -07:00
Ethan Dalool
d924e273b9 Let quickid functions take pathclass paths as arguments. 2020-09-09 14:34:08 -07:00
Ethan Dalool
52afa2489a Let quickid use any of hashlib.algorithms_guaranteed. 2020-09-09 14:33:45 -07:00
Ethan Dalool
7052b2c432 Fix __str__ in case self.token is not a string. 2020-09-09 12:55:09 -07:00
Ethan Dalool
bab1c2bffc Add callback_pre_directory, callback_pre_file.
These callbacks lets you inspect the anticipated copy operation before
it actually occurs. My particular motivation for adding this was to
search for source files which have been renamed, so I can rename the
matching destination file before it attempts to make a duplicate.
With the addition of the BAIL sentinel, you could also use this to
dynamically decide which dirs to skip, if the exclude_ sets are not
good enough for you.
2020-09-06 11:23:49 -07:00
Ethan Dalool
9ef8254191 Let walk_generator follow links, add skip_symlinks to copy_dir. 2020-09-06 11:17:30 -07:00
Ethan Dalool
5e5f55cf1f Remove "Default" from docstrings.
These are already visible with help(), and are cluttering the page.
2020-09-06 11:13:44 -07:00
Ethan Dalool
7840d53212 Move source.correct_case out of conditional.
The reasoning was that in the non-new_root case, the destination supplied
by the user would either be the exact path of the new file, or the parent
of the new file. Either way, the casing of source shouldn't matter since
only the file itself is being created with the given casing.

But, I realized, since we are doing makedirs in the event that the whole
path leading up to destination doesn't exist, we should create it with
the correct case according to the source. And that can occur regardless
of new_root.
2020-09-06 09:54:28 -07:00
Ethan Dalool
26eaac7119 Simplify this call to makedirs with destination.parent. 2020-09-06 09:32:12 -07:00
Ethan Dalool
b34f6fafed Rename callback_file -> callback_file_progress. 2020-09-05 21:21:56 -07:00
Ethan Dalool
36c7b5447f Rename callback_directory -> callback_directory_progress. 2020-09-05 21:21:38 -07:00
Ethan Dalool
9874f12a2a Return self from correct_case instead of self.absolute_path. 2020-09-02 09:11:49 -07:00
Ethan Dalool
27291df9ad Sort Pathclass by normcase instead of absolute path. 2020-09-02 08:52:13 -07:00
Ethan Dalool
53528741b2 Assert exists before trying to calculate size; explicit elif.
If you have a path that you expect to expect to be a file, it is
confusing to see the traceback where it attempts to listdir because
self.is_file failed and the else was to treat it as a dir.
I prefer this explicit exists check, and explicit dir check.
Yeah, potential for race conditions exists.
2020-08-28 17:31:31 -07:00
Ethan Dalool
8e7eeb9c3f Use code from cmd's sdate.py which is better. 2020-08-26 10:07:06 -07:00
Ethan Dalool
e23bd1131f Add support for path-mounted drives. 2020-08-25 18:43:39 -07:00
Ethan Dalool
03256e2a79 Split long parameter list onto lines. 2020-08-25 18:43:12 -07:00
Ethan Dalool
09437547f0 Fix handling of destination replacement when source is drive root. 2020-08-20 13:53:29 -07:00
Ethan Dalool
c0736e8acb Add windrives.py. 2020-08-20 11:39:49 -07:00
Ethan Dalool
a14902d124 Fix can_use_bare incorrectly handling nargs=* arguments. 2020-08-20 11:39:04 -07:00
Ethan Dalool
3c8b76051f Remove unused import safeprint. 2020-07-29 17:55:20 -07:00
Ethan Dalool
9b5c6732d8 Move kbps.py from cmd repository to voussoirkit. 2020-07-20 18:06:45 -07:00
Ethan Dalool
16b1d85dbd Add main function to hms.py. 2020-07-20 18:04:12 -07:00
Ethan Dalool
3a5e160504 Add Path.__lt__ so they are sortable. 2020-07-20 18:03:48 -07:00
Ethan Dalool
5412af7d90 Return touched filenames instead of printing them. 2020-07-16 21:31:16 -07:00
Ethan Dalool
453b2d2fb1 Fix pathclass.glob actually only looking at cwd.
In other news, don't ever trust anything I do.
2020-07-16 21:28:40 -07:00
Ethan Dalool
ff9895a390 Add method glob to get child items with a glob pattern. 2020-07-07 22:25:44 -07:00
Ethan Dalool
df02690361 Add touch.py. 2020-06-22 17:45:54 -07:00
Ethan Dalool
39c09be237 Add configlayers.py. 2020-04-03 17:07:51 -07:00
Ethan Dalool
fb61212395 Add some example calls to bytestring docstring. 2020-03-15 15:21:00 -07:00
Ethan Dalool
417c14a023 Let directories @size property work also.
I was hesitant at first because of course it can be slow to run,
but previously you'd have to have an if so that you know whether you've
got int or none corresponding to files and folders anyway, so I don't
think the caller-side complexity is increased at all, and saves me from
having to write this as a helper function outside the class.
Only for special cases like preparing to handle permissionerror
would you need to use your own function.
2020-03-12 18:18:59 -07:00
Ethan Dalool
081313d7ef Add pathclass.drive for convenient splitdrive. 2020-03-12 17:45:42 -07:00
Ethan Dalool
f090d7d83d Rename args -> argv. 2020-03-12 14:04:23 -07:00
Ethan Dalool
430002f914 Rename variables byte_value -> number, string -> unit_string. 2020-03-12 14:03:56 -07:00
Ethan Dalool
46487d7a31 Add periods to these exception messages. 2020-03-12 14:03:10 -07:00
Ethan Dalool
f897abce11 Simplify this regex, remove unnecessary capture group. 2020-03-12 14:02:25 -07:00