ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
946 commits 5 branches 43 tags 2.8 MiB
Commit graph

171 commits

Author SHA1 Message Date
Con Kolivas 754088feb9 Must give the memcpy the sb low offset to work properly. 2012-03-07 23:14:04 +11:00
Con Kolivas 086265c83f Update copyright notices and style to 1TBS for return codes. 2012-03-07 22:04:24 +11:00
ckolivas 1abacd82af Use the single_mcpy function even if we need the sliding mmap buffer if we won't need to be pageing buf_low. 2012-03-07 16:53:18 +11:00
ckolivas 43f407aa04 Remove global variables sliding_buffer get_sb and do_mcpy and put them in rzip_control. 2012-03-07 14:48:21 +11:00
ckolivas a62e78186e Merge branch 'liblrzip' of github.com:ckolivas/lrzip into liblrzip 
Conflicts:
	rzip.c
 2012-03-07 11:41:35 +11:00
Con Kolivas f496e0705d get_sb only allows accessing one byte at a time, yet we don't need that functionality when sliding mmap is not in use. 
User different versions of the function and the larger memcpys depending on whether sliding mmap is in use or not.
This affords a substantial speedup in the rzip phase of files not requiring sliding mmap.
A small optimisation of moving the check for remapping the low buffer also speeds up the sliding mmap version slightly.
 2011-09-18 22:29:09 +10:00
discomfitor 3316dd5505 remove trailing whitespace 2011-08-17 20:59:41 +10:00
discomfitor 223a1107ea add info callback which provides percentage completion of operation 2011-08-17 20:59:41 +10:00
discomfitor 63141a1035 add some gross bucketing to implement a queue for freeing stream_info structs after an operation has completed (in library mode ONLY; no changes will occur during regular execution) to fix some leaks 2011-08-17 20:59:41 +10:00
discomfitor 949c6d7a97 another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above 2011-08-17 20:59:41 +10:00
discomfitor 6dd20d735b giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error 2011-08-17 20:59:40 +10:00
discomfitor 133b201867 move more code out of main.c, allocate outfile suffix 2011-08-17 20:59:40 +10:00
discomfitor cbff8faef7 move liblrzip.h contents to lrzip_private.h global variables----- 2011-08-17 20:59:40 +10:00
Con Kolivas 3e10b27422 More divide by zero output fixes. 2011-05-08 00:45:56 +10:00
Con Kolivas 0d86b8ba0e More percentage display fixes for divide by zero errors. 2011-05-08 00:38:33 +10:00
Con Kolivas a1359276f4 Fix nan when no hashes found in verbose message. 2011-05-08 00:34:09 +10:00
Con Kolivas 1ea75154d6 Silence compiler warnings. 2011-05-08 00:05:46 +10:00
Con Kolivas 55dc9b0095 Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. 
Put special case management of zero sized files.
Modify the percentage calculation to be able to cope with small numbers and zeroes.
 2011-05-07 18:37:25 +10:00
Con Kolivas 6ceaeb5f93 Progress update percent was being zeroed in rzip.c thus displaying on every loop. Move it higher, fixing the amount of output to the terminal. 2011-05-07 10:19:12 +10:00
ckolivas 127a101b14 Disable md5 checking on apple for now till someone fixes it. 2011-04-13 14:50:26 +10:00
Con Kolivas 70e7866271 Include arpa/inet.h for those OSs that have endian details elsewhere. 2011-03-22 21:47:24 +11:00
ckolivas f56e48c4f2 Modify maxram to mean the most we'll allocate in one chunk and add usable_ram for the most we'll allocate to one task. 
Limit all maxram on 32bits to 1GB since most don't like mallocing more than this.
Update testing to allow larger total amount based on usable ram while sticking to maxram limit per alloc.
 2011-03-22 12:10:21 +11:00
Con Kolivas 2736118f82 Check for different endian.h includes and variants of le32toh. 2011-03-22 07:57:32 +11:00
Con Kolivas a30efcaa55 Portable generic byteswap for BE. 2011-03-22 00:13:29 +11:00
Con Kolivas c533b031bc Fix trivial compiler warnings. 2011-03-21 23:36:39 +11:00
Con Kolivas 1647b4036d Unused variable. 2011-03-21 21:29:01 +11:00
Con Kolivas 43eb2e4594 Revert "yet yet yet more BE fixes." 
This reverts commit 44b93068a3.

Value was not stored but ysed for a memcpy.
 2011-03-21 21:27:25 +11:00
Con Kolivas 8c8f5bdd48 lrz_crypt tweaks courtesy of Serge Belyshev. 2011-03-21 20:54:53 +11:00
Con Kolivas 44b93068a3 yet yet yet more BE fixes. 2011-03-21 18:58:38 +11:00
Con Kolivas 2982cb7121 Yet yet more BE fixes. 2011-03-21 16:12:51 +11:00
Con Kolivas ff5a5ed054 Use separate lrz_encrypt and lrz_decrypt wrappers to lrz_crypt. 2011-03-19 14:04:22 +11:00
Con Kolivas 0b1db75a44 Cleanups. 2011-03-19 13:53:43 +11:00
ckolivas 5005c2dff5 Get rid of extra data in new header and stick to old header size, compacting extra info into existing space. 2011-03-18 17:32:47 +11:00
ckolivas b8d8ecfe33 Fix warnings. 
Patch ~ by Serge Belyshev.
 2011-03-18 10:37:09 +11:00
ckolivas e856d481bf Discard hash_iv entirely, using passphrase hash and centralise key generation. 
Patch by Serge Belyshev.
 2011-03-18 10:22:58 +11:00
Con Kolivas a3e80871c2 Add 8 bytes of random data to the start of every encrypted compressed block and use it along with 8 bytes from the stored salt to make a unique key and IV for encryption. 2011-03-18 00:06:11 +11:00
ckolivas 4bc23ee001 Centralise where all encryption/decryption is done, and whether to carry over the IV or not. 2011-03-17 14:10:58 +11:00
Con Kolivas f61632670e Encrypt the md5 value as well to make the file unidentifiable. 2011-03-16 00:29:10 +11:00
ckolivas cb1f630fe0 Tidy. 2011-03-15 10:41:17 +11:00
ckolivas 091909b552 Refuse to compress a file smaller than a minimum size. 2011-03-15 10:40:14 +11:00
Con Kolivas 9e772d3140 Make ALL decompression use temporary in-ram buffer whenever possible. 2011-03-14 12:48:40 +11:00
Con Kolivas b644240152 write_1g always uses control->fd_out so don't pass fd to it. 2011-03-14 11:15:35 +11:00
Con Kolivas 19ef7298c8 Cast fstat values to i64 to prevent 32 bit overflows. 2011-03-14 10:21:10 +11:00
Con Kolivas f2ddd9022c Ignore the eof flag if the file size is known. 2011-03-13 21:31:03 +11:00
Con Kolivas 6ac74aa9f0 Create a flag to know when the temporary output buffer is in use, in preparation for when we use it on decompression. 2011-03-13 08:34:06 +11:00
Con Kolivas d067a6ea9e Implement the real workings of writing to a temporary buffer before flushing to stdout. 2011-03-12 22:46:57 +11:00
Con Kolivas 7fbec0a783 Prepare to write compressed output by flushing stdout after each chunk is compressed. 2011-03-12 19:56:08 +11:00
Con Kolivas d87df4f44e Store the file size as zero when the magic header will be written before the size is known. 2011-03-11 23:45:22 +11:00
Con Kolivas 81ab444971 Remove unused flags by fake mremap. 2011-03-09 13:13:33 +11:00
Con Kolivas dc92d477d1 Remove unused best_h. 2011-03-09 13:10:00 +11:00
Con Kolivas 1a7c409e10 header-mangling-part-X-move-all-headers-defines-into 2011-03-09 08:37:26 +11:00
Con Kolivas 99c3ea2ab9 header-mangling-part-3-remove-ugly-hacks-for-liblrzi 2011-03-09 08:36:07 +11:00
Con Kolivas 1511c27aad header-mangling-part-2-move-all-function-prototypes 2011-03-09 08:34:44 +11:00
Con Kolivas f6f0a25ef6 rebase-of-function-split-and-control-additions-to-fu 2011-03-09 08:32:14 +11:00
ckolivas 59e5bc0492 Remove all open coded strerrors when they're going to be shown by fatal(). 2011-03-08 12:05:03 +11:00
ckolivas 82a3bd3028 Clean up unused variables. 2011-03-08 11:43:02 +11:00
Con Kolivas 44ed5727b3 Clean up and fix times displayed. 
Merge fixes.
 2011-03-07 21:05:20 +11:00
Con Kolivas 9fc551c159 Merge branch 'temp' 
Conflicts:
	main.c
 2011-03-07 17:25:10 +11:00
Con Kolivas 32e182c95a Check for free space in the right place for compression and give the right message. 2011-03-07 17:00:28 +11:00
Con Kolivas 38eca38743 Unify maxram allocation and limit threads when there isn't enough ram. 2011-03-07 13:23:14 +11:00
Con Kolivas a4ac1b8221 Display the progress percentage and ETA when it's known on STDIN compress. 2011-03-07 13:03:43 +11:00
ckolivas 63566a2992 Control.window was broken comparing to zero max_chunk. 2011-02-28 14:10:46 +11:00
ckolivas ab656aa98a Wrong check in rzip.c will leave a small chunk with unlimited mode. 2011-02-28 14:05:33 +11:00
Con Kolivas 8bdd5688c8 Check for free space before compression/decompression and abort if -f option is not enabled. 2011-02-26 23:10:28 +11:00
Con Kolivas db4f9f5fea Only retry mmapping if it's due to a memory allocation error. 2011-02-26 20:00:26 +11:00
Con Kolivas 7acd389382 Trivial cast to uchar * 2011-02-26 19:45:00 +11:00
Con Kolivas 8bf60c3114 Don't round to page size if the entire buffer will fit, otherwise this generates one extra pass of a tiny chunk. 2011-02-24 11:38:31 +11:00
Con Kolivas 9c8b1ee795 Deprecate the -M option as it is no longer of any significant utility compared to the -U option. 2011-02-22 20:38:39 +11:00
Con Kolivas bb33f7571c Multi-threading speed ups. 
Add one more thread on compression and decompression to account for the staggered nature of thread recruitment.
Make the initial buffer slightly smaller and make it progressively larger, thus recruiting threads sooner and more evenly.
This also speeds up decompression for the same reason.
Check the amount of memory being used by each thread on decompression to ensure we don't try to recruit too much ram.
 2011-02-22 00:49:50 +11:00
Con Kolivas a7b4708bd2 Use a different failure mode for when perror is unlikely to be set. 
Add 2 unlikely wrappers.
 2011-02-21 14:51:20 +11:00
Con Kolivas 57e25da244 Update copyright yeah in updated files. 2011-02-20 23:04:44 +11:00
Con Kolivas 8a27dc5057 Changes to make md5 be used for integrity testing. 
Add the md5 value to the end of each archive.
This can then be used for integrity testing instead of crc32.
Keep crc in new archives to maintain compatibility with version 0.5 files.
Use md5 integrity testing on decompression when available in preference, and disable calculation of crc32.
Display the choice of integrity testing in verbose output and when -i is used.
Display the md5 and crc values when max verbosity, file info, or display hash is enabled.
Store a new flag in the magic header to show that the md5 value is stored at the end of the file.
Update the magic header information document.
 2011-02-20 18:01:19 +11:00
Con Kolivas 44a279579e Add option to display hash information without enabling verbose mode. 2011-02-20 12:20:05 +11:00
Con Kolivas 4036125f94 Make the buffer passed to hash checking page sized which gives a minor speedup. 2011-02-20 12:16:45 +11:00
Con Kolivas 7287ab8a66 Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time. 2011-02-19 10:34:45 +11:00
Con Kolivas d1a70bbb90 Remove the sb.low_top idea by trimming the size of sb.size_low when it gets to the end of the file instead. 2011-02-18 21:14:16 +11:00
Con Kolivas fb2a12744a Implement md5 checking on decompression. 
Implement hash check flag to determine whether to show md5sum on compression/decompression or not.
 2011-02-18 15:16:13 +11:00
Con Kolivas c2417877bc Being implementing md5 hash checking by importing the md5 code from coreutils. 
Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
 2011-02-18 09:09:40 +11:00
Con Kolivas 2cabb335cb Update copyright notices courtesy of Jari Aalto. 2010-12-16 09:45:21 +11:00
Con Kolivas 427df4cd4a Fix stdin failing due to getting sizes all wrong. 
Fix stdin compression values not being shown at end.
Fix inappropriate failure when lzma doesn't compress block.
 2010-12-12 17:40:58 +11:00
Con Kolivas 4331ca4058 Move message to after size is finalised. 2010-12-11 12:25:43 +11:00
Con Kolivas 6c33071118 Rationalise the testing now that the default lzma settings use a lot less ram by default. 2010-12-11 01:07:43 +11:00
Con Kolivas 50437a8447 Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. 
This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup.
Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring.
Fix a few dodgy callocs that may have been overflowing!
The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
 2010-12-10 23:51:59 +11:00
Con Kolivas 8dd9b00496 Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." 
This reverts commit 688aa55c79.
 2010-12-08 21:25:00 +11:00
Con Kolivas 5be0f6c5c9 Revert "Make sure chunk to compress is of a minimum size." 
This reverts commit 8dd1b6ea58.
 2010-12-08 21:23:41 +11:00
Con Kolivas dc42add0cf Since 32 bit seems to fall over so easily, get ruthless with limits on it, while still maintaining a large window with sliding mmap. 2010-12-04 08:39:52 +11:00
Con Kolivas 8dd1b6ea58 Make sure chunk to compress is of a minimum size. 2010-12-03 20:02:39 +11:00
Con Kolivas 688aa55c79 Make threads spawn at regular intervals along chunk size thus speeding up compression. 
One more fix for unnecessary argument passing.
 2010-12-03 19:33:56 +11:00
Con Kolivas 2a4892afa1 Streams passed are constant to some functions, don't pass stream argument to them. 2010-12-03 19:30:56 +11:00
Con Kolivas f2b5852872 Fix divide by zero error when a file is smaller than the page size and compressed with -U. 2010-11-30 11:22:02 +11:00
Con Kolivas f04b8aa15b Tweak defaults some more. 
Set maximum size of compression window to chunk size in the case of -M mode and mmap size normally.
Set maximum window to 1.5x ram in -M mode.
 2010-11-20 18:29:28 +11:00
Con Kolivas 6929492874 Fix the sliding mmap buffer not working on 2nd pass onwards which was causing a slowdon of death. 2010-11-20 09:44:54 +11:00
Con Kolivas 13ca377279 More tidying up and ensure control window is obeyed. 2010-11-20 09:22:47 +11:00
Con Kolivas 75e675e6dd Bump version number to 0.542. 
Choose sane defaults for memory usage since linux ludicriously overcommits.
Use sliding mmap for any compression windows greater than 2/3 ram.
Consolidate and simplify testing of allocatable ram.
Minor tweaks to output.
Round up the size of the high buffer in sliding mmap to one page.
Squeeze a little more out of 32 bit compression windows.
 2010-11-20 01:23:08 +11:00
Con Kolivas 25e053ed49 Wrong number of passes. 2010-11-18 23:44:22 +11:00
Con Kolivas 591d791791 Bump version to 0.541. 
Limit LZMA window to 300MB on 32 bit as per reports of failure when larger.
Minor documentation and display clean ups.
 2010-11-18 23:33:43 +11:00
Con Kolivas 6e4fdc97f8 Implement -p option to specify number of processors to determine thread count. 
Remove -P option as failing to set permissions only issues a warning now, removing any requirement for -P.
Change default compression level back to 7 as 9 was not giving significantly better compression but was slowing things down.
 2010-11-13 17:36:21 +11:00
Con Kolivas 02de002c58 Reworked the multithreading massively. 
Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating.
Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible.
This causes a big speed up on the compression side on SMP machines.
Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows.
Alter the progress output to match some of the changes in verbose modes.
 2010-11-13 01:26:09 +11:00
Con Kolivas ead0e54182 Drop the upper limit on lzma compression window on 64 bit. It is not necessary. 
zpaq will fail with windows bigger than 600MB on 32 bit due to failing testing the 3* malloc test, so limit it to 600MB as well as lzma on 32 bit.
 2010-11-07 01:57:23 +11:00
Con Kolivas b8528abee9 Consciously check page size, even though no one's going to build this on a machine with a different page size. 
Clean up builds, removing ifdefs from main code.
Make output choices consistent.
 2010-11-06 18:17:33 +11:00