ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
684 commits 5 branches 43 tags 2.8 MiB
Commit graph

190 commits

Author SHA1 Message Date
Con Kolivas 06a76bc4aa Revert "More big endian fixes." 
This reverts commit edbcc797e0.

Broke regression text. Needs more attention.
 2011-03-21 19:01:24 +11:00
Con Kolivas edbcc797e0 More big endian fixes. 2011-03-21 12:41:36 +11:00
Con Kolivas 280254b8cb Forgot to remove the initial pos offset. 2011-03-21 12:34:57 +11:00
Con Kolivas fd9bbfe64a One last check on limit. 2011-03-21 07:33:35 +11:00
Con Kolivas f87528d14d Forgot to include salt length for next head. 2011-03-20 17:01:33 +11:00
Con Kolivas 30b70dc884 Fixes for non little endian machines. 2011-03-20 16:16:38 +11:00
Con Kolivas 0dad7dcc5a Modify the encrypted version to not write down the rzip chunk size, then keep track of where each block header is written, go back and write 8 bytes of salt before it and encrypt the headers in place. 2011-03-20 15:45:44 +11:00
Con Kolivas e7d33e82d0 Make read_1g able to read from the tmp outbuf if we decide to read from fd_out. 2011-03-19 14:46:06 +11:00
Con Kolivas ff5a5ed054 Use separate lrz_encrypt and lrz_decrypt wrappers to lrz_crypt. 2011-03-19 14:04:22 +11:00
Con Kolivas 63fb1bafea Modify the file format further to make all block header information only encode the number of bytes determined in chunk_bytes instead of 8 full bytes all the time. 2011-03-18 23:18:36 +11:00
ckolivas b8d8ecfe33 Fix warnings. 
Patch ~ by Serge Belyshev.
 2011-03-18 10:37:09 +11:00
ckolivas e856d481bf Discard hash_iv entirely, using passphrase hash and centralise key generation. 
Patch by Serge Belyshev.
 2011-03-18 10:22:58 +11:00
Con Kolivas a3e80871c2 Add 8 bytes of random data to the start of every encrypted compressed block and use it along with 8 bytes from the stored salt to make a unique key and IV for encryption. 2011-03-18 00:06:11 +11:00
ckolivas e78069c0d7 Consolidate magic writing to be able to seek to read and encrypt block header data later. 2011-03-17 16:59:27 +11:00
ckolivas 4bc23ee001 Centralise where all encryption/decryption is done, and whether to carry over the IV or not. 2011-03-17 14:10:58 +11:00
ckolivas 660243b8dc Pad out first block if less than CBC_LEN and encrypting or else encryption will fail. 2011-03-17 11:19:34 +11:00
Con Kolivas 76280ffec5 Mlock ivec for it will hold the injection vector. 2011-03-17 00:25:15 +11:00
Con Kolivas a7468ce6e4 Fail if block to encrypt ends up being less than one full block for now, and tidy up code. 2011-03-16 23:13:09 +11:00
Con Kolivas c41cfe9e83 Some decompression code needs more room so simply allocate u_len for s_buf. 2011-03-16 21:48:21 +11:00
Con Kolivas 5fd7c82e1f Implement ciphertext stealing to not have to pad compressed blocks courtesy of Serge Belyshev who saved me hours of debugging thanks! 2011-03-16 21:24:05 +11:00
Con Kolivas 928a0a098a Consolidate padded length throughout. 2011-03-16 10:29:16 +11:00
Con Kolivas bedea4dbec Consolidate cbc padding into a macro. 2011-03-16 10:17:48 +11:00
Con Kolivas ddcc45ebf0 Revert "aes_crypt_cbc always returns zero so ignore its return value." 
This reverts commit 4314970b0d.

Oops, it can return invalid length.
 2011-03-16 10:00:20 +11:00
Con Kolivas 2d2a9bc1e9 We're freeing the wrong buffer by mistake should decompression fail. 2011-03-16 09:50:30 +11:00
Con Kolivas 4314970b0d aes_crypt_cbc always returns zero so ignore its return value. 2011-03-16 08:39:05 +11:00
Con Kolivas 519123966a total_read field is no longer used. 2011-03-16 08:32:16 +11:00
Con Kolivas 5da0633893 aes_crypt_cbc will allow you to work on the same buffer, so don't bother allocating a separate buffer. 
Allocate slightly more on the buffer that might be used for encryption rather than reallocing.
 2011-03-16 00:46:39 +11:00
Con Kolivas c5938c6a8b Implement the actual aes cbc encryption and decryption. 2011-03-15 23:52:39 +11:00
ckolivas 65a681a254 Relative offset is not required in tmp inbuf, it can be safely zeroed. 2011-03-15 11:21:26 +11:00
Con Kolivas 3a8c0b6689 Remove seeks that aren't required and don't work on tmp input buffers. 
Clean up open_stream_in.
 2011-03-14 21:51:27 +11:00
Con Kolivas c832e80085 More infrastructure to read from temporary buffers on stdin decompression. 2011-03-14 21:19:57 +11:00
Con Kolivas 36e09f206e Begin massive read changes to support using temporary file for STDIN. 2011-03-14 20:22:45 +11:00
Con Kolivas 684959efed Add fields for temporary input buffer and clamp memory accordingly in preparation. 2011-03-14 14:47:26 +11:00
Con Kolivas 5f7a03932b Calculate the total expected size progressively and show it when it's not known. 2011-03-14 13:32:36 +11:00
Con Kolivas 9e772d3140 Make ALL decompression use temporary in-ram buffer whenever possible. 2011-03-14 12:48:40 +11:00
Con Kolivas 27d7c2a031 Tidy. 2011-03-14 12:23:12 +11:00
Con Kolivas 0fe3213a47 Write to physical files if we cannot fit the decompression in ram. 2011-03-14 12:15:54 +11:00
Con Kolivas 37009e2ac5 Make sure to read on an fd if that's what we're supposed to be doing. 2011-03-14 11:25:04 +11:00
Con Kolivas b644240152 write_1g always uses control->fd_out so don't pass fd to it. 2011-03-14 11:15:35 +11:00
Con Kolivas 4a6fa7602f Begin decompressing to stdout via temporary buffer by creating a read/write wrapper. 2011-03-14 10:07:51 +11:00
Con Kolivas f2ddd9022c Ignore the eof flag if the file size is known. 2011-03-13 21:31:03 +11:00
Con Kolivas 8756fe91e2 Enable decompression when file has been chunked stdout and display progress only when expected size is known. 2011-03-13 17:52:23 +11:00
Con Kolivas 6d0ac95170 Remove extra locking that does nothing. 2011-03-13 12:12:37 +11:00
Con Kolivas 6ac74aa9f0 Create a flag to know when the temporary output buffer is in use, in preparation for when we use it on decompression. 2011-03-13 08:34:06 +11:00
Con Kolivas 2f87f62696 Make the tmp out buf slightly larger to account for incompressible data, and check for buffer overflows. 2011-03-13 08:16:46 +11:00
Con Kolivas 11ea12d3ce Forgot to test for TEST_ONLY. 2011-03-12 23:07:52 +11:00
Con Kolivas d067a6ea9e Implement the real workings of writing to a temporary buffer before flushing to stdout. 2011-03-12 22:46:57 +11:00
Con Kolivas 7fbec0a783 Prepare to write compressed output by flushing stdout after each chunk is compressed. 2011-03-12 19:56:08 +11:00
Con Kolivas c75a50f723 Being modifying write_1g function to be able to write to a temporary buffer instead of straight to fd_out. 
Split out make_magic to be able to write magic wherever we want later.
 2011-03-12 14:13:28 +11:00
Con Kolivas 9444441d51 Modify maximum ram usable when stdout is being used in preparation for temporary in-ram file during stdout and fix summary shown and 32 bit limits. 2011-03-12 12:19:02 +11:00
Con Kolivas fe68b9a3f7 Institute writing and reading of 0.6 file format for compress/decompress. 2011-03-12 11:17:11 +11:00
Con Kolivas 3a00735c24 Fix locking. 
Patch by <mike@zentific.com>
 2011-03-11 12:29:27 +11:00
Con Kolivas a8dcecd721 fix-undefined-mutex-behavior-rename-stream-variable. 
Patch by <mike@zentific.com>
 2011-03-11 08:35:15 +11:00
Con Kolivas 643054ae22 Fix threading errors. 
Patch by <mike@zentific.com>
 2011-03-11 08:33:35 +11:00
Con Kolivas fb2de8cb35 Remove unused offset variable. 2011-03-09 13:33:53 +11:00
Con Kolivas 1b965167ff Remove unused offset variable. 2011-03-09 13:30:20 +11:00
Con Kolivas 11052f56f3 Ignore the lzo_1 return value entirely. 2011-03-09 13:25:33 +11:00
Con Kolivas 2db75fe408 Get rid of trailing whitespace 2011-03-09 08:50:46 +11:00
Con Kolivas 1a7c409e10 header-mangling-part-X-move-all-headers-defines-into 2011-03-09 08:37:26 +11:00
Con Kolivas 99c3ea2ab9 header-mangling-part-3-remove-ugly-hacks-for-liblrzi 2011-03-09 08:36:07 +11:00
Con Kolivas 1511c27aad header-mangling-part-2-move-all-function-prototypes 2011-03-09 08:34:44 +11:00
Con Kolivas f6f0a25ef6 rebase-of-function-split-and-control-additions-to-fu 2011-03-09 08:32:14 +11:00
Con Kolivas 38eca38743 Unify maxram allocation and limit threads when there isn't enough ram. 2011-03-07 13:23:14 +11:00
Con Kolivas 3433438a8e Structs in stream.c can be static. 2011-02-26 20:11:43 +11:00
ckolivas f9f880908c Remove the slightly fragile exponential growth buffer size. 
It was only speeding up compression a small amount, yet adversely affected compression and would segfault due to the size not being consistent on successive passes.
 2011-02-25 10:10:22 +11:00
Con Kolivas dcf62d11a0 Make sure not to make the bufsize larger than the limit. 
Drop the page rounding since it is of no demonstrable benefit but adds complexity.
 2011-02-24 12:20:06 +11:00
Con Kolivas 22ae326d01 Make it always clear that a failure to allocate a buffer has occurred on compression. 2011-02-24 11:52:30 +11:00
Con Kolivas 402dbbed65 Make sure we don't start shrinking the buffer size. 2011-02-23 15:34:43 +11:00
Con Kolivas 94673d3fe3 Change the LZO testing option to be a bool on/off instead of taking a confusing parameter. 
Make the lzo testing message simpler and only appear when max verbose mode is enabled.
 2011-02-23 01:15:18 +11:00
Con Kolivas 011344753a With lzma and zpaq, the compression overhead per thread is significant. 
As we can work out what that compression overhead is, we can factor that into testing how much ram we can allocate.
There is no advantage to running multiple threads when there is no compression back end so drop to 1 only.
Limit ram for compression back end to 1/3 ram regardless for when OSs lie due to heavy overcommit.
 2011-02-22 15:19:31 +11:00
Con Kolivas bcb857d934 Don't add extra threads for single-threaded decompression case. 2011-02-22 00:58:55 +11:00
Con Kolivas bb33f7571c Multi-threading speed ups. 
Add one more thread on compression and decompression to account for the staggered nature of thread recruitment.
Make the initial buffer slightly smaller and make it progressively larger, thus recruiting threads sooner and more evenly.
This also speeds up decompression for the same reason.
Check the amount of memory being used by each thread on decompression to ensure we don't try to recruit too much ram.
 2011-02-22 00:49:50 +11:00
Con Kolivas 88e3df6af1 Print perror before unlinking files. 
Join common parts of fatal errors.
Update copyright notices.
Small improvement to visual output.
 2011-02-21 16:11:59 +11:00
Con Kolivas a7b4708bd2 Use a different failure mode for when perror is unlikely to be set. 
Add 2 unlikely wrappers.
 2011-02-21 14:51:20 +11:00
Con Kolivas 74df2b5973 Minor updates to man pages, lrzip.conf example file. 
Update main help screen to include environment settings.
Update to respect $TMP environment variable for TMP files.
Updated control structure to include tmpdir pointer.
Update lrzip.conf parser to respect -U -M options.
Update lrzip.conf example to include new parameters.
Reorder main Switch loop in main.c for readability.
Have MAXRAM and control.window be exclusive. MAXRAM wins.
Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
Have UNLIMITED and MAXRAM be exclusive. UNLIMITED wins.
Corrects heuristic computation in rzip.c which would override
MAXRAM or UNLIMITED if control.window set
Show heuristically computed control.window when computed.
Remove display compression level from control.window verbose output.
Update print_verbose format for Testing for incompressible data in stream.c
to omit extra \n.

Changes by Peter Hyman <pete@peterhyman.com>
 2011-02-21 12:03:08 +11:00
Con Kolivas 57e25da244 Update copyright yeah in updated files. 2011-02-20 23:04:44 +11:00
Con Kolivas 626e0be281 Convert semaphore primitives to pthread_mutexes making them more portable, thus allowing multithreading to work on OSX. 2011-02-17 00:24:28 +11:00
Con Kolivas 05c5326df3 Revert "OSX doesn't support unnamed semaphores so to make it work, fake the threading by just creating the threads and waiting for them to finish." 
This reverts commit b81542cea4.

Revert the change bypassing semaphores in OSX in preparation for changing the semaphores to mutexes.
 2011-02-16 17:40:50 +11:00
Con Kolivas b81542cea4 OSX doesn't support unnamed semaphores so to make it work, fake the threading by just creating the threads and waiting for them to finish. 
This is done by making the semaphore wrappers null functions on osx and closing the thread in the creation wrapper.
Move the wrappers to rzip.h to make this change clean.
 2011-02-11 12:22:09 +11:00
Con Kolivas f2d33c00f8 Cast the mallocs to their variable type. 
Check that read and write actually return greater than zero.
 2011-02-11 11:46:58 +11:00
Con Kolivas 3879807865 Try limiting stream_read in unzip_literal and just returning how much was read. 2011-02-10 16:57:22 +11:00
Con Kolivas 9a3bfe33d1 Revert "Make sure to read the full length asked of unzip_literal." 
This reverts commit 499ae18cef.

Wrong fix, revert it.
 2011-02-10 16:46:35 +11:00
Con Kolivas 499ae18cef Make sure to read the full length asked of unzip_literal. 2011-02-10 15:30:31 +11:00
Con Kolivas 2a0553bc54 Revert "Decompress more than one stream at a time if there are threads free and the end of one stream is reached." 
This reverts commit 8ee9ef64f5.

This change is unreliable. Hence revert it and all dependent patches.
 2011-02-09 12:39:15 +11:00
Con Kolivas 8239635038 Revert "Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data." 
This reverts commit 0b0f6db606.
 2011-02-09 12:39:02 +11:00
Con Kolivas 44399d88ac Revert "Check we dont go over control.threads when trying to fill the other stream." 
This reverts commit 0bded0a7d9.
 2011-02-09 12:38:32 +11:00
Con Kolivas 0bded0a7d9 Check we dont go over control.threads when trying to fill the other stream. 
Update comments to reflect when we can go over control.threads.
 2011-02-09 11:07:38 +11:00
Con Kolivas 0b0f6db606 Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data. 2011-02-09 10:45:03 +11:00
Con Kolivas 8ee9ef64f5 Decompress more than one stream at a time if there are threads free and the end of one stream is reached. 
Still limit total threads running to control.threads.
This affords a small speedup on decompression.
 2011-02-08 11:58:01 +11:00
Con Kolivas aa00c29fba The free semaphore is now only updated from the main process on decompression so there are no synchronisation concerns. 
Remove the free semaphore and the fragile use of sem_trywait and replace it with a simple busy flag for threads on decompression.
 2011-02-08 08:55:36 +11:00
Con Kolivas 9c2b86fec6 We are flushing the wrong file on decompression. Make sure we flush the file out. 2011-02-08 08:27:22 +11:00
Con Kolivas 6b73eb1394 We do not need to wait for the main process to be ready to receive the data from the threads before shutting them down. 
Only wait for the "ready" semaphore if we've failed to decompress in parallel.
This affords a small speedup on decompression.
 2011-02-06 11:58:47 +11:00
Con Kolivas 8d110e3366 Remove the check for interleaved streams as it wasn't achieving anything. 
Make sure the thread has really exited before moving on, and set the free semaphore from outside the thread once it has joined.
 2011-02-06 11:21:36 +11:00
Con Kolivas 4efa81b1d0 Relax the memory allocation testing when no back end compression will be used. 2011-02-06 09:00:43 +11:00
Con Kolivas f7a27cbde5 Flush writes to disk before allocating more ram, avoids one more fsync at the end of compression. 
Add fsyncing to decompression as well, improving reliability of allocating more ram.
 2010-12-18 10:02:19 +11:00
Con Kolivas e1bb9ebda5 There's a bug where archives with multiple stream 0 entries per chunk can be corrupted on decompression with threading enabled. 
Until the bug is tracked down, disable multithreaded decompression when an archive with interleaved stream 0/1 entries is detected.
 2010-12-18 00:50:49 +11:00
Con Kolivas 2cabb335cb Update copyright notices courtesy of Jari Aalto. 2010-12-16 09:45:21 +11:00
Con Kolivas becb08129c gotti spotted that bzip2 and gzip would fail if they encountered an incompressible block instead of returning it uncompressed, and provided a patch for it. 
Tweaked the patch slightly for style.
 2010-12-15 22:54:15 +11:00
Con Kolivas 427df4cd4a Fix stdin failing due to getting sizes all wrong. 
Fix stdin compression values not being shown at end.
Fix inappropriate failure when lzma doesn't compress block.
 2010-12-12 17:40:58 +11:00
Con Kolivas 34f76fa73c Don't wait on a semaphore the 2nd time attempting to compress/decompress. 
Wait for all ucompthreads to be free.
Destroy the semaphores used by ucompthreads when a stream_in is closed.
 2010-12-12 10:39:52 +11:00