ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
139 commits 5 branches 43 tags 2.8 MiB
Commit graph

139 commits

This branch
This branch
All branches
Author SHA1 Message Date
Con Kolivas f2d33c00f8 Cast the mallocs to their variable type. 
Check that read and write actually return greater than zero.
 2011-02-11 11:46:58 +11:00
Con Kolivas 3879807865 Try limiting stream_read in unzip_literal and just returning how much was read. 2011-02-10 16:57:22 +11:00
Con Kolivas 9a3bfe33d1 Revert "Make sure to read the full length asked of unzip_literal." 
This reverts commit 499ae18cef.

Wrong fix, revert it.
 2011-02-10 16:46:35 +11:00
Con Kolivas 499ae18cef Make sure to read the full length asked of unzip_literal. 2011-02-10 15:30:31 +11:00
Con Kolivas 0a32b5f72d Convert mmaps to malloc in runzip as they may fail if not a multiple of page size! 2011-02-10 13:53:42 +11:00
Con Kolivas bece82a593 Trivial documentation fixes courtesy of Laszlo Ersek. 2011-02-10 13:14:36 +11:00
Con Kolivas 2a0553bc54 Revert "Decompress more than one stream at a time if there are threads free and the end of one stream is reached." 
This reverts commit 8ee9ef64f5.

This change is unreliable. Hence revert it and all dependent patches.
 2011-02-09 12:39:15 +11:00
Con Kolivas 8239635038 Revert "Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data." 
This reverts commit 0b0f6db606.
 2011-02-09 12:39:02 +11:00
Con Kolivas 44399d88ac Revert "Check we dont go over control.threads when trying to fill the other stream." 
This reverts commit 0bded0a7d9.
 2011-02-09 12:38:32 +11:00
Con Kolivas 0bded0a7d9 Check we dont go over control.threads when trying to fill the other stream. 
Update comments to reflect when we can go over control.threads.
 2011-02-09 11:07:38 +11:00
Con Kolivas 0b0f6db606 Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data. 2011-02-09 10:45:03 +11:00
Con Kolivas 8ee9ef64f5 Decompress more than one stream at a time if there are threads free and the end of one stream is reached. 
Still limit total threads running to control.threads.
This affords a small speedup on decompression.
 2011-02-08 11:58:01 +11:00
Con Kolivas aa00c29fba The free semaphore is now only updated from the main process on decompression so there are no synchronisation concerns. 
Remove the free semaphore and the fragile use of sem_trywait and replace it with a simple busy flag for threads on decompression.
 2011-02-08 08:55:36 +11:00
Con Kolivas 9c2b86fec6 We are flushing the wrong file on decompression. Make sure we flush the file out. 2011-02-08 08:27:22 +11:00
Con Kolivas 6b73eb1394 We do not need to wait for the main process to be ready to receive the data from the threads before shutting them down. 
Only wait for the "ready" semaphore if we've failed to decompress in parallel.
This affords a small speedup on decompression.
 2011-02-06 11:58:47 +11:00
Con Kolivas 8d110e3366 Remove the check for interleaved streams as it wasn't achieving anything. 
Make sure the thread has really exited before moving on, and set the free semaphore from outside the thread once it has joined.
 2011-02-06 11:21:36 +11:00
Con Kolivas 4efa81b1d0 Relax the memory allocation testing when no back end compression will be used. 2011-02-06 09:00:43 +11:00
Con Kolivas 191a55d03c Apply correct fix for freeBSD not implementing mremap thanks to Brian Carlson. 2011-01-02 14:23:29 +11:00
Con Kolivas f7a27cbde5 Flush writes to disk before allocating more ram, avoids one more fsync at the end of compression. 
Add fsyncing to decompression as well, improving reliability of allocating more ram.
 2010-12-18 10:02:19 +11:00
Con Kolivas e1bb9ebda5 There's a bug where archives with multiple stream 0 entries per chunk can be corrupted on decompression with threading enabled. 
Until the bug is tracked down, disable multithreaded decompression when an archive with interleaved stream 0/1 entries is detected.
 2010-12-18 00:50:49 +11:00
Con Kolivas 2cabb335cb Update copyright notices courtesy of Jari Aalto. 2010-12-16 09:45:21 +11:00
Con Kolivas e83612fc0a Freebsd doesn't do mremap. Use the fake one. 2010-12-16 09:42:34 +11:00
Con Kolivas becb08129c gotti spotted that bzip2 and gzip would fail if they encountered an incompressible block instead of returning it uncompressed, and provided a patch for it. 
Tweaked the patch slightly for style.
 2010-12-15 22:54:15 +11:00
Con Kolivas 654ae02ba6 Quick bump to version 0.551. 
Update docs.
 2010-12-12 17:48:07 +11:00
Con Kolivas 427df4cd4a Fix stdin failing due to getting sizes all wrong. 
Fix stdin compression values not being shown at end.
Fix inappropriate failure when lzma doesn't compress block.
 2010-12-12 17:40:58 +11:00
Con Kolivas 981859fdba Version to 0.550. 2010-12-12 10:51:29 +11:00
Con Kolivas d1db088986 Bump version to 0.550. 
Typo.
 2010-12-12 10:49:54 +11:00
Con Kolivas 28079083c1 Update docs. 2010-12-12 10:46:22 +11:00
Con Kolivas 34f76fa73c Don't wait on a semaphore the 2nd time attempting to compress/decompress. 
Wait for all ucompthreads to be free.
Destroy the semaphores used by ucompthreads when a stream_in is closed.
 2010-12-12 10:39:52 +11:00
Con Kolivas 351cfced8a Return the correct buffer on decompression failure in case decompression will be attempted again. 2010-12-11 14:25:07 +11:00
Con Kolivas aeeeedcab2 Cope with multithreaded memory failures better. 
Instead of failing completely, detect when a failure has occurred, and serialise work for that thread to decrease the memory required to complete compression / decompression.
Do that by waiting for the thread before it to complete before trying to work on that block again.
Check internally when lzma compress has failed and try a lower compression level before falling back to bzip2 compression.
 2010-12-11 13:19:34 +11:00
Con Kolivas 4331ca4058 Move message to after size is finalised. 2010-12-11 12:25:43 +11:00
Con Kolivas 6c33071118 Rationalise the testing now that the default lzma settings use a lot less ram by default. 2010-12-11 01:07:43 +11:00
Con Kolivas a6ab7c875b Limit the number of threads decompressing stream 0 to just 1 since it's always followed by stream 1 chunks, and it may lead to failure to decompress due to running out of memory by running too many threads. 2010-12-11 00:04:30 +11:00
Con Kolivas 50437a8447 Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. 
This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup.
Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring.
Fix a few dodgy callocs that may have been overflowing!
The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
 2010-12-10 23:51:59 +11:00
Con Kolivas 8dd9b00496 Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." 
This reverts commit 688aa55c79.
 2010-12-08 21:25:00 +11:00
Con Kolivas 5be0f6c5c9 Revert "Make sure chunk to compress is of a minimum size." 
This reverts commit 8dd1b6ea58.
 2010-12-08 21:23:41 +11:00
Con Kolivas e0265b33e1 Default compression level and window size on lzma is set to 7 which is the highest it goes. 
Scale the 9 levels down to the 7 that lzma has.
This makes the default lzma compression level 5 which is what lzma normally has, and uses a lot less ram and is significantly faster than previously, but at the cost of giving slightly less compression.
 2010-12-08 20:53:26 +11:00
Con Kolivas 13e443e23c Update docs. 2010-12-04 21:49:54 +11:00
Con Kolivas c3dfcfcec1 Update version number to 0.544. 
Change suggested maximum compression in README to disable threading with -p 1.
Use bzip2 as a fallback compression when lzma fails due to internal memory errors as may happen on 32 bits.
 2010-12-04 21:36:51 +11:00
Con Kolivas 23e89b06af Minor output consistency tidying. 2010-12-04 09:04:38 +11:00
Con Kolivas dc42add0cf Since 32 bit seems to fall over so easily, get ruthless with limits on it, while still maintaining a large window with sliding mmap. 2010-12-04 08:39:52 +11:00
Con Kolivas 8dd1b6ea58 Make sure chunk to compress is of a minimum size. 2010-12-03 20:02:39 +11:00
Con Kolivas 22c87ed3ee Fix missing define. 2010-12-03 19:38:21 +11:00
Con Kolivas 2da407a178 Change decompression threading to have a group of threads for each stream (2 in total), thus making mulithreaded decompression more robust. 2010-12-03 19:35:48 +11:00
Con Kolivas 688aa55c79 Make threads spawn at regular intervals along chunk size thus speeding up compression. 
One more fix for unnecessary argument passing.
 2010-12-03 19:33:56 +11:00
Con Kolivas 2a4892afa1 Streams passed are constant to some functions, don't pass stream argument to them. 2010-12-03 19:30:56 +11:00
Con Kolivas db95eba695 Take into account VM split on 32 bits again. 2010-12-03 19:30:27 +11:00
Con Kolivas d39d3959e8 uclibc doesn't return valid values via sysconf for ram, so work around it by reading /proc if possible. 2010-11-30 11:24:29 +11:00
Con Kolivas f2b5852872 Fix divide by zero error when a file is smaller than the page size and compressed with -U. 2010-11-30 11:22:02 +11:00