ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
152 commits 5 branches 43 tags 2.8 MiB
Commit graph

152 commits

This branch
This branch
All branches
Author SHA1 Message Date
Con Kolivas b94f68ba41 Remove unused md5_stream function. 2011-02-19 22:14:47 +11:00
Con Kolivas 744202a47f Remove unused variable. 2011-02-19 10:39:07 +11:00
Con Kolivas 9e589b3d2e Revert "Forgot the md5.x changes to support md5 on decompression." 
This reverts commit 006691ca8e.

md5_midstream no longer needed with decompression using existing buffers too now.
 2011-02-19 10:35:24 +11:00
Con Kolivas 7287ab8a66 Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time. 2011-02-19 10:34:45 +11:00
Con Kolivas d1a70bbb90 Remove the sb.low_top idea by trimming the size of sb.size_low when it gets to the end of the file instead. 2011-02-18 21:14:16 +11:00
Con Kolivas 006691ca8e Forgot the md5.x changes to support md5 on decompression. 2011-02-18 15:21:34 +11:00
Con Kolivas fb2a12744a Implement md5 checking on decompression. 
Implement hash check flag to determine whether to show md5sum on compression/decompression or not.
 2011-02-18 15:16:13 +11:00
Con Kolivas c2417877bc Being implementing md5 hash checking by importing the md5 code from coreutils. 
Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
 2011-02-18 09:09:40 +11:00
Con Kolivas cd8b086bf2 Minimise the number of mallocs in unzip_match. 2011-02-17 09:32:01 +11:00
Con Kolivas 626e0be281 Convert semaphore primitives to pthread_mutexes making them more portable, thus allowing multithreading to work on OSX. 2011-02-17 00:24:28 +11:00
Con Kolivas 05c5326df3 Revert "OSX doesn't support unnamed semaphores so to make it work, fake the threading by just creating the threads and waiting for them to finish." 
This reverts commit b81542cea4.

Revert the change bypassing semaphores in OSX in preparation for changing the semaphores to mutexes.
 2011-02-16 17:40:50 +11:00
Con Kolivas ea9b00c839 Update version number to 0.552 and update documentation. 2011-02-11 13:22:29 +11:00
Con Kolivas b81542cea4 OSX doesn't support unnamed semaphores so to make it work, fake the threading by just creating the threads and waiting for them to finish. 
This is done by making the semaphore wrappers null functions on osx and closing the thread in the creation wrapper.
Move the wrappers to rzip.h to make this change clean.
 2011-02-11 12:22:09 +11:00
Con Kolivas f2d33c00f8 Cast the mallocs to their variable type. 
Check that read and write actually return greater than zero.
 2011-02-11 11:46:58 +11:00
Con Kolivas 3879807865 Try limiting stream_read in unzip_literal and just returning how much was read. 2011-02-10 16:57:22 +11:00
Con Kolivas 9a3bfe33d1 Revert "Make sure to read the full length asked of unzip_literal." 
This reverts commit 499ae18cef.

Wrong fix, revert it.
 2011-02-10 16:46:35 +11:00
Con Kolivas 499ae18cef Make sure to read the full length asked of unzip_literal. 2011-02-10 15:30:31 +11:00
Con Kolivas 0a32b5f72d Convert mmaps to malloc in runzip as they may fail if not a multiple of page size! 2011-02-10 13:53:42 +11:00
Con Kolivas bece82a593 Trivial documentation fixes courtesy of Laszlo Ersek. 2011-02-10 13:14:36 +11:00
Con Kolivas 2a0553bc54 Revert "Decompress more than one stream at a time if there are threads free and the end of one stream is reached." 
This reverts commit 8ee9ef64f5.

This change is unreliable. Hence revert it and all dependent patches.
 2011-02-09 12:39:15 +11:00
Con Kolivas 8239635038 Revert "Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data." 
This reverts commit 0b0f6db606.
 2011-02-09 12:39:02 +11:00
Con Kolivas 44399d88ac Revert "Check we dont go over control.threads when trying to fill the other stream." 
This reverts commit 0bded0a7d9.
 2011-02-09 12:38:32 +11:00
Con Kolivas 0bded0a7d9 Check we dont go over control.threads when trying to fill the other stream. 
Update comments to reflect when we can go over control.threads.
 2011-02-09 11:07:38 +11:00
Con Kolivas 0b0f6db606 Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data. 2011-02-09 10:45:03 +11:00
Con Kolivas 8ee9ef64f5 Decompress more than one stream at a time if there are threads free and the end of one stream is reached. 
Still limit total threads running to control.threads.
This affords a small speedup on decompression.
 2011-02-08 11:58:01 +11:00
Con Kolivas aa00c29fba The free semaphore is now only updated from the main process on decompression so there are no synchronisation concerns. 
Remove the free semaphore and the fragile use of sem_trywait and replace it with a simple busy flag for threads on decompression.
 2011-02-08 08:55:36 +11:00
Con Kolivas 9c2b86fec6 We are flushing the wrong file on decompression. Make sure we flush the file out. 2011-02-08 08:27:22 +11:00
Con Kolivas 6b73eb1394 We do not need to wait for the main process to be ready to receive the data from the threads before shutting them down. 
Only wait for the "ready" semaphore if we've failed to decompress in parallel.
This affords a small speedup on decompression.
 2011-02-06 11:58:47 +11:00
Con Kolivas 8d110e3366 Remove the check for interleaved streams as it wasn't achieving anything. 
Make sure the thread has really exited before moving on, and set the free semaphore from outside the thread once it has joined.
 2011-02-06 11:21:36 +11:00
Con Kolivas 4efa81b1d0 Relax the memory allocation testing when no back end compression will be used. 2011-02-06 09:00:43 +11:00
Con Kolivas 191a55d03c Apply correct fix for freeBSD not implementing mremap thanks to Brian Carlson. 2011-01-02 14:23:29 +11:00
Con Kolivas f7a27cbde5 Flush writes to disk before allocating more ram, avoids one more fsync at the end of compression. 
Add fsyncing to decompression as well, improving reliability of allocating more ram.
 2010-12-18 10:02:19 +11:00
Con Kolivas e1bb9ebda5 There's a bug where archives with multiple stream 0 entries per chunk can be corrupted on decompression with threading enabled. 
Until the bug is tracked down, disable multithreaded decompression when an archive with interleaved stream 0/1 entries is detected.
 2010-12-18 00:50:49 +11:00
Con Kolivas 2cabb335cb Update copyright notices courtesy of Jari Aalto. 2010-12-16 09:45:21 +11:00
Con Kolivas e83612fc0a Freebsd doesn't do mremap. Use the fake one. 2010-12-16 09:42:34 +11:00
Con Kolivas becb08129c gotti spotted that bzip2 and gzip would fail if they encountered an incompressible block instead of returning it uncompressed, and provided a patch for it. 
Tweaked the patch slightly for style.
 2010-12-15 22:54:15 +11:00
Con Kolivas 654ae02ba6 Quick bump to version 0.551. 
Update docs.
 2010-12-12 17:48:07 +11:00
Con Kolivas 427df4cd4a Fix stdin failing due to getting sizes all wrong. 
Fix stdin compression values not being shown at end.
Fix inappropriate failure when lzma doesn't compress block.
 2010-12-12 17:40:58 +11:00
Con Kolivas 981859fdba Version to 0.550. 2010-12-12 10:51:29 +11:00
Con Kolivas d1db088986 Bump version to 0.550. 
Typo.
 2010-12-12 10:49:54 +11:00
Con Kolivas 28079083c1 Update docs. 2010-12-12 10:46:22 +11:00
Con Kolivas 34f76fa73c Don't wait on a semaphore the 2nd time attempting to compress/decompress. 
Wait for all ucompthreads to be free.
Destroy the semaphores used by ucompthreads when a stream_in is closed.
 2010-12-12 10:39:52 +11:00
Con Kolivas 351cfced8a Return the correct buffer on decompression failure in case decompression will be attempted again. 2010-12-11 14:25:07 +11:00
Con Kolivas aeeeedcab2 Cope with multithreaded memory failures better. 
Instead of failing completely, detect when a failure has occurred, and serialise work for that thread to decrease the memory required to complete compression / decompression.
Do that by waiting for the thread before it to complete before trying to work on that block again.
Check internally when lzma compress has failed and try a lower compression level before falling back to bzip2 compression.
 2010-12-11 13:19:34 +11:00
Con Kolivas 4331ca4058 Move message to after size is finalised. 2010-12-11 12:25:43 +11:00
Con Kolivas 6c33071118 Rationalise the testing now that the default lzma settings use a lot less ram by default. 2010-12-11 01:07:43 +11:00
Con Kolivas a6ab7c875b Limit the number of threads decompressing stream 0 to just 1 since it's always followed by stream 1 chunks, and it may lead to failure to decompress due to running out of memory by running too many threads. 2010-12-11 00:04:30 +11:00
Con Kolivas 50437a8447 Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. 
This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup.
Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring.
Fix a few dodgy callocs that may have been overflowing!
The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
 2010-12-10 23:51:59 +11:00
Con Kolivas 8dd9b00496 Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." 
This reverts commit 688aa55c79.
 2010-12-08 21:25:00 +11:00
Con Kolivas 5be0f6c5c9 Revert "Make sure chunk to compress is of a minimum size." 
This reverts commit 8dd1b6ea58.
 2010-12-08 21:23:41 +11:00