Con Kolivas
8239635038
Revert "Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data."
...
This reverts commit 0b0f6db606
2011-02-09 12:39:02 +11:00
Con Kolivas
44399d88ac
Revert "Check we dont go over control.threads when trying to fill the other stream."
...
This reverts commit 0bded0a7d9
2011-02-09 12:38:32 +11:00
Con Kolivas
0bded0a7d9
Check we dont go over control.threads when trying to fill the other stream.
...
Update comments to reflect when we can go over control.threads.
2011-02-09 11:07:38 +11:00
Con Kolivas
0b0f6db606
Limit the maximum number of threads on stream 0 to 1 again as stream 1 data always appear after a chunk of stream 0 data.
2011-02-09 10:45:03 +11:00
Con Kolivas
8ee9ef64f5
Decompress more than one stream at a time if there are threads free and the end of one stream is reached.
...
Still limit total threads running to control.threads.
This affords a small speedup on decompression.
2011-02-08 11:58:01 +11:00
Con Kolivas
aa00c29fba
The free semaphore is now only updated from the main process on decompression so there are no synchronisation concerns.
...
Remove the free semaphore and the fragile use of sem_trywait and replace it with a simple busy flag for threads on decompression.
2011-02-08 08:55:36 +11:00
Con Kolivas
9c2b86fec6
We are flushing the wrong file on decompression. Make sure we flush the file out.
2011-02-08 08:27:22 +11:00
Con Kolivas
6b73eb1394
We do not need to wait for the main process to be ready to receive the data from the threads before shutting them down.
...
Only wait for the "ready" semaphore if we've failed to decompress in parallel.
This affords a small speedup on decompression.
2011-02-06 11:58:47 +11:00
Con Kolivas
8d110e3366
Remove the check for interleaved streams as it wasn't achieving anything.
...
Make sure the thread has really exited before moving on, and set the free semaphore from outside the thread once it has joined.
2011-02-06 11:21:36 +11:00
Con Kolivas
4efa81b1d0
Relax the memory allocation testing when no back end compression will be used.
2011-02-06 09:00:43 +11:00
Con Kolivas
191a55d03c
Apply correct fix for freeBSD not implementing mremap thanks to Brian Carlson.
2011-01-02 14:23:29 +11:00
Con Kolivas
f7a27cbde5
Flush writes to disk before allocating more ram, avoids one more fsync at the end of compression.
...
Add fsyncing to decompression as well, improving reliability of allocating more ram.
2010-12-18 10:02:19 +11:00
Con Kolivas
e1bb9ebda5
There's a bug where archives with multiple stream 0 entries per chunk can be corrupted on decompression with threading enabled.
...
Until the bug is tracked down, disable multithreaded decompression when an archive with interleaved stream 0/1 entries is detected.
2010-12-18 00:50:49 +11:00
Con Kolivas
2cabb335cb
Update copyright notices courtesy of Jari Aalto.
2010-12-16 09:45:21 +11:00
Con Kolivas
e83612fc0a
Freebsd doesn't do mremap. Use the fake one.
2010-12-16 09:42:34 +11:00
Con Kolivas
becb08129c
gotti spotted that bzip2 and gzip would fail if they encountered an incompressible block instead of returning it uncompressed, and provided a patch for it.
...
Tweaked the patch slightly for style.
2010-12-15 22:54:15 +11:00
Con Kolivas
654ae02ba6
Quick bump to version 0.551.
...
Update docs.
2010-12-12 17:48:07 +11:00
Con Kolivas
427df4cd4a
Fix stdin failing due to getting sizes all wrong.
...
Fix stdin compression values not being shown at end.
Fix inappropriate failure when lzma doesn't compress block.
2010-12-12 17:40:58 +11:00
Con Kolivas
981859fdba
Version to 0.550.
2010-12-12 10:51:29 +11:00
Con Kolivas
d1db088986
Bump version to 0.550.
...
Typo.
2010-12-12 10:49:54 +11:00
Con Kolivas
28079083c1
Update docs.
2010-12-12 10:46:22 +11:00
Con Kolivas
34f76fa73c
Don't wait on a semaphore the 2nd time attempting to compress/decompress.
...
Wait for all ucompthreads to be free.
Destroy the semaphores used by ucompthreads when a stream_in is closed.
2010-12-12 10:39:52 +11:00
Con Kolivas
351cfced8a
Return the correct buffer on decompression failure in case decompression will be attempted again.
2010-12-11 14:25:07 +11:00
Con Kolivas
aeeeedcab2
Cope with multithreaded memory failures better.
...
Instead of failing completely, detect when a failure has occurred, and serialise work for that thread to decrease the memory required to complete compression / decompression.
Do that by waiting for the thread before it to complete before trying to work on that block again.
Check internally when lzma compress has failed and try a lower compression level before falling back to bzip2 compression.
2010-12-11 13:19:34 +11:00
Con Kolivas
4331ca4058
Move message to after size is finalised.
2010-12-11 12:25:43 +11:00
Con Kolivas
6c33071118
Rationalise the testing now that the default lzma settings use a lot less ram by default.
2010-12-11 01:07:43 +11:00
Con Kolivas
a6ab7c875b
Limit the number of threads decompressing stream 0 to just 1 since it's always followed by stream 1 chunks, and it may lead to failure to decompress due to running out of memory by running too many threads.
2010-12-11 00:04:30 +11:00
Con Kolivas
50437a8447
Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished.
...
This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup.
Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring.
Fix a few dodgy callocs that may have been overflowing!
The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
2010-12-10 23:51:59 +11:00
Con Kolivas
8dd9b00496
Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression."
...
This reverts commit 688aa55c79
2010-12-08 21:25:00 +11:00
Con Kolivas
5be0f6c5c9
Revert "Make sure chunk to compress is of a minimum size."
...
This reverts commit 8dd1b6ea58
2010-12-08 21:23:41 +11:00
Con Kolivas
e0265b33e1
Default compression level and window size on lzma is set to 7 which is the highest it goes.
...
Scale the 9 levels down to the 7 that lzma has.
This makes the default lzma compression level 5 which is what lzma normally has, and uses a lot less ram and is significantly faster than previously, but at the cost of giving slightly less compression.
2010-12-08 20:53:26 +11:00
Con Kolivas
13e443e23c
Update docs.
2010-12-04 21:49:54 +11:00
Con Kolivas
c3dfcfcec1
Update version number to 0.544.
...
Change suggested maximum compression in README to disable threading with -p 1.
Use bzip2 as a fallback compression when lzma fails due to internal memory errors as may happen on 32 bits.
2010-12-04 21:36:51 +11:00
Con Kolivas
23e89b06af
Minor output consistency tidying.
2010-12-04 09:04:38 +11:00
Con Kolivas
dc42add0cf
Since 32 bit seems to fall over so easily, get ruthless with limits on it, while still maintaining a large window with sliding mmap.
2010-12-04 08:39:52 +11:00
Con Kolivas
8dd1b6ea58
Make sure chunk to compress is of a minimum size.
2010-12-03 20:02:39 +11:00
Con Kolivas
22c87ed3ee
Fix missing define.
2010-12-03 19:38:21 +11:00
Con Kolivas
2da407a178
Change decompression threading to have a group of threads for each stream (2 in total), thus making mulithreaded decompression more robust.
2010-12-03 19:35:48 +11:00
Con Kolivas
688aa55c79
Make threads spawn at regular intervals along chunk size thus speeding up compression.
...
One more fix for unnecessary argument passing.
2010-12-03 19:33:56 +11:00
Con Kolivas
2a4892afa1
Streams passed are constant to some functions, don't pass stream argument to them.
2010-12-03 19:30:56 +11:00
Con Kolivas
db95eba695
Take into account VM split on 32 bits again.
2010-12-03 19:30:27 +11:00
Con Kolivas
d39d3959e8
uclibc doesn't return valid values via sysconf for ram, so work around it by reading /proc if possible.
2010-11-30 11:24:29 +11:00
Con Kolivas
f2b5852872
Fix divide by zero error when a file is smaller than the page size and compressed with -U.
2010-11-30 11:22:02 +11:00
Con Kolivas
22da2ee76d
Push version number to 0.543.
...
Update docs.
2010-11-24 21:08:35 +11:00
Con Kolivas
6f2b94be3b
Fix the case where a compressed file has more than one stream 0 entry per block.
...
Limit lzma windows to 300MB in the right place on 32 bit only.
Make the main process less nice than the backend threads since it tends to be the rate limiting step.
2010-11-24 20:12:19 +11:00
Con Kolivas
180f5340fe
Bring version up to 0.542.
2010-11-20 18:34:45 +11:00
Con Kolivas
f04b8aa15b
Tweak defaults some more.
...
Set maximum size of compression window to chunk size in the case of -M mode and mmap size normally.
Set maximum window to 1.5x ram in -M mode.
2010-11-20 18:29:28 +11:00
Con Kolivas
6929492874
Fix the sliding mmap buffer not working on 2nd pass onwards which was causing a slowdon of death.
2010-11-20 09:44:54 +11:00
Con Kolivas
13ca377279
More tidying up and ensure control window is obeyed.
2010-11-20 09:22:47 +11:00
Con Kolivas
75e675e6dd
Bump version number to 0.542.
...
Choose sane defaults for memory usage since linux ludicriously overcommits.
Use sliding mmap for any compression windows greater than 2/3 ram.
Consolidate and simplify testing of allocatable ram.
Minor tweaks to output.
Round up the size of the high buffer in sliding mmap to one page.
Squeeze a little more out of 32 bit compression windows.
2010-11-20 01:23:08 +11:00
