ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
lrzip/WHATS-NEW

237 lines
8.5 KiB
Plaintext
Raw Normal View History

Push version number to 0.543. Update docs.
2010-11-24 11:08:35 +01:00
lrzip-0.543
A fix for when large files being decompressed fail with multithreaded
decompression.
Slight speedup on multithreaded workloads by decreasing the nice value of the
main process compared to the back end threads as it tends to be the rate
limiting component.
Fixed lzma compression windows being set way too small by default.
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
lrzip-0.542
Lrzip will now try to select sane defaults for memory usage in cases where the
virtual memory heavily overcommits (eg. Linux) as this seriously slows down
compression.
For compression windows larger than 2/3 ram, lrzip will now use a sliding mmap
buffer for better performance.
The progress output is more informative in max verbose mode, and will no longer
do more passes than it estimates.
32 bit machines should be able to use slightly larger windows.
Fix the sliding mmap buffer not working on 2nd pass onwards which was causing a slowdon of death.
2010-11-19 23:44:54 +01:00
The sliding mmap not working on 2nd pass onwards has been fixed which should
speed up the slowdown of death.
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
Increase version number to 0.540. Fix compression type reporting on -i. Remove some unnecessary output when -i is used. Update docs and benchmarks.
2010-11-16 13:14:35 +01:00
lrzip-0.540
MASSIVE MULTITHREADING on the decompression phase. Provided there are enough
chunks of data in the archived file, lrzip will use as many threads as there
are CPUs for the backend decompression. Much like the multithreading on the
compression side, it makes the slower compression algorithms speed up the most.
Fixed output from being scrambled and consuming a lot of CPU time on threaded
zpaq compression.
Further fixes to ensure window sizes work on 32 bit machines.
Be more careful about testing for how much ram lrzip can use.
Minor build warning fixes.
Minor tweaks to screen output.
Updated benchmarks.
Bump version number up to 0.530. Update all documentation. Minor fixes by Jari Aalto for build and docs.
2010-11-13 11:37:17 +01:00
lrzip-0.530
MASSIVE MULTITHREADING on the compression phase. Lrzip will now use as many
threads as you have CPU cores for the back end compression, and even continue
doing the rzip preprocessing stage as long as it can which the other threads
continue. This makes the slower compression algorithms (lzma and zpaq) much
faster on multicore machines, to the point of making zpaq compression almost
as fast as single threaded lzma compression.
-p option added to allow you to specify number of processors to override the
built-in test, or if you wish to disable threading.
-P option to not set permissions has now been removed since failing to set
permissions is only a warning now and not a failure.
Further improvements to the progress output.
Updated benchmarks and docs.
Distros don't like 3 point versioning scheme so change back to 2 point and change version to 0.520.
2010-11-08 00:43:05 +01:00
lrzip-0.520
Just changed version numbering back to 2 point.
Bump version number to 0.5.2. Update docs.
2010-11-07 05:33:07 +01:00
lrzip-0.5.2
Fixed the Darwin build again.
Fix the corner case of big ram usage on 32 bit zpaq failing due to the
compression window not being limited by limiting zpaq to 600MB windows on 32
bits as well.
Some previous failures now only induce warnings.
Improved progress output.
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
lrzip-0.5.1
Fixed the build on Darwin.
Rewrote the rzip compression phase to make it possible to use unlimited sized
Update changelogs.
2010-11-05 14:36:56 +01:00
windows now, not limited by ram. Unfortunately it gets progressively slower in
this mode the bigger the file gets but you can compress a file of any size as
one big compression window with it using the new -U option. Suggest you try
the new improved -M mode first or in combination. See the docs for more
information.
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
Changed the memory selection system to simply find the largest reasonable sized
window and use that by default instead of guessing the window size.
Setting -M now only affects the window size, trying to find the largest
unreasonably sized window that will still work.
The default compression level is now 9 and affects the rzip compression stage
as well as the backend compression.
Update changelogs.
2010-11-05 14:36:56 +01:00
Fixed some potential failures during compression.
Improved screen output with more reporting in verbose mode, and chunk size
percentage update.
Fixed file size reporting on compressed files generated from stdin.
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
Changed to 3 point releases in case we get more than 9 subversions ;)
Reinstate the temporary file for decompression from stdin. Seeking makes that option broken without the temporary file. Start documenting the new features in this version. Minor tidying.
2010-11-01 11:37:55 +01:00
lrzip-0.50
Rewrote the file format to be up to 5% more compact and slightly faster.
Made the memory initialisation much more robust, with attempted fallback
to still work even when initial settings fail.
Reinstate the temporary files for decompression to stdout and testing as the damaged line reinstated last commit meant it looked like those combinations worked when they actually didn't. Compression from stdin still works without temporary files.
2010-11-02 00:52:21 +01:00
Updated a lot of the stdin code.
The most common scenario of compression from stdin now works without
temporary files.
Reinstate the temporary file for decompression from stdin. Seeking makes that option broken without the temporary file. Start documenting the new features in this version. Minor tidying.
2010-11-01 11:37:55 +01:00
Lots more meaningful warnings if failure occurs.
Update docs. Bump version number. Tidy.
2010-11-02 01:14:00 +01:00
May be able to decompress files on 32 bit machines that were compressed on 64
bit machines with >2GB windows now if there is enough ram.
Reinstate the temporary file for decompression from stdin. Seeking makes that option broken without the temporary file. Start documenting the new features in this version. Minor tidying.
2010-11-01 11:37:55 +01:00
Typo.
2010-05-22 02:09:27 +02:00
lrzip-0.46
Import Jari Aalto's patches and prepare for 0.45
2010-03-30 09:45:48 +02:00
Add lrzuntar man, update manpages and prep for 0.46 release.
2010-05-22 02:07:18 +02:00
Added lrzuntar which works the same as lrztar -d.
Con Kolivas
May 2010
lrzip-0.45
Import Jari Aalto's patches and prepare for 0.45
2010-03-30 09:45:48 +02:00
Added docs for lrztar and lrunzip.
Added distclean and maintainer-clean make targets.
Created git repo: http://github.com/ckolivas/lrzip
Con Kolivas
March 2010
Initial import
2010-03-29 01:07:08 +02:00
lrzip-0.44
Added an lrztar wrapper to compress / decompress whole directories (finally).
Added -i option to give information about a compressed file.
lrzip-0.43
Darwin support updated. Should build on OSX v10.5+
Finally, stdin/stdout support.
Test archive integrity support.
ZPAQ support in config files.
lrzip-0.42
ZPAQ compression update now shows which rzip stream it's currently compressing
making the update more useful. It also doesn't update unnecessarily with every
byte compressed which was slowing it down a LOT.
lrzip-0.41
ZPAQ compression backend! ZPAQ is from the family of "paq" compressors that
have some of the best compression ratios around, but at the cost of extremely
long compression and equally long decompression times. This can be enabled
with the -z option and makes lrzip archives made with this not backwardly
compatible.
lrzip-0.40
Compression windows should be limited by available ram now on 64bit. The limit
on 32bit is still 2GB.
The compression advantages on large files on 64bit machines with large ram
should be substantially better.
The file format is no longer compatible with earlier versions of lrzip.
Support for decompressing older formats is present, but all new files will
be generated in the new format.
Minor speedups.
Decompression should no longer stall at 4GB boundaries for extended periods
making decompression much faster on files >4GB in size.
Documentation and benchmark updates galore.
lrzip-0.31
The window size limit is now 2GB on both 32bit and 64bit. While it appears to be
smaller than the old windows, only 900MB was being used on .30 even though it
claimed to use more. This can cause huge improvements in the compression of very
large files. Flushing of data to disk between compression windows was
implemented to minimise disk thrashing of read vs write.
Con Kolivas
November 2009
lrzip-0.30
-P option to not set permissions on output files allowing you to write to
braindead filesystems (eg fat32).
Probably other weird and wonderful bugs have been introduced.
Con Kolivas
November 2009
lrzip-0.24 has updated functionality
FEATURE ENHANCEMENTS
lrzip.conf file may be used to set default parameters.
Omit conf using environment: LRZIP=NOCONFIG lrzip.....
LRZIP environment variable may be used in the future
to store certain types of parameters.
LZMA SDK has been upgraded to version 4.63. This
version fixes some problems certain users observed,
and is much simpler using a C-only wrapper
interface.
lrzip now is able to compute an ETA for completion.
In order to do this, the file must be larger than
one compression window in size. That is, is the
compression window is 500MB, and the file is 1GB,
then after the first pass, an ETA will be computed.
If the file is smaller, then no estimate can be made.
lrzip is now able to compute MB/s transfer speeds
for both compression and decompression.
CLEANUPS
Some file cleanups have been done.
Peter Hyman
January 2009
pete@peterhyman.com
lrzip-0.22 update
FEATURE ENHANCEMENTS
-g option. Now supports gzip compression. Very fast!
Expanded dictionary buffer size in lzma compressor.
Variable, expanded dictionary size buffer in both lzma
compressor and decompressor.
Improved output during compression when using -vv.
Multi-threading supprt when using multiple processors
or dual core processors when using lzma compression.
This results in a nearly 2x speed improvement.
Assembler module support to speed up CRC checking.
Improvements in autotools usage, system detection
and Makefile enhancements.
Lrzip now has a timer that will print total time
at the end of a compression or decompression if
-q command line option is not used.
BUG FIX!!!
Even though lrzip uses a compression threshold to
prevent the lzma compressor from getting data that
may not be compressible, there was still a possibility
that lrzip could hang. This was because a data chunk
could contain an uncompressible segment and if the
lzma compressor got it, it would hang.
THANKS TO LASSE COLLIN for uncovering the error in
the lzma wrapper code that was causing the hangup.
January 2008
Peter Hyman
pete@peterhyman.com
Reference in a new issue Copy permalink