diff --git a/ChangeLog b/ChangeLog index f8d280e..2914a73 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,4 +1,13 @@ lrzip ChangeLog +NOVEMBER 2010, version 0.5.2 Con Kolivas +* Fixed the Darwin build, again. +* Fixed cases of extreme ram usage on 32 bit failing by limiting zpaq to 600MB +windows as well. +* Check page size if we can instead of assuming it's always 4k. +* Improve the progress output. +* Change failure to chmod and failure to set nice level to warnings only. +* Standardise what's a stderr message and what's output. + NOVEMBER 2010, version 0.5.1 Con Kolivas * Fix Darwin build - Darwin doesn't support mremap so introduce a fake wrapper for it. diff --git a/README b/README index d2445cb..2ea30c2 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ -lrzip v0.5.1 +lrzip README Long Range ZIP or Lzma RZIP @@ -123,7 +123,8 @@ As you can see, the -U option gives the same compression in this case as the -M option, and for about 50% more time. The advantage to using -U is that it will work even when the size can't be encompassed by -M, but progressively slower. Why isn't it on by default? If the compression window is a LOT larger -than ram, with a lot of redundant information it can be drastically slower. +than ram, with a lot of redundant information it can be drastically slower. I +may revisit this possibility in the future if I can make it any faster. Q. Can I use your tool for even more compression than lzma offers? A. Yes, the rzip preparation of files makes them more compressible by every @@ -328,11 +329,12 @@ A. See http://www.7-zip.org and http://www.p7zip.org. Also, see the file LIMITATIONS Due to mmap limitations the maximum size a window can be set to is currently -2GB on 32bit. Files generated on 64 bit machines with windows >2GB in size -may not be decompressible on 32bit machines. +2GB on 32bit unless the -U option is specified. Files generated on 64 bit +machines with windows >2GB in size might not be decompressible on 32bit +machines. BUGS: -Probably lots. +Probably lots. Tell me if you spot any :) Links: @@ -357,7 +359,7 @@ Ed Avis for various fixes. Thanks to Matt Mahoney for zpaq code. Thanks to Jukka Laurila for Darwin support. Thanks to George Makrydakis for lrztar. Con Kolivas -Mon, 4 Nov 2010 +Mon, 7 Nov 2010 Also documented by Peter Hyman diff --git a/TODO b/TODO index c902d8c..3e0e702 100644 --- a/TODO +++ b/TODO @@ -10,9 +10,19 @@ for a current file without doing any writes. Consider ncurses version or even GUI one. -Consider using LZMA Filters for processor-optimized +Consider using LZMA Filters for processor-optimised coding to increase compression. Get the ASM working on 64bit. -Clean up the config system since it's a mystery to me. \ No newline at end of file +Clean up the config system since it's a mystery to me. + +Increased multi-threading. + +Make stdout work without a temporary file. + +Make stdin on decompression work without a temporary file. + +Make testing file integrity work without a temporary file. + +Stop breaking Darwin builds :P diff --git a/WHATS-NEW b/WHATS-NEW index 6e2d034..276ed97 100644 --- a/WHATS-NEW +++ b/WHATS-NEW @@ -1,3 +1,12 @@ +lrzip-0.5.2 + +Fixed the Darwin build again. +Fix the corner case of big ram usage on 32 bit zpaq failing due to the +compression window not being limited by limiting zpaq to 600MB windows on 32 +bits as well. +Some previous failures now only induce warnings. +Improved progress output. + lrzip-0.5.1 Fixed the build on Darwin. diff --git a/configure b/configure index b587f8e..57ad399 100755 --- a/configure +++ b/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.67 for lrzip 0.5.1. +# Generated by GNU Autoconf 2.67 for lrzip 0.5.2. # # Report bugs to . # @@ -551,9 +551,9 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='lrzip' -PACKAGE_TARNAME='lrzip-0.5.1' -PACKAGE_VERSION='0.5.1' -PACKAGE_STRING='lrzip 0.5.1' +PACKAGE_TARNAME='lrzip-0.5.2' +PACKAGE_VERSION='0.5.2' +PACKAGE_STRING='lrzip 0.5.2' PACKAGE_BUGREPORT='kernel@kolivas.org' PACKAGE_URL='' @@ -1221,7 +1221,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures lrzip 0.5.1 to adapt to many kinds of systems. +\`configure' configures lrzip 0.5.2 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1269,7 +1269,7 @@ Fine tuning of the installation directories: --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] - --docdir=DIR documentation root [DATAROOTDIR/doc/lrzip-0.5.1] + --docdir=DIR documentation root [DATAROOTDIR/doc/lrzip-0.5.2] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] @@ -1286,7 +1286,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of lrzip 0.5.1:";; + short | recursive ) echo "Configuration of lrzip 0.5.2:";; esac cat <<\_ACEOF @@ -1375,7 +1375,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -lrzip configure 0.5.1 +lrzip configure 0.5.2 generated by GNU Autoconf 2.67 Copyright (C) 2010 Free Software Foundation, Inc. @@ -2014,7 +2014,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by lrzip $as_me 0.5.1, which was +It was created by lrzip $as_me 0.5.2, which was generated by GNU Autoconf 2.67. Invocation command line was $ $0 $@ @@ -5324,7 +5324,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by lrzip $as_me 0.5.1, which was +This file was extended by lrzip $as_me 0.5.2, which was generated by GNU Autoconf 2.67. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -5386,7 +5386,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -lrzip config.status 0.5.1 +lrzip config.status 0.5.2 configured by $0, generated by GNU Autoconf 2.67, with options \\"\$ac_cs_config\\" diff --git a/configure.ac b/configure.ac index f6cbf1a..8fdba23 100644 --- a/configure.ac +++ b/configure.ac @@ -1,5 +1,5 @@ dnl Process this file with autoconf to produce a configure script. -AC_INIT([lrzip],[0.5.1],[kernel@kolivas.org],[lrzip-0.5.1]) +AC_INIT([lrzip],[0.5.2],[kernel@kolivas.org],[lrzip-0.5.2]) AC_CONFIG_HEADER(config.h) # see what our system is! AC_CANONICAL_HOST diff --git a/doc/README.benchmarks b/doc/README.benchmarks index 600f0d0..b94c15e 100644 --- a/doc/README.benchmarks +++ b/doc/README.benchmarks @@ -17,14 +17,14 @@ using lrzip v0.42. Compression Size Percentage Compress Decompress None 365711360 100 -7z 53315279 14.6 2m4.770s 0m5.360s -lrzip 52372722 14.3 2m48.477s 0m8.336s -lrzip -z 43455498 11.9 10m11.335 10m14.296 -lrzip -l 112151676 30.7 0m14.913s 0m5.063s -lrzip -g 73476127 20.1 0m29.628s 0m5.591s -lrzip -b 60851152 16.6 0m43.539s 0m12.244s -bzip2 62416571 17.1 0m44.493s 0m9.819s -gzip 80563601 22.0 0m14.343s 0m2.781s +7z 53315279 14.6 2m4s 0m5.4s +lrzip 52372722 14.3 2m48s 0m8.3s +lrzip -z 43455498 11.9 10m11s 10m14s +lrzip -l 112151676 30.7 0m14s 0m5.1s +lrzip -g 73476127 20.1 0m29s 0m5.6s +lrzip -b 60851152 16.6 0m43s 0m12.2s +bzip2 62416571 17.1 0m44s 0m9.8s +gzip 80563601 22.0 0m14s 0m2.8s These results are interesting to note the compression of lrzip by default is @@ -53,8 +53,11 @@ Tarball of 6 consecutive kernel trees. Compression Size Percentage Compress Decompress None 2373713920 100 7z 344088002 14.5 17m26s 1m22s +lrzip 104874109 4.4 11m37s 56s +lrzip -l 223130711 9.4 05m21s 1m01s lrzip -U 73356070 3.1 08m53s 43s lrzip -Ul 158851141 6.7 04m31s 35s +lrzip -Uz 62614573 2.6 24m42s 25m30s Things start getting very interesting now when lrzip is really starting to shine. Note how it's not that much larger for 6 kernel trees than it was for @@ -62,16 +65,21 @@ one. That's because all the similar data in both kernel trees is being compressed as one copy and only the differences really make up the extra size. All compression software does this, but not over such large distances. If you copy the same data over multiple times, the resulting lrzip archive doesn't -get much larger at all. +get much larger at all. You might find this example interesting because the +-U option is actually faster as well as providing better compression. The +reason is that the window is not much larger than the amount of ram addressable +(2GB), and it compresses so much more in the rzip stage that it makes up the +time by not needing to compress anywhere near as much data with the backend +compressor. Using the first example (linux-2.6.31.tar) and simply copying the data multiple times over gives these results with lrzip(lzo): Copies Size Compressed Compress Decompress -1 365711360 112151676 0m14.913s 0m5.063s -2 731422720 112151829 0m16.174s 0m6.543s -3 1097134080 112151832 0m17.466s 0m8.115s +1 365711360 112151676 0m14.9s 0m5.1s +2 731422720 112151829 0m16.2s 0m6.5s +3 1097134080 112151832 0m17.5s 0m8.1s I had the amusing thought that this compression software could be used as a @@ -92,14 +100,14 @@ These benchmarks were done on the quad core with version 0.5.1 Compression Size Percentage Compress Time Decompress Time None 10737418240 100.0 -gzip 2772899756 25.8 05m47.35s 2m46.77s -bzip2 2704781700 25.2 16m15.603s 6m19.718s -xz 2272322208 21.2 50m58.437s 3m52.734s -7z 2242897134 20.9 26m36.333s 5m41.198s -lrzip 1354237684 12.6 29m13.402s 6m55.441s -lrzip -M 1079528708 10.1 23m44.226s 4m05.461s -lrzip -l 1793312108 16.7 05m13.246s 3m12.886s -lrzip -lM 1413268368 13.2 04m18.338s 2m54.650s +gzip 2772899756 25.8 05m47s 2m46s +bzip2 2704781700 25.2 16m15s 6m19s +xz 2272322208 21.2 50m58s 3m52s +7z 2242897134 20.9 26m36s 5m41s +lrzip 1354237684 12.6 29m13s 6m55s +lrzip -M 1079528708 10.1 23m44s 4m05s +lrzip -l 1793312108 16.7 05m13s 3m12s +lrzip -lM 1413268368 13.2 04m18s 2m54s lrzip -z 1299844906 12.1 04h32m14s 04h33m lrzip -zM 1066902006 9.9 04h07m14s 04h08m @@ -123,4 +131,4 @@ Or, to make things easier, just use the default settings all the time and be happy as lzma gives good results. :D Con Kolivas -Tue, 5th Nov 2010 +Tue, 7th Nov 2010