ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2026-04-04 22:07:42 +00:00
Code Issues Projects Releases Packages Wiki Activity

Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of

Browse source 
accessing the buffer directly, thus allowing us to have window sizes larger than
available ram. This is implemented through the use of a "sliding mmap"
implementation. Sliding mmap uses two mmapped buffers, one large one as
previously, and one page sized smaller one. When an attempt is made to read
beyond the end of the large buffer, the small buffer is remapped to the file
area that's being accessed. While this implementation is 100x slower than direct
mmapping, it allows us to implement unlimited sized compression windows.
Implement the -U option with unlimited sized windows.
Rework the selection of compression windows. Instead of trying to guess how
much ram the machine might be able to access, we try to safely buffer as much
ram as we can, and then use that to determine the file buffer size. Do not
choose an arbitrary upper window limit unless -w is specified.
Rework the -M option to try to buffer the entire file, reducing the buffer
size until we succeed.
Align buffer sizes to page size.
Clean up lots of unneeded variables.
Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
and the compression backends.
More error handling.
Change -L to affect rzip compression level directly as well as backend
compression level and use 9 by default now.
More cleanups of information output.
Use 3 point release numbering in case one minor version has many subversions.
Numerous minor cleanups and tidying.
Updated docs and manpages.
This commit is contained in:
Con Kolivas 2010-11-04 21:14:55 +11:00
parent c106128d1a
commit 29b166629a
12 changed files with 400 additions and 256 deletions
Download patch file Download diff file Expand all files Collapse all files

8
doc/README.benchmarks
View file

@ -89,12 +89,14 @@ gzip 2772899756 25.8 7m52.667s 4m8.661s
bzip2 2704781700 25.2 20m34.269s 7m51.362s
xz 2272322208 21.2 58m26.829s 4m46.154s
7z 2242897134 20.9 29m28.152s 6m35.952s
lrzip 1361276826 12.7 27m45.874s 9m20.046
lrzip(lzo) 1837206675 17.1 4m48.167s 8m28.842s
lrzip* 1354237684 12.6 29m13.402s 6m55.441s
lrzip(lzo)* 1828073980 17.0 3m34.816s 5m06.266s
lrzip(zpaq) 1341008779 12.5 4h11m14s
lrzip(zpaq)M 1270134391 11.8 4h30m14
lrzip(zpaq)MW 1066902006 9.9
(The benchmarks with * were done with version 0.5)
At this end of the spectrum things really start to heat up. The compression
advantage is massive, with the lzo backend even giving much better results
than 7z, and over a ridiculously short time. Note that it's not much longer
@ -117,4 +119,4 @@ Or, to make things easier, just use the default settings all the time and be
happy as lzma gives good results. :D
Con Kolivas
Sat, 19 Dec 2009
Tue, 2nd Nov 2010