mirror of
https://github.com/ckolivas/lrzip.git
synced 2025-12-06 07:12:00 +01:00
Full Rewrite (except for FAQ, minor changes there)
- Cleanup - Rewrite of most of document - Added GitHub Flavored Markdown - Contributors Table - Tl;DR - Misc. - etc.
This commit is contained in:
Haneef Mubarak
parent
acf81ead70
commit
c1f445308c
356
README
356
README
|
|
@ -1,356 +0,0 @@
|
|||
lrzip README
|
||||
|
||||
Long Range ZIP or Lzma RZIP
|
||||
|
||||
This is a compression program optimised for large files. The larger the file
|
||||
and the more memory you have, the better the compression advantage this will
|
||||
provide, especially once the files are larger than 100MB. The advantage can
|
||||
be chosen to be either size (much smaller than bzip2) or speed (much faster
|
||||
than bzip2).
|
||||
|
||||
|
||||
Quick lowdown of the most used options:
|
||||
|
||||
lrztar directory
|
||||
This will produce an archive directory.tar.lrz compressed with lzma.
|
||||
|
||||
lrzuntar directory.tar.lrz
|
||||
This will completely extract an archived directory.
|
||||
|
||||
lrzip filename
|
||||
This will produce an archive filename.lrz compressed with lzma (best all
|
||||
round) giving slow compression and fast decompression
|
||||
|
||||
lrzip -z filename
|
||||
This will produce an archive filename.lrz compressed with ZPAQ giving extreme
|
||||
compression but which takes ages to compress and decompress
|
||||
|
||||
lrzip -l filename
|
||||
This will produce an archive filename.lrz compressed with LZO giving very
|
||||
fast compression and fast decompression
|
||||
|
||||
lrunzip filename.lrz
|
||||
This will decompress filename.lrz into filename
|
||||
|
||||
|
||||
Lrzip uses an extended version of rzip which does a first pass long distance
|
||||
redundancy reduction. The lrzip modifications make it scale according to
|
||||
memory size.
|
||||
The data is then either:
|
||||
1. Compressed by lzma (default) which gives excellent compression
|
||||
at approximately twice the speed of bzip2 compression
|
||||
2. Compressed by a number of other compressors chosen for different reasons,
|
||||
in order of likelihood of usefulness:
|
||||
2a. ZPAQ: Extreme compression up to 20% smaller than lzma but ultra slow
|
||||
at compression AND decompression.
|
||||
2b. LZO: Extremely fast compression and decompression which on most machines
|
||||
compresses faster than disk writing making it as fast (or even faster) than
|
||||
simply copying a large file
|
||||
2c. GZIP: Almost as fast as LZO but with better compression.
|
||||
2d. BZIP2: A defacto linux standard of sorts but is the middle ground between
|
||||
lzma and gzip and neither here nor there.
|
||||
3. Leaving it uncompressed and rzip prepared. This form improves substantially
|
||||
any compression performed on the resulting file in both size and speed (due to
|
||||
the nature of rzip preparation merging similar compressible blocks of data and
|
||||
creating a smaller file). By "improving" I mean it will either speed up the
|
||||
very slow compressors with minor detriment to compression, or greatly increase
|
||||
the compression of simple compression algorithms.
|
||||
|
||||
The major disadvantages are:
|
||||
1. The main lrzip application only works on single files so it requires the
|
||||
lrztar wrapper to fake a complete archiver.
|
||||
2. It requires a lot of memory to get the best performance out of, and is not
|
||||
really usable (for compression) with less than 256MB. Decompression requires
|
||||
less ram and works on smaller ram machines. Sometimes swap may need to be
|
||||
enabled on these lower ram machines for the operating system to be happy.
|
||||
3. STDIN/STDOUT works fine on both compression and decompression, but larger
|
||||
files compressed in this manner will end up being less efficiently compressed.
|
||||
|
||||
The unique feature of lrzip is that it tries to make the most of the available
|
||||
ram in your system at all times for maximum benefit. It does this by default,
|
||||
choosing the largest sized window possible without running out of memory. It
|
||||
also has a unique "sliding mmap" feature which makes it possible to even use
|
||||
a compression window larger than your ramsize, if the file is that large. It
|
||||
does this (with the -U option) by implementing one large mmap buffer as per
|
||||
normal, and a smaller moving buffer to track which part of the file is
|
||||
currently being examined, emulating a much larger single mmapped buffer.
|
||||
Unfortunately this mode can be many times slower.
|
||||
|
||||
See the file README.benchmarks in doc/ for performance examples and what kind
|
||||
of data lrzip is very good with.
|
||||
|
||||
|
||||
Requires:
|
||||
pthreads
|
||||
liblzo2-dev
|
||||
libbz2-dev
|
||||
libz-dev
|
||||
libm
|
||||
tar
|
||||
(nasm on 32bit x86)
|
||||
|
||||
To build/install:
|
||||
./configure
|
||||
make
|
||||
make install
|
||||
|
||||
To build from the git repository do:
|
||||
./autogen.sh
|
||||
before the above steps.
|
||||
|
||||
|
||||
FAQS.
|
||||
Q. What encryption does lrzip use?
|
||||
A. Lrzip uses the best current proven technologies to achieve high grade
|
||||
password protected encryption. It uses a combination of sha512 to multiply
|
||||
hash the password with random salt and aes128 to do block encryption of each
|
||||
block of data with more random salt. The amount of initial hashing of the
|
||||
password increases by the date an lrzip archive is encrypted according to
|
||||
Moore's law, making it harder each year to brute force attack the password
|
||||
to keep up with the increasing computing power each year. It is virtually
|
||||
guaranteed that the same file encrypted with the same password will never
|
||||
be the same twice. The weakest link in this encryption mode by far is the
|
||||
password chosen by the user. There is currently no known attack or backdoor
|
||||
for this encryption mechanism, and there is absolutely no way of retrieving
|
||||
your password should you forget it.
|
||||
|
||||
Q. How do I make a static build?
|
||||
A. ./configure --enable-static-bin
|
||||
|
||||
Q. I want the absolute maximum compression I can possibly get, what do I do?
|
||||
A. Try the command line options "-Uzp 1 -L 9". This uses all available ram and
|
||||
ZPAQ compression, and even uses a compression window larger than you have ram.
|
||||
The -p 1 option disables multithreading which improves compression but at the
|
||||
expense of speed. Expect it to take many times longer.
|
||||
|
||||
Q. I want the absolute fastest decent compression I can possibly get.
|
||||
A. Try the command line option -l. This will use the lzo backend compression,
|
||||
and level 7 compression (1 isn't much faster).
|
||||
|
||||
Q. How much slower is the unlimited mode?
|
||||
A. It depends on 2 things. First, just how much larger than your ram the file
|
||||
is, as the bigger the difference, the slower it will be. The second is how much
|
||||
redundant data there is. The more there is, the slower, but ultimately the
|
||||
better the compression. Why isn't it on by default? If the compression window is
|
||||
a LOT larger than ram, with a lot of redundant information it can be drastically
|
||||
slower. I may revisit this possibility in the future if I can make it any
|
||||
faster.
|
||||
|
||||
Q. Can I use your tool for even more compression than lzma offers?
|
||||
A. Yes, the rzip preparation of files makes them more compressible by most
|
||||
other compression technique I have tried. Using the -n option will generate
|
||||
a .lrz file smaller than the original which should be more compressible, and
|
||||
since it is smaller it will compress faster than it otherwise would have.
|
||||
|
||||
Q. 32bit?
|
||||
A. 32bit machines have a limit of 2GB sized compression windows due to
|
||||
userspace limitations on mmap and malloc, so even if you have much more ram
|
||||
you will not be able to use compression windows larger than 2GB. Also you
|
||||
may be unable to decompress files compressed on 64bit machines which have
|
||||
used windows larger than 2GB.
|
||||
|
||||
Q. How about 64bit?
|
||||
A. 64bit machines with their ability to address massive amounts of ram will
|
||||
excel with lrzip due to being able to use compression windows limited only in
|
||||
size by the amount of physical ram.
|
||||
|
||||
Q. Other operating systems?
|
||||
A. The code is POSIXy with GNU extensions. Patches are welcome. Version 0.43+
|
||||
should build on MacOSX 10.5+
|
||||
|
||||
Q. Does it work on stdin/stdout?
|
||||
A. Yes it does. Compression and decompression work well to/from STDIN/STDOUT.
|
||||
However because lrzip does multiple passes on the data, it has to store a
|
||||
large amount in ram before it dumps it to STDOUT (and vice versa), thus it
|
||||
is unable to work with the massive compression windows regular operation
|
||||
provides. Thus the compression afforded on files larger than approximately
|
||||
25% RAM size will be less efficient (though still benefiting compared to
|
||||
traditional compression formats).
|
||||
|
||||
Q. I have another compression format that is even better than zpaq, can you
|
||||
use that?
|
||||
A. You can use it yourself on rzip prepared files (see above). Alternatively
|
||||
if the source code is compatible with the GPL license it can be added to the
|
||||
lrzip source code. Libraries with functions similar to compress() and
|
||||
decompress() functions of zlib would make the process most painless. Please
|
||||
tell me if you have such a library so I can include it :)
|
||||
|
||||
Q. What's this "Starting lzma back end compression thread..." message?
|
||||
A. While I'm a big fan of progress percentage being visible, unfortunately
|
||||
lzma compression can't currently be tracked when handing over 100+MB chunks
|
||||
over to the lzma library. Therefore you'll see progress percentage until
|
||||
each chunk is handed over to the lzma library.
|
||||
|
||||
Q. What's this "lzo testing for incompressible data" message?
|
||||
A. Other compression is much slower, and lzo is the fastest. To help speed up
|
||||
the process, lzo compression is performed on the data first to test that the
|
||||
data is at all compressible. If a small block of data is not compressible, it
|
||||
tests progressively larger blocks until it has tested all the data (if it fails
|
||||
to compress at all). If no compressible data is found, then the subsequent
|
||||
compression is not even attempted. This can save a lot of time during the
|
||||
compression phase when there is incompressible data. Theoretically it may be
|
||||
possible that data is compressible by the other backend (zpaq, lzma etc) and not
|
||||
at all by lzo, but in practice such data achieves only minuscule amounts of
|
||||
compression which are not worth pursuing. Most of the time it is clear one way
|
||||
or the other that data is compressible or not. If you wish to disable this
|
||||
test and force it to try compressing it anyway, use -T.
|
||||
|
||||
Q. I have truckloads of ram so I can compress files much better, but can my
|
||||
generated file be decompressed on machines with less ram?
|
||||
A. Yes. Ram requirements for decompression go up only by the -L compression
|
||||
option with lzma and are never anywhere near as large as the compression
|
||||
requirements. However if you're on 64bit and you use a compression window
|
||||
greater than 2GB, it might not be possible to decompress it on 32bit machines.
|
||||
|
||||
Q. Why are you including bzip2 compression?
|
||||
A. To maintain a similar compression format to the original rzip (although the
|
||||
other modes are more useful).
|
||||
|
||||
Q. What about multimedia?
|
||||
A. Most multimedia is already in a heavily compressed "lossy" format which by
|
||||
its very nature has very little redundancy. This means that there is not
|
||||
much that can actually be compressed. If your video/audio/picture is in a
|
||||
high bitrate, there will be more redundancy than a low bitrate one making it
|
||||
more suitable to compression. None of the compression techniques in lrzip are
|
||||
optimised for this sort of data. However, the nature of rzip preparation
|
||||
means that you'll still get better compression than most normal compression
|
||||
algorithms give you if you have very large files. ISO images of dvds for
|
||||
example are best compressed directly instead of individual .VOB files. ZPAQ is
|
||||
the only compression format that can do any significant compression of
|
||||
multimedia.
|
||||
|
||||
Q. Is this multithreaded?
|
||||
A. As of version 0.540, it is HEAVILY multithreaded with the back end
|
||||
compression and decompression phase, and will continue to process the rzip
|
||||
pre-processing phase so when using one of the more CPU intensive backend
|
||||
compressions like lzma or zpaq, SMP machines will show massive speed
|
||||
improvements. Lrzip will detect the number of CPUs to use, but it can be
|
||||
overridden with the -p option if the slightly better compression is desired
|
||||
more than speed. -p 1 will give the best compression but also be the slowest.
|
||||
|
||||
Q. This uses heaps of memory, can I make it use less?
|
||||
A. Well you can by setting -w to the lowest value (1) but the huge use of
|
||||
memory is what makes the compression better than ordinary compression
|
||||
programs so it defeats the point. You'll still derive benefit with -w 1 but
|
||||
not as much.
|
||||
|
||||
Q. What CFLAGS should I use?
|
||||
A. With a recent enough compiler (gcc>4) setting both CFLAGS and CXXFLAGS to
|
||||
-O2 -march=native -fomit-frame-pointer
|
||||
|
||||
Q. What compiler does this work with?
|
||||
A. It has been tested on gcc, ekopath and the intel compiler successfully
|
||||
previously. Whether the commercial compilers help or not, I could not tell you.
|
||||
|
||||
Q. What codebase are you basing this on?
|
||||
A. rzip v2.1 and lzma sdk920, but it should be possible to stay in sync with
|
||||
each of these in the future.
|
||||
|
||||
Q. Do we really need yet another compression format?
|
||||
A. It's not really a new one at all; simply a reimplementation of a few very
|
||||
good performing ones that will scale with memory and file size.
|
||||
|
||||
Q. How do you use lrzip yourself?
|
||||
A. Three basic uses. I compress large files currently on my drive with the
|
||||
-l option since it is so quick to get a space saving. When archiving data for
|
||||
permanent storage I compress it with the default options. When compressing
|
||||
small files for distribution I use the -z option for the smallest possible
|
||||
size.
|
||||
|
||||
Q. I found a file that compressed better with plain lzma. How can that be?
|
||||
A. When the file is more than 5 times the size of the compression window
|
||||
you have available, the efficiency of rzip preparation drops off as a means
|
||||
of getting better compression. Eventually when the file is large enough,
|
||||
plain lzma compression will get better ratios. The lrzip compression will be
|
||||
a lot faster though. The only way around this is to use as large compression
|
||||
windows as possible with -U option.
|
||||
|
||||
Q. Can I use swapspace as ram for lrzip with a massive window?
|
||||
A. It will indirectly do this with -U (unlimited) mode enabled. This mode will
|
||||
make the compression window as big as the file itself no matter how big it is,
|
||||
but it will slow down proportionately more the bigger the file is than your ram.
|
||||
|
||||
Q. Why do you nice it to +19 by default? Can I speed up the compression by
|
||||
changing the nice value?
|
||||
A. This is a common misconception about what nice values do. They only tell the
|
||||
cpu process scheduler how to prioritise workloads, and if your application is
|
||||
the _only_ thing running it will be no faster at nice -20 nor will it be any
|
||||
slower at +19.
|
||||
|
||||
Q. What is the LZO Testing option, -T?
|
||||
A. LZO testing is normally performed for the slower back-end compression of LZMA
|
||||
and ZPAQ. The reasoning is that if it is completely incompressible by LZO then
|
||||
it will also be incompressible by them. Thus if a block fails to be compressed
|
||||
by the very fast LZO, lrzip will not attempt to compress that block with the
|
||||
slower compressor, thereby saving time. If this option is enabled, it will
|
||||
bypass the LZO testing and attempt to compress each block regardless.
|
||||
|
||||
Q. Compression and decompression progress on large archives slows down and
|
||||
speeds up. There's also a jump in the percentage at the end?
|
||||
A. Yes, that's the nature of the compression/decompression mechanism. The jump
|
||||
is because the rzip preparation makes the amount of data much smaller than the
|
||||
compression backend (lzma) needs to compress.
|
||||
|
||||
Q. Tell me about patented compression algorithms, GPL, lawyers and copyright.
|
||||
A. No
|
||||
|
||||
Q. I receive an error "LZMA ERROR: 2. Try a smaller compression window."
|
||||
what does this mean?
|
||||
A. LZMA requests large amounts of memory. When a higher compression window is
|
||||
used, there may not be enough contiguous memory for LZMA. LZMA may request
|
||||
up to 25% of TOTAL ram depending on compression level. If contiguous blocks
|
||||
of memory are not free, LZMA will return an error. This is not a fatal
|
||||
error, and a backup mode of compression will be used.
|
||||
|
||||
Q. Where can I get more information about the internals of LZMA?
|
||||
A. See http://www.7-zip.org and http://www.p7zip.org. Also, see the file
|
||||
./lzma/C/lzmalib.h which explains the LZMA properties used and the LZMA
|
||||
memory requirements and computation.
|
||||
|
||||
Q. This version is much slower than the old version?
|
||||
A. Make sure you have set CFLAGS and CXXFLAGS. An unoptimised build will be
|
||||
almost 3 times slower.
|
||||
|
||||
LIMITATIONS
|
||||
Due to mmap limitations the maximum size a window can be set to is currently
|
||||
2GB on 32bit unless the -U option is specified. Files generated on 64 bit
|
||||
machines with windows >2GB in size might not be decompressible on 32bit
|
||||
machines. Large files might not decompress on machines with less RAM if SWAP is
|
||||
disabled.
|
||||
|
||||
BUGS:
|
||||
Probably lots. Tell me if you spot any :) Any known ones should be documented
|
||||
in the file BUGS.
|
||||
|
||||
|
||||
Links:
|
||||
rzip:
|
||||
http://rzip.samba.org/
|
||||
lzo:
|
||||
http://www.oberhumer.com/opensource/lzo/
|
||||
lzma:
|
||||
http://www.7-zip.org/
|
||||
zpaq:
|
||||
http://mattmahoney.net/dc/
|
||||
|
||||
Thanks to Andrew Tridgell for rzip. Thanks to Markus Oberhumer for lzo.
|
||||
Thanks to Igor Pavlov for lzma. Thanks to Jean-loup Gailly and Mark Adler
|
||||
for the zlib compression library. Thanks to Christian Leber for lzma
|
||||
compat layer, Michael J Cohen for Darwin support, Lasse Collin for fix
|
||||
to LZMALib.cpp and for Makefile.in suggestions, and everyone else who coded
|
||||
along the way. Huge thanks to Peter Hyman for most of the 0.19-0.24 changes,
|
||||
and the update to the multithreaded lzma library and all sorts of other
|
||||
features. Thanks to René Rhéaume for fixing executable stacks and
|
||||
Ed Avis for various fixes. Thanks to Matt Mahoney for zpaq code. Thanks to
|
||||
Jukka Laurila for Darwin support. Thanks to George Makrydakis for lrztar.
|
||||
Thanks to Ulrich Drepper for his md5 implementation. Thanks to Michael
|
||||
Blumenkrantz for new configuration tools. Thanks to the PolarSSL authors for
|
||||
encryption code. Thanks to Serge Belyshev for extensive help, advice, and
|
||||
patches to implement encryption. Michael Blumenkrantz also for liblrzip.
|
||||
|
||||
Con Kolivas <kernel@kolivas.org>
|
||||
Sat, 11 March 2011
|
||||
|
||||
Also documented by
|
||||
Peter Hyman <pete@peterhyman.com>
|
||||
Sun, 04 Jan 2009
|
||||
476
README.md
Normal file
476
README.md
Normal file
|
|
@ -0,0 +1,476 @@
|
|||
lrzip - Long Range ZIP or LZMA RZIP
|
||||
===================================
|
||||
|
||||
A compression utility that excels at compressing large files (usually > 10-50 MB).
|
||||
Larger files and/or more free RAM means that the utility will be able to more
|
||||
effectively compress your files (ie: faster / smaller size), especially if the
|
||||
filesize(s) exceed 100 MB. You can either choose to optimize for speed (fast
|
||||
compression / decompression) or size, but not both.
|
||||
|
||||
|
||||
### haneefmubarak's TL;DR for the long explanation:
|
||||
|
||||
Just change the word `directory` to the name of the directory you wish to compress.
|
||||
|
||||
#### Compression:
|
||||
|
||||
```bash
|
||||
lrzdir=directory; tar cvf $lrzdir; lrzip -Ubvvp `nproc` -S .bzip2-lrz -L 9 $lrzdir.tar; rm -fv $lrzdir.tar; unset lrzdir
|
||||
```
|
||||
|
||||
`tar`s the directory, then maxes out all of the system's processor cores
|
||||
along with sliding window RAM to give the best **BZIP2** compression while being as fast as possible,
|
||||
enables max verbosity output, attaches the extension `.bzip2-lrz`, and finally
|
||||
gets rid of the temporary tarfile. Uses a tempvar `lrzdir` which is unset automatically.
|
||||
|
||||
#### Decompression for the kind of file from above:
|
||||
|
||||
```bash
|
||||
lrzdir=directory; lrunzip -cdivvp `nproc` -o $lrzdir.tar $lrzdir.tar.bzip2-lrz; tar xvf $lrzdir.tar; rm -vf $lrzdir.tar
|
||||
```
|
||||
|
||||
Checks integrity, then decompresses the directory using all of the
|
||||
processor cores for max speed, enables max verbosity output, unarchives
|
||||
the resulting tarfile, and finally gets rid of the temporary tarfile. Uses the same kind of tempvar.
|
||||
|
||||
|
||||
### lrzip build/install guide:
|
||||
|
||||
A quick guide on building and installing.
|
||||
|
||||
#### What you will need
|
||||
|
||||
- gcc
|
||||
- bash or zsh
|
||||
- pthreads
|
||||
- tar
|
||||
- libc
|
||||
- libm
|
||||
- libz-dev
|
||||
- libbz2-dev
|
||||
- liblzo2-dev
|
||||
- coreutils
|
||||
- nasm on x86, not needed on x64
|
||||
- git if you want a repo-fresh copy
|
||||
- an OS with the usual *nix headers and libraries
|
||||
|
||||
#### Obtaining the source
|
||||
|
||||
Two different ways of doing this:
|
||||
|
||||
Stable: Packaged tarball that is known to work:
|
||||
|
||||
Go to <https://github.com/ckolivas/lrzip/releases> and downlaod the `tar.gz`
|
||||
file from the top. `cd` to the directory you downloaded, and use `tar xvzf lrzip-X.X.tar.gz`
|
||||
to extract the files (don't forget to replace `X.X` with the correct version). Finally, cd
|
||||
into the directory you just extracted.
|
||||
|
||||
Latest: `git clone -v https://github.com/ckolivas/lrzip.git; cd lrzip`
|
||||
|
||||
#### Build
|
||||
|
||||
```bash
|
||||
./autogen.sh
|
||||
./configure
|
||||
make -j `nproc` # maxes out all cores
|
||||
```
|
||||
|
||||
#### Install
|
||||
|
||||
Simple 'n Easy™: `sudo make install`
|
||||
|
||||
### lrzip 101:
|
||||
|
||||
|Command|Result|
|
||||
|------|------|
|
||||
|`lrztar directory`|An archive `directory.tar.lrz` compressed with **LZMA**.|
|
||||
|`lrzuntar directory.tar.lrz`|A directory extracted from a `lrztar` archive.|
|
||||
|`lrzip filename`|An archive `filename.lrz` compressed with **LZMA**, meaning slow compression and fast decompression.|
|
||||
|`lrzip -z filename`|An archive "filename.lrz" compressed with **ZPAQ** that can give extreme compression, but takes a bit longer than forever to compress and decompress.|
|
||||
|`lrzip -l filename`|An archive lightly compressed with **LZO**, meaning really, really fast compression and decompression.|
|
||||
|`lrunzip filename.lrz`|Decompress filename.lrz to filename.|
|
||||
|
||||
### lrzip internals
|
||||
|
||||
lrzip uses an extended version of [rzip](http://rzip.samba.org/) which does a first pass long distance
|
||||
redundancy reduction. lrzip's modifications allow it to scale to accomodate various memory sizes.
|
||||
|
||||
Then, one of the following scenarios occurs:
|
||||
|
||||
- Compressed
|
||||
- (default) **LZMA** gives excellent compression @ ~2x the speed of bzip2
|
||||
- **ZPAQ** gives extreme compression while taking forever
|
||||
- **LZO** gives insanely fast compression that can actually be faster than simply copying a large file
|
||||
- **GZIP** gives compression almost as fast as LZO but with better compression
|
||||
- **BZIP2** is a defacto linux standard and hacker favorite which usually gives
|
||||
quite good compression (ZPAQ>LZMA>BZIP2>GZIP>LZO) while staying fairly fast (LZO>GZIP>BZIP2>LZMA>ZPAQ);
|
||||
in other words, a good middle-ground and a good choice overall
|
||||
- Uncompressed, in the words of the software's original author:
|
||||
|
||||
> Leaving it uncompressed and rzip prepared. This form improves substantially
|
||||
> any compression performed on the resulting file in both size and speed (due to
|
||||
> the nature of rzip preparation merging similar compressible blocks of data and
|
||||
> creating a smaller file). By "improving" I mean it will either speed up the
|
||||
> very slow compressors with minor detriment to compression, or greatly increase
|
||||
> the compression of simple compression algorithms.
|
||||
>
|
||||
> (Con Kolivas, from the original lrzip README)
|
||||
|
||||
|
||||
The only real disadvantages:
|
||||
|
||||
- The main program, lrzip, only works on single files, and therefore
|
||||
requires the use of an lrztar wrapper to fake a complete archiver.
|
||||
- lrzip requires quite a bit of memory along with a modern processor
|
||||
to get the best performance in reasonable time. This usually means that
|
||||
it is somewhat unusable with less than 256 MB. However, decompression
|
||||
usually requires less RAM and can work on less powerful machines with much
|
||||
less RAM. On machines with less RAM, it may be a good idea to enable swap
|
||||
if you want to keep your operating system happy.
|
||||
- Piping output to and/or from STDIN and/or STDOUT works fine with both
|
||||
compression and decompression, but larger files compressed this way will
|
||||
likely end up being compressed less efficiently. Decompression doesn't
|
||||
really have any issues with piping, though.
|
||||
|
||||
One of the more unique features of lrzip is that it will try to use all of
|
||||
the available RAM as best it can at all times to provide maximum benefit. This
|
||||
is the default operating method, where it will create and use the single
|
||||
largest memory window that will still fit in available memory without freezing
|
||||
up the system. It does this by `mmap`ing the small portions of the file that
|
||||
it is working on. However, it also has a unique "sliding `mmap`" feature, which
|
||||
allows it to use compression windows that far exceed the size of your RAM if
|
||||
the file you are compressing is large. It does this by using one large `mmap`
|
||||
along with a smaller moving `mmap` buffer to track the part of the file that
|
||||
is currently being examined. From a higher level, this can be seen as simply
|
||||
emulating a single, large `mmap` buffer. The unfortunate thing about this
|
||||
feature is that it can become extremely slow. The counter-argument to
|
||||
being slower is that it will usually give a better compression factor.
|
||||
|
||||
The file `doc/README.benchmarks` has some performance examples to show
|
||||
what kind of data lrzip is good with.
|
||||
|
||||
|
||||
|
||||
### FAQ
|
||||
|
||||
> Q: What kind of encryption does lrzip use?
|
||||
|
||||
> A: lrzip uses SHA2-512 repetitive hashing of the password along with a salt
|
||||
> to provide a key which is used by AES-128 to do block encryption. Each block
|
||||
> has more random salts added to the block key. The amount of initial hashing
|
||||
> increases as the timestamp goes forward, in direct relation to Moore's law,
|
||||
> which means that the amount of time required to encrypt/decrypt the file
|
||||
> stays the same on a contemporary computer. It is virtually
|
||||
> guaranteed that the same file encrypted with the same password will never
|
||||
> be the same twice. The weakest link in this encryption mode by far is the
|
||||
> password chosen by the user. There is currently no known attack or backdoor
|
||||
> for this encryption mechanism, and there is absolutely no way of retrieving
|
||||
> your password should you forget it.
|
||||
|
||||
> Q: How do I make a static build?
|
||||
|
||||
> A: `./configure --enable-static-bin`
|
||||
|
||||
> Q: I want the absolute maximum compression I can possibly get, what do I do?
|
||||
|
||||
> A: Try the command line options "-Uzp 1 -L 9". This uses all available ram and
|
||||
> ZPAQ compression, and even uses a compression window larger than you have ram.
|
||||
> The -p 1 option disables multithreading which improves compression but at the
|
||||
> expense of speed. Expect it to take many times longer.
|
||||
|
||||
> Q: I want the absolute fastest decent compression I can possibly get.
|
||||
|
||||
> A: Try the command line option -l. This will use the lzo backend compression,
|
||||
> and level 7 compression (1 isn't much faster).
|
||||
|
||||
> Q: How much slower is the unlimited mode?
|
||||
|
||||
> A: It depends on 2 things. First, just how much larger than your ram the file
|
||||
is, as the bigger the difference, the slower it will be. The second is how much
|
||||
redundant data there is. The more there is, the slower, but ultimately the
|
||||
better the compression. Why isn't it on by default? If the compression window is
|
||||
a LOT larger than ram, with a lot of redundant information it can be drastically
|
||||
slower. I may revisit this possibility in the future if I can make it any
|
||||
faster.
|
||||
|
||||
> Q: Can I use your tool for even more compression than lzma offers?
|
||||
|
||||
> A: Yes, the rzip preparation of files makes them more compressible by most
|
||||
other compression technique I have tried. Using the -n option will generate
|
||||
a .lrz file smaller than the original which should be more compressible, and
|
||||
since it is smaller it will compress faster than it otherwise would have.
|
||||
|
||||
> Q: 32bit?
|
||||
|
||||
> A: 32bit machines have a limit of 2GB sized compression windows due to
|
||||
userspace limitations on mmap and malloc, so even if you have much more ram
|
||||
you will not be able to use compression windows larger than 2GB. Also you
|
||||
may be unable to decompress files compressed on 64bit machines which have
|
||||
used windows larger than 2GB.
|
||||
|
||||
> Q: How about 64bit?
|
||||
|
||||
> A: 64bit machines with their ability to address massive amounts of ram will
|
||||
excel with lrzip due to being able to use compression windows limited only in
|
||||
size by the amount of physical ram.
|
||||
|
||||
> Q: Other operating systems?
|
||||
|
||||
> A: The code is POSIXy with GNU extensions. Patches are welcome. Version 0.43+
|
||||
should build on MacOSX 10.5+
|
||||
|
||||
> Q: Does it work on stdin/stdout?
|
||||
|
||||
> A: Yes it does. Compression and decompression work well to/from STDIN/STDOUT.
|
||||
However because lrzip does multiple passes on the data, it has to store a
|
||||
large amount in ram before it dumps it to STDOUT (and vice versa), thus it
|
||||
is unable to work with the massive compression windows regular operation
|
||||
provides. Thus the compression afforded on files larger than approximately
|
||||
25% RAM size will be less efficient (though still benefiting compared to
|
||||
traditional compression formats).
|
||||
|
||||
> Q: I have another compression format that is even better than zpaq, can you
|
||||
use that?
|
||||
|
||||
> A: You can use it yourself on rzip prepared files (see above). Alternatively
|
||||
if the source code is compatible with the GPL license it can be added to the
|
||||
lrzip source code. Libraries with functions similar to compress() and
|
||||
decompress() functions of zlib would make the process most painless. Please
|
||||
tell me if you have such a library so I can include it :)
|
||||
|
||||
> Q: What's this "Starting lzma back end compression thread..." message?
|
||||
|
||||
> A: While I'm a big fan of progress percentage being visible, unfortunately
|
||||
lzma compression can't currently be tracked when handing over 100+MB chunks
|
||||
over to the lzma library. Therefore you'll see progress percentage until
|
||||
each chunk is handed over to the lzma library.
|
||||
|
||||
> Q: What's this "lzo testing for incompressible data" message?
|
||||
|
||||
> A: Other compression is much slower, and lzo is the fastest. To help speed up
|
||||
the process, lzo compression is performed on the data first to test that the
|
||||
data is at all compressible. If a small block of data is not compressible, it
|
||||
tests progressively larger blocks until it has tested all the data (if it fails
|
||||
to compress at all). If no compressible data is found, then the subsequent
|
||||
compression is not even attempted. This can save a lot of time during the
|
||||
compression phase when there is incompressible dat
|
||||
> A: Theoretically it may be
|
||||
possible that data is compressible by the other backend (zpaq, lzma etc) and not
|
||||
at all by lzo, but in practice such data achieves only minuscule amounts of
|
||||
compression which are not worth pursuing. Most of the time it is clear one way
|
||||
or the other that data is compressible or not. If you wish to disable this
|
||||
test and force it to try compressing it anyway, use -T.
|
||||
|
||||
> Q: I have truckloads of ram so I can compress files much better, but can my
|
||||
generated file be decompressed on machines with less ram?
|
||||
|
||||
> A: Yes. Ram requirements for decompression go up only by the -L compression
|
||||
option with lzma and are never anywhere near as large as the compression
|
||||
requirements. However if you're on 64bit and you use a compression window
|
||||
greater than 2GB, it might not be possible to decompress it on 32bit machines.
|
||||
|
||||
> Q: Why are you including bzip2 compression?
|
||||
|
||||
> A: To maintain a similar compression format to the original rzip (although the
|
||||
other modes are more useful).
|
||||
|
||||
> Q: What about multimedia?
|
||||
|
||||
> A: Most multimedia is already in a heavily compressed "lossy" format which by
|
||||
its very nature has very little redundancy. This means that there is not
|
||||
much that can actually be compressed. If your video/audio/picture is in a
|
||||
high bitrate, there will be more redundancy than a low bitrate one making it
|
||||
more suitable to compression. None of the compression techniques in lrzip are
|
||||
optimised for this sort of dat
|
||||
> A: However, the nature of rzip preparation
|
||||
means that you'll still get better compression than most normal compression
|
||||
algorithms give you if you have very large files. ISO images of dvds for
|
||||
example are best compressed directly instead of individual .VOB files. ZPAQ is
|
||||
the only compression format that can do any significant compression of
|
||||
multimedi
|
||||
> A:
|
||||
|
||||
> Q: Is this multithreaded?
|
||||
|
||||
> A: As of version 0.540, it is HEAVILY multithreaded with the back end
|
||||
compression and decompression phase, and will continue to process the rzip
|
||||
pre-processing phase so when using one of the more CPU intensive backend
|
||||
compressions like lzma or zpaq, SMP machines will show massive speed
|
||||
improvements. Lrzip will detect the number of CPUs to use, but it can be
|
||||
overridden with the -p option if the slightly better compression is desired
|
||||
more than speed. -p 1 will give the best compression but also be the slowest.
|
||||
|
||||
> Q: This uses heaps of memory, can I make it use less?
|
||||
|
||||
> A: Well you can by setting -w to the lowest value (1) but the huge use of
|
||||
memory is what makes the compression better than ordinary compression
|
||||
programs so it defeats the point. You'll still derive benefit with -w 1 but
|
||||
not as much.
|
||||
|
||||
> Q: What CFLAGS should I use?
|
||||
|
||||
> A: With a recent enough compiler (gcc>4) setting both CFLAGS and CXXFLAGS to
|
||||
-O2 -march=native -fomit-frame-pointer
|
||||
|
||||
> Q: What compiler does this work with?
|
||||
|
||||
> A: It has been tested on gcc, ekopath and the intel compiler successfully
|
||||
previously. Whether the commercial compilers help or not, I could not tell you.
|
||||
|
||||
> Q: What codebase are you basing this on?
|
||||
|
||||
> A: rzip v2.1 and lzma sdk920, but it should be possible to stay in sync with
|
||||
each of these in the future.
|
||||
|
||||
> Q: Do we really need yet another compression format?
|
||||
|
||||
> A: It's not really a new one at all; simply a reimplementation of a few very
|
||||
good performing ones that will scale with memory and file size.
|
||||
|
||||
> Q: How do you use lrzip yourself?
|
||||
|
||||
> A: Three basic uses. I compress large files currently on my drive with the
|
||||
-l option since it is so quick to get a space saving. When archiving data for
|
||||
permanent storage I compress it with the default options. When compressing
|
||||
small files for distribution I use the -z option for the smallest possible
|
||||
size.
|
||||
|
||||
> Q: I found a file that compressed better with plain lzm
|
||||
> A: How can that be?
|
||||
|
||||
> A: When the file is more than 5 times the size of the compression window
|
||||
you have available, the efficiency of rzip preparation drops off as a means
|
||||
of getting better compression. Eventually when the file is large enough,
|
||||
plain lzma compression will get better ratios. The lrzip compression will be
|
||||
a lot faster though. The only way around this is to use as large compression
|
||||
windows as possible with -U option.
|
||||
|
||||
> Q: Can I use swapspace as ram for lrzip with a massive window?
|
||||
|
||||
> A: It will indirectly do this with -U (unlimited) mode enabled. This mode will
|
||||
make the compression window as big as the file itself no matter how big it is,
|
||||
but it will slow down proportionately more the bigger the file is than your ram.
|
||||
|
||||
> Q: Why do you nice it to +19 by default? Can I speed up the compression by
|
||||
changing the nice value?
|
||||
|
||||
> A: This is a common misconception about what nice values do. They only tell the
|
||||
cpu process scheduler how to prioritise workloads, and if your application is
|
||||
the _only_ thing running it will be no faster at nice -20 nor will it be any
|
||||
slower at +19.
|
||||
|
||||
> Q: What is the LZO Testing option, -T?
|
||||
|
||||
> A: LZO testing is normally performed for the slower back-end compression of LZMA
|
||||
and ZPA> Q: The reasoning is that if it is completely incompressible by LZO then
|
||||
it will also be incompressible by them. Thus if a block fails to be compressed
|
||||
by the very fast LZO, lrzip will not attempt to compress that block with the
|
||||
slower compressor, thereby saving time. If this option is enabled, it will
|
||||
bypass the LZO testing and attempt to compress each block regardless.
|
||||
|
||||
> Q: Compression and decompression progress on large archives slows down and
|
||||
speeds up. There's also a jump in the percentage at the end?
|
||||
|
||||
> A: Yes, that's the nature of the compression/decompression mechanism. The jump
|
||||
is because the rzip preparation makes the amount of data much smaller than the
|
||||
compression backend (lzma) needs to compress.
|
||||
|
||||
> Q: Tell me about patented compression algorithms, GPL, lawyers and copyright.
|
||||
|
||||
> A: No
|
||||
|
||||
> Q: I receive an error "LZMA ERROR: 2. Try a smaller compression window."
|
||||
what does this mean?
|
||||
|
||||
> A: LZMA requests large amounts of memory. When a higher compression window is
|
||||
used, there may not be enough contiguous memory for LZM
|
||||
> A: LZMA may request
|
||||
up to 25% of TOTAL ram depending on compression level. If contiguous blocks
|
||||
of memory are not free, LZMA will return an error. This is not a fatal
|
||||
error, and a backup mode of compression will be used.
|
||||
|
||||
> Q: Where can I get more information about the internals of LZMA?
|
||||
|
||||
> A: See http://www.7-zip.org and http://www.p7zip.org. Also, see the file
|
||||
./lzma/C/lzmalib.h which explains the LZMA properties used and the LZMA
|
||||
memory requirements and computation.
|
||||
|
||||
> Q: This version is much slower than the old version?
|
||||
|
||||
> A: Make sure you have set CFLAGS and CXXFLAGS. An unoptimized build will be
|
||||
almost 3 times slower.
|
||||
|
||||
#### LIMITATIONS
|
||||
Due to mmap limitations the maximum size a window can be set to is currently
|
||||
2GB on 32bit unless the -U option is specified. Files generated on 64 bit
|
||||
machines with windows >2GB in size might not be decompressible on 32bit
|
||||
machines. Large files might not decompress on machines with less RAM if SWAP is
|
||||
disabled.
|
||||
|
||||
#### BUGS:
|
||||
Probably lots. <https://github.com/ckolivas/lrzip/issues> if you spot any :D
|
||||
|
||||
Any known ones should be documented
|
||||
in the file BUGS.
|
||||
|
||||
|
||||
|
||||
#### Backends:
|
||||
|
||||
rzip:
|
||||
<http://rzip.samba.org/>
|
||||
|
||||
lzo:
|
||||
<http://www.oberhumer.com/opensource/lzo/>
|
||||
|
||||
lzma:
|
||||
<http://www.7-zip.org/>
|
||||
|
||||
zpaq:
|
||||
<http://mattmahoney.net/dc/>
|
||||
|
||||
### Thanks (CONTRIBUTORS)
|
||||
|
||||
|Person(s)|Thanks for|
|
||||
|---|---|
|
||||
|`Andrew Tridgell`|`rzip`|
|
||||
|`Markus Oberhumer`|`lzo`|
|
||||
|`Igor Pavlov`|`lzma`|
|
||||
|`Jean-Loup Gailly & Mark Adler`|`zlib`|
|
||||
|***`Con Kolivas`***|***Original Code, binding all of this together, managing the project, original `README`***|
|
||||
|`Christian Leber`|`lzma` compatibility layer|
|
||||
|`Michael J Cohen`|Darwin/OSX support|
|
||||
|`Lasse Collin`|fixes to `LZMALib.cpp` and `Makefile.in`|
|
||||
|Everyone else who coded along the way (add yourself where appropriate if that's you)|Miscellaneous Coding|
|
||||
|**`Peter Hyman`**|Most of the `0.19` to `0.24` changes|
|
||||
|`^^^^^^^^^^^`|Updating the multithreaded `lzma` lib
|
||||
|`^^^^^^^^^^^`|All sorts of other features
|
||||
|`René Rhéaume`|Fixing executable stacks|
|
||||
|`Ed Avis`|Various fixes|
|
||||
|`Matt Mahoney`|`zpaq` integration code|
|
||||
|`Jukka Laurila`|Additional Darwin/OSX support|
|
||||
|`George Makrydakis`|`lrztar` wrapper|
|
||||
|`Ulrich Drepper`|*special* implementation of md5|
|
||||
|**`Michael Blumenkrantz`**|New config tools|
|
||||
|`^^^^^^^^^^^^^^^^^^^^`|`liblrzip`|
|
||||
|Authors of `PolarSSL`|Encryption code|
|
||||
|`Serge Belyshev`|Extensive help, advice, and patches to implement secure encryption|
|
||||
|`Jari Aalto`|Fixing typos, esp. in code|
|
||||
|`Carlo Alberto Ferraris`|Code cleanup
|
||||
|`Peter Hyman`|Additional documentation|
|
||||
|`Haneef Mubarak`|Cleanup, Rewrite, and GH Markdown of `README` --> `README.md`|
|
||||
|
||||
Persons above are listed in chronological order of first contribution to **lrzip**. Person(s) with names in **bold** have multiple major contributions, person(s) with names in *italics* have made massive contributions, person(s) with names in ***both*** have made innumerable massive contributions.
|
||||
|
||||
#### README Authors
|
||||
|
||||
Con Kolivas (`ckolivas` on GitHub) <kernel@kolivas.org>
|
||||
Sat, 11 March 2011: README
|
||||
|
||||
Also documented by
|
||||
Peter Hyman <pete@peterhyman.com>
|
||||
Sun, 04 Jan 2009: README
|
||||
|
||||
Mostly Rewritten + GFMified:
|
||||
Haneef Mubarak (haneefmubarak on GitHub)
|
||||
Sun/Mon Sep 01-02 2013: README.md
|
||||
Loading…
Reference in a new issue