As we can work out what that compression overhead is, we can factor that into testing how much ram we can allocate.
There is no advantage to running multiple threads when there is no compression back end so drop to 1 only.
Limit ram for compression back end to 1/3 ram regardless for when OSs lie due to heavy overcommit.
Update main help screen to include environment settings.
Update to respect $TMP environment variable for TMP files.
Updated control structure to include tmpdir pointer.
Update lrzip.conf parser to respect -U -M options.
Update lrzip.conf example to include new parameters.
Reorder main Switch loop in main.c for readability.
Have MAXRAM and control.window be exclusive. MAXRAM wins.
Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
Have UNLIMITED and MAXRAM be exclusive. UNLIMITED wins.
Corrects heuristic computation in rzip.c which would override
MAXRAM or UNLIMITED if control.window set
Show heuristically computed control.window when computed.
Remove display compression level from control.window verbose output.
Update print_verbose format for Testing for incompressible data in stream.c
to omit extra \n.
Changes by Peter Hyman <pete@peterhyman.com>
Add the md5 value to the end of each archive.
This can then be used for integrity testing instead of crc32.
Keep crc in new archives to maintain compatibility with version 0.5 files.
Use md5 integrity testing on decompression when available in preference, and disable calculation of crc32.
Display the choice of integrity testing in verbose output and when -i is used.
Display the md5 and crc values when max verbosity, file info, or display hash is enabled.
Store a new flag in the magic header to show that the md5 value is stored at the end of the file.
Update the magic header information document.
Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
Limit lzma windows to 300MB in the right place on 32 bit only.
Make the main process less nice than the backend threads since it tends to be the rate limiting step.
Remove -P option as failing to set permissions only issues a warning now, removing any requirement for -P.
Change default compression level back to 7 as 9 was not giving significantly better compression but was slowing things down.
This facilitates a massive speed up on SMP machines proportional to the number of CPUs during the back end compression phase, but does so at some cost to the final size.
Limit the number of threads to ensure that each thread at least works on a window of STREAM_BUFSIZE.
Disable the lzma threading library as it does not contribute any more to the scalability of this new approach, yet compromises compression.
Increase the size of the windows passed to all the compression back end types now as they need more to split them up into multiple threads, and the number of blocks increases the compressed size slightly.
Prevent failure when offset is not a multiple of page size.
Add chunk percentage complete to output.
Tweak output at various verbosities.
Update documentation to reflect improved performance of unlimited mode.
Update benchmark results.
More tidying.
Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable.
Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c.
Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable.
Minor cleanups.
accessing the buffer directly, thus allowing us to have window sizes larger than
available ram. This is implemented through the use of a "sliding mmap"
implementation. Sliding mmap uses two mmapped buffers, one large one as
previously, and one page sized smaller one. When an attempt is made to read
beyond the end of the large buffer, the small buffer is remapped to the file
area that's being accessed. While this implementation is 100x slower than direct
mmapping, it allows us to implement unlimited sized compression windows.
Implement the -U option with unlimited sized windows.
Rework the selection of compression windows. Instead of trying to guess how
much ram the machine might be able to access, we try to safely buffer as much
ram as we can, and then use that to determine the file buffer size. Do not
choose an arbitrary upper window limit unless -w is specified.
Rework the -M option to try to buffer the entire file, reducing the buffer
size until we succeed.
Align buffer sizes to page size.
Clean up lots of unneeded variables.
Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
and the compression backends.
More error handling.
Change -L to affect rzip compression level directly as well as backend
compression level and use 9 by default now.
More cleanups of information output.
Use 3 point release numbering in case one minor version has many subversions.
Numerous minor cleanups and tidying.
Updated docs and manpages.
fsync after flushing buffer.
Remove unnecessary memset after anonymous mmap.
Do test malloc before compression backend to see how big a chunk can be passed.
Make stdout write directly to stdout on decompression without the need for temporary files since there is no need to seek backwards.
Make file testing not actually write the file during test.
More tidying up.
