accessing the buffer directly, thus allowing us to have window sizes larger than
available ram. This is implemented through the use of a "sliding mmap"
implementation. Sliding mmap uses two mmapped buffers, one large one as
previously, and one page sized smaller one. When an attempt is made to read
beyond the end of the large buffer, the small buffer is remapped to the file
area that's being accessed. While this implementation is 100x slower than direct
mmapping, it allows us to implement unlimited sized compression windows.
Implement the -U option with unlimited sized windows.
Rework the selection of compression windows. Instead of trying to guess how
much ram the machine might be able to access, we try to safely buffer as much
ram as we can, and then use that to determine the file buffer size. Do not
choose an arbitrary upper window limit unless -w is specified.
Rework the -M option to try to buffer the entire file, reducing the buffer
size until we succeed.
Align buffer sizes to page size.
Clean up lots of unneeded variables.
Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
and the compression backends.
More error handling.
Change -L to affect rzip compression level directly as well as backend
compression level and use 9 by default now.
More cleanups of information output.
Use 3 point release numbering in case one minor version has many subversions.
Numerous minor cleanups and tidying.
Updated docs and manpages.
fsync after flushing buffer.
Remove unnecessary memset after anonymous mmap.
Do test malloc before compression backend to see how big a chunk can be passed.
Make stdout write directly to stdout on decompression without the need for temporary files since there is no need to seek backwards.
Make file testing not actually write the file during test.
More tidying up.
Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space.
Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram.
Don't fail if preallocation is unsuccessful.
Add more detailed error reporting.
Minor cleanups.
This will increase speed of compression and generate a smaller file, but not be backward compatible.
Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine.
fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read.
Add lots more information to verbose mode.
Lots of code tidying and minor tweaks.
This reverts commit 603785f783.
Not everyone has a large /tmp directory and may not be able to fit the temporary files in /tmp so it's not a great idea to do this by default.
