Merge pull request #258 from eli-schwartz/dont-use-echo

build: fix incorrect use of shell commands in handmade libtool file

.github/workflows/c-cpp.yml

@@ -0,0 +1,29 @@
+name: check_build
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: prepare repo
+      run: git fetch --prune --unshallow
+    - name: autogen
+      run: ./autogen.sh
+    - name: install liblzo2-dev
+      run: sudo apt install -y liblzo2-dev
+    - name: install liblz4-dev
+      run: sudo apt install -y liblz4-dev
+    - name: configure
+      run: ./configure
+    - name: make
+      run: make
+    - name: make check
+      run: make check

.gitignore

@@ -16,6 +16,7 @@ libtool
 lrzip
 lrzip*.tar.bz2
 lrzip*.tar.gz
+lrzip*.tar.lrz
 ltmain.sh
 missing
 stamp-h1
@@ -24,7 +25,15 @@ ltoptions.m4
 ltsugar.m4
 ltversion.m4
 lt~obsolete.m4
+compile
 man/lrunzip.1
 man/lrzcat.1
 man/lrztar.1
 man/lrzuntar.1
+man/lrz.1
+libzpaq/.dirstamp
+lrzip.pc
+regressiontest.out
+decompress_demo
+liblrzip_demo
+

BUGS

@@ -1,4 +1,8 @@
-BUGME March 2011
+BUGME
 
+Issues can be reported/tracked here:
+https://github.com/ckolivas/lrzip/issues
+
+Known issues:
 Mac may not be able to work with STDIN/STDOUT on very large files.
 MD5 is disabled on Mac due to not working properly.

ChangeLog

@@ -1,905 +1 @@
-lrzip ChangeLog
-
-SEPTEMBER 2013, version 0.616 George Makrydakis
-
-* Making things more homogeneous in checks, some space removed.
-* Simplifying and refactoring logic, alignment edits, empty semantic rewire.
-* Target directories must always exist and -f should not create them.
-* Deferring output directory after -f check has taken place.
-* Preserve extraction semantics without resorting to tar stripping.
-* Making sure last forward slash is removed from input path.
-* Removing pointless forced overwrite point check.
-
-
-SEPTEMBER 2013, version 0.615 Con Kolivas, George Makrydakis
-
-* Adjusting -O flag semantics, options help update.
-* Making -O flag operative for lrztar, whitespace path fix, lrzuntar fix, other
-minor.
-* Further tighten up ram restrictions with stdin/stdout to prevent running out
-of memory with all the buffers involved.
-* Massive files fail with -U due to trying to allocate the whole lot in ram
-while doing checksums. Do it piecemeal to avoid the problem. Patch and debugging
-courtesy of Adam Tkáč.
-* We have to run through the clear buffer function even for empty buffers or
-corrupt archives with empty match streams.
-* MD5 code uses little endian so remove arbitrary SWAP macro and explicitly use
-htole32.
-* Rewrite the magic if we receive lzma properties and have not yet written them
-yet during stdout operation.
-* Set the control lzma properties only once.
-* Add a control lock mutex for protecting certain control variables.
-* Fix stdin fake mremap creating null bytes on osx, patch courtesy of John
-Boyle.
-* Cache frequently used indirectly referenced variables in the sliding mmap
-code.
-* Micro-optimise sliding_get_sb_range.
-* A fix for a bug where large files containing the same non-zero bytes which
-requires a sliding window, courtesy of Serge Belyshev.
-* Put vchar should take a 64 bit integer and is used from more than one call
-site so uninline it.
-* Microoptimise in hash_search.
-* Inline rzip functions used from only one caller.
-* Check for successful calloc of hash table only after performing it.
-* stdin_eof is just a bool
-* hash_bits can only be up to 64 so use a char type.
-* Check endianness of build with autotools to enable md5 support on more
-platforms.
-
-
-JULY 2012, version 0.614 Con Kolivas
-* Fix lrztar failing due to incorrect fix for -S freeing segfault.
-
-
-JULY 2012, version 0.613 Con Kolivas, Serge Belyshev.
-* Fix other non-win32 compilation for libzpaq.
-* Fix libzpaq compilation on other non-win32 OSs.
-* Fix Types.h missing which breaks typedefs on macosx.
-* Cope with missing ffsll() on various platforms.
-* Fix get_ram compilation on osx.
-* Rename lrzip.h to lrzip_core.h to cope with case insensitive filesystems not
-working with Lrzip.h vs lrzip.h
-* Include test for pthread.h and include header in lrzip_private.h.
-* Use system extensions in configure.ac to allow to build on non-gnu-linux
-platforms.
-* Fix invalid free error when -o and -S are used together.
-* Use no back end compression for blocks smaller than 64 bytes to avoid issues
-to do with minimum buffer size and minimum match sizes of the rzip
-preprocessing.
-* Fix md5 calculaton on blocks larger then 2^32. Bug reproduced and debugged by
-Serge Belyshev. Patch by Paul Eggert.
-
-
-MARCH 2012, version 0.612 Con Kolivas
-* Minor formatting improvement to delete misaligned %.
-* Implement progress counter on zpaq compress/decompress per thread within the
-Reader function.
-* Add some lrzip style defines and optimisations to the lrzip functions in
-libzpaq.h
-* Re-instantiate the lzo compresses test for zpaq compression.
-* Remove redundant code.
-* Delete now-unused zpipe code.
-* Implement virtual write support for bufWrite to speed up zpaq support and
-simplify the virtual read function for bufRead.
-* Add virtual read() support to the bufRead class to speed up encoding/decoding
-via the zpaq library.
-* Use libzpaq backend for decompression.
-* Move zpaq compression to new libzpaq library back end.
-* Merge branch 'master' into zpaq
-* Add minimal libzpaq compilation support.
-* Move to libzpaq directory.
-* Import libzapq files.
-* Fix Lrzip.h not being installed into include directory and make distcheck
-failing.
-
-
-MARCH 2012, version 0.611 Con Kolivas.
-* Move the update counter to earlier in the hash_search so it still updates even
-when there are not matches.
-* Perform all checksumming in a separate thread to speed up the hash search in
-the rzip phase.
-* Optimise the hot function empty_hash.
-* Micro-optimise the hot next_tag function.
-* Micro-optimise the match_len function.
-* Pass one less variable to the get_sb function as an optimisation.
-* Cache the hash_entry value passing one less variable to hash_empty as a
-micro-optimisation.
-* Use ffsl for a faster lesser_bitness function.
-* Prevent lrzuntar from automatically overwriting existing directories without
-the -f option being specified.
-* Calling lrzip via lrzcat would fail as the proper parameters weren't being
-passed to the control structure. Fix.
-
-
-MARCH 2012, version 0.610 Con Kolivas, Michael Blumenkrantz
-* Implement complete set of liblrzip libraries, documentation and example uses
-with support for simple lrzip_compress() and lrzip_decompress() or complete
-fine-grained control over all compression and decompression options.
-* Use as much of the low buffer as possible with a single memcopy before going
-fine grained byte by byte.
-* Preserve the compressed time on decompression where suitable.
-* Store a copy of the control struct to be reused on subsequent files to prevent
-variables being modified in the control struct on the first file that corrupt
-compression/decompression of the 2nd file.
-* Explicitly select C99 to avoid certain warnings.
-* Generic modifications to silence -Wextra warnings.
-* Fix typos.
-* Use an array of parameters in lrztar to allow working with directories with
-spaces in their names.
-
-
-SEPTEMBER 2011, version 0.608 Con Kolivas
-* get_sb only allows accessing one byte at a time, yet we don't need that
-functionality when sliding mmap is not in use. Use different versions of the
-function and the larger memcpys depending on whether sliding mmap is in use or
-not. This affords a substantial speedup in the rzip phase of files not requiring
-sliding mmap.
-* A small optimisation of moving the check for remapping the low buffer also
-speeds up the sliding mmap version slightly.
-* Installation of the included bash completion script was disabled to avoid
-conflict with distribution bash completion packages.
-* Minor cleanup of sha4.h
-* Make mremap use the fake mremap on all platforms not linux.
-
-SEPTEMBER 2011, version 0.607 Con Kolivas
-* Updated lzma library to version 920.
-* Fixed a rare unable-to-decompress corner case.
-* Added lrzip completion script.
-* Updated makefile to more portable posix version.
-* Hopefully fixed compilation on FreeBSD not supporting memopen.
-* Added lots more debugging information about offsets in max verbose mode.
-* Removed error and warning messages that are spammy or harmless.
-
-MAY 2011, version 0.606 Con Kolivas
-* lrzuntar broke as lrzip doesn't automatically use stdout now, so use lrzcat
-in the lrztar script instead.
-
-MAY 2011, version 0.605 Con Kolivas
-* Addition of explicit lrzcat symlink which automatically sets "-d -o -"
-* Revert automatic output to stdout due to addition of explicit lrzcat
-executable, and to be consistent with gzip.
-* Fix counter from being zeroed on every loop in rzip.c which was creating
-excess progress output.
-* Update GNU license.
-* Set aside at least page_size of zeroes when compressing a chunk, thus
-allowing files that are otherwise too small to compress with lrzip to be
-compressed. The zeroes will just be dropped on decompression.
-* Add explicit checks to cope with zero sized files as well.
-* Remove the arbitrary 128 byte minimum size restriction on archives now that
-any size file can be safely archived.
-* Set the eof flag when reading older file format files to not hang on zero
-byte sized archives.
-* Update the percentage counter when getting file info to not overflow with
-tiny files.
-* Set the maximum file mode to 0666 to avoid setting the executable bit when
-permissions are not carried over (such as from STDIO).
-
-APRIL 2011, version 0.604 Con Kolivas
-* Detach threads after creating them on the compression side. Not joining them
-meant that compressing massive files requiring hundreds of threads would
-eventually hit the resource limit of number of threads created even though
-the threads themselves would exit.
-
-APRIL 2011, version 0.603 Con Kolivas, George Makrydakis, Jari Aalto.
-* lseek in stream.c wasn't being compiled to the lseek64 variant on Apple
-due to missing includes, breaking >2GB files. Added includes.
-* Detect when stdout is being redirected and automatically direct output to
-stdout unless a filename is specified.
-* Update lrztar to properly support -S -O and -o, and use new syntax not
-requiring '-o -' for stdout.
-* Update lrzip.conf to support encryption.
-* Do a sanity check to ensure lrzip is not attempting to work on a directory.
-* Typo fixes.
-
-APRIL 2011, version 0.602 Con Kolivas
-* Fixed the symlinks breaking package generation.
-* Made maximum chunk allocable on 32bits 2/3 of a GB again limiting total ram
-used to 2GB, regardless of how much crazy ram people have put on such systems
-since the rest is unusable.
-* Now builds on Cygwin courtesy of Тулебаев Салават.
-* Check for podman on ./configure.
-* Disable md5 generation and checking on Apple till it's fixed.
-* Display full version when -V is invoked.
-* Don't restrict file permissions from unknown source, respecting umode
-instead.
-
-MARCH 2011, version 0.601 Con Kolivas
-* Fixed the symlinks for lrzuntar, lrunzip not being installed.
-* Fixed the pod-based manpages not being installed.
-* Clearly demonstrate that ASM is not supported on 64bits.
-
-MARCH 2011, version 0.600 Con Kolivas
-* Massive rewrite with new file format to accomodate new features.
-* Allocate temporary buffers of safely sized ram that can act as temporary
-storage for performing de/compression to/from STDIN/STDOUT without requiring
-temporary physical files. Files compressed on machines with much larger ram
-being decompressed on smaller ram machines may still require temporary files,
-but as much as possible is done using in-ram decompression, and minimally
-sized temporary files. Information displayed is more verbose and accurate in
-STDIN/STDOUT mode.
-* The temporary buffers created for decompressing to STDOUT are also used
-for decompressing regular files now avoiding multiple write/reads when
-re-constructing the file on decompression. This can dramatically speed up
-the rzip phase of decompression on complicated files with many small matches,
-and always makes testing faster.
-* Compress block headers as well now since we know how many bytes can be used
-to describe the length of the block, decreasing overall file size.
-* Store the rzip chunk size per chunk to make it possible to check total file
-size by summating each rzip chunk size when it's not known till the end (as
-happens when compressing from STDIN).
-* Implement password protected encryption. Import the polarssl code for
-sha512 and aes128 routines. Read password without echoing to screen by
-disabling echo via terminfo. Take the password then multiply hash it
-according to the date it was generated. Inrease the number of hashes according
-to Moore's law so it always takes approximately 1 second per password on the
-most modern hardware when first encrypted. Hash the password against 8 bytes
-of salt which is a combination of the 2 byte encoded loop counter (for how 
-many times to hash the password) and 6 random bytes. Take random from
-/dev/urandom if it's available and fall back to random() if not. Encrypt each
-block of compressed data with 8 extra bytes of random salt. Once the headers
-are written, go back and encrypt the headers as well. Then encrypt the md5
-hash value as well. Anything beyond the initial lrzip magic header should
-apppear as random data and no two successive encryptions of the same data with
-the same password should generate the same data.
-* New build system should be more robust and portable.
-* Abstract out functions better into separate files and headers, and remove
-all use of global variables. This will make the generation of an lrzip
-library possible in the future.
-* Prevent testmalloc from coming up with a negative number when determining
-how big a block of memory to allocate by decreasing the number of threads to
-be used and then aborting to a minimum value should it still be too much
-apparent ram.
-* Big Endian fixes.
-* Change lrztar and lrzuntar to not use temporary files.
-* Countless other fixes, documentation and cleanups.
-
-MARCH 2011, version 0.571 Con Kolivas
-* Only retry mmaping if it's a memory error, otherwise it may give spurious
-errors.
-* Check for free space before compression/decompression and abort if there
-is inadequate free space if the -f option is not passed.
-* Fix the wrong check in rzip.c which was rounding down the page size and
-making for one extra small chunk at the end.
-* Check the correct stdout when refusing to pipe to a terminal.
-* Fix windows EOL on lzma.txt.
-* Ignore what stdout is going to when testing from stdin.
-* More verbose summary after we know whether we have stdin/out to more
-accurately reflect the window that will be used.
-* Updated gitignore
-* Unlink temporary files immediately to avoid files lying around.
-* Check free space AFTER reading magic, and not when decompressing to stdout.
-* Don't dump output to stdout when just testing a file.
-* Dump the temporary file generated on emulating stdout on decompression after
-every chunk is decompressed instead of after the whole file is decompressed.
-* Don't attempt to file check decompression when it's not meaningful (test
-only or to stdout).
-* Clean up and simplify the times displayed component. It could potentially
-show thousands of seconds.
-* Remove open coded strerror messages when they're to be printed by fatal().
-
-MARCH 2011, Michael Blumenkrantz
-* Updated autotools/conf build system.
-
-FEBRUARY 2011, version 0.570 Con Kolivas
-* Change the lzo testing to a bool on/off instead of taking a parameter.
-* Clean up the messy help output.
-* Refuse to read from or write to terminal in stdin/stdout mode.
-* Delete temporary files generated when testing from stdin.
-* Don't call perror on failures that aren't system related.
-* Improve visual output.
-* Significant multi-threading speed-ups - as the spawning of threads is
-partially serialised, use one more thread than CPUs.
-* Don't split up the chunks into multiple threads when no back end
-compression is being used.
-* Take into account the precise amount of overhead that the memory hungry
-back-end compression modes lzma and zpaq use when determining how much ram to
-allocate, and limit total back-end compression ram to 1/3.
-* Deprecate the -M option as it is no longer of any significant utility
-compared to the -U option.
-* Strip symbols by default in compilation and add -W -Wall to CXXFLAGS to be
-consistent with CFLAGS.
-* Update benchmarks.
-* Add verbose and max verbose modes for -i information giving a breakdown of
-each rzip chunk and compressed block.
-* Modify purpose of -T threshold option to mean Disable LZO pre-compression
- testing. Threshold level test deprecated.
-* Make failure to allocate a buffer in the compress_ functions put out an
-error message always rather than just in max_verbose mode.
-
-FEBRUARY 2011, Peter Hyman
-* Minor updates to man pages, lrzip.conf example file.
-* Update main help screen to include environment settings.
-* Update to respect $TMP environment variable for TMP files.
-* Updated control structure to include tmpdir pointer.
-* Update lrzip.conf parser to respect -U, -H, -T, k, and -c options.
-* Update lrzip.conf example to include new parameters.
-* Reorder main Switch loop in main.c for readability.
-* Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
-* Corrects heuristic computation in rzip.c which would override
-  UNLIMITED if control.window set
-* Show heuristically computed control.window when computed.
-* Remove display compression level from control.window verbose output.
-* Update print_verbose format for Testing for incompressible data in stream.c
-  to omit extra \n.
-
-FEBRUARY 2011, version 0.560 Con Kolivas
-* Rewrote the locking implementation to use pthread_mutexes instead of
-semaphores. This makes multi-threading work on OSX.
-* Re-use a single malloc in unzip_match instead of multiple malloc/frees.
-* Imported the md5 code from coreutils and modified them to be used by lrzip.
-* Implemented md5 hash checking on compression and decompression by using
-buffers to pass to the existing crc checking and re-using them for the md5
-checking.
-* Sped up the hash checking on compression by not reading the whole chunk at
-once to do the hash check, and instead read one paged size chunk.
-* Store the md5 hash value at the end of the archive, maintaining backward
-compatibility with existing 0.5 versions of lrzip, and place a flag in the
-header denoting its existence.
-* Use the header flag for md5's existence to determine what hash check to
-perform on decompression.
-* Add hash value display to maximum verbose output and in information mode.
-* Add -H option to enable hash value displaying without verbose mode. 
-* Display what hash checking will be used on decompression in verbose mode and
-in information displayed with -i.
-* Add -c option to perform a hash check on the file generated on decompression,
-comparing it to that stored in the lrzip archive, or that generated on
-decompression on archives that do not have it stored.
-* Delete broken or damaged files generated on interruption or when they fail
-integrity testing by default.
-* Implement the -k option to keep broken or damaged files.
-
-FEBRUARY 2011, version 0.552 Con Kolivas
-* Fix incompressible blocks causing failure with bzip2 and gzip.
-* Freebsd doesn't do mremap, use the fake one.
-* Update copyright notices.
-* Flush writes to disk before allocating ram.
-* Relax memory allocation when no back end compression will be used.
-* Clean up unnecessary semaphore usage in decompression.
-* Fix minor typos.
-* Convert mmaps to malloc in runzip as they may silently fail if not a
-multiple of page size, thereby leading to silent corruption on decompression.
-* Return the correct length by unzip_literal.
-* Use wrappers to serialise threads on OSX and bypass use of unnamed semaphores
-which aren't supported.
-
-DECEMBER 2010, version 0.551 Con Kolivas
-* Fix stdin failing due to getting sizes all wrong.
-* Fix stdin compression values not being shown at end.
-* Fix inappropriate failure when lzma doesn't compress block.
-
-DECEMBER 2010, version 0.550 Con Kolivas
-* Move the threading on compression to higher up in the code, allowing the next
-stream to start using compression threads before the previous stream has
-finished. This speeds up compression on files that take more than one pass to
-compress.
-* Limit the number of threads decompressing stream 0 to just 1 since it's always
-followed by stream 1 chunks, and it may lead to failure to decompress due to
-running out of memory by running too many threads. 
-* Default compression level and window size on lzma is set to 7 which is the
-highest it goes. Scale the 9 lrzip levels into 7, thus making the default lzma
-level 5 which uses a lot less memory and is substantially faster at the cost of
-some compression.
-* Rationalise the memory testing now that the default lzma settings use a lot
-less ram by default, and make all systems use no more than 1/3 ram in one mmap.
-This allows larger windows to be used by 32 bit at last without memory
-allocation errors.
-* Revert "Make threads spawn at regular intervals along chunk size thus speeding
-up compression" as it actually slowed it down instead of speeding it up.
-* Cope with compression/decompression threads failing by waiting till the
-previous thread has finished its work, thus serialising the work and using less
-ram, making success more likely.
-* Fix some dodgy callocs which weren't really allocating enough ram.
-* Destroy semaphores used in stream_in on closing the stream.
-* Minor output improvements.
-
-DECEMBER 2010, version 0.544 Con Kolivas
-* Make multiple stream 0 entry decompression more robust by creating separate
-thread groups for stream 0 and stream 1.
-* uclibc fails to return valid values for memory size with sysconf so create
-a workaround.
-* 32 bits has memory allocation errors on large files so be ruthless with
-limiting window sizes and use more sliding mmap.
-* Split up compression streams according to proportion of hash search thus
-evenly distributing work to threads based on raw data length rather than
-rzip pre-processed data. This makes for more CPU usage on compression and
-smoother write out on decompression with small speed ups.
-* Remove unnecessary argument passing.
-* Fix -U causing a floating point error with ultra small sized files.
-* 32 bits still sometimes fails the lzma compression stage due to internal
-memory allocation issues, so use bzip2 as a fallback on blocks that fail to
-compress.
-* Minor cleanups.
-
-NOVEMBER 2010, version 0.543 Con Kolivas
-* Fix for when there are multiple stream 0 entries per chunk, which would
-cause a failure on decompression.
-* Fix lzma windows being set far too small by mistake at all times.
-* Decrease nice level of main process compared to back end threads as it tends
-to be the rate limiting step.
-
-NOVEMBER 2010, version 0.542 Con Kolivas
-* Choose sane defaults for memory usage since linux ludicriously overcommits.
-* Use sliding mmap for any compression windows greater than 2/3 ram.
-* Consolidate and simplify testing of allocatable ram.
-* Minor tweaks to output.
-* Round up the size of the high buffer in sliding mmap to one page.
-* Squeeze a little more out of 32 bit compression windows.
-* Fix sliding mmap from not working on 2nd pass onwards.
-
-NOVEMBER 2010, version 0.541 Con Kolivas
-* Fix wrong number of passes reported.
-* Re-fix the off-by-one that wasn't off-by-one.
-* Limit lzma compression windows to 300MB as per reports of failures with larger
-windows.
-
-NOVEMBER 2010, version 0.540 Con Kolivas
-* Massive rewrite of backend decompression phse, implementing multithreading.
-This is done by taking each stream of data on read in into separate buffers for
-up to as many threads as CPUs. As each thread's data becomes available, feed it
-into runzip once it is requests more of the stream. Provided there are enough
-chunks in the originally compressed data, this provides a massive speedup
-potentially proportional to the number of CPUs. The slower the backend
-compression, the better the speed up (i.e. zpaq is the best sped up).
-* Fix the output of zpaq compress and decompress from trampling on itself and
-racing and consuming a lot of CPU time printing to the console.
-* When limiting cwindow to 6 on 32 bits, ensure that control.window is also set.
-* When testing for the maximum size of testmalloc, the multiple used was out by
-one, so increase it.
-* Minor output tweaks.
-* Build warning fixes.
-* Updated benchmarks.
-
-NOVEMBER 2010, version 0.530 Con Kolivas
-* Massive rewrite of backend compression phase. Now the stream is split up
-into as many chunks as there are CPUs, of at least 10MB in size, that are
-still mallocable. Once the stream has reached a chunk of this size, its buffer
-is handed to a new backend compression thread which works while the rzip stream
-continues processing. This has the effect of parallelising workloads almost
-linearly up to the number of CPUs on the slower compression backends. ZPAQ,
-in particular, is effectively 4x faster on quad core now. Decompression is
-unchanged.
-* Added the -p option to allow the number of processors to be specified to
-override the detected number.
-* Changed the default level back to 7 as 9 wasn't offering significanly more
-compression but was adding time.
-* Increased the size of all the buffers to other backends now as well, since
-each block adds overhead with its header.
-* Numerous alterations to screen output to cope with new threaded compression
-phase.
-* Deprecated the -P option since not setting the file permissions only
-generates a warning now, not a failure.
-* Updated docs and benchmarks.
-
-NOVEMBER 2010, version 0.520 Con Kolivas
-* Distros don't like 3 point version numbering so just repackaged as 0.520.
-
-NOVEMBER 2010, version 0.5.2 Con Kolivas
-* Fixed the Darwin build, again.
-* Fixed cases of extreme ram usage on 32 bit failing by limiting zpaq to 600MB
-windows as well.
-* Check page size if we can instead of assuming it's always 4k.
-* Improve the progress output.
-* Change failure to chmod and failure to set nice level to warnings only.
-* Standardise what's a stderr message and what's output.
-
-NOVEMBER 2010, version 0.5.1 Con Kolivas
-* Fix Darwin build - Darwin doesn't support mremap so introduce a fake wrapper
-for it.
-* Fix the memopen routines, a wrongly implemented wrapper for Darwin equivalents
-was also using the faked versions on all builds.
-* Fix dodgy ordered includes.
-* Clean up excessive use of #ifdefs
-* Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of
-accessing the buffer directly, thus allowing us to have window sizes larger than
-available ram. This is implemented through the use of a "sliding mmap"
-implementation. Sliding mmap uses two mmapped buffers, one large one as
-previously, and one smaller one. When an attempt is made to read beyond the end
-of the large buffer, the small buffer is remapped to the file area that's being
-accessed, while the larger one is remapped as the search progresses along the
-file. While this implementation is potentially much slower than direct mmapping,
-it allows us to implement unlimited sized compression windows.
-* Implement the -U option with unlimited sized compression windows.
-* Rework the selection of compression windows. Instead of trying to guess how
-much ram the machine might be able to access, we try to safely buffer as much
-ram as we can, and then use that to determine the file buffer size. Do not
-choose an arbitrary upper window limit unless -w is specified.
-* Rework the -M option to try to buffer the entire file, reducing the buffer
-size until we succeed.
-* Align buffer sizes to page size.
-* Clean up lots of unneeded variables.
-* Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
-and the compression backends.
-* More error handling.
-* Change -L to affect rzip compression level directly as well as backend
-compression level and use 9 by default now.
-* Fix file size reporting on compressed files generated from stdin.
-* More cleanups of information output and more information.
-* Add chunk percentage to progress update.
-* Reinstated the 2GB buffer limit on 32 bit machines during compression, though
-the -U mode can work around it now.
-* Code micro-optimisations.
-* Use 3 point release numbering in case one minor version has many subversions.
-* Numerous minor cleanups and tidying.
-* Updated docs, manpages, and benchmarks.
-
-NOVEMBER 2010, version 0.5 Con Kolivas
-* Changed offset encoding in rzip stage to use variable byte width offsets
-instead of 64 bits wide. Makes for better compression and slightly faster.
-* Write the byte width into the file before each block.
-* Shrunk match lengths to maximum of 16 bits again as per original rzip as the
-larger offsets did not achieve greater compression and made final size larger.
-* New file format not backwards compatible due to variable byte widths.
-* Rewrote memory initialisation to have a pre-allocation stage to try and
-find the maximum memory usable and defragment ram.
-* Use reduced window size if allocating memory fails at higher size.
-* Change use of malloc to mmap to make it possible to address up to 44 bit
-sized offsets even on 32 bit machines on decompression. Still unable to use
-greater than 2GB windows on 32 bit machines and unsure if this is fixable.
-* Reworked the STDIN code to use an anonymous mmap and read in stdin into this
-to make it possible to compress from STDIN without the need for temporary
-files. As the file size is not known in advance, memory allocation is set to
-large and byte width to equivalent size.
-* Reallocation of ram where possible to minimise risk of running out of memory
-in the middle of a compression phase, and flushing to disk to empty dirty ram
-for the same reason.
-* More robust fatal warnings.
-* Numerous cleanups and tidying of code and addition of comments.
-* Updated documentation to reflect changes.
-
-OCTOBER 2010, version 0.47, Con Kolivas
-* Fix the symlinking problem when DESTDIR is in use reported by a billion
-people.
-
-MAY 2010, version 0.46, Con Kolivas, Ed Avis.
-* Suppress final [OK] message with -q flag EA
-* Handle mkstemp() errors correctly EA
-* Add lrzuntar manpage
-* Update manpages
-
-APRIL 2010, version 0.45, Con Kolivas, Jon Tibble, George Makrydakis
-* Fixes the nasm program test (AC_CHECK_PROG doesn't overwrite a
-variable that is already set so do it manually) JT
-* Fix compiler flags as not all compilers accept -Wall -W (cc on
-Solaris/OpenSolaris) JT
-* Fix lrztar to not try to compress files already with the .lrz extension GM
-* Fix lrztar to decompress files where the pathname is ../* GM
-* Add lrzuntar symlink to call lrztar -d
-
-
-MAR 2010, version 0.45, Con Kolivas, Jari Aalto
-* Fixed reported window size
-* Fixed 32bit windows being attempted to be larger than contiguous amounts
-by taking into account VM kernel/userspace split of 896MB.
-* Minor code cleanups
-* Added lrztar and lrunzip docs
-* Fix minor typos
-* Added distclean and maintainer-clean make targets
-
-
-DEC 2009, version 0.44, Con Kolivas, George Makrydakis
-* Added lrztar wrapper to manage whole directories.
-* Added -i option to provide information about a compressed file.
-* Fixed "nan" showing as Compression speed on very small files.
-* Fixed build for old bz library.
-* Avoid overwriting output file if input doesn't exist.
-* Implement signal handler to delete temporary files.
-
-
-DEC 2009, version 0.43, Con Kolivas, Jukka Laurila
-* Darwin support thanks to Jukka Laurila.
-* Finally added stdin/stdout support due to popular demand. This is done
-by basically using temporary files so is a low performance way of using
-lrzip.
-* Added test function. This just uses a temporary file during decompression.
-* Config files should now accept zpaq options.
-* Minor code style cleanups.
-* Updated benchmarks in docs.
-* Add a warning when attempting to decompress a file from a newer lrzip
-version.
-
-
-NOV 2009, version 0.42, Con Kolivas
-* Changed progress update to show which of 2 chunks are being compressed
-in zpaq.
-* Fixed progress update in ZPAQ to not update with each byte which was
-wasting heaps of CPU time.
-
-
-NOV 2009, version 0.41, Con Kolivas
-* Added zpaq compression backend for extremely good but extremely slow
-compression (incompatible with previous versions if used).
-* Limited chunk size passed to LZMA to 4GB to avoid library overflows.
-* Minor changes to the formatting output
-* Changed lower limit of -T threshhold to 0 to allow disabling it.
-* Added lzo_compresses check into zpaq and bzip2 as well since they're
-slow.
-
-
-NOV 2009, version 0.40, Con Kolivas
-* Massive core code rewrite.
-* All code moved to be 64bit based for compression block addressing and length
-allowing compression windows to be limited by ram only.
-* 64bit userspace should now have no restriction on compression window size,
-32bit is still limited to 2GB windows due to userspace limitations.
-* New file format using the new addressing and data types, incompatible with
-versions prior to 0.40.
-* Support for reading and decompressing older formats.
-* Minor speedups in read/write routines.
-* Countless minor code fixes throughout.
-* Code style cleanups and consistency changes in core code.
-* Configure script improvements.
-
-
-NOV 2009, version 0.31, Con Kolivas
-* Updated to be in sync with lzma SDK 9.07beta.
-* Cleanups and fixes of the configure scripts to use the correct package version
-name.
-* Massive fixes to the memory management code fixing lots of 32bit overflow
-errors. The window size limit is now 2GB on both 32bit and 64bit. While it
-appears to be smaller than the old windows, only 900MB was being used on .30
-even though it claimed to use more. This can cause huge improvements in the
-compression of very large files.
-* The offset when mmap()ing was not being set to a multiple of page size so
-it would fail if the window size was not a multiple of it.
-* Flushing of data to disk between compression windows was implemented to
-minimise disk thrashing of read vs write.
-
-
-NOV 2009, version 0.30, Con Kolivas
-* Numerous bugfixes to try and make the most of 64bit environments with huge
-memory and to barf less on 32bit environments.
-* Executable stacks were fixed.
-* Probably other weird and wonderful bugs have been introduced.
-* -P option to not set permissions on output files allowing you to write to
-braindead filesystems (eg fat32).
-
-
-JAN 2009, version 0.24, Peter Hyman, pete@peterhyman.com
-Happy New Year!
-* Upgrade LZMA SDK to 4.63. Use new C Wrapper. Invalidates
-  LZMA archives created earlier due to new Magic property
-  bytes.
-* New LZMA logic will automatically determine allow LZMA
-  code to determine optimal lc, lp, pb, fb, and dictionary
-  size settings. stream.c will only pass level and thread
-  information. Compress function will return encoded 5 byte
-  data with compression settings. This will be stored in lrz
-  file header.
-* add error messages during LZMA compression. There are some
-  edge cases where LZMA cannot allocate memory. These errors
-  are reported and the user will be advised to use a lower
-  compression window setting.
-* type changes in rzip_fd function for correctness.
-* remove function *Realloc() since it was never used. Cleaned
-  in rzip.h and util.c.
-* apply munmap prior to closing and compressing stream in
-  function rzip_chunk in rzip.c.
-* add realloc function in close_stream_out in stream.c
-  to reclaim some ram and try and allieviate out of memory
-  conditions in LZMA compression.
-* remove file acconfig.h and include DEFINE in configure.in.
-* add lrzip.conf capability.
-* add timer for compression including elapsed time and eta.
-* add compression and decompression MB/s calculation.
-* Updated WHATS-NEW, TODO and created BUGS file.
-* Updated lrzip.1 manpage and created lrzip.conf.5 manpage.
-* Added lrzip.conf.example file in doc directory.
-
-MAR 2008, Con Kolivas, kernel@kolivas.org
-* Numerous changes all over to place restrictions on window
-size to work with 32 bit limitations.
-* Various bugfixes with respect to detecting buffer sizes and
-likelihood of compressibility.
-* Fixed the inappropriate straight copying uncompressed data for
-files larger than 4GB.
-* Re-initiated the 10MB window limits for non-lzma compression.
-I was unable to reproduce any file size savings.
-* Allow compression windows larger than ramsize if people really
-really want them.
-* Decrease thresholds for the test function to a minimum of 5%
-compressibility since the hanging in lzma compression bug has been
-fixed.
-
-JAN 2008, version 0.22, Peter Hyman, pete@peterhyman.com
-* version update
-lzma/LZMALib.cpp
-  Thanks to Lasse Collin for debugging the problem LZMA
-  had with hanging on uncompressable files.
-  Update for control parameters to both compress and
-  decompress functions.
-Makefile.in
-* use of @top_srcdir@ (Lasse Collin). Also moved away
-  more cruft.
-main.c stream.c.rzip.h LZMALib.cpp lzmalib.h
-* addition of three new control structure members.
-  control.lc -- literal context bits
-  control.lp -- literal post state bits
-  control.pb -- post state bits
-  These are needed to ensure decompression will work.
-  These will now be stored along with control.compression_level
-  in the lrz file beginning at offset 0x16 for three bytes.
-  These will be passed to the functions lzma_compresses and
-  lzma_uncompress. Currently, only compression level is
-  needed or used, but the others are stored for possible future
-  use.
-  See magic file for more information.
-stream.c
-* Change to lzo_compresses function that will reject a chunk
-  without testing it if the size of the chunk is greater
-  than the compression window * threshold. This is to avoid
-  a low probability that lzma would still be passed a chunk
-  that contains uncompressible data or barely compressible
-  data. If after rzip hashing the chunk size is still close
-  to the window size, there is hardly anything worth
-  compressing. While there is no reason lzma cannot get the
-  chunk, this will save a lot of time.
-magic.headers.txt
-* updated file to show new layout that includes lzma
-  parameters.
-README-NOT-BACKWARD-COMPATIBLE
-* added warning about using lrzip-0.22 with earlier versions.
-WHATS-NEW
-* highlight of new features.
-
-DEC 2007, version 0.21. Peter Hyman, pete@peterhyman.com
-* version update.
-* Modified to use Assembler routines from lzma SDK for CRC
-  computation when hashing streams in rzip.c and runzip.c.
-  Added files 7zCrcT8.c and 7zCrcT8u.s to lzma tree.
-  Cleaned up source tree. Moved unused files out of the way.
-  Moved non-core docs to doc directory
-configure.in
-* correct AC_INIT to set program variables.
-* modified to add check for nasm assembler.
-* modified syntax of test for errno in error.h to use
-  echo $ECHO_N/$ECHO_C instead of $ac_n/$ac_c which
-  was incorrect.
-Makefile.in, lzma/Makefile
-* modified to add compile instructions for 7zCrcT8.c
-  and 7zCrcT8U.s and Assembler. Cleaned up to remove
-  targets that don't exist or sources that don't
-  exist.
-  Modified to properly set directories. Added doc install.
-  Add link command to symlink lrunzip to lrzip.
-*main.c
-  Add CrcGenerateTable() function to init CRC tables.
-  This is needed for all crc routines including those
-  in MatchFinderMT.
-rzip.c and runzip.c
-* Updated source to change call to crc32_buffer to call
-  CrcUpdate in the assembler code. Changed parameter order
-  to conform.
-stream.c
-* Removed 10MB limit on streams for bzip, gzip, and lzo.
-  This, to improve effeciency of long range analysis. For
-  some files, this could improve results.
-Current-Benchmarks.txt
-* Added file to keep benchmarks current to version.
-  (probably need to update README too).
-README.Assembler
-* Explain how to remove default compile of Assembler
-  modules.
-config.sub config.guess
-* added files for system detection.
-
-DEC 2007, version 0.20. Peter Hyman, pete@peterhyman.com
-
-* Updated to LZMA SDK 4.57.
-* Updated to p7zip POSIX version. (www.p7zip.org)
-* Added multi-threading support (up to 2x speed with LZMA).
-* Edited LZMADecompress.cpp for backward compatibility
-  with decompress function. Needed SetPropertiesRaw function.
-* Repopulated source tree for distribution.
-* Updated Makefile.in to reflect new source files.
-  Updated to include command to link lrunzip to lrzip because
-  lrzip will test if lrunzip was used on command line.
-* Updated Makefile.in for new compile time and linking options.
-* Updated LZMALibs.cpp to include new property members for
-  LZMAEncoders as well as changed default dictionaries to
-  level+16. This would make the default compression level
-  of 7 translate to a dictionary number of 23.
-* Added output to show Nice Level when verbose mode set
-  Initial add of support for zlib which seems to give quite
-  excellent performance.
-* configure.in added AC_CHECK for libz and libm.
-  Added AC_PROG_LN_S for Makefile symlink section.
-* lrzip.1 updated man page for -g option
-* main.c added option test for gzip
-  Added sysconf(_SC_NPROCESSORS_CONF) for CPU detection
-  for threading.
-  Updated verbose output to show whether or not
-  Threading will be used.
-  Added Timer for each file compressed.
-* rzip.h added flags for GZIP compression.
-  Added control member for threads. Arg passed to
-  lzma_conpress.
-* stream.c update to accomodate gzip compress and decompress
-  functions. Cleaned up file by rearranging functions into
-  groups.
-  Removed include of lzmalib.h since it was causing a
-  compile time warning with zlib.h. Prototyped functions
-  manually.
-  Cleanup output from lzo_compresses function so that
-  unnecessary linefeeds are eliminated.
-  lzma_compress function call now uses threads as argument.
-* Added README.benchmarks file to explain a method of
-  comparing results between different methods.
-* LZMALib.cpp, lzmalib.h. Adjust function lzma_compress
-  prototype and function to include new argument threads.
-  This parameter is now placed in properties.
-* lzma/Makefile. Updated to reflect new API library.
-  Updated to include Threading option.
-
-DEC 2007, version 0.19. Con Kolivas.
-* Added nice support, defaulting to nice 19.
-
-DEC 2007, version 0.19. Peter Hyman, pete@peterhyman.com
-
-* Major goal was to stop LZMA from hanging on some files.
-  Accomplished this with a threhold setting that is used by
-  the lzo_compresses function to better analyze chunk data.
-  Threshold makes it less likely that uncompressible data
-  will be passed to the LZMA compressor.
-
-main.c
-* Added Threshold option 1-10 to control LZMA compression attempt.
-  Default value=2. This means that anything over 10% compression
-  as reported by lzo_compresses will return a true value to
-  the LZMA compression function.
-* Added verbosity option and more verbosity option (-v[v]).
-* Added -O option to specify output directory.
-* Updated compress_file and decompress_file functions to handle.
-  output directories and better handle multi files and filename
-  extensions. Optimized some string handling routines.
-  Improved flexibility in determining location of output files
-  when using -O. Added fflush(stdout) to improve printf reliability.
-* decompress_file will accept any filename and will automatically
-  append .lrz if not present. Won't automatically fail.
-* Added logic to protect against conflicting options such as
-  -q and -v, -o and -O.
-* Added printout to screen of options selected. Will display
-  only when -v or -vv used.
-* Adjusted several printf statements to avoid compiler
-  warnings (use %ll for long long int types).
-
-runzip.c
-* Added decompression progress indicator.
-  Will show percent decompressed along with bytes decompressed
-  and total to be decompressed. Will show if -q option NOT used.
-
-rzip.h
-* Version incremented to 0.19.
-* Added flag DEFINESs for verbosity and more verbosity.
-* Updated control struct to include output directory and
-  threshold value. Removed verbosity member.
-
-rzip.c
-* Minor changes to handle display when verbosity set. Changed
-  number format in some printf statements to properly handle
-  unsigned data.
-
-stream.c
-* major overhaul of lzo_compresses function to use a threshold
-  value when testing a data chunk to see if it is suitable for
-  LZMA compression. Optimized test loop to improve performance
-  and reduce number of passes. Improved output reporting depending
-  on verbosity setting.
-* Added print controls for verbosity option.
-* Corrected if statements that tested for error condition of
-  some lzo functions that only return a true value regardless.
-
-lrzip.1
-* updated man page to show new options and explain -T threshold.
-
-README
-* updated README to explain -T threshold option.
-
-README.lzo_compresses.test.txt
-* Added this file to help explain the theory behind the rewrite
-  of the lzo_compresses function and how to use the -T option.
-
-TODO
-* wish list and future enhancements.
-
-ChangeLog
-* added file.
+Check git for changelog: https://github.com/ckolivas/lrzip/commits/master

Dockerfile

@@ -0,0 +1,14 @@
+FROM alpine as builder
+
+RUN apk add --update git autoconf automake libtool gcc musl-dev zlib-dev bzip2-dev lzo-dev coreutils make g++ lz4-dev && \
+    git clone https://github.com/ckolivas/lrzip.git && \
+    cd /lrzip && ./autogen.sh && ./configure && make -j `nproc` && make install
+
+FROM alpine
+
+RUN apk add --update --no-cache lzo libbz2 libstdc++ lz4-dev && \
+    rm -rf /tmp/* /var/tmp/*
+
+COPY --from=builder /usr/local/bin/lrzip /usr/local/bin/lrzip
+
+CMD ["/bin/sh"]

Lrzip.h

@@ -1,765 +0,0 @@
-/*
-   Copyright (C) 2006-2011 Con Kolivas
-   Copyright (C) 2011 Peter Hyman
-   Copyright (C) 1998-2003 Andrew Tridgell
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU General Public License as published by
-   the Free Software Foundation; either version 2 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU General Public License
-   along with this program. If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifndef LIBLRZIP_H
-#define LIBLRZIP_H
-
-#include <stdbool.h>
-#include <stdio.h>
-#ifdef _WIN32
-# include <stddef.h>
-#else
-# include <inttypes.h>
-#endif
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/**
- @brief LRZIP library
-
- @mainpage lrzip
- @version 1.0
- @date 2011
-
- @section intro What is LRZIP?
- 
- LRZIP is a compression program optimised for large files. The larger the file
- and the more memory you have, the better the compression advantage this will
- provide, especially once the files are larger than 100MB. The advantage can
- be chosen to be either size (much smaller than bzip2) or speed (much faster
- than bzip2).
- * @link Lrzip.h LRZIP API @endlink
- */
-
-/** @file Lrzip.h */
-
-/**
- * @typedef Lrzip
- * @brief The overall struct for managing all operations
- */
-typedef struct Lrzip Lrzip;
-
-/**
- * @typedef Lrzip_Log_Level
- * @brief The amount of information to display using logging functions
- * This enum is used when setting or getting the log level of an #Lrzip
- * struct. It determines how much information is shown about the current operation,
- * either in stdout/stderr or using logging callbacks.
- * @see lrzip_log_level_set()
- * @see lrzip_log_level_get()
- */
-typedef enum {
-	LRZIP_LOG_LEVEL_ERROR = 0, /**< Only display errors */
-	LRZIP_LOG_LEVEL_INFO, /**< Display information and errors */
-	LRZIP_LOG_LEVEL_PROGRESS, /**< Display progress updates, information, and errors */
-	LRZIP_LOG_LEVEL_VERBOSE, /**< Display verbose progress updates, information, and errors */
-	LRZIP_LOG_LEVEL_DEBUG /**< Display all possible information */
-} Lrzip_Log_Level;
-
-/**
- * @typedef Lrzip_Mode
- * @brief The mode of operation for an #Lrzip struct
- * This enum is used when setting or getting the operation mode of an #Lrzip
- * struct. It determines what will happen when lrzip_run() is called.
- * @see lrzip_mode_set()
- * @see lrzip_mode_get()
- */
-typedef enum {
-	LRZIP_MODE_NONE = 0, /**< No operation set */
-	LRZIP_MODE_INFO, /**< Retrieve info about an archive */
-	LRZIP_MODE_TEST, /**< Test an archive's integrity */
-	LRZIP_MODE_DECOMPRESS, /**< Decompress an archive */
-	LRZIP_MODE_COMPRESS_NONE, /**< RZIP preprocess only */
-	LRZIP_MODE_COMPRESS_LZO, /**< Use LZO compression */
-	LRZIP_MODE_COMPRESS_ZLIB, /**< Use ZLIB (GZIP) compression */
-	LRZIP_MODE_COMPRESS_BZIP2, /**< Use BZIP2 compression */
-	LRZIP_MODE_COMPRESS_LZMA, /**< Use LZMA compression */
-	LRZIP_MODE_COMPRESS_ZPAQ /**< Use ZPAQ compression */
-} Lrzip_Mode;
-
-/**
- * @typedef Lrzip_Flag
- * @brief The extra params for an #Lrzip struct's operations
- * This enum is used when setting or getting the flags of an #Lrzip
- * struct. It determines some of the miscellaneous extra abilities of LRZIP.
- * @see lrzip_flags_set()
- * @see lrzip_flags_get()
- */
-typedef enum {
-	LRZIP_FLAG_REMOVE_SOURCE = (1 << 0), /**< Remove the input file after the operation completes */
-	LRZIP_FLAG_REMOVE_DESTINATION = (1 << 1), /**< Remove matching destination file if it exists */
-	LRZIP_FLAG_KEEP_BROKEN = (1 << 2), /**< Do not remove broken files */
-	LRZIP_FLAG_VERIFY = (1 << 3), /**< Only verify the archive, do not perform any compression/decompression */
-	LRZIP_FLAG_DISABLE_LZO_CHECK = (1 << 4), /**< Disable test to determine if LZO compression will be useful */
-	LRZIP_FLAG_UNLIMITED_RAM = (1 << 5), /**< Use unlimited ram window size for compression */
-	LRZIP_FLAG_ENCRYPT = (1 << 6) /**< Encrypt archive during compression; @see lrzip_pass_cb_set() */
-} Lrzip_Flag;
-
-/**
- * @typedef Lrzip_Info_Cb
- * @brief The callback to call when an operation's progress changes
- * @param data The data param passed in lrzip_info_cb_set()
- * @param pct The overall operation progress as a percent
- * @param chunk_pct The current chunk's operation progress as a percent
- */
-typedef void (*Lrzip_Info_Cb)(void *data, int pct, int chunk_pct);
-/**
- * @typedef Lrzip_Log_Cb
- * @brief The callback to call when a log message is to be shown
- * @param data The data param passed in lrzip_log_cb_set()
- * @param level The Lrzip_Log_Level of the message
- * @param line The line in LRZIP code where the message originated
- * @param file The file in LRZIP code where the message originated
- * @param format The printf-style format of the message
- * @param args The matching va_list for @p format
- */
-typedef void (*Lrzip_Log_Cb)(void *data, unsigned int level, unsigned int line, const char *file, const char *format, va_list args);
-/**
- * @typedef Lrzip_Password_Cb
- * @brief The callback to call for operations requiring a password
- * @param data The data param passed in lrzip_pass_cb_set()
- * @param buffer The pre-allocated buffer to write the password into
- * @param buf_size The size, in bytes, of @p buffer
- */
-typedef void (*Lrzip_Password_Cb)(void *data, char *buffer, size_t buf_size);
-
-/**
- * @brief Initialize liblrzip
- * This function must be called prior to running any other liblrzip
- * functions to initialize compression algorithms. It does not allocate.
- * @return true on success, false on failure
- */
-bool lrzip_init(void);
-
-/**
- * @brief Create a new #Lrzip struct
- * Use this function to allocate a new struct for immediate or later use,
- * optionally setting flags and changing modes at a later time.
- * @param mode The optional Lrzip_Mode to set, or LRZIP_MODE_NONE to allow
- * setting a mode later.
- * @return The new #Lrzip struct, or NULL on failure
- * @see lrzip_mode_set()
- */
-Lrzip *lrzip_new(Lrzip_Mode mode);
-
-/**
- * @brief Free an #Lrzip struct
- * Use this function to free all memory associated with an existing struct.
- * @param lr The struct to free
- */
-void lrzip_free(Lrzip *lr);
-
-/**
- * @brief Set up an #Lrzip struct using environment settings
- * Use this function to acquire and utilize settings already existing in
- * either environment variables or configuration files for LRZIP. For more detailed
- * information, see the LRZIP manual.
- * @param lr The struct to configure
- * @note This function cannot fail.
- */
-void lrzip_config_env(Lrzip *lr);
-
-/**
- * @brief Retrieve the operation mode of an #Lrzip struct
- * @param lr The struct to query
- * @return The Lrzip_Mode of @p lr, or LRZIP_MODE_NONE on failure
- */
-Lrzip_Mode lrzip_mode_get(Lrzip *lr);
-
-/**
- * @brief Set the operation mode of an #Lrzip struct
- * @param lr The struct to change the mode for
- * @param mode The Lrzip_Mode to set for @p lr
- * @return true on success, false on failure
- */
-bool lrzip_mode_set(Lrzip *lr, Lrzip_Mode mode);
-
-/**
- * @brief Set the compression level of an #Lrzip struct
- * @param lr The struct to change the compression level for
- * @param level The value, 1-9, to use as the compression level for operations with @p lr
- * @return true on success, false on failure
- * @note This function is only valid for compression operations
- */
-bool lrzip_compression_level_set(Lrzip *lr, unsigned int level);
-
-/**
- * @brief Get the compression level of an #Lrzip struct
- * @param lr The struct to get the compression level of
- * @return The value, 1-9, used as the compression level for operations with @p lr,
- * or 0 on failure
- * @note This function is only valid for compression operations
- */
-unsigned int lrzip_compression_level_get(Lrzip *lr);
-
-/**
- * @brief Set the operation specific parameters
- * @param lr The struct to set parameters for
- * @param flags A bitwise ORed set of Lrzip_Flags
- * @note This function does not perform any error checking. Any errors in flags
- * will be determined when lrzip_run() is called.
- */
-void lrzip_flags_set(Lrzip *lr, unsigned int flags);
-
-/**
- * @brief Get the operation specific parameters
- * @param lr The struct to get parameters of
- * @return A bitwise ORed set of Lrzip_Flags
- */
-unsigned int lrzip_flags_get(Lrzip *lr);
-
-/**
- * @brief Set the nice level for operations in a struct
- * @param lr The struct to set the nice level for
- * @param nice The value to use when nicing during operations
- */
-void lrzip_nice_set(Lrzip *lr, int nice);
-
-/**
- * @brief Get the nice level for operations in a struct
- * @param lr The struct to get the nice level of
- * @return The value to use when nicing during operations
- */
-int lrzip_nice_get(Lrzip *lr);
-
-/**
- * @brief Explicitly set the number of threads to use during operations
- * @param lr The struct to set the threads for
- * @param threads The number of threads to use for operations
- * @note LRZIP will automatically determine the optimal number of threads to use,
- * so this function should only be used to specify FEWER than optimal threads.
- */
-void lrzip_threads_set(Lrzip *lr, unsigned int threads);
-
-/**
- * @brief Get the number of threads used during operations
- * @param lr The struct to query
- * @return The number of threads to use for operations
- */
-unsigned int lrzip_threads_get(Lrzip *lr);
-
-/**
- * @brief Set the maximum compression window for operations
- * @param lr The struct to set the maximum compression window for
- * @param size The size (in hundreds of MB) to use for the maximum size of compression
- * chunks.
- * @note LRZIP will automatically determine the optimal maximum compression window to use,
- * so this function should only be used to specify a LOWER value.
- */
-void lrzip_compression_window_max_set(Lrzip *lr, int64_t size);
-
-/**
- * @brief Get the maximum compression window for operations
- * @param lr The struct to query
- * @return The size (in hundreds of MB) to use for the maximum size of compression
- * chunks.
- */
-int64_t lrzip_compression_window_max_get(Lrzip *lr);
-
-/**
- * @brief Return the size of the stream queue in a struct
- * This function returns the current count of streams added for processing
- * using lrzip_file_add. It always returns instantly.
- * @param lr The struct to query
- * @return The current number of streams in the queue
- */
-unsigned int lrzip_files_count(Lrzip *lr);
-
-/**
- * @brief Return the size of the file queue in a struct
- * This function returns the current count of files added for processing
- * using lrzip_filename_add. It always returns instantly.
- * @param lr The struct to query
- * @return The current number of files in the queue
- */
-unsigned int lrzip_filenames_count(Lrzip *lr);
-
-/**
- * @brief Return the array of the stream queue in a struct
- * This function returns the current queue of streams added for processing
- * using lrzip_file_add. It always returns instantly.
- * @param lr The struct to query
- * @return The current stream queue
- */
-FILE **lrzip_files_get(Lrzip *lr);
-
-/**
- * @brief Return the array of the filename queue in a struct
- * This function returns the current queue of files added for processing
- * using lrzip_filename_add. It always returns instantly.
- * @param lr The struct to query
- * @return The current filename queue
- */
-char **lrzip_filenames_get(Lrzip *lr);
-
-/**
- * @brief Add a stream (FILE) to the operation queue
- * This function adds a stream to the input queue. Each time lrzip_run()
- * is called, it will run the current operation (specified by the Lrzip_Mode)
- * on either a stream or file in the queue.
- * @param lr The struct
- * @param file The stream descriptor to queue
- * @return true on success, false on failure
- * @note The file queue will be fully processed prior to beginning processing
- * the stream queue.
- * @warning Any streams added to this queue MUST NOT be closed until they have
- * either been processed or removed from the queue!
- */
-bool lrzip_file_add(Lrzip *lr, FILE *file);
-
-/**
- * @brief Remove a stream from the operation queue
- * This function removes a previously added stream from the operation queue by
- * iterating through the queue and removing the stream if found.
- * @param lr The struct
- * @param file The stream to remove
- * @return true only on successful removal, else false
- */
-bool lrzip_file_del(Lrzip *lr, FILE *file);
-
-/**
- * @brief Pop the current head of the stream queue
- * This function is used to remove the current head of the stream queue. It can be called
- * immediately following any lrzip_run() stream operation to remove the just-processed stream. This
- * function modifies the stream queue array, reordering and updating the index count.
- * @param lr The struct to pop the stream queue of
- * @return The stream removed from the queue, or NULL on failure
- */
-FILE *lrzip_file_pop(Lrzip *lr);
-
-/**
- * @brief Clear the stream queue
- * This function is used to free and reset the stream queue. The streams
- * themselves are untouched.
- * @param lr The struct
- */
-void lrzip_files_clear(Lrzip *lr);
-
-/**
- * @brief Add a file to the operation queue
- * This function adds a file to the input queue. Each time lrzip_run()
- * is called, it will run the current operation (specified by the Lrzip_Mode)
- * on either a stream or file in the queue.
- * @param lr The struct
- * @param file The file (by absolute path) to queue
- * @return true on success, false on failure
- * @note The file queue will be fully processed prior to beginning processing
- * the stream queue.
- */
-bool lrzip_filename_add(Lrzip *lr, const char *file);
-
-/**
- * @brief Remove a filename from the operation queue
- * This function removes a previously added filename from the operation queue by
- * iterating through the queue and removing the filename if found.
- * @param lr The struct
- * @param file The file to remove
- * @return true only on successful removal, else false
- */
-bool lrzip_filename_del(Lrzip *lr, const char *file);
-
-/**
- * @brief Pop the current head of the file queue
- * This function is used to remove the current head of the file queue. It can be called
- * immediately following any lrzip_run() file operation to remove the just-processed file. This
- * function modifies the file queue array, reordering and updating the index count.
- * @param lr The struct to pop the filename queue of
- * @return The filename removed from the queue, or NULL on failure
- */
-const char *lrzip_filename_pop(Lrzip *lr);
-
-/**
- * @brief Clear the file queue
- * This function is used to free and reset the file queue.
- * @param lr The struct
- */
-void lrzip_filenames_clear(Lrzip *lr);
-
-/**
- * @brief Set the default suffix for LRZIP compression operations
- * This function is used to change the default ".lrz" suffix for operations
- * to @p suffix.
- * @param lr The struct
- * @param suffix The suffix to use for compression operations
- */
-void lrzip_suffix_set(Lrzip *lr, const char *suffix);
-
-/**
- * @brief Get the default suffix for LRZIP compression operations
- * @param lr The struct
- * @return The suffix to use for compression operations, or NULL on failure
- */
-const char *lrzip_suffix_get(Lrzip *lr);
-
-/**
- * @brief Set the output directory for operations
- * This function can be used to set the output directory for operations.
- * Files will be stored according to their basename and lrzip suffix where
- * applicable.
- * @param lr The struct
- * @param dir The absolute path of the output directory
- */
-void lrzip_outdir_set(Lrzip *lr, const char *dir);
-
-/**
- * @brief Get the output directory for operations
- * @param lr The struct
- * @return The previously set output directory
- */
-const char *lrzip_outdir_get(Lrzip *lr);
-
-/**
- * @brief Set the output stream for operations
- * This function can be used to set the output stream for operations.
- * Raw data will be written to this stream for the duration of lrzip_run().
- * @param lr The struct
- * @param file The stream to write to
- * @warning @p file is NOT created by this library and must be opened by the user!
- */
-void lrzip_outfile_set(Lrzip *lr, FILE *file);
-
-/**
- * @brief Get the output stream for operations
- * @param lr The struct
- * @return The previously set output stream
- */
-FILE *lrzip_outfile_get(Lrzip *lr);
-
-/**
- * @brief Set the output file for operations
- * This function can be used to set the output file for operations.
- * Raw data will be written to the file with this name for the duration of lrzip_run().
- * @param lr The struct
- * @param file The name of the file to write to
- */
-void lrzip_outfilename_set(Lrzip *lr, const char *file);
-
-/**
- * @brief Get the output filename for operations
- * @param lr The struct
- * @return The previously set output filename
- */
-const char *lrzip_outfilename_get(Lrzip *lr);
-
-/**
- * @brief Retrieve the MD5 digest of an LRZIP file
- * Use this function after calling lrzip_run() to retrieve the digest of
- * the processed archive.
- * @param lr The struct having run an operation
- * @return The MD5 digest of the operation's associated archive
- * @note The return value of this function will change after each operation
- */
-const unsigned char *lrzip_md5digest_get(Lrzip *lr);
-
-/**
- * @brief Run the current operation
- * This function is called when all necessary parameters have been set for an operation.
- * The calling thread will then block until the operation has fully completed, writing
- * output using logging and progress callbacks and calling password callbacks as required.
- * @param lr The struct to run an operation with
- * @return true if the operation successfully completed, else false
- */
-bool lrzip_run(Lrzip *lr);
-
-/**
- * @brief Set the logging level
- * @param lr The struct
- * @param level The #Lrzip_Log_Level to use
- */
-void lrzip_log_level_set(Lrzip *lr, int level);
-
-/**
- * @brief Get the logging level
- * @param lr The struct to query
- * @return The #Lrzip_Log_Level of @p lr
- */
-int lrzip_log_level_get(Lrzip *lr);
-
-/**
- * @brief Set a logging callback for use with all operations
- * This function sets an Lrzip_Log_Cb which will be called any time logging
- * output is to be displayed. The callback will be called as many times as the #Lrzip_Log_Level
- * requires.
- * @param lr The struct
- * @param cb The callback
- * @param log_data The data param to use in the logging callback
- */
-void lrzip_log_cb_set(Lrzip *lr, Lrzip_Log_Cb cb, void *log_data);
-
-/**
- * @brief Redirect stdout log messages to another stream
- * This function sends any logging messages which would normally go to stdout into another stream.
- * Useful for when stdout is the target set by lrzip_outfile_set().
- * @param lr The struct
- * @param out The stream to use instead of stdout
- */
-void lrzip_log_stdout_set(Lrzip *lr, FILE *out);
-
-/**
- * @brief Return the stream currently used as stdout
- * @param lr The struct to query
- * @return A stream where stdout messages will be sent, NULL on failure
- */
-FILE *lrzip_log_stdout_get(Lrzip *lr);
-
-/**
- * @brief Redirect stderr log messages to another stream
- * This function sends any logging messages which would normally go to stderr into another stream.
- * @param lr The struct
- * @param err The stream to use instead of stderr
- */
-void lrzip_log_stderr_set(Lrzip *lr, FILE *err);
-
-/**
- * @brief Return the stream currently used as stderr
- * @param lr The struct to query
- * @return A stream where stderr messages will be sent, NULL on failure
- */
-FILE *lrzip_log_stderr_get(Lrzip *lr);
-
-/**
- * @brief Set a password callback for use with all operations
- * This function sets an Lrzip_Password_Cb which will be used when working with encrypted
- * LRZIP archives. It will be called both when compressing and decompressing archives.
- * @param lr The struct
- * @param cb The callback to set
- * @param data The data param to use in the password callback
- */
-void lrzip_pass_cb_set(Lrzip *lr, Lrzip_Password_Cb cb, void *data);
-
-/**
- * @brief Set an info callback for use with all operations
- * This function sets an Lrzip_Info_Cb which will be called any time there is a
- * progress update in an operation.
- * @param lr The struct
- * @param cb The callback to set
- * @param data The data param to use in the info callback
- */
-void lrzip_info_cb_set(Lrzip *lr, Lrzip_Info_Cb cb, void *data);
-
-/**
- * @brief Quick setup for performing a decompression
- * This function performs all the required allocations and sets necessary parameters
- * to decompress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-bool lrzip_decompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len);
-
-/**
- * @brief Quick setup for performing a compression
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param mode The compression mode to use
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-bool lrzip_compress_full(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, Lrzip_Mode mode, int compress_level);
-
-/**
- * @brief Quick setup for performing a compression using LZMA
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_compress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_LZMA, 7); }
-
-/**
- * @brief Quick setup for performing a compression using LZO
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_lcompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_LZO, 7); }
-
-/**
- * @brief Quick setup for performing a compression using ZLIB (GZIP)
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_gcompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_ZLIB, 7); }
-
-/**
- * @brief Quick setup for performing a compression using ZPAQ
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_zcompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_ZPAQ, 7); }
-
-/**
- * @brief Quick setup for performing a compression using BZIP
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_bcompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_BZIP2, 7); }
-
-/**
- * @brief Quick setup for performing RZIP preprocessing
- * This function performs all the required allocations and sets necessary parameters
- * to preprocess @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @return true on success, else false
- */
-static inline bool lrzip_rcompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_NONE, 7); }
-
-/**
- * @brief Quick setup for performing a compression using LZMA and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_compress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_LZMA, compress_level); }
-
-/**
- * @brief Quick setup for performing a compression using LZO and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_lcompress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_LZO, compress_level); }
-
-/**
- * @brief Quick setup for performing a compression using ZLIB (GZIP) and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_gcompress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_ZLIB, compress_level); }
-
-/**
- * @brief Quick setup for performing a compression using ZPAQ and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_zcompress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_ZPAQ, compress_level); }
-
-/**
- * @brief Quick setup for performing a compression using BZIP and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to compress @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_bcompress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_BZIP2, compress_level); }
-
-/**
- * @brief Quick setup for performing RZIP preprocessing and a user-defined compression level
- * This function performs all the required allocations and sets necessary parameters
- * to preprocess @p source to @p dest. No extra functions are necessary to call, and
- * this function will block until it completes.
- * @param dest A pointer to the LRZIP-allocated destination buffer
- * @param dest_len A pointer to the length of @p dest
- * @param source The allocated source buffer to read from
- * @param source_len The length of @p source
- * @param compress_level The value, 1-9, to use as a compression level
- * @return true on success, else false
- */
-static inline bool lrzip_rcompress2(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, int compress_level)
-{ return lrzip_compress_full(dest, dest_len, source, source_len, LRZIP_MODE_COMPRESS_NONE, compress_level); }
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif
-

Makefile.am

@@ -14,6 +14,8 @@ ltmain.sh \
 missing \
 $(PACKAGE_TARNAME)-$(PACKAGE_VERSION).tar.gz \
 $(PACKAGE_TARNAME)-$(PACKAGE_VERSION).tar.bz2 \
+$(PACKAGE_TARNAME)-$(PACKAGE_VERSION).tar.xz \
+$(PACKAGE_TARNAME)-$(PACKAGE_VERSION).tar.lrz \
 $(PACKAGE_TARNAME)-$(PACKAGE_VERSION)-doc.tar.bz2 \
 m4/libtool.m4 \
 m4/lt~obsolete.m4 \
@@ -32,7 +34,6 @@ lrztar_SCRIPTS = lrztar
 noinst_LTLIBRARIES = libtmplrzip.la
 libtmplrzip_la_SOURCES = \
   lrzip_private.h \
-  liblrzip_private.h \
   lrzip.c \
   lrzip_core.h \
   rzip.h \
@@ -55,13 +56,6 @@ libtmplrzip_la_SOURCES = \
 libtmplrzip_la_LIBADD = lzma/C/liblzma.la
 
 
-lib_LTLIBRARIES = liblrzip.la
-liblrzip_la_SOURCES = \
-  liblrzip.c \
-  liblrzip_private.h
-nodist_EXTRA_liblrzip_la_SOURCES = dummy.cxx
-liblrzip_la_LIBADD = libtmplrzip.la
-
 bin_PROGRAMS = lrzip
 lrzip_SOURCES = \
   main.c
@@ -72,31 +66,19 @@ if STATIC
   lrzip_LDFLAGS = -all-static
 endif
 
-noinst_PROGRAMS = decompress_demo liblrzip_demo
-decompress_demo_SOURCES = decompress_demo.c
-decompress_demo_LDADD = liblrzip.la
-
-liblrzip_demo_SOURCES = liblrzip_demo.c
-liblrzip_demo_LDADD = liblrzip.la
-
 dist_doc_DATA = \
   AUTHORS \
   BUGS \
   ChangeLog \
   COPYING \
-  README \
+  README.md \
   README-NOT-BACKWARD-COMPATIBLE \
   TODO \
   WHATS-NEW
 
-pkgconfigdir = $(libdir)/pkgconfig
-pkgconfig_DATA = lrzip.pc
-
-lrzip_HEADERS = Lrzip.h
 lrzipdir = $(includedir)
 
 EXTRA_DIST = \
-		lrzip.pc.in \
 		lrztar \
 		description-pak \
 		autogen.sh \
@@ -107,12 +89,14 @@ install-exec-hook:
 	$(LN_S) -f lrzip$(EXEEXT) $(DESTDIR)$(bindir)/lrunzip$(EXEEXT)
 	$(LN_S) -f lrzip$(EXEEXT) $(DESTDIR)$(bindir)/lrzcat$(EXEEXT)
 	$(LN_S) -f lrztar$(EXEEXT) $(DESTDIR)$(bindir)/lrzuntar$(EXEEXT)
+	$(LN_S) -f lrzip$(EXEEXT) $(DESTDIR)$(bindir)/lrz$(EXEEXT)
 
 
 uninstall-local:
 	rm -f $(bindir)/lrunzip
 	rm -f $(bindir)/lrzcat
 	rm -f $(bindir)/lrzuntar
+	rm -f $(bindir)/lrz
 
 .PHONY: doc
 

README

@@ -1,356 +0,0 @@
-lrzip README
-
-Long Range ZIP or Lzma RZIP
-
-This is a compression program optimised for large files. The larger the file
-and the more memory you have, the better the compression advantage this will
-provide, especially once the files are larger than 100MB. The advantage can
-be chosen to be either size (much smaller than bzip2) or speed (much faster
-than bzip2).
-
-
-Quick lowdown of the most used options:
-
- lrztar directory
-This will produce an archive directory.tar.lrz compressed with lzma.
-
- lrzuntar directory.tar.lrz
-This will completely extract an archived directory.
-
- lrzip filename
-This will produce an archive filename.lrz compressed with lzma (best all
-round) giving slow compression and fast decompression
-
- lrzip -z filename
-This will produce an archive filename.lrz compressed with ZPAQ giving extreme
-compression but which takes ages to compress and decompress
-
- lrzip -l filename
-This will produce an archive filename.lrz compressed with LZO giving very
-fast compression and fast decompression
-
- lrunzip filename.lrz
-This will decompress filename.lrz into filename
-
-
-Lrzip uses an extended version of rzip which does a first pass long distance
-redundancy reduction. The lrzip modifications make it scale according to
-memory size.
-The data is then either:
-1. Compressed by lzma (default) which gives excellent compression
-at approximately twice the speed of bzip2 compression
-2. Compressed by a number of other compressors chosen for different reasons,
-in order of likelihood of usefulness:
-2a. ZPAQ: Extreme compression up to 20% smaller than lzma but ultra slow
-at compression AND decompression.
-2b. LZO: Extremely fast compression and decompression which on most machines
-compresses faster than disk writing making it as fast (or even faster) than
-simply copying a large file
-2c. GZIP: Almost as fast as LZO but with better compression.
-2d. BZIP2: A defacto linux standard of sorts but is the middle ground between
-lzma and gzip and neither here nor there.
-3. Leaving it uncompressed and rzip prepared. This form improves substantially
-any compression performed on the resulting file in both size and speed (due to
-the nature of rzip preparation merging similar compressible blocks of data and
-creating a smaller file). By "improving" I mean it will either speed up the
-very slow compressors with minor detriment to compression, or greatly increase
-the compression of simple compression algorithms.
-
-The major disadvantages are:
-1. The main lrzip application only works on single files so it requires the
-lrztar wrapper to fake a complete archiver.
-2. It requires a lot of memory to get the best performance out of, and is not
-really usable (for compression) with less than 256MB. Decompression requires
-less ram and works on smaller ram machines. Sometimes swap may need to be
-enabled on these lower ram machines for the operating system to be happy.
-3. STDIN/STDOUT works fine on both compression and decompression, but larger
-files compressed in this manner will end up being less efficiently compressed.
-
-The unique feature of lrzip is that it tries to make the most of the available
-ram in your system at all times for maximum benefit. It does this by default,
-choosing the largest sized window possible without running out of memory. It
-also has a unique "sliding mmap" feature which makes it possible to even use
-a compression window larger than your ramsize, if the file is that large. It
-does this (with the -U option) by implementing one large mmap buffer as per
-normal, and a smaller moving buffer to track which part of the file is
-currently being examined, emulating a much larger single mmapped buffer.
-Unfortunately this mode can be many times slower.
-
-See the file README.benchmarks in doc/ for performance examples and what kind
-of data lrzip is very good with.
-
-
-Requires:
-pthreads
-liblzo2-dev
-libbz2-dev
-libz-dev
-libm
-tar
-(nasm on 32bit x86)
-
-To build/install:
-./configure
-make
-make install
-
-To build from the git repository do:
-./autogen.sh
-before the above steps.
-
-
-FAQS.
-Q. What encryption does lrzip use?
-A. Lrzip uses the best current proven technologies to achieve high grade
-password protected encryption. It uses a combination of sha512 to multiply
-hash the password with random salt and aes128 to do block encryption of each
-block of data with more random salt. The amount of initial hashing of the
-password increases by the date an lrzip archive is encrypted according to
-Moore's law, making it harder each year to brute force attack the password
-to keep up with the increasing computing power each year. It is virtually
-guaranteed that the same file encrypted with the same password will never
-be the same twice. The weakest link in this encryption mode by far is the
-password chosen by the user. There is currently no known attack or backdoor
-for this encryption mechanism, and there is absolutely no way of retrieving
-your password should you forget it.
-
-Q. How do I make a static build?
-A. ./configure --enable-static-bin
-
-Q. I want the absolute maximum compression I can possibly get, what do I do?
-A. Try the command line options "-Uzp 1 -L 9". This uses all available ram and
-ZPAQ compression, and even uses a compression window larger than you have ram.
-The -p 1 option disables multithreading which improves compression but at the
-expense of speed. Expect it to take many times longer.
-
-Q. I want the absolute fastest decent compression I can possibly get.
-A. Try the command line option -l. This will use the  lzo backend compression,
-and level 7 compression (1 isn't much faster).
-
-Q. How much slower is the unlimited mode?
-A. It depends on 2 things. First, just how much larger than your ram the file
-is, as the bigger the difference, the slower it will be. The second is how much
-redundant data there is. The more there is, the slower, but ultimately the
-better the compression. Why isn't it on by default? If the compression window is
-a LOT larger than ram, with a lot of redundant information it can be drastically
-slower. I may revisit this possibility in the future if I can make it any
-faster.
-
-Q. Can I use your tool for even more compression than lzma offers?
-A. Yes, the rzip preparation of files makes them more compressible by most
-other compression technique I have tried. Using the -n option will generate
-a .lrz file smaller than the original which should be more compressible, and
-since it is smaller it will compress faster than it otherwise would have.
-
-Q. 32bit?
-A. 32bit machines have a limit of 2GB sized compression windows due to
-userspace limitations on mmap and malloc, so even if you have much more ram
-you will not be able to use compression windows larger than 2GB. Also you
-may be unable to decompress files compressed on 64bit machines which have
-used windows larger than 2GB.
-
-Q. How about 64bit?
-A. 64bit machines with their ability to address massive amounts of ram will
-excel with lrzip due to being able to use compression windows limited only in
-size by the amount of physical ram.
-
-Q. Other operating systems?
-A. The code is POSIXy with GNU extensions. Patches are welcome. Version 0.43+
-should build on MacOSX 10.5+
-
-Q. Does it work on stdin/stdout?
-A. Yes it does. Compression and decompression work well to/from STDIN/STDOUT.
-However because lrzip does multiple passes on the data, it has to store a
-large amount in ram before it dumps it to STDOUT (and vice versa), thus it
-is unable to work with the massive compression windows regular operation
-provides. Thus the compression afforded on files larger than approximately
-25% RAM size will be less efficient (though still benefiting compared to
-traditional compression formats).
-
-Q. I have another compression format that is even better than zpaq, can you
-use that?
-A. You can use it yourself on rzip prepared files (see above). Alternatively
-if the source code is compatible with the GPL license it can be added to the
-lrzip source code. Libraries with functions similar to compress() and
-decompress() functions of zlib would make the process most painless. Please
-tell me if you have such a library so I can include it :)
-
-Q. What's this "Starting lzma back end compression thread..." message?
-A. While I'm a big fan of progress percentage being visible, unfortunately
-lzma compression can't currently be tracked when handing over 100+MB chunks
-over to the lzma library. Therefore you'll see progress percentage until
-each chunk is handed over to the lzma library.
-
-Q. What's this "lzo testing for incompressible data" message?
-A. Other compression is much slower, and lzo is the fastest. To help speed up
-the process, lzo compression is performed on the data first to test that the
-data is at all compressible. If a small block of data is not compressible, it
-tests progressively larger blocks until it has tested all the data (if it fails
-to compress at all). If no compressible data is found, then the subsequent
-compression is not even attempted. This can save a lot of time during the
-compression phase when there is incompressible data. Theoretically it may be
-possible that data is compressible by the other backend (zpaq, lzma etc) and not
-at all by lzo, but in practice such data achieves only minuscule amounts of
-compression which are not worth pursuing. Most of the time it is clear one way
-or the other that data is compressible or not. If you wish to disable this
-test and force it to try compressing it anyway, use -T.
-
-Q. I have truckloads of ram so I can compress files much better, but can my
-generated file be decompressed on machines with less ram?
-A. Yes. Ram requirements for decompression go up only by the -L compression
-option with lzma and are never anywhere near as large as the compression
-requirements. However if you're on 64bit and you use a compression window
-greater than 2GB, it might not be possible to decompress it on 32bit machines.
-
-Q. Why are you including bzip2 compression?
-A. To maintain a similar compression format to the original rzip (although the
-other modes are more useful).
-
-Q. What about multimedia?
-A. Most multimedia is already in a heavily compressed "lossy" format which by
-its very nature has very little redundancy. This means that there is not
-much that can actually be compressed. If your video/audio/picture is in a
-high bitrate, there will be more redundancy than a low bitrate one making it
-more suitable to compression. None of the compression techniques in lrzip are
-optimised for this sort of data. However, the nature of rzip preparation
-means that you'll still get better compression than most normal compression
-algorithms give you if you have very large files. ISO images of dvds for
-example are best compressed directly instead of individual .VOB files. ZPAQ is
-the only compression format that can do any significant compression of
-multimedia.
-
-Q. Is this multithreaded?
-A. As of version 0.540, it is HEAVILY multithreaded with the back end
-compression and decompression phase, and will continue to process the rzip
-pre-processing phase so when using one of the more CPU intensive backend
-compressions like lzma or zpaq, SMP machines will show massive speed
-improvements. Lrzip will detect the number of CPUs to use, but it can be
-overridden with the -p option if the slightly better compression is desired
-more than speed. -p 1 will give the best compression but also be the slowest.
-
-Q. This uses heaps of memory, can I make it use less?
-A. Well you can by setting -w to the lowest value (1) but the huge use of
-memory is what makes the compression better than ordinary compression
-programs so it defeats the point. You'll still derive benefit with -w 1 but
-not as much.
-
-Q. What CFLAGS should I use?
-A. With a recent enough compiler (gcc>4) setting both CFLAGS and CXXFLAGS to
-	-O2 -march=native -fomit-frame-pointer
-
-Q. What compiler does this work with?
-A. It has been tested on gcc, ekopath and the intel compiler successfully
-previously. Whether the commercial compilers help or not, I could not tell you.
-
-Q. What codebase are you basing this on?
-A. rzip v2.1 and lzma sdk920, but it should be possible to stay in sync with
-each of these in the future.
-
-Q. Do we really need yet another compression format?
-A. It's not really a new one at all; simply a reimplementation of a few very
-good performing ones that will scale with memory and file size.
-
-Q. How do you use lrzip yourself?
-A. Three basic uses. I compress large files currently on my drive with the
--l option since it is so quick to get a space saving. When archiving data for
-permanent storage I compress it with the default options. When compressing
-small files for distribution I use the -z option for the smallest possible
-size.
-
-Q. I found a file that compressed better with plain lzma. How can that be?
-A. When the file is more than 5 times the size of the compression window
-you have available, the efficiency of rzip preparation drops off as a means
-of getting better compression. Eventually when the file is large enough,
-plain lzma compression will get better ratios. The lrzip compression will be
-a lot faster though. The only way around this is to use as large compression
-windows as possible with -U option.
-
-Q. Can I use swapspace as ram for lrzip with a massive window?
-A. It will indirectly do this with -U (unlimited) mode enabled. This mode will
-make the compression window as big as the file itself no matter how big it is,
-but it will slow down proportionately more the bigger the file is than your ram.
-
-Q. Why do you nice it to +19 by default? Can I speed up the compression by
-changing the nice value?
-A. This is a common misconception about what nice values do. They only tell the
-cpu process scheduler how to prioritise workloads, and if your application is
-the _only_ thing running it will be no faster at nice -20 nor will it be any
-slower at +19.
-
-Q. What is the LZO Testing option, -T?
-A. LZO testing is normally performed for the slower back-end compression of LZMA
-and ZPAQ. The reasoning is that if it is completely incompressible by LZO then
-it will also be incompressible by them. Thus if a block fails to be compressed
-by the very fast LZO, lrzip will not attempt to compress that block with the
-slower compressor, thereby saving time. If this option is enabled, it will
-bypass the LZO testing and attempt to compress each block regardless.
-
-Q. Compression and decompression progress on large archives slows down and
-speeds up. There's also a jump in the percentage at the end?
-A. Yes, that's the nature of the compression/decompression mechanism. The jump
-is because the rzip preparation makes the amount of data much smaller than the
-compression backend (lzma) needs to compress.
-
-Q. Tell me about patented compression algorithms, GPL, lawyers and copyright.
-A. No
-
-Q. I receive an error "LZMA ERROR: 2. Try a smaller compression window."
-   what does this mean?
-A. LZMA requests large amounts of memory. When a higher compression window is
-   used, there may not be enough contiguous memory for LZMA. LZMA may request
-   up to 25% of TOTAL ram depending on compression level. If contiguous blocks
-   of memory are not free, LZMA will return an error. This is not a fatal
-   error, and a backup mode of compression will be used.
-
-Q. Where can I get more information about the internals of LZMA?
-A. See http://www.7-zip.org and http://www.p7zip.org. Also, see the file
-   ./lzma/C/lzmalib.h which explains the LZMA properties used and the LZMA
-   memory requirements and computation.
-
-Q. This version is much slower than the old version?
-A. Make sure you have set CFLAGS and CXXFLAGS. An unoptimised build will be
-almost 3 times slower.
-
-LIMITATIONS
-Due to mmap limitations the maximum size a window can be set to is currently
-2GB on 32bit unless the -U option is specified. Files generated on 64 bit
-machines with windows >2GB in size might not be decompressible on 32bit
-machines. Large files might not decompress on machines with less RAM if SWAP is
-disabled.
-
-BUGS:
-Probably lots. Tell me if you spot any :) Any known ones should be documented
-in the file BUGS.
-
-
-Links:
-rzip:
-http://rzip.samba.org/
-lzo:
-http://www.oberhumer.com/opensource/lzo/
-lzma:
-http://www.7-zip.org/
-zpaq:
-http://mattmahoney.net/dc/
-
-Thanks to Andrew Tridgell for rzip. Thanks to Markus Oberhumer for lzo.
-Thanks to Igor Pavlov for lzma. Thanks to Jean-loup Gailly and Mark Adler
-for the zlib compression library. Thanks to Christian Leber for lzma
-compat layer, Michael J Cohen for Darwin support, Lasse Collin for fix
-to LZMALib.cpp and for Makefile.in suggestions, and everyone else who coded
-along the way. Huge thanks to Peter Hyman for most of the 0.19-0.24 changes,
-and the update to the multithreaded lzma library and all sorts of other
-features. Thanks to René Rhéaume for fixing executable stacks and
-Ed Avis for various fixes. Thanks to Matt Mahoney for zpaq code. Thanks to
-Jukka Laurila for Darwin support. Thanks to George Makrydakis for lrztar.
-Thanks to Ulrich Drepper for his md5 implementation. Thanks to Michael
-Blumenkrantz for new configuration tools. Thanks to the PolarSSL authors for
-encryption code. Thanks to Serge Belyshev for extensive help, advice, and
-patches to implement encryption. Michael Blumenkrantz also for liblrzip.
-
-Con Kolivas <kernel@kolivas.org>
-Sat, 11 March 2011
-
-Also documented by
-Peter Hyman <pete@peterhyman.com>
-Sun, 04 Jan 2009

README.md

@@ -0,0 +1,479 @@
+lrzip - Long Range ZIP or LZMA RZIP
+===================================
+
+A compression utility that excels at compressing large files (usually > 10-50 MB).
+Larger files and/or more free RAM means that the utility will be able to more
+effectively compress your files (ie: faster / smaller size), especially if the
+filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast
+compression / decompression) or size, but not both.
+
+
+### haneefmubarak's TL;DR for the long explanation:
+
+Just change the word `directory` to the name of the directory you wish to compress.
+
+#### Compression:
+
+```bash
+lrzdir=directory; tar cvf $lrzdir.tar $lrzdir; lrzip -Ubvvp `nproc` -S .bzip2-lrz -L 9 $lrzdir.tar; rm -fv $lrzdir.tar; unset lrzdir
+```
+
+`tar`s the directory, then maxes out all of the system's processor cores
+along with sliding window RAM to give the best **BZIP2** compression while being as fast as possible,
+enables max verbosity output, attaches the extension `.bzip2-lrz`, and finally
+gets rid of the temporary tarfile. Uses a tempvar `lrzdir` which is unset automatically.
+
+#### Decompression for the kind of file from above:
+
+```bash
+lrzdir=directory; lrunzip -cdivvp `nproc` -o $lrzdir.tar $lrzdir.tar.bzip2-lrz; tar xvf $lrzdir.tar; rm -vf $lrzdir.tar
+```
+
+Checks integrity, then decompresses the directory using all of the 
+processor cores for max speed, enables max verbosity output, unarchives
+the resulting tarfile, and finally gets rid of the temporary tarfile. Uses the same kind of tempvar.
+
+
+### lrzip build/install guide:
+
+A quick guide on building and installing.
+
+#### What you will need
+
+ - gcc
+ - bash or zsh
+ - pthreads
+ - tar
+ - libc
+ - libm
+ - libz-dev
+ - libbz2-dev
+ - liblzo2-dev
+ - liblz4-dev
+ - coreutils
+ - Optional nasm
+ - git if you want a repo-fresh copy
+ - an OS with the usual *nix headers and libraries
+
+#### Obtaining the source
+
+Two different ways of doing this:
+
+Stable: Packaged tarball that is known to work:
+
+Go to <https://github.com/ckolivas/lrzip/releases> and download the `tar.gz`
+file from the top. `cd` to the directory you downloaded, and use `tar xvzf lrzip-X.X.tar.gz`
+to extract the files (don't forget to replace `X.X` with the correct version). Finally, cd
+into the directory you just extracted.
+
+Latest: `git clone -v https://github.com/ckolivas/lrzip.git; cd lrzip`
+
+#### Build
+
+```bash
+./autogen.sh
+./configure
+make -j `nproc` # maxes out all cores
+```
+
+#### Install
+
+Simple 'n Easy™: `sudo make install`
+
+### lrzip 101:
+
+|Command|Result|
+|------|------|
+|`lrztar directory`|An archive `directory.tar.lrz` compressed with **LZMA**.|
+|`lrzuntar directory.tar.lrz`|A directory extracted from a `lrztar` archive.|
+|`lrzip filename`|An archive `filename.lrz` compressed with **LZMA**, meaning slow compression and fast decompression.|
+|`lrzip -z filename`|An archive "filename.lrz" compressed with **ZPAQ** that can give extreme compression, but takes a bit longer than forever to compress and decompress.|
+|`lrzip -l filename`|An archive lightly compressed with **LZO**, meaning really, really fast compression and decompression.|
+|`lrunzip filename.lrz`|Decompress filename.lrz to filename.|
+|`lrz filename`|As per lrzip above but with gzip compatible semantics (i.e. will be quiet and delete original file)
+|`lrz -d filename.lrz`|As per lrunzip above but with gzip compatible semantics (i.e. will be quiet and delete original file)
+
+### lrzip internals
+
+lrzip uses an extended version of [rzip](http://rzip.samba.org/) which does a first pass long distance
+redundancy reduction. lrzip's modifications allow it to scale to accommodate various memory sizes.
+
+Then, one of the following scenarios occurs:
+
+ - Compressed
+  - (default) **LZMA** gives excellent compression @ ~2x the speed of bzip2 
+  - **ZPAQ** gives extreme compression while taking forever
+  - **LZO** gives insanely fast compression that can actually be faster than simply copying a large file
+  - **GZIP** gives compression almost as fast as LZO but with better compression
+  - **BZIP2** is a defacto linux standard and hacker favorite which usually gives
+  quite good compression (ZPAQ>LZMA>BZIP2>GZIP>LZO) while staying fairly fast (LZO>GZIP>BZIP2>LZMA>ZPAQ); 
+  in other words, a good middle-ground and a good choice overall
+ - Uncompressed, in the words of the software's original author:
+
+> Leaving it uncompressed and rzip prepared. This form improves substantially
+> any compression performed on the resulting file in both size and speed (due to
+> the nature of rzip preparation merging similar compressible blocks of data and
+> creating a smaller file). By "improving" I mean it will either speed up the
+> very slow compressors with minor detriment to compression, or greatly increase
+> the compression of simple compression algorithms.
+> 
+> (Con Kolivas, from the original lrzip README)
+
+
+The only real disadvantages:
+
+ - The main program, lrzip, only works on single files, and therefore
+ requires the use of an lrztar wrapper to fake a complete archiver.
+ - lrzip requires quite a bit of memory along with a modern processor
+ to get the best performance in reasonable time. This usually means that
+ it is somewhat unusable with less than 256 MB. However, decompression
+ usually requires less RAM and can work on less powerful machines with much
+ less RAM. On machines with less RAM, it may be a good idea to enable swap
+ if you want to keep your operating system happy.
+ - Piping output to and/or from STDIN and/or STDOUT works fine with both
+ compression and decompression, but larger files compressed this way will
+ likely end up being compressed less efficiently. Decompression doesn't
+ really have any issues with piping, though.
+
+One of the more unique features of lrzip is that it will try to use all of
+the available RAM as best it can at all times to provide maximum benefit. This
+is the default operating method, where it will create and use the single
+largest memory window that will still fit in available memory without freezing
+up the system. It does this by `mmap`ing the small portions of the file that
+it is working on. However, it also has a unique "sliding `mmap`" feature, which
+allows it to use compression windows that far exceed the size of your RAM if
+the file you are compressing is large. It does this by using one large `mmap`
+along with a smaller moving `mmap` buffer to track the part of the file that
+is currently being examined. From a higher level, this can be seen as simply
+emulating a single, large `mmap` buffer. The unfortunate thing about this
+feature is that it can become extremely slow. The counter-argument to
+being slower is that it will usually give a better compression factor.
+
+The file `doc/README.benchmarks` has some performance examples to show
+what kind of data lrzip is good with.
+
+
+
+### FAQ
+
+> Q: What  kind of encryption does lrzip use?
+
+> A: lrzip uses SHA2-512 repetitive hashing of the password along with a salt
+> to provide a key which is used by AES-128 to do block encryption. Each block
+> has more random salts added to the block key. The amount of initial hashing
+> increases as the timestamp goes forward, in direct relation to Moore's law,
+> which means that the amount of time required to encrypt/decrypt the file
+> stays the same on a contemporary computer. It is virtually
+> guaranteed that the same file encrypted with the same password will never
+> be the same twice. The weakest link in this encryption mode by far is the
+> password chosen by the user. There is currently no known attack or backdoor
+> for this encryption mechanism, and there is absolutely no way of retrieving
+> your password should you forget it.
+
+> Q: How do I make a static build?
+
+> A: `./configure --enable-static-bin`
+
+> Q: I want the absolute maximum compression I can possibly get, what do I do?
+
+> A: Try the command line options "-Uzp 1 -L 9". This uses all available ram and
+> ZPAQ compression, and even uses a compression window larger than you have ram.
+> The -p 1 option disables multithreading which improves compression but at the
+> expense of speed. Expect it to take many times longer.
+
+> Q: I want the absolute fastest decent compression I can possibly get.
+
+> A: Try the command line option -l. This will use the  lzo backend compression,
+> and level 7 compression (1 isn't much faster).
+
+> Q: How much slower is the unlimited mode?
+
+> A: It depends on 2 things. First, just how much larger than your ram the file
+is, as the bigger the difference, the slower it will be. The second is how much
+redundant data there is. The more there is, the slower, but ultimately the
+better the compression. Why isn't it on by default? If the compression window is
+a LOT larger than ram, with a lot of redundant information it can be drastically
+slower. I may revisit this possibility in the future if I can make it any
+faster.
+
+> Q: Can I use your tool for even more compression than lzma offers?
+
+> A: Yes, the rzip preparation of files makes them more compressible by most
+other compression technique I have tried. Using the -n option will generate
+a .lrz file smaller than the original which should be more compressible, and
+since it is smaller it will compress faster than it otherwise would have.
+
+> Q: 32bit?
+
+> A: 32bit machines have a limit of 2GB sized compression windows due to
+userspace limitations on mmap and malloc, so even if you have much more ram
+you will not be able to use compression windows larger than 2GB. Also you
+may be unable to decompress files compressed on 64bit machines which have
+used windows larger than 2GB.
+
+> Q: How about 64bit?
+
+> A: 64bit machines with their ability to address massive amounts of ram will
+excel with lrzip due to being able to use compression windows limited only in
+size by the amount of physical ram.
+
+> Q: Other operating systems?
+
+> A: The code is POSIXy with GNU extensions. Patches are welcome. Version 0.43+
+should build on MacOSX 10.5+
+
+> Q: Does it work on stdin/stdout?
+
+> A: Yes it does. Compression and decompression work well to/from STDIN/STDOUT.
+However because lrzip does multiple passes on the data, it has to store a
+large amount in ram before it dumps it to STDOUT (and vice versa), thus it
+is unable to work with the massive compression windows regular operation
+provides. Thus the compression afforded on files larger than approximately
+25% RAM size will be less efficient (though still benefiting compared to
+traditional compression formats).
+
+> Q: I have another compression format that is even better than zpaq, can you
+use that?
+
+> A: You can use it yourself on rzip prepared files (see above). Alternatively
+if the source code is compatible with the GPL license it can be added to the
+lrzip source code. Libraries with functions similar to compress() and
+decompress() functions of zlib would make the process most painless. Please
+tell me if you have such a library so I can include it :)
+
+> Q: What's this "Starting lzma back end compression thread..." message?
+
+> A: While I'm a big fan of progress percentage being visible, unfortunately
+lzma compression can't currently be tracked when handing over 100+MB chunks
+over to the lzma library. Therefore you'll see progress percentage until
+each chunk is handed over to the lzma library.
+
+> Q: What's this "lz4 testing for incompressible data" message?
+
+> A: Other compression is much slower, and lz4 is the fastest. To help speed up
+the process, lz4 compression is performed on the data first to test that the
+data is at all compressible. If a small block of data is not compressible, it
+tests progressively larger blocks until it has tested all the data (if it fails
+to compress at all). If no compressible data is found, then the subsequent
+compression is not even attempted. This can save a lot of time during the
+compression phase when there is incompressible data. Theoretically it may be
+possible that data is compressible by the other backend (zpaq, lzma etc) and
+not at all by lz4, but in practice such data achieves only minuscule amounts of
+compression which are not worth pursuing. Most of the time it is clear one way
+or the other that data is compressible or not. If you wish to disable this test
+and force it to try compressing it anyway, use -T.
+
+> Q: I have truckloads of ram so I can compress files much better, but can my
+generated file be decompressed on machines with less ram?
+
+> A: Yes. Ram requirements for decompression go up only by the -L compression
+option with lzma and are never anywhere near as large as the compression
+requirements. However if you're on 64bit and you use a compression window
+greater than 2GB, it might not be possible to decompress it on 32bit machines.
+
+> Q: Why are you including bzip2 compression?
+
+> A: To maintain a similar compression format to the original rzip (although the
+other modes are more useful).
+
+> Q: What about multimedia?
+
+> A: Most multimedia is already in a heavily compressed "lossy" format which by
+its very nature has very little redundancy. This means that there is not much
+that can actually be compressed. If your video/audio/picture is in a high
+bitrate, there will be more redundancy than a low bitrate one making it more
+suitable to compression. None of the compression techniques in lrzip are
+optimised for this sort of data. However, the nature of rzip preparation means
+that you'll still get better compression than most normal compression
+algorithms give you if you have very large files. ISO images of dvds for
+example are best compressed directly instead of individual .VOB files. ZPAQ is
+the only compression format that can do any significant compression of
+multimedia.
+
+> Q: Is this multithreaded?
+
+> A: As of version 0.540, it is HEAVILY multithreaded with the back end
+compression and decompression phase, and will continue to process the rzip
+pre-processing phase so when using one of the more CPU intensive backend
+compressions like lzma or zpaq, SMP machines will show massive speed
+improvements. Lrzip will detect the number of CPUs to use, but it can be
+overridden with the -p option if the slightly better compression is desired
+more than speed. -p 1 will give the best compression but also be the slowest.
+
+> Q: This uses heaps of memory, can I make it use less?
+
+> A: Well you can by setting -w to the lowest value (1) but the huge use of
+memory is what makes the compression better than ordinary compression
+programs so it defeats the point. You'll still derive benefit with -w 1 but
+not as much.
+
+> Q: What CFLAGS should I use?
+
+> A: With a recent enough compiler (gcc>4) setting both CFLAGS and CXXFLAGS to
+	-O2 -march=native -fomit-frame-pointer
+
+> Q: What compiler does this work with?
+
+> A: It has been tested on gcc, ekopath and the intel compiler successfully
+previously. Whether the commercial compilers help or not, I could not tell you.
+
+> Q: What codebase are you basing this on?
+
+> A: rzip v2.1 and lzma sdk920, but it should be possible to stay in sync with
+each of these in the future.
+
+> Q: Do we really need yet another compression format?
+
+> A: It's not really a new one at all; simply a reimplementation of a few very
+good performing ones that will scale with memory and file size.
+
+> Q: How do you use lrzip yourself?
+
+> A: Three basic uses. I compress large files currently on my drive with the
+-l option since it is so quick to get a space saving. When archiving data for
+permanent storage I compress it with the default options. When compressing
+small files for distribution I use the -z option for the smallest possible
+size.
+
+> Q: I found a file that compressed better with plain lzma. How can that be?
+
+> A: When the file is more than 5 times the size of the compression window
+you have available, the efficiency of rzip preparation drops off as a means
+of getting better compression. Eventually when the file is large enough,
+plain lzma compression will get better ratios. The lrzip compression will be
+a lot faster though. The only way around this is to use as large compression
+windows as possible with -U option.
+
+> Q: Can I use swapspace as ram for lrzip with a massive window?
+
+> A: It will indirectly do this with -U (unlimited) mode enabled. This mode will
+make the compression window as big as the file itself no matter how big it is,
+but it will slow down proportionately more the bigger the file is than your ram.
+
+> Q: Why do you nice it to +19 by default? Can I speed up the compression by
+changing the nice value?
+
+> A: This is a common misconception about what nice values do. They only tell the
+cpu process scheduler how to prioritise workloads, and if your application is
+the _only_ thing running it will be no faster at nice -20 nor will it be any
+slower at +19.
+
+> Q: What is the LZ4 Testing option, -T?
+
+> A: LZ4 testing is normally performed for the slower back-end compression of
+LZMA and ZPAQ. The reasoning is that if it is completely incompressible by LZ4
+then it will also be incompressible by them. Thus if a block fails to be
+compressed by the very fast LZ4, lrzip will not attempt to compress that block
+with the slower compressor, thereby saving time. If this option is enabled, it
+will bypass the LZ4 testing and attempt to compress each block regardless.
+
+> Q: Compression and decompression progress on large archives slows down and
+speeds up. There's also a jump in the percentage at the end?
+
+> A: Yes, that's the nature of the compression/decompression mechanism. The jump
+is because the rzip preparation makes the amount of data much smaller than the
+compression backend (lzma) needs to compress.
+
+> Q: Tell me about patented compression algorithms, GPL, lawyers and copyright.
+
+> A: No
+
+> Q: I receive an error "LZMA ERROR: 2. Try a smaller compression window."
+   what does this mean?
+
+> A: LZMA requests large amounts of memory. When a higher compression window is
+used, there may not be enough contiguous memory for LZMA: LZMA may request up
+to 25% of TOTAL ram depending on compression level. If contiguous blocks of
+memory are not free, LZMA will return an error. This is not a fatal error, and
+a backup mode of compression will be used.
+
+> Q: Where can I get more information about the internals of LZMA?
+
+> A: See http://www.7-zip.org and http://www.p7zip.org. Also, see the file
+   ./lzma/C/lzmalib.h which explains the LZMA properties used and the LZMA
+   memory requirements and computation.
+
+> Q: This version is much slower than the old version?
+
+> A: Make sure you have set CFLAGS and CXXFLAGS. An unoptimised build will be
+almost 3 times slower.
+
+> Q: Why not update to the latest version of libzpaq?
+
+> A: For reasons that are unclear the later versions of libzpaq create
+corrupt archives when included with lrzip
+
+#### LIMITATIONS
+Due to mmap limitations the maximum size a window can be set to is currently
+2GB on 32bit unless the -U option is specified. Files generated on 64 bit
+machines with windows >2GB in size might not be decompressible on 32bit
+machines. Large files might not decompress on machines with less RAM if SWAP is
+disabled.
+
+#### BUGS:
+Probably lots. <https://github.com/ckolivas/lrzip/issues> if you spot any :D
+
+ Any known ones should be documented
+in the file BUGS.
+
+
+
+#### Backends:
+
+rzip:
+<http://rzip.samba.org/>
+
+lzo:
+<http://www.oberhumer.com/opensource/lzo/>
+
+lzma:
+<http://www.7-zip.org/>
+
+zpaq:
+<http://mattmahoney.net/dc/>
+
+### Thanks (CONTRIBUTORS)
+
+|Person(s)|Thanks for|
+|---|---|
+|`Andrew Tridgell`|`rzip`|
+|`Markus Oberhumer`|`lzo`|
+|`Igor Pavlov`|`lzma`|
+|`Jean-Loup Gailly & Mark Adler`|`zlib`|
+|***`Con Kolivas`***|***Original Code, binding all of this together, managing the project, original `README`***|
+|`Christian Leber`|`lzma` compatibility layer|
+|`Michael J Cohen`|Darwin/OSX support|
+|`Lasse Collin`|fixes to `LZMALib.cpp` and `Makefile.in`|
+|Everyone else who coded along the way (add yourself where appropriate if that's you)|Miscellaneous Coding|
+|**`Peter Hyman`**|Most of the `0.19` to `0.24` changes|
+|`^^^^^^^^^^^`|Updating the multithreaded `lzma` lib
+|`^^^^^^^^^^^`|All sorts of other features
+|`René Rhéaume`|Fixing executable stacks|
+|`Ed Avis`|Various fixes|
+|`Matt Mahoney`|`zpaq` integration code|
+|`Jukka Laurila`|Additional Darwin/OSX support|
+|`George Makrydakis`|`lrztar` wrapper|
+|`Ulrich Drepper`|*special* implementation of md5|
+|**`Michael Blumenkrantz`**|New config tools|
+|`^^^^^^^^^^^^^^^^^^^^`|`liblrzip`|
+|Authors of `PolarSSL`|Encryption code|
+|`Serge Belyshev`|Extensive help, advice, and patches to implement secure encryption|
+|`Jari Aalto`|Fixing typos, esp. in code|
+|`Carlo Alberto Ferraris`|Code cleanup
+|`Peter Hyman`|Additional documentation|
+|`Haneef Mubarak`|Cleanup, Rewrite, and GH Markdown of `README` --> `README.md`|
+
+Persons above are listed in chronological order of first contribution to **lrzip**. Person(s) with names in **bold** have multiple major contributions, person(s) with names in *italics* have made massive contributions, person(s) with names in ***both*** have made innumerable massive contributions.
+
+#### README Authors
+
+Con Kolivas (`ckolivas` on GitHub) <kernel@kolivas.org>
+Tuesday, 16 February 2021: README
+
+Also documented by
+Peter Hyman <pete@peterhyman.com>
+Sun, 04 Jan 2009: README
+
+Mostly Rewritten + GFMified:
+Haneef Mubarak (haneefmubarak on GitHub)
+Sun/Mon Sep 01-02 2013: README.md

TODO

@@ -21,5 +21,3 @@ Consider ncurses version or even GUI one.
 
 Consider using LZMA Filters for processor-optimised
 coding to increase compression.
-
-Get the ASM working on 64bit - it's only the CRC check so probably no point.

WHATS-NEW

@@ -1,3 +1,68 @@
+lrzip-0.651
+
+Remove redundant files
+Revert locale dependent output
+Add warnings for low memory and threads
+
+lrzip-0.650
+
+Minor optimisations.
+Exit status fixes.
+Update and beautify information output.
+Fix Android build.
+Enable MD5 on Apple build.
+Deprecate and remove liblrzip which was unused and at risk of bitrot.
+Fix failures with compressing to STDOUT with inadequate memory.
+Fix possible race conditions.
+Fix memory leaks.
+Fix -q to only hide progress.
+Add -Q option for very quiet.
+
+
+lrzip-0.641
+
+Critical bugfix for broken lz4 testing which would prevent secondary
+compression from being enabled.
+
+lrzip-0.640
+
+Numerous bugfixes and build fixes.
+lz4 now used for compressibility testing (only) making lz4-dev a build
+requirement.
+Fixes for handling of corrupt archives without crashing.
+Fixes for creating small lzma based archives to stdout.
+Incomplete files are now deleted on interrupting lrzip unless the keep-broken
+option is enabled.
+Version prints to stdout instead of stderr.
+
+
+lrzip-0.631
+
+Assembler code is back and works with x86_64
+
+lrzip-0.621
+
+Substantial speed ups for the rzip stage in both regular and unlimited modes.
+Lrzip now supports long command line options.
+Proper support for the various forms of TMPDIR environment variables.
+More unix portability fixes.
+OSX fixes.
+Fixed order of lrzip.conf search.
+Addressed all warnings created with pedantic compiler settings and clang
+Fixes for some stderr messages being swallowed up.
+Fixed being unable to decompress to STDOUT when in a non-writable directory.
+Changed broken liblrzip callback function API to match lrzip proper.
+
+
+lrzip-0.620
+
+Fixes display output of lrzip -i for large files greater than one chunk.
+Fixes for various failure to allocate memory conditions when dealing with
+large files and STDIO.
+Fixes for more unix portability.
+Fixes for failure to decompress to STDOUT.
+
+
 lrzip-0.616
 
 Fixes for various issues with -O not working with trailing slashes and

configure.ac

@@ -2,7 +2,7 @@
 ##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##
 m4_define([v_maj], [0])
 m4_define([v_min], [6])
-m4_define([v_mic], [16])
+m4_define([v_mic], [51])
 ##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##
 m4_define([v_v], m4_join([], v_min, v_mic))
 m4_define([v_ver], [v_maj.v_v])
@@ -15,16 +15,16 @@ m4_define([lt_age], v_min)
 
 dnl Process this file with autoconf to produce a configure script.
 AC_INIT([lrzip],[v_ver],[kernel@kolivas.org])
-AC_PREREQ([2.59])
+AC_PREREQ([2.71])
 AC_CONFIG_SRCDIR([configure.ac])
 AC_CONFIG_MACRO_DIR([m4])
 AC_CONFIG_HEADERS([config.h])
 
-AM_INIT_AUTOMAKE([1.6 dist-bzip2 foreign])
+AM_INIT_AUTOMAKE([1.6 dist-bzip2 foreign subdir-objects])
 m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
 AC_USE_SYSTEM_EXTENSIONS
 
-AC_PROG_LIBTOOL
+LT_INIT
 ##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##
 ##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##--##
 m4_ifdef([v_rev], , [m4_define([v_rev], [0])])
@@ -51,23 +51,22 @@ AC_PROG_INSTALL
 AC_PROG_LN_S
 AC_SUBST(SHELL)
 AC_SYS_LARGEFILE
+AC_FUNC_FSEEKO
 AC_FUNC_ALLOCA
-AC_PROG_CC_C99
-AS_IF([test "x$ac_cv_prog_cc_c99" = "xno"], 
-	AC_MSG_ERROR([C compiler does not support C99], 1))
+
 AC_CHECK_PROG([HAVE_POD2MAN], [pod2man], [yes])
 AS_IF([test "$HAVE_POD2MAN" != "yes"],
 	AC_MSG_FAILURE([pod2man is needed to generate manual from POD]))
 
-
- AC_ARG_ENABLE(
+AC_ARG_ENABLE(
 		asm,
-		[AC_HELP_STRING([--enable-asm],[Enable native Assembly code])],
+		[AS_HELP_STRING([--enable-asm],[Enable native Assembly code])],
 		ASM=$enableval,
 		ASM=yes
              )
-if test x"$ASM" = xyes; then
-	AC_CHECK_PROG( ASM_PROG, nasm, yes, no )
+
+if test x"$ASM" = x"yes"; then
+	AC_CHECK_PROG( ASM_PROG, nasm, nasm, no ) # fix to set ASM_PROG to nasm, not yes.
 	if test x"$ASM_PROG" = x"no "; then
 		ASM=no
 	fi
@@ -75,10 +74,10 @@ fi
 
 static=no
 AC_ARG_ENABLE([static-bin],
-	[AC_HELP_STRING([--enable-static-bin],[Build statically linked binary @<:@default=no@:>@])],
+	[AS_HELP_STRING([--enable-static-bin],[Build statically linked binary @<:@default=no@:>@])],
 	[static=$enableval]
 	)
-AM_CONDITIONAL([STATIC], [test "x$static" = "xyes"])
+AM_CONDITIONAL([STATIC], [test x"$static" = x"yes"])
 
 AC_CHECK_HEADERS(fcntl.h sys/time.h unistd.h sys/mman.h)
 AC_CHECK_HEADERS(ctype.h errno.h sys/resource.h)
@@ -92,12 +91,6 @@ AC_CHECK_SIZEOF(int)
 AC_CHECK_SIZEOF(long)
 AC_CHECK_SIZEOF(short)
 
-if test $ac_cv_sizeof_long != 4 -a "x$ASM" = "xyes" ; then
-	AC_MSG_WARN([64bit arch detected, disabling ASM])
-	ASM=no
-fi
-
-
 AC_CACHE_CHECK([for large file support],rzip_cv_HAVE_LARGE_FILES,[
 AC_RUN_IFELSE([AC_LANG_SOURCE([[
 #include <stdio.h>
@@ -121,36 +114,42 @@ AC_CHECK_LIB(bz2, BZ2_bzBuffToBuffCompress, ,
         AC_MSG_ERROR([Could not find bz2 library - please install libbz2-dev]))
 AC_CHECK_LIB(lzo2, lzo1x_1_compress, ,
         AC_MSG_ERROR([Could not find lzo2 library - please install liblzo2-dev]))
+AC_CHECK_LIB(lz4, LZ4_compress_default, ,
+	AC_MSG_ERROR([Could not find lz4 library - please install liblz4-dev]))
 
 AC_CHECK_FUNCS(mmap strerror)
 AC_CHECK_FUNCS(getopt_long)
 
-# final checks for x86 and/or assembler
-if test x"$ASM" = x"no"; then
-	ASM_OBJ=7zCrc.o
-	ASM=no
-else
+AX_PTHREAD
+LIBS="$PTHREAD_LIBS $LIBS"
+CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
+CXXFLAGS="$CXXFLAGS $PTHREAD_CXXFLAGS"
+
+# final checks for assembler
+# ASM is back for x86_64 by using newer CRC code from p7zip-16.02
+# object files handled in lzma/C/Makefile.am
+if test x"$ASM" = x"yes"; then
+	ASM_OPT="-I../ASM/x86/"
 	case $host in
 		i?86-*)
-			ASM_OBJ="7zCrcT8.o 7zCrcT8U.o"
-			ASM_CMD="$ASM_PROG -f elf" ;;
-# x86_64 code is broken still
-#		x86_64-*)
-#			ASM_OBJ="7zCrcT8.o 7zCrcT8U_64.o"
-#			ASM_CMD="$ASM_PROG -f elf64" ;;
-		*) ASM_OBJ=7zCrc.o ;;
+			ASM_OPT="$ASM_OPT -g -f elf" ;;
+		x86_64-*)
+			ASM_OPT="$ASM_OPT -Dx64 -g -f elf64" ;;
+		*) ASM_OPT= ;;
 	esac
+else
+	ASM_OPT=
 fi
-AM_CONDITIONAL([USE_ASM], [test "x$ASM" != "xyes" -a "x$ASM" != "xno"])
-AC_SUBST([ASM_OBJ])
+AM_CONDITIONAL([USE_ASM], [test x"$ASM" = x"yes"])
+AC_SUBST([ASM_OPT])
 AC_SUBST([ASM_CMD])
 EFL_CHECK_DOXYGEN([build_doc="yes"], [build_doc="no"])
 
 AC_CONFIG_FILES([
 Makefile
-lrzip.pc
 lzma/Makefile
 lzma/C/Makefile
+lzma/ASM/x86/Makefile
 doc/Makefile
 man/Makefile
 ])
@@ -166,7 +165,7 @@ echo
 echo
 echo "Configuration Options Summary:"
 echo
-echo "  ASM.(32 bit only)..: $ASM"
+echo "  ASM................: $ASM"
 echo "  Static binary......: $static"
 echo
 echo "Documentation..........: ${build_doc}"

doc/README.Assembler

@@ -1,5 +1,20 @@
 README.Assembler
 
+Update November 2019
+
+Assembler is enabled by
+./configure --enable-asm
+and disabled by
+./configure --disable-asm
+not
+ASM=no ./configure 
+
+New files replace 32 and 64 bit assembler code.
+fixes to lzma/C/Makefile.am permit libtool linking.
+
+Original text follows.
+==========================
+
 Notes about CRC Assembly Language Coding.
 
 lrzip-0.21 makes use of an x86 assembly language file

doc/README.benchmarks

@@ -45,7 +45,7 @@ purpose compressor at the moment:
 
 These are benchmarks performed on a 2.53Ghz dual core Intel Core2 with 4GB ram
 using lrzip v0.5.1. Note that it was running with a 32 bit userspace so only
-2GB addressing was posible. However the benchmark was run with the -U option
+2GB addressing was possible. However the benchmark was run with the -U option
 allowing the whole file to be treated as one large compression window.
 
 Tarball of 6 consecutive kernel trees.

doc/lrzip.conf.example

@@ -1,55 +1,47 @@
 # lrzip.conf example file
 # anything beginning with a # or whitespace will be ignored
 # valid parameters are separated with an = and a value
-# parameters and values are not case sensitive
+# parameters and values are not case sensitive except where specified
 # 
 # lrzip 0.24+, peter hyman, pete@peterhyman.com
 # ignored by earlier versions.
 
-# Compression Window size in 100MB. Normally selected by program.
+# Compression Window size in 100MB. Normally selected by program. (-w)
 # WINDOW = 20
-
-# Compression Level 1-9 (7 Default).
+# Compression Level 1-9 (7 Default). (-L)
 # COMPRESSIONLEVEL = 7
-
 # Use -U setting, Unlimited ram. Yes or No
 # UNLIMITED = NO
-
-# Compression Method, rzip, gzip, bzip2, lzo, or lzma (default), or zpaq.
-# If specified here, command line options not usable.
+# Compression Method, rzip, gzip, bzip2, lzo, or lzma (default), or zpaq. (-n -g -b -l --lzma -z)
+# May be overridden by command line compression choice.
 # COMPRESSIONMETHOD = lzma
-
-# Perform LZO Test. Default = YES (-T option, NO)
+# Perform LZO Test. Default = YES (-T )
 # LZOTEST = NO
-
-# Hash Check on decompression, YES
+# Hash Check on decompression, (-c)
 # HASHCHECK = YES
-
-# Show HASH value on Compression even if Verbose is off, YES
+# Show HASH value on Compression even if Verbose is off, YES (-H)
 # SHOWHASH = YES
 
-# Default output directory
+# Default output directory (-O)
 # OUTPUTDIRECTORY = location
-
-# Verbosity, Yes or Max
+# Verbosity, YES or MAX (v, vv)
 # VERBOSITY = max
+# Show Progress as file is parsed, YES or no (NO = -q option)
+# SHOWPROGRESS = YES
 
-# Show Progress as file is parsed, Yes or no
-# SHOWPROGRESS = true
-
-# Set Niceness. 19 is default. -20 to 19 is the allowable range
+# Set Niceness. 19 is default. -20 to 19 is the allowable range (-N)
 # NICE = 19
 
-# Keep broken or damaged output files, YES
+# Keep broken or damaged output files, YES (-K)
 # KEEPBROKEN = YES
 
-# Delete source file after compression
+# Delete source file after compression (-D)
 # this parameter and value are case sensitive
 # value must be YES to activate
 
 # DELETEFILES = NO
 
-# Replace existing lrzip file when compressing
+# Replace existing lrzip file when compressing (-f)
 # this parameter and value are case sensitive
 # value must be YES to activate
 
@@ -58,6 +50,6 @@
 # Override for Temporary Directory. Only valid when stdin/out or Test is used
 # TMPDIR = /tmp
 
+# Whether to use encryption on compression YES, NO (-e)
 # ENCRYPT = NO
 
-# Whether to use encryption on compression

liblrzip.c

@@ -1,743 +0,0 @@
-/*
-   Copyright (C) 2012 Con Kolivas
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU General Public License as published by
-   the Free Software Foundation; either version 2 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU General Public License
-   along with this program. If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <liblrzip_private.h>
-
-#ifdef HAVE_SYS_TIME_H
-# include <sys/time.h>
-#endif
-#ifdef HAVE_SYS_RESOURCE_H
-# include <sys/resource.h>
-#endif
-#ifdef HAVE_SYS_TYPES_H
-# include <sys/types.h>
-#endif
-#ifdef HAVE_SYS_STAT_H
-# include <sys/stat.h>
-#endif
-
-/* needed for CRC routines */
-#include "lzma/C/7zCrc.h"
-#include "util.h"
-#include "lrzip_core.h"
-#include "rzip.h"
-
-#if defined(__APPLE__) || defined(__FreeBSD__)
-# define fmemopen(s, len, modes) fake_fmemopen((s), (len), (modes))
-static FILE *fake_fmemopen(void *buf, size_t buflen, const char *mode)
-{
-	FILE *in;
-
-	in = tmpfile();
-	if (!in)
-		return NULL;
-	if (fwrite(buf, buflen, 1, in) != 1) {
-		fclose(in);
-		return NULL;
-	}
-	rewind(in);
-	return in;
-}
-#endif
-
-static void liblrzip_index_update(size_t x, size_t *idx, void **queue)
-{
-	for (; x < *idx; x++)
-		queue[x] = queue[x + 1];
-	(*idx)--;
-}
-
-static bool liblrzip_setup_flags(Lrzip *lr)
-{
-	if (!lr)
-		return false;
-#define MODE_CHECK(X) \
-	case LRZIP_MODE_COMPRESS_##X: \
-	lr->control->flags ^= FLAG_NOT_LZMA; \
-	lr->control->flags |= FLAG_##X##_COMPRESS; \
-	break
-
-
-	switch (lr->mode) {
-	case LRZIP_MODE_DECOMPRESS:
-		lr->control->flags |= FLAG_DECOMPRESS;
-		break;
-	case LRZIP_MODE_TEST:
-		lr->control->flags |= FLAG_TEST_ONLY;
-		break;
-	case LRZIP_MODE_INFO:
-		lr->control->flags |= FLAG_INFO;
-		break;
-	case LRZIP_MODE_COMPRESS_NONE:
-		lr->control->flags ^= FLAG_NOT_LZMA;
-		lr->control->flags |= FLAG_NO_COMPRESS;
-		break;
-	case LRZIP_MODE_COMPRESS_LZMA:
-		lr->control->flags ^= FLAG_NOT_LZMA;
-		break;
-	MODE_CHECK(LZO);
-	MODE_CHECK(BZIP2);
-	MODE_CHECK(ZLIB);
-	MODE_CHECK(ZPAQ);
-#undef MODE_CHECK
-	default:
-		return false;
-	}
-	setup_overhead(lr->control);
-	if (lr->flags & LRZIP_FLAG_VERIFY) {
-		lr->control->flags |= FLAG_CHECK;
-		lr->control->flags |= FLAG_HASH;
-	}
-	if (lr->flags & LRZIP_FLAG_REMOVE_DESTINATION)
-		lr->control->flags |= FLAG_FORCE_REPLACE;
-	if (lr->flags & LRZIP_FLAG_REMOVE_SOURCE)
-		lr->control->flags &= ~FLAG_KEEP_FILES;
-	if (lr->flags & LRZIP_FLAG_KEEP_BROKEN)
-		lr->control->flags |= FLAG_KEEP_BROKEN;
-	if (lr->flags & LRZIP_FLAG_DISABLE_LZO_CHECK)
-		lr->control->flags &= ~FLAG_THRESHOLD;
-	if (lr->flags & LRZIP_FLAG_UNLIMITED_RAM)
-		lr->control->flags |= FLAG_UNLIMITED;
-	if (lr->flags & LRZIP_FLAG_ENCRYPT)
-		lr->control->flags |= FLAG_ENCRYPT;
-	if (lr->control->log_level > 0) {
-		lr->control->flags |= FLAG_SHOW_PROGRESS;
-		if (lr->control->log_level > 1) {
-			lr->control->flags |= FLAG_VERBOSITY;
-			if (lr->control->log_level > 2)
-				lr->control->flags |= FLAG_VERBOSITY_MAX;
-		}
-	} else lr->control->flags ^= (FLAG_VERBOSE | FLAG_SHOW_PROGRESS);
-	return true;
-}
-
-
-bool lrzip_init(void)
-{
-	/* generate crc table */
-	CrcGenerateTable();
-	return true;
-}
-
-void lrzip_config_env(Lrzip *lr)
-{
-	const char *eptr;
-	/* Get Preloaded Defaults from lrzip.conf
-	 * Look in ., $HOME/.lrzip/, /etc/lrzip.
-	 * If LRZIP=NOCONFIG is set, then ignore config
-	 */
-	eptr = getenv("LRZIP");
-	if (!eptr)
-		read_config(lr->control);
-	else if (!strstr(eptr,"NOCONFIG"))
-		read_config(lr->control);
-}
-
-void lrzip_free(Lrzip *lr)
-{
-	size_t x;
-
-	if ((!lr) || (!lr->infilename_buckets))
-		return;
-	rzip_control_free(lr->control);
-	for (x = 0; x < lr->infilename_idx; x++)
-		free(lr->infilenames[x]);
-	free(lr->infilenames);
-	free(lr->infiles);
-	free(lr);
-}
-
-Lrzip *lrzip_new(Lrzip_Mode mode)
-{
-	Lrzip *lr;
-
-	lr = calloc(1, sizeof(Lrzip));
-	if (!lr)
-		return NULL;
-	lr->control = calloc(1, sizeof(rzip_control));
-	if (!lr->control)
-		goto error;
-	if (!initialize_control(lr->control))
-		goto error;
-	lr->mode = mode;
-	lr->control->library_mode = 1;
-	return lr;
-error:
-	lrzip_free(lr);
-	return NULL;
-}
-
-Lrzip_Mode lrzip_mode_get(Lrzip *lr)
-{
-	if (!lr)
-		return LRZIP_MODE_NONE;
-	return lr->mode;
-}
-
-bool lrzip_mode_set(Lrzip *lr, Lrzip_Mode mode)
-{
-	if ((!lr) || (mode > LRZIP_MODE_COMPRESS_ZPAQ))
-		return false;
-	lr->mode = mode;
-	return true;
-}
-
-bool lrzip_compression_level_set(Lrzip *lr, unsigned int level)
-{
-	if ((!lr) || (!level) || (level > 9))
-		return false;
-	lr->control->compression_level = level;
-	return true;
-}
-
-unsigned int lrzip_compression_level_get(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->control->compression_level;
-}
-
-void lrzip_flags_set(Lrzip *lr, unsigned int flags)
-{
-	if (!lr)
-		return;
-	lr->flags = flags;
-}
-
-unsigned int lrzip_flags_get(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->flags;
-}
-
-void lrzip_nice_set(Lrzip *lr, int nice)
-{
-	if ((!lr) || (nice < -19) || (nice > 20))
-		return;
-	lr->control->nice_val = nice;
-}
-
-int lrzip_nice_get(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->control->nice_val;
-}
-
-void lrzip_threads_set(Lrzip *lr, unsigned int threads)
-{
-	if ((!lr) || (!threads))
-		return;
-	lr->control->threads = threads;
-}
-
-unsigned int lrzip_threads_get(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->control->threads;
-}
-
-void lrzip_compression_window_max_set(Lrzip *lr, int64_t size)
-{
-	if (!lr)
-		return;
-	lr->control->window = size;
-}
-
-int64_t lrzip_compression_window_max_get(Lrzip *lr)
-{
-	if (!lr)
-		return -1;
-	return lr->control->window;
-}
-
-unsigned int lrzip_files_count(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->infile_idx;
-}
-
-unsigned int lrzip_filenames_count(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->infilename_idx;
-}
-
-FILE **lrzip_files_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->infiles;
-}
-
-char **lrzip_filenames_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->infilenames;
-}
-
-bool lrzip_file_add(Lrzip *lr, FILE *file)
-{
-	if ((!lr) || (!file))
-		return false;
-	if (lr->infilenames)
-		return false;
-	if (!lr->infile_buckets) {
-		/* no files added */
-		lr->infiles = calloc(INFILE_BUCKET_SIZE + 1, sizeof(void*));
-		lr->infile_buckets++;
-	} else if (lr->infile_idx == INFILE_BUCKET_SIZE * lr->infile_buckets + 1) {
-		/* all buckets full, create new bucket */
-		FILE **tmp;
-
-		tmp = realloc(lr->infiles, (++lr->infile_buckets * INFILE_BUCKET_SIZE + 1) * sizeof(void*));
-		if (!tmp)
-			return false;
-		lr->infiles = tmp;
-	}
-
-	lr->infiles[lr->infile_idx++] = file;
-	return true;
-}
-
-bool lrzip_file_del(Lrzip *lr, FILE *file)
-{
-	size_t x;
-
-	if ((!lr) || (!file))
-		return false;
-	if (!lr->infile_buckets)
-		return true;
-
-	for (x = 0; x <= lr->infile_idx + 1; x++) {
-		if (!lr->infiles[x])
-			return true; /* not found */
-		if (lr->infiles[x] != file)
-			continue; /* not a match */
-		break;
-	}
-	/* update index */
-	liblrzip_index_update(x, &lr->infile_idx, (void**)lr->infiles);
-	return true;
-}
-
-FILE *lrzip_file_pop(Lrzip *lr)
-{
-	FILE *ret;
-	if ((!lr) || (!lr->infile_buckets))
-		return NULL;
-	ret = lr->infiles[0];
-	lrzip_file_del(lr, ret);
-	return ret;
-}
-
-void lrzip_files_clear(Lrzip *lr)
-{
-	if ((!lr) || (!lr->infile_buckets))
-		return;
-	free(lr->infiles);
-	lr->infiles = NULL;
-}
-
-bool lrzip_filename_add(Lrzip *lr, const char *file)
-{
-	struct stat st;
-
-	if ((!lr) || (!file) || (!file[0]) || (!strcmp(file, "-")))
-		return false;
-	if (lr->infiles)
-		return false;
-	if (stat(file, &st))
-		return false;
-	if (S_ISDIR(st.st_mode))
-		return false;
-
-	if (!lr->infilename_buckets) {
-		/* no files added */
-		lr->infilenames = calloc(INFILE_BUCKET_SIZE + 1, sizeof(void*));
-		lr->infilename_buckets++;
-	} else if (lr->infilename_idx == INFILE_BUCKET_SIZE * lr->infilename_buckets + 1) {
-		/* all buckets full, create new bucket */
-		char **tmp;
-
-		tmp = realloc(lr->infilenames, (++lr->infilename_buckets * INFILE_BUCKET_SIZE + 1) * sizeof(void*));
-		if (!tmp)
-			return false;
-		lr->infilenames = tmp;
-	}
-
-	lr->infilenames[lr->infilename_idx++] = strdup(file);
-	return true;
-}
-
-bool lrzip_filename_del(Lrzip *lr, const char *file)
-{
-	size_t x;
-
-	if ((!lr) || (!file) || (!file[0]))
-		return false;
-	if (!lr->infilename_buckets)
-		return true;
-
-	for (x = 0; x <= lr->infilename_idx + 1; x++) {
-		if (!lr->infilenames[x])
-			return true; /* not found */
-		if (strcmp(lr->infilenames[x], file))
-			continue; /* not a match */
-		free(lr->infilenames[x]);
-		break;
-	}
-	/* update index */
-	liblrzip_index_update(x, &lr->infilename_idx, (void**)lr->infilenames);
-	return true;
-}
-
-const char *lrzip_filename_pop(Lrzip *lr)
-{
-	static char buf[4096];
-	if ((!lr) || (!lr->infilename_buckets))
-		return NULL;
-	strcat(buf, lr->infilenames[0]);
-	lrzip_filename_del(lr, buf);
-	return &buf[0];
-}
-
-void lrzip_filenames_clear(Lrzip *lr)
-{
-	size_t x;
-	if ((!lr) || (!lr->infilename_buckets))
-		return;
-	for (x = 0; x < lr->infilename_idx; x++)
-		free(lr->infilenames[x]);
-	free(lr->infilenames);
-	lr->infilenames = NULL;
-}
-
-void lrzip_suffix_set(Lrzip *lr, const char *suffix)
-{
-	if ((!lr) || (!suffix) || (!suffix[0]))
-		return;
-	free(lr->control->suffix);
-	lr->control->suffix = strdup(suffix);
-}
-
-const char *lrzip_suffix_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->suffix;
-}
-
-void lrzip_outdir_set(Lrzip *lr, const char *dir)
-{
-	const char *slash;
-	char *buf;
-	size_t len;
-	if ((!lr) || (!dir) || (!dir[0]))
-		return;
-	free(lr->control->outdir);
-	slash = strrchr(dir, '/');
-	if (slash && (slash[1] == 0)) {
-		lr->control->outdir = strdup(dir);
-		return;
-	}
-	len = strlen(dir);
-	buf = malloc(len + 2);
-	if (!buf)
-		return;
-	memcpy(buf, dir, len);
-	buf[len] = '/';
-	buf[len + 1] = 0;
-	lr->control->outdir = buf;
-}
-
-const char *lrzip_outdir_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->outdir;
-}
-
-void lrzip_outfile_set(Lrzip *lr, FILE *file)
-{
-	if ((!lr) || (file && (file == stderr)))
-		return;
-	if (lr->control->outname)
-		return;
-	lr->control->outFILE = file;
-}
-
-FILE *lrzip_outfile_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->outFILE;
-}
-
-void lrzip_outfilename_set(Lrzip *lr, const char *file)
-{
-	if ((!lr) || (file && (!file[0])))
-		return;
-	if (lr->control->outFILE)
-		return;
-	if (lr->control->outname && file && (!strcmp(lr->control->outname, file)))
-		return;
-	free(lr->control->outname);
-	lr->control->outname = file ? strdup(file) : NULL;
-}
-
-const char *lrzip_outfilename_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->outname;
-}
-
-const unsigned char *lrzip_md5digest_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->md5_resblock;
-}
-
-bool lrzip_run(Lrzip *lr)
-{
-	struct timeval start_time, end_time;
-	rzip_control *control;
-	double seconds,total_time; // for timers
-	int hours,minutes;
-
-	if (!liblrzip_setup_flags(lr))
-		return false;
-	control = lr->control;
-
-	if ((!lr->infile_idx) && (!lr->infilename_idx))
-		return false;
-	if (lr->control->outFILE) {
-		if (lr->control->outFILE == lr->control->msgout)
-			lr->control->msgout = stderr;
-		lr->control->flags |= FLAG_STDOUT;
-		register_outputfile(lr->control, lr->control->msgout);
-	}
-
-	if (lr->infilenames)
-		lr->control->infile = lr->infilenames[0];
-	else {
-		lr->control->inFILE = lr->infiles[0];
-		control->flags |= FLAG_STDIN;
-	}
-
-	if ((!STDOUT) && (!lr->control->msgout)) lr->control->msgout = stdout;
-	register_outputfile(lr->control, lr->control->msgout);
-
-	setup_ram(lr->control);
-
-	gettimeofday(&start_time, NULL);
-
-	if (ENCRYPT && (!lr->control->pass_cb)) {
-		print_err("No password callback set!\n");
-		return false;
-	}
-
-	if (DECOMPRESS || TEST_ONLY) {
-		if (!decompress_file(lr->control))
-			return false;
-	} else if (INFO) {
-		if (!get_fileinfo(lr->control))
-			return false;
-	} else if (!compress_file(lr->control))
-		return false;
-
-	/* compute total time */
-	gettimeofday(&end_time, NULL);
-	total_time = (end_time.tv_sec + (double)end_time.tv_usec / 1000000) -
-	             (start_time.tv_sec + (double)start_time.tv_usec / 1000000);
-	hours = (int)total_time / 3600;
-	minutes = (int)(total_time / 60) % 60;
-	seconds = total_time - hours * 3600 - minutes * 60;
-	if (!INFO)
-		print_progress("Total time: %02d:%02d:%05.2f\n", hours, minutes, seconds);
-
-	return true;
-}
-
-void lrzip_log_level_set(Lrzip *lr, int level)
-{
-	if (!lr)
-		return;
-	lr->control->log_level = level;
-}
-
-int lrzip_log_level_get(Lrzip *lr)
-{
-	if (!lr)
-		return 0;
-	return lr->control->log_level;
-}
-
-void lrzip_log_cb_set(Lrzip *lr, Lrzip_Log_Cb cb, void *log_data)
-{
-	if (!lr)
-		return;
-	lr->control->log_cb = (void*)cb;
-	lr->control->log_data = log_data;
-}
-
-void lrzip_log_stdout_set(Lrzip *lr, FILE *out)
-{
-	if (!lr)
-		return;
-	lr->control->msgout = out;
-}
-
-FILE *lrzip_log_stdout_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->msgout;
-}
-
-void lrzip_log_stderr_set(Lrzip *lr, FILE *err)
-{
-	if (!lr)
-		return;
-	lr->control->msgerr = err;
-}
-
-FILE *lrzip_log_stderr_get(Lrzip *lr)
-{
-	if (!lr)
-		return NULL;
-	return lr->control->msgerr;
-}
-
-void lrzip_pass_cb_set(Lrzip *lr, Lrzip_Password_Cb cb, void *data)
-{
-	if (!lr)
-		return;
-	lr->control->pass_cb = (void*)cb;
-	lr->control->pass_data = data;
-}
-
-void lrzip_info_cb_set(Lrzip *lr, Lrzip_Info_Cb cb, void *data)
-{
-	if (!lr)
-		return;
-	lr->control->info_cb = (void*)cb;
-	lr->control->info_data = data;
-}
-
-bool lrzip_compress_full(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len, Lrzip_Mode mode, int compress_level)
-{
-	Lrzip *lr;
-	FILE *s, *d;
-	struct stat st;
-	int fd;
-
-	if ((!dest) || (!dest_len) || (!source) || (!source_len) || (mode < LRZIP_MODE_COMPRESS_NONE))
-		return false;
-
-	lrzip_init();
-	if (!mode) mode = LRZIP_MODE_COMPRESS_LZMA;
-	lr = lrzip_new(mode);
-	if (!lr)
-		return false;
-	lrzip_config_env(lr);
-
-	s = fmemopen((void*)source, source_len, "r");
-	d = tmpfile();
-	if ((!s) || (!d))
-		goto error;
-
-	if (!lrzip_file_add(lr, s))
-		goto error;
-	lrzip_outfile_set(lr, d);
-	if (!lrzip_compression_level_set(lr, compress_level))
-		goto error;
-	if (!lrzip_run(lr))
-		goto error;
-
-	fd = fileno(d);
-	if (fstat(fd, &st))
-		goto error;
-	*dest_len = st.st_size;
-	if (unlikely((i64)fread(dest, sizeof(char), st.st_size, d) != st.st_size))
-		goto error;
-	if (unlikely(ferror(d)))
-		goto error;
-	fclose(s);
-	fclose(d);
-	return true;
-
-error:
-	if (s) fclose(s);
-	if (d) fclose(d);
-	lrzip_free(lr);
-	return false;
-}
-
-
-bool lrzip_decompress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)
-{
-	Lrzip *lr;
-	FILE *s, *d;
-	struct stat st;
-	int fd;
-
-	if ((!dest) || (!dest_len) || (!source) || (!source_len))
-		return false;
-
-	lrzip_init();
-	lr = lrzip_new(LRZIP_MODE_DECOMPRESS);
-	if (!lr)
-		return false;
-	lrzip_config_env(lr);
-
-	s = fmemopen((void*)source, source_len, "r");
-	d = tmpfile();
-	if ((!s) || (!d))
-		goto error;
-
-	if (!lrzip_file_add(lr, s))
-		goto error;
-	lrzip_outfile_set(lr, d);
-	if (!lrzip_run(lr))
-		goto error;
-
-	fd = fileno(d);
-	if (fstat(fd, &st))
-		goto error;
-	*dest_len = st.st_size;
-	if (unlikely((i64)fread(dest, sizeof(char), st.st_size, d) != st.st_size))
-		goto error;
-	if (unlikely(ferror(d)))
-		goto error;
-	fclose(s);
-	fclose(d);
-	return true;
-
-error:
-	if (s) fclose(s);
-	if (d) fclose(d);
-	lrzip_free(lr);
-	return false;
-}

liblrzip_demo.c

@@ -1,345 +0,0 @@
-/*
-   Copyright (C) 2012 Con Kolivas
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU General Public License as published by
-   the Free Software Foundation; either version 2 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU General Public License
-   along with this program. If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifdef HAVE_CONFIG_H
-# include "config.h"
-#endif
-#undef NDEBUG
-#include <stdlib.h>
-#include <stdio.h>
-#ifdef HAVE_STRING_H
-# include <string.h>
-#endif
-#include <assert.h>
-#ifdef HAVE_ERRNO_H
-# include <errno.h>
-#else
-extern int errno;
-#endif
-#ifdef HAVE_UNISTD_H
-# include <unistd.h>
-#endif
-#include <termios.h>
-#include <Lrzip.h>
-
-#define failure(...) do { \
-	fprintf(stderr, __VA_ARGS__); \
-	exit(1); \
-} while (0)
-
-static void usage(void)
-{
-	printf("lrzip version %s\n", PACKAGE_VERSION);
-	printf("Copyright (C) Con Kolivas 2006-2011\n");
-	printf("Based on rzip ");
-	printf("Copyright (C) Andrew Tridgell 1998-2003\n\n");
-	printf("Usage: lrzip [options] <file...>\n");
-	printf("General options:\n");
-	printf("     -c            check integrity of file written on decompression\n");
-	printf("     -d            decompress\n");
-	printf("     -e            password protected sha512/aes128 encryption on compression\n");
-	printf("     -h|-?         show help\n");
-	printf("     -H            display md5 hash integrity information\n");
-	printf("     -i            show compressed file information\n");
-	printf("     -q            don't show compression progress\n");
-	printf("     -t            test compressed file integrity\n");
-	printf("     -v[v]         Increase verbosity\n");
-	printf("     -V            show version\n");
-	printf("Options affecting output:\n");
-	printf("     -D            delete existing files\n");
-	printf("     -f            force overwrite of any existing files\n");
-	printf("     -k            keep broken or damaged output files\n");
-	printf("     -o filename   specify the output file name and/or path\n");
-	printf("     -O directory  specify the output directory when -o is not used\n");
-	printf("     -S suffix     specify compressed suffix (default '.lrz')\n");
-	printf("Options affecting compression:\n");
-	printf("     -b            bzip2 compression\n");
-	printf("     -g            gzip compression using zlib\n");
-	printf("     -l            lzo compression (ultra fast)\n");
-	printf("     -n            no backend compression - prepare for other compressor\n");
-	printf("     -z            zpaq compression (best, extreme compression, extremely slow)\n");
-	printf("Low level options:\n");
-	printf("     -L level      set lzma/bzip2/gzip compression level (1-9, default 7)\n");
-	printf("     -N value      Set nice value to value (default 19)\n");
-	printf("     -p value      Set processor count to override number of threads\n");
-	printf("     -T            Disable LZO compressibility testing\n");
-	printf("     -U            Use unlimited window size beyond ramsize (potentially much slower)\n");
-	printf("     -w size       maximum compression window in hundreds of MB\n");
-	printf("                   default chosen by heuristic dependent on ram and chosen compression\n");
-	printf("\nLRZIP=NOCONFIG environment variable setting can be used to bypass lrzip.conf.\n");
-	printf("TMP environment variable will be used for storage of temporary files when needed.\n");
-	printf("TMPDIR may also be stored in lrzip.conf file.\n");
-	printf("\nIf no filenames or \"-\" is specified, stdin/out will be used.\n");
-}
-
-static int get_pass(char *s, size_t slen)
-{
-	int len;
-
-	memset(s, 0, slen);
-	if (!fgets(s, slen, stdin)) {
-		fprintf(stderr, "Failed to retrieve passphrase\n");
-		return -1;
-	}
-	len = strlen(s);
-	if (len > 0 && ('\r' ==  s[len - 1] || '\n' == s[len - 1]))
-		s[len - 1] = '\0';
-	if (len > 1 && ('\r' ==  s[len - 2] || '\n' == s[len - 2]))
-		s[len - 2] = '\0';
-	len = strlen(s);
-	if (!len) {
-		fprintf(stderr, "Empty passphrase\n");
-		return -1;
-	}
-	return len;
-}
-
-static void pass_cb(void *data __UNUSED__, char *pass_string, size_t pass_len)
-{
-	int len;
-	struct termios termios_p;
-	/* Disable stdin echo to screen */
-	tcgetattr(fileno(stdin), &termios_p);
-	termios_p.c_lflag &= ~ECHO;
-	tcsetattr(fileno(stdin), 0, &termios_p);
-
-	printf("Enter passphrase: ");
-	len = get_pass(pass_string, pass_len);
-	printf("\n");
-
-	if (len < 1) exit(1);
-
-	termios_p.c_lflag |= ECHO;
-	tcsetattr(fileno(stdin), 0, &termios_p);
-}
-
-static void mode_check(Lrzip *lr, Lrzip_Mode mode)
-{
-	Lrzip_Mode current = lrzip_mode_get(lr);
-	if (current && (current != mode))
-		failure("Can only use one of -l, -b, -g, -z or -n\n");
-	lrzip_mode_set(lr, mode);
-}
-
-int main(int argc, char *argv[])
-{
-	Lrzip *lr;
-	extern int optind;
-	extern char *optarg;
-	int64_t x;
-	int c;
-	bool get_hash = false;
-
-	lrzip_init();
-	lr = lrzip_new(LRZIP_MODE_NONE);
-	assert(lr);
-	lrzip_config_env(lr);
-	lrzip_log_level_set(lr, LRZIP_LOG_LEVEL_PROGRESS);
-	while ((c = getopt(argc, argv, "bcdDefghHiklL:nN:o:O:p:qS:tTUvVw:z?")) != -1) {
-		switch (c) {
-		case 'b':
-			mode_check(lr, LRZIP_MODE_COMPRESS_BZIP2);
-			break;
-		case 'c':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_VERIFY);
-			break;
-		case 'd':
-			mode_check(lr, LRZIP_MODE_DECOMPRESS);
-			break;
-		case 'D':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_REMOVE_SOURCE);
-			break;
-		case 'e':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_ENCRYPT);
-			break;
-		case 'f':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_REMOVE_DESTINATION);
-			break;
-		case 'g':
-			mode_check(lr, LRZIP_MODE_COMPRESS_ZLIB);
-			break;
-		case 'h':
-		case '?':
-			usage();
-			return -1;
-		case 'H':
-			get_hash = true;
-			break;
-		case 'i':
-			mode_check(lr, LRZIP_MODE_INFO);
-			break;
-		case 'k':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_KEEP_BROKEN);
-			break;
-		case 'l':
-			mode_check(lr, LRZIP_MODE_COMPRESS_LZO);
-			break;
-		case 'L':
-			errno = 0;
-			x = strtol(optarg, NULL, 10);
-			if (errno || ((x < 1) || (x > 9)))
-				failure("Invalid compression level (must be 1-9)\n");
-			lrzip_compression_level_set(lr, (unsigned int)x);
-			break;
-		case 'n':
-			mode_check(lr, LRZIP_MODE_COMPRESS_NONE);
-			break;
-		case 'N':
-			errno = 0;
-			x = strtol(optarg, NULL, 10);
-			if (errno || (x < -20 || x > 19))
-				failure("Invalid nice value (must be -20..19)\n");
-			lrzip_nice_set(lr, x);
-			break;
-		case 'o':
-			if (lrzip_outdir_get(lr))
-				failure("Cannot have -o and -O together\n");
-			if (!strcmp(optarg, "-"))
-				lrzip_outfile_set(lr, stdout);
-			else
-				lrzip_outfilename_set(lr, optarg);
-			break;
-		case 'O':
-			if (lrzip_outfilename_get(lr))	/* can't mix -o and -O */
-				failure("Cannot have options -o and -O together\n");
-			if (lrzip_outfile_get(lr))
-				failure("Cannot specify an output directory when outputting to stdout\n");
-			lrzip_outdir_set(lr, optarg);
-			break;
-		case 'p':
-			errno = 0;
-			x = strtol(optarg, NULL, 10);
-			if (errno || (x < 1))
-				failure("Must have at least one thread\n");
-			lrzip_threads_set(lr, (unsigned int)x);
-			break;
-		case 'q':
-			lrzip_log_level_set(lr, lrzip_log_level_get(lr) - 1);
-			break;
-		case 'S':
-			if (lrzip_outfilename_get(lr))
-				failure("Specified output filename already, can't specify an extension.\n");
-			if (lrzip_outfile_get(lr))
-				failure("Cannot specify a filename suffix when outputting to stdout\n");
-			lrzip_suffix_set(lr, optarg);
-			break;
-		case 't':
-			if (lrzip_outfilename_get(lr))
-				failure("Cannot specify an output file name when just testing.\n");
-			if (lrzip_flags_get(lr) & LRZIP_FLAG_REMOVE_SOURCE)
-				failure("Doubt that you want to delete a file when just testing.\n");
-			mode_check(lr, LRZIP_MODE_TEST);
-			break;
-		case 'T':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_DISABLE_LZO_CHECK);
-			break;
-		case 'U':
-			lrzip_flags_set(lr, lrzip_flags_get(lr) | LRZIP_FLAG_UNLIMITED_RAM);
-			break;
-		case 'v':
-			lrzip_log_level_set(lr, lrzip_log_level_get(lr) + 1);
-			break;
-		case 'V':
-			printf("lrzip version %s\n", PACKAGE_VERSION);
-			exit(0);
-			break;
-		case 'w':
-			errno = 0;
-			x = strtoll(optarg, NULL, 10);
-			if (errno || (x < 1))
-				failure("Invalid compression window '%s'!\n", optarg);
-			lrzip_compression_window_max_set(lr, x);
-			break;
-		case 'z':
-			mode_check(lr, LRZIP_MODE_COMPRESS_ZPAQ);
-			break;
-		}
-	}
-	/* LZMA is the default */
-	if (!lrzip_mode_get(lr)) lrzip_mode_set(lr, LRZIP_MODE_COMPRESS_LZMA);
-	argc -= optind, argv += optind;
-
-	if (lrzip_outfilename_get(lr) && (argc > 1))
-		failure("Cannot specify output filename with more than 1 file\n");
-
-	if ((lrzip_flags_get(lr) & LRZIP_FLAG_UNLIMITED_RAM) && lrzip_compression_window_max_get(lr)) {
-		fprintf(stderr, "If -U used, cannot specify a window size with -w.\n");
-		lrzip_compression_window_max_set(lr, 0);
-	}
-
-	if (argc < 1) lrzip_file_add(lr, stdin);
-
-	if ((lrzip_flags_get(lr) & LRZIP_FLAG_UNLIMITED_RAM) && lrzip_files_count(lr)) {
-		fprintf(stderr, "Cannot have -U and stdin, unlimited mode disabled.\n");
-		lrzip_flags_set(lr, lrzip_flags_get(lr) & ~LRZIP_FLAG_UNLIMITED_RAM);
-	}
-
-	/* If no output filename is specified, and we're using stdin,
-	 * use stdout */
-	if (lrzip_files_count(lr) && (!lrzip_outfilename_get(lr)))
-		lrzip_outfile_set(lr, stdout);
-
-	if (lrzip_flags_get(lr) & LRZIP_FLAG_VERIFY) {
-		if (lrzip_mode_get(lr) != LRZIP_MODE_DECOMPRESS) {
-			fprintf(stderr, "Can only check file written on decompression.\n");
-			lrzip_flags_set(lr, lrzip_flags_get(lr) & ~LRZIP_FLAG_VERIFY);
-		} else if (lrzip_outfile_get(lr)) {
-			fprintf(stderr, "Can't check file written when writing to stdout. Checking disabled.\n");
-			lrzip_flags_set(lr, lrzip_flags_get(lr) & ~LRZIP_FLAG_VERIFY);
-		}
-	}
-
-	for (x = 0; x < argc; x++) {
-		if (argv[x][0] != '-') {
-			assert(lrzip_filename_add(lr, argv[x]));
-			continue;
-		}
-		if (argv[x][1] == 0) {
-			assert(lrzip_file_add(lr, stdin));
-			continue;
-		}
-	}
-	if (argc == 1) {
-		if (!lrzip_files_count(lr)) lrzip_file_add(lr, stdin);
-		if (lrzip_filenames_count(lr)) {
-			if (!lrzip_outfilename_get(lr)) {
-				char buf[4096];
-				const char *infile;
-				size_t len;
-
-				infile = lrzip_filenames_get(lr)[0];
-				len = strlen(infile);
-				if (!strcmp(infile + len - 4, ".lrz"))
-					strncat(buf, infile, len - 4);
-				else
-					snprintf(buf, sizeof(buf), "%s.out", infile);
-				lrzip_outfilename_set(lr, buf);
-			}
-		} else if (!lrzip_outfile_get(lr)) lrzip_outfile_set(lr, stdout);
-	}
-	lrzip_log_stdout_set(lr, stdout);
-	lrzip_log_stderr_set(lr, stderr);
-	lrzip_pass_cb_set(lr, pass_cb, NULL);
-	if (!lrzip_run(lr)) exit(1);
-	if (get_hash) {
-		const unsigned char *digest = lrzip_md5digest_get(lr);
-		for (x = 0; x < 16; x++)
-			fprintf(stdout, "%02x", digest[x] & 0xFF);
-	}
-	lrzip_free(lr);
-	return 0;
-}

liblrzip_private.h

@@ -1,22 +0,0 @@
-#ifdef HAVE_CONFIG_H
-# include "config.h"
-#endif
-#include <lrzip_private.h>
-#include <Lrzip.h>
-
-#define INFILE_BUCKET_SIZE 10
-
-struct Lrzip
-{
-	Lrzip_Mode mode;
-	unsigned int flags;
-	rzip_control *control;
-
-	/* bucket allocation is used here to avoid frequent calls to realloc */
-	char **infilenames;
-	size_t infilename_idx;
-	size_t infilename_buckets;
-	FILE **infiles;
-	size_t infile_idx;
-	size_t infile_buckets;
-};

lrzip.pc.in

@@ -1,10 +0,0 @@
-prefix=@prefix@
-exec_prefix=@exec_prefix@
-libdir=@libdir@
-includedir=@includedir@
-
-Name: lrzip
-Description: lrzip compression library
-Version: @VERSION@
-Libs: -L${libdir} -llrzip
-Libs.private: @LIBS@

lrzip_core.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2011 Con Kolivas
+   Copyright (C) 2006-2016,2022 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998-2003 Andrew Tridgell
 
@@ -27,7 +27,7 @@ bool write_magic(rzip_control *control);
 bool read_magic(rzip_control *control, int fd_in, i64 *expected_size);
 bool preserve_perms(rzip_control *control, int fd_in, int fd_out);
 int open_tmpoutfile(rzip_control *control);
-bool dump_tmpoutfile(rzip_control *control, int fd_out);
+bool flush_tmpout(rzip_control *control);
 int open_tmpinfile(rzip_control *control);
 bool read_tmpinfile(rzip_control *control, int fd_in);
 bool decompress_file(rzip_control *control);
@@ -36,14 +36,15 @@ bool get_fileinfo(rzip_control *control);
 bool compress_file(rzip_control *control);
 bool write_fdout(rzip_control *control, void *buf, i64 len);
 bool write_fdin(rzip_control *control);
-bool flush_tmpoutbuf(rzip_control *control);
 void close_tmpoutbuf(rzip_control *control);
 void clear_tmpinbuf(rzip_control *control);
 bool clear_tmpinfile(rzip_control *control);
 void close_tmpinbuf(rzip_control *control);
-bool initialize_control(rzip_control *control);
+bool initialise_control(rzip_control *control);
+#define initialize_control(_control) initialise_control(_control)
 extern void zpaq_compress(uchar *c_buf, i64 *c_len, uchar *s_buf, i64 s_len, int level,
 			  FILE *msgout, bool progress, long thread);
 extern void zpaq_decompress(uchar *s_buf, i64 *d_len, uchar *c_buf, i64 c_len,
 			    FILE *msgout, bool progress, long thread);
+
 #endif

lrzip_private.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2013 Con Kolivas
+   Copyright (C) 2006-2016,2018,2021-2022 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998-2003 Andrew Tridgell
 
@@ -30,6 +30,7 @@
 #include <stdio.h>
 #include <stdbool.h>
 #include <stdarg.h>
+#include <semaphore.h>
 
 #ifdef HAVE_PTHREAD_H
 # include <pthread.h>
@@ -66,6 +67,10 @@ void *alloca (size_t);
 # include <sys/endian.h>
 #endif
 #ifndef __BYTE_ORDER
+# ifndef __BIG_ENDIAN
+#  define __BIG_ENDIAN	4321
+#  define __LITTLE_ENDIAN	1234
+# endif
 # ifdef WORDS_BIGENDIAN
 #  define __BYTE_ORDER __BIG_ENDIAN
 # else
@@ -139,24 +144,30 @@ extern int errno;
 #define unlikely(x)	__builtin_expect(!!(x), 0)
 #define __maybe_unused	__attribute__((unused))
 
-#if defined(__MINGW32__) || defined(__CYGWIN__) || defined(ANDROID) || defined(__APPLE__)
+#if defined(__MINGW32__) || defined(__CYGWIN__) || defined(__ANDROID__) || defined(__APPLE__) || defined(__OpenBSD__)
 # define ffsll __builtin_ffsll
 #endif
 
-typedef long long int i64;
+typedef int64_t i64;
 typedef uint32_t u32;
 
 typedef struct rzip_control rzip_control;
 typedef struct md5_ctx md5_ctx;
 
-#if !defined(__linux)
- #define mremap fake_mremap
+/* ck specific unnamed semaphore implementations to cope with osx not
+ * implementing them. */
+#ifdef __APPLE__
+struct cksem {
+	int pipefd[2];
+};
+
+typedef struct cksem cksem_t;
+#else
+typedef sem_t cksem_t;
 #endif
 
-#if defined(__APPLE__)
-# define MD5_RELIABLE (0)
-#else
-# define MD5_RELIABLE (1)
+#if !defined(__linux)
+ #define mremap fake_mremap
 #endif
 
 #define bswap_32(x) \
@@ -218,6 +229,7 @@ typedef struct md5_ctx md5_ctx;
 #define FLAG_TMP_OUTBUF		(1 << 21)
 #define FLAG_TMP_INBUF		(1 << 22)
 #define FLAG_ENCRYPT		(1 << 23)
+#define FLAG_OUTPUT		(1 << 24)
 
 #define NO_MD5		(!(HASH_CHECK) && !(HAS_MD5))
 
@@ -243,12 +255,19 @@ typedef struct md5_ctx md5_ctx;
 # define PROCESSORS (sysconf(_SC_NPROCESSORS_ONLN))
 #endif
 
-#ifdef _SC_PAGE_SIZE
-# define PAGE_SIZE (sysconf(_SC_PAGE_SIZE))
-#else
-# define PAGE_SIZE (4096)
+#ifndef PAGE_SIZE
+# ifdef _SC_PAGE_SIZE
+#  define PAGE_SIZE (sysconf(_SC_PAGE_SIZE))
+# else
+#  define PAGE_SIZE (4096)
+# endif
 #endif
 
+#define dealloc(ptr) do { \
+	free(ptr); \
+	ptr = NULL; \
+} while (0)
+
 /* Determine how many times to hash the password when encrypting, based on
  * the date such that we increase the number of loops according to Moore's
  * law relative to when the data is encrypted. It is then stored as a two
@@ -286,10 +305,13 @@ typedef struct md5_ctx md5_ctx;
 #define HAS_MD5		(control->flags & FLAG_MD5)
 #define CHECK_FILE	(control->flags & FLAG_CHECK)
 #define KEEP_BROKEN	(control->flags & FLAG_KEEP_BROKEN)
-#define LZO_TEST	(control->flags & FLAG_THRESHOLD)
+#define LZ4_TEST	(control->flags & FLAG_THRESHOLD)
 #define TMP_OUTBUF	(control->flags & FLAG_TMP_OUTBUF)
 #define TMP_INBUF	(control->flags & FLAG_TMP_INBUF)
 #define ENCRYPT		(control->flags & FLAG_ENCRYPT)
+#define SHOW_OUTPUT	(control->flags & FLAG_OUTPUT)
+
+#define IS_FROM_FILE ( !!(control->inFILE) && !STDIN )
 
 
 /* Structure to save state of computation between the single steps.  */
@@ -325,6 +347,49 @@ struct checksum {
 	i64 len;
 };
 
+typedef i64 tag;
+
+struct node {
+	void *data;
+	struct node *prev;
+};
+
+struct runzip_node {
+	struct stream_info *sinfo;
+	pthread_t *pthreads;
+	struct runzip_node *prev;
+};
+
+struct rzip_state {
+	void *ss;
+	struct node *sslist;
+	struct node *head;
+	struct level *level;
+	tag hash_index[256];
+	struct hash_entry *hash_table;
+	char hash_bits;
+	i64 hash_count;
+	i64 hash_limit;
+	tag minimum_tag_mask;
+	i64 tag_clean_ptr;
+	i64 last_match;
+	i64 chunk_size;
+	i64 mmap_size;
+	char chunk_bytes;
+	uint32_t cksum;
+	int fd_in, fd_out;
+	char stdin_eof;
+	struct {
+		i64 inserts;
+		i64 literals;
+		i64 literal_bytes;
+		i64 matches;
+		i64 match_bytes;
+		i64 tag_hits;
+		i64 tag_misses;
+	} stats;
+};
+
 struct rzip_control {
 	char *infile;
 	FILE *inFILE; // if a FILE is being read from
@@ -358,6 +423,7 @@ struct rzip_control {
 	i64 max_mmap;
 	int threads;
 	char nice_val;		// added for consistency
+	int current_priority;
 	char major_version;
 	char minor_version;
 	i64 st_size;
@@ -373,13 +439,14 @@ struct rzip_control {
 	uchar *salt_pass;
 	int salt_pass_len;
 	uchar *hash;
+	char *passphrase;
 
 	pthread_mutex_t control_lock;
 	unsigned char eof;
 	unsigned char magic_written;
 	bool lzma_prop_set;
 
-	pthread_mutex_t cksumlock;
+	cksem_t cksumsem;
 	md5_ctx ctx;
 	uchar md5_resblock[MD5_DIGEST_SIZE];
 	i64 md5_read; // How far into the file the md5 has done so far
@@ -388,22 +455,33 @@ struct rzip_control {
 	const char *util_infile;
 	char delete_infile;
 	const char *util_outfile;
-#define STREAM_BUCKET_SIZE 20
-	size_t sinfo_buckets;
-	size_t sinfo_idx;
-	struct stream_info **sinfo_queue;
 	char delete_outfile;
 	FILE *outputfile;
 	char library_mode;
 	int log_level;
 	void (*info_cb)(void *data, int pct, int chunk_pct);
 	void *info_data;
-	void (*log_cb)(void *data, unsigned int level, unsigned int line, const char *file, const char *func, const char *format, va_list);
+	void (*log_cb)(void *data, unsigned int level, unsigned int line, const char *file, const char *func, const char *format, va_list args);
 	void *log_data;
 
+	char chunk_bytes;
 	struct sliding_buffer sb;
-	uchar *(*get_sb)(rzip_control *control, i64 p);
-	void (*do_mcpy)(rzip_control *control, unsigned char *buf, i64 offset, i64 len);
+	void (*do_mcpy)(rzip_control *, unsigned char *, i64, i64);
+	void (*next_tag)(rzip_control *, struct rzip_state *, i64, tag *);
+	tag (*full_tag)(rzip_control *, struct rzip_state *, i64);
+	i64 (*match_len)(rzip_control *, struct rzip_state *, i64, i64, i64, i64 *);
+
+	pthread_t *pthreads;
+	struct runzip_node *ruhead;
+};
+
+struct uncomp_thread {
+	uchar *s_buf;
+	i64 u_len, c_len;
+	i64 last_head;
+	uchar c_type;
+	int busy;
+	int streamno;
 };
 
 struct stream {
@@ -429,6 +507,7 @@ struct stream_info {
 	i64 total_read;
 	i64 ram_alloced;
 	i64 size;
+	struct uncomp_thread *ucthreads;
 	long thread_no;
 	long next_thread;
 	int chunks;
@@ -445,8 +524,8 @@ static inline void print_stuff(const rzip_control *control, int level, unsigned
 	} else if (control->msgout) {
 		va_start(ap, format);
 		vfprintf(control->msgout, format, ap);
-		fflush(control->msgout);
 		va_end(ap);
+		fflush(control->msgout);
 	}
 }
 
@@ -461,6 +540,7 @@ static inline void print_err(const rzip_control *control, unsigned int line, con
 		va_start(ap, format);
 		vfprintf(control->msgerr, format, ap);
 		va_end(ap);
+		fflush(control->msgerr);
 	}
 }
 
@@ -469,6 +549,7 @@ static inline void print_err(const rzip_control *control, unsigned int line, con
 } while (0)
 
 #define print_output(...)	do {\
+	if (SHOW_OUTPUT)	\
 	print_stuff(1, __VA_ARGS__); \
 } while (0)
 

lrztar

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 # Copyright (C) George Makrydakis 2009-2011,2013
-# Copyright (C) Con Kolivas 2011-2012
+# Copyright (C) Con Kolivas 2011-2012,2016,2018,2021
 
 # A bash wrapper for Con Kolivas' excellent lrzip utility. For the time
 # being, lrzip does not like pipes, so we had to do this. It is kind of
@@ -24,7 +24,7 @@ function lrztar_local() {
     local hv="\
 lrztar GNU/bash wrapper script for lrzip and tar input/output over directories.
 Copyright (C) George Makrydakis 2009-2011,2013
-Copyright (C) Con Kolivas       2011,2012
+Copyright (C) Con Kolivas       2011-2012,2016,2018,2021
 
 Usage : lrztar [lrzip options] <directory>
 Result: a lrzip tarball is produced.
@@ -43,18 +43,22 @@ Notice:
  - This script exists because of how lrzip behaves.
  - Beware the -f flag, it stands for what it says...
 "
+    [[ $1 == "" ]] && {
+        printf "lrztar: no arguments given\n";
+        return 1;
+    }
     local p=("${@:1:$(($#-1))}") s="${!#}" vopt=("lrz") \
           v_w=0 v_S=0 v_D=0 v_p=0 v_q=0 v_L=0 \
           v_n=0 v_l=0 v_b=0 v_g=0 v_z=0 v_U=0 \
           v_T=0 v_N=0 v_v=0 v_f=0 v_d=0 v_h=0 \
-          v_H=0 v_c=0 v_k=0 v_o=0 v_O=0 x= i="$(pwd)"
-    which tar   &> /dev/null \
+          v_H=0 v_c=0 v_k=0 v_o=0 v_O=0 v_m=0 x= i="$(pwd)"
+    tar --version &> /dev/null \
         || { printf "lrztar: no tar in your path\n"; return 1; }
-    which lrzip &> /dev/null \
+    lrzip --version &> /dev/null \
         || { printf "lrztar: no lrzip in your path\n"; return 1; }
-    which lrzcat &> /dev/null \
+    lrzcat --version &> /dev/null \
         || { printf "lrztar: no lrzcat in your path\n"; return 1; }
-    while getopts w:O:S:DqL:nlbgzUTN:p:vfo:d:tVhHck x; do
+    while getopts w:O:S:DqL:nlbgzUm:TN:p:vfo:d:tVhHck x; do
         [[ $x == [tV] ]] && {
             printf "lrztar: invalid option for lrztar: %s\n" "$x";
             return 1;
@@ -133,12 +137,14 @@ Notice:
             s="${s%/}"
             p+=(-o "$i/${s##*/}.tar.${vopt[v_S]}");
         fi
-        ! ((v_f)) && [[ -e $i/${s##*/}.tar.${vopt[v_S]} ]] && {
-            printf "lrztar: %s exists, use -f to overwrite\n" \
-                   "$i/${s##*/}.tar.${vopt[v_S]}"
-            return 1
-        }
-        tar c "$s" | lrzip ${p[@]// /\\ }
+        if ! ((v_o)); then 
+            ! ((v_f)) && [[ -e $i/${s##*/}.tar.${vopt[v_S]} ]] && {
+                printf "lrztar: %s exists, use -f to overwrite\n" \
+                    "$i/${s##*/}.tar.${vopt[v_S]}"
+                return 1
+            }
+        fi
+        tar c "$s" | lrzip "${p[@]}"
         x=$?
     }
     return $x

lzma/ASM/x86/7zAsm.asm

@@ -0,0 +1,100 @@
+; 7zAsm.asm -- ASM macros
+; 2009-12-12 : Igor Pavlov : Public domain
+; 2011-10-12 : P7ZIP       : Public domain
+
+%define NOT ~
+
+%macro MY_ASM_START 0
+  SECTION .text
+%endmacro
+
+%macro MY_PROC 2 ; macro name:req, numParams:req
+  align 16
+  %define proc_numParams %2 ; numParams
+    global %1
+    global _%1
+    %1:
+    _%1:
+%endmacro
+
+%macro  MY_ENDP 0
+  %ifdef x64
+    ret
+    ; proc_name ENDP
+  %else
+    ret ; (proc_numParams - 2) * 4
+  %endif
+%endmacro
+
+%ifdef x64
+  REG_SIZE equ 8
+%else
+  REG_SIZE equ 4
+%endif
+
+  %define x0  EAX
+  %define x1  ECX
+  %define x2  EDX
+  %define x3  EBX
+  %define x4  ESP
+  %define x5  EBP
+  %define x6  ESI
+  %define x7  EDI
+
+  %define x0_L  AL
+  %define x1_L  CL
+  %define x2_L  DL
+  %define x3_L  BL
+
+  %define x0_H  AH
+  %define x1_H  CH
+  %define x2_H  DH
+  %define x3_H  BH
+
+%ifdef x64
+  %define r0  RAX
+  %define r1  RCX
+  %define r2  RDX
+  %define r3  RBX
+  %define r4  RSP
+  %define r5  RBP
+  %define r6  RSI
+  %define r7  RDI
+%else
+  %define r0  x0
+  %define r1  x1
+  %define r2  x2
+  %define r3  x3
+  %define r4  x4
+  %define r5  x5
+  %define r6  x6
+ %define  r7  x7
+%endif
+
+%macro MY_PUSH_4_REGS 0
+    push    r3
+    push    r5
+%ifdef x64
+  %ifdef CYGWIN64
+    push    r6
+    push    r7
+  %endif
+%else
+    push    r6
+    push    r7
+%endif
+%endmacro
+
+%macro MY_POP_4_REGS 0
+%ifdef x64
+  %ifdef CYGWIN64
+    pop     r7
+    pop     r6
+  %endif
+%else
+    pop     r7
+    pop     r6
+%endif
+    pop     r5
+    pop     r3
+%endmacro

lzma/ASM/x86/7zCrcOpt_asm.asm

@@ -0,0 +1,147 @@
+; 7zCrcOpt.asm -- CRC32 calculation : optimized version
+; 2009-12-12 : Igor Pavlov : Public domain
+
+%include "7zAsm.asm"
+
+MY_ASM_START
+
+%define rD    r2
+%define rN     r7
+
+%ifdef x64
+    %define num_VAR      r8
+    %define table_VAR    r9
+%else
+    data_size   equ (REG_SIZE * 7)
+    crc_table   equ (REG_SIZE + data_size)
+    %define num_VAR     [r4 + data_size]
+    %define table_VAR   [r4 + crc_table]
+%endif
+
+%define SRCDAT  rN + rD + 4 *
+
+%macro CRC 4  ;CRC macro op:req, dest:req, src:req, t:req
+    %1      %2, DWORD [r5 + %3 * 4 + 0400h * %4]  ; op      dest, DWORD [r5 + src * 4 + 0400h * t]
+%endmacro
+
+%macro CRC_XOR 3  ; CRC_XOR macro dest:req, src:req, t:req
+    CRC xor, %1, %2, %3
+%endmacro
+
+%macro CRC_MOV 3  ; CRC_MOV macro dest:req, src:req, t:req
+    CRC mov, %1, %2, %3   ; CRC mov, dest, src, t
+%endmacro
+
+%macro CRC1b 0
+    movzx   x6, BYTE [rD]
+    inc     rD
+    movzx   x3, x0_L
+    xor     x6, x3
+    shr     x0, 8
+    CRC     xor, x0, r6, 0
+    dec     rN
+%endmacro
+
+%macro  MY_PROLOG 1 ; MY_PROLOG macro crc_end:req
+    MY_PUSH_4_REGS
+
+
+%ifdef x64
+  %ifdef CYGWIN64
+    ;ECX=CRC, RDX=buf, R8=size R9=table
+    ; already in R8  : mov num_VAR,R8 ; LEN
+    ; already in RDX : mov rD, RDX ; BUF
+    ; already in R9  : mov table_VAR,R9; table
+    mov x0, ECX ; CRC
+  %else
+    ;EDI=CRC, RSI=buf, RDX=size RCX=table
+    mov num_VAR,RDX ; LEN
+    mov rD, RSI ; BUF
+    mov table_VAR,RCX; table
+    mov x0, EDI ; CRC
+  %endif
+%else
+    mov     x0, [r4 + 20]  ; CRC
+    mov     rD, [r4 + 24]  ; buf
+%endif
+    mov     rN, num_VAR
+    mov     r5, table_VAR
+    test    rN, rN
+    jz   near   %1 ; crc_end
+  %%sl:
+    test    rD, 7
+    jz     %%sl_end
+    CRC1b
+    jnz     %%sl
+  %%sl_end:
+    cmp     rN, 16
+    jb   near   %1; crc_end
+    add     rN, rD
+    mov     num_VAR, rN
+    sub     rN, 8
+    and     rN, NOT 7
+    sub     rD, rN
+    xor     x0, [SRCDAT 0]
+%endmacro
+
+%macro MY_EPILOG 1  ; MY_EPILOG macro crc_end:req
+    xor     x0, [SRCDAT 0]
+    mov     rD, rN
+    mov     rN, num_VAR
+    sub     rN, rD
+  %1:  ; crc_end:
+    test    rN, rN
+    jz      %%end ; @F
+    CRC1b
+    jmp     %1 ; crc_end
+  %%end:
+    MY_POP_4_REGS
+%endmacro
+
+MY_PROC CrcUpdateT8, 4
+    MY_PROLOG crc_end_8
+    mov     x1, [SRCDAT 1]
+    align 16
+  main_loop_8:
+    mov     x6, [SRCDAT 2]
+    movzx   x3, x1_L
+    CRC_XOR x6, r3, 3
+    movzx   x3, x1_H
+    CRC_XOR x6, r3, 2
+    shr     x1, 16
+    movzx   x3, x1_L
+    movzx   x1, x1_H
+    CRC_XOR x6, r3, 1
+    movzx   x3, x0_L
+    CRC_XOR x6, r1, 0
+
+    mov     x1, [SRCDAT 3]
+    CRC_XOR x6, r3, 7
+    movzx   x3, x0_H
+    shr     x0, 16
+    CRC_XOR x6, r3, 6
+    movzx   x3, x0_L
+    CRC_XOR x6, r3, 5
+    movzx   x3, x0_H
+    CRC_MOV x0, r3, 4
+    xor     x0, x6
+    add     rD, 8
+    jnz     main_loop_8
+
+    MY_EPILOG crc_end_8
+MY_ENDP
+
+; T4 CRC deleted
+
+; end
+
+%ifidn __OUTPUT_FORMAT__,elf
+section .note.GNU-stack noalloc noexec nowrite progbits
+%endif
+%ifidn __OUTPUT_FORMAT__,elf32
+section .note.GNU-stack noalloc noexec nowrite progbits
+%endif
+%ifidn __OUTPUT_FORMAT__,elf64
+section .note.GNU-stack noalloc noexec nowrite progbits
+%endif
+

lzma/ASM/x86/7zCrcT8U.s

@@ -1,102 +0,0 @@
-
-SECTION .text
-
-%macro CRC1b 0
-    movzx EDX, BYTE [ESI]
-    inc ESI
-    movzx EBX, AL
-    xor EDX, EBX 
-    shr EAX, 8
-    xor EAX, [EBP + EDX * 4]
-    dec EDI
-%endmacro
-
-data_size equ (28)
-crc_table equ (data_size + 4)
-
-	align 16
-	global CrcUpdateT8
-	global _CrcUpdateT8
-CrcUpdateT8:
-_CrcUpdateT8:
-    push EBX
-    push ESI
-    push EDI
-    push EBP
-
-    mov	EAX, [ESP + 20]        
-    mov	ESI, [ESP + 24]        
-    mov	EDI, [ESP + data_size] 
-    mov EBP, [ESP + crc_table] 
-
-    test EDI, EDI
-    jz sl_end
-  sl:
-    test ESI, 7
-    jz sl_end
-    CRC1b
-    jnz sl
-  sl_end:
-
-    cmp EDI, 16
-    jb NEAR crc_end
-    mov [ESP + data_size], EDI
-    sub EDI, 8
-    and EDI, ~ 7
-    sub [ESP + data_size], EDI
-
-    add EDI, ESI
-    xor EAX, [ESI]
-    mov EBX, [ESI + 4]
-    movzx ECX, BL
-    align 16
-  main_loop:
-    mov EDX, [EBP + ECX*4 + 0C00h]
-    movzx ECX, BH
-    xor EDX, [EBP + ECX*4 + 0800h]
-    shr EBX, 16
-    movzx ECX, BL
-    xor EDX, [EBP + ECX*4 + 0400h]
-    xor EDX, [ESI + 8]
-    movzx ECX, AL
-    movzx EBX, BH
-    xor EDX, [EBP + EBX*4 + 0000h]
-
-    mov EBX, [ESI + 12]
-
-    xor EDX, [EBP + ECX*4 + 01C00h]
-    movzx ECX, AH
-    add ESI, 8
-    shr EAX, 16
-    xor EDX, [EBP + ECX*4 + 01800h]
-    movzx ECX, AL
-    xor EDX, [EBP + ECX*4 + 01400h]
-    movzx ECX, AH
-    mov EAX, [EBP + ECX*4 + 01000h]
-    movzx ECX, BL
-    xor EAX,EDX
-
-    cmp ESI, EDI
-    jne	main_loop
-    xor	EAX, [ESI]
-
-    mov EDI, [ESP + data_size]
-
-  crc_end:
-
-    test EDI, EDI
-    jz fl_end
-  fl:
-    CRC1b
-    jnz fl
-  fl_end:
-
-    pop EBP
-    pop EDI
-    pop ESI
-    pop EBX
-    ret 
-
-%ifidn __OUTPUT_FORMAT__,elf
-    section .note.GNU-stack noalloc noexec nowrite progbits
-%endif

lzma/ASM/x86/Makefile.am

@@ -0,0 +1,7 @@
+MAINTAINERCLEANFILES = Makefile.in
+
+noinst_LTLIBRARIES = liblzmaasm.la
+
+liblzmaasm_la_SOURCES = \
+	7zAsm.asm \
+	7zCrcOpt_asm.asm

lzma/ASM/x86_64/7zCrcT8U_64.s

@@ -1,105 +0,0 @@
-
-SECTION .text
-
-
-
-
-%macro CRC1b 0
-    movzx EDX, BYTE [RSI]
-    inc RSI
-    movzx EBX, AL
-    xor EDX, EBX
-    shr EAX, 8
-    xor EAX, [RDI + RDX * 4]
-    dec R8
-%endmacro
-
-
-
-
-align 16
-global CrcUpdateT8
-
-CrcUpdateT8:
-
-    push RBX
-    push RSI
-    push RDI
-    push RBP
-
-    mov EAX, ECX
-    mov RSI, RDX
-    mov RDI, R9
-
-
-    test R8, R8
-    jz sl_end
-  sl:
-    test RSI, 7
-    jz sl_end
-    CRC1b
-    jnz sl
-  sl_end:
-
-    cmp R8, 16
-    jb crc_end
-    mov R9, R8
-    and R8, 7
-    add R8, 8
-    sub R9, R8
-            
-    add R9, RSI
-    xor EAX, [RSI]
-    mov EBX, [RSI + 4]
-    movzx ECX, BL
-    align 16
-  main_loop:
-    mov EDX, [RDI + RCX*4 + 0C00h]
-    movzx EBP, BH
-    xor EDX, [RDI + RBP*4 + 0800h]
-    shr EBX, 16
-    movzx ECX, BL
-    xor EDX, [RSI + 8]
-    xor EDX, [RDI + RCX*4 + 0400h]
-    movzx ECX, AL
-    movzx EBP, BH
-    xor EDX, [RDI + RBP*4 + 0000h]
-
-    mov EBX, [RSI + 12]
-
-    xor EDX, [RDI + RCX*4 + 01C00h]
-    movzx EBP, AH
-    shr EAX, 16
-    movzx ECX, AL
-    xor EDX, [RDI + RBP*4 + 01800h]
-    movzx EBP, AH
-    mov EAX, [RDI + RCX*4 + 01400h]
-    add RSI, 8
-    xor EAX, [RDI + RBP*4 + 01000h]
-    movzx ECX, BL
-    xor EAX,EDX
-
-    cmp RSI, R9
-    jne main_loop
-    xor EAX, [RSI]
-
-
-  
-  crc_end:
-
-    test R8, R8
-    jz fl_end
-  fl:
-    CRC1b
-    jnz fl
-  fl_end:
-
-    pop RBP
-    pop RDI
-    pop RSI
-    pop RBX
-    ret
-
-%ifidn __OUTPUT_FORMAT__,elf
-    section .note.GNU-stack noalloc noexec nowrite progbits
-%endif

lzma/C/LzmaLib.c

@@ -8,8 +8,8 @@ Public domain */
 #include "Alloc.h"
 #include "LzmaLib.h"
 
-static void *SzAlloc(void *p, size_t size) { p = p; return MyAlloc(size); }
-static void SzFree(void *p, void *address) { p = p; MyFree(address); }
+static void *SzAlloc(void __attribute__((unused)) *p, size_t size) { return MyAlloc(size); }
+static void SzFree(void __attribute__((unused)) *p, void *address) { MyFree(address); }
 static ISzAlloc g_Alloc = { SzAlloc, SzFree };
 
 MY_STDAPI LzmaCompress(unsigned char *dest, size_t  *destLen, const unsigned char *src, size_t  srcLen,

lzma/C/Makefile.am

@@ -1,22 +1,28 @@
 MAINTAINERCLEANFILES = Makefile.in
 
+# Update -D
 AM_CFLAGS = \
-  -DCOMPRESS_MF_MT \
   -D_REENTRANT \
   -I@top_builddir@ \
   -I@top_srcdir@
 
-ASM_LIBS =
 ASM_S =
+ASM_7z =
+C_S =
 if USE_ASM
-  ASM_LIBS += @ASM_OBJ@
+  ASM_7z += 7zCrcOpt_asm
+  ASM_S += @abs_top_srcdir@/lzma/ASM/x86/$(ASM_7z).asm
+  C_S += 7zCrcT8.c
 else
-  ASM_S += 7zCrc.c 7zCrc.h
+  C_S += 7zCrc.c 
 endif
 
 noinst_LTLIBRARIES = liblzma.la
+# need separate variable for ASM so that make will compile later
+# to prevent an error even if -j## is used.
 liblzma_la_SOURCES = \
-  $(ASM_S) \
+	$(C_S) \
+	7zCrc.h \
 	LzmaDec.h \
 	LzmaEnc.h \
 	LzFind.c \
@@ -37,10 +43,22 @@ liblzma_la_SOURCES = \
 	basetyps.h \
 	MyWindows.h \
 	MyGuidDef.h
-liblzma_so_LIBS = $(ASM_LIBS)
 
-7zCrcT8U.o: @top_srcdir@/lzma/ASM/x86/7zCrcT8U.s
-	@ASM_CMD@ -o 7zCrcT8U.o @top_srcdir@/lzma/ASM/x86/7zCrcT8U.s
+## hack to force asm compilation and to trick libtool with .lo file
+if USE_ASM 
+liblzma_la_LIBADD = $(ASM_7z).lo
 
-7zCrcT8U_64.o: @top_srcdir@/lzma/ASM/x86_64/7zCrcT8U_64.s
-	@ASM_CMD@ -o 7zCrcT8U_64.o @top_srcdir@/lzma/ASM/x86_64/7zCrcT8U_64.s
+7ZIPASMLOFILE := \
+\# $(ASM_7z).lo - a libtool object file\
+\n\# Generated by libtool -- hack to allow asm linking\
+\n\# Peter Hyman\
+\npic_object='.libs/$(ASM_7z).o'\
+\nnon_pic_object='$(ASM_7z).o'\
+\n
+
+$(ASM_7z).lo: $(ASM_S)
+	$(ASM_PROG) $(ASM_OPT) -o $(ASM_7z).o $(ASM_S)
+	mkdir -p .libs
+	cp $(ASM_7z).o .libs/
+	@printf "$(7ZIPASMLOFILE)" > $(ASM_7z).lo
+endif

lzma/C/Threads.c

@@ -198,7 +198,7 @@ WRes Thread_Create(CThread *thread, THREAD_FUNC_RET_TYPE (THREAD_FUNC_CALL_TYPE
 	ret = pthread_attr_setdetachstate(&attr,PTHREAD_CREATE_JOINABLE);
 	if (ret) return ret;
 
-	ret = pthread_create(&thread->_tid, &attr, (void * (*)(void *))startAddress, parameter);
+	ret = pthread_create(&thread->_tid, &attr, (void *)startAddress, parameter);
 
 	/* ret2 = */ pthread_attr_destroy(&attr);
 

lzma/Makefile.am

@@ -1,4 +1,4 @@
-SUBDIRS = C
+SUBDIRS = C ASM/x86
 MAINTAINERCLEANFILES = Makefile.in
 
 lzmadocdir = @docdir@/lzma

m4/ac_attribute.m4

@@ -18,8 +18,7 @@ AC_DEFUN([AC_C___ATTRIBUTE__],
 AC_MSG_CHECKING([for __attribute__])
 
 AC_CACHE_VAL([ac_cv___attribute__],
-   [AC_TRY_COMPILE(
-       [
+   [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 #include <stdlib.h>
 
 int func(int x);
@@ -27,11 +26,8 @@ int foo(int x __attribute__ ((unused)))
 {
    exit(1);
 }
-       ],
-       [],
-       [ac_cv___attribute__="yes"],
-       [ac_cv___attribute__="no"]
-    )])
+       ]], [[]])],[ac_cv___attribute__="yes"],[ac_cv___attribute__="no"
+    ])])
 
 AC_MSG_RESULT($ac_cv___attribute__)
 

m4/efl_doxygen.m4

@@ -15,9 +15,7 @@ dnl
 dnl Disable the build of the documentation
 dnl
 AC_ARG_ENABLE([doc],
-   [AC_HELP_STRING(
-       [--disable-doc],
-       [Disable documentation build @<:@default=enabled@:>@])],
+   [AS_HELP_STRING([--disable-doc],[Disable documentation build @<:@default=enabled@:>@])],
    [
     if test "x${enableval}" = "xyes" ; then
        efl_enable_doc="yes"
@@ -39,9 +37,7 @@ dnl
    efl_doxygen="doxygen"
 
    AC_ARG_WITH([doxygen],
-      [AC_HELP_STRING(
-          [--with-doxygen=FILE],
-          [doxygen program to use @<:@default=doxygen@:>@])],
+      [AS_HELP_STRING([--with-doxygen=FILE],[doxygen program to use @<:@default=doxygen@:>@])],
 dnl
 dnl Check the given doxygen program.
 dnl

main.c

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2013 Con Kolivas
+   Copyright (C) 2006-2016,2021-2022 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998-2003 Andrew Tridgell
 
@@ -49,6 +49,10 @@
 # include <arpa/inet.h>
 #endif
 
+#include <dirent.h>
+#include <getopt.h>
+#include <libgen.h>
+
 #include "rzip.h"
 #include "lrzip_core.h"
 #include "util.h"
@@ -57,47 +61,72 @@
 /* needed for CRC routines */
 #include "lzma/C/7zCrc.h"
 
+#define MAX_PATH_LEN 4096
+
 static rzip_control base_control, local_control, *control;
 
-static void usage(void)
+static void usage(bool compat)
 {
-	print_output("lrzip version %s\n", PACKAGE_VERSION);
-	print_output("Copyright (C) Con Kolivas 2006-2013\n");
+	print_output("lrz%s version %s\n", compat ? "" : "ip", PACKAGE_VERSION);
+	print_output("Copyright (C) Con Kolivas 2006-2022\n");
 	print_output("Based on rzip ");
 	print_output("Copyright (C) Andrew Tridgell 1998-2003\n\n");
-	print_output("Usage: lrzip [options] <file...>\n");
+	print_output("Usage: lrz%s [options] <file...>\n", compat ? "" : "ip");
 	print_output("General options:\n");
-	print_output("     -c            check integrity of file written on decompression\n");
-	print_output("     -d            decompress\n");
-	print_output("     -e            password protected sha512/aes128 encryption on compression\n");
-	print_output("     -h|-?         show help\n");
-	print_output("     -H            display md5 hash integrity information\n");
-	print_output("     -i            show compressed file information\n");
-	print_output("     -q            don't show compression progress\n");
-	print_output("     -t            test compressed file integrity\n");
-	print_output("     -v[v]         Increase verbosity\n");
-	print_output("     -V            show version\n");
+	if (compat) {
+		print_output("	-c, --stdout		output to STDOUT\n");
+		print_output("	-C, --check		check integrity of file written on decompression\n");
+	} else
+		print_output("	-c, -C, --check		check integrity of file written on decompression\n");
+	print_output("	-d, --decompress	decompress\n");
+	print_output("	-e, --encrypt[=password] password protected sha512/aes128 encryption on compression\n");
+	print_output("	-h, -?, --help		show help\n");
+	print_output("	-H, --hash		display md5 hash integrity information\n");
+	print_output("	-i, --info		show compressed file information\n");
+	if (compat) {
+		print_output("	-L, --license		display software version and license\n");
+		print_output("	-P, --progress		show compression progress\n");
+	} else {
+		print_output("	-q, --quiet		don't show compression progress\n");
+		print_output("	-Q, --very-quiet	don't show any output\n");
+	}
+	print_output("	-r, --recursive		operate recursively on directories\n");
+	print_output("	-t, --test		test compressed file integrity\n");
+	print_output("	-v[v%s], --verbose	Increase verbosity\n", compat ? "v" : "");
+	print_output("	-V, --version		show version\n");
 	print_output("Options affecting output:\n");
-	print_output("     -D            delete existing files\n");
-	print_output("     -f            force overwrite of any existing files\n");
-	print_output("     -k            keep broken or damaged output files\n");
-	print_output("     -o filename   specify the output file name and/or path\n");
-	print_output("     -O directory  specify the output directory when -o is not used\n");
-	print_output("     -S suffix     specify compressed suffix (default '.lrz')\n");
+	if (!compat)
+		print_output("	-D, --delete		delete existing files\n");
+	print_output("	-f, --force		force overwrite of any existing files\n");
+	if (compat)
+		print_output("	-k, --keep		don't delete source files on de/compression\n");
+	print_output("	-K, --keep-broken	keep broken or damaged output files\n");
+	print_output("	-o, --outfile filename	specify the output file name and/or path\n");
+	print_output("	-O, --outdir directory	specify the output directory when -o is not used\n");
+	print_output("	-S, --suffix suffix	specify compressed suffix (default '.lrz')\n");
 	print_output("Options affecting compression:\n");
-	print_output("     -b            bzip2 compression\n");
-	print_output("     -g            gzip compression using zlib\n");
-	print_output("     -l            lzo compression (ultra fast)\n");
-	print_output("     -n            no backend compression - prepare for other compressor\n");
-	print_output("     -z            zpaq compression (best, extreme compression, extremely slow)\n");
+	print_output("	--lzma			lzma compression (default)\n");
+	print_output("	-b, --bzip2		bzip2 compression\n");
+	print_output("	-g, --gzip		gzip compression using zlib\n");
+	print_output("	-l, --lzo		lzo compression (ultra fast)\n");
+	print_output("	-n, --no-compress	no backend compression - prepare for other compressor\n");
+	print_output("	-z, --zpaq		zpaq compression (best, extreme compression, extremely slow)\n");
 	print_output("Low level options:\n");
-	print_output("     -L level      set lzma/bzip2/gzip compression level (1-9, default 7)\n");
-	print_output("     -N value      Set nice value to value (default 19)\n");
-	print_output("     -p value      Set processor count to override number of threads\n");
-	print_output("     -T            Disable LZO compressibility testing\n");
-	print_output("     -U            Use unlimited window size beyond ramsize (potentially much slower)\n");
-	print_output("     -w size       maximum compression window in hundreds of MB\n");
-	print_output("                   default chosen by heuristic dependent on ram and chosen compression\n");
+	if (compat) {
+		print_output("	-1 .. -9		set lzma/bzip2/gzip compression level (1-9, default 7)\n");
+		print_output("	--fast			alias for -1\n");
+		print_output("	--best			alias for -9\n");
+	}
+	if (!compat)
+		print_output("	-L, --level level	set lzma/bzip2/gzip compression level (1-9, default 7)\n");
+	print_output("	-N, --nice-level value	Set nice value to value (default %d)\n", compat ? 0 : 19);
+	print_output("	-p, --threads value	Set processor count to override number of threads\n");
+	print_output("	-m, --maxram size	Set maximum available ram in hundreds of MB\n");
+	print_output("				overrides detected amount of available ram\n");
+	print_output("	-T, --threshold		Disable LZ4 compressibility testing\n");
+	print_output("	-U, --unlimited		Use unlimited window size beyond ramsize (potentially much slower)\n");
+	print_output("	-w, --window size	maximum compression window in hundreds of MB\n");
+	print_output("				default chosen by heuristic dependent on ram and chosen compression\n");
 	print_output("\nLRZIP=NOCONFIG environment variable setting can be used to bypass lrzip.conf.\n");
 	print_output("TMP environment variable will be used for storage of temporary files when needed.\n");
 	print_output("TMPDIR may also be stored in lrzip.conf file.\n");
@@ -105,17 +134,25 @@ static void usage(void)
 
 }
 
+static void license(void)
+{
+	print_output("lrz version %s\n", PACKAGE_VERSION);
+	print_output("Copyright (C) Con Kolivas 2006-2016\n");
+	print_output("Based on rzip ");
+	print_output("Copyright (C) Andrew Tridgell 1998-2003\n\n");
+	print_output("This is free software.  You may redistribute copies of it under the terms of\n");
+	print_output("the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.\n");
+	print_output("There is NO WARRANTY, to the extent permitted by law.\n");
+}
+
 static void sighandler(int sig __UNUSED__)
 {
-	struct termios termios_p;
-
-	/* Make sure we haven't died after disabling stdin echo */
-	tcgetattr(fileno(stdin), &termios_p);
-	termios_p.c_lflag |= ECHO;
-	tcsetattr(fileno(stdin), 0, &termios_p);
-
-	unlink_files(control);
-	exit(0);
+	signal(sig, SIG_IGN);
+	signal(SIGTERM, SIG_IGN);
+	signal(SIGTTIN, SIG_IGN);
+	signal(SIGTTOU, SIG_IGN);
+	print_err("Interrupted\n");
+	fatal_exit(&local_control);
 }
 
 static void show_summary(void)
@@ -150,15 +187,15 @@ static void show_summary(void)
 		if (!DECOMPRESS && !TEST_ONLY) {
 			print_verbose("Compression mode is: ");
 			if (LZMA_COMPRESS)
-				print_verbose("LZMA. LZO Compressibility testing %s\n", (LZO_TEST? "enabled" : "disabled"));
+				print_verbose("LZMA. LZ4 Compressibility testing %s\n", (LZ4_TEST? "enabled" : "disabled"));
 			else if (LZO_COMPRESS)
 				print_verbose("LZO\n");
 			else if (BZIP2_COMPRESS)
-				print_verbose("BZIP2. LZO Compressibility testing %s\n", (LZO_TEST? "enabled" : "disabled"));
+				print_verbose("BZIP2. LZ4 Compressibility testing %s\n", (LZ4_TEST? "enabled" : "disabled"));
 			else if (ZLIB_COMPRESS)
 				print_verbose("GZIP\n");
 			else if (ZPAQ_COMPRESS)
-				print_verbose("ZPAQ. LZO Compressibility testing %s\n", (LZO_TEST? "enabled" : "disabled"));
+				print_verbose("ZPAQ. LZ4 Compressibility testing %s\n", (LZ4_TEST? "enabled" : "disabled"));
 			else if (NO_COMPRESS)
 				print_verbose("RZIP pre-processing only\n");
 			if (control->window)
@@ -184,26 +221,122 @@ static void show_summary(void)
 	}
 }
 
+static struct option long_options[] = {
+	{"bzip2",	no_argument,	0,	'b'}, /* 0 */
+	{"check",	no_argument,	0,	'c'},
+	{"check",	no_argument,	0,	'C'},
+	{"decompress",	no_argument,	0,	'd'},
+	{"delete",	no_argument,	0,	'D'},
+	{"encrypt",	optional_argument,	0,	'e'}, /* 5 */
+	{"force",	no_argument,	0,	'f'},
+	{"gzip",	no_argument,	0,	'g'},
+	{"help",	no_argument,	0,	'h'},
+	{"hash",	no_argument,	0,	'H'},
+	{"info",	no_argument,	0,	'i'}, /* 10 */
+	{"keep-broken",	no_argument,	0,	'k'},
+	{"keep-broken",	no_argument,	0,	'K'},
+	{"lzo",		no_argument,	0,	'l'},
+	{"lzma",       	no_argument,	0,	'/'},
+	{"level",	optional_argument,	0,	'L'}, /* 15 */
+	{"license",	no_argument,	0,	'L'},
+	{"maxram",	required_argument,	0,	'm'},
+	{"no-compress",	no_argument,	0,	'n'},
+	{"nice-level",	required_argument,	0,	'N'},
+	{"outfile",	required_argument,	0,	'o'},
+	{"outdir",	required_argument,	0,	'O'}, /* 20 */
+	{"threads",	required_argument,	0,	'p'},
+	{"progress",	no_argument,	0,	'P'},
+	{"quiet",	no_argument,	0,	'q'},
+	{"very-quiet",	no_argument,	0,	'Q'},
+	{"recursive",	no_argument,	0,	'r'},
+	{"suffix",	required_argument,	0,	'S'},
+	{"test",	no_argument,	0,	't'},  /* 25 */
+	{"threshold",	required_argument,	0,	'T'},
+	{"unlimited",	no_argument,	0,	'U'},
+	{"verbose",	no_argument,	0,	'v'},
+	{"version",	no_argument,	0,	'V'},
+	{"window",	required_argument,	0,	'w'},  /* 30 */
+	{"zpaq",	no_argument,	0,	'z'},
+	{"fast",	no_argument,	0,	'1'},
+	{"best",	no_argument,	0,	'9'},
+	{0,	0,	0,	0},
+};
+
+static void set_stdout(struct rzip_control *control)
+{
+	control->flags |= FLAG_STDOUT;
+	control->outFILE = stdout;
+	control->msgout = stderr;
+	register_outputfile(control, control->msgout);
+}
+
+/* Recursively enter all directories, adding all regular files to the dirlist array */
+static void recurse_dirlist(char *indir, char **dirlist, int *entries)
+{
+	char fname[MAX_PATH_LEN];
+	struct stat istat;
+	struct dirent *dp;
+	DIR *dirp;
+
+	dirp = opendir(indir);
+	if (unlikely(!dirp))
+		failure("Unable to open directory %s\n", indir);
+	while ((dp = readdir(dirp)) != NULL) {
+		if (!strcmp(dp->d_name, ".") || !strcmp(dp->d_name, ".."))
+			continue;
+		sprintf(fname, "%s/%s", indir, dp->d_name);
+		if (unlikely(stat(fname, &istat)))
+			failure("Unable to stat file %s\n", fname);
+		if (S_ISDIR(istat.st_mode)) {
+			recurse_dirlist(fname, dirlist, entries);
+			continue;
+		}
+		if (!S_ISREG(istat.st_mode)) {
+			print_err("Not regular file %s\n", fname);
+			continue;
+		}
+		print_maxverbose("Added file %s\n", fname);
+		*dirlist = realloc(*dirlist, MAX_PATH_LEN * (*entries + 1));
+		strcpy(*dirlist + MAX_PATH_LEN * (*entries)++, fname);
+	}
+	closedir(dirp);
+}
+
+static const char *loptions = "bcCdDefghHiKlL:nN:o:O:p:PqQrS:tTUm:vVw:z?";
+static const char *coptions = "bcCdefghHikKlLnN:o:O:p:PrS:tTUm:vVw:z?123456789";
+
 int main(int argc, char *argv[])
 {
+	bool lrzcat = false, compat = false, recurse = false;
+	bool options_file = false, conf_file_compression_set = false; /* for environment and tracking of compression setting */
 	struct timeval start_time, end_time;
 	struct sigaction handler;
 	double seconds,total_time; // for timers
-	bool lrzcat = false;
+	bool nice_set = false;
 	int c, i;
 	int hours,minutes;
 	extern int optind;
-	char *eptr; /* for environment */
+	char *eptr, *av; /* for environment */
+	char *endptr = NULL;
 
         control = &base_control;
 
-	initialize_control(control);
+	initialise_control(control);
 
-	if (strstr(argv[0], "lrunzip"))
+	av = basename(argv[0]);
+	control->flags |= FLAG_OUTPUT;
+	if (!strcmp(av, "lrunzip"))
 		control->flags |= FLAG_DECOMPRESS;
-	else if (strstr(argv[0], "lrzcat")) {
+	else if (!strcmp(av, "lrzcat")) {
 		control->flags |= FLAG_DECOMPRESS | FLAG_STDOUT;
 		lrzcat = true;
+	} else if (!strcmp(av, "lrz")) {
+		/* Called in gzip compatible command line mode */
+		control->flags &= ~FLAG_SHOW_PROGRESS;
+		control->flags &= ~FLAG_KEEP_FILES;
+		compat = true;
+		long_options[1].name = "stdout";
+		long_options[11].name = "keep";
 	}
 
 	/* generate crc table */
@@ -212,21 +345,57 @@ int main(int argc, char *argv[])
 	/* Get Preloaded Defaults from lrzip.conf
 	 * Look in ., $HOME/.lrzip/, /etc/lrzip.
 	 * If LRZIP=NOCONFIG is set, then ignore config
+	 * If lrzip.conf sets a compression mode, options_file will be true.
+	 * This will allow for a test to permit an override of compression mode.
+	 * If there is an override, then all compression settings will be reset
+	 * and command line switches will prevail, including for --lzma.
 	 */
 	eptr = getenv("LRZIP");
 	if (eptr == NULL)
-		read_config(control);
+		options_file = read_config(control);
 	else if (!strstr(eptr,"NOCONFIG"))
-		read_config(control);
+		options_file = read_config(control);
+	if (options_file && (control->flags & FLAG_NOT_LZMA))		/* if some compression set in lrzip.conf    */
+		conf_file_compression_set = true;			/* need this to allow command line override */
 
-	while ((c = getopt(argc, argv, "bcdDefghHiklL:nN:o:O:p:qS:tTUvVw:z?")) != -1) {
+	while ((c = getopt_long(argc, argv, compat ? coptions : loptions, long_options, &i)) != -1) {
 		switch (c) {
 		case 'b':
-			if (control->flags & FLAG_NOT_LZMA)
+		case 'g':
+		case 'l':
+		case 'n':
+		case 'z':
+			/* If some compression was chosen in lrzip.conf, allow this one time
+			 * because conf_file_compression_set will be true
+			 */
+			if ((control->flags & FLAG_NOT_LZMA) && conf_file_compression_set == false)
 				failure("Can only use one of -l, -b, -g, -z or -n\n");
-			control->flags |= FLAG_BZIP2_COMPRESS;
+			/* Select Compression Mode */
+			control->flags &= ~FLAG_NOT_LZMA; /* must clear all compressions first */
+			if (c == 'b')
+				control->flags |= FLAG_BZIP2_COMPRESS;
+			else if (c == 'g')
+				control->flags |= FLAG_ZLIB_COMPRESS;
+			else if (c == 'l')
+				control->flags |= FLAG_LZO_COMPRESS;
+			else if (c == 'n')
+				control->flags |= FLAG_NO_COMPRESS;
+			else if (c == 'z')
+				control->flags |= FLAG_ZPAQ_COMPRESS;
+			/* now FLAG_NOT_LZMA will evaluate as true */
+			conf_file_compression_set = false;
+			break;
+		case '/':							/* LZMA Compress selected */
+			control->flags &= ~FLAG_NOT_LZMA;			/* clear alternate compression flags */
 			break;
 		case 'c':
+			if (compat) {
+				control->flags |= FLAG_KEEP_FILES;
+				set_stdout(control);
+				break;
+			}
+			/* FALLTHRU */
+		case 'C':
 			control->flags |= FLAG_CHECK;
 			control->flags |= FLAG_HASH;
 			break;
@@ -238,47 +407,54 @@ int main(int argc, char *argv[])
 			break;
 		case 'e':
 			control->flags |= FLAG_ENCRYPT;
+			control->passphrase = optarg;
 			break;
 		case 'f':
 			control->flags |= FLAG_FORCE_REPLACE;
 			break;
-		case 'g':
-			if (control->flags & FLAG_NOT_LZMA)
-				failure("Can only use one of -l, -b, -g, -z or -n\n");
-			control->flags |= FLAG_ZLIB_COMPRESS;
-			break;
 		case 'h':
-		case '?':
-			usage();
-			return -1;
+			usage(compat);
+			exit(0);
+			break;
 		case 'H':
 			control->flags |= FLAG_HASH;
 			break;
 		case 'i':
 			control->flags |= FLAG_INFO;
+			control->flags &= ~FLAG_DECOMPRESS;
 			break;
 		case 'k':
+			if (compat) {
+				control->flags |= FLAG_KEEP_FILES;
+				break;
+			}
+			/* FALLTHRU */
+		case 'K':
 			control->flags |= FLAG_KEEP_BROKEN;
 			break;
-		case 'l':
-			if (control->flags & FLAG_NOT_LZMA)
-				failure("Can only use one of -l, -b, -g, -z or -n\n");
-			control->flags |= FLAG_LZO_COMPRESS;
-			break;
 		case 'L':
-			control->compression_level = atoi(optarg);
+			if (compat) {
+				license();
+				exit(0);
+			}
+			control->compression_level = strtol(optarg, &endptr, 10);
 			if (control->compression_level < 1 || control->compression_level > 9)
 				failure("Invalid compression level (must be 1-9)\n");
+			if (*endptr)
+				failure("Extra characters after compression level: \'%s\'\n", endptr);
 			break;
-		case 'n':
-			if (control->flags & FLAG_NOT_LZMA)
-				failure("Can only use one of -l, -b, -g, -z or -n\n");
-			control->flags |= FLAG_NO_COMPRESS;
+		case 'm':
+			control->ramsize = strtol(optarg, &endptr, 10) * 1024 * 1024 * 100;
+			if (*endptr)
+				failure("Extra characters after ramsize: \'%s\'\n", endptr);
 			break;
 		case 'N':
-			control->nice_val = atoi(optarg);
-			if (control->nice_val < -20 || control->nice_val > 19)
-				failure("Invalid nice value (must be -20..19)\n");
+			nice_set = true;
+			control->nice_val = strtol(optarg, &endptr, 10);
+			if (control->nice_val < PRIO_MIN || control->nice_val > PRIO_MAX)
+				failure("Invalid nice value (must be %d...%d)\n", PRIO_MIN, PRIO_MAX);
+			if (*endptr)
+				failure("Extra characters after nice level: \'%s\'\n", endptr);
 			break;
 		case 'o':
 			if (control->outdir)
@@ -301,13 +477,25 @@ int main(int argc, char *argv[])
 				strcat(control->outdir, "/");
 			break;
 		case 'p':
-			control->threads = atoi(optarg);
+			control->threads = strtol(optarg, &endptr, 10);
 			if (control->threads < 1)
 				failure("Must have at least one thread\n");
+			if (*endptr)
+				failure("Extra characters after number of threads: \'%s\'\n", endptr);
+			break;
+		case 'P':
+			control->flags |= FLAG_SHOW_PROGRESS;
 			break;
 		case 'q':
 			control->flags &= ~FLAG_SHOW_PROGRESS;
 			break;
+		case 'Q':
+			control->flags &= ~FLAG_SHOW_PROGRESS;
+			control->flags &= ~FLAG_OUTPUT;
+			break;
+		case 'r':
+			recurse = true;
+			break;
 		case 'S':
 			if (control->outname)
 				failure("Specified output filename already, can't specify an extension.\n");
@@ -318,6 +506,8 @@ int main(int argc, char *argv[])
 		case 't':
 			if (control->outname)
 				failure("Cannot specify an output file name when just testing.\n");
+			if (compat)
+				control->flags |= FLAG_KEEP_FILES;
 			if (!KEEP_FILES)
 				failure("Doubt that you want to delete a file when just testing.\n");
 			control->flags |= FLAG_TEST_ONLY;
@@ -330,7 +520,9 @@ int main(int argc, char *argv[])
 			break;
 		case 'v':
 			/* set verbosity flag */
-			if (!(control->flags & FLAG_VERBOSITY) && !(control->flags & FLAG_VERBOSITY_MAX))
+			if (!(control->flags & FLAG_SHOW_PROGRESS))
+				control->flags |= FLAG_SHOW_PROGRESS;
+			else if (!(control->flags & FLAG_VERBOSITY) && !(control->flags & FLAG_VERBOSITY_MAX))
 				control->flags |= FLAG_VERBOSITY;
 			else if ((control->flags & FLAG_VERBOSITY)) {
 				control->flags &= ~FLAG_VERBOSITY;
@@ -338,25 +530,46 @@ int main(int argc, char *argv[])
 			}
 			break;
 		case 'V':
+			control->msgout = stdout;
 			print_output("lrzip version %s\n", PACKAGE_VERSION);
 			exit(0);
 			break;
 		case 'w':
-			control->window = atol(optarg);
+			control->window = strtol(optarg, &endptr, 10);
+			if (control->window < 1)
+				failure("Window must be positive\n");
+			if (*endptr)
+				failure("Extra characters after window size: \'%s\'\n", endptr);
 			break;
-		case 'z':
-			if (control->flags & FLAG_NOT_LZMA)
-				failure("Can only use one of -l, -b, -g, -z or -n\n");
-			control->flags |= FLAG_ZPAQ_COMPRESS;
+		case '1':
+		case '2':
+		case '3':
+		case '4':
+		case '5':
+		case '6':
+		case '7':
+		case '8':
+		case '9':
+			control->compression_level = c - '0';
 			break;
+		default:
+			usage(compat);
+			return 2;
 		}
 	}
 
+	if (compat && !SHOW_PROGRESS)
+		control->flags &= ~FLAG_OUTPUT;
+
 	argc -= optind;
 	argv += optind;
 
-	if (control->outname && argc > 1)
-		failure("Cannot specify output filename with more than 1 file\n");
+	if (control->outname) {
+		if (argc > 1)
+			failure("Cannot specify output filename with more than 1 file\n");
+		if (recurse)
+			failure("Cannot specify output filename with recursive\n");
+	}
 
 	if (VERBOSE && !SHOW_PROGRESS) {
 		print_err("Cannot have -v and -q options. -v wins.\n");
@@ -380,45 +593,75 @@ int main(int argc, char *argv[])
 
 	/* Set the main nice value to half that of the backend threads since
 	 * the rzip stage is usually the rate limiting step */
-	if (control->nice_val > 0 && !NO_COMPRESS) {
-		if (unlikely(setpriority(PRIO_PROCESS, 0, control->nice_val / 2) == -1))
-			print_err("Warning, unable to set nice value\n");
-	} else {
-		if (unlikely(setpriority(PRIO_PROCESS, 0, control->nice_val) == -1))
-			print_err("Warning, unable to set nice value\n");
+	control->current_priority = getpriority(PRIO_PROCESS, 0);
+	if (nice_set) {
+		if (!NO_COMPRESS) {
+			/* If niceness can't be set. just reset process priority */
+			if (unlikely(setpriority(PRIO_PROCESS, 0, control->nice_val/2) == -1)) {
+				print_err("Warning, unable to set nice value %d...Resetting to %d\n",
+					control->nice_val, control->current_priority);
+				setpriority(PRIO_PROCESS, 0, (control->nice_val=control->current_priority));
+			}
+		} else {
+			if (unlikely(setpriority(PRIO_PROCESS, 0, control->nice_val) == -1)) {
+				print_err("Warning, unable to set nice value %d...Resetting to %d\n",
+					control->nice_val, control->current_priority);
+				setpriority(PRIO_PROCESS, 0, (control->nice_val=control->current_priority));
+			}
+		}
 	}
 
 	/* One extra iteration for the case of no parameters means we will default to stdin/out */
 	for (i = 0; i <= argc; i++) {
+		char *dirlist = NULL, *infile = NULL;
+		int direntries = 0, curentry = 0;
+
 		if (i < argc)
-			control->infile = argv[i];
+			infile = argv[i];
 		else if (!(i == 0 && STDIN))
 			break;
-		if (control->infile) {
-			if ((strcmp(control->infile, "-") == 0))
+		if (infile) {
+			if ((strcmp(infile, "-") == 0))
 				control->flags |= FLAG_STDIN;
 			else {
-				struct stat infile_stat;
+				bool isdir = false;
+				struct stat istat;
 
-				stat(control->infile, &infile_stat);
-				if (unlikely(S_ISDIR(infile_stat.st_mode)))
-					failure("lrzip only works directly on FILES.\n"
-					"Use lrztar or pipe through tar for compressing directories.\n");
+				if (unlikely(stat(infile, &istat)))
+					failure("Failed to stat %s\n", infile);
+				isdir = S_ISDIR(istat.st_mode);
+				if (!recurse && (isdir || !S_ISREG(istat.st_mode))) {
+					failure("lrzip only works directly on regular FILES.\n"
+					"Use -r recursive, lrztar or pipe through tar for compressing directories.\n");
+				}
+				if (recurse && !isdir)
+					failure("%s not a directory, -r recursive needs a directory\n", infile);
 			}
 		}
 
+		if (recurse) {
+			if (unlikely(STDIN || STDOUT))
+				failure("Cannot use -r recursive with STDIO\n");
+			recurse_dirlist(infile, &dirlist, &direntries);
+		}
+
 		if (INFO && STDIN)
 			failure("Will not get file info from STDIN\n");
-
-		if ((control->outname && (strcmp(control->outname, "-") == 0)) ||
-			/* If no output filename is specified, and we're using
-			 * stdin, use stdout */
-			(!control->outname && STDIN) || lrzcat ) {
-				control->flags |= FLAG_STDOUT;
-				control->outFILE = stdout;
-				control->msgout = stderr;
-				register_outputfile(control, control->msgout);
+recursion:
+		if (recurse) {
+			if (curentry >= direntries) {
+				infile = NULL;
+				continue;
+			}
+			infile = dirlist + MAX_PATH_LEN * curentry++;
 		}
+		control->infile = infile;
+
+		/* If no output filename is specified, and we're using
+		 * stdin, use stdout */
+		if ((control->outname && (strcmp(control->outname, "-") == 0)) ||
+		    (!control->outname && STDIN) || lrzcat)
+				set_stdout(control);
 
 		if (lrzcat) {
 			control->msgout = stderr;
@@ -433,7 +676,10 @@ int main(int argc, char *argv[])
 
 		if (STDIN)
 			control->inFILE = stdin;
+
 		/* Implement signal handler only once flags are set */
+		sigemptyset(&handler.sa_mask);
+		handler.sa_flags = 0;
 		handler.sa_handler = &sighandler;
 		sigaction(SIGTERM, &handler, 0);
 		sigaction(SIGINT, &handler, 0);
@@ -441,12 +687,12 @@ int main(int argc, char *argv[])
 		if (!FORCE_REPLACE) {
 			if (STDIN && isatty(fileno((FILE *)stdin))) {
 				print_err("Will not read stdin from a terminal. Use -f to override.\n");
-				usage();
+				usage(compat);
 				exit (1);
 			}
-			if (!TEST_ONLY && STDOUT && isatty(fileno((FILE *)stdout))) {
+			if (!TEST_ONLY && STDOUT && isatty(fileno((FILE *)stdout)) && !compat) {
 				print_err("Will not write stdout to a terminal. Use -f to override.\n");
-				usage();
+				usage(compat);
 				exit (1);
 			}
 		}
@@ -466,8 +712,8 @@ int main(int argc, char *argv[])
 
 		gettimeofday(&start_time, NULL);
 
-		if (unlikely(STDIN && ENCRYPT))
-			failure("Unable to work from STDIN while reading password\n");
+		if (!control->passphrase && (unlikely((STDIN || STDOUT) && ENCRYPT)))
+			failure("Unable to work from STDIO while reading password\n");
 
 		memcpy(&local_control, &base_control, sizeof(rzip_control));
 		if (DECOMPRESS || TEST_ONLY)
@@ -485,7 +731,9 @@ int main(int argc, char *argv[])
 		minutes = (int)(total_time / 60) % 60;
 		seconds = total_time - hours * 3600 - minutes * 60;
 		if (!INFO)
-			print_progress("Total time: %02d:%02d:%05.2f\n", hours, minutes, seconds);
+			print_output("Total time: %02d:%02d:%05.2f\n", hours, minutes, seconds);
+		if (recurse)
+			goto recursion;
 	}
 
 	return 0;

man/Makefile.am

@@ -1,12 +1,12 @@
-MAINTAINERCLEANFILES = Makefile.in lrunzip.1 lrztar.1 lrzuntar.1
+MAINTAINERCLEANFILES = Makefile.in lrunzip.1 lrztar.1 lrzuntar.1 lrz.1
 
-man1_MANS = lrzip.1 lrunzip.1 lrzcat.1 lrztar.1 lrzuntar.1
+man1_MANS = lrzip.1 lrunzip.1 lrzcat.1 lrztar.1 lrzuntar.1 lrz.1
 man5_MANS = lrzip.conf.5
 
-BUILT_SOURCES = lrunzip.1 lrzcat.1 lrztar.1 lrzuntar.1
+BUILT_SOURCES = lrunzip.1 lrzcat.1 lrztar.1 lrzuntar.1 lrz.1
 CLEANFILES = $(BUILT_SOURCES)
 
-EXTRA_DIST = lrzip.1 lrunzip.1.pod lrzcat.1.pod lrztar.1.pod lrzuntar.1.pod $(man5_MANS)
+EXTRA_DIST = lrzip.1 lrunzip.1.pod lrzcat.1.pod lrztar.1.pod lrzuntar.1.pod lrz.1.pod $(man5_MANS)
 
 SUFFIXES = .1 .1.pod
 .1.pod.1:

man/lrunzip.1.pod

@@ -1,6 +1,6 @@
 #   Copyright
 #
-#      Copyright (C) 2010-2011 Con Kolivas
+#      Copyright (C) 2010-2016 Con Kolivas
 #      Copyright (C) 2009-2009 Jari Aalto
 #
 #   License
@@ -70,6 +70,7 @@ lrzip(1),
 lrzcat(1),
 lrztar(1),
 lrzuntar(1),
+lrz(1),
 bzip2(1),
 gzip(1),
 lzop(1),

man/lrz.1.pod

@@ -0,0 +1,346 @@
+#!/usr/bin/perl -w
+
+#   Copyright
+#
+#      Copyright (C) 2021 Con Kolivas
+#
+#   License
+#
+#       This program is free software; you can redistribute it and/or modify
+#       it under the terms of the GNU General Public License as published by
+#       the Free Software Foundation; either version 2 of the License, or
+#       (at your option) any later version.
+#
+#       This program is distributed in the hope that it will be useful,
+#       but WITHOUT ANY WARRANTY; without even the implied warranty of
+#       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+#       GNU General Public License for more details.
+#
+#       You should have received a copy of the GNU General Public License
+#       along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+#   Description
+#
+#	To learn what TOP LEVEL section to use in manual pages,
+#	see POSIX/Susv standard and "tility Description Defaults" at
+#	http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap01.html#tag_01_11
+#
+#	This is manual page in Perl POD format. Read more at
+#	http://perldoc.perl.org/perlpod.html or run command:
+#
+#	    perldoc perlpod | less
+#
+#	To check the syntax:
+#
+#	    podchecker *.pod
+#
+#	Create manual page with command:
+#
+#	    pod2man PAGE.N.pod > PAGE.N
+
+=pod
+
+=encoding utf8
+
+=head1 NAME
+
+lrz - gzip compatible command line variant of lrzip
+
+
+=head1 SYNOPSIS
+
+B<lrz> [options] I<file>
+
+
+=head1 DESCRIPTION
+
+B<lrz> is identical to the B<lrzip> application, however, its command
+line options and behaviour are made to be as compatible with B<gzip>
+as possible.
+
+=head1 OPTIONS
+
+=head2 General options
+
+=over 9
+
+=item B<--stdout>
+
+=item B<-c>
+
+Output to STDOUT.
+
+
+=item B<--check>
+
+=item B<-C>
+
+Check integrity of file written on decompression.
+
+
+=item B<--decompress>
+
+=item B<-d>
+
+Decompress.
+
+
+=item B<--encrypt>[=I<password>]
+
+=item B<-e>
+
+Password protect sha512/aes128 encryption on compression.
+
+
+=item B<--help>
+
+=item B<-h>
+
+=item B<-?>
+
+Show help.
+
+
+=item B<--hash>
+
+=item B<-H>
+
+Display md5 hash integrity information.
+
+
+=item B<--info>
+
+=item B<-i>
+
+Show compressed file information.
+
+
+=item B<--license>
+
+=item B<-L>
+
+Display software version and license.
+
+
+=item B<--progress>
+
+=item B<-P>
+
+Show compression progress.
+
+
+=item B<--recursive>
+
+=item B<-r>
+
+Operate recursively on directories.
+
+
+=item B<--test>
+
+=item B<-t>
+
+Test compressed file integrity.
+
+
+=item B<--verbose>
+
+=item B<-v[vv]>
+
+Increase verbosity.
+
+
+=item B<--version>
+
+=item B<-V>
+
+Show version.
+
+
+=back
+
+
+=head2 Options affecting output
+
+=over 9
+
+=item B<--force>
+
+=item B<-f>
+
+Force overwrite of any existing files.
+
+
+=item B<--keep>
+
+=item B<-k>
+
+Don't delete source files on de/compression.
+
+
+=item B<--keep-broken>
+
+=item B<-K>
+
+Keep broken or damaged output files.
+
+
+=item B<--outfile>  I<name>
+
+=item B<-o> I<name>
+
+Specify the output file name and/or path.
+
+
+=item B<--outdir> I<dir>
+
+=item B<-O> I<dir>
+
+Specify the output directory when B<-o> is not used.
+
+
+=item B<--suffix> I<suffix>
+
+=item B<-S> I<suffix>
+
+Specify compressed suffix (default '.lrz').
+
+
+=back
+
+=head2 Options affecting compression
+
+=over 9
+
+=item B<--bzip2>
+
+=item B<-b>
+
+Bzip2 compression.
+
+=item B<--gzip>
+
+=item B<-g>
+
+Gzip compression using zlib.
+
+=item B<--lzo>
+
+=item B<-l>
+
+Lzo compression (ultra fast).
+
+=item B<--lzma>
+
+Lzma compression (default).
+
+=item B<--no-compress>
+
+=item B<-n>
+
+No backend compression - prepare for other compressor.
+
+=item B<--zpaq>
+
+=item B<-z>
+
+Zpaq compression (best, extreme compression, extremely slow).
+
+
+=back
+
+=head2 Low level options
+
+=over 9
+
+=item B<-1> .. B<-9>
+
+=item B<--level> I<level>
+
+=item B<-L> I<level>
+
+Set lzma/bzip2/gzip compression level (1-9, default 7).
+
+
+=item B<--fast>
+
+Alias for B<-1>.
+
+
+=item B<--best>
+
+Alias for B<-9>.
+
+
+=item B<--nice-level> I<value>
+
+=item B<-N> I<value>
+
+Set nice value to I<value> (default 0).
+
+
+=item B<--threads> I<value>
+
+=item B<-P> I<value>
+
+Set processor count to override number of threads.
+
+=item B<--maxram> I<size>
+
+=item B<-m> I<size>
+
+Set maximum available ram as I<size> * 100 MB.
+Overrides detected amount of available ram.
+
+=item B<--threshold>
+
+=item B<-T>
+
+Disable LZ4 compressibility testing.
+
+=item B<--unlimited>
+
+=item B<-U>
+
+Use unlimited window size beyond ramsize (potentially much slower).
+
+=item B<--window> I<size>
+
+=item B<-w> I<size>
+
+Set maximum compression window as I<size> * 100 MB.
+Default chosen by heuristic dependent on ram and chosen compression.
+
+=back
+
+See also lrzip(1)
+
+=head1 ENVIRONMENT
+
+lrz uses the same environment and configuration files as lrzip(1)
+
+=head1 FILES
+
+See lrzip(1)
+
+=head1 SEE ALSO
+
+lrzip.conf(5),
+lrzip(1),
+lrunzip(1),
+lrztar(1),
+lrzuntar(1),
+bzip2(1),
+gzip(1),
+lzop(1),
+rzip(1),
+zip(1)
+
+=head1 AUTHORS
+
+This manual page was written by Con Kolivas <kernel@kolivas.org> (but
+may be used by others). Released under license GNU GPL version 2 or (at
+your option) any later version. For more information about license,
+visit <http://www.gnu.org/copyleft/gpl.html>.
+
+=cut

man/lrzcat.1.pod

@@ -1,6 +1,6 @@
 #   Copyright
 #
-#      Copyright (C) 2011 Con Kolivas
+#      Copyright (C) 2011-2016 Con Kolivas
 #
 #   License
 #
@@ -69,6 +69,7 @@ lrzip(1),
 lrunzip(1),
 lrztar(1),
 lrzuntar(1),
+lrz(1),
 bzip2(1),
 gzip(1),
 lzop(1),

man/lrzip.1

@@ -1,4 +1,4 @@
-.TH "lrzip" "1" "May 2011" "" ""
+.TH "lrzip" "1" "February 2022" "" ""
 .SH "NAME"
 lrzip \- a large-file compression program
 .SH "SYNOPSIS"
@@ -17,6 +17,8 @@ lrztar \-d [lrzip options] <directory>
 .br
 lrzuntar [lrzip options] <directory>
 .br
+lrz [lrz options] <directory>
+.br
 LRZIP=NOCONFIG [lrzip|lrunzip] [OPTIONS] <file>
 .PP
 .SH "DESCRIPTION"
@@ -34,37 +36,41 @@ Here is a summary of the options to lrzip\&.
 
 
 General options:
-  \-c            check integrity of file written on decompression
-  \-d            decompress
-  \-e            password protected sha512/aes128 encryption on compression
-  \-h|\-?         show help
-  \-H            display md5 hash integrity information
-  \-i            show compressed file information
-  \-q            don't show compression progress
-  \-t            test compressed file integrity
-  \-v[v]         Increase verbosity
-  \-V            show version
+ \-c, \-\-check             check integrity of file written on decompression
+ \-d, \-\-decompress        decompress
+ \-e, \-\-encrypt[=password] password protected sha512/aes128 encryption on compression
+ \-h, \-?, \-\-help          show help
+ \-H, \-\-hash              display md5 hash integrity information
+ \-i, \-\-info              show compressed file information
+ \-q, \-\-quiet             don't show compression progress
+ \-Q, \-\-very-quiet        don't show any output
+ \-r, \-\-recursive         operate recursively on directories
+ \-t, \-\-test              test compressed file integrity
+ \-v[v], \-\-verbose        Increase verbosity
+ \-V, \-\-version           show version
 Options affecting output:
-  \-D            delete existing files
-  \-f            force overwrite of any existing files
-  \-k            keep broken or damaged output files
-  \-o filename   specify the output file name and/or path
-  \-O directory  specify the output directory when \-o is not used
-  \-S suffix     specify compressed suffix (default '.lrz')
+ \-D, \-\-delete            delete existing files
+ \-f, \-\-force             force overwrite of any existing files
+ \-k, \-\-keep-broken       keep broken or damaged output files
+ \-o, \-\-outfile filename  specify the output file name and/or path
+ \-O, \-\-outdir directory  specify the output directory when -o is not used
+ \-S, \-\-suffix suffix     specify compressed suffix (default '.lrz')
 Options affecting compression:
-  \-b            bzip2 compression
-  \-g            gzip compression using zlib
-  \-l            lzo compression (ultra fast)
-  \-n            no backend compression - prepare for other compressor
-  \-z            zpaq compression (best, extreme compression, extremely slow)
+ \-b, \-\-bzip2             bzip2 compression
+ \-g, \-\-gzip              gzip compression using zlib
+ \-l, \-\-lzo               lzo compression (ultra fast)
+ \-n, \-\-no-compress       no backend compression - prepare for other compressor
+ \-z, \-\-zpaq              zpaq compression (best, extreme compression, extremely slow)
 Low level options:
-  \-L level      set lzma/bzip2/gzip compression level (1\-9, default 7)
-  \-N value      Set nice value to value (default 19)
-  \-p value      Set processor count to override number of threads
-  \-T            Disable LZO compressibility testing
-  \-U            Use unlimited window size beyond ramsize (potentially much slower)
-  \-w size       maximum compression window in hundreds of MB
-                default chosen by heuristic dependent on ram and chosen compression
+ \-L, \-\-level level       set lzma/bzip2/gzip compression level (1-9, default 7)
+ \-N, \-\-nice-level value  Set nice value to value (default 19)
+ \-p, \-\-threads value     Set processor count to override number of threads
+ \-m, \-\-maxram size       Set maximum available ram in hundreds of MB
+                         overrides detected amount of available ram
+ \-T, \-\-threshold         Disable LZ4 compressibility testing
+ \-U, \-\-unlimited         Use unlimited window size beyond ramsize (potentially much slower)
+ \-w, \-\-window size       maximum compression window in hundreds of MB
+                         default chosen by heuristic dependent on ram and chosen compression
 
 LRZIP=NOCONFIG environment variable setting can be used to bypass lrzip.conf.
 TMP environment variable will be used for storage of temporary files when needed.
@@ -97,6 +103,7 @@ the name used to launch the program. If it contains the string
 "lrzcat" then the \-d \-o \- options are automatically set.
 .IP
 .IP "\fB-e\fP"
+.IP "\fB\-\-encrypt\fP[=\fIpassword\fP]"
 Encrypt. This option enables high grade password encryption using a combination
 of multiply sha512 hashed password, random salt and aes128 CBC encryption.
 Passwords up to 500 characters long are supported, and the encryption mechanism
@@ -136,6 +143,17 @@ bursts with lzma compression which is the default compression. This means
 that it will progress very rapidly for short periods and then stop for
 long periods.
 .IP
+.IP "\fB-Q\fP"
+If this option is specified then lrzip will not show any output to the console
+except for error messages.
+.IP
+.IP "\fB-r\fP"
+If this option is specified, lrzip will recursively enter the directories
+specified, compressing or decompressing every file individually in the same
+directory. Note for better compression it is recommended to instead combine
+files in a tar file rather than compress them separately, either manually
+or with the lrztar helper.
+.IP
 .IP "\fB-t\fP"
 This tests the compressed file integrity. It does this by decompressing it
 to a temporary file and then deleting it.
@@ -227,13 +245,13 @@ decrease the load on your machine, or to improve compression. Setting it to
 1 will maximise compression but will not attempt to use more than one CPU.
 .IP
 .IP "\fB-T\fP"
-Disables the LZO compressibility threshold testing when a slower compression
-back-end is used. LZO testing is normally performed for the slower back-end
+Disables the LZ4 compressibility threshold testing when a slower compression
+back-end is used. LZ4 testing is normally performed for the slower back-end
 compression of LZMA and ZPAQ. The reasoning is that if it is completely
-incompressible by LZO then it will also be incompressible by them. Thus if a
-block fails to be compressed by the very fast LZO, lrzip will not attempt to
+incompressible by LZ4 then it will also be incompressible by them. Thus if a
+block fails to be compressed by the very fast LZ4, lrzip will not attempt to
 compress that block with the slower compressor, thereby saving time. If this
-option is enabled, it will bypass the LZO testing and attempt to compress each
+option is enabled, it will bypass the LZ4 testing and attempt to compress each
 block regardless.
 .IP
 .IP "\fB-U \fP"
@@ -329,12 +347,19 @@ lrunzip(1),
 lrzcat(1),
 lrztar(1),
 lrzuntar(1),
+lrz(1),
 bzip2(1),
 gzip(1),
 lzop(1),
 rzip(1),
 zip(1)
 
+.PP
+.SH "DIAGNOSTICS"
+.PP
+Exit status is normally 0; if an error occurs, exit status is 1, usage errors
+is 2.
+
 .PP
 .SH "AUTHOR and CREDITS"
 .br
@@ -351,8 +376,9 @@ zpaq was written by Matt Mahoney.
 Peter Hyman added informational output, updated LZMA SDK,
 and added lzma multi-threading capabilities.
 .PP
-If you wish to report a problem, or make a suggestion, then please email Con at
-kernel@kolivas.org
+If you wish to report a problem, or make a suggestion, then please consult the
+git repository at:
+https://github.com/ckolivas/lrzip
 .PP
 lrzip is released under the GNU General Public License version 2.
 Please see the file COPYING for license details.

man/lrzip.conf.5

@@ -1,4 +1,4 @@
-.TH "lrzip.conf" "5" "January 2009" "" ""
+.TH "lrzip.conf" "5" "January 2009, updated May 2019" "" ""
 .SH "NAME"
 lrzip.conf \- Configuration File for lrzip
 .SH "DESCRIPTION"
@@ -13,54 +13,63 @@ three places\&:
 .nf
 $PWD \- Current Directory
 /etc/lrzip
-$HOME/\&./lrzip
+$HOME/\&.lrzip
 .PP
 Parameters are set in \fBPARAMETER\&=VALUE\fP fashion where any line
 beginning with a \fB#\fP or that is blank will be ignored\&.
-Parameter values are not case sensitive\&.
+Parameter values are not case sensitive except where specified\&.
 .PP
 .SH "CONFIG FILE EXAMPLE"
 .nf
 # This is a comment.
 
-# Compression Window size in 100MB. Normally selected by program.
-# WINDOW = 5
-# Compression Level 1-9 (7 Default).
+# Compression Window size in 100MB. Normally selected by program. (-w)
+# WINDOW = 20
+# Compression Level 1-9 (7 Default). (-L)
 # COMPRESSIONLEVEL = 7
-# Unlimited Ram Compression
-# UNLIMITED = YES
-# Compression Method, rzip, gzip, bzip2, lzo, or lzma (default), zpaq.
-# COMPRESSIONMETHOD = LZMA
-# Perform LZO Test. Default = YES (\-T option, NO)
+# Use -U setting, Unlimited ram. Yes or No
+# UNLIMITED = NO
+# Compression Method, rzip, gzip, bzip2, lzo, or lzma (default), or zpaq. (-n -g -b -l --lzma -z)
+# If specified here, command line options not usable.
+# COMPRESSIONMETHOD = lzma
+# Perform LZO Test. Default = YES (-T )
 # LZOTEST = NO
-# Hash Check on decompression, YES
+# Hash Check on decompression, (-c)
 # HASHCHECK = YES
-# Show HASH value on Compression even if Verbose is off, YES
+# Show HASH value on Compression even if Verbose is off, YES (-H)
 # SHOWHASH = YES
 
-# Default output directory
+# Default output directory (-O)
 # OUTPUTDIRECTORY = location
-# Verbosity, Yes or Max
-# VERBOSITY = MAX
-# Show Progress as file is parsed, YES, NO (yes is default)
-# SHOWPROGRESS = NO
-# Set Niceness. 19 is default. \-20 to 19 is the allowable range
+# Verbosity, YES or MAX (v, vv)
+# VERBOSITY = max
+# Show Progress as file is parsed, YES or no (NO = -q option)
+# SHOWPROGRESS = YES
+
+# Set Niceness. 19 is default. -20 to 19 is the allowable range (-N)
 # NICE = 19
 
-# Keep broken or damaged output files, YES
+# Keep broken or damaged output files, YES (-K)
 # KEEPBROKEN = YES
-# Delete source file after compression
+
+# Delete source file after compression (-D)
 # this parameter and value are case sensitive
 # value must be YES to activate
+
 # DELETEFILES = NO
 
-# Replace existing lrzip file when compressing
+# Replace existing lrzip file when compressing (-f)
 # this parameter and value are case sensitive
 # value must be YES to activate
-# REPLACEFILE = NO
 
-# Select Temporary Directory when stdin/stdout or Test file is used
+# REPLACEFILE = YES 
+
+# Override for Temporary Directory. Only valid when stdin/out or Test is used
 # TMPDIR = /tmp
+
+# Whether to use encryption on compression YES, NO (-e)
+# ENCRYPT = NO
+
 .fi
 .PP
 .SH "NOTES"

man/lrztar.1.pod

@@ -1,6 +1,6 @@
 #   Copyright
 #
-#      Copyright (C) 2010-2011 Con Kolivas
+#      Copyright (C) 2010-2016 Con Kolivas
 #      Copyright (C) 2009-2010 Jari Aalto
 #
 #   License
@@ -74,6 +74,7 @@ lrzuntar(1),
 lrzip(1),
 lrunzip(1),
 lrzcat(1),
+lrz(1),
 bzip2(1),
 gzip(1),
 lzop(1),

man/lrzuntar.1.pod

@@ -1,6 +1,6 @@
 #   Copyright
 #
-#      Copyright (C) 2010-2011 Con Kolivas
+#      Copyright (C) 2010-2016 Con Kolivas
 #
 #   License
 #
@@ -52,6 +52,7 @@ lrztar(1),
 lrzip(1),
 lrunzip(1),
 lrzcat(1),
+lrz(1),
 bzip2(1),
 gzip(1),
 lzop(1),

regression_test.sh

@@ -1,239 +0,0 @@
-#!/bin/bash
-#Very basic regression testing does a number of regular compression /
-#decompression / test cycles +/- STDIN +/- STDOUT and with the different
-#compression backends.
-#Run it with 
-# regression_test.sh filename
-#where filename is any random file to test with (big or small depending on
-#what's being tested.
-
-infile=$1
-
-end(){
-	rm -f lrztest lrztest.lrz
-}
-
-if [ ! -e $infile ]; then
-        echo $infile does not exist, exiting
-        exit 1
-fi
-
-if [ -f lrztest ]; then
-	echo lrztest file exists, exiting
-	exit 1
-fi
-
-if [ -f lrztest.lrz ]; then
-	echo lrztest.lrz file exists, exiting
-	exit 1
-fi
-
-trap 'echo "ABORTING";end;exit' 1 2 15
-
-echo testing compression from stdin
-./lrzip -vvlfo lrztest.lrz < $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing compression from stdin
-        end
-	exit 1
-fi
-rm lrztest.lrz
-
-echo testing compression to stdout
-./lrzip -vvlo - $infile > lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing compression to stdout
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing compression from stdin to stdout
-./lrzip -vvl < $infile > lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing compression from stdin to stdout
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing standard compression
-./lrzip -vvlfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing standard compression
-        end
-	exit 1
-fi
-
-echo testing standard decompression
-./lrzip -vvdo lrztest lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest ];then
-	echo FAILED testing standard decompression
-        end
-	exit 1
-fi
-
-rm lrztest
-echo testing standard decompression with file checking
-./lrzip -vvdfco lrztest lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest ];then
-	echo FAILED testing standard decompression with file checking
-        end
-	exit 1
-fi
-
-rm lrztest
-echo testing decompression from stdin
-./lrzip -vvfo lrztest -d < lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest ];then
-	echo FAILED testing decompression from stdin
-        end
-	exit 1
-fi
-
-rm lrztest
-echo testing decompression to stdout
-./lrzip -vvdo - lrztest.lrz > lrztest
-
-if [ $? -ne 0 ] || [ ! -f lrztest ];then
-	echo FAILED testing decompression to stdout
-        end
-	exit 1
-fi
-
-rm lrztest
-echo testing decompression from stdin to stdout
-./lrzip -vvd < lrztest.lrz > lrztest
-
-if [ $? -ne 0 ] || [ ! -f lrztest ];then
-	echo FAILED testing decompression from stdin to stdout
-        end
-	exit 1
-fi
-
-rm lrztest
-echo testing testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing testing
-        end
-	exit 1
-fi
-
-echo testing testing from stdin
-./lrzip -vvt < lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing testing from stdin
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing rzip only compression
-./lrzip -vvnfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing rzip only compression
-        end
-	exit 1
-fi
-
-echo testing rzip only testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing rzip only testing
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing lzma compression
-./lrzip -vvfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing lzma compression
-        end
-	exit 1
-fi
-
-echo testing lzma testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing lzma testing
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing gzip compression
-./lrzip -vvgfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing gzip compression
-        end
-	exit 1
-fi
-
-echo testing gzip testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing gzip testing
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing bzip2 compression
-./lrzip -vvbfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing bzip2 compression
-        end
-	exit 1
-fi
-
-echo testing bzip2 testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing bzip2 testing
-        end
-	exit 1
-fi
-
-rm lrztest.lrz
-echo testing zpaq compression
-./lrzip -vvzfo lrztest.lrz $infile
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing zpaq compression
-        end
-	exit 1
-fi
-
-echo testing zpaq testing
-./lrzip -vvt lrztest.lrz
-
-if [ $? -ne 0 ] || [ ! -f lrztest.lrz ];then
-	echo FAILED testing zpaq testing
-        end
-	exit 1
-fi
-
-end
-
-echo ALL TESTS SUCCESSFUL
-
-exit 0

regressiontest.good

@@ -0,0 +1,49 @@
+Test basic use
+Test decompression in read-only dir
+   1000    1000    3893
+this should be silent
+man page for lrz should exist
+0
+compress stdin to stdout
+Respect $TMPDIR
+   1000    1000    3893
+Decompress in read only dir
+   1000    1000    3893
+Test -cd
+   1000    1000    3893
+Test -cfd should not remove testfile.lrz
+   1000    1000    3893
+testfile.lrz
+Test -1c
+   1002    1002    3975
+Test -r
+t10.lrz
+t1.lrz
+t2.lrz
+t3.lrz
+t4.lrz
+t5.lrz
+t6.lrz
+t7.lrz
+t8.lrz
+t9.lrz
+Test tar compatibility
+t/
+t/t8
+t/t7
+t/t3
+t/t5
+t/t2
+t/t6
+t/t10
+t/t4
+t/t9
+t/t1
+11
+test compress of 1 GB data with parallel --pipe --compress
+1073741824
+test compress of 1 GB with sort --compress-program
+1073741825
+test should not lrz -dc removes file
+OK
+testfile.lrz

regressiontest.sh

@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Regression test.
+#
+# Copyright (C) 2016
+# Ole Tange and Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>
+# or write to the Free Software Foundation, Inc., 51 Franklin St,
+# Fifth Floor, Boston, MA 02110-1301 USA
+
+bash > regressiontest.out 2>&1  <<'_EOS'
+  rm -f testfile.lrz
+  seq 1000 > testfile
+  
+  echo 'Test basic use'
+    lrz testfile
+  
+  echo 'Test decompression in read-only dir'
+    mkdir -p ro
+    cp testfile.lrz ro
+    chmod 500 ro
+    cd ro
+    lrz -dc testfile.lrz | wc
+    cd ..
+  
+  echo 'this should be silent'
+    lrz -d testfile.lrz
+  
+  echo 'man page for lrz should exist'
+    man lrz >/dev/null
+    echo $?
+  
+  echo 'compress stdin to stdout'
+    cat testfile | lrz | cat > testfile.lrz
+  
+  echo 'Respect $TMPDIR'
+    mkdir -p t
+    chmod 111 t
+    cd t
+    TMPDIR=.. lrz -d < ../testfile.lrz | wc
+    cd ..
+    rm -rf t
+  
+  echo 'Decompress in read only dir'
+    mkdir -p t
+    chmod 111 t
+    cd t
+    lrz -d < ../testfile.lrz | wc
+    cd ..
+    rm -rf t
+  
+  echo 'Test -cd'
+    mkdir -p t
+    chmod 111 t
+    cd t
+    lrz -cd  ../testfile.lrz | wc
+    cd ..
+    rm -rf t
+
+  echo 'Test -cfd should not remove testfile.lrz'
+    mkdir -p t
+    chmod 111 t
+    cd t
+    lrz -cfd  ../testfile.lrz | wc
+    cd ..
+    rm -rf t
+    ls testfile.lrz
+
+  echo 'Test -1c'
+    lrz -1c testfile | wc
+
+  echo 'Test -r'
+    mkdir t
+    touch t/t{1..10}
+    lrz -r t
+    ls t
+    rm -r t
+
+  echo 'Test tar compatibility'
+    mkdir t
+    touch t/t{1..10}
+    tar --use-compress-program lrz -cvf testfile.tar.lrz t
+    tar --use-compress-program lrz -tvf testfile.tar.lrz | wc -l
+    rm -r t
+
+  echo 'test compress of 1 GB data with parallel --pipe --compress'
+    yes "`echo {1..100}`" |
+      head -c 1G |
+      parallel --pipe --block 100m --compress-program lrz cat |
+      wc -c
+
+  echo 'test compress of 1 GB with sort --compress-program'
+    yes "`echo {1..100}`" |
+      head -c 1G |
+      sort --compress-program lrz |
+      wc -c
+
+  echo 'test should not lrz -dc removes file'
+    rm testfile.lrz
+    echo OK > testfile
+    lrz testfile
+    lrz -dc testfile.lrz
+    ls testfile.lrz
+
+_EOS
+
+diff regressiontest.good regressiontest.out

runzip.c

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2012 Con Kolivas
+   Copyright (C) 2006-2016,2018,2021-2022 Con Kolivas
    Copyright (C) 1998-2003 Andrew Tridgell
 
    This program is free software; you can redistribute it and/or modify
@@ -139,16 +139,12 @@ static i64 seekto_fdinend(rzip_control *control)
 
 static i64 read_header(rzip_control *control, void *ss, uchar *head)
 {
-	int chunk_bytes = 2;
 	bool err = false;
 
-	/* All chunks were unnecessarily encoded 8 bytes wide version 0.4x */
-	if (control->major_version == 0 && control->minor_version == 4)
-		chunk_bytes = 8;
 	*head = read_u8(control, ss, 0, &err);
 	if (err)
 		return -1;
-	return read_vchars(control, ss, 0, chunk_bytes);
+	return read_vchars(control, ss, 0, control->chunk_bytes);
 }
 
 static i64 unzip_literal(rzip_control *control, void *ss, i64 len, uint32 *cksum)
@@ -165,12 +161,12 @@ static i64 unzip_literal(rzip_control *control, void *ss, i64 len, uint32 *cksum
 
 	stream_read = read_stream(control, ss, 1, buf, len);
 	if (unlikely(stream_read == -1 )) {
-		free(buf);
+		dealloc(buf);
 		fatal_return(("Failed to read_stream in unzip_literal\n"), -1);
 	}
 
 	if (unlikely(write_1g(control, buf, (size_t)stream_read) != (ssize_t)stream_read)) {
-		free(buf);
+		dealloc(buf);
 		fatal_return(("Failed to write literal buffer of size %lld\n", stream_read), -1);
 	}
 
@@ -179,7 +175,7 @@ static i64 unzip_literal(rzip_control *control, void *ss, i64 len, uint32 *cksum
 	if (!NO_MD5)
 		md5_process_bytes(buf, stream_read, &control->ctx);
 
-	free(buf);
+	dealloc(buf);
 	return stream_read;
 }
 
@@ -198,7 +194,7 @@ static i64 read_fdhist(rzip_control *control, void *buf, i64 len)
 static i64 unzip_match(rzip_control *control, void *ss, i64 len, uint32 *cksum, int chunk_bytes)
 {
 	i64 offset, n, total, cur_pos;
-	uchar *buf, *off_buf;
+	uchar *buf;
 
 	if (unlikely(len < 0))
 		failure_return(("len %lld is negative in unzip_match!\n",len), -1);
@@ -216,34 +212,39 @@ static i64 unzip_match(rzip_control *control, void *ss, i64 len, uint32 *cksum,
 		fatal_return(("Seek failed by %d from %d on history file in unzip_match\n",
 		      offset, cur_pos), -1);
 
-	buf = (uchar *)malloc(len);
+	n = MIN(len, offset);
+	if (unlikely(n < 1))
+		fatal_return(("Failed fd history in unzip_match due to corrupt archive\n"), -1);
+
+	buf = (uchar *)malloc(n);
 	if (unlikely(!buf))
 		fatal_return(("Failed to malloc match buffer of size %lld\n", len), -1);
-	off_buf = buf;
+
+	if (unlikely(read_fdhist(control, buf, (size_t)n) != (ssize_t)n)) {
+		dealloc(buf);
+		fatal_return(("Failed to read %d bytes in unzip_match\n", n), -1);
+	}
 
 	while (len) {
 		n = MIN(len, offset);
+		if (unlikely(n < 1))
+			fatal_return(("Failed fd history in unzip_match due to corrupt archive\n"), -1);
 
-		if (unlikely(read_fdhist(control, off_buf, (size_t)n) != (ssize_t)n)) {
-			free(buf);
-			fatal_return(("Failed to read %d bytes in unzip_match\n", n), -1);
-		}
-		if (unlikely(write_1g(control, off_buf, (size_t)n) != (ssize_t)n)) {
-			free(buf);
+		if (unlikely(write_1g(control, buf, (size_t)n) != (ssize_t)n)) {
+			dealloc(buf);
 			fatal_return(("Failed to write %d bytes in unzip_match\n", n), -1);
 		}
 
 		if (!HAS_MD5)
-			*cksum = CrcUpdate(*cksum, off_buf, n);
+			*cksum = CrcUpdate(*cksum, buf, n);
 		if (!NO_MD5)
-			md5_process_bytes(off_buf, n, &control->ctx);
+			md5_process_bytes(buf, n, &control->ctx);
 
 		len -= n;
-		off_buf += n;
 		total += n;
 	}
 
-	free(buf);
+	dealloc(buf);
 
 	return total;
 }
@@ -309,6 +310,12 @@ static i64 runzip_chunk(rzip_control *control, int fd_in, i64 expected_size, i64
 	if (unlikely(!ss))
 		failure_return(("Failed to open_stream_in in runzip_chunk\n"), -1);
 
+	/* All chunks were unnecessarily encoded 8 bytes wide version 0.4x */
+	if (control->major_version == 0 && control->minor_version == 4)
+		control->chunk_bytes = 8;
+	else
+		control->chunk_bytes = 2;
+
 	while ((len = read_header(control, ss, &head)) || head) {
 		i64 u;
 		if (unlikely(len == -1))
@@ -365,11 +372,12 @@ static i64 runzip_chunk(rzip_control *control, int fd_in, i64 expected_size, i64
 /* Decompress an open file. Call fatal_return(() on error
    return the number of bytes that have been retrieved
  */
-i64 runzip_fd(rzip_control *control, int fd_in, int fd_out, int fd_hist, i64 expected_size)
+i64 runzip_fd(rzip_control *control, int fd_in, int fd_hist, i64 expected_size)
 {
 	uchar md5_stored[MD5_DIGEST_SIZE];
 	struct timeval start,end;
 	i64 total = 0, u;
+	double tdiff;
 
 	if (!NO_MD5)
 		md5_init_ctx (&control->ctx);
@@ -377,26 +385,36 @@ i64 runzip_fd(rzip_control *control, int fd_in, int fd_out, int fd_hist, i64 exp
 
 	do {
 		u = runzip_chunk(control, fd_in, expected_size, total);
-		if (unlikely(u == -1))
-			return -1;
+		if (u < 1) {
+			if (u < 0 || total < expected_size) {
+				print_err("Failed to runzip_chunk in runzip_fd\n");
+				return -1;
+			}
+		}
 		total += u;
-		if (TMP_OUTBUF)
-			{ if (unlikely(!flush_tmpoutbuf(control)))
-			return -1; }
-		else if (STDOUT)
-			{ if (unlikely(!dump_tmpoutfile(control, fd_out)))
-			return -1; }
+		if (unlikely(!flush_tmpout(control))) {
+			print_err("Failed to flush_tmpout in runzip_fd\n");
+				return -1;
+		}
+
 		if (TMP_INBUF)
 			clear_tmpinbuf(control);
-		else if (STDIN)
-			if (unlikely(!clear_tmpinfile(control)))
+		else if (STDIN && !DECOMPRESS) {
+			if (unlikely(!clear_tmpinfile(control))) {
+				print_err("Failed to clear_tmpinfile in runzip_fd\n");
 				return -1;
+			}
+		}
 	} while (total < expected_size || (!expected_size && !control->eof));
 
 	gettimeofday(&end,NULL);
-	if (!ENCRYPT)
-		print_progress("\nAverage DeCompression Speed: %6.3fMB/s\n",
-			(total / 1024 / 1024) / (double)((end.tv_sec-start.tv_sec)? : 1));
+	if (!ENCRYPT) {
+		tdiff = end.tv_sec - start.tv_sec;
+		if (!tdiff)
+			tdiff = 1;
+		print_output("\nAverage DeCompression Speed: %6.3fMB/s\n",
+			       (total / 1024 / 1024) / tdiff);
+	}
 
 	if (!NO_MD5) {
 		int i,j;

runzip.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2011 Con Kolivas
+   Copyright (C) 2006-2011,2022 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998-2003 Andrew Tridgell
 
@@ -22,6 +22,6 @@
 
 #include "lrzip_private.h"
 
-i64 runzip_fd(rzip_control *control, int fd_in, int fd_out, int fd_hist, i64 expected_size);
+i64 runzip_fd(rzip_control *control, int fd_in, int fd_hist, i64 expected_size);
 
 #endif

rzip.c

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2013 Con Kolivas
+   Copyright (C) 2006-2016,2018,2022 Con Kolivas
    Copyright (C) 1998 Andrew Tridgell
 
    Modified to use flat hash, memory limit and variable hash culling
@@ -50,6 +50,7 @@
 #ifdef HAVE_ARPA_INET_H
 # include <arpa/inet.h>
 #endif
+#include <inttypes.h>
 
 #include "md5.h"
 #include "stream.h"
@@ -73,7 +74,6 @@
  * even tags, then all tags divisible by four, etc.).  This ensures
  * that on average, all parts of the file are covered by the hash, if
  * sparsely. */
-typedef i64 tag;
 
 /* All zero means empty.  We might miss the first chunk this way. */
 struct hash_entry {
@@ -99,35 +99,7 @@ static struct level {
 	{ 64, 1, 128 },
 };
 
-struct rzip_state {
-	void *ss;
-	struct level *level;
-	tag hash_index[256];
-	struct hash_entry *hash_table;
-	char hash_bits;
-	i64 hash_count;
-	i64 hash_limit;
-	tag minimum_tag_mask;
-	i64 tag_clean_ptr;
-	i64 last_match;
-	i64 chunk_size;
-	i64 mmap_size;
-	char chunk_bytes;
-	uint32_t cksum;
-	int fd_in, fd_out;
-	char stdin_eof;
-	struct {
-		i64 inserts;
-		i64 literals;
-		i64 literal_bytes;
-		i64 matches;
-		i64 match_bytes;
-		i64 tag_hits;
-		i64 tag_misses;
-	} stats;
-};
-
-static bool remap_low_sb(rzip_control *control, struct sliding_buffer *sb)
+static void remap_low_sb(rzip_control *control, struct sliding_buffer *sb)
 {
 	i64 new_offset;
 
@@ -135,20 +107,19 @@ static bool remap_low_sb(rzip_control *control, struct sliding_buffer *sb)
 	round_to_page(&new_offset);
 	print_maxverbose("Sliding main buffer to offset %lld\n", new_offset);
 	if (unlikely(munmap(sb->buf_low, sb->size_low)))
-		fatal_return(("Failed to munmap in remap_low_sb\n"), false);
+		failure("Failed to munmap in remap_low_sb\n");
 	if (new_offset + sb->size_low > sb->orig_size)
 		sb->size_low = sb->orig_size - new_offset;
 	sb->offset_low = new_offset;
 	sb->buf_low = (uchar *)mmap(sb->buf_low, sb->size_low, PROT_READ, MAP_SHARED, sb->fd, sb->orig_offset + sb->offset_low);
 	if (unlikely(sb->buf_low == MAP_FAILED))
-		fatal_return(("Failed to re mmap in remap_low_sb\n"), false);
-	return true;
+		failure("Failed to re mmap in remap_low_sb\n");
 }
 
-static inline bool remap_high_sb(rzip_control *control, struct sliding_buffer *sb, i64 p)
+static inline void remap_high_sb(rzip_control *control, struct sliding_buffer *sb, i64 p)
 {
 	if (unlikely(munmap(sb->buf_high, sb->size_high)))
-		fatal_return(("Failed to munmap in remap_high_sb\n"), false);
+		failure("Failed to munmap in remap_high_sb\n");
 	sb->size_high = sb->high_length; /* In case we shrunk it when we hit the end of the file */
 	sb->offset_high = p;
 	/* Make sure offset is rounded to page size of total offset */
@@ -157,8 +128,7 @@ static inline bool remap_high_sb(rzip_control *control, struct sliding_buffer *s
 		sb->size_high = sb->orig_size - sb->offset_high;
 	sb->buf_high = (uchar *)mmap(sb->buf_high, sb->size_high, PROT_READ, MAP_SHARED, sb->fd, sb->orig_offset + sb->offset_high);
 	if (unlikely(sb->buf_high == MAP_FAILED))
-		fatal_return(("Failed to re mmap in remap_high_sb\n"), false);
-	return true;
+		failure("Failed to re mmap in remap_high_sb\n");
 }
 
 /* We use a "sliding mmap" to effectively read more than we can fit into the
@@ -182,17 +152,11 @@ static uchar *sliding_get_sb(rzip_control *control, i64 p)
 	if (p >= sbo && p < (sbo + sb->size_high))
 		return (sb->buf_high + (p - sbo));
 	/* p is not within the low or high buffer range */
-	if (unlikely(!remap_high_sb(control, &control->sb, p)))
-		return NULL;
+	remap_high_sb(control, &control->sb, p);
 	/* Use sb->offset_high directly since it will have changed */
 	return (sb->buf_high + (p - sb->offset_high));
 }
 
-static uchar *single_get_sb(__maybe_unused rzip_control *control, i64 p)
-{
-	return (control->sb.buf_low + p);
-}
-
 /* The length of continous range of the sliding buffer,
  * starting from the offset P.
  */
@@ -235,40 +199,31 @@ static void sliding_mcpy(rzip_control *control, unsigned char *buf, i64 offset,
 }
 
 /* All put_u8/u32/vchars go to stream 0 */
-static inline bool put_u8(rzip_control *control, void *ss, uchar b)
+static inline void put_u8(rzip_control *control, void *ss, uchar b)
 {
-	if (unlikely(write_stream(control, ss, 0, &b, 1)))
-		fatal_return(("Failed to put_u8\n"), false);
-	return true;
+	write_stream(control, ss, 0, &b, 1);
 }
 
-static inline bool put_u32(rzip_control *control, void *ss, uint32_t s)
+static inline void put_u32(rzip_control *control, void *ss, uint32_t s)
 {
 	s = htole32(s);
-	if (unlikely(write_stream(control, ss, 0, (uchar *)&s, 4)))
-		fatal_return(("Failed to put_u32\n"), false);
-	return true;
+	write_stream(control, ss, 0, (uchar *)&s, 4);
 }
 
 /* Put a variable length of bytes dependant on how big the chunk is */
-static bool put_vchars(rzip_control *control, void *ss, i64 s, i64 length)
+static void put_vchars(rzip_control *control, void *ss, i64 s, i64 length)
 {
 	s = htole64(s);
-	if (unlikely(write_stream(control, ss, 0, (uchar *)&s, length)))
-		fatal_return(("Failed to put_vchars\n"), false);
-	return true;
+	write_stream(control, ss, 0, (uchar *)&s, length);
 }
 
-static bool put_header(rzip_control *control, void *ss, uchar head, i64 len)
+static void put_header(rzip_control *control, void *ss, uchar head, i64 len)
 {
-	if (unlikely(!put_u8(control, ss, head)))
-		return false;
-	if (unlikely(!put_vchars(control, ss, len, 2)))
-		return false;
-	return true;
+	put_u8(control, ss, head);
+	put_vchars(control, ss, len, 2);
 }
 
-static inline bool put_match(rzip_control *control, struct rzip_state *st,
+static inline void put_match(rzip_control *control, struct rzip_state *st,
 			     i64 p, i64 offset, i64 len)
 {
 	do {
@@ -278,22 +233,18 @@ static inline bool put_match(rzip_control *control, struct rzip_state *st,
 			n = 0xFFFF;
 
 		ofs = (p - offset);
-		if (unlikely(!put_header(control, st->ss, 1, n)))
-			return false;
-		if (unlikely(!put_vchars(control, st->ss, ofs, st->chunk_bytes)))
-			return false;
-
+		put_header(control, st->ss, 1, n);
+		put_vchars(control, st->ss, ofs, st->chunk_bytes);
 		st->stats.matches++;
 		st->stats.match_bytes += n;
 		len -= n;
 		p += n;
 		offset += n;
 	} while (len);
-	return true;
 }
 
 /* write some data to a stream mmap encoded. Return -1 on failure */
-static inline int write_sbstream(rzip_control *control, void *ss, int stream,
+static inline void write_sbstream(rzip_control *control, void *ss, int stream,
 				 i64 p, i64 len)
 {
 	struct stream_info *sinfo = ss;
@@ -308,30 +259,27 @@ static inline int write_sbstream(rzip_control *control, void *ss, int stream,
 		len -= n;
 
 		if (sinfo->s[stream].buflen == sinfo->bufsize)
-			if (unlikely(!flush_buffer(control, sinfo, stream)))
-				return -1;
+			flush_buffer(control, sinfo, stream);
 	}
-	return 0;
 }
 
-static bool put_literal(rzip_control *control, struct rzip_state *st, i64 last, i64 p)
+static void put_literal(rzip_control *control, struct rzip_state *st, i64 last, i64 p)
 {
 	do {
 		i64 len = p - last;
+
 		if (len > 0xFFFF)
 			len = 0xFFFF;
 
 		st->stats.literals++;
 		st->stats.literal_bytes += len;
 
-		if (unlikely(!put_header(control, st->ss, 0, len)))
-			return false;
+		put_header(control, st->ss, 0, len);
 
-		if (unlikely(len && write_sbstream(control, st->ss, 1, last, len)))
-			fatal_return(("Failed to write_stream in put_literal\n"), false);
+		if (len)
+			write_sbstream(control, st->ss, 1, last, len);
 		last += len;
 	} while (p > last);
-	return true;
 }
 
 /* Could give false positive on offset 0.  Who cares. */
@@ -452,44 +400,63 @@ again:
 	goto again;
 }
 
-static inline void next_tag(rzip_control *control, struct rzip_state *st, i64 p, tag *t)
+static void single_next_tag(rzip_control *control, struct rzip_state *st, i64 p, tag *t)
+{
+	uchar u;
+
+	u = control->sb.buf_low[p - 1];
+	*t ^= st->hash_index[u];
+	u = control->sb.buf_low[p + MINIMUM_MATCH - 1];
+	*t ^= st->hash_index[u];
+}
+
+static void sliding_next_tag(rzip_control *control, struct rzip_state *st, i64 p, tag *t)
 {
 	uchar *u;
 
-	u = control->get_sb(control, p - 1);
+	u = sliding_get_sb(control, p - 1);
 	*t ^= st->hash_index[*u];
-	u = control->get_sb(control, p + MINIMUM_MATCH - 1);
+	u = sliding_get_sb(control, p + MINIMUM_MATCH - 1);
 	*t ^= st->hash_index[*u];
 }
 
-static inline tag full_tag(rzip_control *control, struct rzip_state *st, i64 p)
+static tag single_full_tag(rzip_control *control, struct rzip_state *st, i64 p)
+{
+	tag ret = 0;
+	int i;
+	uchar u;
+
+	for (i = 0; i < MINIMUM_MATCH; i++) {
+		u = control->sb.buf_low[p + i];
+		ret ^= st->hash_index[u];
+	}
+	return ret;
+}
+
+static tag sliding_full_tag(rzip_control *control, struct rzip_state *st, i64 p)
 {
 	tag ret = 0;
 	int i;
 	uchar *u;
 
 	for (i = 0; i < MINIMUM_MATCH; i++) {
-		u = control->get_sb(control, p + i);
-		if (unlikely(!u))
-			return -1;
+		u = sliding_get_sb(control, p + i);
 		ret ^= st->hash_index[*u];
 	}
 	return ret;
 }
 
-static inline i64
-match_len(rzip_control *control, struct rzip_state *st, i64 p0, i64 op,
-	  i64 end, i64 *rev)
+static i64
+single_match_len(rzip_control *control, struct rzip_state *st, i64 p0, i64 op,
+		 i64 end, i64 *rev)
 {
-	uchar *(*csb)(rzip_control *, i64);
 	i64 p, len;
 
 	if (op >= p0)
 		return 0;
 
 	p = p0;
-	csb = control->get_sb;
-	while (p < end && (*csb(control, p) == *csb(control, op))) {
+	while (p < end && control->sb.buf_low[p] == control->sb.buf_low[op]) {
 		p++;
 		op++;
 	}
@@ -499,7 +466,39 @@ match_len(rzip_control *control, struct rzip_state *st, i64 p0, i64 op,
 
 	end = MAX(0, st->last_match);
 
-	while (p > end && op > 0 && *csb(control, op - 1) == *csb(control, p - 1)) {
+	while (p > end && op > 0 && control->sb.buf_low[op - 1] == control->sb.buf_low[p - 1]) {
+		op--;
+		p--;
+	}
+
+	len += *rev = p0 - p;
+	if (len < MINIMUM_MATCH)
+		return 0;
+
+	return len;
+}
+
+static i64
+sliding_match_len(rzip_control *control, struct rzip_state *st, i64 p0, i64 op,
+		  i64 end, i64 *rev)
+{
+	i64 p, len;
+
+	if (op >= p0)
+		return 0;
+
+	p = p0;
+	while (p < end && *sliding_get_sb(control, p) == *sliding_get_sb(control, op)) {
+		p++;
+		op++;
+	}
+	len = p - p0;
+	p = p0;
+	op -= len;
+
+	end = MAX(0, st->last_match);
+
+	while (p > end && op > 0 && *sliding_get_sb(control, op - 1) == *sliding_get_sb(control, p - 1)) {
 		op--;
 		p--;
 	}
@@ -531,19 +530,17 @@ find_best_match(rzip_control *control, struct rzip_state *st, tag t, i64 p,
 		i64 mlen;
 
 		if (t == he->t) {
-			mlen = match_len(control, st, p, he->offset, end,
-					 &rev);
-
-			if (mlen)
+			mlen = control->match_len(control, st, p, he->offset, end,
+						  &rev);
+			if (mlen) {
+				if (mlen > length) {
+					length = mlen;
+					(*offset) = he->offset - rev;
+					(*reverse) = rev;
+				}
 				st->stats.tag_hits++;
-			else
+			} else
 				st->stats.tag_misses++;
-
-			if (mlen >= length) {
-				length = mlen;
-				(*offset) = he->offset - rev;
-				(*reverse) = rev;
-			}
 		}
 
 		h++;
@@ -573,8 +570,12 @@ static void show_distrib(rzip_control *control, struct rzip_state *st)
 	if (total != st->hash_count)
 		print_err("WARNING: hash_count says total %lld\n", st->hash_count);
 
-	print_output("%lld total hashes -- %lld in primary bucket (%-2.3f%%)\n", total, primary,
-		     primary * 100.0 / (total ? : 1));
+	if (!total)
+		print_output("0 total hashes\n");
+	else {
+		print_output("%lld total hashes -- %lld in primary bucket (%-2.3f%%)\n",
+			     total, primary, primary * 100.0 / total);
+	}
 }
 
 /* Perform all checksumming in a separate thread to speed up the hash search. */
@@ -587,8 +588,8 @@ static void *cksumthread(void *data)
 	*control->checksum.cksum = CrcUpdate(*control->checksum.cksum, control->checksum.buf, control->checksum.len);
 	if (!NO_MD5)
 		md5_process_bytes(control->checksum.buf, control->checksum.len, &control->ctx);
-	free(control->checksum.buf);
-	unlock_mutex(control, &control->cksumlock);
+	dealloc(control->checksum.buf);
+	cksem_post(control, &control->cksumsem);
 	return NULL;
 }
 
@@ -599,21 +600,19 @@ static inline void cksum_update(rzip_control *control)
 	create_pthread(control, &thread, NULL, cksumthread, control);
 }
 
-static inline bool hash_search(rzip_control *control, struct rzip_state *st,
+static inline void hash_search(rzip_control *control, struct rzip_state *st,
 			       double pct_base, double pct_multiple)
 {
 	i64 cksum_limit = 0, p, end, cksum_chunks, cksum_remains, i;
+	tag t = 0, tag_mask = (1 << st->level->initial_freq) - 1;
 	struct sliding_buffer *sb = &control->sb;
 	int lastpct = 0, last_chunkpct = 0;
-	tag t = 0;
 	struct {
 		i64 p;
 		i64 ofs;
 		i64 len;
 	} current;
 
-	tag tag_mask = (1 << st->level->initial_freq) - 1;
-
 	if (st->hash_table)
 		memset(st->hash_table, 0, sizeof(st->hash_table[0]) * (1<<st->hash_bits));
 	else {
@@ -628,7 +627,7 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 		st->hash_limit = (1 << st->hash_bits) / 3 * 2;
 		st->hash_table = calloc(sizeof(st->hash_table[0]), (1 << st->hash_bits));
 		if (unlikely(!st->hash_table))
-			fatal_return(("Failed to allocate hash table in hash_search\n"), false);
+			failure("Failed to allocate hash table in hash_search\n");
 	}
 
 	st->minimum_tag_mask = tag_mask;
@@ -643,11 +642,8 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 	current.p = p;
 	current.ofs = 0;
 
-	if (likely(end > 0)) {
-		t = full_tag(control, st, p);
-		if (unlikely(t == -1))
-			return false;
-	}
+	if (likely(end > 0))
+		t = control->full_tag(control, st, p);
 
 	while (p < end) {
 		i64 reverse, mlen, offset;
@@ -656,12 +652,12 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 		if (unlikely(sb->offset_search > sb->offset_low + sb->size_low))
 			remap_low_sb(control, &control->sb);
 
-		if (unlikely(p % 128 == 0)) {
-			int pct, chunk_pct;
+		if (unlikely(p % 128 == 0 && st->chunk_size)) {
+			i64 chunk_pct;
+			int pct;
 
-			pct = pct_base + (pct_multiple * (100.0 * p) /
-			      (st->chunk_size ? : 1));
-			chunk_pct = p / ((end / 100) ? : 1);
+			pct = pct_base + (pct_multiple * (100.0 * p) / st->chunk_size );
+			chunk_pct = p * 100 / end;
 			if (pct != lastpct || chunk_pct != last_chunkpct) {
 				if (!STDIN || st->stdin_eof)
 					print_progress("Total: %2d%%  ", pct);
@@ -674,7 +670,7 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 			}
 		}
 
-		next_tag(control, st, p, &t);
+		control->next_tag(control, st, p, &t);
 
 		/* Don't look for a match if there are no tags with
 		   this number of bits in the hash table. */
@@ -702,16 +698,12 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 		if ((current.len >= GREAT_MATCH || p >= current.p + MINIMUM_MATCH)
 		    && current.len >= MINIMUM_MATCH) {
 			if (st->last_match < current.p)
-				if (unlikely(!put_literal(control, st, st->last_match, current.p)))
-					return false;
-			if (unlikely(!put_match(control, st, current.p, current.ofs, current.len)))
-				return false;
+				put_literal(control, st, st->last_match, current.p);
+			put_match(control, st, current.p, current.ofs, current.len);
 			st->last_match = current.p + current.len;
 			current.p = p = st->last_match;
 			current.len = 0;
-			t = full_tag(control, st, p);
-			if (unlikely(t == -1))
-				return false;
+			t = control->full_tag(control, st, p);
 		}
 
 		if (p > cksum_limit) {
@@ -719,15 +711,15 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 			 * cksumthread. This lock protects all the data in
 			 * control->checksum.
 			 */
-			lock_mutex(control, &control->cksumlock);
+			cksem_wait(control, &control->cksumsem);
 			control->checksum.len = MIN(st->chunk_size - p, control->page_size);
 			control->checksum.buf = malloc(control->checksum.len);
 			if (unlikely(!control->checksum.buf))
-				fatal_return(("Failed to malloc ckbuf in hash_search\n"), false);
+				failure("Failed to malloc ckbuf in hash_search\n");
 			control->do_mcpy(control, control->checksum.buf, cksum_limit, control->checksum.len);
 			control->checksum.cksum = &st->cksum;
-			cksum_update(control);
 			cksum_limit += control->checksum.len;
+			cksum_update(control);
 		}
 	}
 
@@ -738,39 +730,50 @@ static inline bool hash_search(rzip_control *control, struct rzip_state *st,
 		put_literal(control, st, st->last_match, st->chunk_size);
 
 	if (st->chunk_size > cksum_limit) {
+		i64 cksum_len = control->maxram;
+		void *buf;
+
+		while (42) {
+			round_to_page(&cksum_len);
+			buf = malloc(cksum_len);
+			if (buf) {
+				print_maxverbose("Malloced %"PRId64" for checksum ckbuf\n", cksum_len);
+				break;
+			}
+			cksum_len = cksum_len / 3 * 2;
+			if (cksum_len < control->page_size)
+				failure("Failed to malloc any ram for checksum ckbuf\n");
+		}
+
 		/* Compute checksum. If the entire chunk is longer than maxram,
 		 * do it "per-partes" */
-		lock_mutex(control, &control->cksumlock);
+		cksem_wait(control, &control->cksumsem);
+		control->checksum.buf = buf;
 		control->checksum.len = st->chunk_size - cksum_limit;
-		cksum_chunks = control->checksum.len / control->maxram;
-		cksum_remains = control->checksum.len % control->maxram;
-
-		control->checksum.buf = malloc(control->checksum.len);
-		if (unlikely(!control->checksum.buf))
-			fatal_return(("Failed to malloc ckbuf in hash_search2\n"), false);
+		cksum_chunks = control->checksum.len / cksum_len;
+		cksum_remains = control->checksum.len % cksum_len;
 
 		for (i = 0; i < cksum_chunks; i++) {
-			control->do_mcpy(control, control->checksum.buf, cksum_limit, control->maxram);
-			cksum_limit += control->maxram;
-			st->cksum = CrcUpdate(st->cksum, control->checksum.buf, control->maxram);
+			control->do_mcpy(control, control->checksum.buf, cksum_limit, cksum_len);
+			cksum_limit += cksum_len;
+			st->cksum = CrcUpdate(st->cksum, control->checksum.buf, cksum_len);
 			if (!NO_MD5)
-				md5_process_bytes(control->checksum.buf, control->maxram, &control->ctx);
+				md5_process_bytes(control->checksum.buf, cksum_len, &control->ctx);
 		}
 		/* Process end of the checksum buffer */
 		control->do_mcpy(control, control->checksum.buf, cksum_limit, cksum_remains);
 		st->cksum = CrcUpdate(st->cksum, control->checksum.buf, cksum_remains);
 		if (!NO_MD5)
 			md5_process_bytes(control->checksum.buf, cksum_remains, &control->ctx);
-		free(control->checksum.buf);
-		unlock_mutex(control, &control->cksumlock);
-	} else
-		wait_mutex(control, &control->cksumlock);
+		dealloc(control->checksum.buf);
+		cksem_post(control, &control->cksumsem);
+	} else {
+		cksem_wait(control, &control->cksumsem);
+		cksem_post(control, &control->cksumsem);
+	}
 
-	if (unlikely(!put_literal(control, st, 0, 0)))
-		return false;
-	if (unlikely(!put_u32(control, st->ss, st->cksum)))
-		return false;
-	return true;
+	put_literal(control, st, 0, 0);
+	put_u32(control, st->ss, st->cksum);
 }
 
 
@@ -782,9 +785,8 @@ static inline void init_hash_indexes(struct rzip_state *st)
 		st->hash_index[i] = ((random() << 16) ^ random());
 }
 
-#if defined(__APPLE__) || defined(__FreeBSD__)
+#if !defined(__linux)
 # define mremap fake_mremap
-#endif
 
 static inline void *fake_mremap(void *old_address, size_t old_size, size_t new_size, int flags __UNUSED__)
 {
@@ -804,12 +806,13 @@ static inline void *fake_mremap(void *old_address, size_t old_size, size_t new_s
 		return old_address;
 	}
 }
+#endif
 
 /* stdin is not file backed so we have to emulate the mmap by mapping
  * anonymous ram and reading stdin into it. It means the maximum ram
  * we can use will be less but we will already have determined this in
  * rzip_chunk */
-static inline bool mmap_stdin(rzip_control *control, uchar *buf,
+static inline void mmap_stdin(rzip_control *control, uchar *buf,
 			      struct rzip_state *st)
 {
 	i64 len = st->chunk_size;
@@ -822,7 +825,7 @@ static inline bool mmap_stdin(rzip_control *control, uchar *buf,
 		ret = MIN(len, one_g);
 		ret = read(fileno(control->inFILE), offset_buf, (size_t)ret);
 		if (unlikely(ret < 0))
-			fatal_return(("Failed to read in mmap_stdin\n"), false);
+			failure("Failed to read in mmap_stdin\n");
 		total += ret;
 		if (ret == 0) {
 			/* Should be EOF */
@@ -837,7 +840,7 @@ static inline bool mmap_stdin(rzip_control *control, uchar *buf,
 				st->chunk_size = 0;
 			}
 			if (unlikely(buf == MAP_FAILED))
-				fatal_return(("Failed to remap to smaller buf in mmap_stdin\n"), false);
+				failure("Failed to remap to smaller buf in mmap_stdin\n");
 			control->eof = st->stdin_eof = 1;
 			break;
 		}
@@ -845,24 +848,20 @@ static inline bool mmap_stdin(rzip_control *control, uchar *buf,
 		len -= ret;
 	}
 	control->st_size += total;
-	return true;
 }
 
-static inline bool
+static inline void
 init_sliding_mmap(rzip_control *control, struct rzip_state *st, int fd_in,
 		  i64 offset)
 {
 	struct sliding_buffer *sb = &control->sb;
 
-	/* Initialise the high buffer */
+	/* Initialise the high buffer. One page size is fastest to manipulate */
 	if (!STDIN) {
-		sb->high_length = 65536;
-		/* Round up to the next biggest page size */
-		if (sb->high_length % control->page_size)
-			sb->high_length += control->page_size - (sb->high_length % control->page_size);
+		sb->high_length = control->page_size;
 		sb->buf_high = (uchar *)mmap(NULL, sb->high_length, PROT_READ, MAP_SHARED, fd_in, offset);
 		if (unlikely(sb->buf_high == MAP_FAILED))
-			fatal_return(("Unable to mmap buf_high in init_sliding_mmap\n"), false);
+			failure("Unable to mmap buf_high in init_sliding_mmap\n");
 		sb->size_high = sb->high_length;
 		sb->offset_high = 0;
 	}
@@ -871,49 +870,71 @@ init_sliding_mmap(rzip_control *control, struct rzip_state *st, int fd_in,
 	sb->size_low = st->mmap_size;
 	sb->orig_size = st->chunk_size;
 	sb->fd = fd_in;
-	return true;
+}
+
+static void add_to_sslist(rzip_control *control, struct rzip_state *st)
+{
+	struct node *node = calloc(sizeof(struct node), 1);
+
+	if (unlikely(!node))
+		failure("Failed to calloc struct node in add_to_sslist\n");
+	node->data = st->ss;
+	node->prev = st->sslist;
+	st->head = node;
 }
 
 /* compress a chunk of an open file. Assumes that the file is able to
    be mmap'd and is seekable */
-static inline bool
+static inline void
 rzip_chunk(rzip_control *control, struct rzip_state *st, int fd_in, int fd_out,
 	   i64 offset, double pct_base, double pct_multiple)
 {
 	struct sliding_buffer *sb = &control->sb;
 
-	if (unlikely(!init_sliding_mmap(control, st, fd_in, offset)))
-		return false;
+	init_sliding_mmap(control, st, fd_in, offset);
 
 	st->ss = open_stream_out(control, fd_out, NUM_STREAMS, st->chunk_size, st->chunk_bytes);
 	if (unlikely(!st->ss))
-		fatal_return(("Failed to open streams in rzip_chunk\n"), false);
+		failure("Failed to open streams in rzip_chunk\n");
 
 	print_verbose("Beginning rzip pre-processing phase\n");
-	if (unlikely(!hash_search(control, st, pct_base, pct_multiple))) {
-		close_stream_out(control, st->ss);
-		return false;
-	}
+	hash_search(control, st, pct_base, pct_multiple);
 
 	/* unmap buffer before closing and reallocating streams */
 	if (unlikely(munmap(sb->buf_low, sb->size_low))) {
 		close_stream_out(control, st->ss);
-		fatal_return(("Failed to munmap in rzip_chunk\n"), false);
+		failure("Failed to munmap in rzip_chunk\n");
 	}
 	if (!STDIN) {
 		if (unlikely(munmap(sb->buf_high, sb->size_high))) {
 			close_stream_out(control, st->ss);
-			fatal_return(("Failed to munmap in rzip_chunk\n"), false);
+			failure("Failed to munmap in rzip_chunk\n");
 		}
 	}
 
 	if (unlikely(close_stream_out(control, st->ss)))
-		fatal_return(("Failed to flush/close streams in rzip_chunk\n"), false);
-	return true;
+		failure("Failed to flush/close streams in rzip_chunk\n");
+
+	/* Save the sinfo data to a list to be safely released after all
+	 * threads have been shut down. */
+	add_to_sslist(control, st);
+}
+
+static void clear_sslist(struct rzip_state *st)
+{
+	while (st->head) {
+		struct node *node = st->head;
+		struct stream_info *sinfo = node->data;
+
+		dealloc(sinfo->s);
+		dealloc(sinfo);
+		st->head = node->prev;
+		dealloc(node);
+	}
 }
 
 /* compress a whole file chunks at a time */
-bool rzip_fd(rzip_control *control, int fd_in, int fd_out)
+void rzip_fd(rzip_control *control, int fd_in, int fd_out)
 {
 	struct sliding_buffer *sb = &control->sb;
 
@@ -926,31 +947,32 @@ bool rzip_fd(rzip_control *control, int fd_in, int fd_out)
 	struct timeval current, start, last;
 	i64 len = 0, last_chunk = 0;
 	int pass = 0, passes, j;
+	double chunkmbs, tdiff;
 	struct rzip_state *st;
 	struct statvfs fbuf;
 	struct stat s, s2;
-	double chunkmbs;
 	i64 free_space;
 
 	init_mutex(control, &control->control_lock);
 	if (!NO_MD5)
 		md5_init_ctx(&control->ctx);
-	init_mutex(control, &control->cksumlock);
+	cksem_init(control, &control->cksumsem);
+	cksem_post(control, &control->cksumsem);
 
 	st = calloc(sizeof(*st), 1);
 	if (unlikely(!st))
-		fatal_return(("Failed to allocate control state in rzip_fd\n"), false);
+		failure("Failed to allocate control state in rzip_fd\n");
 
 	if (LZO_COMPRESS) {
 		if (unlikely(lzo_init() != LZO_E_OK)) {
-			free(st);
-			fatal_return(("lzo_init() failed\n"), false);
+			dealloc(st);
+			failure("lzo_init() failed\n");
 		}
 	}
 
 	if (unlikely(fstat(fd_in, &s))) {
-		free(st);
-		fatal_return(("Failed to stat fd_in in rzip_fd\n"), false);
+		dealloc(st);
+		failure("Failed to stat fd_in in rzip_fd\n");
 	}
 
 	if (!STDIN) {
@@ -964,16 +986,16 @@ bool rzip_fd(rzip_control *control, int fd_in, int fd_out)
 		* compressed file, based on the compressed file being as large as the
 		* uncompressed file. */
 		if (unlikely(fstatvfs(fd_out, &fbuf))) {
-			free(st);
-			fatal_return(("Failed to fstatvfs in compress_file\n"), false);
+			dealloc(st);
+			failure("Failed to fstatvfs in compress_file\n");
 		}
 		free_space = (i64)fbuf.f_bsize * (i64)fbuf.f_bavail;
 		if (free_space < control->st_size) {
 			if (FORCE_REPLACE)
-				print_err("Warning, possibly inadequate free space detected, but attempting to compress due to -f option being used.\n");
+				print_output("Warning, possibly inadequate free space detected, but attempting to compress due to -f option being used.\n");
 			else {
-				free(st);
-				failure_return(("Possibly inadequate free space to compress file, use -f to override.\n"), false);
+				dealloc(st);
+				failure("Possibly inadequate free space to compress file, use -f to override.\n");
 			}
 		}
 	}
@@ -1019,8 +1041,10 @@ bool rzip_fd(rzip_control *control, int fd_in, int fd_out)
 	gettimeofday(&start, NULL);
 
 	prepare_streamout_threads(control);
-	control->get_sb = single_get_sb;
 	control->do_mcpy = single_mcpy;
+	control->next_tag = &single_next_tag;
+	control->full_tag = &single_full_tag;
+	control->match_len = &single_match_len;
 
 	while (!pass || len > 0 || (STDIN && !st->stdin_eof)) {
 		double pct_base, pct_multiple;
@@ -1045,51 +1069,48 @@ retry:
 			if (sb->buf_low == MAP_FAILED) {
 				if (unlikely(errno != ENOMEM)) {
 					close_streamout_threads(control);
-					free(st->hash_table);
-					free(st);
-					fatal_return(("Failed to mmap %s\n", control->infile), false);
+					dealloc(st->hash_table);
+					dealloc(st);
+					failure("Failed to mmap %s\n", control->infile);
 				}
 				st->mmap_size = st->mmap_size / 10 * 9;
 				round_to_page(&st->mmap_size);
 				if (unlikely(!st->mmap_size)) {
 					close_streamout_threads(control);
-					free(st->hash_table);
-					free(st);
-					fatal_return(("Unable to mmap any ram\n"), false);
+					dealloc(st->hash_table);
+					dealloc(st);
+					failure("Unable to mmap any ram\n");
 				}
 				goto retry;
 			}
 			st->chunk_size = st->mmap_size;
-			if (unlikely(!mmap_stdin(control, sb->buf_low, st))) {
-				close_streamout_threads(control);
-				free(st->hash_table);
-				free(st);
-				return false;
-			}
+			mmap_stdin(control, sb->buf_low, st);
 		} else {
 			/* NOTE The buf is saved here for !STDIN mode */
 			sb->buf_low = (uchar *)mmap(sb->buf_low, st->mmap_size, PROT_READ, MAP_SHARED, fd_in, offset);
 			if (sb->buf_low == MAP_FAILED) {
 				if (unlikely(errno != ENOMEM)) {
 					close_streamout_threads(control);
-					free(st->hash_table);
-					free(st);
-					fatal_return(("Failed to mmap %s\n", control->infile), false);
+					dealloc(st->hash_table);
+					dealloc(st);
+					failure("Failed to mmap %s\n", control->infile);
 				}
 				st->mmap_size = st->mmap_size / 10 * 9;
 				round_to_page(&st->mmap_size);
 				if (unlikely(!st->mmap_size)) {
 					close_streamout_threads(control);
-					free(st->hash_table);
-					free(st);
-					fatal_return(("Unable to mmap any ram\n"), false);
+					dealloc(st->hash_table);
+					dealloc(st);
+					failure("Unable to mmap any ram\n");
 				}
 				goto retry;
 			}
 			if (st->mmap_size < st->chunk_size) {
 				print_maxverbose("Enabling sliding mmap mode and using mmap of %lld bytes with window of %lld bytes\n", st->mmap_size, st->chunk_size);
-				control->get_sb = &sliding_get_sb;
 				control->do_mcpy = &sliding_mcpy;
+				control->next_tag = &sliding_next_tag;
+				control->full_tag = &sliding_full_tag;
+				control->match_len = &sliding_match_len;
 			}
 		}
 		print_maxverbose("Succeeded in testing %lld sized mmap for rzip pre-processing\n", st->mmap_size);
@@ -1131,12 +1152,12 @@ retry:
 
 		gettimeofday(&current, NULL);
 		/* this will count only when size > window */
-		if (last.tv_sec > 0) {
+		if (last.tv_sec > 0 && pct_base > 100) {
 			unsigned int eta_hours, eta_minutes, eta_seconds, elapsed_time, finish_time,
 				elapsed_hours, elapsed_minutes, elapsed_seconds, diff_seconds;
 
 			elapsed_time = current.tv_sec - start.tv_sec;
-			finish_time = elapsed_time / ((pct_base / 100.0) ? : 1);
+			finish_time = elapsed_time / (pct_base / 100.0);
 			elapsed_hours = elapsed_time / 3600;
 			elapsed_minutes = (elapsed_time / 60) % 60;
 			elapsed_seconds = elapsed_time % 60;
@@ -1159,29 +1180,24 @@ retry:
 
 		if (st->chunk_size == len)
 			control->eof = 1;
-		if (unlikely(!rzip_chunk(control, st, fd_in, fd_out, offset, pct_base, pct_multiple))) {
-			close_streamout_threads(control);
-			free(st->hash_table);
-			free(st);
-			return false;
-		}
+		rzip_chunk(control, st, fd_in, fd_out, offset, pct_base, pct_multiple);
 
 		/* st->chunk_size may be shrunk in rzip_chunk */
 		last_chunk = st->chunk_size;
 		len -= st->chunk_size;
 		if (unlikely(len > 0 && control->eof)) {
 			close_streamout_threads(control);
-			free(st->hash_table);
-			free(st);
-			failure_return(("Wrote EOF to file yet chunk_size was shrunk, corrupting archive.\n"), false);
+			dealloc(st->hash_table);
+			dealloc(st);
+			failure("Wrote EOF to file yet chunk_size was shrunk, corrupting archive.\n");
 		}
 	}
 
 	if (likely(st->hash_table))
-		free(st->hash_table);
+		dealloc(st->hash_table);
 	if (unlikely(!close_streamout_threads(control))) {
-		free(st);
-		return false;
+		dealloc(st);
+		failure("Failed to close_streamout_threads in rzip_fd\n");
 	}
 
 	if (!NO_MD5) {
@@ -1196,25 +1212,27 @@ retry:
 		/* When encrypting data, we encrypt the MD5 value as well */
 		if (ENCRYPT)
 			if (unlikely(!lrz_encrypt(control, control->md5_resblock, MD5_DIGEST_SIZE, control->salt_pass))) {
-				free(st);
-				return false;
+				dealloc(st);
+				failure("Failed to lrz_encrypt in rzip_fd\n");
 			}
 		if (unlikely(write_1g(control, control->md5_resblock, MD5_DIGEST_SIZE) != MD5_DIGEST_SIZE)) {
-			free(st);
-			fatal_return(("Failed to write md5 in rzip_fd\n"), false);
+			dealloc(st);
+			failure("Failed to write md5 in rzip_fd\n");
 		}
 	}
 
-	if (TMP_OUTBUF)
-		if (unlikely(!flush_tmpoutbuf(control))) {
-			free(st);
-			return false;
-		}
+	if (unlikely(!flush_tmpout(control))) {
+			dealloc(st);
+			failure("Failed to flush_tmpout in rzip_fd\n");
+	}
 
 	gettimeofday(&current, NULL);
 	if (STDIN)
 		s.st_size = control->st_size;
-	chunkmbs = (s.st_size / 1024 / 1024) / ((double)(current.tv_sec-start.tv_sec)? : 1);
+	tdiff = current.tv_sec - start.tv_sec;
+	if (!tdiff)
+		tdiff = 1;
+	chunkmbs = (s.st_size / 1024 / 1024) / tdiff;
 
 	fstat(fd_out, &s2);
 
@@ -1229,29 +1247,10 @@ retry:
 	       (1.0 + st->stats.match_bytes) / st->stats.literal_bytes);
 
 	if (!STDIN)
-		print_progress("%s - ", control->infile);
-	print_progress("Compression Ratio: %.3f. Average Compression Speed: %6.3fMB/s.\n",
+		print_output("%s - ", control->infile);
+	print_output("Compression Ratio: %.3f. Average Compression Speed: %6.3fMB/s.\n",
 		       1.0 * s.st_size / s2.st_size, chunkmbs);
 
-	free(st);
-	return true;
-}
-
-void rzip_control_free(rzip_control *control)
-{
-	size_t x;
-	if (!control)
-		return;
-
-	free(control->tmpdir);
-	free(control->outname);
-	free(control->outdir);
-	if (control->suffix && control->suffix[0]) free(control->suffix);
-
-	for (x = 0; x < control->sinfo_idx; x++) {
-		free(control->sinfo_queue[x]->s);
-		free(control->sinfo_queue[x]);
-	}
-	free(control->sinfo_queue);
-	free(control);
+	clear_sslist(st);
+	dealloc(st);
 }

rzip.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2011 Con Kolivas
+   Copyright (C) 2006-2016,2022 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998 Andrew Tridgell
 
@@ -21,7 +21,6 @@
 #define RZIP_H
 #include "lrzip_private.h"
 
-bool rzip_fd(rzip_control *control, int fd_in, int fd_out);
-void rzip_control_free(rzip_control *control);
+void rzip_fd(rzip_control *control, int fd_in, int fd_out);
 
 #endif

stream.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2011 Con Kolivas
+   Copyright (C) 2006-2016 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998-2003 Andrew Tridgell
 
@@ -29,7 +29,6 @@ bool join_pthread(pthread_t th, void **thread_return);
 bool init_mutex(rzip_control *control, pthread_mutex_t *mutex);
 bool unlock_mutex(rzip_control *control, pthread_mutex_t *mutex);
 bool lock_mutex(rzip_control *control, pthread_mutex_t *mutex);
-bool wait_mutex(rzip_control *control, pthread_mutex_t *mutex);
 ssize_t write_1g(rzip_control *control, void *buf, i64 len);
 ssize_t read_1g(rzip_control *control, int fd, void *buf, i64 len);
 i64 get_readseek(rzip_control *control, int fd);
@@ -37,8 +36,8 @@ bool prepare_streamout_threads(rzip_control *control);
 bool close_streamout_threads(rzip_control *control);
 void *open_stream_out(rzip_control *control, int f, unsigned int n, i64 chunk_limit, char cbytes);
 void *open_stream_in(rzip_control *control, int f, int n, char cbytes);
-bool flush_buffer(rzip_control *control, struct stream_info *sinfo, int stream);
-int write_stream(rzip_control *control, void *ss, int streamno, uchar *p, i64 len);
+void flush_buffer(rzip_control *control, struct stream_info *sinfo, int stream);
+void write_stream(rzip_control *control, void *ss, int streamno, uchar *p, i64 len);
 i64 read_stream(rzip_control *control, void *ss, int streamno, uchar *p, i64 len);
 int close_stream_out(rzip_control *control, void *ss);
 int close_stream_in(rzip_control *control, void *ss);

util.c

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2013 Con Kolivas
+   Copyright (C) 2006-2016,2021-2022 Con Kolivas
    Copyright (C) 2011 Serge Belyshev
    Copyright (C) 2008, 2011 Peter Hyman
    Copyright (C) 1998 Andrew Tridgell
@@ -100,6 +100,13 @@ void fatal_exit(rzip_control *control)
 	tcsetattr(fileno(stdin), 0, &termios_p);
 
 	unlink_files(control);
+	if (!STDOUT && !TEST_ONLY && control->outfile) {
+		if (!KEEP_BROKEN) {
+			print_verbose("Deleting broken file %s\n", control->outfile);
+			unlink(control->outfile);
+		} else
+			print_verbose("Keeping broken file %s as requested\n", control->outfile);
+	}
 	fprintf(control->outputfile, "Fatal error - exiting\n");
 	fflush(control->outputfile);
 	exit(1);
@@ -110,11 +117,16 @@ void setup_overhead(rzip_control *control)
 	/* Work out the compression overhead per compression thread for the
 	 * compression back-ends that need a lot of ram */
 	if (LZMA_COMPRESS) {
-		int level = control->compression_level * 7 / 9 ? : 1;
+		int level = control->compression_level * 7 / 9;
+
+		if (!level)
+			level = 1;
 		i64 dictsize = (level <= 5 ? (1 << (level * 2 + 14)) :
 				(level == 6 ? (1 << 25) : (1 << 26)));
 
-		control->overhead = (dictsize * 23 / 2) + (4 * 1024 * 1024);
+		control->overhead = (dictsize * 23 / 2) + (6 * 1024 * 1024) + 16384;
+		/* LZMA spec shows memory requirements as 6MB, not 4MB and state size
+		 * where default is 16KB */
 	} else if (ZPAQ_COMPRESS)
 		control->overhead = 112 * 1024 * 1024;
 }
@@ -149,6 +161,15 @@ void round_to_page(i64 *size)
 		*size = PAGE_SIZE;
 }
 
+size_t round_up_page(rzip_control *control, size_t len)
+{
+	int rem = len % control->page_size;
+
+	if (rem)
+		len += control->page_size - rem;
+	return len;
+}
+
 bool get_rand(rzip_control *control, uchar *buf, int len)
 {
 	int fd, i;
@@ -164,7 +185,9 @@ bool get_rand(rzip_control *control, uchar *buf, int len)
 			fatal_return(("Failed to close fd in get_rand\n"), false);
 	}
 	return true;
-}bool read_config(rzip_control *control)
+}
+
+bool read_config(rzip_control *control)
 {
 	/* check for lrzip.conf in ., $HOME/.lrzip and /etc/lrzip */
 	char *HOME, homeconf[255];
@@ -176,11 +199,6 @@ bool get_rand(rzip_control *control, uchar *buf, int len)
 	fp = fopen("lrzip.conf", "r");
 	if (fp)
 		fprintf(control->msgout, "Using configuration file ./lrzip.conf\n");
-	if (fp == NULL) {
-		fp = fopen("/etc/lrzip/lrzip.conf", "r");
-		if (fp)
-			fprintf(control->msgout, "Using configuration file /etc/lrzip/lrzip.conf\n");
-	}
 	if (fp == NULL) {
 		HOME=getenv("HOME");
 		if (HOME) {
@@ -190,8 +208,13 @@ bool get_rand(rzip_control *control, uchar *buf, int len)
 				fprintf(control->msgout, "Using configuration file %s\n", homeconf);
 		}
 	}
+	if (fp == NULL) {
+		fp = fopen("/etc/lrzip/lrzip.conf", "r");
+		if (fp)
+			fprintf(control->msgout, "Using configuration file /etc/lrzip/lrzip.conf\n");
+	}
 	if (fp == NULL)
-		return true;
+		return false;
 
 	/* if we get here, we have a file. read until no more. */
 

util.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (C) 2006-2011 Con Kolivas
+   Copyright (C) 2006-2016 Con Kolivas
    Copyright (C) 2011 Peter Hyman
    Copyright (C) 1998 Andrew Tridgell
 
@@ -20,7 +20,11 @@
 #define LRZIP_UTIL_H
 
 #include "lrzip_private.h"
+#include <errno.h>
+#include <semaphore.h>
 #include <stdarg.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 void register_infile(rzip_control *control, const char *name, char delete);
 void register_outfile(rzip_control *control, const char *name, char delete);
@@ -82,6 +86,7 @@ static inline void failure(const rzip_control *control, unsigned int line, const
 void setup_overhead(rzip_control *control);
 void setup_ram(rzip_control *control);
 void round_to_page(i64 *size);
+size_t round_up_page(rzip_control *control, size_t len);
 bool get_rand(rzip_control *control, uchar *buf, int len);
 bool read_config(rzip_control *control);
 void lrz_stretch(rzip_control *control);
@@ -101,4 +106,66 @@ static inline bool lrz_decrypt(const rzip_control *control, uchar *buf, i64 len,
 	return lrz_crypt(control, buf, len, salt, LRZ_DECRYPT);
 }
 
+/* ck specific wrappers for true unnamed semaphore usage on platforms
+ * that support them and for apple which does not. We use a single byte across
+ * a pipe to emulate semaphore behaviour there. */
+#ifdef __APPLE__
+static inline void cksem_init(const rzip_control *control, cksem_t *cksem)
+{
+	int flags, fd, i;
+
+	if (pipe(cksem->pipefd) == -1)
+		fatal("Failed pipe errno=%d", errno);
+
+	/* Make the pipes FD_CLOEXEC to allow them to close should we call
+	 * execv on restart. */
+	for (i = 0; i < 2; i++) {
+		fd = cksem->pipefd[i];
+		flags = fcntl(fd, F_GETFD, 0);
+		flags |= FD_CLOEXEC;
+		if (fcntl(fd, F_SETFD, flags) == -1)
+			fatal("Failed to fcntl errno=%d", errno);
+	}
+}
+
+static inline void cksem_post(const rzip_control *control, cksem_t *cksem)
+{
+	const char buf = 1;
+	int ret;
+
+	ret = write(cksem->pipefd[1], &buf, 1);
+	if (unlikely(ret == 0))
+		fatal("Failed to write in cksem_post errno=%d", errno);
+}
+
+static inline void cksem_wait(const rzip_control *control, cksem_t *cksem)
+{
+	char buf;
+	int ret;
+
+	ret = read(cksem->pipefd[0], &buf, 1);
+	if (unlikely(ret == 0))
+		fatal("Failed to read in cksem_post errno=%d", errno);
+}
+#else
+static inline void cksem_init(const rzip_control *control, cksem_t *cksem)
+{
+	int ret;
+	if ((ret = sem_init(cksem, 0, 0)))
+		fatal("Failed to sem_init ret=%d errno=%d", ret, errno);
+}
+
+static inline void cksem_post(const rzip_control *control, cksem_t *cksem)
+{
+	if (unlikely(sem_post(cksem)))
+		fatal("Failed to sem_post errno=%d cksem=0x%p", errno, cksem);
+}
+
+static inline void cksem_wait(const rzip_control *control, cksem_t *cksem)
+{
+	if (unlikely(sem_wait(cksem)))
+		fatal("Failed to sem_wait errno=%d cksem=0x%p", errno, cksem);
+}
+#endif
+
 #endif

util/gitdesc.sh

@@ -0,0 +1,105 @@
+#!/bin/bash
+# Peter Hyman, pete@peterhyman.com
+# December 2020
+
+# This program will return commit references based on Tags and Annotated Tags from git describe
+
+usage() {
+cat >&2  <<EOF
+$(basename $0) command [-r]
+all - entire git describe
+commit - commit, omitting v
+tagrev - tag revision count
+major - major release version
+ninor - minor release version
+micro - micro release version
+version - M.mic + [tag release count-HEAD commit]
+-r -- get release tag only
+EOF
+exit 1
+}
+
+# showw message and usage
+die() {
+	echo "$1"
+	usage
+}
+
+# return variables
+# everything, with leading `v' and leading `g' for commits
+describe_tag=
+# abbreviated commit
+commit=
+# count of commits from last tag
+tagrev=
+# major version
+major=
+# minor version
+minor=
+# micro version
+micro=
+# get release or tag?
+tagopt="--tags"
+
+# get whole commit and parse
+# if tagrev > 0 then add it and commit to micro version
+# Expected format is:
+# v#.###-g#######
+init() {
+	describe_tag=$(git describe $tagopt --long --abbrev=7)
+	describe_tag=${describe_tag/v/}
+	describe_tag=${describe_tag/g/}
+	commit=$(echo $describe_tag | cut -d- -f3)
+	tagrev=$(echo $describe_tag | cut -d- -f2)
+	version=$(echo $describe_tag | cut -d- -f1)
+	micro=${version: -2}
+	[ $tagrev -gt 0 ] && micro=$micro-$tagrev-$commit
+	minor=${version: -3:1}
+	major=$(echo $version | cut -d. -f1)
+}
+
+[ ! $(which git) ] && die "Something very wrong: git not found."
+
+[ $# -eq 0 ] && die "Must provide a command and optional argument."
+
+# are we getting a release only?
+if [ $# -eq 2 ]; then
+	if [ $2 = "-r" ]; then
+		tagopt=""
+	else
+		die "Invalid option. Must be -r or nothing."
+	fi
+fi
+
+init
+
+case "$1" in
+	"all" )
+		retval=$describe_tag
+		;;
+	"commit" )
+		retval=$commit
+		;;
+	"tagrev" )
+		retval=$tagrev
+		;;
+	"version" )
+		retval=$version
+		;;
+	"major" )
+		retval=$major
+		;;
+	"minor" )
+		retval=$minor
+		;;
+	"micro" )
+		retval=$micro
+		;;
+	* )
+		die "Invalid command."
+		;;
+esac
+
+echo $retval
+
+exit 0