From f0c31e7ebb73c2199446409c8b96203eb1f79e94 Mon Sep 17 00:00:00 2001 From: Con Kolivas Date: Sun, 27 Feb 2022 21:14:10 +1100 Subject: [PATCH] Update documentation. --- ChangeLog | 1035 +-------------------------------------------------- WHATS-NEW | 15 + man/lrzip.1 | 3 +- 3 files changed, 18 insertions(+), 1035 deletions(-) diff --git a/ChangeLog b/ChangeLog index 82f0cfc..30e7699 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,1034 +1 @@ -lrzip ChangeLog - -NOVEMBER 2019, updates to version 0.631, Peter Hyman - -* Fixups to Assembler code in configure.ac lzma/C/Makefile.am by using -optimized code from p7zip LZMA SDK 16.02. Now works for 64 and 32 bit using -one optimized source. Hack to allow libtool to compile and link assembler -code. - - -JUNE 2016, version 0.630 Con Kolivas, Peter Hyman, Petr Písař, Joel Fredrikson - -* checksum.buf should only be changed after the semaphore wait -* Update README -* Add documentation for recursive mode -* Implement gzip compatible -r recursive option -* Add initial argument processing for recursive option -* Tidy -* Add one more verbose for compat mode -* Add support for various combinations in compat mode -* models is array of chars. char's signess is implementation specific. It's -unsigned on ARMv7. Unsigned char cannot represent negative values. GCC 6 -complains about it: -* Fix decompression of multiple chunk encrypted archives -* Tidy gotos -* Show correct lengths during testing on big endian and compressed archives -* Update copyright dates -* Allow less than maxram to be malloced for checksum to fix Failed to malloc -ckbuf in hash_search2 -* Base temporary output buffer on maximum mallocable, not maxram -* Enable subdir objects for future automake compatibility -* Add support for -m option in lrztar -* Big endian fix for Solaris Sparc courtesy of joelfredrikson. -* Fixed typographical error, changed accomodate to accommodate in README. -* A whitespace fix on lrztar. -* Add sanity check to prevent trying to malloc more ram than a -system/environment is capable of -* Cosmetic help change for compat -* Add rudimentary manpage for lrz -* Fix lrz symbolic linkage -* Do not fail if we are unable to write temporary files, giving a warning only -that it might fail if we don't have enough ram -* Try /tmp/ if none of the temporary environment directories or the current -directory are writeable -* Set STDOUT correctly in compat mode -* Style police -* Fix false warning on decompressing from stdin without keep files -* Fix false warning on compressing from stdin without keep files -* Don't show extra message in compat mode decompress -* Show correct appname when called in compat mode -* Add support for progress, fast and best flags in compat mode -* Add compatibility mode with gzip when called as lrz -* Correct adding slash to control->tmpdir. off-by-one error. -* Update manpage for long options - - -MARCH 2015, version 0.621 Con Kolivas, Peter Hyman - -* Fix libzpaq.cpp warnings -* Fix warnings in LzmaLib.c -* Making the high buffer only one page size is faster for sliding mmap mode -* Fix incompatible log callback in liblrzip -* Use PRId64 instead of lld in fscanf -* Use int64_t for i64 -* Fix ISO C warning in lrzip.c -* Fix ISO C warning in lrzip.c -* Fix ISO C warning in runzip.c -* Fix ISO C warnings in rzip.c -* Fix iso c warning in util.c -* Fix EOL to unix on libzpaq -* control->tmpdir fixes -* Null terminate long options to find unhandled options -* Trivial date -* Add long option support -* Brace failure lead to broken dump to stdout -* fflush messages on print_err -* Fix inverse logic -* Honour the proper unix environment variable for temporary directory and then -try variations -* Attempt to decompress file in ram only if we are unable to create temporary -files when decompressing to stdout -* Avoid checking return values repeatedly on compress that only lead to failures -anyway -* Microoptimisation -* Don't check for failure condition that can no longer occur -* Don't check twice for failure conditions that are fatal in hot paths -* Cache the chunk bytes value to avoid setting it on each read_header call -* fake_mremap is only used when defined to mremap -* Remove unused cksem functions -* Fix remaining use of mutexes lock/unlocking in different threads with cksems, -correcting cksem usage on osx -* Update copyright dates -* Make match_len a function completely removing all indirect calls to get_sb, -significantly speeding up the single_get_sb case -* Make full_tag a pointer allowing us to avoid a function call for get_sb -* Call sliding_get_sb directly in sliding_next_tag -* Make next_tag a pointer to allow ordinary mapping to avoid an extra function -call -* fix order of lrzip.conf search - - -MARCH 2015, version 0.620 Con Kolivas, Peter Hyman, Chris Spiegel, Elmir Jagudin - -* Increase maxram when we abandon use of temporary input/output buffers -* Don't delete the tmpinfile when decompressing from stdin before allowing seek -to end to succeed in checking md5 -* Use temporary file from read_seekto when STDIN will not fit in the ram input -buffer -* Remove unused read_i64 function -* Add message about issue tracker in BUGS -* Use a common exit path iin lrzip_compress/decompress and fix lr leak on -successful return -* Fix parenthesis placement inside of unlikely(). -* Clear sa_mask and sa_handler before calling sigaction(). -* Fix for lrzip -i. Decompressed size wrong -* added '-m' command line option -* Fix wrong README file being included in Makefile -* Pass strict sizes to decompress length, rounding up only the amount we're -allocating to not confuse decompression libraries -* Convert the thread locking to use cksems -* Add cksems to util.h -* Fix 'Failed to malloc ckbuf in hash_search2' with very large files. -* Round up compression and decompression buffers to page size since malloc will -allocate them that large anyway. -* Increase the compressed buffer size given to libzpaq in case of incompressible -data since it does not check if it's trying to write beyond the end of the -buffer. -* Provide a helper function to round a value up to the nearest page size for -malloc optimisations. - - -SEPTEMBER 2013, version 0.616 George Makrydakis - -* Making things more homogeneous in checks, some space removed. -* Simplifying and refactoring logic, alignment edits, empty semantic rewire. -* Target directories must always exist and -f should not create them. -* Deferring output directory after -f check has taken place. -* Preserve extraction semantics without resorting to tar stripping. -* Making sure last forward slash is removed from input path. -* Removing pointless forced overwrite point check. - - -SEPTEMBER 2013, version 0.615 Con Kolivas, George Makrydakis - -* Adjusting -O flag semantics, options help update. -* Making -O flag operative for lrztar, whitespace path fix, lrzuntar fix, other -minor. -* Further tighten up ram restrictions with stdin/stdout to prevent running out -of memory with all the buffers involved. -* Massive files fail with -U due to trying to allocate the whole lot in ram -while doing checksums. Do it piecemeal to avoid the problem. Patch and debugging -courtesy of Adam Tkáč. -* We have to run through the clear buffer function even for empty buffers or -corrupt archives with empty match streams. -* MD5 code uses little endian so remove arbitrary SWAP macro and explicitly use -htole32. -* Rewrite the magic if we receive lzma properties and have not yet written them -yet during stdout operation. -* Set the control lzma properties only once. -* Add a control lock mutex for protecting certain control variables. -* Fix stdin fake mremap creating null bytes on osx, patch courtesy of John -Boyle. -* Cache frequently used indirectly referenced variables in the sliding mmap -code. -* Micro-optimise sliding_get_sb_range. -* A fix for a bug where large files containing the same non-zero bytes which -requires a sliding window, courtesy of Serge Belyshev. -* Put vchar should take a 64 bit integer and is used from more than one call -site so uninline it. -* Microoptimise in hash_search. -* Inline rzip functions used from only one caller. -* Check for successful calloc of hash table only after performing it. -* stdin_eof is just a bool -* hash_bits can only be up to 64 so use a char type. -* Check endianness of build with autotools to enable md5 support on more -platforms. - - -JULY 2012, version 0.614 Con Kolivas -* Fix lrztar failing due to incorrect fix for -S freeing segfault. - - -JULY 2012, version 0.613 Con Kolivas, Serge Belyshev. -* Fix other non-win32 compilation for libzpaq. -* Fix libzpaq compilation on other non-win32 OSs. -* Fix Types.h missing which breaks typedefs on macosx. -* Cope with missing ffsll() on various platforms. -* Fix get_ram compilation on osx. -* Rename lrzip.h to lrzip_core.h to cope with case insensitive filesystems not -working with Lrzip.h vs lrzip.h -* Include test for pthread.h and include header in lrzip_private.h. -* Use system extensions in configure.ac to allow to build on non-gnu-linux -platforms. -* Fix invalid free error when -o and -S are used together. -* Use no back end compression for blocks smaller than 64 bytes to avoid issues -to do with minimum buffer size and minimum match sizes of the rzip -preprocessing. -* Fix md5 calculaton on blocks larger then 2^32. Bug reproduced and debugged by -Serge Belyshev. Patch by Paul Eggert. - - -MARCH 2012, version 0.612 Con Kolivas -* Minor formatting improvement to delete misaligned %. -* Implement progress counter on zpaq compress/decompress per thread within the -Reader function. -* Add some lrzip style defines and optimisations to the lrzip functions in -libzpaq.h -* Re-instantiate the lzo compresses test for zpaq compression. -* Remove redundant code. -* Delete now-unused zpipe code. -* Implement virtual write support for bufWrite to speed up zpaq support and -simplify the virtual read function for bufRead. -* Add virtual read() support to the bufRead class to speed up encoding/decoding -via the zpaq library. -* Use libzpaq backend for decompression. -* Move zpaq compression to new libzpaq library back end. -* Merge branch 'master' into zpaq -* Add minimal libzpaq compilation support. -* Move to libzpaq directory. -* Import libzapq files. -* Fix Lrzip.h not being installed into include directory and make distcheck -failing. - - -MARCH 2012, version 0.611 Con Kolivas. -* Move the update counter to earlier in the hash_search so it still updates even -when there are not matches. -* Perform all checksumming in a separate thread to speed up the hash search in -the rzip phase. -* Optimise the hot function empty_hash. -* Micro-optimise the hot next_tag function. -* Micro-optimise the match_len function. -* Pass one less variable to the get_sb function as an optimisation. -* Cache the hash_entry value passing one less variable to hash_empty as a -micro-optimisation. -* Use ffsl for a faster lesser_bitness function. -* Prevent lrzuntar from automatically overwriting existing directories without -the -f option being specified. -* Calling lrzip via lrzcat would fail as the proper parameters weren't being -passed to the control structure. Fix. - - -MARCH 2012, version 0.610 Con Kolivas, Michael Blumenkrantz -* Implement complete set of liblrzip libraries, documentation and example uses -with support for simple lrzip_compress() and lrzip_decompress() or complete -fine-grained control over all compression and decompression options. -* Use as much of the low buffer as possible with a single memcopy before going -fine grained byte by byte. -* Preserve the compressed time on decompression where suitable. -* Store a copy of the control struct to be reused on subsequent files to prevent -variables being modified in the control struct on the first file that corrupt -compression/decompression of the 2nd file. -* Explicitly select C99 to avoid certain warnings. -* Generic modifications to silence -Wextra warnings. -* Fix typos. -* Use an array of parameters in lrztar to allow working with directories with -spaces in their names. - - -SEPTEMBER 2011, version 0.608 Con Kolivas -* get_sb only allows accessing one byte at a time, yet we don't need that -functionality when sliding mmap is not in use. Use different versions of the -function and the larger memcpys depending on whether sliding mmap is in use or -not. This affords a substantial speedup in the rzip phase of files not requiring -sliding mmap. -* A small optimisation of moving the check for remapping the low buffer also -speeds up the sliding mmap version slightly. -* Installation of the included bash completion script was disabled to avoid -conflict with distribution bash completion packages. -* Minor cleanup of sha4.h -* Make mremap use the fake mremap on all platforms not linux. - -SEPTEMBER 2011, version 0.607 Con Kolivas -* Updated lzma library to version 920. -* Fixed a rare unable-to-decompress corner case. -* Added lrzip completion script. -* Updated makefile to more portable posix version. -* Hopefully fixed compilation on FreeBSD not supporting memopen. -* Added lots more debugging information about offsets in max verbose mode. -* Removed error and warning messages that are spammy or harmless. - -MAY 2011, version 0.606 Con Kolivas -* lrzuntar broke as lrzip doesn't automatically use stdout now, so use lrzcat -in the lrztar script instead. - -MAY 2011, version 0.605 Con Kolivas -* Addition of explicit lrzcat symlink which automatically sets "-d -o -" -* Revert automatic output to stdout due to addition of explicit lrzcat -executable, and to be consistent with gzip. -* Fix counter from being zeroed on every loop in rzip.c which was creating -excess progress output. -* Update GNU license. -* Set aside at least page_size of zeroes when compressing a chunk, thus -allowing files that are otherwise too small to compress with lrzip to be -compressed. The zeroes will just be dropped on decompression. -* Add explicit checks to cope with zero sized files as well. -* Remove the arbitrary 128 byte minimum size restriction on archives now that -any size file can be safely archived. -* Set the eof flag when reading older file format files to not hang on zero -byte sized archives. -* Update the percentage counter when getting file info to not overflow with -tiny files. -* Set the maximum file mode to 0666 to avoid setting the executable bit when -permissions are not carried over (such as from STDIO). - -APRIL 2011, version 0.604 Con Kolivas -* Detach threads after creating them on the compression side. Not joining them -meant that compressing massive files requiring hundreds of threads would -eventually hit the resource limit of number of threads created even though -the threads themselves would exit. - -APRIL 2011, version 0.603 Con Kolivas, George Makrydakis, Jari Aalto. -* lseek in stream.c wasn't being compiled to the lseek64 variant on Apple -due to missing includes, breaking >2GB files. Added includes. -* Detect when stdout is being redirected and automatically direct output to -stdout unless a filename is specified. -* Update lrztar to properly support -S -O and -o, and use new syntax not -requiring '-o -' for stdout. -* Update lrzip.conf to support encryption. -* Do a sanity check to ensure lrzip is not attempting to work on a directory. -* Typo fixes. - -APRIL 2011, version 0.602 Con Kolivas -* Fixed the symlinks breaking package generation. -* Made maximum chunk allocable on 32bits 2/3 of a GB again limiting total ram -used to 2GB, regardless of how much crazy ram people have put on such systems -since the rest is unusable. -* Now builds on Cygwin courtesy of Тулебаев Салават. -* Check for podman on ./configure. -* Disable md5 generation and checking on Apple till it's fixed. -* Display full version when -V is invoked. -* Don't restrict file permissions from unknown source, respecting umode -instead. - -MARCH 2011, version 0.601 Con Kolivas -* Fixed the symlinks for lrzuntar, lrunzip not being installed. -* Fixed the pod-based manpages not being installed. -* Clearly demonstrate that ASM is not supported on 64bits. - -MARCH 2011, version 0.600 Con Kolivas -* Massive rewrite with new file format to accomodate new features. -* Allocate temporary buffers of safely sized ram that can act as temporary -storage for performing de/compression to/from STDIN/STDOUT without requiring -temporary physical files. Files compressed on machines with much larger ram -being decompressed on smaller ram machines may still require temporary files, -but as much as possible is done using in-ram decompression, and minimally -sized temporary files. Information displayed is more verbose and accurate in -STDIN/STDOUT mode. -* The temporary buffers created for decompressing to STDOUT are also used -for decompressing regular files now avoiding multiple write/reads when -re-constructing the file on decompression. This can dramatically speed up -the rzip phase of decompression on complicated files with many small matches, -and always makes testing faster. -* Compress block headers as well now since we know how many bytes can be used -to describe the length of the block, decreasing overall file size. -* Store the rzip chunk size per chunk to make it possible to check total file -size by summating each rzip chunk size when it's not known till the end (as -happens when compressing from STDIN). -* Implement password protected encryption. Import the polarssl code for -sha512 and aes128 routines. Read password without echoing to screen by -disabling echo via terminfo. Take the password then multiply hash it -according to the date it was generated. Inrease the number of hashes according -to Moore's law so it always takes approximately 1 second per password on the -most modern hardware when first encrypted. Hash the password against 8 bytes -of salt which is a combination of the 2 byte encoded loop counter (for how -many times to hash the password) and 6 random bytes. Take random from -/dev/urandom if it's available and fall back to random() if not. Encrypt each -block of compressed data with 8 extra bytes of random salt. Once the headers -are written, go back and encrypt the headers as well. Then encrypt the md5 -hash value as well. Anything beyond the initial lrzip magic header should -apppear as random data and no two successive encryptions of the same data with -the same password should generate the same data. -* New build system should be more robust and portable. -* Abstract out functions better into separate files and headers, and remove -all use of global variables. This will make the generation of an lrzip -library possible in the future. -* Prevent testmalloc from coming up with a negative number when determining -how big a block of memory to allocate by decreasing the number of threads to -be used and then aborting to a minimum value should it still be too much -apparent ram. -* Big Endian fixes. -* Change lrztar and lrzuntar to not use temporary files. -* Countless other fixes, documentation and cleanups. - -MARCH 2011, version 0.571 Con Kolivas -* Only retry mmaping if it's a memory error, otherwise it may give spurious -errors. -* Check for free space before compression/decompression and abort if there -is inadequate free space if the -f option is not passed. -* Fix the wrong check in rzip.c which was rounding down the page size and -making for one extra small chunk at the end. -* Check the correct stdout when refusing to pipe to a terminal. -* Fix windows EOL on lzma.txt. -* Ignore what stdout is going to when testing from stdin. -* More verbose summary after we know whether we have stdin/out to more -accurately reflect the window that will be used. -* Updated gitignore -* Unlink temporary files immediately to avoid files lying around. -* Check free space AFTER reading magic, and not when decompressing to stdout. -* Don't dump output to stdout when just testing a file. -* Dump the temporary file generated on emulating stdout on decompression after -every chunk is decompressed instead of after the whole file is decompressed. -* Don't attempt to file check decompression when it's not meaningful (test -only or to stdout). -* Clean up and simplify the times displayed component. It could potentially -show thousands of seconds. -* Remove open coded strerror messages when they're to be printed by fatal(). - -MARCH 2011, Michael Blumenkrantz -* Updated autotools/conf build system. - -FEBRUARY 2011, version 0.570 Con Kolivas -* Change the lzo testing to a bool on/off instead of taking a parameter. -* Clean up the messy help output. -* Refuse to read from or write to terminal in stdin/stdout mode. -* Delete temporary files generated when testing from stdin. -* Don't call perror on failures that aren't system related. -* Improve visual output. -* Significant multi-threading speed-ups - as the spawning of threads is -partially serialised, use one more thread than CPUs. -* Don't split up the chunks into multiple threads when no back end -compression is being used. -* Take into account the precise amount of overhead that the memory hungry -back-end compression modes lzma and zpaq use when determining how much ram to -allocate, and limit total back-end compression ram to 1/3. -* Deprecate the -M option as it is no longer of any significant utility -compared to the -U option. -* Strip symbols by default in compilation and add -W -Wall to CXXFLAGS to be -consistent with CFLAGS. -* Update benchmarks. -* Add verbose and max verbose modes for -i information giving a breakdown of -each rzip chunk and compressed block. -* Modify purpose of -T threshold option to mean Disable LZO pre-compression - testing. Threshold level test deprecated. -* Make failure to allocate a buffer in the compress_ functions put out an -error message always rather than just in max_verbose mode. - -FEBRUARY 2011, Peter Hyman -* Minor updates to man pages, lrzip.conf example file. -* Update main help screen to include environment settings. -* Update to respect $TMP environment variable for TMP files. -* Updated control structure to include tmpdir pointer. -* Update lrzip.conf parser to respect -U, -H, -T, k, and -c options. -* Update lrzip.conf example to include new parameters. -* Reorder main Switch loop in main.c for readability. -* Have UNLIMITED and control.window be exclusive. UNLIMITED wins. -* Corrects heuristic computation in rzip.c which would override - UNLIMITED if control.window set -* Show heuristically computed control.window when computed. -* Remove display compression level from control.window verbose output. -* Update print_verbose format for Testing for incompressible data in stream.c - to omit extra \n. - -FEBRUARY 2011, version 0.560 Con Kolivas -* Rewrote the locking implementation to use pthread_mutexes instead of -semaphores. This makes multi-threading work on OSX. -* Re-use a single malloc in unzip_match instead of multiple malloc/frees. -* Imported the md5 code from coreutils and modified them to be used by lrzip. -* Implemented md5 hash checking on compression and decompression by using -buffers to pass to the existing crc checking and re-using them for the md5 -checking. -* Sped up the hash checking on compression by not reading the whole chunk at -once to do the hash check, and instead read one paged size chunk. -* Store the md5 hash value at the end of the archive, maintaining backward -compatibility with existing 0.5 versions of lrzip, and place a flag in the -header denoting its existence. -* Use the header flag for md5's existence to determine what hash check to -perform on decompression. -* Add hash value display to maximum verbose output and in information mode. -* Add -H option to enable hash value displaying without verbose mode. -* Display what hash checking will be used on decompression in verbose mode and -in information displayed with -i. -* Add -c option to perform a hash check on the file generated on decompression, -comparing it to that stored in the lrzip archive, or that generated on -decompression on archives that do not have it stored. -* Delete broken or damaged files generated on interruption or when they fail -integrity testing by default. -* Implement the -k option to keep broken or damaged files. - -FEBRUARY 2011, version 0.552 Con Kolivas -* Fix incompressible blocks causing failure with bzip2 and gzip. -* Freebsd doesn't do mremap, use the fake one. -* Update copyright notices. -* Flush writes to disk before allocating ram. -* Relax memory allocation when no back end compression will be used. -* Clean up unnecessary semaphore usage in decompression. -* Fix minor typos. -* Convert mmaps to malloc in runzip as they may silently fail if not a -multiple of page size, thereby leading to silent corruption on decompression. -* Return the correct length by unzip_literal. -* Use wrappers to serialise threads on OSX and bypass use of unnamed semaphores -which aren't supported. - -DECEMBER 2010, version 0.551 Con Kolivas -* Fix stdin failing due to getting sizes all wrong. -* Fix stdin compression values not being shown at end. -* Fix inappropriate failure when lzma doesn't compress block. - -DECEMBER 2010, version 0.550 Con Kolivas -* Move the threading on compression to higher up in the code, allowing the next -stream to start using compression threads before the previous stream has -finished. This speeds up compression on files that take more than one pass to -compress. -* Limit the number of threads decompressing stream 0 to just 1 since it's always -followed by stream 1 chunks, and it may lead to failure to decompress due to -running out of memory by running too many threads. -* Default compression level and window size on lzma is set to 7 which is the -highest it goes. Scale the 9 lrzip levels into 7, thus making the default lzma -level 5 which uses a lot less memory and is substantially faster at the cost of -some compression. -* Rationalise the memory testing now that the default lzma settings use a lot -less ram by default, and make all systems use no more than 1/3 ram in one mmap. -This allows larger windows to be used by 32 bit at last without memory -allocation errors. -* Revert "Make threads spawn at regular intervals along chunk size thus speeding -up compression" as it actually slowed it down instead of speeding it up. -* Cope with compression/decompression threads failing by waiting till the -previous thread has finished its work, thus serialising the work and using less -ram, making success more likely. -* Fix some dodgy callocs which weren't really allocating enough ram. -* Destroy semaphores used in stream_in on closing the stream. -* Minor output improvements. - -DECEMBER 2010, version 0.544 Con Kolivas -* Make multiple stream 0 entry decompression more robust by creating separate -thread groups for stream 0 and stream 1. -* uclibc fails to return valid values for memory size with sysconf so create -a workaround. -* 32 bits has memory allocation errors on large files so be ruthless with -limiting window sizes and use more sliding mmap. -* Split up compression streams according to proportion of hash search thus -evenly distributing work to threads based on raw data length rather than -rzip pre-processed data. This makes for more CPU usage on compression and -smoother write out on decompression with small speed ups. -* Remove unnecessary argument passing. -* Fix -U causing a floating point error with ultra small sized files. -* 32 bits still sometimes fails the lzma compression stage due to internal -memory allocation issues, so use bzip2 as a fallback on blocks that fail to -compress. -* Minor cleanups. - -NOVEMBER 2010, version 0.543 Con Kolivas -* Fix for when there are multiple stream 0 entries per chunk, which would -cause a failure on decompression. -* Fix lzma windows being set far too small by mistake at all times. -* Decrease nice level of main process compared to back end threads as it tends -to be the rate limiting step. - -NOVEMBER 2010, version 0.542 Con Kolivas -* Choose sane defaults for memory usage since linux ludicriously overcommits. -* Use sliding mmap for any compression windows greater than 2/3 ram. -* Consolidate and simplify testing of allocatable ram. -* Minor tweaks to output. -* Round up the size of the high buffer in sliding mmap to one page. -* Squeeze a little more out of 32 bit compression windows. -* Fix sliding mmap from not working on 2nd pass onwards. - -NOVEMBER 2010, version 0.541 Con Kolivas -* Fix wrong number of passes reported. -* Re-fix the off-by-one that wasn't off-by-one. -* Limit lzma compression windows to 300MB as per reports of failures with larger -windows. - -NOVEMBER 2010, version 0.540 Con Kolivas -* Massive rewrite of backend decompression phse, implementing multithreading. -This is done by taking each stream of data on read in into separate buffers for -up to as many threads as CPUs. As each thread's data becomes available, feed it -into runzip once it is requests more of the stream. Provided there are enough -chunks in the originally compressed data, this provides a massive speedup -potentially proportional to the number of CPUs. The slower the backend -compression, the better the speed up (i.e. zpaq is the best sped up). -* Fix the output of zpaq compress and decompress from trampling on itself and -racing and consuming a lot of CPU time printing to the console. -* When limiting cwindow to 6 on 32 bits, ensure that control.window is also set. -* When testing for the maximum size of testmalloc, the multiple used was out by -one, so increase it. -* Minor output tweaks. -* Build warning fixes. -* Updated benchmarks. - -NOVEMBER 2010, version 0.530 Con Kolivas -* Massive rewrite of backend compression phase. Now the stream is split up -into as many chunks as there are CPUs, of at least 10MB in size, that are -still mallocable. Once the stream has reached a chunk of this size, its buffer -is handed to a new backend compression thread which works while the rzip stream -continues processing. This has the effect of parallelising workloads almost -linearly up to the number of CPUs on the slower compression backends. ZPAQ, -in particular, is effectively 4x faster on quad core now. Decompression is -unchanged. -* Added the -p option to allow the number of processors to be specified to -override the detected number. -* Changed the default level back to 7 as 9 wasn't offering significanly more -compression but was adding time. -* Increased the size of all the buffers to other backends now as well, since -each block adds overhead with its header. -* Numerous alterations to screen output to cope with new threaded compression -phase. -* Deprecated the -P option since not setting the file permissions only -generates a warning now, not a failure. -* Updated docs and benchmarks. - -NOVEMBER 2010, version 0.520 Con Kolivas -* Distros don't like 3 point version numbering so just repackaged as 0.520. - -NOVEMBER 2010, version 0.5.2 Con Kolivas -* Fixed the Darwin build, again. -* Fixed cases of extreme ram usage on 32 bit failing by limiting zpaq to 600MB -windows as well. -* Check page size if we can instead of assuming it's always 4k. -* Improve the progress output. -* Change failure to chmod and failure to set nice level to warnings only. -* Standardise what's a stderr message and what's output. - -NOVEMBER 2010, version 0.5.1 Con Kolivas -* Fix Darwin build - Darwin doesn't support mremap so introduce a fake wrapper -for it. -* Fix the memopen routines, a wrongly implemented wrapper for Darwin equivalents -was also using the faked versions on all builds. -* Fix dodgy ordered includes. -* Clean up excessive use of #ifdefs -* Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of -accessing the buffer directly, thus allowing us to have window sizes larger than -available ram. This is implemented through the use of a "sliding mmap" -implementation. Sliding mmap uses two mmapped buffers, one large one as -previously, and one smaller one. When an attempt is made to read beyond the end -of the large buffer, the small buffer is remapped to the file area that's being -accessed, while the larger one is remapped as the search progresses along the -file. While this implementation is potentially much slower than direct mmapping, -it allows us to implement unlimited sized compression windows. -* Implement the -U option with unlimited sized compression windows. -* Rework the selection of compression windows. Instead of trying to guess how -much ram the machine might be able to access, we try to safely buffer as much -ram as we can, and then use that to determine the file buffer size. Do not -choose an arbitrary upper window limit unless -w is specified. -* Rework the -M option to try to buffer the entire file, reducing the buffer -size until we succeed. -* Align buffer sizes to page size. -* Clean up lots of unneeded variables. -* Fix lots of minor logic issues to do with window sizes accepted/passed to rzip -and the compression backends. -* More error handling. -* Change -L to affect rzip compression level directly as well as backend -compression level and use 9 by default now. -* Fix file size reporting on compressed files generated from stdin. -* More cleanups of information output and more information. -* Add chunk percentage to progress update. -* Reinstated the 2GB buffer limit on 32 bit machines during compression, though -the -U mode can work around it now. -* Code micro-optimisations. -* Use 3 point release numbering in case one minor version has many subversions. -* Numerous minor cleanups and tidying. -* Updated docs, manpages, and benchmarks. - -NOVEMBER 2010, version 0.5 Con Kolivas -* Changed offset encoding in rzip stage to use variable byte width offsets -instead of 64 bits wide. Makes for better compression and slightly faster. -* Write the byte width into the file before each block. -* Shrunk match lengths to maximum of 16 bits again as per original rzip as the -larger offsets did not achieve greater compression and made final size larger. -* New file format not backwards compatible due to variable byte widths. -* Rewrote memory initialisation to have a pre-allocation stage to try and -find the maximum memory usable and defragment ram. -* Use reduced window size if allocating memory fails at higher size. -* Change use of malloc to mmap to make it possible to address up to 44 bit -sized offsets even on 32 bit machines on decompression. Still unable to use -greater than 2GB windows on 32 bit machines and unsure if this is fixable. -* Reworked the STDIN code to use an anonymous mmap and read in stdin into this -to make it possible to compress from STDIN without the need for temporary -files. As the file size is not known in advance, memory allocation is set to -large and byte width to equivalent size. -* Reallocation of ram where possible to minimise risk of running out of memory -in the middle of a compression phase, and flushing to disk to empty dirty ram -for the same reason. -* More robust fatal warnings. -* Numerous cleanups and tidying of code and addition of comments. -* Updated documentation to reflect changes. - -OCTOBER 2010, version 0.47, Con Kolivas -* Fix the symlinking problem when DESTDIR is in use reported by a billion -people. - -MAY 2010, version 0.46, Con Kolivas, Ed Avis. -* Suppress final [OK] message with -q flag EA -* Handle mkstemp() errors correctly EA -* Add lrzuntar manpage -* Update manpages - -APRIL 2010, version 0.45, Con Kolivas, Jon Tibble, George Makrydakis -* Fixes the nasm program test (AC_CHECK_PROG doesn't overwrite a -variable that is already set so do it manually) JT -* Fix compiler flags as not all compilers accept -Wall -W (cc on -Solaris/OpenSolaris) JT -* Fix lrztar to not try to compress files already with the .lrz extension GM -* Fix lrztar to decompress files where the pathname is ../* GM -* Add lrzuntar symlink to call lrztar -d - - -MAR 2010, version 0.45, Con Kolivas, Jari Aalto -* Fixed reported window size -* Fixed 32bit windows being attempted to be larger than contiguous amounts -by taking into account VM kernel/userspace split of 896MB. -* Minor code cleanups -* Added lrztar and lrunzip docs -* Fix minor typos -* Added distclean and maintainer-clean make targets - - -DEC 2009, version 0.44, Con Kolivas, George Makrydakis -* Added lrztar wrapper to manage whole directories. -* Added -i option to provide information about a compressed file. -* Fixed "nan" showing as Compression speed on very small files. -* Fixed build for old bz library. -* Avoid overwriting output file if input doesn't exist. -* Implement signal handler to delete temporary files. - - -DEC 2009, version 0.43, Con Kolivas, Jukka Laurila -* Darwin support thanks to Jukka Laurila. -* Finally added stdin/stdout support due to popular demand. This is done -by basically using temporary files so is a low performance way of using -lrzip. -* Added test function. This just uses a temporary file during decompression. -* Config files should now accept zpaq options. -* Minor code style cleanups. -* Updated benchmarks in docs. -* Add a warning when attempting to decompress a file from a newer lrzip -version. - - -NOV 2009, version 0.42, Con Kolivas -* Changed progress update to show which of 2 chunks are being compressed -in zpaq. -* Fixed progress update in ZPAQ to not update with each byte which was -wasting heaps of CPU time. - - -NOV 2009, version 0.41, Con Kolivas -* Added zpaq compression backend for extremely good but extremely slow -compression (incompatible with previous versions if used). -* Limited chunk size passed to LZMA to 4GB to avoid library overflows. -* Minor changes to the formatting output -* Changed lower limit of -T threshhold to 0 to allow disabling it. -* Added lzo_compresses check into zpaq and bzip2 as well since they're -slow. - - -NOV 2009, version 0.40, Con Kolivas -* Massive core code rewrite. -* All code moved to be 64bit based for compression block addressing and length -allowing compression windows to be limited by ram only. -* 64bit userspace should now have no restriction on compression window size, -32bit is still limited to 2GB windows due to userspace limitations. -* New file format using the new addressing and data types, incompatible with -versions prior to 0.40. -* Support for reading and decompressing older formats. -* Minor speedups in read/write routines. -* Countless minor code fixes throughout. -* Code style cleanups and consistency changes in core code. -* Configure script improvements. - - -NOV 2009, version 0.31, Con Kolivas -* Updated to be in sync with lzma SDK 9.07beta. -* Cleanups and fixes of the configure scripts to use the correct package version -name. -* Massive fixes to the memory management code fixing lots of 32bit overflow -errors. The window size limit is now 2GB on both 32bit and 64bit. While it -appears to be smaller than the old windows, only 900MB was being used on .30 -even though it claimed to use more. This can cause huge improvements in the -compression of very large files. -* The offset when mmap()ing was not being set to a multiple of page size so -it would fail if the window size was not a multiple of it. -* Flushing of data to disk between compression windows was implemented to -minimise disk thrashing of read vs write. - - -NOV 2009, version 0.30, Con Kolivas -* Numerous bugfixes to try and make the most of 64bit environments with huge -memory and to barf less on 32bit environments. -* Executable stacks were fixed. -* Probably other weird and wonderful bugs have been introduced. -* -P option to not set permissions on output files allowing you to write to -braindead filesystems (eg fat32). - - -JAN 2009, version 0.24, Peter Hyman, pete@peterhyman.com -Happy New Year! -* Upgrade LZMA SDK to 4.63. Use new C Wrapper. Invalidates - LZMA archives created earlier due to new Magic property - bytes. -* New LZMA logic will automatically determine allow LZMA - code to determine optimal lc, lp, pb, fb, and dictionary - size settings. stream.c will only pass level and thread - information. Compress function will return encoded 5 byte - data with compression settings. This will be stored in lrz - file header. -* add error messages during LZMA compression. There are some - edge cases where LZMA cannot allocate memory. These errors - are reported and the user will be advised to use a lower - compression window setting. -* type changes in rzip_fd function for correctness. -* remove function *Realloc() since it was never used. Cleaned - in rzip.h and util.c. -* apply munmap prior to closing and compressing stream in - function rzip_chunk in rzip.c. -* add realloc function in close_stream_out in stream.c - to reclaim some ram and try and allieviate out of memory - conditions in LZMA compression. -* remove file acconfig.h and include DEFINE in configure.in. -* add lrzip.conf capability. -* add timer for compression including elapsed time and eta. -* add compression and decompression MB/s calculation. -* Updated WHATS-NEW, TODO and created BUGS file. -* Updated lrzip.1 manpage and created lrzip.conf.5 manpage. -* Added lrzip.conf.example file in doc directory. - -MAR 2008, Con Kolivas, kernel@kolivas.org -* Numerous changes all over to place restrictions on window -size to work with 32 bit limitations. -* Various bugfixes with respect to detecting buffer sizes and -likelihood of compressibility. -* Fixed the inappropriate straight copying uncompressed data for -files larger than 4GB. -* Re-initiated the 10MB window limits for non-lzma compression. -I was unable to reproduce any file size savings. -* Allow compression windows larger than ramsize if people really -really want them. -* Decrease thresholds for the test function to a minimum of 5% -compressibility since the hanging in lzma compression bug has been -fixed. - -JAN 2008, version 0.22, Peter Hyman, pete@peterhyman.com -* version update -lzma/LZMALib.cpp - Thanks to Lasse Collin for debugging the problem LZMA - had with hanging on uncompressable files. - Update for control parameters to both compress and - decompress functions. -Makefile.in -* use of @top_srcdir@ (Lasse Collin). Also moved away - more cruft. -main.c stream.c.rzip.h LZMALib.cpp lzmalib.h -* addition of three new control structure members. - control.lc -- literal context bits - control.lp -- literal post state bits - control.pb -- post state bits - These are needed to ensure decompression will work. - These will now be stored along with control.compression_level - in the lrz file beginning at offset 0x16 for three bytes. - These will be passed to the functions lzma_compresses and - lzma_uncompress. Currently, only compression level is - needed or used, but the others are stored for possible future - use. - See magic file for more information. -stream.c -* Change to lzo_compresses function that will reject a chunk - without testing it if the size of the chunk is greater - than the compression window * threshold. This is to avoid - a low probability that lzma would still be passed a chunk - that contains uncompressible data or barely compressible - data. If after rzip hashing the chunk size is still close - to the window size, there is hardly anything worth - compressing. While there is no reason lzma cannot get the - chunk, this will save a lot of time. -magic.headers.txt -* updated file to show new layout that includes lzma - parameters. -README-NOT-BACKWARD-COMPATIBLE -* added warning about using lrzip-0.22 with earlier versions. -WHATS-NEW -* highlight of new features. - -DEC 2007, version 0.21. Peter Hyman, pete@peterhyman.com -* version update. -* Modified to use Assembler routines from lzma SDK for CRC - computation when hashing streams in rzip.c and runzip.c. - Added files 7zCrcT8.c and 7zCrcT8u.s to lzma tree. - Cleaned up source tree. Moved unused files out of the way. - Moved non-core docs to doc directory -configure.in -* correct AC_INIT to set program variables. -* modified to add check for nasm assembler. -* modified syntax of test for errno in error.h to use - echo $ECHO_N/$ECHO_C instead of $ac_n/$ac_c which - was incorrect. -Makefile.in, lzma/Makefile -* modified to add compile instructions for 7zCrcT8.c - and 7zCrcT8U.s and Assembler. Cleaned up to remove - targets that don't exist or sources that don't - exist. - Modified to properly set directories. Added doc install. - Add link command to symlink lrunzip to lrzip. -*main.c - Add CrcGenerateTable() function to init CRC tables. - This is needed for all crc routines including those - in MatchFinderMT. -rzip.c and runzip.c -* Updated source to change call to crc32_buffer to call - CrcUpdate in the assembler code. Changed parameter order - to conform. -stream.c -* Removed 10MB limit on streams for bzip, gzip, and lzo. - This, to improve effeciency of long range analysis. For - some files, this could improve results. -Current-Benchmarks.txt -* Added file to keep benchmarks current to version. - (probably need to update README too). -README.Assembler -* Explain how to remove default compile of Assembler - modules. -config.sub config.guess -* added files for system detection. - -DEC 2007, version 0.20. Peter Hyman, pete@peterhyman.com - -* Updated to LZMA SDK 4.57. -* Updated to p7zip POSIX version. (www.p7zip.org) -* Added multi-threading support (up to 2x speed with LZMA). -* Edited LZMADecompress.cpp for backward compatibility - with decompress function. Needed SetPropertiesRaw function. -* Repopulated source tree for distribution. -* Updated Makefile.in to reflect new source files. - Updated to include command to link lrunzip to lrzip because - lrzip will test if lrunzip was used on command line. -* Updated Makefile.in for new compile time and linking options. -* Updated LZMALibs.cpp to include new property members for - LZMAEncoders as well as changed default dictionaries to - level+16. This would make the default compression level - of 7 translate to a dictionary number of 23. -* Added output to show Nice Level when verbose mode set - Initial add of support for zlib which seems to give quite - excellent performance. -* configure.in added AC_CHECK for libz and libm. - Added AC_PROG_LN_S for Makefile symlink section. -* lrzip.1 updated man page for -g option -* main.c added option test for gzip - Added sysconf(_SC_NPROCESSORS_CONF) for CPU detection - for threading. - Updated verbose output to show whether or not - Threading will be used. - Added Timer for each file compressed. -* rzip.h added flags for GZIP compression. - Added control member for threads. Arg passed to - lzma_conpress. -* stream.c update to accomodate gzip compress and decompress - functions. Cleaned up file by rearranging functions into - groups. - Removed include of lzmalib.h since it was causing a - compile time warning with zlib.h. Prototyped functions - manually. - Cleanup output from lzo_compresses function so that - unnecessary linefeeds are eliminated. - lzma_compress function call now uses threads as argument. -* Added README.benchmarks file to explain a method of - comparing results between different methods. -* LZMALib.cpp, lzmalib.h. Adjust function lzma_compress - prototype and function to include new argument threads. - This parameter is now placed in properties. -* lzma/Makefile. Updated to reflect new API library. - Updated to include Threading option. - -DEC 2007, version 0.19. Con Kolivas. -* Added nice support, defaulting to nice 19. - -DEC 2007, version 0.19. Peter Hyman, pete@peterhyman.com - -* Major goal was to stop LZMA from hanging on some files. - Accomplished this with a threhold setting that is used by - the lzo_compresses function to better analyze chunk data. - Threshold makes it less likely that uncompressible data - will be passed to the LZMA compressor. - -main.c -* Added Threshold option 1-10 to control LZMA compression attempt. - Default value=2. This means that anything over 10% compression - as reported by lzo_compresses will return a true value to - the LZMA compression function. -* Added verbosity option and more verbosity option (-v[v]). -* Added -O option to specify output directory. -* Updated compress_file and decompress_file functions to handle. - output directories and better handle multi files and filename - extensions. Optimized some string handling routines. - Improved flexibility in determining location of output files - when using -O. Added fflush(stdout) to improve printf reliability. -* decompress_file will accept any filename and will automatically - append .lrz if not present. Won't automatically fail. -* Added logic to protect against conflicting options such as - -q and -v, -o and -O. -* Added printout to screen of options selected. Will display - only when -v or -vv used. -* Adjusted several printf statements to avoid compiler - warnings (use %ll for long long int types). - -runzip.c -* Added decompression progress indicator. - Will show percent decompressed along with bytes decompressed - and total to be decompressed. Will show if -q option NOT used. - -rzip.h -* Version incremented to 0.19. -* Added flag DEFINESs for verbosity and more verbosity. -* Updated control struct to include output directory and - threshold value. Removed verbosity member. - -rzip.c -* Minor changes to handle display when verbosity set. Changed - number format in some printf statements to properly handle - unsigned data. - -stream.c -* major overhaul of lzo_compresses function to use a threshold - value when testing a data chunk to see if it is suitable for - LZMA compression. Optimized test loop to improve performance - and reduce number of passes. Improved output reporting depending - on verbosity setting. -* Added print controls for verbosity option. -* Corrected if statements that tested for error condition of - some lzo functions that only return a true value regardless. - -lrzip.1 -* updated man page to show new options and explain -T threshold. - -README -* updated README to explain -T threshold option. - -README.lzo_compresses.test.txt -* Added this file to help explain the theory behind the rewrite - of the lzo_compresses function and how to use the -T option. - -TODO -* wish list and future enhancements. - -ChangeLog -* added file. +Check git for changelog: https://github.com/ckolivas/lrzip/commits/master diff --git a/WHATS-NEW b/WHATS-NEW index 2a7b2b6..b708d54 100644 --- a/WHATS-NEW +++ b/WHATS-NEW @@ -1,5 +1,20 @@ Changelog will be moved to git entirely from this point forward. +lrzip-0.650 + +Minor optimisations. +Exit status fixes. +Update and beautify information output. +Fix Android build. +Enable MD5 on Apple build. +Deprecate and remove liblrzip which was unused and at risk of bitrot. +Fix failures with compressing to STDOUT with inadequate memory. +Fix possible race conditions. +Fix memory leaks. +Fix -q to only hide progress. +Add -Q option for very quiet. + + lrzip-0.641 Critical bugfix for broken lz4 testing which would prevent secondary diff --git a/man/lrzip.1 b/man/lrzip.1 index 75fe4db..b2c1436 100644 --- a/man/lrzip.1 +++ b/man/lrzip.1 @@ -357,7 +357,8 @@ zip(1) .PP .SH "DIAGNOSTICS" .PP -Exit status is normally 0; if an error occurs, exit status is 1. +Exit status is normally 0; if an error occurs, exit status is 1, usage errors +is 2. .PP .SH "AUTHOR and CREDITS"