From b074fa70b0528b8084701a32af60d27a508f2e5c Mon Sep 17 00:00:00 2001 From: Con Kolivas Date: Mon, 21 Mar 2011 23:00:06 +1100 Subject: [PATCH] Documentation. --- ChangeLog | 41 ++++++++++++++++++++++++++++++++++++++++- WHATS-NEW | 28 +++++++++++++++++++++++++++- 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index a1de274..971e467 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,6 +1,45 @@ lrzip ChangeLog MARCH 2011, version 0.600 Con Kolivas -* Massive rewrite +* Massive rewrite with new file format to accomodate new features. +* Allocate temporary buffers of safely sized ram that can act as temporary +storage for performing de/compression to/from STDIN/STDOUT without requiring +temporary physical files. Files compressed on machines with much larger ram +being decompressed on smaller ram machines may still require temporary files, +but as much as possible is done using in-ram decompression, and minimally +sized temporary files. Information displayed is more verbose and accurate in +STDIN/STDOUT mode. +* The temporary buffers created for decompressing to STDOUT are also used +for decompressing regular files now avoiding multiple write/reads when +re-constructing the file on decompression. This can dramatically speed up +the rzip phase of decompression on complicated files with many small matches. +* Compress block headers as well now since we know how many bytes can be used +to describe the length of the block, decreasing overall file size. +* Store the rzip chunk size per chunk to make it possible to check total file +size by summating each rzip chunk size when it's not known till the end (as +happens when compressing from STDIN). +* Implement password protected encryption. Import the polarssl code for +sha512 and aes128 routines. Read password without echoing to screen by +disabling echo via terminfo. Take the password then multiply hash it +according to the date it was generated. Inrease the number of hashes according +to Moore's law so it always takes approximately 1 second per password on the +most modern hardware when first encrypted. Hash the password against 8 bytes +of salt which is a combination of the 2 byte encoded loop counter (for how +many times to hash the password) and 6 random bytes. Take random from +/dev/urandom if it's available and fall back to random() if not. Encrypt each +block of compressed data with 8 extra bytes of random salt. Once the headers +are written, go back and encrypt the headers as well. Then encrypt the md5 +hash value as well. Anything beyond the initial lrzip magic header should +apppear as random data and no two successive encryptions of the same data with +the same password should generate the same data. +* New build system should be more robust and portable. +* Abstract out functions better into separate files and headers, and remove +all use of global variables. This will make the generation of an lrzip +library possible in the future. +* Prevent testmalloc from coming up with a negative number when determining +how big a block of memory to allocate by decreasing the number of threads to +be used and then aborting to a minimum value should it still be too much +apparent ram. +* Numerous other fixes, documentation and cleanups. MARCH 2011, version 0.571 Con Kolivas * Only retry mmaping if it's a memory error, otherwise it may give spurious diff --git a/WHATS-NEW b/WHATS-NEW index b2fa40e..bbe3c28 100644 --- a/WHATS-NEW +++ b/WHATS-NEW @@ -1,6 +1,32 @@ +lrzip-0.600 + +Compressing/decompressing to/from STDIN/STDOUT now works without generating +any temporary files. Very large files compressed in this way will be less +efficiently compressed than if the whole solid file is presented to lrzip, +but it is guaranteed not to generate temporary files on compression. +Decompressing files on a machine with the same amount of ram will also not +generate temporary files, but if a file was generated on a larger ram machine, +lrzip might employ temporary files, but they will not be the full size of the +final file. +Decompression should now be faster as the rzip reconstruction stage is mostly +performed in ram before being written to disk. +Final file sizes should be slightly smaller as block headers are now also +compressed. +Heavy grade encryption is now provided with the -e option. A combination of +a time scaled multiply hashed sha512 password with random salt followed by +aes128 block encryption of all data, including the data headers, provides for +extremely secure encryption. Passwords up to 500 characters in length are +supported, and the same file encrypted with the same password is virtually +guaranteed to never produce the same data twice. All data beyond the basic +lrzip opening header is completely obscured. Don't lose your password! +Lrzip will not try to malloc a negative amount of ram on smaller ram machines, +preferring to decrease the number of threads used when compressing, and then +aborting to a nominal minimum. +A new build configuration system which should be more robust and provides +neater output during compilation. + lrzip-0.571 -A new build configuration system. Avoid spurious errors on failing to mmap a file. Fee space will now be checked to ensure there is enough room for the compressed or decompressed file and lrzip will abort unless the -f option is