ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
832 commits 5 branches 43 tags 2.8 MiB
Commit graph

179 commits

Author SHA1 Message Date
Con Kolivas c1055d173e Clamp largest 32bit usable chunk to 2/3 of 1GB. 2011-04-12 14:40:51 +10:00
Con Kolivas e66d97ab19 Show full version properly when -V is invoked with new versioning system. 2011-04-03 20:13:16 +10:00
Con Kolivas b27e1df879 Revert "Modify the file format to not include rzip chunk size except for STDOUT chunked files thus decreasing the file format size further." 
This reverts commit 131d4c92c2.

Breaks STDIN/OUT so revert this change and cope with the slightly extra file size on non-chunked files.
 2011-03-23 07:59:04 +11:00
Con Kolivas 8077412ee5 Allow files with non-standard extensions to be tested or decompressed if they actually exist. 
Don't allow a suffix if an actual output filename has been specified.
Convert ints to chars when they can be.
 2011-03-23 01:51:40 +11:00
Con Kolivas 70e7866271 Include arpa/inet.h for those OSs that have endian details elsewhere. 2011-03-22 21:47:24 +11:00
Con Kolivas 4bc8faa526 Storage time is only meaningful on compression. 2011-03-22 21:37:10 +11:00
Con Kolivas bdf19e2654 nloop cleanup courtesy Serge Belyshev. 2011-03-22 21:34:04 +11:00
ckolivas f56e48c4f2 Modify maxram to mean the most we'll allocate in one chunk and add usable_ram for the most we'll allocate to one task. 
Limit all maxram on 32bits to 1GB since most don't like mallocing more than this.
Update testing to allow larger total amount based on usable ram while sticking to maxram limit per alloc.
 2011-03-22 12:10:21 +11:00
Con Kolivas 68cca4dea5 More streamlining of version number. Fixes problems with osx version displayed. 2011-03-22 09:06:13 +11:00
Con Kolivas 131d4c92c2 Modify the file format to not include rzip chunk size except for STDOUT chunked files thus decreasing the file format size further. 2011-03-22 09:04:24 +11:00
Con Kolivas 2736118f82 Check for different endian.h includes and variants of le32toh. 2011-03-22 07:57:32 +11:00
Con Kolivas 93f7c602d0 Fixes for 32 bit limits and osx compilation. 2011-03-22 00:46:12 +11:00
Con Kolivas 0b1db75a44 Cleanups. 2011-03-19 13:53:43 +11:00
Con Kolivas 53a6dfd723 Add manpage entry for encryption option. 2011-03-18 23:38:02 +11:00
ckolivas 5005c2dff5 Get rid of extra data in new header and stick to old header size, compacting extra info into existing space. 2011-03-18 17:32:47 +11:00
ckolivas b8d8ecfe33 Fix warnings. 
Patch ~ by Serge Belyshev.
 2011-03-18 10:37:09 +11:00
Con Kolivas f758773cd0 Fix one minor leak. 2011-03-16 13:29:47 +11:00
Con Kolivas 412cf013c4 Implement passphrase input. 2011-03-15 16:32:32 +11:00
Con Kolivas 202d972a6c Add encryption -e option and flag encryption with a control flag, removing encrypt field. 2011-03-15 15:04:58 +11:00
Con Kolivas 65f901a83c Get the seconds, salt and encryption loop data before compressing. 
Store seconds in only 5 bytes which is enough for 400 years, leaving more room for random data.
 2011-03-15 14:41:47 +11:00
ckolivas fca5d4232c Refuse to perform file info from STDIN. 2011-03-15 09:56:45 +11:00
Con Kolivas 00be150cfb Add flag tests for TMP_INBUF. 2011-03-14 14:56:59 +11:00
Con Kolivas 684959efed Add fields for temporary input buffer and clamp memory accordingly in preparation. 2011-03-14 14:47:26 +11:00
Con Kolivas 9e772d3140 Make ALL decompression use temporary in-ram buffer whenever possible. 2011-03-14 12:48:40 +11:00
Con Kolivas d9e9f0d71d No need to keep usable_ram as a rzip_control variable. 2011-03-13 22:28:28 +11:00
Con Kolivas 66fe056f30 Don't use open coded flags. 2011-03-13 22:22:46 +11:00
Con Kolivas 17c35cb331 Use the usable_ram variable just to determine the maximum size allocatable. 2011-03-13 22:16:46 +11:00
Con Kolivas 6ac74aa9f0 Create a flag to know when the temporary output buffer is in use, in preparation for when we use it on decompression. 2011-03-13 08:34:06 +11:00
Con Kolivas d2ae072a3c Round to page for optimum efficiency. 2011-03-12 22:58:39 +11:00
Con Kolivas 9444441d51 Modify maximum ram usable when stdout is being used in preparation for temporary in-ram file during stdout and fix summary shown and 32 bit limits. 2011-03-12 12:19:02 +11:00
Con Kolivas 34a44d16b3 Close config file. 2011-03-09 13:41:16 +11:00
Con Kolivas e22967db11 Remove unused s char. 2011-03-09 13:32:37 +11:00
Con Kolivas 2db75fe408 Get rid of trailing whitespace 2011-03-09 08:50:46 +11:00
Con Kolivas 1a7c409e10 header-mangling-part-X-move-all-headers-defines-into 2011-03-09 08:37:26 +11:00
Con Kolivas 99c3ea2ab9 header-mangling-part-3-remove-ugly-hacks-for-liblrzi 2011-03-09 08:36:07 +11:00
Con Kolivas 1511c27aad header-mangling-part-2-move-all-function-prototypes 2011-03-09 08:34:44 +11:00
Con Kolivas e5dfd2d9d8 header-mangling-part-1-move-functions-out-of-headers 2011-03-09 08:33:44 +11:00
Con Kolivas f6f0a25ef6 rebase-of-function-split-and-control-additions-to-fu 2011-03-09 08:32:14 +11:00
ckolivas 59e5bc0492 Remove all open coded strerrors when they're going to be shown by fatal(). 2011-03-08 12:05:03 +11:00
ckolivas 2a8cad1a28 Check for any unusual entries in get_fileinfo and abort. 2011-03-08 11:43:56 +11:00
ckolivas 9499a7b453 STDOUT does not affect the compression window in master branch. 2011-03-08 08:24:05 +11:00
Con Kolivas ef15ea28a8 Display outputting message only if not testing. 2011-03-07 21:18:15 +11:00
Con Kolivas 44ed5727b3 Clean up and fix times displayed. 
Merge fixes.
 2011-03-07 21:05:20 +11:00
ckolivas c6199980f0 Disable file checking when not possible or meaningful. 2011-03-07 20:53:59 +11:00
Con Kolivas 10b1201f82 Fix more of the dodgy merge. 2011-03-07 17:28:50 +11:00
Con Kolivas 9fc551c159 Merge branch 'temp' 
Conflicts:
	main.c
 2011-03-07 17:25:10 +11:00
Con Kolivas 13a6fb5b43 Dump the temporary file generated to emulate stdout at the end of each chunk on decompression and then truncate the file instead of writing the whole file before dumping it. 2011-03-07 17:14:07 +11:00
Con Kolivas 8a4df9774f Forgot two instances of temporary outfiles that need to be unlinked. 2011-03-07 13:44:29 +11:00
Con Kolivas 68469c2b6f Don't dump output to stdout when testing. 2011-03-07 13:41:00 +11:00
Con Kolivas 06fd0a3528 Unlink files in safe places. 2011-03-07 13:40:43 +11:00
Con Kolivas f9046e6756 Check free space after reading magic, and not when decompressing to stdout. 2011-03-07 13:36:03 +11:00
Con Kolivas 8fa01248e7 Unlink temporary files immediately to avoid files lying around in cases of unexpected/uncaught failure. 2011-03-07 13:33:14 +11:00
Con Kolivas 38eca38743 Unify maxram allocation and limit threads when there isn't enough ram. 2011-03-07 13:23:14 +11:00
Con Kolivas e42916e51f Revert "Make sure to realloc filenames when making physical temp files to not have memory leaks." 
This reverts commit 9c3d492524.

Oops, dirty commit.
 2011-03-07 13:08:40 +11:00
Con Kolivas 8a4814081b Must read magic before testing free space. 
Resolve conflict.
 2011-03-07 13:06:30 +11:00
ckolivas d1dc3dd6b1 Don't bother zeroing variables that are already zeroed by memset. 2011-03-07 13:04:01 +11:00
Con Kolivas 9c3d492524 Make sure to realloc filenames when making physical temp files to not have memory leaks. 
Fix conflict in merge.
 2011-03-07 13:02:17 +11:00
ckolivas 81500282cc Move verbose summary till after we know if we have STDIN and STDOUT to more accurately reflect the window chosen. 2011-03-07 12:57:53 +11:00
Con Kolivas 96efe033fd Ignore what stdout is going to when testing from stdin. 2011-03-07 12:56:10 +11:00
Con Kolivas 624de254ce Revert "Fallocate space in advance on decompression." 
This reverts commit cf03cb0c04.

Fallocating for some reason fails to write the end of the file! Revert change pending further investigation.
 2011-02-28 22:13:36 +11:00
Con Kolivas 9e548448d6 Checked for the wrong standard. 2011-02-28 21:18:53 +11:00
Con Kolivas cf03cb0c04 Fallocate space in advance on decompression. 
This often can speed up decompression of large files and decreases fragmentation.
 2011-02-26 23:26:00 +11:00
Con Kolivas fd5a77ac21 Check on the fd_out on decompression, NOT on fd_in. 2011-02-26 23:19:50 +11:00
Con Kolivas 8bdd5688c8 Check for free space before compression/decompression and abort if -f option is not enabled. 2011-02-26 23:10:28 +11:00
ckolivas 9b60d6e703 Update manpage to new, cleaner help format. 2011-02-25 10:55:27 +11:00
ckolivas 3a818196cc Perform sanity testing on file information output which detects corrupted archives. 2011-02-25 09:31:35 +11:00
Con Kolivas 573aa534ac Display the offset of each block in verbose information as well. 2011-02-25 09:21:30 +11:00
Con Kolivas d90f670fbd The output from information is confusing on 32 bit when it cuts the available ram for internal use. 
Display the actual amount of ram detected, and then decrease the value for calculating what the calculated window will be.
Move the variables to where they're only used locally.
 2011-02-23 23:58:50 +11:00
Con Kolivas 48e7e31dad Make sure to not delete files that already exist and we've refused to overwrite! 2011-02-23 15:01:43 +11:00
Con Kolivas 9fef6ab803 Add verbose and max verbose modes for -i information. 
This will give a breakdown of each rzip chunk and compressed block by jumping to each head offset and reading the header information.
 2011-02-23 12:59:59 +11:00
Con Kolivas 8edc4bd052 Trivial 2011-02-23 01:29:17 +11:00
Con Kolivas 32ff6edf9a Only set the main process nice value to less when a back-end compression is enabled. 2011-02-23 01:26:51 +11:00
Con Kolivas e79127ace8 Trivial changes. 2011-02-23 01:22:54 +11:00
Con Kolivas 94673d3fe3 Change the LZO testing option to be a bool on/off instead of taking a confusing parameter. 
Make the lzo testing message simpler and only appear when max verbose mode is enabled.
 2011-02-23 01:15:18 +11:00
Con Kolivas 9c8b1ee795 Deprecate the -M option as it is no longer of any significant utility compared to the -U option. 2011-02-22 20:38:39 +11:00
Con Kolivas 011344753a With lzma and zpaq, the compression overhead per thread is significant. 
As we can work out what that compression overhead is, we can factor that into testing how much ram we can allocate.
There is no advantage to running multiple threads when there is no compression back end so drop to 1 only.
Limit ram for compression back end to 1/3 ram regardless for when OSs lie due to heavy overcommit.
 2011-02-22 15:19:31 +11:00
Con Kolivas 88e3df6af1 Print perror before unlinking files. 
Join common parts of fatal errors.
Update copyright notices.
Small improvement to visual output.
 2011-02-21 16:11:59 +11:00
Con Kolivas a7b4708bd2 Use a different failure mode for when perror is unlikely to be set. 
Add 2 unlikely wrappers.
 2011-02-21 14:51:20 +11:00
Con Kolivas 5c80a0957d Add usage() to failure to read stdin/out. 2011-02-21 13:29:45 +11:00
Con Kolivas b06287cce5 Make it not read stdin from a terminal or write stdout to a terminal unless forced with -f. 2011-02-21 13:26:51 +11:00
Con Kolivas 29dfb16c7a Clean up the messy help output. 2011-02-21 12:49:44 +11:00
Con Kolivas a85b17a463 Minor help change. 2011-02-21 12:06:49 +11:00
Con Kolivas 74df2b5973 Minor updates to man pages, lrzip.conf example file. 
Update main help screen to include environment settings.
Update to respect $TMP environment variable for TMP files.
Updated control structure to include tmpdir pointer.
Update lrzip.conf parser to respect -U -M options.
Update lrzip.conf example to include new parameters.
Reorder main Switch loop in main.c for readability.
Have MAXRAM and control.window be exclusive. MAXRAM wins.
Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
Have UNLIMITED and MAXRAM be exclusive. UNLIMITED wins.
Corrects heuristic computation in rzip.c which would override
MAXRAM or UNLIMITED if control.window set
Show heuristically computed control.window when computed.
Remove display compression level from control.window verbose output.
Update print_verbose format for Testing for incompressible data in stream.c
to omit extra \n.

Changes by Peter Hyman <pete@peterhyman.com>
 2011-02-21 12:03:08 +11:00
Con Kolivas d4fb4d025b Update manpage. 2011-02-20 23:22:45 +11:00
Con Kolivas 57e25da244 Update copyright yeah in updated files. 2011-02-20 23:04:44 +11:00
Con Kolivas c9863e0e60 Change default behaviour to deleting broken or damaged files that occur by interrupting lrzip or that fail integrity testing. 
Implement the -k option to optionally keep broken or damaged files.
 2011-02-20 23:02:15 +11:00
Con Kolivas 9b264959f5 Implement the ability to test the integrity of the file written to disk on decompression. 2011-02-20 22:29:49 +11:00
Con Kolivas 8a27dc5057 Changes to make md5 be used for integrity testing. 
Add the md5 value to the end of each archive.
This can then be used for integrity testing instead of crc32.
Keep crc in new archives to maintain compatibility with version 0.5 files.
Use md5 integrity testing on decompression when available in preference, and disable calculation of crc32.
Display the choice of integrity testing in verbose output and when -i is used.
Display the md5 and crc values when max verbosity, file info, or display hash is enabled.
Store a new flag in the magic header to show that the md5 value is stored at the end of the file.
Update the magic header information document.
 2011-02-20 18:01:19 +11:00
Con Kolivas 44a279579e Add option to display hash information without enabling verbose mode. 2011-02-20 12:20:05 +11:00
Con Kolivas c2417877bc Being implementing md5 hash checking by importing the md5 code from coreutils. 
Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
 2011-02-18 09:09:40 +11:00
Con Kolivas 9c2b86fec6 We are flushing the wrong file on decompression. Make sure we flush the file out. 2011-02-08 08:27:22 +11:00
Con Kolivas 2cabb335cb Update copyright notices courtesy of Jari Aalto. 2010-12-16 09:45:21 +11:00
Con Kolivas db95eba695 Take into account VM split on 32 bits again. 2010-12-03 19:30:27 +11:00
Con Kolivas 6f2b94be3b Fix the case where a compressed file has more than one stream 0 entry per block. 
Limit lzma windows to 300MB in the right place on 32 bit only.
Make the main process less nice than the backend threads since it tends to be the rate limiting step.
 2010-11-24 20:12:19 +11:00
Con Kolivas 0a4f6807e5 Increase version number to 0.540. 
Fix compression type reporting on -i.
Remove some unnecessary output when -i is used.
Update docs and benchmarks.
 2010-11-16 23:14:35 +11:00
Con Kolivas 3e77c702bb Fix typo and manpage format. 
Fix warnings on newer gcc with attribute unused.
 2010-11-15 11:35:37 +11:00
Con Kolivas 1637598c3f Bump version number up to 0.530. 
Update all documentation.
Minor fixes by Jari Aalto for build and docs.
 2010-11-13 21:37:17 +11:00
Con Kolivas 6e4fdc97f8 Implement -p option to specify number of processors to determine thread count. 
Remove -P option as failing to set permissions only issues a warning now, removing any requirement for -P.
Change default compression level back to 7 as 9 was not giving significantly better compression but was slowing things down.
 2010-11-13 17:36:21 +11:00
Con Kolivas 5505097b2f Implement multithreaded back end compression by splitting up the compression stream into multiple threads, dependant on the number of CPUs detected. 
This facilitates a massive speed up on SMP machines proportional to the number of CPUs during the back end compression phase, but does so at some cost to the final size.
Limit the number of threads to ensure that each thread at least works on a window of STREAM_BUFSIZE.
Disable the lzma threading library as it does not contribute any more to the scalability of this new approach, yet compromises compression.
Increase the size of the windows passed to all the compression back end types now as they need more to split them up into multiple threads, and the number of blocks increases the compressed size slightly.
 2010-11-10 20:56:17 +11:00
Con Kolivas b469e7b56c Distros don't like 3 point versioning scheme so change back to 2 point and change version to 0.520. 2010-11-08 10:43:05 +11:00