ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
953 commits 5 branches 43 tags 2.8 MiB
Commit graph

244 commits

Author SHA1 Message Date
Con Kolivas a334444977 Remove unused read_i64 function 2015-03-03 09:35:35 +11:00
Con Kolivas 4aa69a104a Pass strict sizes to decompress length, rounding up only the amount we're allocating to not confuse decompression libraries 2014-05-30 22:05:51 +10:00
Con Kolivas 22496bdd5a Convert the thread locking to use cksems 2014-05-30 21:27:21 +10:00
Con Kolivas 6c8525893b Round up compression and decompression buffers to page size since malloc will allocate them that large anyway. 2013-09-14 12:58:31 +10:00
Con Kolivas 47de07affb Increase the compressed buffer size given to libzpaq in case of incompressible data since it does not check if it's trying to write beyond the end of the buffer. 2013-09-14 11:45:23 +10:00
Con Kolivas d8df50c9b7 Update various copyright dates. 2013-09-02 12:21:54 +10:00
Con Kolivas 5c811a9311 We have to run through the clear buffer function even for empty buffers or corrupt archives with empty match streams. 2013-09-01 15:02:52 +10:00
Con Kolivas eff1efdbc5 Rewrite the magic if we receive lzma properties and have not yet written them yet during stdout operation. 2013-09-01 12:25:06 +10:00
Con Kolivas b076dbb59d Set the control lzma properties only once. 2013-09-01 12:17:51 +10:00
Con Kolivas 184a8f0025 Rename lrzip.h to lrzip_core.h to cope with case insensitive filesystems not working with Lrzip.h vs lrzip.h 2012-07-07 21:39:40 +10:00
Con Kolivas 081265f1c9 Use no back end compression for blocks smaller than 64 bytes to avoid issues to do with minimum buffer size and minimum match sizes of the rzip preprocessing. 2012-07-07 20:43:44 +10:00
Con Kolivas 9e33cfc24a Implement progress counter on zpaq compress/decompress per thread within the Reader function. 2012-03-17 15:30:43 +11:00
Con Kolivas 77f6965f25 Re-instantiate the lzo compresses test for zpaq compression. 2012-03-16 23:20:54 +11:00
Con Kolivas 8b680e72ac Remove redundant code. 2012-03-16 23:14:49 +11:00
Con Kolivas fbcf0fcfd4 Delete now-unused zpipe code. 2012-03-16 23:04:20 +11:00
Con Kolivas a55250cf16 Use libzpaq backend for decompression. 2012-03-16 14:14:22 +11:00
Con Kolivas f8d05b9a66 Move zpaq compression to new libzpaq library back end. 2012-03-16 14:04:16 +11:00
Con Kolivas 5edf8471d1 Perform all checksumming in a separate thread to speed up the hash search in the rzip phase. 2012-03-11 11:51:36 +11:00
Con Kolivas 086265c83f Update copyright notices and style to 1TBS for return codes. 2012-03-07 22:04:24 +11:00
Con Kolivas 18105bbf7c Merge branch 'master' into liblrzip 
Conflicts:
	lrzip.c
 2011-09-16 22:55:34 +10:00
Con Kolivas 271a407765 Make lzma back end message maxverbose to minimise spamming. 2011-09-15 13:29:09 +10:00
Con Kolivas a0fe1a8c8e Merge branch 'master' into liblrzip 
Conflicts:
	runzip.c
	stream.h
 2011-09-15 08:31:00 +10:00
Con Kolivas e1f2138060 Make sure to read to the end of all streams when closing down an rzip stream. 
This would have made lrzip unable to decompress some rare archives if they ended up stream 0.
 2011-09-15 08:28:30 +10:00
Con Kolivas a39b57a1e7 Add lots more debugging in maxverbose mode about where data is being written. 2011-09-15 08:26:07 +10:00
Con Kolivas 44fa63bf11 Merge branch 'master' into liblrzip 
Conflicts:
	stream.c
 2011-08-17 21:19:13 +10:00
discomfitor 3316dd5505 remove trailing whitespace 2011-08-17 20:59:41 +10:00
discomfitor 63141a1035 add some gross bucketing to implement a queue for freeing stream_info structs after an operation has completed (in library mode ONLY; no changes will occur during regular execution) to fix some leaks 2011-08-17 20:59:41 +10:00
discomfitor 949c6d7a97 another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above 2011-08-17 20:59:41 +10:00
discomfitor 6dd20d735b giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error 2011-08-17 20:59:40 +10:00
discomfitor cbff8faef7 move liblrzip.h contents to lrzip_private.h global variables----- 2011-08-17 20:59:40 +10:00
discomfitor 1c64dcfb31 more initialization stuff out of main(), remove weird coincidentally successful one_g variable 2011-08-17 20:59:40 +10:00
Con Kolivas 7104809cd1 FreeBSD needs fake memopen too. 2011-08-17 20:54:23 +10:00
Con Kolivas 1ea75154d6 Silence compiler warnings. 2011-05-08 00:05:46 +10:00
Con Kolivas 55dc9b0095 Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. 
Put special case management of zero sized files.
Modify the percentage calculation to be able to cope with small numbers and zeroes.
 2011-05-07 18:37:25 +10:00
Con Kolivas 7ed977b1c1 Detach threads from the compression side since we don't explicitly join them as they may count towards max thread count otherwise. 2011-04-23 08:15:44 +10:00
ckolivas 6ddd2e8e7e Correct fix to ensure 64 bit lseek is used on apple in stream.c 2011-04-18 23:13:11 +10:00
Con Kolivas 373cfa25e4 limit must still be clamped to chunk_limit at most. 2011-03-23 08:44:52 +11:00
Con Kolivas 915edb78a2 Remove unused variables and functions. 2011-03-23 08:16:07 +11:00
Con Kolivas b27e1df879 Revert "Modify the file format to not include rzip chunk size except for STDOUT chunked files thus decreasing the file format size further." 
This reverts commit 131d4c92c2.

Breaks STDIN/OUT so revert this change and cope with the slightly extra file size on non-chunked files.
 2011-03-23 07:59:04 +11:00
Con Kolivas 8fcedbc43e Fix big endian decompression from lzma. 2011-03-23 07:42:45 +11:00
Con Kolivas f7a1c14e28 Put warning about possibly wrong password at appropriate place in stream.c 
Use char to be consistent in open_stream_in.
Fail to failure() instead of fatal() should open_stream_in fail.
 2011-03-23 01:13:15 +11:00
Con Kolivas 70e7866271 Include arpa/inet.h for those OSs that have endian details elsewhere. 2011-03-22 21:47:24 +11:00
ckolivas eebba614bc Don't adjust endianness in read_val. 2011-03-22 17:00:37 +11:00
ckolivas 36cc464d2b Yet more 32 bit clamping. 2011-03-22 14:44:56 +11:00
ckolivas f56e48c4f2 Modify maxram to mean the most we'll allocate in one chunk and add usable_ram for the most we'll allocate to one task. 
Limit all maxram on 32bits to 1GB since most don't like mallocing more than this.
Update testing to allow larger total amount based on usable ram while sticking to maxram limit per alloc.
 2011-03-22 12:10:21 +11:00
Con Kolivas 131d4c92c2 Modify the file format to not include rzip chunk size except for STDOUT chunked files thus decreasing the file format size further. 2011-03-22 09:04:24 +11:00
Con Kolivas 2736118f82 Check for different endian.h includes and variants of le32toh. 2011-03-22 07:57:32 +11:00
Con Kolivas a30efcaa55 Portable generic byteswap for BE. 2011-03-22 00:13:29 +11:00
Con Kolivas 25a018fd9d Update Authors to include Serge Belyshev. 2011-03-21 21:02:05 +11:00
Con Kolivas 95f8b9fd90 Forgot return value. 2011-03-21 20:52:11 +11:00
Con Kolivas 135067a0b4 Yet more #6 BE fixes. 2011-03-21 20:33:43 +11:00
Con Kolivas 279b745cd7 Yet #5 more BE fixes. 2011-03-21 20:31:33 +11:00
Con Kolivas 637b0d5f7c yet yet yet yet more BE fixes. 2011-03-21 20:29:24 +11:00
Con Kolivas 04d3d6a1f5 Read BE fixes. 2011-03-21 20:23:36 +11:00
Con Kolivas 06a76bc4aa Revert "More big endian fixes." 
This reverts commit edbcc797e0.

Broke regression text. Needs more attention.
 2011-03-21 19:01:24 +11:00
Con Kolivas edbcc797e0 More big endian fixes. 2011-03-21 12:41:36 +11:00
Con Kolivas 280254b8cb Forgot to remove the initial pos offset. 2011-03-21 12:34:57 +11:00
Con Kolivas fd9bbfe64a One last check on limit. 2011-03-21 07:33:35 +11:00
Con Kolivas f87528d14d Forgot to include salt length for next head. 2011-03-20 17:01:33 +11:00
Con Kolivas 30b70dc884 Fixes for non little endian machines. 2011-03-20 16:16:38 +11:00
Con Kolivas 0dad7dcc5a Modify the encrypted version to not write down the rzip chunk size, then keep track of where each block header is written, go back and write 8 bytes of salt before it and encrypt the headers in place. 2011-03-20 15:45:44 +11:00
Con Kolivas e7d33e82d0 Make read_1g able to read from the tmp outbuf if we decide to read from fd_out. 2011-03-19 14:46:06 +11:00
Con Kolivas ff5a5ed054 Use separate lrz_encrypt and lrz_decrypt wrappers to lrz_crypt. 2011-03-19 14:04:22 +11:00
Con Kolivas 63fb1bafea Modify the file format further to make all block header information only encode the number of bytes determined in chunk_bytes instead of 8 full bytes all the time. 2011-03-18 23:18:36 +11:00
ckolivas b8d8ecfe33 Fix warnings. 
Patch ~ by Serge Belyshev.
 2011-03-18 10:37:09 +11:00
ckolivas e856d481bf Discard hash_iv entirely, using passphrase hash and centralise key generation. 
Patch by Serge Belyshev.
 2011-03-18 10:22:58 +11:00
Con Kolivas a3e80871c2 Add 8 bytes of random data to the start of every encrypted compressed block and use it along with 8 bytes from the stored salt to make a unique key and IV for encryption. 2011-03-18 00:06:11 +11:00
ckolivas e78069c0d7 Consolidate magic writing to be able to seek to read and encrypt block header data later. 2011-03-17 16:59:27 +11:00
ckolivas 4bc23ee001 Centralise where all encryption/decryption is done, and whether to carry over the IV or not. 2011-03-17 14:10:58 +11:00
ckolivas 660243b8dc Pad out first block if less than CBC_LEN and encrypting or else encryption will fail. 2011-03-17 11:19:34 +11:00
Con Kolivas 76280ffec5 Mlock ivec for it will hold the injection vector. 2011-03-17 00:25:15 +11:00
Con Kolivas a7468ce6e4 Fail if block to encrypt ends up being less than one full block for now, and tidy up code. 2011-03-16 23:13:09 +11:00
Con Kolivas c41cfe9e83 Some decompression code needs more room so simply allocate u_len for s_buf. 2011-03-16 21:48:21 +11:00
Con Kolivas 5fd7c82e1f Implement ciphertext stealing to not have to pad compressed blocks courtesy of Serge Belyshev who saved me hours of debugging thanks! 2011-03-16 21:24:05 +11:00
Con Kolivas 928a0a098a Consolidate padded length throughout. 2011-03-16 10:29:16 +11:00
Con Kolivas bedea4dbec Consolidate cbc padding into a macro. 2011-03-16 10:17:48 +11:00
Con Kolivas ddcc45ebf0 Revert "aes_crypt_cbc always returns zero so ignore its return value." 
This reverts commit 4314970b0d.

Oops, it can return invalid length.
 2011-03-16 10:00:20 +11:00
Con Kolivas 2d2a9bc1e9 We're freeing the wrong buffer by mistake should decompression fail. 2011-03-16 09:50:30 +11:00
Con Kolivas 4314970b0d aes_crypt_cbc always returns zero so ignore its return value. 2011-03-16 08:39:05 +11:00
Con Kolivas 519123966a total_read field is no longer used. 2011-03-16 08:32:16 +11:00
Con Kolivas 5da0633893 aes_crypt_cbc will allow you to work on the same buffer, so don't bother allocating a separate buffer. 
Allocate slightly more on the buffer that might be used for encryption rather than reallocing.
 2011-03-16 00:46:39 +11:00
Con Kolivas c5938c6a8b Implement the actual aes cbc encryption and decryption. 2011-03-15 23:52:39 +11:00
ckolivas 65a681a254 Relative offset is not required in tmp inbuf, it can be safely zeroed. 2011-03-15 11:21:26 +11:00
Con Kolivas 3a8c0b6689 Remove seeks that aren't required and don't work on tmp input buffers. 
Clean up open_stream_in.
 2011-03-14 21:51:27 +11:00
Con Kolivas c832e80085 More infrastructure to read from temporary buffers on stdin decompression. 2011-03-14 21:19:57 +11:00
Con Kolivas 36e09f206e Begin massive read changes to support using temporary file for STDIN. 2011-03-14 20:22:45 +11:00
Con Kolivas 684959efed Add fields for temporary input buffer and clamp memory accordingly in preparation. 2011-03-14 14:47:26 +11:00
Con Kolivas 5f7a03932b Calculate the total expected size progressively and show it when it's not known. 2011-03-14 13:32:36 +11:00
Con Kolivas 9e772d3140 Make ALL decompression use temporary in-ram buffer whenever possible. 2011-03-14 12:48:40 +11:00
Con Kolivas 27d7c2a031 Tidy. 2011-03-14 12:23:12 +11:00
Con Kolivas 0fe3213a47 Write to physical files if we cannot fit the decompression in ram. 2011-03-14 12:15:54 +11:00
Con Kolivas 37009e2ac5 Make sure to read on an fd if that's what we're supposed to be doing. 2011-03-14 11:25:04 +11:00
Con Kolivas b644240152 write_1g always uses control->fd_out so don't pass fd to it. 2011-03-14 11:15:35 +11:00
Con Kolivas 4a6fa7602f Begin decompressing to stdout via temporary buffer by creating a read/write wrapper. 2011-03-14 10:07:51 +11:00
Con Kolivas f2ddd9022c Ignore the eof flag if the file size is known. 2011-03-13 21:31:03 +11:00
Con Kolivas 8756fe91e2 Enable decompression when file has been chunked stdout and display progress only when expected size is known. 2011-03-13 17:52:23 +11:00
Con Kolivas 6d0ac95170 Remove extra locking that does nothing. 2011-03-13 12:12:37 +11:00
Con Kolivas 6ac74aa9f0 Create a flag to know when the temporary output buffer is in use, in preparation for when we use it on decompression. 2011-03-13 08:34:06 +11:00
Con Kolivas 2f87f62696 Make the tmp out buf slightly larger to account for incompressible data, and check for buffer overflows. 2011-03-13 08:16:46 +11:00
Con Kolivas 11ea12d3ce Forgot to test for TEST_ONLY. 2011-03-12 23:07:52 +11:00