Con Kolivas
63fb1bafea
Modify the file format further to make all block header information only encode the number of bytes determined in chunk_bytes instead of 8 full bytes all the time.
2011-03-18 23:18:36 +11:00
ckolivas
e856d481bf
Discard hash_iv entirely, using passphrase hash and centralise key generation.
...
Patch by Serge Belyshev.
2011-03-18 10:22:58 +11:00
Con Kolivas
a3e80871c2
Add 8 bytes of random data to the start of every encrypted compressed block and use it along with 8 bytes from the stored salt to make a unique key and IV for encryption.
2011-03-18 00:06:11 +11:00
ckolivas
4bc23ee001
Centralise where all encryption/decryption is done, and whether to carry over the IV or not.
2011-03-17 14:10:58 +11:00
ckolivas
ae63349e3c
We do need to seek to read the MD5 correctly on very small files so support seeking to end and then seek to the md5 start.
2011-03-17 11:53:29 +11:00
Con Kolivas
f61632670e
Encrypt the md5 value as well to make the file unidentifiable.
2011-03-16 00:29:10 +11:00
ckolivas
e26d0d1381
Clear the temporary infile where possible as well when we have falen back to it.
2011-03-15 11:27:07 +11:00
ckolivas
65a681a254
Relative offset is not required in tmp inbuf, it can be safely zeroed.
2011-03-15 11:21:26 +11:00
Con Kolivas
8a83741c32
Clear tmpinbuf between chunks.
2011-03-14 22:03:13 +11:00
Con Kolivas
3a8c0b6689
Remove seeks that aren't required and don't work on tmp input buffers.
...
Clean up open_stream_in.
2011-03-14 21:51:27 +11:00
Con Kolivas
c832e80085
More infrastructure to read from temporary buffers on stdin decompression.
2011-03-14 21:19:57 +11:00
Con Kolivas
36e09f206e
Begin massive read changes to support using temporary file for STDIN.
2011-03-14 20:22:45 +11:00
Con Kolivas
684959efed
Add fields for temporary input buffer and clamp memory accordingly in preparation.
2011-03-14 14:47:26 +11:00
Con Kolivas
9e772d3140
Make ALL decompression use temporary in-ram buffer whenever possible.
2011-03-14 12:48:40 +11:00
Con Kolivas
b644240152
write_1g always uses control->fd_out so don't pass fd to it.
2011-03-14 11:15:35 +11:00
Con Kolivas
7eabb5e7de
Check seek hasn't made the pseudo file longer.
2011-03-14 11:12:12 +11:00
Con Kolivas
4a6fa7602f
Begin decompressing to stdout via temporary buffer by creating a read/write wrapper.
2011-03-14 10:07:51 +11:00
Con Kolivas
f2ddd9022c
Ignore the eof flag if the file size is known.
2011-03-13 21:31:03 +11:00
Con Kolivas
b17540d32e
Fix decompressing single chunk stdout chunked archives.
2011-03-13 18:14:50 +11:00
Con Kolivas
8756fe91e2
Enable decompression when file has been chunked stdout and display progress only when expected size is known.
2011-03-13 17:52:23 +11:00
Con Kolivas
c75a50f723
Being modifying write_1g function to be able to write to a temporary buffer instead of straight to fd_out.
...
Split out make_magic to be able to write magic wherever we want later.
2011-03-12 14:13:28 +11:00
Con Kolivas
1a7c409e10
header-mangling-part-X-move-all-headers-defines-into
2011-03-09 08:37:26 +11:00
Con Kolivas
99c3ea2ab9
header-mangling-part-3-remove-ugly-hacks-for-liblrzi
2011-03-09 08:36:07 +11:00
Con Kolivas
1511c27aad
header-mangling-part-2-move-all-function-prototypes
2011-03-09 08:34:44 +11:00
Con Kolivas
f6f0a25ef6
rebase-of-function-split-and-control-additions-to-fu
2011-03-09 08:32:14 +11:00
ckolivas
59e5bc0492
Remove all open coded strerrors when they're going to be shown by fatal().
2011-03-08 12:05:03 +11:00
Con Kolivas
13a6fb5b43
Dump the temporary file generated to emulate stdout at the end of each chunk on decompression and then truncate the file instead of writing the whole file before dumping it.
2011-03-07 17:14:07 +11:00
Con Kolivas
a7b4708bd2
Use a different failure mode for when perror is unlikely to be set.
...
Add 2 unlikely wrappers.
2011-02-21 14:51:20 +11:00
Con Kolivas
57e25da244
Update copyright yeah in updated files.
2011-02-20 23:04:44 +11:00
Con Kolivas
7b073160a3
Can't always open fd_out in runzip for integrity testing, so use fd_hist.
2011-02-20 22:44:10 +11:00
Con Kolivas
9b264959f5
Implement the ability to test the integrity of the file written to disk on decompression.
2011-02-20 22:29:49 +11:00
Con Kolivas
8a27dc5057
Changes to make md5 be used for integrity testing.
...
Add the md5 value to the end of each archive.
This can then be used for integrity testing instead of crc32.
Keep crc in new archives to maintain compatibility with version 0.5 files.
Use md5 integrity testing on decompression when available in preference, and disable calculation of crc32.
Display the choice of integrity testing in verbose output and when -i is used.
Display the md5 and crc values when max verbosity, file info, or display hash is enabled.
Store a new flag in the magic header to show that the md5 value is stored at the end of the file.
Update the magic header information document.
2011-02-20 18:01:19 +11:00
Con Kolivas
44a279579e
Add option to display hash information without enabling verbose mode.
2011-02-20 12:20:05 +11:00
Con Kolivas
744202a47f
Remove unused variable.
2011-02-19 10:39:07 +11:00
Con Kolivas
7287ab8a66
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 10:34:45 +11:00
Con Kolivas
fb2a12744a
Implement md5 checking on decompression.
...
Implement hash check flag to determine whether to show md5sum on compression/decompression or not.
2011-02-18 15:16:13 +11:00
Con Kolivas
cd8b086bf2
Minimise the number of mallocs in unzip_match.
2011-02-17 09:32:01 +11:00
Con Kolivas
f2d33c00f8
Cast the mallocs to their variable type.
...
Check that read and write actually return greater than zero.
2011-02-11 11:46:58 +11:00
Con Kolivas
3879807865
Try limiting stream_read in unzip_literal and just returning how much was read.
2011-02-10 16:57:22 +11:00
Con Kolivas
9a3bfe33d1
Revert "Make sure to read the full length asked of unzip_literal."
...
This reverts commit 499ae18cef
2011-02-10 16:46:35 +11:00
Con Kolivas
499ae18cef
Make sure to read the full length asked of unzip_literal.
2011-02-10 15:30:31 +11:00
Con Kolivas
0a32b5f72d
Convert mmaps to malloc in runzip as they may fail if not a multiple of page size!
2011-02-10 13:53:42 +11:00
Con Kolivas
bece82a593
Trivial documentation fixes courtesy of Laszlo Ersek.
2011-02-10 13:14:36 +11:00
Con Kolivas
2cabb335cb
Update copyright notices courtesy of Jari Aalto.
2010-12-16 09:45:21 +11:00
Con Kolivas
2b08c6e280
Implement massive multithreading decompression.
...
This is done by taking each stream of data on read in into separate buffers for up to as many threads as CPUs.
As each thread's data becomes available, feed it into runzip once it is requests more of the stream.
Provided there are enough chunks in the originally compressed data, this provides a massive speedup potentially proportional to the number of CPUs. The slower the backend compression, the better the speed up (i.e. zpaq is the best sped up).
Fix the output of zpaq compress and decompress from trampling on itself and racing and consuming a lot of CPU time printing to the console.
When limiting cwindow to 6 on 32 bits, ensure that control.window is also set.
When testing for the maximum size of testmalloc, the multiple used was out by one, so increase it.
Minor output tweaks.
2010-11-16 21:25:32 +11:00
Con Kolivas
a66dafe66a
Updated benchmark results.
...
More tidying up.
2010-11-05 14:52:14 +11:00
Con Kolivas
296534921a
Unlimited mode is now usable in a meaningful timeframe!
...
Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable.
Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c.
Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable.
Minor cleanups.
2010-11-05 12:16:43 +11:00
Con Kolivas
102140dc2b
Reinstate the temporary files for decompression to stdout and testing as the damaged line reinstated last commit meant it looked like those combinations worked when they actually didn't.
...
Compression from stdin still works without temporary files.
2010-11-02 10:52:21 +11:00
Con Kolivas
1e88273ffc
Minor fixes.
2010-11-02 00:08:35 +11:00
Con Kolivas
772fbf602e
Reinstitute 2GB window limit on 32 bit. It still doesn't work. However we can now decompress larger windows.
...
Do more mmap in place of malloc.
Update docs.
Remove redundant code.
2010-11-01 22:55:59 +11:00
