ckolivas/lrzip
1
0
Fork 0
mirror of https://github.com/ckolivas/lrzip.git synced 2025-12-06 07:12:00 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
lrzip/rzip.c

1138 lines
32 KiB
C
Raw Normal View History

Initial import
2010-03-29 01:07:08 +02:00
/*
Update copyright yeah in updated files.
2011-02-20 13:04:44 +01:00
Copyright (C) 2006-2011 Con Kolivas
Update copyright notices courtesy of Jari Aalto.
2010-12-15 23:45:21 +01:00
Copyright (C) 1998 Andrew Tridgell
Initial import
2010-03-29 01:07:08 +02:00
Modified to use flat hash, memory limit and variable hash culling
by Rusty Russell copyright (C) 2003.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
Update copyright notices courtesy of Jari Aalto.
2010-12-15 23:45:21 +01:00
along with this program. If not, see <http://www.gnu.org/licenses/>.
Initial import
2010-03-29 01:07:08 +02:00
*/
/* rzip compression algorithm */
header-mangling-part-2-move-all-function-prototypes
2011-03-08 22:34:44 +01:00
#ifdef HAVE_CONFIG_H
# include "config.h"
#endif
header-mangling-part-X-move-all-headers-defines-into
2011-03-08 22:37:26 +01:00
#ifdef HAVE_SYS_MMAN_H
# include <sys/mman.h>
#endif
#include <sys/statvfs.h>
#ifdef HAVE_SYS_STAT_H
# include <sys/stat.h>
#endif
#ifdef HAVE_SYS_TIME_H
# include <sys/time.h>
#endif
#ifdef HAVE_UNISTD_H
# include <unistd.h>
#endif
#include <lzo/lzoconf.h>
#include <lzo/lzo1x.h>
#ifdef HAVE_ERRNO_H
# include <errno.h>
#endif
Check for different endian.h includes and variants of le32toh.
2011-03-21 21:57:32 +01:00
#ifdef HAVE_ENDIAN_H
# include <endian.h>
#elif HAVE_SYS_ENDIAN_H
# include <sys/endian.h>
#endif
Include arpa/inet.h for those OSs that have endian details elsewhere.
2011-03-22 11:47:24 +01:00
#ifdef HAVE_ARPA_INET_H
# include <arpa/inet.h>
#endif
header-mangling-part-X-move-all-headers-defines-into
2011-03-08 22:37:26 +01:00
#include "md5.h"
header-mangling-part-2-move-all-function-prototypes
2011-03-08 22:34:44 +01:00
#include "stream.h"
#include "util.h"
Fix warnings. Patch ~ by Serge Belyshev.
2011-03-18 00:37:09 +01:00
#include "lrzip.h"
header-mangling-part-X-move-all-headers-defines-into
2011-03-08 22:37:26 +01:00
/* needed for CRC routines */
#include "lzma/C/7zCrc.h"
#ifndef MAP_ANONYMOUS
# define MAP_ANONYMOUS MAP_ANON
#endif
Initial import
2010-03-29 01:07:08 +02:00
#define CHUNK_MULTIPLE (100 * 1024 * 1024)
#define CKSUM_CHUNK 1024*1024
#define GREAT_MATCH 1024
#define MINIMUM_MATCH 31
/* Hash table works as follows. We start by throwing tags at every
* offset into the table. As it fills, we start eliminating tags
* which don't have lower bits set to one (ie. first we eliminate all
* even tags, then all tags divisible by four, etc.). This ensures
* that on average, all parts of the file are covered by the hash, if
* sparsely. */
typedef i64 tag;
/* All zero means empty. We might miss the first chunk this way. */
struct hash_entry {
i64 offset;
tag t;
};
/* Levels control hashtable size and bzip2 level. */
static struct level {
unsigned long mb_used;
unsigned initial_freq;
unsigned max_chain_len;
} levels[10] = {
{ 1, 4, 1 },
{ 2, 4, 2 },
{ 4, 4, 2 },
{ 8, 4, 2 },
{ 16, 4, 3 },
{ 32, 4, 4 },
{ 32, 2, 6 },
{ 64, 1, 16 }, /* More MB makes sense, but need bigger test files */
{ 64, 1, 32 },
{ 64, 1, 128 },
};
struct rzip_state {
void *ss;
struct level *level;
tag hash_index[256];
struct hash_entry *hash_table;
i64 hash_bits;
i64 hash_count;
i64 hash_limit;
tag minimum_tag_mask;
i64 tag_clean_ptr;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 last_match;
Initial import
2010-03-29 01:07:08 +02:00
i64 chunk_size;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
i64 mmap_size;
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
char chunk_bytes;
Initial import
2010-03-29 01:07:08 +02:00
uint32_t cksum;
int fd_in, fd_out;
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
int stdin_eof;
Initial import
2010-03-29 01:07:08 +02:00
struct {
i64 inserts;
i64 literals;
i64 literal_bytes;
i64 matches;
i64 match_bytes;
i64 tag_hits;
i64 tag_misses;
} stats;
};
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
struct sliding_buffer {
uchar *buf_low; /* The low window buffer */
uchar *buf_high;/* "" high "" */
i64 orig_offset;/* Where the original buffer started */
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
i64 offset_low; /* What the current offset the low buffer has */
i64 offset_high;/* "" high buffer "" */
i64 offset_search;/* Where the search is up to */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 orig_size; /* How big the full buffer would be */
i64 size_low; /* How big the low buffer is */
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
i64 size_high; /* "" high "" */
i64 high_length;/* How big the high buffer should be */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
int fd; /* The fd of the mmap */
} sb; /* Sliding buffer */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool remap_low_sb(rzip_control *control)
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
{
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
i64 new_offset;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
new_offset = sb.offset_search;
Bump version to 0.541. Limit LZMA window to 300MB on 32 bit as per reports of failure when larger. Minor documentation and display clean ups.
2010-11-18 13:33:43 +01:00
round_to_page(&new_offset);
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
print_maxverbose("Sliding main buffer to offset %lld\n", new_offset);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(munmap(sb.buf_low, sb.size_low)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to munmap in remap_low_sb\n"), false);
Remove the sb.low_top idea by trimming the size of sb.size_low when it gets to the end of the file instead.
2011-02-18 11:14:16 +01:00
if (new_offset + sb.size_low > sb.orig_size)
sb.size_low = sb.orig_size - new_offset;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.offset_low = new_offset;
sb.buf_low = (uchar *)mmap(sb.buf_low, sb.size_low, PROT_READ, MAP_SHARED, sb.fd, sb.orig_offset + sb.offset_low);
if (unlikely(sb.buf_low == MAP_FAILED))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to re mmap in remap_low_sb\n"), false);
return true;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static inline bool remap_high_sb(rzip_control *control, i64 p)
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
{
Updated benchmark results. More tidying up.
2010-11-05 04:52:14 +01:00
if (unlikely(munmap(sb.buf_high, sb.size_high)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to munmap in remap_high_sb\n"), false);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.size_high = sb.high_length; /* In case we shrunk it when we hit the end of the file */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
sb.offset_high = p;
Updated benchmark results. More tidying up.
2010-11-05 04:52:14 +01:00
/* Make sure offset is rounded to page size of total offset */
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
sb.offset_high -= (sb.offset_high + sb.orig_offset) % control->page_size;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(sb.offset_high + sb.size_high > sb.orig_size))
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
sb.size_high = sb.orig_size - sb.offset_high;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.buf_high = (uchar *)mmap(sb.buf_high, sb.size_high, PROT_READ, MAP_SHARED, sb.fd, sb.orig_offset + sb.offset_high);
if (unlikely(sb.buf_high == MAP_FAILED))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to re mmap in remap_high_sb\n"), false);
return true;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
}
/* We use a "sliding mmap" to effectively read more than we can fit into the
* compression window. This is done by using a maximally sized lower mmap at
Updated benchmark results. More tidying up.
2010-11-05 04:52:14 +01:00
* the beginning of the block which slides up once the hash search moves beyond
* it, and a 64k mmap block that slides up and down as is required for any
* offsets outside the range of the lower one. This is much slower than mmap
* but makes it possible to have unlimited sized compression windows. */
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static uchar *get_sb(rzip_control *control, i64 p)
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
{
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
i64 low_end = sb.offset_low + sb.size_low;
Remove the sb.low_top idea by trimming the size of sb.size_low when it gets to the end of the file instead.
2011-02-18 11:14:16 +01:00
if (unlikely(sb.offset_search > low_end))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!remap_low_sb(control))) return NULL;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (p >= sb.offset_low && p < low_end)
return (sb.buf_low + p - sb.offset_low);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
if (p >= sb.offset_high && p < (sb.offset_high + sb.size_high))
return (sb.buf_high + (p - sb.offset_high));
Updated benchmark results. More tidying up.
2010-11-05 04:52:14 +01:00
/* p is not within the low or high buffer range */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!remap_high_sb(control, p))) return NULL;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
return (sb.buf_high + (p - sb.offset_high));
}
Streams passed are constant to some functions, don't pass stream argument to them.
2010-12-03 09:30:56 +01:00
/* All put_u8/u32/vchars go to stream 0 */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static inline bool put_u8(rzip_control *control, void *ss, uchar b)
Initial import
2010-03-29 01:07:08 +02:00
{
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
if (unlikely(write_stream(control, ss, 0, &b, 1)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to put_u8\n"), false);
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static inline bool put_u32(rzip_control *control, void *ss, uint32_t s)
Initial import
2010-03-29 01:07:08 +02:00
{
Yet yet more BE fixes.
2011-03-21 06:12:51 +01:00
s = htole32(s);
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
if (unlikely(write_stream(control, ss, 0, (uchar *)&s, 4)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to put_u32\n"), false);
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
Add more robust checking. Premalloc ram to improve early detection of being unable to allocate that much ram. Make sure to always make chunk size a multiple of page size for mmap to work. Begin changes to make variable byte width offsets in rzip chunks. Decrease header entries to only 2 byte wide as per original rzip. Random other tidying.
2010-10-31 01:35:04 +02:00
/* Put a variable length of bytes dependant on how big the chunk is */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static inline bool put_vchars(rzip_control *control, void *ss, i64 s, int length)
Initial import
2010-03-29 01:07:08 +02:00
{
Yet yet more BE fixes.
2011-03-21 06:12:51 +01:00
s = htole64(s);
if (unlikely(write_stream(control, ss, 0, (uchar *)&s, length)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to put_vchars\n"), false);
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool put_header(rzip_control *control, void *ss, uchar head, i64 len)
Initial import
2010-03-29 01:07:08 +02:00
{
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!put_u8(control, ss, head))) return false;
if (unlikely(!put_vchars(control, ss, len, 2))) return false;
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool put_match(rzip_control *control, struct rzip_state *st, i64 p, i64 offset, i64 len)
Initial import
2010-03-29 01:07:08 +02:00
{
do {
i64 ofs;
i64 n = len;
We were attempting to truncate mmap to page size when only the offset needed to be. Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space. Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram. Don't fail if preallocation is unsuccessful. Add more detailed error reporting. Minor cleanups.
2010-10-31 14:19:39 +01:00
if (n > 0xFFFF)
n = 0xFFFF;
Initial import
2010-03-29 01:07:08 +02:00
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
ofs = (p - offset);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!put_header(control, st->ss, 1, n))) return false;
if (unlikely(!put_vchars(control, st->ss, ofs, st->chunk_bytes))) return false;
Initial import
2010-03-29 01:07:08 +02:00
st->stats.matches++;
st->stats.match_bytes += n;
len -= n;
p += n;
offset += n;
} while (len);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." This reverts commit 688aa55c7930c883d5a8ab9d963af673ad805445.
2010-12-08 11:25:00 +01:00
/* write some data to a stream mmap encoded. Return -1 on failure */
header-mangling-part-2-move-all-function-prototypes
2011-03-08 22:34:44 +01:00
static int write_sbstream(rzip_control *control, void *ss, int stream, i64 p, i64 len)
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
{
struct stream_info *sinfo = ss;
while (len) {
i64 n, i;
Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." This reverts commit 688aa55c7930c883d5a8ab9d963af673ad805445.
2010-12-08 11:25:00 +01:00
n = MIN(sinfo->bufsize - sinfo->s[stream].buflen, len);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
for (i = 0; i < n; i++) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
uchar *u;
u = get_sb(control, p + i);
if (unlikely(!u)) return -1;
memcpy(sinfo->s[stream].buf + sinfo->s[stream].buflen + i, u, 1);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
}
Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." This reverts commit 688aa55c7930c883d5a8ab9d963af673ad805445.
2010-12-08 11:25:00 +01:00
sinfo->s[stream].buflen += n;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
p += n;
len -= n;
Reworked the multithreading massively. Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating. Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible. This causes a big speed up on the compression side on SMP machines. Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows. Alter the progress output to match some of the changes in verbose modes.
2010-11-12 15:26:09 +01:00
Revert "Make threads spawn at regular intervals along chunk size thus speeding up compression." This reverts commit 688aa55c7930c883d5a8ab9d963af673ad805445.
2010-12-08 11:25:00 +01:00
if (sinfo->s[stream].buflen == sinfo->bufsize)
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!flush_buffer(control, sinfo, stream))) return -1;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
}
return 0;
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool put_literal(rzip_control *control, struct rzip_state *st, i64 last, i64 p)
Initial import
2010-03-29 01:07:08 +02:00
{
do {
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 len = p - last;
We were attempting to truncate mmap to page size when only the offset needed to be. Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space. Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram. Don't fail if preallocation is unsuccessful. Add more detailed error reporting. Minor cleanups.
2010-10-31 14:19:39 +01:00
if (len > 0xFFFF)
len = 0xFFFF;
Initial import
2010-03-29 01:07:08 +02:00
st->stats.literals++;
st->stats.literal_bytes += len;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!put_header(control, st->ss, 0, len))) return false;
Initial import
2010-03-29 01:07:08 +02:00
Revert "yet yet yet more BE fixes." This reverts commit 44b93068a3871e0338431b7830f1a770b31d3b3f. Value was not stored but ysed for a memcpy.
2011-03-21 11:27:25 +01:00
if (unlikely(len && write_sbstream(control, st->ss, 1, last, len)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to write_stream in put_literal\n"), false);
Initial import
2010-03-29 01:07:08 +02:00
last += len;
} while (p > last);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
/* Could give false positive on offset 0. Who cares. */
static int empty_hash(struct rzip_state *st, i64 h)
{
return !st->hash_table[h].offset && !st->hash_table[h].t;
}
static i64 primary_hash(struct rzip_state *st, tag t)
{
return t & ((1 << st->hash_bits) - 1);
}
static inline tag increase_mask(tag tag_mask)
{
/* Get more precise. */
return (tag_mask << 1) | 1;
}
static int minimum_bitness(struct rzip_state *st, tag t)
{
tag better_than_min = increase_mask(st->minimum_tag_mask);
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
Initial import
2010-03-29 01:07:08 +02:00
if ((t & better_than_min) != better_than_min)
return 1;
return 0;
}
/* Is a going to be cleaned before b? ie. does a have fewer low bits
* set than b? */
static int lesser_bitness(tag a, tag b)
{
tag mask;
Add more robust checking. Premalloc ram to improve early detection of being unable to allocate that much ram. Make sure to always make chunk size a multiple of page size for mmap to work. Begin changes to make variable byte width offsets in rzip chunks. Decrease header entries to only 2 byte wide as per original rzip. Random other tidying.
2010-10-31 01:35:04 +02:00
for (mask = 0; mask != (tag) - 1; mask = ((mask << 1) | 1)) {
Initial import
2010-03-29 01:07:08 +02:00
if ((a & b & mask) != mask)
break;
}
return ((a & mask) < (b & mask));
}
/* If hash bucket is taken, we spill into next bucket(s). Secondary hashing
works better in theory, but modern caches make this 20% faster. */
static void insert_hash(struct rzip_state *st, tag t, i64 offset)
{
i64 h, victim_h = 0, round = 0;
/* If we need to kill one, this will be it. */
static i64 victim_round = 0;
h = primary_hash(st, t);
while (!empty_hash(st, h)) {
/* If this due for cleaning anyway, just replace it:
rehashing might move it behind tag_clean_ptr. */
if (minimum_bitness(st, st->hash_table[h].t)) {
st->hash_count--;
break;
}
/* If we are better than current occupant, we can't
jump over it: it will be cleaned before us, and
noone would then find us in the hash table. Rehash
it, then take its place. */
if (lesser_bitness(st->hash_table[h].t, t)) {
insert_hash(st, st->hash_table[h].t,
st->hash_table[h].offset);
break;
}
/* If we have lots of identical patterns, we end up
with lots of the same hash number. Discard random. */
if (st->hash_table[h].t == t) {
We were attempting to truncate mmap to page size when only the offset needed to be. Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space. Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram. Don't fail if preallocation is unsuccessful. Add more detailed error reporting. Minor cleanups.
2010-10-31 14:19:39 +01:00
if (round == victim_round)
Initial import
2010-03-29 01:07:08 +02:00
victim_h = h;
if (++round == st->level->max_chain_len) {
h = victim_h;
st->hash_count--;
victim_round++;
if (victim_round == st->level->max_chain_len)
victim_round = 0;
break;
}
}
h++;
h &= ((1 << st->hash_bits) - 1);
}
st->hash_table[h].t = t;
st->hash_table[h].offset = offset;
}
/* Eliminate one hash entry with minimum number of lower bits set.
Returns tag requirement for any new entries. */
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static tag clean_one_from_hash(rzip_control *control, struct rzip_state *st)
Initial import
2010-03-29 01:07:08 +02:00
{
tag better_than_min;
again:
better_than_min = increase_mask(st->minimum_tag_mask);
Start cleaning up all the flag testing with neat macros.
2010-10-31 18:53:53 +01:00
if (!st->tag_clean_ptr)
Reworked the multithreading massively. Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating. Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible. This causes a big speed up on the compression side on SMP machines. Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows. Alter the progress output to match some of the changes in verbose modes.
2010-11-12 15:26:09 +01:00
print_maxverbose("Starting sweep for mask %u\n", (unsigned int)st->minimum_tag_mask);
Initial import
2010-03-29 01:07:08 +02:00
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
for (; st->tag_clean_ptr < (1U << st->hash_bits); st->tag_clean_ptr++) {
Initial import
2010-03-29 01:07:08 +02:00
if (empty_hash(st, st->tag_clean_ptr))
continue;
if ((st->hash_table[st->tag_clean_ptr].t & better_than_min)
!= better_than_min) {
st->hash_table[st->tag_clean_ptr].offset = 0;
st->hash_table[st->tag_clean_ptr].t = 0;
st->hash_count--;
return better_than_min;
}
}
/* We hit the end: everthing in hash satisfies the better mask. */
st->minimum_tag_mask = better_than_min;
st->tag_clean_ptr = 0;
goto again;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static inline tag next_tag(rzip_control *control, struct rzip_state *st, i64 p, tag t)
Initial import
2010-03-29 01:07:08 +02:00
{
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
uchar *u;
u = get_sb(control, p - 1);
if (unlikely(!u)) return -1;
t ^= st->hash_index[*u];
u = get_sb(control, p + MINIMUM_MATCH - 1);
if (unlikely(!u)) return -1;
t ^= st->hash_index[*u];
Initial import
2010-03-29 01:07:08 +02:00
return t;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static inline tag full_tag(rzip_control *control, struct rzip_state *st, i64 p)
Initial import
2010-03-29 01:07:08 +02:00
{
tag ret = 0;
int i;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
uchar *u;
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
for (i = 0; i < MINIMUM_MATCH; i++) {
u = get_sb(control, p + i);
if (unlikely(!u)) return -1;
ret ^= st->hash_index[*u];
}
Initial import
2010-03-29 01:07:08 +02:00
return ret;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static inline i64 match_len(rzip_control *control, struct rzip_state *st, i64 p0, i64 op, i64 end,
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 *rev)
Initial import
2010-03-29 01:07:08 +02:00
{
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 p = p0;
Initial import
2010-03-29 01:07:08 +02:00
i64 len = 0;
if (op >= p0)
return 0;
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
while ((*get_sb(control, p) == *get_sb(control, op)) && (p < end)) {
Initial import
2010-03-29 01:07:08 +02:00
p++;
op++;
}
len = p - p0;
p = p0;
op -= len;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
end = 0;
Initial import
2010-03-29 01:07:08 +02:00
if (end < st->last_match)
end = st->last_match;
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
while (p > end && op > 0 && *get_sb(control, op - 1) == *get_sb(control, p - 1)) {
Initial import
2010-03-29 01:07:08 +02:00
op--;
p--;
}
(*rev) = p0 - p;
len += p0 - p;
if (len < MINIMUM_MATCH)
return 0;
return len;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static i64 find_best_match(rzip_control *control, struct rzip_state *st, tag t, i64 p, i64 end,
Initial import
2010-03-29 01:07:08 +02:00
i64 *offset, i64 *reverse)
{
i64 length = 0;
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
i64 rev;
Remove unused best_h.
2011-03-09 03:10:00 +01:00
i64 h;
Initial import
2010-03-29 01:07:08 +02:00
rev = 0;
*reverse = 0;
/* Could optimise: if lesser goodness, can stop search. But
* chains are usually short anyway. */
h = primary_hash(st, t);
while (!empty_hash(st, h)) {
i64 mlen;
if (t == st->hash_table[h].t) {
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
mlen = match_len(control, st, p, st->hash_table[h].offset, end,
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
&rev);
Initial import
2010-03-29 01:07:08 +02:00
if (mlen)
st->stats.tag_hits++;
else
st->stats.tag_misses++;
if (mlen >= length) {
length = mlen;
(*offset) = st->hash_table[h].offset - rev;
(*reverse) = rev;
}
}
h++;
h &= ((1 << st->hash_bits) - 1);
}
return length;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
static void show_distrib(rzip_control *control, struct rzip_state *st)
Initial import
2010-03-29 01:07:08 +02:00
{
i64 primary = 0;
Change byte width to be dependant on file size. This will increase speed of compression and generate a smaller file, but not be backward compatible. Tweak the way memory is allocated to optimise chances of success and minimise slowdown for the machine. fsync to empty dirty data before allocating large ram to increase chance of mem allocation and decrease disk thrash of write vs read. Add lots more information to verbose mode. Lots of code tidying and minor tweaks.
2010-10-31 05:09:05 +01:00
i64 total = 0;
i64 i;
Initial import
2010-03-29 01:07:08 +02:00
for (i = 0; i < (1U << st->hash_bits); i++) {
if (empty_hash(st, i))
continue;
total++;
if (primary_hash(st, st->hash_table[i].t) == i)
primary++;
}
if (total != st->hash_count)
Reworked the multithreading massively. Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating. Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible. This causes a big speed up on the compression side on SMP machines. Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows. Alter the progress output to match some of the changes in verbose modes.
2010-11-12 15:26:09 +01:00
print_err("WARNING: hash_count says total %lld\n", st->hash_count);
Initial import
2010-03-29 01:07:08 +02:00
Reworked the multithreading massively. Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating. Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible. This causes a big speed up on the compression side on SMP machines. Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows. Alter the progress output to match some of the changes in verbose modes.
2010-11-12 15:26:09 +01:00
print_output("%lld total hashes -- %lld in primary bucket (%-2.3f%%)\n", total, primary,
Fix nan when no hashes found in verbose message.
2011-05-07 16:34:09 +02:00
primary * 100.0 / (total ? : 1));
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool hash_search(rzip_control *control, struct rzip_state *st, double pct_base, double pct_multiple)
Initial import
2010-03-29 01:07:08 +02:00
{
Progress update percent was being zeroed in rzip.c thus displaying on every loop. Move it higher, fixing the amount of output to the terminal.
2011-05-07 02:19:12 +02:00
int lastpct = 0, last_chunkpct = 0;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
i64 cksum_limit = 0, p, end;
Initial import
2010-03-29 01:07:08 +02:00
tag t = 0;
struct {
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 p;
Initial import
2010-03-29 01:07:08 +02:00
i64 ofs;
i64 len;
} current;
tag tag_mask = (1 << st->level->initial_freq) - 1;
We were attempting to truncate mmap to page size when only the offset needed to be. Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space. Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram. Don't fail if preallocation is unsuccessful. Add more detailed error reporting. Minor cleanups.
2010-10-31 14:19:39 +01:00
if (st->hash_table)
Initial import
2010-03-29 01:07:08 +02:00
memset(st->hash_table, 0, sizeof(st->hash_table[0]) * (1<<st->hash_bits));
We were attempting to truncate mmap to page size when only the offset needed to be. Fix the longstanding limit on 32 bits that allowed us to allocate only 2GB of ram by moving the big malloc calls to mmap equivalents which allow us to mmap up to 2^44 bytes of anonymous space. Use progressively smaller preallocation to try and defragment ram prior to real mmap call to increase success rate of allocating ram when it's a significant proportion of total ram. Don't fail if preallocation is unsuccessful. Add more detailed error reporting. Minor cleanups.
2010-10-31 14:19:39 +01:00
else {
Initial import
2010-03-29 01:07:08 +02:00
i64 hashsize = st->level->mb_used *
(1024 * 1024 / sizeof(st->hash_table[0]));
for (st->hash_bits = 0; (1U << st->hash_bits) < hashsize; st->hash_bits++);
Start cleaning up all the flag testing with neat macros.
2010-10-31 18:53:53 +01:00
print_maxverbose("hashsize = %lld. bits = %lld. %luMB\n",
hashsize, st->hash_bits, st->level->mb_used);
Initial import
2010-03-29 01:07:08 +02:00
/* 66% full at max. */
st->hash_limit = (1 << st->hash_bits) / 3 * 2;
st->hash_table = calloc(sizeof(st->hash_table[0]), (1 << st->hash_bits));
}
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(!st->hash_table))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to allocate hash table in hash_search\n"), false);
Initial import
2010-03-29 01:07:08 +02:00
st->minimum_tag_mask = tag_mask;
st->tag_clean_ptr = 0;
st->cksum = 0;
st->hash_count = 0;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
p = 0;
end = st->chunk_size - MINIMUM_MATCH;
Initial import
2010-03-29 01:07:08 +02:00
st->last_match = p;
current.len = 0;
current.p = p;
current.ofs = 0;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (likely(end > 0)) {
Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. Put special case management of zero sized files. Modify the percentage calculation to be able to cope with small numbers and zeroes.
2011-05-07 10:37:25 +02:00
t = full_tag(control, st, p);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(t == -1)) return false;
}
Initial import
2010-03-29 01:07:08 +02:00
while (p < end) {
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
i64 reverse, mlen, offset = 0;
Initial import
2010-03-29 01:07:08 +02:00
p++;
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.offset_search = p;
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
t = next_tag(control, st, p, t);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(t == -1)) return false;
Initial import
2010-03-29 01:07:08 +02:00
/* Don't look for a match if there are no tags with
this number of bits in the hash table. */
if ((t & st->minimum_tag_mask) != st->minimum_tag_mask)
continue;
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
mlen = find_best_match(control, st, t, p, end, &offset, &reverse);
Initial import
2010-03-29 01:07:08 +02:00
/* Only insert occasionally into hash. */
if ((t & tag_mask) == tag_mask) {
st->stats.inserts++;
st->hash_count++;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
insert_hash(st, t, p);
Initial import
2010-03-29 01:07:08 +02:00
if (st->hash_count > st->hash_limit)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
tag_mask = clean_one_from_hash(control, st);
Initial import
2010-03-29 01:07:08 +02:00
}
if (mlen > current.len) {
current.p = p - reverse;
current.len = mlen;
current.ofs = offset;
}
if ((current.len >= GREAT_MATCH || p >= current.p + MINIMUM_MATCH)
&& current.len >= MINIMUM_MATCH) {
if (st->last_match < current.p)
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!put_literal(control, st, st->last_match, current.p))) return false;
if (unlikely(!put_match(control, st, current.p, current.ofs, current.len))) return false;
Initial import
2010-03-29 01:07:08 +02:00
st->last_match = current.p + current.len;
current.p = p = st->last_match;
current.len = 0;
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
t = full_tag(control, st, p);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(t == -1)) return false;
Initial import
2010-03-29 01:07:08 +02:00
}
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
if (unlikely(p % 128 == 0)) {
int pct, chunk_pct;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
pct = pct_base + (pct_multiple * (100.0 * p) /
More percentage display fixes for divide by zero errors.
2011-05-07 16:38:33 +02:00
(st->chunk_size ? : 1));
chunk_pct = p / ((end / 100) ? : 1);
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
if (pct != lastpct || chunk_pct != last_chunkpct) {
Display the progress percentage and ETA when it's known on STDIN compress.
2011-03-02 03:39:21 +01:00
if (!STDIN || st->stdin_eof)
Consciously check page size, even though no one's going to build this on a machine with a different page size. Clean up builds, removing ifdefs from main code. Make output choices consistent.
2010-11-06 08:17:33 +01:00
print_progress("Total: %2d%% ", pct);
print_progress("Chunk: %2d%%\r", chunk_pct);
add info callback which provides percentage completion of operation
2011-08-13 10:05:01 +02:00
if (control->info_cb)
control->info_cb(control->info_data,
(!STDIN || st->stdin_eof) ? pct : -1, chunk_pct);
Initial import
2010-03-29 01:07:08 +02:00
lastpct = pct;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
last_chunkpct = chunk_pct;
Initial import
2010-03-29 01:07:08 +02:00
}
}
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
if (p > (i64)cksum_limit) {
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
i64 i, n = MIN(st->chunk_size - p, control->page_size);
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
uchar *ckbuf = malloc(n);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
Use a different failure mode for when perror is unlikely to be set. Add 2 unlikely wrappers.
2011-02-21 04:51:20 +01:00
if (unlikely(!ckbuf))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to malloc ckbuf in hash_search\n"), false);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
for (i = 0; i < n; i++)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
memcpy(ckbuf + i, get_sb(control, cksum_limit + i), 1);
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
st->cksum = CrcUpdate(st->cksum, ckbuf, n);
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
if (!NO_MD5)
md5_process_bytes(ckbuf, n, &control->ctx);
Initial import
2010-03-29 01:07:08 +02:00
cksum_limit += n;
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
free(ckbuf);
Initial import
2010-03-29 01:07:08 +02:00
}
}
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
if (MAX_VERBOSE)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
show_distrib(control, st);
Initial import
2010-03-29 01:07:08 +02:00
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
if (st->last_match < st->chunk_size)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
put_literal(control, st, st->last_match, st->chunk_size);
Initial import
2010-03-29 01:07:08 +02:00
if (st->chunk_size > cksum_limit) {
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
i64 i, n = st->chunk_size - cksum_limit;
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
uchar *ckbuf = malloc(n);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
Use a different failure mode for when perror is unlikely to be set. Add 2 unlikely wrappers.
2011-02-21 04:51:20 +01:00
if (unlikely(!ckbuf))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to malloc ckbuf in hash_search\n"), false);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
for (i = 0; i < n; i++)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
memcpy(ckbuf + i, get_sb(control, cksum_limit + i), 1);
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
st->cksum = CrcUpdate(st->cksum, ckbuf, n);
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
if (!NO_MD5)
md5_process_bytes(ckbuf, n, &control->ctx);
Initial import
2010-03-29 01:07:08 +02:00
cksum_limit += n;
Fix md5 process bytes to occur at the same time as crc with the same buffer, saving time.
2011-02-19 00:34:45 +01:00
free(ckbuf);
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!put_literal(control, st, 0, 0))) return false;
if (unlikely(!put_u32(control, st->ss, st->cksum))) return false;
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
static void init_hash_indexes(struct rzip_state *st)
{
int i;
for (i = 0; i < 256; i++)
st->hash_index[i] = ((random() << 16) ^ random());
}
Remove unused flags by fake mremap.
2011-03-09 03:13:33 +01:00
static inline void *fake_mremap(void *old_address, size_t old_size, size_t new_size, int flags __UNUSED__)
Fix darwin build and clean up ifdef and incorrect ordered includes at the same time. All builds were using fakememopen due to incorrect ifdef usage, so make GNU builds actually use memopen again.
2010-11-03 03:14:46 +01:00
{
munmap(old_address, old_size);
return mmap(old_address, new_size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
}
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
/* stdin is not file backed so we have to emulate the mmap by mapping
* anonymous ram and reading stdin into it. It means the maximum ram
* we can use will be less but we will already have determined this in
* rzip_chunk */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool mmap_stdin(rzip_control *control, uchar *buf, struct rzip_state *st)
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
{
i64 len = st->chunk_size;
uchar *offset_buf = buf;
ssize_t ret;
i64 total;
total = 0;
while (len > 0) {
Tidy.
2011-03-15 00:41:17 +01:00
ret = MIN(len, one_g);
another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above
2011-08-13 08:21:45 +02:00
ret = read(fileno(control->inFILE), offset_buf, (size_t)ret);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(ret < 0))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to read in mmap_stdin\n"), false);
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
total += ret;
if (ret == 0) {
/* Should be EOF */
print_maxverbose("Shrinking chunk to %lld\n", total);
Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. Put special case management of zero sized files. Modify the percentage calculation to be able to cope with small numbers and zeroes.
2011-05-07 10:37:25 +02:00
if (likely(total)) {
buf = (uchar *)mremap(buf, st->chunk_size, total, 0);
st->mmap_size = st->chunk_size = total;
} else {
/* Empty file */
buf = (uchar *)mremap(buf, st->chunk_size, control->page_size, 0);
st->mmap_size = control->page_size;
st->chunk_size = 0;
}
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(buf == MAP_FAILED))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to remap to smaller buf in mmap_stdin\n"), false);
Store the file size as zero when the magic header will be written before the size is known.
2011-03-11 13:45:22 +01:00
control->eof = st->stdin_eof = 1;
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
break;
}
offset_buf += ret;
len -= ret;
}
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
control->st_size += total;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
return true;
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool init_sliding_mmap(rzip_control *control, struct rzip_state *st, int fd_in, i64 offset)
Initial import
2010-03-29 01:07:08 +02:00
{
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
/* Initialise the high buffer */
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
if (!STDIN) {
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.high_length = 65536;
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
/* Round up to the next biggest page size */
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
if (sb.high_length % control->page_size)
sb.high_length += control->page_size - (sb.high_length % control->page_size);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.buf_high = (uchar *)mmap(NULL, sb.high_length, PROT_READ, MAP_SHARED, fd_in, offset);
if (unlikely(sb.buf_high == MAP_FAILED))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Unable to mmap buf_high in init_sliding_mmap\n"), false);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.size_high = sb.high_length;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
sb.offset_high = 0;
}
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
sb.offset_low = 0;
sb.offset_search = 0;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
sb.size_low = st->mmap_size;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
sb.orig_size = st->chunk_size;
sb.fd = fd_in;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
return true;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
}
Initial import
2010-03-29 01:07:08 +02:00
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
/* compress a chunk of an open file. Assumes that the file is able to
be mmap'd and is seekable */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
static bool rzip_chunk(rzip_control *control, struct rzip_state *st, int fd_in, int fd_out, i64 offset,
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
double pct_base, double pct_multiple)
{
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!init_sliding_mmap(control, st, fd_in, offset))) return false;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
st->ss = open_stream_out(control, fd_out, NUM_STREAMS, st->chunk_size, st->chunk_bytes);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(!st->ss))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to open streams in rzip_chunk\n"), false);
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
print_verbose("Beginning rzip pre-processing phase\n");
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!hash_search(control, st, pct_base, pct_multiple))) {
close_stream_out(control, st->ss);
return false;
}
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
Initial import
2010-03-29 01:07:08 +02:00
/* unmap buffer before closing and reallocating streams */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(munmap(sb.buf_low, sb.size_low))) {
close_stream_out(control, st->ss);
fatal_return(("Failed to munmap in rzip_chunk\n"), false);
}
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
if (!STDIN) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(munmap(sb.buf_high, sb.size_high))) {
close_stream_out(control, st->ss);
fatal_return(("Failed to munmap in rzip_chunk\n"), false);
}
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
}
Initial import
2010-03-29 01:07:08 +02:00
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
if (unlikely(close_stream_out(control, st->ss)))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to flush/close streams in rzip_chunk\n"), false);
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
/* compress a whole file chunks at a time */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
bool rzip_fd(rzip_control *control, int fd_in, int fd_out)
Initial import
2010-03-29 01:07:08 +02:00
{
/* add timers for ETA estimates
* Base it off the file size and number of iterations required
* depending on compression window size
* Track elapsed time and estimated time to go
* If file size < compression window, can't do
*/
struct timeval current, start, last;
More tidying up and ensure control window is obeyed.
2010-11-19 23:22:47 +01:00
i64 len = 0, last_chunk = 0;
Being implementing md5 hash checking by importing the md5 code from coreutils. Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
2011-02-17 23:09:40 +01:00
int pass = 0, passes, j;
Clean up unused variables.
2011-03-08 01:36:56 +01:00
struct rzip_state *st;
Check for free space before compression/decompression and abort if -f option is not enabled.
2011-02-26 13:10:28 +01:00
struct statvfs fbuf;
Clean up unused variables.
2011-03-08 01:36:56 +01:00
struct stat s, s2;
double chunkmbs;
Check for free space before compression/decompression and abort if -f option is not enabled.
2011-02-26 13:10:28 +01:00
i64 free_space;
Being implementing md5 hash checking by importing the md5 code from coreutils. Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
2011-02-17 23:09:40 +01:00
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
if (!NO_MD5)
md5_init_ctx (&control->ctx);
Initial import
2010-03-29 01:07:08 +02:00
st = calloc(sizeof(*st), 1);
Unlimited mode is now usable in a meaningful timeframe! Modify the sliding mmap window to have a 64k smaller buffer which matches the size of the search size, and change the larger lower buffer to make it slide with the main hash search progress. This makes for a MUCH faster unlimited mode, making it actually usable. Limit windows to 2GB again on 32 bit, but do it when determining the largest size possible in rzip.c. Implement a linux-kernel like unlikely() wrapper for inbuilt expect, and modify most fatal warnings to be unlikely, and a few places where it's also suitable. Minor cleanups.
2010-11-05 02:16:43 +01:00
if (unlikely(!st))
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
fatal_return(("Failed to allocate control state in rzip_fd\n"), false);
Initial import
2010-03-29 01:07:08 +02:00
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
if (LZO_COMPRESS) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(lzo_init() != LZO_E_OK)) {
free(st);
fatal_return(("lzo_init() failed\n"), false);
}
Initial import
2010-03-29 01:07:08 +02:00
}
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(fstat(fd_in, &s))) {
free(st);
fatal_return(("Failed to stat fd_in in rzip_fd\n"), false);
}
Initial import
2010-03-29 01:07:08 +02:00
Fix file size recording in the magic header when stdin is used.
2010-11-01 09:18:20 +01:00
if (!STDIN) {
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
len = control->st_size = s.st_size;
Fix file size recording in the magic header when stdin is used.
2010-11-01 09:18:20 +01:00
print_verbose("File size: %lld\n", len);
} else
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
control->st_size = 0;
Initial import
2010-03-29 01:07:08 +02:00
Implement the real workings of writing to a temporary buffer before flushing to stdout.
2011-03-12 12:46:57 +01:00
if (!STDOUT) {
/* Check if there's enough free space on the device chosen to fit the
* compressed file, based on the compressed file being as large as the
* uncompressed file. */
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(fstatvfs(fd_out, &fbuf))) {
free(st);
fatal_return(("Failed to fstatvfs in compress_file\n"), false);
}
Cast fstat values to i64 to prevent 32 bit overflows.
2011-03-14 00:21:10 +01:00
free_space = (i64)fbuf.f_bsize * (i64)fbuf.f_bavail;
Implement the real workings of writing to a temporary buffer before flushing to stdout.
2011-03-12 12:46:57 +01:00
if (free_space < control->st_size) {
if (FORCE_REPLACE)
print_err("Warning, possibly inadequate free space detected, but attempting to compress due to -f option being used.\n");
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
else {
free(st);
failure_return(("Possibly inadequate free space to compress file, use -f to override.\n"), false);
}
Implement the real workings of writing to a temporary buffer before flushing to stdout.
2011-03-12 12:46:57 +01:00
}
Check for free space before compression/decompression and abort if -f option is not enabled.
2011-02-26 13:10:28 +01:00
}
Rationalise the testing now that the default lzma settings use a lot less ram by default.
2010-12-10 15:07:43 +01:00
/* Optimal use of ram involves using no more than 2/3 of it, so we
* allocate 1/3 of it to the main buffer and use a sliding mmap
* buffer to work on 2/3 ram size, leaving enough ram for the
* compression backends */
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
control->max_mmap = control->maxram;
round_to_page(&control->max_mmap);
More tidying up and ensure control window is obeyed.
2010-11-19 23:22:47 +01:00
Don't round to page size if the entire buffer will fit, otherwise this generates one extra pass of a tiny chunk.
2011-02-24 01:38:31 +01:00
/* Set maximum chunk size to 2/3 of ram if not unlimited or specified
* by a control window. When it's smaller than the file size, round it
* to page size for efficiency. */
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
if (UNLIMITED)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
control->max_chunk = control->st_size;
else if (control->window)
control->max_chunk = control->window * CHUNK_MULTIPLE;
Don't round to page size if the entire buffer will fit, otherwise this generates one extra pass of a tiny chunk.
2011-02-24 01:38:31 +01:00
else
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
control->max_chunk = control->ramsize / 3 * 2;
control->max_mmap = MIN(control->max_mmap, control->max_chunk);
if (control->max_chunk < control->st_size)
round_to_page(&control->max_chunk);
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
if (!STDIN)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
st->chunk_size = MIN(control->max_chunk, len);
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
else
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
st->chunk_size = control->max_mmap;
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
if (st->chunk_size < len)
round_to_page(&st->chunk_size);
Initial import
2010-03-29 01:07:08 +02:00
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
st->level = &levels[control->compression_level];
Initial import
2010-03-29 01:07:08 +02:00
st->fd_in = fd_in;
st->fd_out = fd_out;
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
st->stdin_eof = 0;
Initial import
2010-03-29 01:07:08 +02:00
init_hash_indexes(st);
Bump version to 0.541. Limit LZMA window to 300MB on 32 bit as per reports of failure when larger. Minor documentation and display clean ups.
2010-11-18 13:33:43 +01:00
passes = 0;
Initial import
2010-03-29 01:07:08 +02:00
/* set timers and chunk counter */
last.tv_sec = last.tv_usec = 0;
gettimeofday(&start, NULL);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!prepare_streamout_threads(control))) {
free(st->hash_table);
free(st);
return false;
}
Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup. Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring. Fix a few dodgy callocs that may have been overflowing! The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
2010-12-10 13:51:59 +01:00
Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. Put special case management of zero sized files. Modify the percentage calculation to be able to cope with small numbers and zeroes.
2011-05-07 10:37:25 +02:00
while (!pass || len > 0 || (STDIN && !st->stdin_eof)) {
Initial import
2010-03-29 01:07:08 +02:00
double pct_base, pct_multiple;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
i64 offset = s.st_size - len;
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
int bits = 8;
Initial import
2010-03-29 01:07:08 +02:00
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
st->chunk_size = control->max_chunk;
st->mmap_size = control->max_mmap;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
if (!STDIN) {
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
st->chunk_size = MIN(st->chunk_size, len);
Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. Put special case management of zero sized files. Modify the percentage calculation to be able to cope with small numbers and zeroes.
2011-05-07 10:37:25 +02:00
if (likely(st->chunk_size))
st->mmap_size = MIN(st->mmap_size, len);
else
st->mmap_size = control->page_size;
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
}
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
retry:
if (STDIN) {
/* NOTE the buf is saved here for STDIN mode */
sb.buf_low = mmap(NULL, st->mmap_size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
/* Better to shrink the window to the largest size that works than fail */
if (sb.buf_low == MAP_FAILED) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(errno != ENOMEM)) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
fatal_return(("Failed to mmap %s\n", control->infile), false);
}
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
st->mmap_size = st->mmap_size / 10 * 9;
round_to_page(&st->mmap_size);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!st->mmap_size)) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
fatal_return(("Unable to mmap any ram\n"), false);
}
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
goto retry;
}
Fix stdin failing due to getting sizes all wrong. Fix stdin compression values not being shown at end. Fix inappropriate failure when lzma doesn't compress block.
2010-12-12 07:40:58 +01:00
st->chunk_size = st->mmap_size;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!mmap_stdin(control, sb.buf_low, st))) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
return false;
}
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
} else {
Move message to after size is finalised.
2010-12-11 02:25:43 +01:00
/* NOTE The buf is saved here for !STDIN mode */
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
sb.buf_low = (uchar *)mmap(sb.buf_low, st->mmap_size, PROT_READ, MAP_SHARED, fd_in, offset);
if (sb.buf_low == MAP_FAILED) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(errno != ENOMEM)) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
fatal_return(("Failed to mmap %s\n", control->infile), false);
}
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
st->mmap_size = st->mmap_size / 10 * 9;
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
round_to_page(&st->mmap_size);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!st->mmap_size)) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
fatal_return(("Unable to mmap any ram\n"), false);
}
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
goto retry;
}
Move message to after size is finalised.
2010-12-11 02:25:43 +01:00
if (st->mmap_size < st->chunk_size)
print_maxverbose("Enabling sliding mmap mode and using mmap of %lld bytes with window of %lld bytes\n", st->mmap_size, st->chunk_size);
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
}
print_maxverbose("Succeeded in testing %lld sized mmap for rzip pre-processing\n", st->mmap_size);
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
if (st->chunk_size > control->ramsize)
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
print_verbose("Compression window is larger than ram, will proceed with unlimited mode possibly much slower\n");
Implement -p option to specify number of processors to determine thread count. Remove -P option as failing to set permissions only issues a warning now, removing any requirement for -P. Change default compression level back to 7 as 9 was not giving significantly better compression but was slowing things down.
2010-11-13 07:36:21 +01:00
Fix the arbitrary lower limit of 128 bytes by compressing zeroes beyond the size of the chunk. Put special case management of zero sized files. Modify the percentage calculation to be able to cope with small numbers and zeroes.
2011-05-07 10:37:25 +02:00
if (!passes && !STDIN && st->chunk_size) {
Bump version number to 0.542. Choose sane defaults for memory usage since linux ludicriously overcommits. Use sliding mmap for any compression windows greater than 2/3 ram. Consolidate and simplify testing of allocatable ram. Minor tweaks to output. Round up the size of the high buffer in sliding mmap to one page. Squeeze a little more out of 32 bit compression windows.
2010-11-19 15:23:08 +01:00
passes = s.st_size / st->chunk_size + !!(s.st_size % st->chunk_size);
if (passes == 1)
print_verbose("Will take 1 pass\n");
else
print_verbose("Will take %d passes\n", passes);
}
Move the ram allocation phase into rzip_fd to be able to get a more accurate measure of percentage done. Prevent failure when offset is not a multiple of page size. Add chunk percentage complete to output. Tweak output at various verbosities. Update documentation to reflect improved performance of unlimited mode. Update benchmark results. More tidying.
2010-11-05 13:02:58 +01:00
sb.orig_offset = offset;
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
print_maxverbose("Chunk size: %lld\n", st->chunk_size);
Initial import
2010-03-29 01:07:08 +02:00
Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup. Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring. Fix a few dodgy callocs that may have been overflowing! The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
2010-12-10 13:51:59 +01:00
/* Determine the chunk byte width to write to the file
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
* This allows archives of different chunk sizes to have
Do real stdin without the stupid temporary files by mmapping into anonymous ram.
2010-11-01 06:45:48 +01:00
* optimal byte width entries. When working with stdin we
* won't know in advance how big it is so it will always be
* rounded up to the window size. */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
while (st->chunk_size >> bits > 0)
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
bits++;
st->chunk_bytes = bits / 8;
if (bits % 8)
st->chunk_bytes++;
print_maxverbose("Byte width: %d\n", st->chunk_bytes);
Display the progress percentage and ETA when it's known on STDIN compress.
2011-03-02 03:39:21 +01:00
if (STDIN)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
pct_base = (100.0 * -len) / control->st_size;
Display the progress percentage and ETA when it's known on STDIN compress.
2011-03-02 03:39:21 +01:00
else
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
pct_base = (100.0 * (control->st_size - len)) / control->st_size;
pct_multiple = ((double)st->chunk_size) / control->st_size;
Initial import
2010-03-29 01:07:08 +02:00
pass++;
Display the progress percentage and ETA when it's known on STDIN compress.
2011-03-02 03:39:21 +01:00
if (st->stdin_eof)
passes = pass;
Initial import
2010-03-29 01:07:08 +02:00
gettimeofday(&current, NULL);
/* this will count only when size > window */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
if (last.tv_sec > 0) {
Clean up and fix times displayed. Merge fixes.
2011-03-02 04:16:57 +01:00
unsigned int eta_hours, eta_minutes, eta_seconds, elapsed_time, finish_time,
elapsed_hours, elapsed_minutes, elapsed_seconds, diff_seconds;
Change the byte width to be variable depending on chunk size, and write it as a single char describing the next byte width for decompression. More minor tidying.
2010-11-01 03:28:49 +01:00
elapsed_time = current.tv_sec - start.tv_sec;
More divide by zero output fixes.
2011-05-07 16:45:56 +02:00
finish_time = elapsed_time / ((pct_base / 100.0) ? : 1);
Clean up and fix times displayed. Merge fixes.
2011-03-02 04:16:57 +01:00
elapsed_hours = elapsed_time / 3600;
elapsed_minutes = (elapsed_time / 60) % 60;
elapsed_seconds = elapsed_time % 60;
diff_seconds = finish_time - elapsed_time;
eta_hours = diff_seconds / 3600;
eta_minutes = (diff_seconds / 60) % 60;
eta_seconds = diff_seconds % 60;
Updated benchmark results. More tidying up.
2010-11-05 04:52:14 +01:00
chunkmbs = (last_chunk / 1024 / 1024) / (double)(current.tv_sec-last.tv_sec);
Display the progress percentage and ETA when it's known on STDIN compress.
2011-03-02 03:39:21 +01:00
if (!STDIN || st->stdin_eof)
Ensure file size is correct on compressed file from stdin. Improve visual output. Update manpage.
2010-11-05 14:00:44 +01:00
print_verbose("\nPass %d / %d -- Elapsed Time: %02d:%02d:%02d. ETA: %02d:%02d:%02d. Compress Speed: %3.3fMB/s.\n",
pass, passes, elapsed_hours, elapsed_minutes, elapsed_seconds,
eta_hours, eta_minutes, eta_seconds, chunkmbs);
else
Bump version to 0.541. Limit LZMA window to 300MB on 32 bit as per reports of failure when larger. Minor documentation and display clean ups.
2010-11-18 13:33:43 +01:00
print_verbose("\nPass %d -- Elapsed Time: %02d:%02d:%02d. Compress Speed: %3.3fMB/s.\n",
pass, elapsed_hours, elapsed_minutes, elapsed_seconds, chunkmbs);
Initial import
2010-03-29 01:07:08 +02:00
}
last.tv_sec = current.tv_sec;
last.tv_usec = current.tv_usec;
Clean up and fix times displayed. Merge fixes.
2011-03-02 04:16:57 +01:00
Get rid of extra data in new header and stick to old header size, compacting extra info into existing space.
2011-03-18 07:32:47 +01:00
if (st->chunk_size == len)
control->eof = 1;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!rzip_chunk(control, st, fd_in, fd_out, offset, pct_base, pct_multiple))) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
return false;
}
Clean up and fix times displayed. Merge fixes.
2011-03-02 04:16:57 +01:00
Bump version to 0.541. Limit LZMA window to 300MB on 32 bit as per reports of failure when larger. Minor documentation and display clean ups.
2010-11-18 13:33:43 +01:00
/* st->chunk_size may be shrunk in rzip_chunk */
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
last_chunk = st->chunk_size;
len -= st->chunk_size;
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(len > 0 && control->eof)) {
close_streamout_threads(control);
free(st->hash_table);
free(st);
failure_return(("Wrote EOF to file yet chunk_size was shrunk, corrupting archive.\n"), false);
}
Initial import
2010-03-29 01:07:08 +02:00
}
Cleanups.
2011-03-19 03:53:43 +01:00
if (likely(st->hash_table))
free(st->hash_table);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!close_streamout_threads(control))) {
free(st);
return false;
}
Move the threading on compression to higher up in the code, allowing the next stream to start using compression threads before the previous stream has finished. This overlapping of compressing streams means that when files are large enough to be split into multiple blocks, all CPUs will be used more effectively throughout the compression, affording a nice speedup. Move the writing of the chunk byte size and initial headers into the compthread to prevent any races occurring. Fix a few dodgy callocs that may have been overflowing! The previous commit reverts were done because the changes designed to speed it up actually slowed it down instead.
2010-12-10 13:51:59 +01:00
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
if (!NO_MD5) {
/* Temporary workaround till someone fixes apple md5 */
another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above
2011-08-13 08:21:45 +02:00
md5_finish_ctx(&control->ctx, control->md5_resblock);
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
if (HASH_CHECK || MAX_VERBOSE) {
print_output("MD5: ");
for (j = 0; j < MD5_DIGEST_SIZE; j++)
another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above
2011-08-13 08:21:45 +02:00
print_output("%02x", control->md5_resblock[j] & 0xFF);
Disable md5 checking on apple for now till someone fixes it.
2011-04-13 06:50:26 +02:00
print_output("\n");
}
/* When encrypting data, we encrypt the MD5 value as well */
if (ENCRYPT)
another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above
2011-08-13 08:21:45 +02:00
if (unlikely(!lrz_encrypt(control, control->md5_resblock, MD5_DIGEST_SIZE, control->salt_pass))) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
free(st);
return false;
}
another semi-clusterfuck commit: *new logging macros *allow compression to and from any FILE, not just stdin/stdout *update main.c to work with the above
2011-08-13 08:21:45 +02:00
if (unlikely(write_1g(control, control->md5_resblock, MD5_DIGEST_SIZE) != MD5_DIGEST_SIZE)) {
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
free(st);
fatal_return(("Failed to write md5 in rzip_fd\n"), false);
}
Implement md5 checking on decompression. Implement hash check flag to determine whether to show md5sum on compression/decompression or not.
2011-02-18 05:16:13 +01:00
}
Being implementing md5 hash checking by importing the md5 code from coreutils. Implement md5 hash checking on compression by doing the md5 hash check as each sb low buffer has been allocated to avoid going over the file again where possible.
2011-02-17 23:09:40 +01:00
Create a flag to know when the temporary output buffer is in use, in preparation for when we use it on decompression.
2011-03-12 22:34:06 +01:00
if (TMP_OUTBUF)
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
if (unlikely(!flush_tmpoutbuf(control))) {
free(st);
return false;
}
Implement the real workings of writing to a temporary buffer before flushing to stdout.
2011-03-12 12:46:57 +01:00
Initial import
2010-03-29 01:07:08 +02:00
gettimeofday(&current, NULL);
Fix stdin failing due to getting sizes all wrong. Fix stdin compression values not being shown at end. Fix inappropriate failure when lzma doesn't compress block.
2010-12-12 07:40:58 +01:00
if (STDIN)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
s.st_size = control->st_size;
Initial import
2010-03-29 01:07:08 +02:00
chunkmbs = (s.st_size / 1024 / 1024) / ((double)(current.tv_sec-start.tv_sec)? : 1);
fstat(fd_out, &s2);
Start cleaning up all the flag testing with neat macros.
2010-10-31 18:53:53 +01:00
print_maxverbose("matches=%u match_bytes=%u\n",
(unsigned int)st->stats.matches, (unsigned int)st->stats.match_bytes);
print_maxverbose("literals=%u literal_bytes=%u\n",
(unsigned int)st->stats.literals, (unsigned int)st->stats.literal_bytes);
print_maxverbose("true_tag_positives=%u false_tag_positives=%u\n",
(unsigned int)st->stats.tag_hits, (unsigned int)st->stats.tag_misses);
print_maxverbose("inserts=%u match %.3f\n",
(unsigned int)st->stats.inserts,
(1.0 + st->stats.match_bytes) / st->stats.literal_bytes);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
if (!STDIN)
rebase-of-function-split-and-control-additions-to-fu
2011-03-08 22:32:14 +01:00
print_progress("%s - ", control->infile);
Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of accessing the buffer directly, thus allowing us to have window sizes larger than available ram. This is implemented through the use of a "sliding mmap" implementation. Sliding mmap uses two mmapped buffers, one large one as previously, and one page sized smaller one. When an attempt is made to read beyond the end of the large buffer, the small buffer is remapped to the file area that's being accessed. While this implementation is 100x slower than direct mmapping, it allows us to implement unlimited sized compression windows. Implement the -U option with unlimited sized windows. Rework the selection of compression windows. Instead of trying to guess how much ram the machine might be able to access, we try to safely buffer as much ram as we can, and then use that to determine the file buffer size. Do not choose an arbitrary upper window limit unless -w is specified. Rework the -M option to try to buffer the entire file, reducing the buffer size until we succeed. Align buffer sizes to page size. Clean up lots of unneeded variables. Fix lots of minor logic issues to do with window sizes accepted/passed to rzip and the compression backends. More error handling. Change -L to affect rzip compression level directly as well as backend compression level and use 9 by default now. More cleanups of information output. Use 3 point release numbering in case one minor version has many subversions. Numerous minor cleanups and tidying. Updated docs and manpages.
2010-11-04 11:14:55 +01:00
print_progress("Compression Ratio: %.3f. Average Compression Speed: %6.3fMB/s.\n",
1.0 * s.st_size / s2.st_size, chunkmbs);
Initial import
2010-03-29 01:07:08 +02:00
free(st);
giant clusterfuck commit: *add and use alloca/strdupa in some spots *implement return values on almost every function *free() is now a macro which also nulls the freed pointer *when in 'library mode', fatal_exit() will not be called *fatal()/failure() are now macros which add line number and file for logging *all occurrences of fatal/failure in files other than main.c are now wrapped to use a logging callback (if specified) and then return *functions now clean up resources on returning with an error
2011-08-12 08:43:42 +02:00
return true;
Initial import
2010-03-29 01:07:08 +02:00
}
move liblrzip.h contents to lrzip_private.h global variables-----
2011-08-11 09:06:32 +02:00
void rzip_control_free(rzip_control *control)
{
add some gross bucketing to implement a queue for freeing stream_info structs after an operation has completed (in library mode ONLY; no changes will occur during regular execution) to fix some leaks
2011-08-13 09:35:36 +02:00
size_t x;
if (!control) return;
move liblrzip.h contents to lrzip_private.h global variables-----
2011-08-11 09:06:32 +02:00
add some gross bucketing to implement a queue for freeing stream_info structs after an operation has completed (in library mode ONLY; no changes will occur during regular execution) to fix some leaks
2011-08-13 09:35:36 +02:00
free(control->tmpdir);
free(control->outname);
free(control->outdir);
if (control->suffix && control->suffix[0]) free(control->suffix);
for (x = 0; x < control->sinfo_idx; x++) {
free(control->sinfo_queue[x]->s);
free(control->sinfo_queue[x]);
}
free(control->sinfo_queue);
free(control);
move liblrzip.h contents to lrzip_private.h global variables-----
2011-08-11 09:06:32 +02:00
}
Reference in a new issue Copy permalink