Commit graph

7 commits

Author SHA1 Message Date
Con Kolivas 2b08c6e280 Implement massive multithreading decompression.
This is done by taking each stream of data on read in into separate buffers for up to as many threads as CPUs.
As each thread's data becomes available, feed it into runzip once it is requests more of the stream.
Provided there are enough chunks in the originally compressed data, this provides a massive speedup potentially proportional to the number of CPUs. The slower the backend compression, the better the speed up (i.e. zpaq is the best sped up).
Fix the output of zpaq compress and decompress from trampling on itself and racing and consuming a lot of CPU time printing to the console.
When limiting cwindow to 6 on 32 bits, ensure that control.window is also set.
When testing for the maximum size of testmalloc, the multiple used was out by one, so increase it.
Minor output tweaks.
2010-11-16 21:25:32 +11:00
Con Kolivas b17ad2fa41 Minor warning fix. 2010-11-15 19:54:02 +11:00
Con Kolivas e9957e1115 Fix zpaq compression now updating the console too much because it's now so much faster it uses up a lot of CPU time just ouputting to the screen.
Do this by updating only every 10%, and print separate updates for each thread.
2010-11-13 08:33:30 +11:00
Con Kolivas 02de002c58 Reworked the multithreading massively.
Place the data from each stream into a buffer that then is handed over to one thread which is allowed to begin doing the backend compression while the main rzip stream continues operating.
Fork up to as many threads as CPUs and feed data to them in a ring fashion, parallelising the workload as much as possible.
This causes a big speed up on the compression side on SMP machines.
Thread compression is limited to a minimum of 10MB compressed per thread to minimise the compromise to compression of smaller windows.
Alter the progress output to match some of the changes in verbose modes.
2010-11-13 01:26:09 +11:00
Con Kolivas d972496aa8 Make messages come via stdout instead of stderror courtesy of Alexander Saprykin 2010-04-25 16:26:00 +10:00
Con Kolivas e3916090a4 Fix more CRLF EOL 2010-04-25 16:20:51 +10:00
Con Kolivas 6dcceb0b1b Initial import 2010-03-29 10:07:08 +11:00