Zip File created contains 0 byte files

Nov 28, 2010 at 4:28 PM

I haven't a clue what I'm doing wrong.  I'm trying to create a zip file of a LOT (65,000+) of small files, so I wanted to multi-thread if possible.  The problem is the zip it creates all the files are 0 bytes.  I pretty much copied the code from one of your examples, but it isn't working.  A little help please!

I already had a simple .zip option working, but it took quite a long time so I wanted to try to use a multi-threaded option.

Thanks Much!

 


 

 

' Create List of Files to Zip
Dim FilesToZip As New ArrayList
Dim FilterList() As String = {"*.txt", "*.properties"}
FilesToZip.AddRange(Directory.GetFiles(ServerWorldPath, "*.*", SearchOption.AllDirectories))
For Each Filter As String In FilterList
     FilesToZip.AddRange(Directory.GetFiles(My.Settings.strHey0ModFolder, Filter, SearchOption.TopDirectoryOnly))
Next

' Zip the Files
Using Output As ZipOutputStream = New ZipOutputStream(BackupFilename)
     Output.CompressionLevel = Zlib.CompressionLevel.Default
Output.EnableZip64 = Zip64Option.Always Output.ParallelDeflateThreshold = 0 For Each inputFile As String In FilesToZip Console.WriteLine(inputFile) Output.PutNextEntry(inputFile) Using Input As FileStream = File.Open(inputFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite) Dim n As Integer Dim buffer As Byte() = New Byte(2048) {} Do While (n = Input.Read(buffer, 0, buffer.Length) > 0) Output.Write(buffer, 0, n) Loop End Using Next End Using
Coordinator
Nov 30, 2010 at 10:21 PM

I don't know what the problem is; it could be a bug in the library.

But using parallel deflation in your scenario won't speed things up. Parallel deflation will speed things up if you have files that are larger than... say, 32k or 64k at a minimum. This is because this is the default size of the moving window in the DEFLATE implementation in the DotNetZip library.  If you have lots of "small" files, where small means less than 16k, then using parallel deflation will be detrimental to performance. I think this is all explained in the documentation.

Check it out.

 

Dec 1, 2010 at 1:20 AM

These are small files and LOTS of them...  The folder I'm compressing contains:
72,487 files & 4,161 folders -- Yes you read the correctly.    The files are ~3-4kb each

As I couldn't get the above code to work, using this until I can find a better solution:

 


 

 

        ' Create Zip of World Files
        Using ZipF As New ZipFile(ZipFileName)
            ZipF.CompressionLevel = Zlib.CompressionLevel.Default
            ZipF.UseZip64WhenSaving = Zip64Option.AsNecessary
            ZipF.ParallelDeflateThreshold = 0
            ZipF.BufferSize = 4096
            ZipF.AddDirectory(ServerWorldPath, ServerWorldName)
            ZipF.AddSelectedFiles("*.txt", My.Settings.strHey0ModFolder, "", False)
            ZipF.AddSelectedFiles("*.properties", My.Settings.strHey0ModFolder, "", False)
            ZipF.Save()
        End Using

        ' Print File Info
        Dim fi As New FileInfo(ZipFileName)
        stdIn.WriteLine("say * Zipping Finished - Size: " & Math.Round(fi.Length / 1024 ^ 2, 0, MidpointRounding.ToEven) & " Mb *")

 
Now I would love to get more threads working to speed things up, I only see 20-25% CPU usage when it is zipping. Now with 7zip using Deflate64 & 4 threads it uses 100% and finishes very quickly!

DotNetZip (1 Thread): 81 seconds
7zip (4 Threads): 14 seconds

 

Coordinator
Dec 3, 2010 at 3:28 PM

yep, there's a big opportunity for efficiency if I parallelize the zipping on a file-by-file basis.

In v1.9, I created a parallel stream that can deflate large files much faster. This doesn't help when there are many many files.  

That's something I had hoped to address in v2.0.  For now I don't have any good options for you, to decrease that 81 seconds.