7

Closed

Splitting Large (~1gb+) files leads to an exception

description

I'm running into an issue when taking a large file and splitting it.
 
When the file gets close to 1gb or over (I've tested on a 980mb file and a 2.5gb file), splitting the zip file doesn't seem to work. I get an exception as follows:
 
{"Could not find file 'C:\ziptest\zipped\SQL Server Developer 2005.z01'.":"C:\ziptest\zipped\SQL Server Developer 2005.z01"}
 
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at Ionic.Zip.ZipSegmentedStream.TruncateBackward(UInt32 diskNumber, Int64 offset)
at Ionic.Zip.ZipEntry.Write(Stream s)
at Ionic.Zip.ZipFile.Save()
at Ionic.Zip.ZipFile.Save(String fileName)
at DBBackupUploader.Program.ProcessBackupDir(String strDBBackupDir, String strZipDir, String strProcessedDir) in C:\Users\bholt\Documents\Visual Studio 2005\Projects\DBBackupUploader\DBBackupUploader\Program.cs:line 188
 

 
If I watch it when its going, the file is created in the directory (along with a tmp file). It churns away for a bit and then produces that error message. The files also disappear from the directory.
 
If I remove the MaxOutputSegmentSize line, it works just fine on all the large files. Splitting works fine for me on small files.
 
Changing the MaxOutputSegmentSize value seems to make no difference. So it seems to me like it is a combo of large files and splitting.
 

 
Code:
 
zip.AddFile(strFilePath);zip.Comment = "This zip was created at " + System.DateTime.Now.ToString("G");zip.MaxOutputSegmentSize = 500000 * 1024; string strOutputFileName = fiFileToProcess.Name.Substring(0, fiFileToProcess.Name.Length - 3);zip.Save(strZipDir + "\" + strOutputFileName + "zip");
 
Any ideas?

file attachments

Closed Jun 18, 2011 at 4:28 AM by Cheeso
fixed in changeset 79356. Sorry for the delay, everyone. I know you've been waiting a while. The first binary to contain this fix will be v1.9.1.6.

comments

Protoman wrote Feb 3, 2010 at 2:29 PM

Stack Trace:
" at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.File.Move(String sourceFileName, String destFileName)\r\n at Ionic.Zip.ZipSegmentedStream.TruncateBackward(UInt32 diskNumber, Int64 offset)\r\n at Ionic.Zip.ZipEntry.Write(Stream s)\r\n at Ionic.Zip.ZipFile.Save()\r\n at Ionic.Zip.ZipFile.Save(String fileName)\r\n at DBBackupUploader.Program.ProcessBackupDir(String strDBBackupDir, String strZipDir, String strProcessedDir) in C:\Users\bholt\Documents\Visual Studio 2005\Projects\DBBackupUploader\DBBackupUploader\Program.cs:line 188" string

Can you replicate?

Protoman wrote Feb 3, 2010 at 2:34 PM

using version : 1.9.1.2

Cheeso wrote Feb 3, 2010 at 9:32 PM

I'll look into it.

Protoman wrote Feb 12, 2010 at 2:18 PM

Just curious if you've had a opportunity to look into this and if you can replicate. Let me know if you need anything.

Cheeso wrote Feb 12, 2010 at 9:54 PM

Protoman - I took a few days off - our power and internet was off for several days after a snowstorm. Catching up now. I'm running tests, but have not been able to replicate this problem yet. On my machine it takes a while to generate a zip of that size - an hour or more. Then if I don't get the problem, I tweak the test a little, and run the test again. As you can see, each iteration takes quite a while. I will let you know, on this workitem, if I need anything.

Cheeso wrote Feb 13, 2010 at 3:18 PM

I haven't been able to reproduce the problem. Is there anything more about your environment you can tell me? Is it possible that you ran out of disk space? How many files are you zipping? When you say "the file gets oer 1gb" - which file is that? The agrgegate size of the zip file? Or.... the size of the file you are zipping? etc etc.

Cheeso wrote Feb 15, 2010 at 8:06 PM

Still cannot reproduce this problem. Need some guidance.

Protoman wrote Feb 16, 2010 at 2:29 PM

I'm zipping just one file, it's 920,000 KB. Lots of space on my drive, over 680 GB free.

Any files larger then that fail, a small file (30mb) work fine. Will test some files with sizes in the middle to try and figure out where it starts to fail.

Also, it only fails when trying to create a spanned/split archive. The set size of each segment doesn't seem to matter. I've tried it at 500,000 KB and 5,000 KB and it fails just the same on both. One thing that is interesting though, when I set the segment size to 5,000 KB and watch the output directory I can see it creating each segment. It seems to make all the segments ok (when I set it to 5,000 KB it makes ~184 zips), but it fails right after it makes the last one. The zips are then removed automatically it seems.

Do you want more of my code?

Cheeso wrote Feb 17, 2010 at 3:30 AM

If you had a succint app that reproduces the error reliably, it would be useful. I tried following your instructions and could not reproduce it.

Protoman wrote Feb 17, 2010 at 4:02 PM

I made a little app that reproduces the error.

Need a directory called C:\ziptest
with sub directories : \processed and \zipped

file to zip goes in \ziptest

I've narrowed the file size down : 717mb seems to work while 740mb does not.

Protoman wrote Feb 17, 2010 at 7:49 PM

also, checking the events of ZipProgressEventType during the save, it hits the error before getting to "Saving_AfterWriteEntry".

Cheeso wrote Feb 17, 2010 at 9:35 PM

Can you upload the source code to a standalone app that produces the problem? When run on a clean system, it would produce the problem you see. This might include: creating directories and subdirectories, creating files or placing the files into those directories, and zipping up the directory or files.

I have tried repeatedly to produce this problem. I know you see the problem. That hasn't helped me reproduce it. I need something more concrete from you. I need a repro.

Protoman wrote Feb 17, 2010 at 9:42 PM

the source code should all be there in that zip file that i posted.

ZiadJ wrote Feb 25, 2010 at 5:08 PM

I'm getting exactly the same behavior as well. Here is the test code I'm using:
        using (ZipFile zip = new ZipFile())
        {
            ZipEntry e1 = zip.AddFile("c:\\Output\\Movie253.avi");
            zip.MaxOutputSegmentSize = 50 * 1024 * 1024;
            try
            {
                zip.Save("c:\\ServerCookies\\test.zip");
            }
            catch (Exception Exception)
            {
                MessageBox.Show(Exception.Message);
            }
        }
In my case it works fine for a 195MB file but fails for a 253MB file. The files are created successfully till it reaches the last one at which point all created files are deleted and I get the following error:
Could not find file 'c:\Output\Movie253.z01'.

Stack Trace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at Ionic.Zip.ZipSegmentedStream.TruncateBackward(UInt32 diskNumber, Int64 offset) in c:\dinoch\dev\dotnet\zip\DotNetZip\Zip Partial DLL\ZipSegmentedStream.cs:line 364
at Ionic.Zip.ZipEntry.Write(Stream s) in c:\dinoch\dev\dotnet\zip\DotNetZip\Zip Partial DLL\ZipEntry.Write.cs:line 1914
at Ionic.Zip.ZipFile.Save() in c:\dinoch\dev\dotnet\zip\DotNetZip\Zip Partial DLL\ZipFile.Save.cs:line 123
at Ionic.Zip.ZipFile.Save(String fileName) in c:\dinoch\dev\dotnet\zip\DotNetZip\Zip Partial DLL\ZipFile.Save.cs:line 371
....

Im using Version 1.9.1.2 on VS 2010.

ZiadJ wrote Feb 25, 2010 at 5:23 PM

Btw a friend of mine reported the same error while trying to compress a 100MB file!! As for me I have 3.4GB free on my primary hard disk with 1GB free memory and the operation took ~40s to complete.
Code:
using (ZipFile zip = new ZipFile())
{
ZipEntry e1 = zip.AddFile("c:\Output\Movie253.avi");
zip.MaxOutputSegmentSize = 50 * 1024 * 1024;
try
{
zip.Save("c:\Output\test.zip");
}
catch (Exception Exception)
{
MessageBox.Show(Exception.Message);
}
}

Cheeso wrote Feb 26, 2010 at 3:36 AM

I'll try again.... so far, no luck reproducing it after many trials.

Protoman wrote Feb 26, 2010 at 3:27 PM

So my app works fine on your system for files > 750mb?

Another thing that I noticed that's happening in the output directory while the program is running:
You can see it making the z01, z02, z03 files and there is also tmp file (ex: DotNetZip-xxxxxx.tmp) that I am assuming is used to build each of the segments (which are then renamed to .zXX files).
After the exception is thrown and the zXX files disappear from the output directory, I have noticed that a DotNetZip-xxxxxx.tmp file appears in the running directory (ex: bin\Debug). The file size is the same of the max segment size.

Protoman wrote Feb 26, 2010 at 7:57 PM

I made a file watch to watch the directory, log results are below:

26/02/2010 2:20:06 PM - Created: C:\ziptest\zipped\DotNetZip-riatlxn0.tmp
26/02/2010 2:20:06 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:16 PM - Changed: C:\ziptest\zipped\DotNetZip-riatlxn0.tmp
26/02/2010 2:20:16 PM - Renamed: C:\ziptest\zipped\DotNetZip-riatlxn0.tmp renamed to C:\ziptest\zipped\ziptest.z01
26/02/2010 2:20:16 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:16 PM - Created: C:\ziptest\zipped\DotNetZip-wb4jcsnj.tmp
26/02/2010 2:20:16 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:24 PM - Changed: C:\ziptest\zipped\DotNetZip-wb4jcsnj.tmp
26/02/2010 2:20:24 PM - Renamed: C:\ziptest\zipped\DotNetZip-wb4jcsnj.tmp renamed to C:\ziptest\zipped\ziptest.z02
26/02/2010 2:20:24 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:24 PM - Created: C:\ziptest\zipped\DotNetZip-jdwrucqm.tmp
26/02/2010 2:20:24 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:32 PM - Changed: C:\ziptest\zipped\DotNetZip-jdwrucqm.tmp
26/02/2010 2:20:32 PM - Renamed: C:\ziptest\zipped\DotNetZip-jdwrucqm.tmp renamed to C:\ziptest\zipped\ziptest.z03
26/02/2010 2:20:32 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:32 PM - Created: C:\ziptest\zipped\DotNetZip-zrp1apoy.tmp
26/02/2010 2:20:32 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:39 PM - Changed: C:\ziptest\zipped\DotNetZip-zrp1apoy.tmp
26/02/2010 2:20:39 PM - Renamed: C:\ziptest\zipped\DotNetZip-zrp1apoy.tmp renamed to C:\ziptest\zipped\ziptest.z04
26/02/2010 2:20:39 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:39 PM - Created: C:\ziptest\zipped\DotNetZip-w3g3l1ip.tmp
26/02/2010 2:20:39 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:47 PM - Changed: C:\ziptest\zipped\DotNetZip-w3g3l1ip.tmp
26/02/2010 2:20:47 PM - Renamed: C:\ziptest\zipped\DotNetZip-w3g3l1ip.tmp renamed to C:\ziptest\zipped\ziptest.z05
26/02/2010 2:20:47 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:47 PM - Created: C:\ziptest\zipped\DotNetZip-hdczibt0.tmp
26/02/2010 2:20:47 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:54 PM - Changed: C:\ziptest\zipped\DotNetZip-hdczibt0.tmp
26/02/2010 2:20:54 PM - Renamed: C:\ziptest\zipped\DotNetZip-hdczibt0.tmp renamed to C:\ziptest\zipped\ziptest.z06
26/02/2010 2:20:54 PM - Changed: C:\ziptest\zipped
26/02/2010 2:20:54 PM - Created: C:\ziptest\zipped\DotNetZip-atxgob2w.tmp
26/02/2010 2:20:54 PM - Changed: C:\ziptest\zipped
26/02/2010 2:21:02 PM - Changed: C:\ziptest\zipped\DotNetZip-atxgob2w.tmp
26/02/2010 2:21:02 PM - Renamed: C:\ziptest\zipped\DotNetZip-atxgob2w.tmp renamed to C:\ziptest\zipped\ziptest.z07
26/02/2010 2:21:02 PM - Changed: C:\ziptest\zipped
26/02/2010 2:21:02 PM - Created: C:\ziptest\zipped\DotNetZip-hvr4zugj.tmp
26/02/2010 2:21:02 PM - Changed: C:\ziptest\zipped
26/02/2010 2:21:05 PM - Changed: C:\ziptest\zipped\ziptest.z01
26/02/2010 2:21:05 PM - Changed: C:\ziptest\zipped\DotNetZip-hvr4zugj.tmp
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\DotNetZip-hvr4zugj.tmp
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z07
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z06
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z05
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z04
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z03
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z02
26/02/2010 2:21:05 PM - Deleted: C:\ziptest\zipped\ziptest.z01

26/02/2010 2:21:06 PM - Changed: C:\ziptest\zipped

One thing that looks out of place is "Changed: C:\ziptest\zipped\ziptest.z01" right before the deletes start. Why would that file be changed at that point in the process?

Protoman wrote Feb 26, 2010 at 8:59 PM

I found another FileWatcher that shows things in more detail
(http://www.codeproject.com/KB/files/FileSystemWatcherChaos1.aspx)

What causes the change event to fire on that z01 file is : Attributes

So something is causing the attributes to change I guess...

ZiadJ wrote Feb 27, 2010 at 9:29 AM

I partly fix the problem by commenting line 1883 in ZipEntry.Write.cs:
zss.TruncateBackward(_diskNumber, _RelativeOffsetOfLocalHeader);
This function is being called with _diskNumber having a value of zero. This deleted all the created files in question. The output files now seem to be valid but I get twice as many files as expected. The last file has the extension z11 instead of the expected z05 but after I deleted files z01 to z05 the windows zip extractor yielded a valid file! I will investigate further in this issue when I find the time...Any ideas meanwhile?

ZiadJ wrote Feb 27, 2010 at 10:03 AM

Could it be due to the fact that I'm using a 64bit version of Windows? Anyway I thought you might want to get into my PC remotely via, say, TeamViewer since you are not being able to reproduce the problem over there. I'd be more than glad to let you in if that's the easiest way to go.

Protoman wrote Mar 1, 2010 at 2:17 PM

I'm also using a 64bit version of windows (win7 pro)

johnkel2000 wrote Mar 8, 2010 at 10:24 PM

Hello,

I am having the same problem (I think). But I am seeing it with relatively small files; JPGS of about 1M or so. I am attempting to create a split archive of a bunch of JPG files. I get the following stack trace:

System.IO.FileNotFoundException: Could not find file 'c:\image\resize\hugephoto\hugephoto_ywszej4l.z01'.
File name: 'c:\image\resize\hugephoto\hugephoto_ywszej4l.z01'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at Ionic.Zip.ZipSegmentedStream.TruncateBackward(UInt32 diskNumber, Int64 offset) in c:\DotNetZip\v1.9.1.4\DotNetZip\Zip Partial DLL\ZipSegmentedStream.cs:line 368
at Ionic.Zip.ZipEntry.Write(Stream s) in c:\DotNetZip\v1.9.1.4\DotNetZip\Zip Partial DLL\ZipEntry.Write.cs:line 1912
at Ionic.Zip.ZipFile.Save() in c:\DotNetZip\v1.9.1.4\DotNetZip\Zip Partial DLL\ZipFile.Save.cs:line 123
at Ionic.Zip.ZipFile.Save(String fileName) in c:\DotNetZip\v1.9.1.4\DotNetZip\Zip Partial DLL\ZipFile.Save.cs:line 371
at adppu.CompressedArchive_Filter.Process(WorkItem input) in C:\projects\ADPhotoProcessor\trunk\CompressedArchive_Filter\CompressedArchive_Filter.cs:line 66

The code is pretty standard, being taken largely from the spanning example:

snip...
ZipFile zf = new ZipFile();
        zf.MaxOutputSegmentSize = max_zipfile_segment_size;

        try
        {
            foreach (var f in raw)
            {
                zf.AddFile(f.FullName);
            }

            zf.Save(fullzippath);
        }
        catch (Exception e)
snip...

Has anyone made any progress on this?

Fitzlestick wrote Mar 16, 2010 at 3:07 PM

I am experiencing the same problem with splitted zipfiles.

I'm trying to zip 11 files, totaling about 3.98mb.
When zipped to 1 file it's about 1.15mb.
I try to zip it to parts of 1mb maximum when I get the error.

Code:
    Dim zip1 As ZipFile = New ZipFile
    zip1.AddDirectory("c:\temp")
    zip1.MaxOutputSegmentSize = 1*1024*1024
    zip1.Save("c:\zipit.zip")
Error:
"System.IO.FileNotFoundException: Could not find file 'c:\zipit.z01'.
File name: 'c:\zipit.z01'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at Ionic.Zip.ZipSegmentedStream.TruncateBackward(UInt32 diskNumber, Int64 offset)
at Ionic.Zip.ZipEntry.Write(Stream s)
at Ionic.Zip.ZipFile.Save()
at Ionic.Zip.ZipFile.Save(String fileName)
at WindowsApplication1.Form1.Button2_Click(Object sender, EventArgs e) in C:\DOTNET\Projects\WinForms-DotNetZip\WinForms-ZipUp\ZipUp-Form1.vb:line 46"

miliu99 wrote Apr 18, 2010 at 7:19 PM

Any progress on this issue? I got the same error when I try to split into 1.2 MB files. However, the same source works with 1.44 MB files without problems. My total zip file is only 2.12 MB.

miliu99 wrote Apr 18, 2010 at 7:23 PM

Any progress on this issue? I got the same error when I try to split into 1.2 MB files. However, the same source works with 1.44 MB files without problems. My total zip file is only 2.12 MB.

lxitixl wrote Jun 10, 2010 at 9:51 AM

I have the exact same problem, zipping a directory of 104mb and splitting into any file size except 32MB will cause an exception.
It happens whether I'm using Zip64 or not.
Note that I am using Intel core 2 extreme (quad core).

Robert_Novak wrote Aug 24, 2010 at 9:01 PM

I have the same issue and I tried to download your source code and little investigate it and it looks that there is problem with TruncateBackward method in ZipSegmentedStream class.
// get a new temp file, try 3 times:
        for (int i = 0; i < 3; i++)
        {
            try
            {
                _currentTempName = SharedUtilities.InternalGetTempFileName();
                File.Move(CurrentName, _currentTempName);  // move the .z0x file back to a temp name
                break; // possible fix of this issue
            }
            catch(IOException)
            {
                if (i == 2) throw;
            }
        }

phoog wrote Nov 30, 2010 at 9:37 PM

This code is triggering the exception:

public void CreateMultipartArchive(DirectoryInfo parentDirectory)
    {
        var sourceFile = CreateRandomFileInfo(parentDirectory, 0x1000000, 42);

        var zipFilePath = Path.Combine(parentDirectory.FullName, "test.zip");

        using (var zipFile = new ZipFile(zipFilePath))
        {
            zipFile.AddFiles(new [] { sourceFile.FullName }, false, string.Empty);
            zipFile.MaxOutputSegmentSize = 65536;
            zipFile.Save();
        }
    }

    private static FileInfo CreateRandomFileInfo(DirectoryInfo parentDirectory, int lengthOfRandomContent, int randomSeed)
    {
        var folderPath = parentDirectory == null ? Path.GetTempPath() : parentDirectory.FullName;

        var newName = Path.Combine(folderPath, Path.GetRandomFileName());
        var result = new FileInfo(newName);

        const int BUFFER_SIZE = 0x4000;
        var bytesWritten = 0;

        using (var stream = result.OpenWrite())
        {
            var random = new Random(randomSeed);
            var buffer = new byte[BUFFER_SIZE];

            for (var i = BUFFER_SIZE; i < lengthOfRandomContent; i += BUFFER_SIZE)
            {
                random.NextBytes(buffer);
                stream.Write(buffer, 0, BUFFER_SIZE);
                bytesWritten += BUFFER_SIZE;
            }

            var bytesRemaining = lengthOfRandomContent - bytesWritten;

            buffer = new byte[bytesRemaining];
            random.NextBytes(buffer);
            stream.Write(buffer, 0, bytesRemaining);

            stream.Close();
        }

        return result;
    }

Cheeso wrote Jun 18, 2011 at 4:26 AM

This is a real problem; I've reproduced this (and others similar) here, and I've now posted changes to the ZipSegmentedStream.cs and callers of that class, to correct these problems. I had tests in the test tree to track this, and I believe they all passed, but on re-running the tests with my new copy of vS2010, many of those tests failed in exactly the way described here, and in other bugs.

This will be fixed in v1.9.1.6 of the library.
Coming soon.