2
Vote

Mac OSX Archive Utility can not recognize/unzip streamed zip files

description

If Zip64Option.AsNecessary (or Zip64Option.Always but not Zip64Option.Never) is used when streaming a zip file (to a non-seekable stream), the generated zip can not be unzipped with Mac OSX Archive Utility. A .cpgz file is created instead as Archive Utility thinks the zip is a corrupted or uncompressed file so it tries to recompress it. However the same zip file can be opened/unzipped on Windows so it's actually not corrupted. There is no problem with a zip file if it's written to a seekable stream even if Zip64 is turned on. I guess it should be about the "bit 3, streamed output" which is active when a non-seekable stream is used. I see this was discussed before here:
 
http://dotnetzip.codeplex.com/discussions/59740
 
I have tested on Mac OSX 10.5.5 (purposely on relatively an old version), here are my findings:
 
  • If I use Safari (v3.1.2), there is no problem with the streamed zip file. Safari automatically extracts the streamed zip to the Downloads folder when the download is completed. Safari seems to have a built-in zip module which doesn’t cause any problems.
     
  • If I use Firefox (v3.6.13), the streamed zip is saved and there is no automatic extraction. When I try to open the downloaded zip with “Archive Utility”, it doesn’t extract and instead creates a .cpgz file.
     
  • The problem is obviously with Archive Utility which cannot understand this kind of zip files. I have not tested with recent versions of Mac OSX so maybe newer versions of Archive Utility on these systems will not cause the problem. However I need to be compatible with all platforms so I need to find a way to make these generated zip files work out of the box without requiring the user to install an advanced/updated unzip utility.
     
    I know that you suggested MemoryStream or FileStream workarounds to some people before but they are not much suitable for web applications where memory and disk space is important. Also with those methods, the user will have to wait for the zip generation to be completed before starting the download. However I need to stream the zip on the fly so that the user immediately sees the browser's "Open/Save As" dialog. Also I need to do it without turning off Zip64 feature so that I can allow zip files larger than 4GB.
     
    I have found that there was a similar problem with the other zip library SharpZibLib and suggested fix was setting entry size before writing to the stream as discussed here:
     
    http://igorbrejc.net/development/c/sharpziplib-making-it-work-for-linuxmac
     
    I tried to apply the similar fix to DotNetZip. I first made read-only property ZipEntry.UncompressedSize settable and tried to set it to the actual file size in my code like this:
     
    ZipFile zipFile = new ZipFile();
    zipFile.UseZip64WhenSaving = Zip64Option.AsNecessary;
    zipFile.AlternateEncoding = Encoding.UTF8;
    zipFile.AlternateEncodingUsage = ZipOption.AsNecessary;
     
    ZipEntry zipEntry = zipFile.AddFile(itemPath, basePath);
    zipEntry.UncompressedSize = fileInfo.Length;
zipFile.Save(zipFileStream);
 
However this didn't have an effect. Later I fiddled with ZipEntry.Write.cs and tried to change some entry header values like CompressedSize, UnCompressedSize and Crc with no results. I think it must be about these 3 values set in header or in descriptor following the file data. When I use Zip64Option.Never, the file is ok. In my tests, I am zipping small files, I mean no big files to even require Zip64. However whether I use Zip64Option.AsNecessary or Zip64Option.Always, the generated zip causes the same problem.
 
Can you figure out what maybe be confusing utilities like Archive Utility in this specific case?

comments

gtdev wrote Feb 2, 2012 at 5:17 AM

I did further testings and it turns out there is nothing wrong with DotNetZip.

I first saw the below comment in ZipEntry.Write.cs:
        // There is one additional wrinkle: using zip64 as necessary conflicts
        // with output to non-seekable devices.  The header is emitted and
        // must indicate whether zip64 is in use, before we know if zip64 is
        // necessary.  Because there is no seeking, the header can never be
        // changed.  Therefore, on non-seekable devices,
        // Zip64Option.AsNecessary is the same as Zip64Option.Always.
So even I use Zip64Option.AsNecessary with a non-seekable stream, a zip64 file was being generated.
That's why the problem had to be about zip64 rather than the non-seekable stream.
I created a seekable zip64 file and Mac OS X Archive Utility couldn't recognize it too so this confirmed it's not about non-seekable streams. The problem is simply Archive Utility can not unzip zip64 files whether they are seekable or non-seekable. I found this link:

http://superuser.com/questions/114011/extract-large-zip-file-50-gb-on-mac-os-x

This post from the page confirmed my findings:

"After some poking around, the problem turns out to be that OS X ships with v3.0 of Info-Zip's zip, which supports the Zip64 extensions and uses them automatically when needed (so I was creating Zip64 files without realizing it), but only v5.52 of Info-Zip's unzip, which doesn't support Zip64 - Zip64 support was added in unzip's v6.0."

In short, maybe I should use another archive format like Tar instead for Mac platforms.

dave000 wrote Jun 15, 2012 at 1:03 PM

I use it this way:
myZip.UseZip64WhenSaving = _file_size >= ZIP64_SIZE_LIMIT || file_count > ZIP64_COUNT_LIMIT ? Zip64Option.AsNecessary : Zip64Option.Never;

where ZIP64_SIZE_LIMIT = 4GB, and ZIP64_COUNT_LIMIT is 65k

Sign in to add a comment or to set email notifications