Error when unzipping folder on the Mac

Jun 17, 2009 at 1:42 AM

I am using DotNetZip to dynamically generate zip music for download, this works great for windows users,however, mac users are getting an error when they tried to open the zip folder.   

below is the code that I am using,

 Try
            Dim i As Integer
          
            Response.Clear()
            Dim archiveName As String
            archiveName = String.Format("HHSongZip-{0}.zip", DateTime.Now.ToString("yyy-MMM-dd"))
            Context.Response.ContentType = "application/x-zip-compressed"
            Context.Response.ContentEncoding = Encoding.Default
            Context.Response.Charset = ""
            'Response.ContentType = "application/zip"
            Response.AddHeader("content-disposition", "filename=" + archiveName)

            Using zip As ZipFile = New ZipFile
                For i = 0 To ds.Tables(0).Rows.Count - 1
                    If type = "MP3:320" Then
                        zip.AddFile("C:\samples_full\mp3_320\" + ds.Tables(0).Rows(i)("Path").ToString(), "MP3")
                    Else
                        zip.AddFile("C:\samples_full\aif\" + ds.Tables(0).Rows(i)("Path").ToString(), "AIF")
                    End If

                Next
                 zip.Save(Response.OutputStream)

            End Using
        Catch ex As Exception
            Response.Write(ex.Message.ToString())
        End Try

Coordinator
Jun 17, 2009 at 2:05 PM

Can you insert a Response.End() right after the End Using?  (before the Catch) ?

 

Jun 18, 2009 at 6:10 PM

Thanks, Cheeso for your response, unfortunaly your suggestion did not work.  I test it out on a Mac Machine, and it fail to open the zip file.  Note, this works on a windows box.

Coordinator
Jun 18, 2009 at 8:52 PM

Ok, next idea:  Check your MIME type mappings on the browser on the MAC.

Why are you using application/x-zip-compressed instead of application/zip, anyway?  You may want to try both.

 

Jun 20, 2009 at 2:33 AM

Chesso, I  was originally using  the MIME { application/zip }, I figure, that if change the MIME to { application/x-zip-compressed}  but issue did not go away.  I tried SharpZipLib to zip a few file, and it work fine on the MAC.   The only problem with SharpZipLib if requires more coding to generated zip files on the fly.  I really like how the DotNetZip library easily accomplish this. 

Any other suggestions.

Coordinator
Jun 20, 2009 at 6:31 PM

If I were you I might try...

  1. attaching a tracing HTTP proxy to the exchange. Whatever can run on the Mac. The equivalent for Windows+IE would be Fiddler. Looking at the trace and comparing to the exchange for windows, see if there is a difference.
  2. downloading the zip rather than clicking on "open". Is the zip still invalid?
  3. downloading a static zip file, rather than one dynamically produced. IS the static zip file valid? If it succeeds, again compare the trace between the successful scenario and the one that doesn't work.
  4. download some other type of binary file. Again, look at the traces. See if the file is valid when downloaded.

In the trace, look for headers that differ: caching, content-type, content-length, and any negotiation that might affect the HTTP exchange. Even the user-agent might make a difference.

This is just off the top of my head. I would focus on the transaction between the Mac browser and the server, because you have already told me that you know the code on the server works with a Windows/IE browser.

Jul 15, 2009 at 4:25 PM

Did you ever find a solution to unarchiving files on the Mac? I have run into this issues as well. The code I am using is below:


            Response.Clear();
            string zipFileName = String.Format("{0}.zip", lblTATitle.Text.Replace(' ', '_'));
            Response.ContentType = "application/zip";
            Response.AddHeader("content-disposition", "filename=" + zipFileName);

            using (ZipFile zip = new ZipFile())
            {
                zip.TempFileFolder = Server.MapPath("~/PublishedFiles/");
                zip.AddDirectory(Server.MapPath("~/PublishedFiles/temp_ta_" + taId));
                zip.Save(Response.OutputStream);
            }

            // After zip is sent to user, delete temp files
            System.IO.Directory.Delete(Server.MapPath(tempdir), true);

            Response.End();

Coordinator
Jul 15, 2009 at 6:52 PM

#1, don't use Response.End().  This is often seen in code and examples, but in general it's not the right thing to do.  See http://support.microsoft.com/kb/312629 .   Swap out Response.End() and insert  HttpContext.Current.ApplicationInstance.CompleteRequest().

#2, I learned that "application/zip" is a good content-type if the browser is IE.  Otherwise you should try something else, like perhaps application/octet-stream". 

You will have to check User-Agent to learn if IE is the requesting browser.

Try that and see.

 

Jul 15, 2009 at 8:13 PM

Cheeso, thanks for your quick response. I tried your suggestions but still get the same error. I am able to download the zip file on the Mac, but Stuffit cannot open it. I am able to open using Terminal but that is not really an acceptable solution for our users. Also, if I copy the downloaded zip file to Windows, I am able to unzip with no problems. Any other thoughts? Thanks so much for your help!

Jul 15, 2009 at 8:23 PM

Also, the Archive Utility on the Mac does not support password protected zip files. Could this pose a problem even though I am not trying to password protect my zip files?

Jul 15, 2009 at 8:45 PM

Another find:


From:

http://forums.macrumors.com/showthread.php?t=144786

I ran into this problem tonight and tracked it down to Archive Utility. If you ask it to extract a zip file containing a member with a name of the form "foo" rather than "dirname/foo", you will get the message "Error 1 - Operation not permitted". This is true even if you have write permission on the current directory. Unzip from the command line doesn't have this problem, so that's a workaround.

While it's bad form for an archive to extract into the current directory, polluting whatever may already be there, it's also bad form for a program to bomb out with such a cryptic and misleading error message. It should tell you what's going on and ask you to approve the extraction or have you specify another folder.


 

This could be causing the issue since the zip archive does not seem to be unarchiving into a folder. Any ideas on how to accomplish this?

Coordinator
Jul 16, 2009 at 6:20 AM

Sorry, I'm not much help with this.  I don't know Archive Utility, Terminal or any of those other things.

The ZIP you download to Windows is valid, right?  What if you transfer that to the Mac?   Can you open it?  I guess what I am suggesting is, continue testing different scenarios until you can eliminate some things as causes.  If you can just copy over a zip file from Windows to Mac, and the zip file still does not open, then it's not the transfer.  If it opens fine, then maybe it's the transfer/dwnload.

what if you try the other way?  copy the downloaded file from the Mac to Windows?

etc

I don't have a mac so I cannot help test this.

Jul 16, 2009 at 1:33 PM

I've narrowed it down to the zip file itself. The transfer is working correctly, if I copy it to a Windows machine, it will unzip with no problems. Terminal is a command line tool on the Mac - I am able to extract the zip file using this program, but most of my users won't know how to use it since it is command line. Also, I downloaded the third party Stuffit Expander 2009 for the Mac and it will also unzip the downloaded zip file. For now, this is a somewhat acceptable solution, but I would like the users to be able to unzip with the default zip program on the Mac which is Archive Utility.

I think I've narrow down the problem to be with how the zip file extracts itself. For example, if a regular zip file is named Sample.zip, it will usually unzip to a folder with the same name. So if sample.zip contains help.html and logo.gif the extracted contents will go to a folder called Sample (Sample/help.html and Sample/logo.gif).  When I extract the DotNetZip zip file, it does not put the contents into a new folder. The archive extracts into the current directory (same directory as the zip file) which the Mac does not allow. It's like the zip needs to say something like extract to 'ThisParticular' folder.

This is where my knowledge stops. I'm assuming it's an issue with how the zip file is created, but I'm not sure if there is any way to fix it. Any thoughts?

Thanks!!

Coordinator
Jul 16, 2009 at 7:04 PM

Ahh, terrific.  OK.

You can affect the folder used internally in the zip file by adding a string param when you call AddDirectory(). 

So,

    AddDirectory(Server.MapPath("whatever"), "dirUsedInZipFile");

Is it a rule that the folder used inside the zip file must match the zipfile name? This is the convention on Windows for "compressed folders" but not sure if it is a rule. Anyway, you will figure it out now, I'm sure.

 

Jul 20, 2009 at 6:41 PM

Still having issues :(

I seem to be having trouble with folders in the zip file now. Everytime I try to unzip a file that contains folders inside, I get the following Stuffit error: The structure of the archive is damaged (Error #17540)

This does not occur if the zip archive contains only files, not folders.

Any more suggestions?

Coordinator
Jul 20, 2009 at 7:50 PM

I don't know what you mean by "only files, not folders". 

Can you open these zipfiles on windows with "compressed folders" or with the DotNetZip gui tool?  Or IZArc or any other common zip tool?  If so, then it seems to me that Stuffit cannot process any general zip file.  If the zip file can be opened by other tools, but not by Stuffit, then it seems clear that Stuffit places special restrictions on a zipfile.  One of them might be, the required folder names.

If I were doing this I would figure out just exactly what Stuffit expects in a zip file, either by reading technical documents or just by examining zipfiles that stuffit produces.  And then I'd figure out how to replicate that structure with DotNetZip.

 

Jul 20, 2009 at 8:59 PM

Yes, I can open these zip files on a pc. I cannot open them on the Mac. I have tried 2 different programs on the mac which are the default Archive Utility and Stuffit - both programs will not open the zip file.

I meant to say I'm having trouble with subfolders: if I use the zip.AddDirectory("samplefolder") function, it will add any files and any subfolders found in the "samplefolder" directory to the zip file. I am having a problem opening the zip files if the "samplefolder" contains subfolders. If the directory I'm trying to add contains subfolders, I will get the following error when I try to open the zip file: The structure of the archive is damaged. Again, this is only on the mac.

 

Coordinator
Jul 20, 2009 at 9:58 PM

ok - well I think this is the same problem we just covered. ??   I suggested then that you use the overload of AddDirectory that allows you to specify the name of a folder to use within the archive.  Did you try that?  The code snip you showed is the AddDirectory method that accepts a single argument.  Did we just cover this?

I have another hint - if you use the unzip command line utility that is included in the DotNetZip utils download, you can get a look at the paths used by the entries within a zip file. The -l option to that tool will list the entries in an archive:

    unzip -l archive.zip 

I suggest you use that tool to examine the zip files that stuffit or Archive Utility can open, and then also examine the zip files that you are producing with DotNetZip. Compare the two, and modify your DotNetZip code to produce what you need for Stuffit or Archive Utility.

if you post a version of the output of the output from unzip , I can have a look at it with you.

If you don't do command line you can also use the DotNetZip Winforms tool to look at the paths. But it will be harder to post the output of that tool to let me analyze it with you. you would need to do a screenshot - which is possible but not as simple.

Jul 21, 2009 at 1:44 PM

Yes, I have already tried using the overload of AddDirectory as you suggested. I still come up with the same error when trying to unzip on the mac: The structure of the archive is damaged.

I will try your suggestion of unzipping with the command line utility to see the paths. I will post the results shortly.

Jul 21, 2009 at 4:20 PM

Here is the result of unzipping the DotNetZip-produced zip file:

Zipfile: new_employee.zip

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------

2009-07-21 06:51:12             0     0%           0    N 00000000 New_Employee/
2009-07-21 06:51:10           253    38%         158    N 5505BB2C New_Employee/ta_2.ditamap
2009-07-21 06:51:12           676    55%         304    N C950A705 New_Employee/ta_2_spo_11.dita
2009-07-21 06:51:12           697    58%         290    N 2C8A08C9 New_Employee/ta_2_spo_11_eo_18_detail.dita
2009-07-21 06:51:12             0     0%           0    N 00000000 New_Employee/assets/
2009-07-21 06:50:26         28521     0%       28521    N 245A97A2 New_Employee/assets/EO_200972165027_Bluehills.jpg
--------------------------------------------------------------------------------

                            30147                                  6 files

 

Here are the results of the same folder, zipped using the Mac's Archive utility:

Zipfile: new_employee_mac.zip

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------

2009-07-21 12:14:50             0     0%           0    N 00000000 New_Employee/
2009-07-21 12:14:50             0     0%           0    N 00000000 New_Employee/assets/
2009-07-21 06:50:26         28521     6%       26683    N 245A97A2 New_Employee/assets/EO_200972165027_Bluehills.jpg
2009-07-21 06:51:10           253    38%         158    N 5505BB2C New_Employee/ta_2.ditamap
2009-07-21 06:51:12           676    55%         304    N C950A705 New_Employee/ta_2_spo_11.dita
2009-07-21 06:51:12           697    58%         290    N 2C8A08C9 New_Employee/ta_2_spo_11_eo_18_detail.dita
--------------------------------------------------------------------------------

                            30147                                  6 files

 

 

 

Coordinator
Jul 21, 2009 at 5:10 PM

Well that is interesting.  I don't see any major differences in the two listings.  There are a few small ones:

  • the order of the entries differs.  In the MAC zip file, the directories appear first in the zip file. 
  • The timestamps on the directories in the mac zip file are apparently not set according to the timestamp on the directory in the filesystem.
  • the jpg file is not compressed in the DotNetZip version of the zip file.

You can try several things to get the dotnetzip file to look closer to the MAC-produced zip file.

  1. add the directories by name, explicitly, first, before adding any files.    New_Employee and New_Employee\assets.  Use the ZipFile.AddDirectoryByName method. If you call AddDirectory() to add the files, it's no problem to call AddDirectoryByName() first. 
  2. Set the timestamp on the directory entries explicitly.  Use the ZipEntry.LastModified property.
  3. Set the ZipFile.CompressionLevel to BestCompression to insure best compression of the .jpg.

If none of those things work to allow the Archive  Utility to read a zipfile produced by DotNetZip, then if you can post the respective files somewhere, I will have a look at the binaries to see if I can discern any further difference that way.   Sometimes a zip tool will fail on an "extra" field included in the zip header.   Unrecognized extra fields are supposed to be ignored.  Maybe the stuffit tool is not ignoring them.  DotNetZip adds in NTFS times in an extra field according to the spec.  Sometimes this extra field is not interpreted by other tools.  I've never seen a tool that refuses to open a valid zip file because of an unrecognized header, but maybe this is what Stuffit is doing.   Anyway I can have a look if you make the files available.

 

Jul 21, 2009 at 6:21 PM

 

I tried all of your suggestions above, but still no luck. I've been able to manipulate the DotNetZip file so that the output looks exactly like the Mac Archive Utility output, but I still cannot open the file in the Archive Utility or Stuffit. The only difference is the compression level for the jpg file. I set the compression level to BestCompression, but it still does not seem to be compressing it. I have also modified the code so that there are no subfolders in the directories that I add (thought this might have been causing an issue, but didn't solve it). So the 'assets' directory is on the same level as the 'temp_ta_x' directory.

I don't have a place to put the 2 different zip files, may I email them to you? Thanks for your help, it is much appreciated!!

 

Here is my code:

 

            // Create zip file and send to user
            Response.Clear();
            string zipFileName = String.Format("{0}.zip", lblTATitle.Text.Replace(' ', '_'));
            Response.ContentType = "application/zip";
            Response.AddHeader("content-disposition", "filename=" + zipFileName);

            using (ZipFile zip = new ZipFile())
            {
                zip.CompressionLevel = Ionic.Zlib.CompressionLevel.BestCompression;
                zip.AddDirectoryByName(lblTATitle.Text.Replace(' ', '_'));
                zip.AddDirectoryByName(lblTATitle.Text.Replace(' ', '_') + "/assets");
                zip.Entries[0].LastModified = DateTime.Now;
                zip.Entries[1].LastModified = DateTime.Now;
                zip.AddDirectory(Server.MapPath("~/PublishedFiles/temp_ta_" + taId + "_assets"), lblTATitle.Text.Replace(' ', '_') + "/assets");
                zip.AddDirectory(Server.MapPath("~/PublishedFiles/temp_ta_" + taId), lblTATitle.Text.Replace(' ', '_'));
                zip.Save(Response.OutputStream);
            }

            HttpContext.Current.ApplicationInstance.CompleteRequest();

            // After zip is sent to user, delete temp files
            System.IO.Directory.Delete(Server.MapPath(tempdir), true);

 

 

 

Here is the DotNetZip output:

 

Zipfile: new_employee.zip

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------

2009-07-21 14:03:54             0     0%           0    N 00000000 New_Employee/
2009-07-21 14:03:54             0     0%           0    N 00000000 New_Employee/assets/
2009-07-21 13:40:52         28521     0%       28521    N 245A97A2 New_Employee/assets/EO_2009721134053_Bluehills.jpg
2009-07-21 14:03:54           253    38%         158    N 7873F9F1 New_Employee/ta_2.ditamap
2009-07-21 14:03:54           651    56%         288    N 1F222259 New_Employee/ta_2_spo_12.dita
2009-07-21 14:03:54           698    58%         291    N DAEDD584 New_Employee/ta_2_spo_12_eo_18_detail.dita
--------------------------------------------------------------------------------

                            30123                                  6 files

 

 

 

Here is the ArchiveUtility output:

Zipfile: new_employee_mac.zip

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------

2009-07-21 14:03:10             0     0%           0    N 00000000 New_Employee/
2009-07-21 14:03:10             0     0%           0    N 00000000 New_Employee/assets/
2009-07-21 13:40:54         28521     6%       26683    N 245A97A2 New_Employee/assets/EO_2009721134053_Bluehills.jpg
2009-07-21 14:03:54           253    38%         158    N 7873F9F1 New_Employee/ta_2.ditamap
2009-07-21 14:03:54           651    56%         288    N 1F222259 New_Employee/ta_2_spo_12.dita
2009-07-21 14:03:54           698    58%         291    N DAEDD584 New_Employee/ta_2_spo_12_eo_18_detail.dita
--------------------------------------------------------------------------------

                            30123                                  6 files

Coordinator
Jul 21, 2009 at 6:31 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jul 21, 2009 at 6:33 PM

I suspect the problem may be related to one of the extra fields inserted by DotNetZip. 

Upload both files to this workitem:  http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8092

and I will have a look.

Coordinator
Jul 21, 2009 at 8:28 PM

I found what I expected.

The so-called NTFS file times are present in each entry in the zip file generated by DotNetZip, but they are not present in the zip file generated by Stuffit (or whatever you used). The file times are embedded in the "extra field" as described by the zip specification. According to the zip spec, zip applications and libraries should ignore headers they don't recognize. It looks like Stuffit is being a bit too strict, and concluding that the zipfile is corrupt, when in actual fact, stuffit is just not recognizing an extra field which is clearly documented in the spec.

The zip file produce by DotNetZip complies with the specification, and well-designed zip tools won't have a problem with any zip file from DotNetZip. Right now there is no way to tell DotNetZip to NOT include that extra field in the zip entry. Seems like DotNetZip could be a bit more flexible and optionally not embed that field. It's a simple change. I can make a updated version of the library available to you for you to try.

I also found something else in the stuffit-generated archive. Another "extra" field, you might say an alternative to the NTFS file times extra field. It is documented as outdated, and specifically,

Archiving software should recognize "Unix1" extra fields for timestamp comparison but never create it for updated, freshened or new archive members. When copying existing members to a new archive, any "Unix1" extra field blocks should be converted to the new "time" and/or "Unix2" types.

The full documentation is included below. Clearly stuffit (or whatever you used to produce the archive) is doing something against the specification. What version of stuffit are you using? If it is out of date, maybe you should consider updating?

         -Info-ZIP Unix Extra Field (type 1):
          ==================================

          The following is the layout of the old Info-ZIP extra block for
          Unix.  It has been replaced by the extended-timestamp extra block
          (0x5455) and the Unix type 2 extra block (0x7855).
          (Last Revision 970118)

          Local-header version:

          Value         Size            Description
          -----         ----            -----------
  (Unix1) 0x5855        Short           tag for this extra block type
          TSize         Short           total data size for this block
          ModTime       Long            time of last modification (UTC/GMT)
          AcTime        Long            time of last access (UTC/GMT)
          UID           Short           Unix user ID
          GID           Short           Unix group ID

          Central-header version:

          Value         Size            Description
          -----         ----            -----------
  (Unix1) 0x5855        Short           tag for this extra block type
          TSize         Short           total data size for this block
          ModTime       Long            time of last modification (GMT/UTC)
          AcTime        Long            time of last access (GMT/UTC)

          The file modification and access times are in standard Unix signed-
          long format, indicating the number of seconds since 1 January 1970
          00:00:00.  The times are relative to Coordinated Universal Time
          (UTC), also sometimes referred to as Greenwich Mean Time (GMT).  To
          convert to local time, the software must know the local timezone
          offset from UTC/GMT.  The modification time may be used by non-Unix
          systems to support inter-timezone freshening and updating of zip
          archives.

          The local-header extra block may optionally contain UID and GID
          info for the file.  The local-header TSize value is the only
          indication of this.  Note that Unix UIDs and GIDs are usually
          specific to a particular machine, and they generally require root
          access to restore.

          This extra field type is obsolete, but it has been in use since
          mid-1994.  Therefore future archiving software should continue to
          support it.  Some guidelines:

              An archive member should either contain the old "Unix1"
              extra field block or the new extra field types "time" and/or
              "Unix2".

              If both the old "Unix1" block type and one or both of the new
              block types "time" and "Unix2" are found, the "Unix1" block
              should be considered invalid and ignored.

              Unarchiving software should recognize both old and new extra
              field block types, but the info from new types overrides the
              old "Unix1" field.

              Archiving software should recognize "Unix1" extra fields for
              timestamp comparison but never create it for updated, freshened
              or new archive members.  When copying existing members to a new
              archive, any "Unix1" extra field blocks should be converted to
              the new "time" and/or "Unix2" types.

Jul 21, 2009 at 9:00 PM

Cheeso, thank you for taking the time to look into this for me. I would definitely try a version of DotNetZip that didn't include the extra fields. Also, I did not use Stuffit to create the zip file you used for testing. I used the default archive utility built into the Mac OS. I was using Stuffit (2009 version) as an second program to try and open the DotNetZip zip file.

One other thing I've noticed is that I am able to open DotNetZip zip files that do not contain any directories with no problem. If I use the command Zip.AddFile() instead of Zip.AddDirectory("Folder", "FolderInZIP"), I am able to open the zip file with Stuffit (but not the Mac OS Archive Utility). Does this relate to the extra headers? My problem comes in that I need to create directories in the zip file.

Thanks so much!!

Coordinator
Jul 21, 2009 at 9:44 PM

I don't know for sure if the problem you are having - failing to open zip files that were generated by DotNetZip and that contain folders - is related to the extra headers.  That is my theory and that is what we are testing out. 

I've uploaded a test build to the workitem.  You should be able to download it and try that new assembly.  You will need to use the new property, like this:

using (var zip = new ZipFile())
{
  zip.EmitNtfsTimesWhenSaving = false;
  zip.AddDirectory("whatver", "whatever");
  zip.Save(newFileName);
}

That new property should allow DotNetZip to save a zip file without the extra field. We'll see if that's the problem.

I have to caution you that I put this together really quickly and didn't test it very well. So bear with me. I have to go somewhere and wanted to get it out to you before leaving. If it doesn't work, I'll have a look tomorrow.

Jul 22, 2009 at 2:16 PM

Cheeso,

Thanks for the update. Unfortunately, it did not solve my issue. I'm uploading the new zip to the workitem for you to look at. Also, this new build does not seem to work under Medium trust.

Here is my code for review:

            // Create zip file and send to user
            Response.Clear();
            string title = lblTATitle.Text.Replace(' ', '_');
            string zipFileName = String.Format("{0}.zip", title);
            Response.ContentType = "application/zip";
            Response.AddHeader("content-disposition", "filename=" + zipFileName);

            using (ZipFile zip = new ZipFile())
            {  
                zip.EmitNtfsTimesWhenSaving = false;
                zip.AddDirectory(Server.MapPath(tempdir), title);
                zip.Save(Response.OutputStream);
            }

            HttpContext.Current.ApplicationInstance.CompleteRequest();

            // After zip is sent to user, delete temp files
            System.IO.Directory.Delete(Server.MapPath(tempdir), true);

Coordinator
Jul 22, 2009 at 2:55 PM

I think we're looking in the wrong place. 

I looked again at your zip files.   You're trying to zip up a total of 1638 bytes of content, yet the zip files are 28k in size, 57k in size.  If I look in the zip file, I can see a bunch of HTML source code, which seems wrong to me.  This HTML code is present in the files you uploaded earlier, as well as this most recent file.  It's the output of a web page generated by ASP.NET.  Maybe you do a server.transfer after the code above? 

It's pretty clear that the zip file is not being created correctly.  It does not look like a bug in DotNetZip that is causing it.  And it now seems less likely that the problem is with the NTFS file times.  There is junk in the ZIP file, which DotNetZip tolerates, but I think the Mac program doesn't tolerate it so well. 

How did the junk get there?  I don't know but I'm guessing it has something to do with the larger context of how you are using DotNetZip within ASP.NET, writing to the Response outputstream.  There is other data in that output stream.

What if you:  removed the call to ApplicationInstance.CompleteRequest() and put a call to Response.End() directly AFTER Directory.Delete() ?    does that work?

What do you mean "it does not work in medium trust".  Did you get the same exception?  That would be surprising because you sent me a zip file - how did you create the zip file if you got an exception? 

If including Response.End does not work, then we need to simplify this.   I want to do some more basic tests.    Create a standalone program that produces the zip file the way you want it.  Not within an ASP.NET app?  Let's see whether you can create a zip file, with the folders you need, that can be read on the Mac.  Like a simple 10-line console app, and then manually transfer the zip file over to the Mac.  Once we get that working, we can see if we can get it working in ASP.NET. 

 

Jul 22, 2009 at 3:45 PM

I'm not sure exactly what junk you're talking about. The zip file I uploaded today should contain a .ditamap and .dita files (all xml files). These are valid files and exactly what I am trying to zip. I looked at the zip file on the mac and it seems that the extra field headers are still there. At least it is saying so from what I'm looking at.

Here are the results I'm looking at:

-rw----     4.5 fat        0 bX stor 21-Jul-09 14:03 New_Employee/
-rw----     4.5 fat        0 bX stor 21-Jul-09 14:03 New_Employee/assets/
-rw-a--     4.5 fat    28521 bX stor 21-Jul-09 13:40 New_Employee/assets/EO_2009721134053_Bluehills.jpg
-rw-a--     4.5 fat      253 bX defN 21-Jul-09 14:03 New_Employee/ta_2.ditamap
-rw-a--     4.5 fat      651 bX defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12.dita
-rw-a--     4.5 fat      698 bX defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12_eo_18_detail.dita

The 'X' in the bX entry identifies it as a extra header/field.

As for the Medium trust - I have two different machines to test on. I received an exception on the machine that has Medium trust. I was able to create the zip file on my local machine which has full trust. I believe the error came from the fact that the first time I tested I forgot to add the 

zip.EmitNtfsTimesWhenSaving = false;




Jul 22, 2009 at 3:47 PM

Here are the results from a zip file I created in Windows, without DotNetZip:

 

Archive:  new_employee_pc.zip   28157 bytes   5 files
drwx---     2.0 ntf        0 b- stor 22-Jul-09 11:04 New_Employee/assets/
-rw-a--     2.0 ntf    28521 b- defN 21-Jul-09 13:40 New_Employee/assets/EO_2009721134053_Bluehills.jpg
-rw-a--     2.0 ntf      253 t- defN 21-Jul-09 14:03 New_Employee/ta_2.ditamap
-rw-a--     2.0 ntf      651 t- defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12.dita
-rw-a--     2.0 ntf      698 t- defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12_eo_18_detail.dita
5 files, 30123 bytes uncompressed, 27423 bytes compressed:  9.0%

 

And here are the results from a zip file I created on the Mac:

Archive:  new_employee_mac.zip   28488 bytes   6 files
drwxr-xr-x  2.1 unx        0 bx stor 21-Jul-09 14:03 New_Employee/
drwxr-xr-x  2.1 unx        0 bx stor 21-Jul-09 14:03 New_Employee/assets/
-rw-r--r--  2.1 unx    28521 bX defN 21-Jul-09 13:40 New_Employee/assets/EO_2009721134053_Bluehills.jpg
-rw-r--r--  2.1 unx      253 bX defN 21-Jul-09 14:03 New_Employee/ta_2.ditamap
-rw-r--r--  2.1 unx      651 bX defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12.dita
-rw-r--r--  2.1 unx      698 bX defN 21-Jul-09 14:03 New_Employee/ta_2_spo_12_eo_18_detail.dita
6 files, 30123 bytes uncompressed, 27420 bytes compressed:  9.0%

Coordinator
Jul 22, 2009 at 4:23 PM
Edited Jul 22, 2009 at 4:34 PM

There is junk in all of the zip files you have uploaded, that were created with DotNetZip.  sorry I have not been clear.  I'm not really interested in the  output of a zipfile created by Windows Compressed Folders.  What I am saying is, your use of DotNetZip within an ASP.NET program is causing junk to appear in the zip files you create.  I would like to see a zipfile created with DotNEtZip, but not within ASP.NET. 

There is junk in the zipfiles.  If I open the file in a hex editor, I can see HTML code.  It is not XML, and it is not JPG data.  It is the output of an ASPNET page.  It looks like this (in part)

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>
	Task Analysis
</title>

You can see there is junk just by examining the sizes of the file. The file you initially uploaded, called New_Employee.zip, has a size of 57157.  Unzip says these are the contents:

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------
2009-07-21 14:03:54             0     0%           0    N 00000000 New_Employee/
2009-07-21 14:03:54             0     0%           0    N 00000000 New_Employee/assets/
2009-07-21 13:40:52         28521     0%       28521    N 245A97A2 New_Employee/assets/EO_2009721134053_Bluehills.jpg
2009-07-21 14:03:54           253    38%         158    N 7873F9F1 New_Employee/ta_2.ditamap
2009-07-21 14:03:54           651    56%         288    N 1F222259 New_Employee/ta_2_spo_12.dita
2009-07-21 14:03:54           698    58%         291    N DAEDD584 New_Employee/ta_2_spo_12_eo_18_detail.dita
--------------------------------------------------------------------------------
                            30123                                  6 files

The total size of that zip file should be around 30k, but instead it is 57k.   If I just extract the contents of that zip file, and then re-zip those contents, using the unzip.exe and zipit.exe tools, the resulting re-packed zipfile is size 30526.   If I look at the file New_Employee_Mac.zip, there is no junk in the file.  The size of the file is in the expected range: 28488 bytes.

Based on all of this, I am suggesting that there is an interaction between DotNetZIp, ASPNET, and your code, which is causing junk to appear in the zip file. 

I suggest that to troubleshoot that, you should remove or isolate some of the variables.  The obvious variable to remove is ASPNET.  To do this, create a standalone program that uses DotNetZip and creates the zip file in the same way you create them within the ASP.NET app. The key point is this app must not run within ASPNET, because that is the variable you're trying to remove.  Then transfer that file to the Mac to see if you can open it.

If you can open it, you can conclude that it is something in the combination of DotNetZip and ASP.NET that is causing the problem. We can eliminate NTFS file times as a theory for the reason that Mac's Archive Utility is complaining.

I also suggested some changes in your ASP.NET code, to try to figure this out.   Did you try that?

This is just how I would go about troubleshooting it. Of course it's your decision how you want to proceed. 

 

Coordinator
Jul 22, 2009 at 4:49 PM
Edited Jul 22, 2009 at 4:55 PM

Jess, also, I looked at the most recent File you uploaded, New_Employee_1.zip. I agree it still seems to have the NTFS file times. Are you certain that you've created it using the new property? (You seem to have had some doubts about this)

Also, that file seems to lack the JPG file that was present in all the other zipfiles. This is the list of entries:

Modified                     Size  Ratio      Packed  pw?      CRC Filename
--------------------------------------------------------------------------------
2009-07-22 10:17:26             0     0%           0    N 00000000 New_Employee/
2009-07-22 10:17:26           253    37%         160    N 38CCE70D New_Employee/ta_2.ditamap
2009-07-22 10:17:26           687    55%         308    N F5DBB576 New_Employee/ta_2_spo_38.dita
2009-07-22 10:17:26           698    58%         291    N E7E72B2D New_Employee/ta_2_spo_38_eo_86_detail.dita
--------------------------------------------------------------------------------
                             1638                                  4 files

Despite containing just 4 files totalling 1638 bytes uncompressed, the total size of the zipfile is 28446. I can look and see that after byte 1423, there are some newlines and then the HTML text - the same HTML code as described in my prior post.

If I truncate the file and remove all the bytes after 1423, the zip file remains valid, and contains the same contents. I can unzip it successfully.

I've uploaded the truncated file to the workitem. It might be interesting to learn if the Mac utility can unzip that truncated zip file.

Coordinator
Jul 22, 2009 at 5:17 PM

One more thing - I wrote a very simple app to test out the  zip.EmitNtfsTimesWhenSaving = false; thing, and it works - it creates a zip file that does not include the extra field that contains the NTFS file times.   You observed that the files you created included NTFS file times.  If that's the case I'd suggest maybe you are not using the test assembly properly, or there is some other problem.

I've uploaded that zip file to the workitem - if you like you can copy it to a Mac and confirm that you can, or cannot open it. 

as I said earlier though, I think the NTFS file times is a red herring. The idea that it was the NTFS file times that was causing the Mac  Archive Utility a problem, was a theory I held before I saw the "junk" in the zip files you were posting. 

 

Jul 23, 2009 at 1:49 PM

Cheeso, thanks so much for all your help with this. That is very weird about the HTML code, which looks to me to be the actual page source. I will try creating a standalone application that does not use ASP.NET. Also, I still was not able to open the New_Employee_1_truncated.zip, but I was able to open the nontfstimes.zip successfully on the Mac. Hooray! :) I will try the test assembly again. Thanks again!

Coordinator
Jul 23, 2009 at 2:57 PM

Ok, let me know.

The data in the zip file appears to be page output, not page source.  There are no runat="server" tags in it.  There is __VIEWSTATE and other stuff that you would expect to find in a generated ASPNET page.

Good to hear that the nontfstimes.zip could be opened successfully.  Surprising, though. 

Sep 10, 2009 at 4:07 PM

Hi Dino!

Did you ever solve this?

I'm having the same problem. Cannot open in Mac, can when copied to PC.
Same type of creation - created by adding directory and then saving to output stream. Same 

Response.ContentType = "application/zip";
Response.AddHeader("content-disposition", "filename=" + zipFileName)

So it seems to me (I'm postulating) that the problem is in the "Save to Response.Output" perhaps it sends HTML info into the zipfile, which is tolerated on the PC but not on the mac zip opening software?
Or maybe its the need for both  ContentType="octet/stream" And  EmitNtfsTimesWhenSaving = false
Or the need to have a Response.End rather than a CompleteRequest() or maybe neither?!


I'll check with a hex editor to see if I have the same "junk" problem, and nothing to do with the "header" problem.
Could you send a hex modified zipfile, that was not working on the Mac, but now has the "junk" section removed, but still DOES have the headers.
I can check that on the Mac.

In my country we don't work on Friday and Saturday (the Sabbath day), but do on Sunday, I'll get to it then.

Thanks!  Moshe

 

Sep 10, 2009 at 4:32 PM

Looks like no html problem. I do have HTML code, but its way inside the zip, and that's because there in fact ARE HTML files included.
I'll make a test app that just does an AddDirectory. (Already have an app like that, with two simple text files)
I"ll check that it opens on PC and does not on Mac.

Will post. Thanks Cheeso for the great support. Your program is great!

Coordinator
Sep 10, 2009 at 5:55 PM

> Did you ever solve this?

There isn't a problem on my side, nothing for me to solve.

> So it seems to me (I'm postulating) that the problem is in the "Save to Response.Output" perhaps it sends HTML info into the zipfile,

No.  It is very possible to produce corrupt output when saving to Response.OutputStream, if there's a bug in the ASPNET code.  I've seen this many times. If the ASPNET page doesn't call Response.Clear() first, then the output page may have generated HTML code within it.  There's nothing in the library that could cause this. 

It's more likely that it is pilot error.

Sep 10, 2009 at 6:26 PM

Cheeso you'll see the same result too!
There's a difference on the PC too between a saved file and a streamed file (just that on the PC both open, while on the Mac only the saved opens).

OK Checked and the main difference is the "junk" at the end. Also there's a tiny difference in each entry, I think the download is not sending "stream octet"
or maybe changing the endline.

Here's my code.  Didn't set any HTTP headers or anything just this in the Page_Load:

line 1 toFile Header: '52E' 01 02 - 0 14 00000 Noe*;GOk 11 000 11 000 19 0 $ 0000000    
using (ZipFile zip = new ZipFile())
{
    // in c:\temp\bla I have 3 text files which are copies of each other
    zip.AddDirectory(@"c:\temp\bla", "blu");
    zip.Save(@"c:\temp\blaSave.zip"); // opens on PC and Mac
    zip.Save(Response.OutputStream); // saved to c:\temp\blaDown.zip opens on PC, fails on Mac.
}


______________
Differences:
toFile PK 03 04 14 0 0 000 Ooe*; ..... 040$0bla/
toStream PK 03 04 14 0 8 000 Ooe*; ..... 040$0bla/

same PK 03 04 14 0 0 000 Noe*;G OK 11 000 11 000 %0$0 bla/text1.txt 0 00000 1 0 18 0 oJa#52E 01 ?s O652E 01 u 12 > &52E 01 Hi there everyone

but then almost all entries have:
toFile: PK 03 04 14 0 0 000 Noe*;G OK 11 000 11 000 %0$0 bla/text2.txt 0 00000 1 0 18 0 oJa#52E 01 .u 0652E 01 is_ &52E 01 Hi there everyone
toStream: PK 07 08 G OK 11 000 11 000 // extra line in each entry...
and on next line
PK 03 04 14 .... with the correct entry

At the bottom of the file

           <!DOCTYPE html PUBLIC ....
           <html xmlns="http://www.w3.org/1999/xhtml" >
           <head><title> ....  
As described by you.
_______________________________________________________________________
Hope this helps you assist us.
Moshe

Coordinator
Sep 10, 2009 at 6:53 PM

You can get that kind of junk in your output if you don't call Response.End or Response.Close() after saving the zip file.

 

Sep 10, 2009 at 7:06 PM

Just tried with added following code:

 

Response.BufferOutput = false;
Response.Clear();

Response.ContentType = "application/octet-stream"; // also tried /zip
Response.AddHeader("Content-Disposition", "filename="zip2.zip");
Response.Flush();

Maybe this the problem on the Mac is happening because I don't give the contentSize?

___________________
The results: This time the file has no "impurities" inside, but still has the HTML at the end.
Opens on PC but not on Mac.
Can't it be a problem with the Zipfile.Save(Stream s)?
Could you please check a zipfile that you create by saving to OutputStream and then saving the download?
And then looking at the stream?

Thanks, Moshe


Sep 10, 2009 at 7:12 PM

Sorry for all the noise. Its not the Response.End()...

I found this on a forum 

  Hello Todd,

   This extra HTML is coming from the .aspx file hosting your webform. In your
   case this file is only a placeholder, something with the appropriate
   extension to drive the request to your class. Just remove the HTML code from
   your .aspx file.

   Be sure to leave the <%Page directive, otherwise your page won't be
   associated with your .vb code-behind file and your form class.

   Cheers,
   Samu

Sorry for making the trouble!
Moshe

Sep 10, 2009 at 7:46 PM

Still not working. WITH Response.End and Removed the HTML code.

The diff between the zip2file, and zip2stream (and then in saved in client) is now only INSIDE the stream.
I DO get the "impurities"
Each entry has an extra: PK 07 08 GOk 11 00 00 00 11 00 00 00
inserted before the        PK 03 04 14 00 08 ...

And the entries all have  PK 03 04 14 00 08 00 00 00 Noe*;GOk...
while the 2file has:          PK 03 04 14 00 00 00 00 00 Noe*;GOK...

Also the file consistently end differently:

 2file:        PK 05 06 00 00 00 00 06 00 06 00 99 02 00 00 8E 02 00 00 00 00
 2stream:  PK 05 06 00 00 00 00 06 00 06 00 99 02 00 00 DE 02 00 00 00 00

Sorry for making more trouble, Moshe

Coordinator
Sep 10, 2009 at 9:32 PM

Did you try with Response.Close() as I suggested?

We have this funny way of communicating.  You describe a problem.  I make a suggestion.  You then try something totally different.  And you tell me the results.   What happened to my suggestion?  Did you try it? 

It seems kind of pointless, for me continue to read your posts and make suggestions if you just ignore what I write.

What do you mean "still not working"?   Does this mean the file does not open?   Or do you mean that two zip files are not identical?

And about the "impurities".  I don't know what you mean, exactly by that, but consider this - a zipfile that is produced on a non-seekable stream will have slightly different content than a zipfile that is produced on a seekable stream.  These are not "impurities".   the reason is some of the metadata that must be present in the zipfile can be placed before or after the compressed data, but it is known only after the compressed data is produced. On a seekable stream the library seeks back and writes the metadata before the compressed data.  on a non-seekable stream, the library writes the metadata after the compressed data.  This is all in compliance with the zip spec.  It is also described in the doc.

> Sorry for all the noise. Its not the Response.End()...
> I found this on a forum 

Honestly I have no idea what that random post has to do with anything.   ???

What do you mean "It's not the Response.End(). "   WHAT is not Response.End()?  

> Could you please check a zipfile that you create by saving to OutputStream and then saving the download?  And then looking at the stream?

Ok, I've done that and the file opens just fine. 

I'm not sure how to help you.

 

Sep 10, 2009 at 11:41 PM
Edited Sep 10, 2009 at 11:43 PM

>Did you try Response.Close...
No, you suggested Response.End() or Response.Close(),  And in any case it was to remove the "junk" HTML at the end.
Since currently I am not getting "junk HTML" anymore, I don't think that would make a difference.  In any case I'm not at work now, so cannot test it till Sunday.

>We have this funny way of communicating... it seems quite pointless...
I'm sorry. I continued testing before receiving your reply. For some reason, although I refreshed the page (9:20) before replying, codeplex wasn't showing me your reply,
This happened several times. So each posting of mine referred only to two replies of yours before.  
I appologize, I just thought I discovered a solution, then was dissapointed to see its not working.
Mac support IS a problem, since there was no reply here that it succeeded for the first asker (who seemingly moved to SharpZipLib because of this)
and a similar problem occured in another forum (possibly solved, but possibly only because without directories the problem does not occur, and originally caused from similar reason). 

>What do you mean still not working?
Cannot open in the Mac.

>And about "impurities" ... I don't know what you mean, exactly, by that.
>... this is all in compliance with the Zip spec.
I'm not sure about that.
What I found is that there seems to be:  
a. EXTRA metadata that's being inserted for each entry. (unless your saying this is a placeholder for the seekable stream)
b. There is a difference in single bytes at key locations. I was thinking that this is either a mistake, or an inherant feature of the output stream, which would give a line break or some special char
instead of what should be there.
What I'm saying is, that perhaps if you look at the differences I found, (by running code of this sort yourself, and then viewing the result in an editor, or by debugging the output stream in the two cases)
you may have some understanding of what can be changed so that it DOES open in Mac.  I'm hoping for your insight. I could then test the result of any change you tell me of, on a mac .

> Honestly I have no idea what that random post has to do with anything...
> What do you mean...     WHAT is not Response.End()?
I was trying to stop the "junk" html code that appeared at the end of the stream from being appended to the stream.
You had suggested that Response.End() would help. It didn't, and I was looking for some other way.
The "random" post says simply: "When you have html code in your ASPx page,  you'll get it appended to your stream! Erase it! "
I went to the aspx page and removed the code except for the <%page directive. Presto! The HTML code was gone.
But sadly the file still did open on the Mac.

>>Could you please check...  and then looking at the stream. 
>OK I've done that and the file opens just fine.
Yes it opens on the PC, and you don't have a Mac to check. But you can look at the inside and you'll see the same as I described:
   An extra line for each entry, which I suspect is not valid for zip (looks to my layman eyes like its missing some info)
   Some important chars (like in the zip footer) are changed, although the rest of the file seems the same. Could it be caused by the code,
   or perhaps this char change is caused by some setting of the output stream?
   (Actually I'll check if I change this chars manually if they cause the zip to open.

In a former discussion on saving to output stream, I was deaf to your replies, because of a misconception. I read them, but did not "get it".
I believe and hope this is not the case in this discussion. 

Thanks, Moshe

Coordinator
Sep 11, 2009 at 12:20 AM

I don't know the why you're having a problem opening zips on the Mac.  Other people have had problems and have tried different tools on the mac.  Apparently there is a built-in archive utility, and there is a stuffit tool, and other tools too.  The other people have reported differences in the behavior of the various tools.

One of the things that presented a problem was the time formats.  The ZIP spec allows different ways of formatting timestamps for entries included in the zip file.  The zip spec calls the different options "DOS format", "Unix format" and "Windows format" but these names are misleading.  There is nothing about the formats that limits their use to the particular operating systems they are named after.  Timestamps in those formats can, in theory, be read and written by any computer, any operating system.  But, not all tools do.  Some tools on Mac do not correctly read the "Windows" format timestamps, and actually report the zip file as corrupted if it includes those timestamps, even though the format for "windows" timestamps is specified in the PKWARE specification.  And there may be examples of the converse too,  There may be examples of tools that run on Windows that cannot read zip files if they include "Unix-formatted" timestamps. I don't know of examples like this, but it wouldn't surprise me if there were some files.

Now because DotNetZip runs primarily on windows and the zip files it produces are primarily consumed on Windows, by default DotNetZip produces zip files with timestamps formatted in "Windows" format.  If you use the properties EmitTimesInUnixFormatWhenSaving  and EmitTimesInWindowsFormatWhenSaving  you can alter this default behavior. 

One of the tools on the Mac could support neither of those - and instead could read and write zipfiles only with an older, deprecated timestamp format, which, since 2003 I think, the PKWare specification has said should *not* be supported.  This particular tool, I don't recall what it was, reported perfectly compliant zip files to be corrupted.

Now, is this the problem you are having?  I don't know. There's no way for you to know unless you perform lots of testing.

The "impurities" you have focused on are likely not the source of the problem.    As I explained, there are differences in the zip file if it is written to a non-seekable stream.  It is not "an extra line."  It is just a different set of metadata, formatted differently.  But this option is also described in the zpi spec in detail.

If I were you I would produce a test harness that created zip files using vavrious tools, including dotnetzip, and various options within dotnetzip - encryption, time formatting, unicode, zip64, and etc - and then I would test opening each of these zip files on the Mac. At some point using this approach you may find out what things work with your particular archive tool on the Mac, and which don't. 

I do know that if you use ASPNET, you will get junk in your zip file if you Save the zip file to Response.OutputStream, and if you do not call Response.End or Response.Close afterwards.  I just tested it here and reproduced that problem.  That may or may not cause a problem for a Mac utility.

 

Coordinator
Sep 11, 2009 at 2:15 PM

ps:  the interop implications around time formats is described in the doc for EmitTimesInUnixFormatWhenSaving and also in the doc for the EmitTimesInWindowsFormatWhenSaving.

Sep 11, 2009 at 2:55 PM

There is a real problem of DotNetZip on the Mac!  I think its a small one, that wont take much time from you, and will be worth your effort.
Proof: I'm getting mails offline telling me that people switched to SharpZipLib, even though it takes an extra few lines of unintuitive code.
Also this discussion was abandoned by jessdy. He never followed up on your suggestions. I'll explain what I think happened:

I read well the discussion above (this is not my thread) and IMHO you were running after the wrong symptom.
The Mac was not having a problem with the extra info dates.

That was your idea after looking at the difference between the file zipped by the Mac default tool
and a zipfile saved on the client from a download created by jessdy's code which is aspx zip to stream only.
He never tried zipping to  a file with DotNetZip and then checking that on the Mac.
I did. And found that the regular zip (with the "extra" date attribute) DOES OPEN ON THE MAC. So that is not the problem.
Also, using the EmitNtfsTimesWhenSaving = false didn't solve it. IMHO, because that was never a problem for the Mac.
It was only something you noticed as a difference between "what works" from jessdy, and what doesn't.

>Apparently there is a built-in archive utility, and there is a stuffit tool...
Mac has a default unzip action when double clicking, and there's "stuffit" which is more like winzip and gives options.
These two are the common Mac interfaces.

>One of the things that presented a problem was the time formats.
IMHO they did not. When saving a file in a regular DotNetZip ZipFile.Save() action on a PC, the file has these same timestamps and opens OK. 
It is only when you CREATE a zip on the Mac, which is what jessdy did, that you get old unix timestamps, and perhaps only in some versions of the OS.
That is NOT the cause of the problem.

>If you use the properties EmitTimesInUnixFormatWhenSaving  and EmitTimesInWindowsFormatWhenSaving  you can alter this default behavior.
Which you wrote so that the problem will be solved, but, as I explained, that was never the problem, so no need.

>One of the tools on the Mac could support neither of those
No that was not the problem, its a wrong conclusion of looking first time at two files that significantly differed.
But, as I said, when using a DotNetZip created zipfile saved directly to the filesystem without going thru the output stream,
we got a file with the WindowsFormat, and the "one of the tools" - which happens to be the default unzip on Mac, COULD read it.
>and write zipfiles only with an older, deprecated timestamp format, which, since 2003 I think, the PKWare specification has said should *not* be supported. 
right
>This particular tool, I don't recall what it was, reported perfectly compliant zip files to be corrupted.
Wrong. You found when looking in the hex that there was "junk" information appended at the end of the file.
Still, windows knew how to open it.
I had the same problem with the stream. Found out how to eliminate this extra "junk" info, and then tried again.
Mac still has the problem opening the file. So its not the "junk" appended, which is tolerated by windows, and probably not a problem for MacOS either.

>Now, is this the problem you are having?  I don't know. There's no way for you to know unless you perform lots of testing.
No there's only a tiny bit of testing needed. If we focus on the "impurities".

>The "impurities" you have focused on are likely not the source of the problem.   
>As I explained, there are differences in the zip file if it is written to a non-seekable stream. 
>It is not "an extra line." 
>It is just a different set of metadata, formatted differently.  But this option is also described in the zpi spec in detail.
So the "different set of metadata" is described in the zip spec. Great!

What about the extra 8, and the ending of the zipfile.
Its easy for me to check that. I'll change it with a binary editor and run it on Sunday.  

>If I were you I would produce a test harness that created zip files using vavrious tools,
>including dotnetzip, and various options within dotnetzip - encryption, time formatting, unicode, zip64, and etc -
>and then I would test opening each of these zip files on the Mac.
>At some point using this approach you may find out what things work with your particular archive tool on the Mac, and which don't.
I think it will be a lot easier for you than for me, and all that's needed is to simply check those two tiny differences of output,
when streaming and when saving to file.  As I just said, I will check if that fixes the problem on Sunday.

>I do know that if you use ASPNET, you will get junk in your zip file if you Save the zip file to Response.OutputStream,
>and if you do not call Response.End or Response.Close afterwards. 
>I just tested it here and reproduced that problem.  That may or may not cause a problem for a Mac utility.
Yes, but if you erase the form, and just leave the page directive in the main .aspx file, you don't get any junk, anymore.
But the problem in the Mac persists.

Moshe

Sep 11, 2009 at 2:58 PM

and opens ok... 
I meant
and opens ok on the Mac

Sep 11, 2009 at 3:17 PM

I am just now seeing all of these posts from the last few days and thought I'd share what I found to work for me. Sorry I am so late in posting any results! I was never able to get the zip.Save(Response.OutputStream) to work on my end. My expertise does not lie in ASP.NET or standalone apps so I wasn't quite sure how to do all of the testing required to see how to get the stream to work or why it was not working. My solution was to use zip.Save() to save the zip file on the server first. I then sent the saved file using the Response stream. After, I deleted the zip file from the server.

By doing it this way, the zip file opens perfectly on the Mac using any zip program, even the default Archive Utility. This may not be the most ideal solution, but it works for me for now.

If you figure out how to send the zip using the Response.OutputStream, please let me know!

For now, here is the code that I'm using. I hope this might be helpful.

 

            using (ZipFile zip = new ZipFile())
            {
                zip.EmitNtfsTimesWhenSaving = false;
                zip.AddDirectory(dir);
                zip.Save(Server.MapPath(zipFilePath));
            }

            // Send the file to the user
            try
            {
                string path = Server.MapPath(zipFilePath);
                FileInfo file = new FileInfo(path);
                if (file.Exists)
                {
                    Response.Clear();
                    Response.AddHeader("Content-Disposition", "attachment; filename=" + file.Name);
                    Response.AddHeader("Content-Length", file.Length.ToString());
                    Response.ContentType = "application/zip";
                    Response.WriteFile(file.FullName);
                    Response.Flush();
                    Response.Close();
                    file.Delete();
                }
                else
                {
                    Response.Write("This file does not exist.");
                }
            }
            catch (Exception ex)
            {
                error(ex);
            }

 

 

Coordinator
Sep 11, 2009 at 3:59 PM
pashute wrote:

There is a real problem of DotNetZip on the Mac!  I think its a small one, that wont take much time from you, and will be worth your effort.
Proof: I'm getting mails offline telling me that people switched to SharpZipLib, even though it takes an extra few lines of unintuitive code.
Also this discussion was abandoned by jessdy. He never followed up on your suggestions. I'll explain what I think happened:

....

>Now, is this the problem you are having?  I don't know. There's no way for you to know unless you perform lots of testing.
No there's only a tiny bit of testing needed. If we focus on the "impurities".

>The "impurities" you have focused on are likely not the source of the problem.   
>As I explained, there are differences in the zip file if it is written to a non-seekable stream. 
>It is not "an extra line." 
>It is just a different set of metadata, formatted differently.  But this option is also described in the zpi spec in detail.
So the "different set of metadata" is described in the zip spec. Great!

What about the extra 8, and the ending of the zipfile.
Its easy for me to check that. I'll change it with a binary editor and run it on Sunday.  

...

Moshe

 Moshe, look, I understand you're having problems.

I cannot reproduce the problems you have, because I don't have a mac. 

I cannot fix the problems, if they are not described clearly to me. 

When I left off with jess, the conclusion I had come to was that the timestamps were the culprit.  You are now telling me that is incorrect.  Fine.  I believe you.

I stick by my suggestion that the only way to know what works and what does not, is to test the combinations.  With no Mac, I can't do that testing.  maybe you can.   

you asked about "the extra 8".  This is not an "impurity".  As I told you, there is a difference in format for zipfiles written to seekable vs non-seekable output streams. the "extra 8" is a bit that indicates the file was written to a non-seekable stream, and the reader of the zip file should look for the alternative metadata.  It is quite possible that the mac utilities don't handle this properly, even though it is clearly specified in the zip spec.  The only way to know is to test it.  I cannot do that testing.

You cannot just "change the 8" in the binary file.  The 8 indicates the presence of other data.  Changing that byte to a zero will produce an invalid zip file.

You are telling me that "fixing the problem on the Mac won't take much of my time."  But there is nothing to fix.  I don't have a clear description of the problem from you, or from any Mac user.   I don't have a Mac, so I can't investigate this on my own.  With no clear description of the problem, there is nothing to fix in DotNetZip. 

If you are telling me that zip files produced via streaming to Response.OutputStream do not open on the Mac, I'll believe you.  But, they are legal, compliant zip files, and they can be opened by every tool I ever tried.  If the Mac utilities don't open them, this tells me there is a problem in the Mac tool, not in DotNetZip, and there is nothing to fix in DotNetZip.  I can document the interop issue, but I can't fix the Mac utilities.   And taking out the streaming capability from DotNetZip, just because of a shortcoming in the Mac tools, is wrong.   (There are at least some definite problems in the Mac tools, because the zip files I got from jess showed unsupported extra fields for the timestamps - stuff that the zip specification deprecated in 1996.  No zip tool should write zip files that way, but the Mac does.  This is clearly a problem with the Mac tool.)

There are other features of zip files that can cause interoperability problems.  AES encryption, for example.  If you produce a zipfile with AES encryption, you can't extract it from Windows Explorer, nor from the Java classes, nor from the PHP zipfile library, as well as other tools.  This is not a problem in DotNetZip. It's a limitation of Windows Explorer, and those other libraries.  The best I can do is document it:  If you want to open the file in Windows Explorer (etc), then don't use AES encryption. And I have clearly documented this.  Something similar applies to other zip features, like Unicode, spanned zip files, and ZIP64.  I document the interop issues that I know about.  I can do the same on the Mac interop issues, if you tell me what to document.   If I get a clear description of what works and what doesn't work on the Mac, I will be happy to document it, as clearly as I can.  If the existing documentation suggesting that the time formats can cause interop problems is incorrect, I will be glad to change it.

I can only work with what you Mac users provide me.  I don't have a Mac.

 

Coordinator
Sep 11, 2009 at 4:00 PM
jessdv wrote:

I am just now seeing all of these posts from the last few days and thought I'd share what I found to work for me.

 Thanks, Jess, very helpful.

Sep 12, 2009 at 3:07 PM

Thanks Jess for sharing the solution.    One of the neat features that DotNet Zip has;  is  the ability to read from the stream and zip the files as they come.  I find this feature very usefull since in the real world,

real files would be retrieve from different places, and the path for this places are usually store in a database.   It would be nice , if this feature can be enhance to take into account that the mac behaves differently

when zip files are created this way.    One of the drawbacks that  I see with Jess solution , "by the way , I am using the same solution as Jess given that I could not figure out how to stream those files",

if files exist on different places them whatever files need to be zip , need to be copy to a temp directory, and them zip, which I think is fine for a small site with limited users, however, for larger sites with alot  of

traffic and a lot zipping going on, this has to put an overhead on the server, and potentially slowing the server.

I don't know how hard is it woud be to enhance this , since I my self don't have a mac, and it seems that to propertly trouble shoot this issue a mac is required.   

 

Sep 13, 2009 at 10:36 AM

Cheeso thanks! 

The problem is simple, and now jessdv confirmed it as well as jozcar:
   DotNetZip save to output stream does not work on the Mac. (With the default system unzip, or with a well known Mac utility)

It DOES open on the PC. But
> ... but they are legal compliant zip files...
well, not always.  Unless the trailing html that you found in jesdv's zipfile is compliant in zip. That same file opens OK on the PC. Is anything allowed after the zip footer?
This extra html footer inserted by mistake by both jesdv and myself, because we had a form listed in the .aspx file, in fact DID open OK in the PC,
and even in the Mac command line utility, without complaint, by the unzipping programs. 
You can easily verify this, by making a fast default.aspx app from the visual studio that saves to the OutputStream in the codebehind. 
Even with the Response.End (didn't try Response.Close) you'll get the form html at the foot of the zip, and surprise surprise, no problem  opening it on the PC.
But as we have since found out this is NOT "the Mac problem".  We settled the form html problem, (by deleting the form code) and still cannot open the output zipfile on the Mac. 


So there is only one unaccounted for difference which you didn't explicitly tell me is zip compliant:
The files consistently end with the following difference:

 2file:        PK 05 06 00 00 00 00 06 00 06 00 99 02 00 00 8E 02 00 00 00 00
 2stream:  PK 05 06 00 00 00 00 06 00 06 00 99 02 00 00 DE 02 00 00 00 00

1. Is the DE correctly zip compliant.
2. Can I change the DE to 8E and will it still be compliant.

That is the "small amount of time" I meant. There were three differences, and I was asking if you could account for them:
a. "extra line" - your reply: part of the spec for streaming. 
b. The 8 in the entry header - your reply: part of the spec (pointing to the "extra line")
c. The DE vs 8E at the end - waiting for your reply.

Also, thanks jessdv for your reply!  As with jozcar, I too cannot use the server temporary file solution, and need the output stream.
I'll be at work in an hour or so, not able to connect to my PC for some reason. 

Of course it may be that the two Mac utilities have a bug, and are not supporting compliant zipfiles.
If the 8E/DE is not the solution, I can try saving from SharpZip stream to the Mac as suggested by several emails I received - and check the differences.
Cheeso and jozcar, I do have a Mac, and can test anything you wish, and tell you the results.
I'm sure the SharpZip puts out compliant code, and it will be easy for me to compare the differences (for a very simple zipfile). 

Moshe

Sep 13, 2009 at 3:08 PM

Just changing DE to 8E did not help.
Continuing to plans b  and c (comparing with sharpzip, trying with no compression)

 

Coordinator
Sep 13, 2009 at 3:22 PM
pashute wrote:
It DOES open on the PC. But
> ... but they are legal compliant zip files...
well, not always. Unless the trailing html that you found in jesdv's zipfile is compliant in zip.

As I told you, the HTML comes from omitting the Response.Close(). When you save to a stream, like Response.OutputStream(), and then if you write additional bytes to the same stream, you will get junk in the zip file. This is not a problem in DotNetZip. It's a problem with the design of the application.

pashute wrote:

Even with the Response.End (didn't try Response.Close) you'll get the form html at the foot of the zip,

No. This is not the result I see.

The DE is an offset. You cannot simply change bytes in the zip file.

You have asked about the 8, the DE, and so on.  If you are concerned about the bytes in the file, you may want to read the zip spec to understand the significance of these bytes.

It's quite possible that the zip files produced by DotNetZip when writing to a non-seekable stream are incorrect, but so far I have not seen any indication of that.

 

Sep 13, 2009 at 3:40 PM

Just tried: Forcing no compression didn't help either.

I only brought up the HTML issue to show that the unzip utilities are very tolerant, even when the zipfile has code not according to the spec.
So it gave me reason to think that perhaps DotNetZip IS making a mistake - appraently not - I agree, there is no indication otherwise.

So I will now try and compare to a sharpZip download, that supposedly does open ok on the Mac.
Just an interesting point: The code given to me is for a flat file with no directories in it. This succeeded for jessdv too (see beginning of this thread).

To summarize:  
 Streamed flat file                    - opens on PC and Mac
 Saved      deep (with dir) file    - opens on PC and Mac
 Streamed deep (with dir) file   - opens on PC fails on mac

Stay tuned for updates!

BTW does DotNetZip set a "general purpose bit" anywhere in the code? According to this forum Mac interprets this bit a bit differently from the PC.

Coordinator
Sep 13, 2009 at 4:42 PM

> even when the zipfile has code not according to the spec.

There's nothing in the zip spec that says you cannot have extra data in the zip file.  This is how a SFX works.   Extra HTML at the end of the zip file does not render the zip file invalid, although it serves no purpose and can be confusing.

> BTW does DotNetZip set a "general purpose bit" anywhere in the code

The general purpose bit field in the zip spec is used for a number of things.  And yes, DotNetZip sets it.  The "08" you noticed near the beginning of the file is the bit field, with bit 3 set, indicating that the zipfile has been written to a non-seekable stream, and that the "data descriptor" containing the lengths and the CRC follow the compressed data.  In the forum post you mentioned, the code apparently did not set bit 3 in the bitfield, but did emit the data descriptor.  This is inconsistent and results in a non-compliant zip file.   A compliant zip file will have bit 3 of the bit field set, and a data descriptor, or bit 3 in the bitfield unset, and no data descriptor.

The bitfield is used for other things, too.

> To summarize:...

I think you have something there - you said the mac has a problem opening the zip file only when the zip file has been written to a non-seekable stream and also when the zipfile includes a directory.  It's possible the data descriptor attached to the entry in the zipfile for the directory, is confusing the mac archive tool.  The data descriptor contains a CRC, and two lengths, all of which are meaningless for a directory.  Normally these fields are set to zero.  But it could be that the mac tool assumes that no data descriptor is ever present for a directory entry, while DotNetZip emits it.  I will look into this and get back to you.

 

Coordinator
Sep 13, 2009 at 8:32 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Sep 13, 2009 at 10:20 PM
Edited Sep 13, 2009 at 10:21 PM

Just of curiosity… why anyone would use ASPX page to generate binary content (zip file), instead of using ASHX handler?

Coordinator
Sep 13, 2009 at 10:40 PM

It's a good question.  I think the .ashx is the right way to go, but unfortunately I haven't provided an example .ashx, so most people use the .aspx.

Hey can you write an .ashx for me, something I can use as an example?  The usage model would be, a web page with a list of files, each with a checkbox.  The user checks the files he wants, then clicks a button.  This (I guess) invokes the .ashx which produces the .zip file.

 

Coordinator
Sep 13, 2009 at 10:43 PM
Edited Sep 14, 2009 at 12:09 AM

Pashute, an update. Turns out that DotNetZip was doing something inconsistent.  When setting bit 3 in the general purpose BitField, it did not always include a data descriptor.  In particular, when writing a zip file to a non-seekable stream, the library would set bit 3 even on directory entries, which did not have a data descriptor.  I've modified the library to never set bit 3 for a directory. So now when bit 3 is set, a data descriptor is always present.   

I just posted v1.9.0.6, which includes this change.  I'd like to know if the Mac likes zip files produced this way.   

I will also back-port this change to v1.8, because it is low impact and is widely interesting.

 

Sep 14, 2009 at 10:10 AM
Cheeso wrote:

It's a good question.  I think the .ashx is the right way to go, but unfortunately I haven't provided an example .ashx, so most people use the .aspx.

Hey can you write an .ashx for me, something I can use as an example?  The usage model would be, a web page with a list of files, each with a checkbox.  The user checks the files he wants, then clicks a button.  This (I guess) invokes the .ashx which produces the .zip file.

 

 Here you are: http://www.sendspace.com/file/v7ole8 :)

Sep 14, 2009 at 10:16 AM
OKay! I'll try it!

On Mon, Sep 14, 2009 at 1:43 AM, Cheeso <notifications@codeplex.com> wrote:

From: Cheeso

Pashute, an update. Turns out that DotNetZip was doing something inconsistent.  It was setting bit 3, when writing a zip file to a non-seekable stream, even on directory entries which did not have a data descriptor.  I've modified the library to never set bit 3 for a directory. 

I just posted v1.9.0.6, which includes this change.  I'd like to know if the Mac likes zip files produced this way.   

 

Read the full discussion online.

To add a post to this discussion, reply to this email (DotNetZip@discussions.codeplex.com)

To start a new discussion for this project, email DotNetZip@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Sep 14, 2009 at 4:44 PM

Works! 

Summary: DotNetZip now works with streaming to Mac 

DotNetZip rocks, and nothing compares to it!
Thanks Cheeso!

Sep 15, 2009 at 12:52 PM
Edited Sep 22, 2009 at 10:02 AM

... I wrote of a further problem, but its probably something in my code failing...

Sep 22, 2009 at 11:22 AM

I hope you still have some time to look into it further. There's still a problem in SOME cases.
I tried following the Appnote.txt on the PKArc spec but I'm a bit rusty, and don't have the tools.
BTW from Wikipedia I read that the Stuffit guy (the zip tool for Mac)  is the one who argued with P.K. before he pronounced kaput.

Following is tree:
grandpa
...   pa
.....    son
.......     grandson
...............   a,b,c,d (files)
(in extra folder zipfile only:)
secondRoot
... subdir
.....   subsub1 (and a vast tree of subdirectories)
.........subsub2 (ditto)
.........Subsub3 (ditto)

1. Works: zip from stream, grandpa only (no secondRoot).
2. Works: zip to file same.
3. Fails:    zip from stream with second root.
4. Works: zip from file, with second root

So it seems that's not the error.

I'll continue testing till I fine tune on the error.
And send you.  Could it be that when doing a recursive add dir, somehow again there is the problem of the general bit?

Moshe

Sep 22, 2009 at 1:12 PM
Edited Sep 22, 2009 at 3:22 PM

Hi Cheeso!

Recursive directories add in with no problem.
The problem seems to be with PNG files and some other types of files (yet to be determined).

Could you check the PNG file addition for anomalies, that could cause the Mac to disagree with the PKArk compliance?

Also adding a zipfile to a zipfile. I have an exising zipfile, and it seems that adding it during streaming to outputStream causes a problem.
The same zipfile when zipped with dotNetZip directly to a file has no problem opening on a Mac, and of course none of these problems show up on a PC.
I suppose we are back to the "general info" bit issue - when these types of files are encountered inside the AddDirectory() method..

I'm still checking for other types of files. I had removed MP3 and M4a files. Will return them to the dir and see if they effect the Mac zip..

Thanks! Moshe

 

Coordinator
Sep 22, 2009 at 3:02 PM

Your reports appear to be pretty erratic.

At 722am you concluded the problem is with "second root".  And then later it is PNG files and "some other files".

I'll need you to be much much more specific, and very detailed, if you want me to fix something.  I don't handle PNG files differently, though they are typically STORED and not DEFLATED, because they are already compressed.  You can do something similar by setting the ZipFile.CompressionLevel property.  Or the ZipEntry.CompressionMethod property.

I await your detailed report.

 

Sep 22, 2009 at 3:35 PM
Edited Sep 23, 2009 at 10:11 AM

At 7:22 it was not a conclusion just a recorded difference. I was giving the simptoms. I even negated the "second root" theory on the spot.
I postulated that perhaps something is going wrong in the recursive AddDirectory() when adding nested directories.
Then in the next entry I reported negative to the nested directory theory.

I am now after testing and sure the PNG is the problem, also trying to add an existing zipfile  to the outstream.
 Can I attach the files here so you see the problem for yourself?
I edited the above entry and changed it a bit. Wasn't expecting you to answer till I finished checking.

I checked for problems with periods or hyphens in the filename. Negative. No problem for Mac zip (or PC).
I gotta leave now, but maybe later or early tomorrow I'll be online to complete the checking.

To conclude:
The mac can read a PNG and a ZIP stored inside a zipfile, if that was created directly to a file.
The Mac CANNOT read PNG and ZIP files nested in a directory and added via AddDirectory() and then saved to the OutputStream.

 

 

Coordinator
Sep 22, 2009 at 4:55 PM

You can create a workitem and attach files.

I'll be interested to see the files.

Sep 23, 2009 at 10:13 AM
Edited Sep 23, 2009 at 10:15 AM

How do I make a workitem?

Final conclusions:
 Adding mp3 file, png file and .zip file all cause the zip to fail on the mac (only when done via outputStream).
 Adding .xml, .txt .asf .gif .mp4 or other types does not have this problem.
Could it have to do with filesize? 

I"m offline for next two hours. Will be back and check the file sizes, if there's any consistency.

 

Moshe

Coordinator
Sep 23, 2009 at 3:04 PM

To create a workitem, click on the "Issue Tracker" tab, then click the "Create New Item" link.

Sep 29, 2009 at 8:25 AM

I opened Issue http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8871 
With the problematic directory and a code snippet of Default.aspx.
Take out the problems directory, and it opens OK on a Mac. Put in Any of the files in the Problems directory, and the Mac fails.

Sep 29, 2009 at 10:23 AM

BTW I checked the filesize and it makes no difference.

These files cause a failure, while the other files do not (including .ASF and .XML)

Oct 1, 2009 at 3:00 PM

I am having a similar problem

Folder stucture in not there in MAC

Coordinator
Oct 1, 2009 at 3:28 PM
Edited Oct 1, 2009 at 3:29 PM

that sounds like a completely separate problem.

why don't you open a new thread?

Oct 8, 2009 at 2:18 PM

Cheeso, thanks! Any estimate when you'll get to it? (issue 8871)

For us its urgent, we thought we are almost there, then this bug creeped in.
I wrote a comment there but did not see your response.

Oct 12, 2009 at 2:41 PM

The issue is gone. What's happening?

 

Oct 12, 2009 at 2:46 PM

I mean to say that when I click on the issue, I get a message that it does not exist anymore.

The issue itself, of a problem with certain filetypes zipped with DotNetZip not opening on the Mac, is very much still there.
Any input on this?
Moshe

Coordinator
Oct 12, 2009 at 6:07 PM

It's still there.

http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8871

I haven't gotten to it yet.

Coordinator
Oct 29, 2009 at 11:35 AM
Edited Oct 29, 2009 at 12:10 PM

Moshe, sorry this has taken me so long.

I'm looking at your code, and your zipfile.   The zipfile you had attached to that workitem - where did you get it?  What did you use to create it?  It does not appear to have been written by DotNetZip. Some of the metadata in that zipfile is not generated by DotNetZip. Could not have been generated by DotNetZip. Was never generated by DotNetZip.

Each ZipEntry in the zip file starts with a 4-byte signature 0x504b0304. The next byte is the "version needed to extract". In that zip file, there were entries that had 0x0A in that byte. DotNetZip puts one of two values in that byte: 0x14 or 0x2D. It never inserts 0x0A, and as far as I can recall, has never done so. Is it possible that your zip file, the one you attached to the workitem, was produced by something other than DotNetZip?

When I run your code, I do not get a zipfile structured in the same way.  I get a zip file structured the way I would expect - with 0x14 in that first byte after the entry signature.

And, more interestingly, Is it possible that the problems you are having on the Mac are related to this other thing? The 0x0A byte refers to version 1.0 of the zip specification, which is very old, dating to 1989.  After several iterations on the v1.0 spec, the v2.0 spec was released in 1993.  I think you can agree that v1.0 is very old.

Is it possible that you are using an old, or possibly out-of-date tool on the Mac that is not able to support current, documented features of the zip spec?  Is it possible that the zip files are fine, but the tool you are using on the Mac is the problem?

With DotNetZip I try to produce zip files that are compliant to the spec. I test with 7zip, winzip, Windows compressed folders, perl's IO::Uncompress::Unzip , Java's jar.exe, the visual studio (internal) ZIP dll, InfoZip, MS-Word (.docx), and other tools.  In all cases I get good compatibility, whether I use streaming output or not.  As I told you before, I don't have a Mac, unfortunately, so I cannot test Mac utilities.  But I make a good effort with other tools. 

DotNetZip doesn't insert anything "exotic" or non-compliant in the zip files it generates.  The Unicode stuff is freshest part of the zip spec implemented by DotNetZip - This part was first described in v6.3.2 of the spec, from September 2007.   But because Unicode support in .ZIP files is relatively new, DotNetZip does not use it unless you specifically ask for it, in order to maintain compatibility with the widest set of zip tools. 

Aside from Unicode, DotNetZip uses a few other "semi fresh" options from the zip spec:  NTFS timestamps on file entries, and the "bit 3" streaming support, with a data descriptor of 0x504b0708.  Those things are "relatively new" - meaning, they are about 10 years old, as far as I can tell.

Not long ago I received an uploaded zip file from a Mac user, I don't recall if it was you.  It contained an extra field ID of 0x5855, which was specified as "deprecated" by PKZIP no later than 1997, twelve years ago.  The person told me they had just created this zip file on their Mac.  I then realized that there are tools out there, still in use, that are very much outdated.

Is it possible that this is what's happening in your case? 

You said the problem only occurs when writing to Response.OutputStream .  When that happens, DotNetZip uses "bit 3" streaming support and emits a data descriptor of 0x504b0708, as I mentioned previously.  But apparently this was a source of compatibility problems in days gone by, as indicated by this statement in the zip spec:

Although not originally assigned a signature, the value 0x08074b50 [ignore byte ordering - it's the same sig -Cheeso] has commonly been adopted as a signature value for the data descriptor record. Implementers should be aware that ZIP files may be encountered with or without this signature marking data descriptors and should account for either case when reading ZIP files to ensure compatibility. When writing ZIP files, it is recommended to include the signature value marking the data descriptor record. When the signature is used, the fields currently defined for the data descriptor record will immediately follow the signature.

If the problem is in the zip tool on the Mac, you should be able to verify it by using a more recent tool, or by transferring the zip file to a different platform and testing and opening the zipfile there.   This would confirm that the problem is with the tool on the Mac, not with the zipfile itself.  If that's the case, then you have two workarounds:  don't use streaming output, OR, use a more up-to-date tool on the Mac.

----

Moshe, the upshot here is that I still don't see a problem in the zipfiles generated by DotNetZip.  I looked very closely at them, as you can tell from my comments above.  I may be missing something, but I don't think so.  Without some clearer indication of a real problem, there's nothing for me to fix or correct in DotNetZip. 

 

Oct 29, 2009 at 5:25 PM

>Sorry for
I'm the one to apologize for "making trouble" on a free tool. I wish to thank you again for creating and supporting  this great code.
>...could not be generated from dotnetzip...
In fact, it was.  I had run the attached code, and saved the stream to a file. opened OK on PC, OK, failed on Mac.

>... When run with your code... I get
I'll try the latest DotNetZip with same code and same files, and re-post.

>... bit 3... outdated .
Yes it was me. Even if it's correct that the problem is with the ziptool on the Mac, that's what's out there, and working! 
These are relatively (2 years old) new mac machines and we have thousands of students using them, so we support Mac users too.
Would it be possible to work with the option not used currently? (I'm confused about what to request, but I'm asking if you can change it from the way it is working now?)
After the file I gave you, you changed the directory to work without the 3 bit (or with it... not sure) and that caused the zip to open OK on the Mac
(As long as there was no Gif, Zip or mp3 file inside).

So, two things: a. I'll make a zip with the latest code.   b. I'm asking if you can do a change relating to the bit 3, in the file headers (best if its optional).
That would give Mac support "out of the box".

Thanks again, Moshe

Coordinator
Oct 30, 2009 at 10:23 PM

I hear what you're saying but I don't think the solution is as simple as "do a change in bit 3". 

First, I'm not certain that's the problem.  It was a guess. Something to test out. 

Second, if that's the problem, then the solution is not to break the correct software.  It's to fix the thing that's really broken.  So far I think it's not DotNetZip.  If I change what works for every other tool, to make something that works for one broken tool, then of course I've got a maintenance problem on my hands. The things you are asking for - and again, this is based on a *guess* of mine, I still don't have a definitive proof that the thing I suggested is what the real problem is - these things were changed in the zip specification 15 years ago or more.  What do I do about all the tools that do the right thing?   How many of them will break, if I make a change to DotNetZip in this way?

Now Add to those things, the confusion around the zip file.  I don't see how the zip file you posted could have been generated by DotNetZip.  You can tell me it was, but the code in DotNetZip is not capable of generating what I downloaded.  If you can generate that zip again, and show me how to do the same with DotNetZip, I'd be interested in that.  Right now it sure looks like that zip file was generated by a very old tool. 

So no, at this point I'm not going to "do a change in bit 3". I have no confidence that's the right thing to do.

-----

I've had similar situations, in particular dealing with zip64, and there was a guy who showed me why DotNetZip was wrong.  It took a few exchanges back-and-forth, but in the end he showed me what was wrong and I was able to fix it.  In this case, I don't see what is wrong in DotNetZip.  You haven't persuaded me.

> Even if it's correct that the problem is with the ziptool on the Mac, that's what's out there, and working!

But if what I understand is correct, it's not working.  It won't work with *any* compliant tool that streams the output.  It (apparently) works with zip files that have not been streamed.   Have you tested this?  If you can show me other tools that produce streamed files that work fine on the Mac as well as Windows, that would help convince me that DotNetZip is in need of a change.  

Seems like these are your options:

  • do the research and present to me exactly what the problem is, or make it simple for me to duplicate the problem on my own.  Does SharpZipLib cause this problem?  (it uses a streaming interface exclusively, I think)  What about commercial ZIP libraries?  If a different tool or library doesn't produce zips that cause this problem, that would indicate that the problem is in DotNetZip.  If there's a problem, I'll fix it.
  • If the research you do shows the problem is in the Mac tool, get the manufacturer that produced the zip tool on the Mac to fix it.
  • consider using a different tool or library.  This will solve the problem, if the problem is in fact in DotNetZip, or if it is in the Mac, and if it is just too hard to prove that. 
  • if in your research you find that it is the mac tool that is broken, and cannot convince the manufacturer to fix it, or you don't want to, and you don't like the workaround, then maybe you should consider maintaining yourself, a special version of DotNetZip for your users.
  • I'm not sure but I think you have a workaround: don't use the streaming interface of DotNetZip.  Save the zip file to to a regular filesystem file. 

 

Nov 1, 2009 at 8:35 AM
Edited Nov 1, 2009 at 11:21 AM
Could you just tell me where to look in the source code, so that I can try to  "change the bit 3" myself, and test it, that way won't waste your time, and I'll tell you the results, you could then decide how to continue.
True, it was a guess, but (a) it worked for directories, and now at least streamed zipfiles without a PNG file in them DO open on the mac, which did not before. and (b) the guess was based on findings in another (not DotNetZip) tool, that had a similar problem.

It won't break anything for the other tools especially on the PC, since it will be an option for Mac compatibility. You can call the option UseMacOutdatedIncompatibleAndStupidlyChosenDefaultToolFormat .

You are correct that saving a file to the server has none of these problems, but saving a file on the server is not an option, not for me, and not for the others asking this question, as I proved at length, and as some other people have confirmed here.

AFAIK, Mac support is important for many businesses, who have clients on macs, and MS has been promoting some kind of Mac support.
We are MS partners,  we make software used by university students, most of whom use PCs with MS OS - Window et al.  Still we need to support Mac users.

I realize what you say, that I can always use another tool. I just hoped that as per last time when the issue seemed solved, since the bit 3 solved the directory extra info, it would solve the file problem too.

Moshe
Coordinator
Nov 1, 2009 at 3:14 PM
Edited Mar 15, 2010 at 7:17 PM

> True, it [is] a guess... You can call the option UseMacOutdatedIncompatibleAndStupidlyChosenDefaultToolFormat .

I won't add stuff based on a guess.  This is not how I do development.  *You* could do it. 

> it worked for directories...

No - i made the change on directories after looking very closely at the format.  I found a change that needed to be made according to the spec.  I made the change to comply with an edge case in the spec, and that solved a problem.  What you suggest here is entirely different.  You are suggesting I go against the spec.

> Still we need to support Mac users.

I've no problem supporting Mac users.  I cannot do it on my own.  I cannot do the tests.  I've suggested that you do the tests. 

Moshe, this discussion isn't working.  I'm repeating myself.  It's always the same.  I've repeatedly suggested that you do the research, figure it out, and explain it to me.  If you show me the clear problem, I can and will fix it.  If you just say "it doesn't work", there's nothing I can do with that information. I've repeatedly made guesses, and suggestions to you.  From what I can tell, you don't seem to want to do the research. That's up to you. But I can't "fix" something you haven't described clearly.  I'm going to stop repeating myself now.  

> Could you just tell me where to look in the source code, so that I can try to

Search for "BitField" in the source.   You'll have to deal with both reading and writing zip files. In other words, you want to insure that a zipfile generated by DotNetZip can also be read by DotNetZip.  Currently DotNetZip reads compliant zip files. I don't have any good ideas for how to get it to successfully read the non-compliant files that are generated the way you suggest.

Or, send me a MacBook, and I'll do the tests myself.

 

Nov 3, 2009 at 4:24 AM

I've verified that bug repros on all versions of OSX, including Snow Leopard with the Tue-10-27-2009--33136.29 build.

If my testing is correct, the problem is that the default zip handler in OSX doesn't work right unless the entry header precedes the compressed content.  In my case, I can tolerate the perf of doing two compression passes per file so that this can work.  My plan is to pass a null stream to ZipEntry::Write inside ZipFile::Save as a first pass, plus some small tinkering inside ZipEntry::Write to use the pre-calculated header data instead of falling back to the old path.

Does this sound reasonable?  Or are there potential side-effects I'm unaware of?

Nov 3, 2009 at 8:16 AM
Edited Nov 3, 2009 at 8:32 AM
:-) 
Will do... (the testing, not the sending of a macbook yet).

Moshe


Nov 3, 2009 at 8:31 AM

John (Ketchpaw), did you check if you CAN open a streamed zipfile if there are no PNG, ZIP or MP3 files inside?

It worked with the fixed version Cheeso gave me (see above in this forum entry search for "Works!").

For that reason, I doubt that the problem is the entry order, unless when adding a PNG file the order of entry vs data changes?

Moshe

 

Coordinator
Nov 3, 2009 at 11:04 AM

John, what you describe sounds reasonable,.  It might be easier to just NOT save to a streamed output when the zipfile is going to be used by a machine running OSX. 

If I were doing this, I would also test other zip libraries that generate streamed output, like SharpZipLib.  If it really is the case that OSX cannot tolerate the data descriptor (bit 3, streamed output), then I would also raise a bug with apple. 

I'll be interested in what you find.

Nov 6, 2009 at 7:45 PM

Turns out my proposed fix is not necessary.

The problem I ran into the other day is that OSX seems to additionally not support CompressionLevel.None.  We set that because we're working with large video files, but it's not really a performance issue.  If i set it to any other value, it's fine.  Could you add that to the intellisense for .None?

Coordinator
Nov 7, 2009 at 9:23 AM

Ok just clarifyng.

are you asking me to add to the documentation for CompressionLevel.None, that Mac OSX doesn't like it?

It sounds like a good thing to include in the doc. I Want to be sure that's what you're asking for.

a second q - you said your proposed fix is not necessary.  What do you need to do, then, in order to get a zip file that works for Mac OSX ?

Nov 8, 2009 at 4:28 AM
In the docs would be great, adding it to intellisense would be even better since it's such a nuanced thing.

I just wanted to be specific: the checkin you've already made was enough to fix compat. My "double pump" idea is what's unecessary.

-john

On Nov 7, 2009, at 2:24 AM, Cheeso <notifications@codeplex.com> wrote:

From: Cheeso

Ok just clarifyng.

are you asking me to add to the documentation for CompressionLevel.None, that Mac OSX doesn't like it?

It sounds like a good thing to include in the doc. I Want to be sure that's what you're asking for.

a second q - you said your proposed fix is not necessary. What do you need to do, then, in order to get a zip file that works for Mac OSX ?

Coordinator
Nov 8, 2009 at 4:34 AM

John - what checkin fixed it?  I'd like to know what problem I fixed, and when....

The intellisense - this is what shows up when you hover, right?  I guess it's the summary of the property, right? 

Nov 9, 2009 at 10:29 PM
http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8770

-john


On Sat, Nov 7, 2009 at 9:34 PM, Cheeso <notifications@codeplex.com> wrote:

From: Cheeso

John - what checkin fixed it?  I'd like to know what problem I fixed, and when....

The intellisense - this is what shows up when you hover, right?  I guess it's the summary of the property, right? 

Read the full discussion online.

To add a post to this discussion, reply to this email (DotNetZip@discussions.codeplex.com)

To start a new discussion for this project, email DotNetZip@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Nov 19, 2009 at 11:19 AM

My issue is closed. It works with Mac (version 1.9.0.30 if you use the reduced version).
http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8871

I have not tested with "Comperssion.None" and have not tested with full version.

Thanks Dino (aka Cheeso)! DotNetZip is great!

Coordinator
Nov 19, 2009 at 5:00 PM
Edited Mar 15, 2010 at 7:19 PM

Moshe, thanks for the compliments. I'm glad it's working for you.

You had problems opening zip files on the Mac, in some cases, when the zip files were generated by this library.   You now report that the problems are resolved, "if you use the reduced version" of the library.  But you yourself point out that you did not test the full version of the library, so the conclusion you draw - "the reduced version fixes it" - is unwarranted.  How would you know, if you didn't test the alternative?

Beyond the fact that your own tests don't support the conclusion you drew, I am 100% certain that the problems you saw are completely independent of the use of the "full" or "Reduced" DotNetZip library.  The two are the same, except that the full library include extra methods you did not call, dealing with the creation of self-extracting archives.  If you create a normal zip, which is what you were doing, your application will follow EXACTLY the same code paths in DotNetZip, whether you use the "reduced" library or the "normal" library.  There is no difference in code, and the zip files will be exactly equivalent.

I still don't understand the problems you had.  We went through many exchanges, and I found and fixed one clear problem in DotNetZip: http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=8770   This fix was included in v1.9.0.6.  One other user confirmed that this fix corrected his compat issues.  But you reported that v1.9.0.6 did not fix your problem.  

It remains a mystery.   I believe you were having problems, I just don't know enough to conclude that the problem was in DotNetZip.

In summary, the statement that "using the Reduced version fixes the problem" is not correct.  

On the other hand, v1.9.0.30 is the most tested release of DotNetZip ever, and it does include a fix for what I believe was an issue Mac compatibility  in the generated zip files.

Thanks for all the effort!

 

 

Dec 22, 2009 at 4:42 PM

Cheeso,

I was having the same problem on the Mac using v1.8, but downloading v1.9.0.31 solved this problem for me.  I was able to build a zip file on the fly and save it to the OutputStream and the Mac Unarchiver as well as Stuffit was able to open the zip file without any problems.  Thanks for making the fix.

Note: The zip file opened without having to change any of the default time formats.  I tested the zips using Safari and Mac OS X 10.6.2

Coordinator
Dec 22, 2009 at 5:36 PM

Terrific!  Thanks for the feedback.

 

Coordinator
Mar 15, 2010 at 8:04 PM

Just following up on this.  I think Mac's may have a problem opening ZIP files generated with "bit 3" set.  Bit 3 can eb set when saving the zip file directly to the Response.OutputStream, or any non-seekable output stream. 

The way to get a zip file without bit 3 is to write the zipfile to a seekable stream.  Two examples of this are:  a MemoryStream, or a FileStream (representing a filesystem file). 

This is worth a try.  If you use the   MemoryStream idea, then your code in VB looks like this:

Private Sub ZipFolder(ByVal FolderToZip As String)
    Response.Clear()
    Response.BufferOutput = False

    Dim archiveName As String = _
        Path.GetFileName(FolderToZip) & "_" & (DateTime.Now.ToString("yyyyMMMdd_HHmmss")) & ".zip"

    Response.ContentType = "application/zip"
    Response.AddHeader("Content-Disposition", "filename=" & chr(34) & archiveName & chr(34))

    Using ms as New System.IO.MemoryStream()
        Using zip As ZipFile = New ZipFile()
            zip.AddDirectory(FolderToZip)
            '' save the zip fiel to a memory stream
            zip.Save(ms)
        End Using
        '' now copy the memory stream to the Response OutputStream
        ms.Position = 0
        dim b(1024) as Byte
        dim n as new Int32
        n=-1
        While (n <> 0)
            n = ms.Read(b,0,b.Length)
            If (n <> 0)
                Response.OutputStream.Write(b,0,n)
            End If
        End While
        Response.Close
    End Using
    Response.Close()
End Sub

In this case, the entire contents of the compressed zip will be placed into memory, before writing it out to Response.OutputStream.  This may or may not be acceptable to you, depending on how large the zip file is.

If you use the FileStream idea, then your code would look like this:

Private Sub ZipFolder(ByVal FolderToZip As String)
    Response.Clear()
    Response.BufferOutput = False

    Dim archiveName As String = _
        Path.GetFileName(FolderToZip) & "_" & (DateTime.Now.ToString("yyyyMMMdd_HHmmss")) & ".zip"

    Response.ContentType = "application/zip"
    Response.AddHeader("Content-Disposition", "filename=" & chr(34) & archiveName & chr(34))
    Dim tempfile As String = "c:\temp\" & archiveName
    Using zip As ZipFile = New ZipFile()
        zip.AddDirectory(FolderToZip)
        '' save the zip fiel to a memory stream
        zip.Save(tempfile)
    End Using
    ' open and read the file, and copy it to Response.OutputStream
    Using fs as System.IO.FileStream = System.IO.File.OpenRead(tempfile)
        dim b(1024) as Byte
        dim n as New Int32
        n=-1
        While (n <> 0)
            n = fs.Read(b,0,b.Length)
            If (n <> 0)
                Response.OutputStream.Write(b,0,n)
            End If
        End While
    End Using
    Response.Close
    System.IO.File.Delete(tempfile)
End Sub

Try one of those and see if it works for you.

 

Coordinator
Mar 15, 2010 at 8:16 PM

The corresponding code in C# would look like this. For the workaround that uses MemoryStream :

private void EmitZip()
{
    Response.Clear();
    Response.BufferOutput= false;

    System.Web.HttpContext c= System.Web.HttpContext.Current;
    String ReadmeText= String.Format("README.TXT\n\nHello!\n\n" +
                                     "This is a zip file that was dynamically generated at {0}\n" +
                                     "by an ASP.NET Page running on the machine named '{1}'.\n" +
                                     "The server type is: {2}\n"+
                                     "The password used: '{3}'\n" +
                                     "Encryption: {4}\n",
                                     System.DateTime.Now.ToString("G"),
                                     System.Environment.MachineName,
                                     c.Request.ServerVariables["SERVER_SOFTWARE"],
                                     tbPassword.Text,
                                     (chkUseAes.Checked)?EncryptionAlgorithm.WinZipAes256.ToString() : "None"
                                     );
    string archiveName= String.Format("archive-{0}.zip", DateTime.Now.ToString("yyyy-MMM-dd-HHmmss"));
    Response.ContentType = "application/zip";
    Response.AddHeader("content-disposition", "inline; filename=\"" + archiveName + "\"");

    using (var ms = new System.IO.MemoryStream())
    {
        using (ZipFile zip = new ZipFile())
        {
            // the Readme.txt file will not be password-protected.
            zip.AddEntry("Readme.txt", ReadmeText, Encoding.Default);
            if (!String.IsNullOrEmpty(tbPassword.Text))
            {
                zip.Password = tbPassword.Text;
                if (chkUseAes.Checked)
                    zip.Encryption = EncryptionAlgorithm.WinZipAes256;
            }

            // filesToInclude is a string[] or List<String>
            zip.AddFiles(filesToInclude, "files");

            zip.Save(ms);
        }
        ms.Position = 0;
        var b = new byte[1024];
        int n;
        while ((n = ms.Read(b,0,b.Length)) > 0)
            Response.OutputStream.Write(b,0,n);
    }
    Response.Close();
}

For the workaround using the FileStream :

private void EmitZip()
{
    Response.Clear();
    Response.BufferOutput= false;

    System.Web.HttpContext c= System.Web.HttpContext.Current;
    String ReadmeText= String.Format("README.TXT\n\nHello!\n\n" +
                                     "This is a zip file that was dynamically generated at {0}\n" +
                                     "by an ASP.NET Page running on the machine named '{1}'.\n" +
                                     "The server type is: {2}\n"+
                                     "The password used: '{3}'\n" +
                                     "Encryption: {4}\n",
                                     System.DateTime.Now.ToString("G"),
                                     System.Environment.MachineName,
                                     c.Request.ServerVariables["SERVER_SOFTWARE"],
                                     tbPassword.Text,
                                     (chkUseAes.Checked)?EncryptionAlgorithm.WinZipAes256.ToString() : "None"
                                     );
    string archiveName= String.Format("archive-{0}.zip", DateTime.Now.ToString("yyyy-MMM-dd-HHmmss"));
    Response.ContentType = "application/zip";
    Response.AddHeader("content-disposition", "inline; filename=\"" + archiveName + "\"");

    string tempfile = "c:\\temp\\" + archiveName;
    using (ZipFile zip = new ZipFile())
    {
        // the Readme.txt file will not be password-protected.
        zip.AddEntry("Readme.txt", ReadmeText, Encoding.Default);
        if (!String.IsNullOrEmpty(tbPassword.Text))
        {
            zip.Password = tbPassword.Text;
            if (chkUseAes.Checked)
                zip.Encryption = EncryptionAlgorithm.WinZipAes256;
        }

        // filesToInclude is a string[] or List<String>
        zip.AddFiles(filesToInclude, "files");

        zip.Save(tempfile);
    }
    using (var fs = System.IO.File.OpenRead(tempfile))
    {
        var b = new byte[1024];
        int n;
        while ((n = fs.Read(b,0,b.Length)) > 0)
            Response.OutputStream.Write(b,0,n);
    }
    Response.Close();
    System.IO.File.Delete(tempfile);
}