AddEntry or UpdateEntry causing some kind of corruption?

Jan 7, 2010 at 5:22 PM

I am creating a Windows 7 sidebar gadget to interface with our ASP.NET application. The user goes to a page in our application and fills out some fields then presses a button and we stream the gadget to them (it's just a zip file with .gadget file extension). I need to store some data in a text file (ex. the URL to use to get back to the ASP.NET application, an encrypted user ticket, etc) within this archive, which is why I'm generating it dynamically (I call this file Settings.txt). I have tried a number of methods to do this but every time I try to install the gadget I get an error message "Not a valid gadget package."

If I rename the file to .zip and unzip it this works fine. Then I can re-zip it using WinRAR and rename it back to .gadget and this works with no problem. If I don't try dynamically generating Settings.txt then it also works fine (for example, if I put my own Settings.txt into the template directory and simply zip all files in the directory then it works fine). Here is my code:

    Response.Clear()
    Response.ContentType = "application/zip"
    Response.AddHeader("content-disposition", "attachment; filename=Test.gadget")

    ' Zip the contents of the gadget template directory
    Using zip As New ZipFile(Encoding.UTF8)

        zip.CompressionLevel = Ionic.Zlib.CompressionLevel.None

        Dim di As New DirectoryInfo(strPath)

        ' Add all the sub-directories to the zip archive
        For Each item As DirectoryInfo In di.GetDirectories
            zip.AddDirectory(item.FullName, item.Name)
        Next

        ' Add all the files to the zip archive
        For Each item As FileInfo In di.GetFiles
            zip.AddFile(item.FullName, "\")
        Next

        zip.AddEntry("Settings.txt", "\", GenerateSettingsFile(), Encoding.UTF8)

        zip.Save(Response.OutputStream)

    End Using

    Response.End()

I've also tried pre-zipping all files (including an empty Settings.txt file) and then opening the archive and updating the contents of Settings.txt before streaming the file down, which also does not work.

    zip.UpdateEntry("Settings.txt", "\", GenerateSettingsFile(), Encoding.UTF8)

I have also tried writing the Settings.txt file to a different location, then adding that to the archive:

    Dim sw As New StreamWriter("C:\Settings.txt", False, Encoding.UTF8)
    sw.Write(GenerateSettingsFile())
    sw.Close()

    zip.AddFile("C:\Settings.txt", "\") 
I've tried using different encodings and even writing an empty Settings.txt file (ex. replacing GenerateSettingsFile() with ""). I'm not sure how to troubleshoot this or compare the differences between the dynamically generated zip file versus using WinRAR. Does anyone have any ideas on how to approach this?

Coordinator
Jan 7, 2010 at 6:10 PM

The first thing I think of is "bit 3".

When DotNetZip writes to a non-seekable output stream, like Response.OutputStream, it formats the zip differently than when it writes to a seekable stream, like a filesystem file. The metadata surrounding the compressed entry data is shaped differently.  The ZIP specification makes allowance for this. 

Metadata is data about the data.  when I talk about metadata In the zip file, I refer to the information about the individual zip entries.  The filename, the timestamp, the compressed and uncompressed file sizes, the CRC.  All of this is data about the file squished into the zip.

There are some metadata that are not knowable until the entry is compressed - specifically CRC and compressed size. Even the uncompressed file size is not knowable until the library reads through the entire thing-to-be-compressed.  But, in the "normal" zip file structure, all of this metadata lies before the actual compressed data.  The way most libraries make this possible is they write dummy metadata, then write the compressed blob, then Seek back to the location of the metadata and write the actual metadata.  Are you following?

This is possible only when the output stream is seekable. PKWARE modified the zip spec to allow writing zip files to non-seekable output streams (like standard output, or Response.OutputStream). There's a single bit that needs to be set in the up-front metadata, "bit 3", and that signals that the three amigos (CRC, Compressed and Uncompressed size) will be placed immediately following the compressed blob.  This is the format that DotNetZip uses when writing to Response.OutputStream.  It's legal and compliant and some zip libraries don't handle it very well.  The archive tool on the Mac is one such example.  I've seen others.

If this is the cause of the problem, it would be consistent with your observations - that any time you edit and save the zip (.gadget) file manually, using a zip tool, it works.  This is because when re-saving a zip file to a filesystem file, a zip tool (like WinRar) can simply unset bit 3, and put the three amigos where they normally reside.

So... I suggest that you write the .gadget to a filesystem file, from ASP.NET, then send *that* down to the system.  Doing it this way will avoid bit 3, and it may solve your problem. It introduces a new problem though: how to manage the temp file generated by ASP.NET.  That normally isn't a big deal; just delete the temp file after streaming it.  You must not call Response.End() before attempting to delete the file.  Response.End() throws an exception (maybe you didn't know that), and any code following Response.End() won't be executed.  So it's important to use Response.Close() if you have meaningful code that you want to execute after  completing the communication with the requesting system.  

The other option you can use, if the .gadget file is small - write to a memory stream.  This avoids the creation of a temp file, but instead creates the zip in a memory blob that is held in memory in its entirety.  This works if the .gadget file is small.  A MemoryStream is seekable, so here again you wold avoid the bit 3 bogeyman. After saving to the stream, just seek to the beginning of the MemoryStream, and then write the contents of *that* stream to the Response.OutputStream.  Obviously, this will not work if the zip file is very large.   How large is large? that's up to you to decide.

Good luck.

Jan 7, 2010 at 6:29 PM

Thanks very much Cheeso, that was exactly it! Changing my code to use a MemoryStream worked:

    Dim mem_stream As New MemoryStream
    zip.Save(mem_stream)

    Response.BinaryWrite(mem_stream.ToArray())
The archive is small IMO, less than 30kb (with no compression) so I think this solution will be fine.