Unable to (PK)unzip on IBM Mainframe

May 21, 2009 at 7:05 AM

I have tried the v1.7 zip classes to pack huge txt files.
It works fine on any win PCs and with any unpacking apps.

The problem we are facing is on Mainframe:
The file cannot be unpacked with “corrupt file” error message…

The older version v1.5 created correct files for the MF, but I changed to newer version to use the events on saving to notify the user on zip progress (in SaveProgress:Saving_EntryBytesRead)
Now the file must be unpacked and repacked with PKZIP (on an XP PC) to be able to unzip on MF's PKUNZIP...

Please help me to solve the problem:
- How can I configure the v1.7 (or above) to create compatible zip files for MF (IBM437) like older PKUNZIP or like the v1.5 does or
- How can I use the v1.5 to catch events like Saving_EntryBytesRead

Coordinator
May 21, 2009 at 12:52 PM

Hello V, thanks for using the forums.

Does the problem occur with all zips, of any size?  Or does it require a certain number of zipped entries, or a certain size of zip file?   What size?

Can you tell me about the file entries in the zip?  how large are they?  The filenames - are they ASCII or do they have filenames with extended characters that require a separate code page?

Also, what do you mean by IBM437?  The normal encoding for the mainframe is IBM437. Unless you take special action, the encoding used for the file should be IBM437.  Are you specifying an encoding in the application code?  How are you zipping the files, anyway?  With a custom application?  Or with the DotNetZip command-line tools?  Or something else?

Can you open the zip files in Windows Explorer?  Apparently you can open these files in pkunzip on the PC, is that correct?  So the zipfiles are valid and readable, in that case. But not readable on the mainframe.   Have you contacted PKZIP about the problem?  If the zipfile is readable by PKUnzip on the PC , but not readable by PKUnzip on the Mainframe, then the problem appears to be not with the zip file itself, but rather with the PKUnzip on the Mainframe.  Do you agree?

Nevertheless, I am willing to take a look at the problem to better understand it, and maybe figure a workaround.   Is it possible for you to give me a sample of a zip file that works, and a sample of a zip file that does not work?   I can create a workitem for you where you can post the files, if you are willing and able to share those files.  If you cannot share those files with me, then I am may not be able to help you further.  

 

May 21, 2009 at 1:47 PM

Hi Cheeso,

Yes, any size. Only one entry and no metter of size. ex: 4.6MB txt (Ansi enc. > the MF requires very "basic" charset) to ~100KB zipped.
The file name: CBL_AE_042009_Imp.txt. No ext chars,
and as I said, PKUNZIP (from the last century:1993) on PC and PKZIP re-zip creates a similar zip file, except the MF can process (PKUNZIP) it...
No special encoding, PKUNZIP (on PC) and re-zip does not alters the entry data, so I think there's no problem with the file to be packed.
Just create a zip with v1.5 and v1.7 with same compression level! I guess they will be (binary) different.
application code
 for v1.7:
 zipFile = new ZipFile(zipPath);
   try {
    zipFile.CompressionLevel = CompressionLevel.BEST_COMPRESSION;
    zipFile.SaveProgress += new EventHandler<SaveProgressEventArgs>(zipFile_SaveProgress);
    zipFile.UpdateFile(sfdMain.FileName, string.Empty);
    zipFile.TempFileFolder = Path.GetTempPath();
    zipFile.Save();
    return true;
   }
 for v1.5:
 zipFile = new ZipFile(zipPath);
 zipFile.AddFile(sfdMain.FileName, string.Empty);
 zipFile.TempFileFolder = Path.GetTempPath();
 zipFile.SaveStarted += new SaveStartedEventHandler(zipFile_SaveStarted);
 zipFile.SaveCompleted += new SaveCompletedEventHandler(zipFile_SaveCompleted);
 zipFile.Save();
 return true;

Yes, on win pcs IE and all applications can read, unpack those files.
Yes the zip file is valid (not corrupt), but I think there is some encoding problem.
The strange is the v1.5 has less feature but more compatibility.
I have tried to set zipFile.UseUnicodeAsNecessary = false and not helped.

I can post files, but first please check the v1.5 vs v1.7 zip files with an ansi text file. I think you could have test files by then.
If this "theory" is dead-end I will post samples...

Thanks for you kind help

regards Zsolt

Coordinator
May 21, 2009 at 2:52 PM

What is the problem you get in PKUnzip on the mainframe? Have you reported this problem to PKWare? What did they say?

In DotNetZip v1.7, the library will use IBM437 encoding unless it is necessary to use something else.  It is necessary only when the filenames include characters that cannot be represented in the IBM437 code page. So I suspect that encoding is not the problem.

Try this test :  set the zip.CompressionLevel to CompressionLevel.NONE, and CompressionLevel.DEFAULT, and see if you get the same results.

Does the problem occur only when updating a zipfile? Or does it also occur when creatiing a new zipfile? Can you test that, too?

 

May 21, 2009 at 9:42 PM

Sorry, I have no access to the MF. The error was someting like this: the file is corrupt. It was just reported to me.
I am at the dev site and hard to test. That's why I suggested to create files with both versions and compare.

No I did not contacted to PKWare. Because the older version works, I thought it's about the changed defaults or more functionalities resulted different behavior.

Sorry, I've tried all compression level (inc DEFAULT eq level 6) except NONE. I will...

I thought update will be faster when I try to compress the same file (because of checking the source and the target).
Earlier I used the Add method, and in 99% the app should create a new file, so yes, I can't create compatible file with v1.7.

I will create zip files with both versions on same comp level, compare them and let you know.

Coordinator
May 21, 2009 at 10:17 PM
Edited May 21, 2009 at 10:34 PM
Because the older version works, I thought it's about the changed defaults or more functionalities resulted different behavior.

I'm not so sure that is a valid conclusion. It seems that the zip file is readable everywhere except the mainframe. Clearly there is some negative interaction between the DotNetZip v1.7 and the mainframe pkunzip. But there is no such negative symptom on any other platform or zip tool, on the PC, when the zip file is created by DotNetZip v1.7. Pkunzip on the PC can read the zip file. Windows Explorer can read the zip file. Only Pkunzip on the mainframe cannot read the file. To me this says that pkunzip on the mainframe is not properly handling a valid zip file.

The zip files created by DotNetZip conform to the PKZIP specification. There are provisions in the specification to allow backward compatibility. In other words, when a ZIP file uses a "new feature", older tools should be able to gracefully degrade their behavior. For example, an older tool or library can detect that the header code that indicates that Unicode is in use, is out of the range it can handle. In this case the older tool should emit an error stating that.

On the other hand, it is possible that older tools don't behave well with header codes they do not understand. In that case the older tool needs to be updated.

Edit: I had originally written:

One change for v1.7 beyond v1.5 is the timestamp data. After Windows NT was released, PKWARE extended the zip spec to include high-resolution timestamps, compatible with NTFS. This feature has been in the specification since, oh, I guess 1996. But DotNetZip added support for it in v1.7.  There are other changes, but this is a key one.  One possible cause fo the problem you are seeing  is that the high-resolution time stamps embedded into the zip file by DotNetZip v1.7 are not handled well by the Mainframe pkunzip.  This could cause pkunzip to report the zip file as invalid.

But this is not correct - the change in the timestamps came in v1.8.  There is a similar change though, that came in v1.7 - that dealing with the ZIP64 extension to the ZIP file.  ZIP64 was added after the year 2000, so I am sure it is not handled by your ancient MF pkunzip. On the other hand, your ZIP files do not use ZIP64. But, as part of the new ZIP64 supported that was added in v1.7 of DotNetZip, DotNetZip produces its files with "dummy" header codes.  Normal ZIP applications can and should safely ignore these dummy codes.  But if a zip tool is overly rigid about handling unknown header codes, it could report the zip file as invalid, as I described above. 

A technically easy solution to this is to replace the pkunzip on the mainframe with a more modern version that supports the high-res timestamps, or a more resilient version that ignores header codes it does not understand. This is why I asked you if you contacted PKWare.  I understand this proposed solution may not be so easy in practice.

On the hunch that it is the timestamps causing the difficulty, I can propose to build a patched v1.7 of DotNetZip for you, which does not emit the high-resoultion timestamps dummy header codes into the zip file.  You can try creating a zip archive with that version, and see if the mainframe likes it.  what do you think?

 

Coordinator
May 21, 2009 at 10:43 PM

Now that I looked at the code again, the dummy header code would be in there only if you used a streaming approach when you created the zip file.

Can you show me the DotNetZip code you use to create your zip files?

May 22, 2009 at 9:58 AM

I think there is no chance to replace the PKUNZIP on the MF...

Please tell me which part to show in which file? I am using Ionic.Utils.Zip.dll (v1.5.2.1) I can send you all cs files.

However I've created zip files with both version without compression and there are two bytes differs (case sens; bin compare)
After the entry at the end of the file at the "footer" from offset 5 and 37:

offset 1-5:
- v1.5:(P K=50 4B 01 02 14) <> v1.7:(50 4B 01 02 00)

offset 36-43
- v1.5:(00 01 00 20 00 B6 81 00) <> v1.7:(00 00 00 20 00 B6 81 00)

Coordinator
May 22, 2009 at 1:14 PM

Thanks for the diffs. 

I'm not sure if those differences would cause a problem. We can test it.  I will make available a modified v1.7 assembly that sets those bits to the values in the 1.5 assembly.

Also, I have another idea - can you try DotNetZip v1.8?  There is an option to set the compression strategy on the ZipFile, in v1.8, via the ZipFile.Strategy property.  This property affects the output compressed bytestream, and it could be that the pkunzip on the mainframe does not understand the compressed bytestream using the default strategy.  Try creating a zipfile with CompressionStrategy.HUFFMAN_ONLY and CompressionStrategy.FILTERED  , and see if the MF can unpack those zip files.  You should set the strategy before adding any entries to the ZipFile instance.

     using (var zip = GetZipFile())
     {
         zip.Strategy = Ionic.Zlib.CompressionStrategy.HUFFMAN_ONLY;
         //zip.CompressionLevel = Ionic.Zlib.CompressionLevel.BEST_COMPRESSION;
         zip.UpdateFile("Whatever.txt", "");
         //zip.TempFileFolder = Path.GetTempPath();
         zip.Save(zipPath);
     }

 

Coordinator
May 22, 2009 at 1:54 PM

ok, here is the workitem http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=7801

there is a zipfile attached to that workitem.  It contains a modified v1.7 DLL, that I would like you to try.  There's a new method :  ZipEntry.MarkAsText().

Please use it this way:

     using (var zip = GetZipFile())
     {
         //zip.CompressionLevel = Ionic.Zlib.CompressionLevel.BEST_COMPRESSION;
         var e = zip.UpdateFile("Whatever.txt", "");
         e.MarkAsText();
         //zip.TempFileFolder = Path.GetTempPath();
         zip.Save(zipPath);
     }

Then, see if your MF PKunzip can read the resulting zip file.

 

May 22, 2009 at 11:12 PM

test result for v1.7 MarkAsText:

only one different in the header and footer:

at offset 14 (offset 1-2: P K=50 4B...)

v1.5: 44 <> v1.7a: 4C

I cant send to test on MF right now; and I will test the v1.8 later and let  you know.

thanks anyway

May 25, 2009 at 8:27 AM

The MarkAsText works fine: on the MF it can be unpacked!

The v1.8 HUFFMAN_ONLY creates low compression level files, which is bad on slow network :(. I did tried it.

I have tried Compression Strategy : FILTERED which creates "incompatible" files too.

I think the solution is arround those two bytes at offset 5 and 37. In the "FILTERED strat" the byte at offs 37 is still 00 (at 5>2D)...

With MarkAstext those bytes are identical to the bytes in the file created with v1.5.

I can use this solution; the compression is at max and there are feedbacks during the progress.

Thanks

Coordinator
May 25, 2009 at 8:57 AM
Edited May 25, 2009 at 9:17 AM

Glad to hear the MarkAsText worked for you.

NB: you are using a custom, one-off version of the v1.7 library.  The MarkAsText capability will be added to v1.8, but after looking at it further, it made sense to make it a Property.  If you upgrade to v1.8 in the future, you'll need to modify your code to use e.IsText = true, rather than MarkAsText().  It will work the same, and you will get compatible ZIP files.

Glad it helped.

Remember, DotNetZip is donationware!

http://cheeso.members.winisp.net/DotNetZipDonate.aspx