Opening a zip stream with a non zero offset

Jul 21, 2009 at 8:08 PM

What I'd like to do is concatenate several files together and then open a zip stream at an arbitrary point.

For example, if I combined stub.exe and data.zip into one file, then opened a stream to the combined file, seeked to the start of the zip and passed that into ZipFile.Read() it would be nice if it just worked (and would probably be fairly easy to implement). One way would be to have the stub exe save out the zip file to a separate file, but that would be a time consuming event. Another way would be to make my own Stream class that handled an offset from the start of the file.

The SFX handling seems to load the zip into memory, and that would be inappropriate for a 2-3 GB file.

Cheers

John

Coordinator
Jul 21, 2009 at 10:49 PM
Edited Jul 21, 2009 at 10:50 PM

What problem are you trying to solve, John?  

Your understanding that the SFX is loading the full zip content into memory, is not right.

If you are trying to come up with ideas on how to work around that design flaw, it's not necessary, because the DotNetZip SFX does not load the zip into memory.

In an SFX created by DotNetZip, the zip content is appended as extra data after the "normal" contents of a .NET EXE.   When the SFX runs, the zip content is not loaded as code, nor as data.  If you run the SFX and extract , the sfx reads itself as a stream (sort of a Escher thing), and unzips while reading.   It's also possible to run the SFX and NOT extract the zip - let's say you just Cancel out of the GUI - in which case the appended zip data is neither read, nor loaded.

But I'm not sure if your idea of concatenating files together is intended to be a solution to this problem.  Which is why I asked, what problem are you trying to solve?

Jul 21, 2009 at 11:24 PM

In short, exactly as you describe the SFX, but with my own extractor program.

I'll look at the code some more to see how you achieve that with the SFX.

Cheers

John

Jul 21, 2009 at 11:37 PM

> In an SFX created by DotNetZip, the zip content is appended as extra data after the "normal" contents of a .NET EXE

...but AFAICT, you open a stream to the start of the entire file (stub exe and all), and it works.... I must be missing something =( When I try something similar, it exceptions in ReadDirEntry with a bad signature.

Cheers

John

Coordinator
Jul 22, 2009 at 1:18 AM

ok, you want to do your own extractor.

Here's the thing with the DotNetZip SFX - the exe *is* a zip.   To read a zip file in ReadDirEntry, the code scans for the zip directory , which lies at the end of the zip file. Inside the directory are offsets that point to the zip entry data, at earlier points in the file.  But the offsets are expressed relative to the beginning of the file, not relative to the beginning of the zip data.  In other words, the offset considers all the exe data, too. 

Suppose the extractor exe is 1000 bytes.  The zip data is 10,000 bytes.  The first directory entry in the SFX will list the offset for the first zip entry as 1000.  Whereas, in a normal zip, the first directory entry normally lists an offset of 0 for the first zip entry. 

The upshot is, if you just append files, as in "copy sfxstub.exe+data.zip  sfx.exe" , then you cannot just open the stream at the start of the file (stub and all) as DotNetZip does.  If you take the "copy a+b" approach to building an SFX, then what you would have to do, to extract, is read from the start of the zip data.   But (here's the key thing) the zero offset of that stream has to be the start of the zip data, not the start of the sfx exe.  In other words, stream.Seek(0, SeekOrigin.Begin) has to put the stream pointer at the beginning of the zip data, not at the beginning of the sfx.exe.   During read and extract of a ZIP file, DotNetZip will seek around in the file.  I think a good way to make this work is to build a OffsetStream class that accepts in its constructor, a regular stream (let's say, opened on your SFX.exe), and an offset.  Then, all the methods are pass-throughs, except for Seek.  In the Seek you would have to subtract the given offset from the offset provided in the constructor.  I hope this is making sense. 

 

Jul 22, 2009 at 1:40 AM

It makes perfect sense; the bit I was missing was the offsets precompensate for the size of the stub exe.

> Another way would be to make my own Stream class that handled an offset from the start of the file.

> work is to build a OffsetStream class that accepts in its constructor

One in the same =)

> stream.Seek(0, SeekOrigin.Begin) has to put the stream pointer at the beginning of the zip data

It would be nice if this wasn't the case, and Stream.Position when the ZipFile constructor was called was treated as the origin. Or maybe add a function to fixup the offsets?

Anyway, thanks! Everything is clear now =)

Cheers

John

Coordinator
Jul 22, 2009 at 2:48 AM

you're welcome. 

let's see, use the stream position when ZipFile ctor is called as the origin... hmmmm....  lemme think, would that work?  I think you're right.  Every Seek inside DotNetZip would have to be relative.  Yep, I think that's a reasonable request.  It would be simpler than an offsetstream class.

 

Coordinator
Jul 22, 2009 at 2:52 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jul 22, 2009 at 6:13 AM

ok , I produced a test build of DotNetZip that includes the change which uses the stream Position when the ZipFile constructor is called, as the origin.  I haven't tested this.  Can you grab it from the workitem and see if it works for you.  let me know.

 

Jul 22, 2009 at 11:04 PM

Worked 100% of the time - once in one try =)

I'll let you know if I find any issues - thanks!

Cheers

John

Coordinator
Jul 22, 2009 at 11:07 PM

ha! That's a good test suite.  ok, I will include that new feature into the main source, and in the next binary release.  let me know if you have more problems.

Sep 8, 2010 at 1:28 AM

I just tried updating to DotNetZip 1.9, and this didn't seem to work any more =( Did anything change with regards to this? I don't see anything in the change log....

 

Cheers

John

Coordinator
Sep 9, 2010 at 10:42 AM

Don't think so, not on purpose anyway. 

I'll need a test case that shows the problem.

 

Sep 9, 2010 at 8:01 PM
Test case attached =)

It basically takes a user supplied Test.zip (can be anything) and packages it as it would normally into Test.pak. It then searches for the signature and tries to open the zip.

Compiled against 1.8 - stream offset remains 8 and all is good
Compiled against 1.9 - bad read exception

If you want a bigger test file, download any of the following - http://udk.com/download

DotNetZip works great btw, we're using it to package iPhone applications, Firefox, Chrome and IE plugins... it's a great piece of software!

Cheers
John

-----Original Message-----
From: Cheeso [mailto:[email removed]
Sent: Thursday, September 09, 2010 6:42 AM
To: John Scott
Subject: Re: Opening a zip stream with a non zero offset [DotNetZip:63121]

From: Cheeso

Don't think so, not on purpose anyway.

I'll need a test case that shows the problem.



Read the full discussion online <http://dotnetzip.codeplex.com/Thread/View.aspx?ThreadId=63121&ANCHOR#Post491168> .

To add a post to this discussion, reply to this email ([email removed] <mailto:[email removed]?subject=[DotNetZip:63121]> )

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe <http://www.codeplex.com/site/discussions/thread/unsubscribe/63121> on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
Oct 25, 2010 at 5:17 PM
... any thoughts on this?

Cheers
John

-----Original Message-----
From: John Scott
Sent: Thursday, September 09, 2010 4:02 PM
To: [email removed]
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

Test case attached =)

It basically takes a user supplied Test.zip (can be anything) and packages it as it would normally into Test.pak. It then searches for the signature and tries to open the zip.

Compiled against 1.8 - stream offset remains 8 and all is good
Compiled against 1.9 - bad read exception

If you want a bigger test file, download any of the following - http://udk.com/download

DotNetZip works great btw, we're using it to package iPhone applications, Firefox, Chrome and IE plugins... it's a great piece of software!

Cheers
John

-----Original Message-----
From: Cheeso [mailto:[email removed]
Sent: Thursday, September 09, 2010 6:42 AM
To: John Scott
Subject: Re: Opening a zip stream with a non zero offset [DotNetZip:63121]

From: Cheeso

Don't think so, not on purpose anyway.

I'll need a test case that shows the problem.



Read the full discussion online <http://dotnetzip.codeplex.com/Thread/View.aspx?ThreadId=63121&ANCHOR#Post491168> .

To add a post to this discussion, reply to this email ([email removed] <mailto:[email removed]?subject=[DotNetZip:63121]> )

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe <http://www.codeplex.com/site/discussions/thread/unsubscribe/63121> on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
Nov 12, 2010 at 12:49 AM
We've just run into a bug with source files > 2GB that seems to be fixed in 1.9. However, we can't upgrade due to the below issue =(

Cheers
John

-----Original Message-----
From: John Scott
Sent: Monday, October 25, 2010 1:17 PM
To: '[email removed]'
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

... any thoughts on this?

Cheers
John

-----Original Message-----
From: John Scott
Sent: Thursday, September 09, 2010 4:02 PM
To: [email removed]
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

Test case attached =)

It basically takes a user supplied Test.zip (can be anything) and packages it as it would normally into Test.pak. It then searches for the signature and tries to open the zip.

Compiled against 1.8 - stream offset remains 8 and all is good
Compiled against 1.9 - bad read exception

If you want a bigger test file, download any of the following - http://udk.com/download

DotNetZip works great btw, we're using it to package iPhone applications, Firefox, Chrome and IE plugins... it's a great piece of software!

Cheers
John

-----Original Message-----
From: Cheeso [mailto:[email removed]
Sent: Thursday, September 09, 2010 6:42 AM
To: John Scott
Subject: Re: Opening a zip stream with a non zero offset [DotNetZip:63121]

From: Cheeso

Don't think so, not on purpose anyway.

I'll need a test case that shows the problem.



Read the full discussion online <http://dotnetzip.codeplex.com/Thread/View.aspx?ThreadId=63121&ANCHOR#Post491168> .

To add a post to this discussion, reply to this email ([email removed] <mailto:[email removed]?subject=[DotNetZip:63121]> )

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe <http://www.codeplex.com/site/discussions/thread/unsubscribe/63121> on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
Nov 16, 2010 at 5:37 PM
The fix that seems to be working for us is -

OffsetStream.cs:91
public override long Seek(long offset, System.IO.SeekOrigin origin)
{
if( origin == SeekOrigin.Begin || origin == SeekOrigin.End )
{
return _innerStream.Seek( _originalPosition + offset, origin ) - _originalPosition;
}
else
{
return _innerStream.Seek( offset, origin ) - _originalPosition;
}
}

... also, I think this change would make things more as intended.

OffsetStream.cs:76
public override long Length
{
get
{
return _innerStream.Length - _originalPosition;
}
}

Cheers
John

-----Original Message-----
From: John Scott
Sent: Thursday, November 11, 2010 8:49 PM
To: '[email removed]'
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

We've just run into a bug with source files > 2GB that seems to be fixed in 1.9. However, we can't upgrade due to the below issue =(

Cheers
John

-----Original Message-----
From: John Scott
Sent: Monday, October 25, 2010 1:17 PM
To: '[email removed]'
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

... any thoughts on this?

Cheers
John

-----Original Message-----
From: John Scott
Sent: Thursday, September 09, 2010 4:02 PM
To: [email removed]
Subject: RE: Opening a zip stream with a non zero offset [DotNetZip:63121]

Test case attached =)

It basically takes a user supplied Test.zip (can be anything) and packages it as it would normally into Test.pak. It then searches for the signature and tries to open the zip.

Compiled against 1.8 - stream offset remains 8 and all is good
Compiled against 1.9 - bad read exception

If you want a bigger test file, download any of the following - http://udk.com/download

DotNetZip works great btw, we're using it to package iPhone applications, Firefox, Chrome and IE plugins... it's a great piece of software!

Cheers
John

-----Original Message-----
From: Cheeso [mailto:[email removed]
Sent: Thursday, September 09, 2010 6:42 AM
To: John Scott
Subject: Re: Opening a zip stream with a non zero offset [DotNetZip:63121]

From: Cheeso

Don't think so, not on purpose anyway.

I'll need a test case that shows the problem.



Read the full discussion online <http://dotnetzip.codeplex.com/Thread/View.aspx?ThreadId=63121&ANCHOR#Post491168> .

To add a post to this discussion, reply to this email ([email removed] <mailto:[email removed]?subject=[DotNetZip:63121]> )

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe <http://www.codeplex.com/site/discussions/thread/unsubscribe/63121> on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com