dotnetzip can't include open files (which others do?)

Jun 3, 2009 at 10:08 PM

 

When I'm creating a zip file with ZipIt, I get an error if I try to include today's (live) log files.  Winzip command line can can work with open files like this, although it throws a warning it works ok. I really want the ability to include open files without having to resort to a kludgy workaround like copying the files to a new location before including them, because windows CAN copy the files even though they are open.

7zip has the same problem as dotnetzip.

I don't see an elegant way to do this unless I use winzip command line, which requires the full winzip installation as a pre-req, but I'd rather use something small/elegant like 7zip or dotnetzip that's a simple single exe. Anyone know of something that could help?

-J

 

adding 20090603\SMSLog_20090603.CSV...
Saving....
Exception: System.IO.IOException: The process cannot access the file because another process has locked a portion of th
 file.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.ReadCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at Ionic.Zip.CrcCalculatorStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Ionic.Zip.ZipEntry._WriteFileData(Stream s)
   at Ionic.Zip.ZipEntry._EmitOne(Stream outstream)
   at Ionic.Zip.ZipEntry.Write(Stream outstream)
   at Ionic.Zip.ZipFile.Save()
   at Ionic.Zip.Examples.ZipIt.Main(String[] args)

Coordinator
Jun 3, 2009 at 10:44 PM

What version of the zipit.exe tool are you using? (zipit -? will tell you).  I implemented a change in DotNetZip v1.7, in the way the library opens files, to better handle open files.

Does this problem occur every time, or just once in a while?

 

Jun 4, 2009 at 1:17 AM
Thanks for responding.

I just downloaded 1.7 today and tried. I tested twice with zipping
blackberry server logs from today (live). Both times failed with
dotnetzip and 7zip but worked with winzip command line.

I tried with poweshell api calls also which is my goal but I'm having
powershell syntax issues

I assume it would have the same result? I can try it regardless
tomorrow to confirm.

J



On 6/3/09, harr2969 <notifications@codeplex.com> wrote:
> From: harr2969
>
> When I'm creating a zip file with ZipIt, I get an error if I try to include
> today's (live) log files. Winzip command line can can work with open files
> like this, although it throws a warning it works ok. I really want the
> ability to include open files without having to resort to a kludgy
> workaround like copying the files to a new location before including them,
> because windows CAN copy the files even though they are open.7zip has the
> same problem as dotnetzip.I don't see an elegant way to do this unless I use
> winzip command line, which requires the full winzip installation as a
> pre-req, but I'd rather use something small/elegant like 7zip or dotnetzip
> that's a simple single exe. Anyone know of something that could help?-J
> adding 20090603\SMSLog_20090603.CSV...
> Saving....
> Exception: System.IO.IOException: The process cannot access the file because
> another process has locked a portion of th
> file.
>
> at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
> at System.IO.FileStream.ReadCore(Byte[] buffer, Int32 offset, Int32
> count)
> at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
> at Ionic.Zip.CrcCalculatorStream.Read(Byte[] buffer, Int32 offset, Int32
> count)
> at Ionic.Zip.ZipEntry._WriteFileData(Stream s)
> at Ionic.Zip.ZipEntry._EmitOne(Stream outstream)
> at Ionic.Zip.ZipEntry.Write(Stream outstream)
> at Ionic.Zip.ZipFile.Save()
> at Ionic.Zip.Examples.ZipIt.Main(String[] args)
>
>
Coordinator
Jun 4, 2009 at 2:28 AM

Nah, if you are doing DotNetZIp from within Powershell you will have the same problem as using zipit.exe from a cmd.exe shell.

So I take it the problem occurs every time you try, eh?

There was a problem where DotNetZip would fail if a file was open for writing.  For example, a spreadsheet file open by Microsoft Excel, or a favorites file open by Firefox browser.   The changes in v1.7 worked around that problem - addressed it, so that it no longer occurs.  With v1.7, DotNetZip can read and open and zip up files that are open in this way. DotNetZip now opens all files in a way that allows sharing, and everything is good.

The problem you are reporting is something different.  After you reported this problem, I learned that an application that has a file open, can set a lock on a file - actually on a region within a file - and can prevent all readers for the period the lock is open.  There is no sharing allowed at all.  And DotNetZip is not handling this condition.  It generates an exception and DotNetZip is not handling it, which I guess causes zipit.exe to fail.  One option is for DotNetZip to handle the exception and just pass over the file, but I don't know if that is what you really want.  The result would be, that the zipfile would not contain that particular file. 

From what I read in the Windows API documentation, the recommended action when this condition occurs is EITHER:

  •  to ignore that file (whch in the case of DotNetZip means, don't open it, don't zip it) ,
  • OR, wait "a little while" and retry.  That's *really* what the documentation says:  "retry the operation later".  How much later?  who knows..

Let me ask you, what does WinZip do in this situation?  You said that WinZip gives you error messages, but what is in the result zip file?  The log file that is "live" - is it present in the resultant zip file?  Or does WinZip pass over it, ignore it? 

What would you like DotNetZip to do?

 

 

Jun 4, 2009 at 6:21 PM

It would be ideal if dotnetzip could read up until or around the region locked.

In my test/review just now, Winzip includes the open file data up to very end of the file. Presumably some of the lines could be truncated but I didn't find this on the blackberry logs I reviewed after compressing them. All the text logs I reviewed showed data up until the minute I started compressing them. I've never had a file skipped, they are always included. But I haven't tried

I'll show you what it outputs when I'm compressing. I could ftp you the actual compressed file if it helped.

PS C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs> & 'C:\Program Files (x86)\Research In Mo
tion\BlackBerry Enterprise Server\Logs\upload.ps1'
No date input detected in format yyyymmdd, uploading logs for 20090604
VERBOSE: logdate: 20090604
VERBOSE: hostname: BESR122
VERBOSE: wzexec: c:\Program Files (x86)\WinZip\wzzip.exe
VERBOSE: filename: BESR122-20090604.zip
VERBOSE: cmdline: c:\Program Files (x86)\WinZip\wzzip.exe BESR122-20090604.zip 20090604

WinZip(R) Command Line Support Add-On Version 2.0 (Build 7041)
Copyright (c) WinZip International LLC 1991-2005 - All Rights Reserved

THANK YOU FOR TRYING WINZIP
This is a fully functional version for EVALUATION USE ONLY
The registered Pro version does not display this notice.
Please go to www.winzip.com for ordering information and to register your
copy of WinZip Pro.

 Adding BESR122_ALRT_01_20090604_0001.txt
Warning: The following file is open by another program.  If that program
Warning:   writes to the file while WinZip is zipping the file, the zipped
Warning:   file may be corrupt: 20090604\BESR122_ALRT_01_20090604_0001.txt
 Adding BESR122_BBIM_01_20090604_0001.txt
 Adding BESR122_CBCK_01_20090604_0001.txt
Warning: The following file is open by another program.  If that program
Warning:   writes to the file while WinZip is zipping the file, the zipped
Warning:   file may be corrupt: 20090604\BESR122_CBCK_01_20090604_0001.txt
 Adding BESR122_CEXC_01_20090604_0001.txt
Warning: The following file is open by another program.  If that program
Warning:   writes to the file while WinZip is zipping the file, the zipped
Warning:   file may be corrupt: 20090604\BESR122_CEXC_01_20090604_0001.txt
 Adding BESR122_CMNG_01_20090604_0001.txt

---skipped a bunch more similar lines [joel] --

creating Zip file BESR122-20090604.zip

========

*****Oddly - testing with dotnetzip today it worked for me with the open files and no errors. !!  Inconsistent. :-/   I'll keep trying and see if it breaks again or is inconsistent on specific servers.

Now I'm having a different issue, I'll post another thread :(

Thanks,

--J



On Wed, Jun 3, 2009 at 9:28 PM, Cheeso <notifications@codeplex.com> wrote:
> From: Cheeso
>
> Nah, if you are doing DotNetZIp from within Powershell you will have the
> same problem as using zipit.exe from a cmd.exe shell.
>
> So I take it the problem occurs every time you try, eh?
>
> There was a problem where DotNetZip would fail if a file was open for
> writing.  For example, a spreadsheet file open by Microsoft Excel, or a
> favorites file open by Firefox browser.   The changes in v1.7 worked around
> that problem - addressed it, so that it no longer occurs.  With v1.7,
> DotNetZip can read and open and zip up files that are open in this way.
> DotNetZip now opens all files in a way that allows sharing, and everything
> is good.
>
> The problem you are reporting is something different.  After you reported
> this problem, I learned that an application that has a file open, can set a
> lock on a file - actually on a region within a file - and can prevent all
> readers for the period the lock is open.  There is no sharing allowed at
> all.  And DotNetZip is not handling this condition.  It generates an
> exception and DotNetZip is not handling it, which I guess causes zipit.exe
> to fail.  One option is for DotNetZip to handle the exception and just pass
> over the file, but I don't know if that is what you really want.  The result
> would be, that the zipfile would not contain that particular file. 
>
> From what I read in the Windows API documentation, the recommended action
> when this condition occurs is EITHER:
>
>  to ignore that file (whch in the case of DotNetZip means, don't open it,
> don't zip it) ,
> OR, wait "a little while" and retry.  That's *really* what the documentation
> says:  "retry the operation later".  How much later?  who knows..
>
> Let me ask you, what does WinZip do in this situation?  You said that WinZip
> gives you error messages, but what is in the result zip file?  The log file
> that is "live" - is it present in the resultant zip file?  Or does WinZip
> pass over it, ignore it? 
>
> What would you like DotNetZip to do?
>
>  
>
>  
>
> Read the full discussion online.
>
> To add a post to this discussion, reply to this email
> (DotNetZip@discussions.codeplex.com)
>
> To start a new discussion for this project, email
> DotNetZip@discussions.codeplex.com
>
> You are receiving this email because you subscribed to this discussion on
> CodePlex. You can unsubscribe on codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any posts
> to this discussion will also be available online at codeplex.com

Coordinator
Jun 13, 2009 at 9:18 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jun 13, 2009 at 10:40 PM

J, I built a special version of zipit.exe and the DotNetZip library (Ionic.Zip.dll) for you to try. You can get it by visiting the workitem.

http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=7889

It does a retry in the event of receiving the error you reported.  Try it out and let me know if that behavior is suitable for you.

Jun 14, 2009 at 2:09 PM
Thanks, I haven't had a reoccurance of that since the first day :/
I'm rolling out this util over about 100 blackberry servers for log
compression/uploading so I'll keep this in mind if it re-occurs.
Thanks!

On 6/13/09, Cheeso <notifications@codeplex.com> wrote:
> From: Cheeso
>
> J, I built a special version of zipit.exe and the DotNetZip library
> (Ionic.Zip.dll) for you to try. You can get it by visiting the
> workitem.http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=7889It
> does a retry in the event of receiving the error you reported. Try it out
> and let me know if that behavior is suitable for you.
>
>
Sep 16, 2009 at 1:26 PM

Hello,

I'm getting this error with rev 1.8.4.23

<ExceptionDetail>
    <Description>Ionic.Zip.ZipException: Cannot extract ---&gt; System.IO.IOException: The process cannot access the file 'C:\Users\Blake Niemyjski\Documents\DocumentFolder' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.OpenHandle(String path)
   at System.IO.Directory.SetLastWriteTimeUtc(String path, DateTime lastWriteTimeUtc)
   at System.IO.Directory.SetLastWriteTime(String path, DateTime lastWriteTime)
   at Ionic.Zip.ZipEntry.a(String A_0, Boolean A_1)
   at Ionic.Zip.ZipEntry.a(String A_0, Stream A_1, String A_2)
   --- End of inner exception stack trace ---
   at Ionic.Zip.ZipEntry.a(String A_0, Stream A_1, String A_2)
   at Ionic.Zip.ZipEntry.Extract(String baseDirectory)
   at Ionic.Zip.ZipFile.a(String A_0, Boolean A_1)
   at Ionic.Zip.ZipFile.ExtractAll(String path, ExtractExistingFileAction extractExistingFile)
   at Production.Configuration.a(String A_0, String A_1, Boolean A_2)
   at Production.Configuration.a(String A_0, String A_1)
   at Production.Configuration.b.a()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</Description>
    <ErrorCode>-2146233088</ErrorCode>
    <ExceptionTypeName>Ionic.Zip.ZipException</ExceptionTypeName>
    <InnerException>
      <Description>System.IO.IOException: The process cannot access the file 'C:\Users\Blake Niemyjski\Documents\DocumentFolder' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.OpenHandle(String path)
   at System.IO.Directory.SetLastWriteTimeUtc(String path, DateTime lastWriteTimeUtc)
   at System.IO.Directory.SetLastWriteTime(String path, DateTime lastWriteTime)
   at Ionic.Zip.ZipEntry.a(String A_0, Boolean A_1)
   at Ionic.Zip.ZipEntry.a(String A_0, Stream A_1, String A_2)</Description>
      <ErrorCode>-2147024864</ErrorCode>
      <ExceptionTypeName>System.IO.IOException</ExceptionTypeName>
      <InnerException i:nil="true" />
      <Message>The process cannot access the file 'C:\Users\Blake Niemyjski\Documents\DocumentFolder' because it is being used by another process.</Message>
      <MethodName>WinIOError</MethodName>
      <ModuleName>mscorlib</ModuleName>
      <ModuleVersion>2.0.0.0</ModuleVersion>
      <Source>mscorlib</Source>
      <StackTrace>   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.OpenHandle(String path)
   at System.IO.Directory.SetLastWriteTimeUtc(String path, DateTime lastWriteTimeUtc)
   at System.IO.Directory.SetLastWriteTime(String path, DateTime lastWriteTime)
   at Ionic.Zip.ZipEntry.a(String A_0, Boolean A_1)
   at Ionic.Zip.ZipEntry.a(String A_0, Stream A_1, String A_2)</StackTrace>
    </InnerException>
    <Message>Cannot extract</Message>
    <MethodName>a</MethodName>
    <ModuleName>Production</ModuleName>
    <ModuleVersion>5.0.0.0</ModuleVersion>
    <Source>Production</Source>
    <StackTrace>   at Ionic.Zip.ZipEntry.a(String A_0, Stream A_1, String A_2)
   at Ionic.Zip.ZipEntry.Extract(String baseDirectory)
   at Ionic.Zip.ZipFile.a(String A_0, Boolean A_1)
   at Ionic.Zip.ZipFile.ExtractAll(String path, ExtractExistingFileAction extractExistingFile)
   at Production.Configuration.a(String A_0, String A_1, Boolean A_2)
   at Production.Configuration.a(String A_0, String A_1)
   at Production.Configuration.b.a()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</StackTrace>
  </ExceptionDetail>

Thanks
-Blake Niemyjski

Coordinator
Sep 17, 2009 at 10:59 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Sep 19, 2009 at 3:52 PM

This is fixed in v1.9.0.9

Dec 16, 2009 at 4:53 PM

I am seeing an identical problem with the 1.8 zipit downloaded recently.  While creating a zip file, Zipit threw an exception: System.IOException: The process cannot access the file 'xxxxxx' because it is being used by another process.

My desired behavior is for zipit to simply not zip anything it can't open or read, and keep on chugging.  Is there a solution?  THANKS!!

 

Coordinator
Dec 18, 2009 at 6:46 AM

The capability to retry or skip was exposed by the API in v1.9.0.9, but I had not exposed it in the zipit.exe command-line tool until now.

The possible solutions are:

  • write a program to do what you want, and use the ZipError property on ZipFile
  • write a script using Powershell, and use the ZipError property
  • download version 1.9.0.32 of the DotNetZip tools - zipit has a new set of -e* options for specifying what to do in this case.
    In your case, you want the -es option, which skips any file for which it encounters an error.

 

Dec 18, 2009 at 11:22 AM

 

Hello,
Thanks
-Blake Niemyjski

Hello,

I think the biggest issue that most of us have is we can't use 1.9.x.x as it isn't an official release and we can't use beta components in our real world applications. Do you know when 1.9 is going to be finalized? I've been waiting for three plus months.

Thanks

-Blake Niemyjski

 

Coordinator
Dec 18, 2009 at 5:24 PM

Whoa!  Sorry, Blake. Sorry that you're waiting so long.  I hadn't realized that it has been 3+ months.  I normally shoot for 3-4 month update cycles.  Shorter than that, and it's too frequent.  Longer than that, and it takes too long to get fixes out. I hear you telling me that it's long enough now.

I'm trying to strike the right balance.  v1.9 includes some pretty big features:  a new implementation for compressing files in parallel, zip splitting, and a couple new classes in ZipOutputStream and ZipInputStream.  These are pretty big changes, and while I think the release is gonna be useful to just about everyone who wants to zip files programmatically, and I'd like to see people use it, it hasn't been tested very broadly.  I test it as much as I can, but that's no substitute for real world usage.  So I'm trying to be prudent in how quickly I promote v1.9 to "released" status.

Aside from the "beta" and "official release" monikers, you might want to just look at the relative quality and stability of the code.  If v1.9 works for you, then it is *just as good* to you as "officially released". no?   It's been very stable, since the early weeks - no great changes in the API, so you can be assured that even if I update v1.9 at this point, the changes will be small. 

But I hear you - it's time to promote v1.9.  I do have some final tests to run before I'm ready to do that.  It should happen soon.

Dec 18, 2009 at 7:39 PM

 

Hello,
Thanks
-Blake Niemyjski

Hello,

It's all good, I have a work around for that fix so I wasn't really saying it literally, but it would be nice to have a new version. I completely know about big releases and open source software. I have been on both sides :). 

Thank you so much for taking the time to develop and test DotNetZip, it is a great piece of software and the community appreciates your work! I will keep an eye out for 1.9 :).

Thanks

-Blake Niemyjski