NTFS file times

Jul 1, 2009 at 9:24 AM

I am seeing some strange behaviour with regard to file times with Zip files that I create with DotNetZip.  For example, if I create a Zip file and then look at it in WinZip some of the file times are an hour different to what I expect.

I think this is due to the way that DotNetZip manipulates the file times when adding a file to the Zip to adjust for daylight saving time.  I can see why this is necessary for the standard Last Modified time as this is stored as a local time.  However, I think it is wrong to do it for the NTFS times as these should be stored as UTC.  The DateTime.ToFileTime and DateTime.FromFileTime methods will handle any translation from UTC to local time for you so I don't think you need to do anything special.

Coordinator
Jul 1, 2009 at 2:07 PM

Hmm, ok, interesting.

If you create a zip with DotNetZip, and then unzip from within WinZip, the file times on the resulting extracted files are incorrect - is that what you are saying?

 

Jul 1, 2009 at 2:46 PM

Yes, if I create a zip now (with DST in effect) and add a file that was last modified when DST wasn't in effect and then use WinZip to extract the file, the modified time is an hour later than it should be.

Also, looking at the code it intuitively seems to be doing the wrong thing.  If I have understood it correctly, it adds an hour to the modified time if DST is currently in effect but the file was last written when DST wasn't in effect.  Similar it subtracts an hour in the opposite situation.  This means that a zip file created when DST is in effect will have different times stored to the exact same zip file created when DST is not in effect.

According to the spec, the NTFS file times should be stored as UTC in the zip so none of this manipulation for DST should be necessary.  UTC is always UTC regardless of whether DST is in effect.

I can understand why this time manipulation may be necessary for the standard Last Modified time as this isn't stored in UTC.  I am referring specifically to the NTFS times.

Coordinator
Jul 1, 2009 at 3:47 PM

I understand what you're saying, but I still have some questions.

I agree that if the times were stored as UTC then none of the back-and-forth with time adjustments should be necessary.  

I also think that you are correct, it makes sense to store the NTFS time in UTC.

Where in the spec does it say that the NTFS times should be stored as UTC?   You are talking about the AppNote.txt document from PKWare, correct?   What version of the spec?  I looked again just now and did not see any description of whether the 8-byte NTFS time quantities were to be stored as UTC or not.   I could not find any mention of UTC, timezone, or "local time", anywhere in the PKWare spec.

The way the DotNetZip library is implemented, it currently stores the NTFS time in "local time", similar to the MSDOS time.  I don't specifically recall my decision on this, but failing any clear direction from the spec, I believe I just went with the same approach I had to take with the MSDOS time, which implies that DotNetZip must do the manipulation to deal with DST.

It wouldn't be too hard to change, but before I make another change I want to make absolutely sure I've got it right.  The time handling in DotNetZip has kept me busy in maintenance.  I've thought I had it right before.   Right now all the time-handling tests pass.  I want to make sure before I change it again.

 

Coordinator
Jul 1, 2009 at 3:51 PM

Looking at it further, I can see it says "NTFS time" and for NTFS, a filetime is stored and managed as the number of 100-ns ticks since Jan 1, 1601, UTC.  So it is very clearly implied by the text in the spec.

Jul 1, 2009 at 3:55 PM

Yes, sorry.  The spec doesn't clearly state that the times are UTC but, as you say, I think it is implied as NTFS file times are UTC.

Coordinator
Jul 1, 2009 at 6:15 PM

I think you're exactly right about this. 

Coordinator
Jul 1, 2009 at 6:17 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jul 2, 2009 at 12:50 AM

Can you try out v1.8.4.1 ?

Jul 2, 2009 at 2:50 PM

That looks OK - the NTFS file times look better.  But it looks as if the standard Last Modified time is also being stored as UTC.  I thought this time was supposed to be local time?

Coordinator
Jul 2, 2009 at 6:09 PM

!!!! Yes.  you are correct.  and very thorough.

Jul 3, 2009 at 2:14 PM

Hi, I've just had a look at 1.8.4.4 and I'm not convinced that it's quite right still.  I think for the Last Modified time you should do what you used to do; add an hour to the time if DST is currently in effect and the file was last modified when DST was not in effect, etc.  I believe this was the correct behaviour for Last Modified and matches what Winzip and Windows Compressed folders do.

As an example, I have a file which was last modifed at 11:00 when DST was not in effect.  In Windows Explorer this time shows up as 12:00 because Windows adjusts the time to the timezone that is currently in effect.  If I add this file to a zip with DotNetZip, the Last Modified is set to 11:00.  I think this too should be adjusted to the current timezone and be set to 12:00 in the zip.

Sorry for being a pain!

Coordinator
Jul 3, 2009 at 2:58 PM
Edited Jul 3, 2009 at 3:18 PM

Obviously I don't have the right tests.  I thought I had tests for this, tests that verified that the behavior with the times was correct.  apparently the tests are incomplete.  But I don't know how.

I have a test that grabs a bunch of files from outside of the current DST realm - if it is currently DST then it grabs files that are stamped from before DST, and vice versa.  Then it zips up these files along with a bunch of files that are from "now".  Then unzips them and checks that the times are still accurate.  This test succeeds.

Are you saying that in prior releases the LastModified time was correct, and in v1.8.4.4 it is not?    And when you say "look at it in Windows Explorer," what does that mean, exactly? How do you "look at it it" ?     I'm trying to understand how to design a test to verify the desired correct behavior.

It seems to me there are three moments where a  test can evaluate the time on a file: before zipping, within the zipfile, and after unzipping.  There are 3 distinct time values for each file at each of those moments, LastModifiedWrite, LastAccessed, and Created.  Within the zip file, there is a fourth value  - LastModified. 

Obviously at the first moment, the test needs only to record the time on a file.  Within the zip file, the test needs to verify that the times match what has been recorded.  And then after unzipping, the test needs to verify that the times on the unpacked file match what was originally recorded. 

I think that I am doing that, but I think you are telling me that the LastModified time of entries within the zipfile is not displayed properly.  I need to know how you display it.  If you extract the file, is the time also incorrect?  

I need to understand this more clearly before fixing it.    

 

 

Jul 3, 2009 at 3:19 PM

If you use DotNetZip to zip and extract files, I think the times will always be correct, hence why your tests pass.  This is because the NTFS times will always be present and will be used on extraction and these are correct.

Where I am seeing an issue is when I open up a zip file produced with DotNetZip in Winzip.  Admittedly I'm currently using an old version of Winzip (v9.0) which probably doesn't understand NTFS times (it certainly doesn't write them into zip files it creates) so is using the Last Modified time.  I don't think this is a big issue because if you use DotNetZip or a newer version of Winzip it works fine because the NTFS times get used.  I'm just nitpicking!

So in my example, I create a zip file and add a file that was last modified when daylight saving time was not in effect.  In Windows Explorer details view the time shows up as 12:00.  If I open the Zip file in Winzip 9 the modified time displays as 11:00.  If I extract it using Winzip 9, the time on the file reported by Windows Explorer is 11:00 (i.e. an hour different to the original file).

If zip the same file with Winzip or Windows compressed folders the Last Modified time stored in the zip file is 12:00.  I think the behaviour you had before was correct with regard to the Last Modified time.  It was only the NTFS times that needed changing to be stored as UTC.  The NTFS times are now correct but the behaviour concerning the Last Modified time has also changed.

I think that maybe you need a test that extracts files from a Zip file without using the NTFS times.  That would test the Last Modified time.  I'm not sure how easy this would be, though, as DotNetZip always seems to add the NTFS times to the Zip and uses them if they are present.

Coordinator
Jul 3, 2009 at 4:38 PM

Don't worry about being picky, I want it to be correct.

Actually I do have compatibility tests where I zip with DotNetZip and unzip with WinZip, and vice versa.  But it is WinZip12, which maybe always uses NTFS times.    I also use other zip libraries, too, in the tests: 7zip, and the built-in zip handing from the Explorer shell.  I try to verify that all the metadata - file size, attributes, file times, internal content, etc - is the same, but obviously I missed this. 

Just now I looked in Explorer itself and saw exactly what you described - the one-hour time shift.    I don't have a way to automate the use of the explorer graphical UI.  So I will find another way  to get those times and check them against what DotNetZip stores.

 

Coordinator
Jul 3, 2009 at 9:01 PM

OK, try v1.8.4.5 .

I put together a new test that I think verifies what you described.  It now passes. 

Jul 3, 2009 at 9:20 PM

This blog entry might provide some interesting background on problems people typically run into when dealing with this issue:

http://blogs.msdn.com/oldnewthing/archive/2003/10/24/55413.aspx

There are also lots of links there as well.

Coordinator
Jul 3, 2009 at 10:40 PM

Thanks, I was aware of that one.

Actually getting it all implemented properly, and getting complete tests covering all the cases, has been tricky for me.

Jul 3, 2009 at 11:18 PM

Perfect!  That all looks good to me now.  Thanks very much.

Coordinator
Jul 4, 2009 at 3:18 PM

thanks for the help