Ever since I've been working with the .NET framework, most of my time was spent on the System.IO namespace. I'm not a UI guy, I'm an IO guy! The most important class in that namespace is System.IO.Stream. And since it was well-designed, probably inspired by other successful stream implementations (Delphi comes to mind), it's very easy to expose features using streams.

My favorite use of streams is for pass-through streams. A pass-through stream is a class which derives from System.IO.Stream, but reads from or writes to an inner stream received at creation. It serves as a data modifyer or data analyser. When reading from a pass-through stream, it first reads from its inner stream, then processes the data read (potentially modifying it) and returns this data. When writing to a pass-through stream, it first processes the provided data (again potentially modifying it), then writes it to its inner stream.

Xceed Zip for .NET and Xceed FTP for .NET both use a pletoria of pass-through streams. The most popular is Xceed.Compression.CompressedStream, the stream responsible for compressing data before writing it to its inner stream, or decompressing data read from its inner stream. But most others are internal. We've been juggling with the idea of exposing them for a long time, but beleive it would only confuse developers to "see" those new namespaces and classes. Another useful thing with internal classes is that we can change their interface without causing breaking changes.

TransientStream

It was a long debate before we decided to go forth with the "transient" keyword. Not only is it used in the TransientStream type name, but also as a property on many of our pass-through streams. What we meant by "transient" is "volatile", or if you prefer more explicit keywords, "does-not-close-its-inner-stream-when-closed". A TransientStream is about the simplest expression of a pass-through stream. All required property and method overrides simply call the inner stream. The only exception is for the Close method, which simply makes sure not to call Close on the inner stream. This is very useful when you need to pass your stream to another routine which closes the stream, while you don't want your stream to get closed.

ChecksumStream

This stream does not modify the data read from or written to, but takes the opportunity to calculate either a CRC32 or an Adler32 on that data. When reading, it can also make sure, upon closing it, that the calculated checksum matches an expected stream, else throw an exception. In this way, we can insert checksum calculation anywhere in a process without interfering nor requiring code changes.

CombinedStream

The deflate compression algorithm has the ability to detect the end of the data when decompressing. The CompressedStream is itself a pass-through stream. When reading from it, it first reads from the inner stream, then decompresses the data. When it reaches the end of the compressed data, the CompressedStream has the ability to return a stream on the remaining data, in case this inner stream contains more data after the compressed block. Why isn't this equivalent to the inner stream you might ask? Let's say the inner stream isn't seekable. The CompressedStream's Read method first reads N bytes from the inner stream, but may have found that the end of the compressed data is after M bytes (M < N). The inner stream is already N-M bytes too far. The CombinedStream receives both a byte array (the unused N-M bytes) and the inner stream as ctor parameters, and will expose those as one contiguous stream. Pretty slick!

HeaderFooterStream

Xceed Streaming Compression for .NET exposes stream-based (as opposed to archive-based) compression formats. Those formats all have one thing in common: they have a header and a footer. Not all of them can depend on the deflate algorithm to automatically detect the end of the stream. That's why they need to make sure to never return the first M bytes and last N bytes from the inner stream, where M is the expected header size and N the expected footer size.

WindowStream

When exposing part of a zip file as a single AbstractFile, we need to make sure we do not read past the boundaries of that file's data in the zip file. The WindowStream exposes a region of its inner stream as a zero-position, N-length stream.

ZCryptStream

This pass-though stream automatically encrypts or decrypts the data written or read, using the basic Zip encryption (which is as weak as me in front of a cheese cake). I will be working on AES encryption very soon, and it will most probably be implemented as a pass-through stream too!

NotifyStream

Though pass-through streams can do much of the task, it is often better for the clarity of the code to have processing done by other classes not deriving from System.IO.Stream. The NotifyStream class exposes three events: ReadingFromStream, WritingToStream and ClosingStream. Any other class can advise for those events to intervene in the reading or writing process. This old class exists since the beginning of Xceed Zip for .NET, but it has proven very useful in the current development we are doing for Tar and GZip support within Xceed Zip for .NET.

ForwardSeekableStream

This new class created for Xceed Zip for .NET 3.0 (Tar and GZip support) can expose a non-seekable stream as a seekable stream when reading, or at least a stream reporting a Position when writing. When reading, you can call Seek with an offset behond the current position, and it will simply read from the non-seekable inner stream until well positioned. And for both reading and writing operations, it counts the number of bytes read or written so it can report a position (granting we knew the original position when created).

FtpAsciiDataStream

Xceed FTP for .NET also uses pass-through streams. For example, the FtpAsciiDataStream wraps the NetworkStream to perform convertion of LF to CR/LF on the fly when sending a file in ASCII mode.

One of the less known features of the Xceed FileSystem is its file filtering capabilities. Not only does it come with built-in support for filtering files based on name, size, attributes and dates, it also lets you easily combine criterias. Furthermore, as for all Xceed components, it's fully extensible.

For example, let's say I want to copy files matching the "*.txt" filter that have the archive attribute on. The following code can be used:

  sourceFolder.CopyFilesTo( destFolder, true, true, "*.txt", FileAttributes.Archive );

What is happening beneath the surface? The fouth parameter is "params object[] filters". This means you can provide any number of any types of parameters. Any types? Not exactly. What you should see is "params Filter[] filters". The Filter class is the base class for any type of filter you could think of. The Xceed FileSystem comes with seven built-in filter classes, divided in two categories:

Operators: AndFilter, OrFilter, NotFilter.
Filters: NameFilter, AttributeFilter, SizeFilter, DateTimeFilter.

So the line of code above can be seen as this:

  sourceFolder.CopyFilesTo( destFolder, true, true,
    new AndFilter( new NameFilter( "*.txt" ), new AttributeFilter( FileAttributes.Archive ) ) );

But we've decided that forcing the creation of a new NameFilter everytime you want to filter on a mask was overkill for such a common operation. That's why we also accept two other types of parameters. Strings are automatically converted to a NameFilter, and FileAttributes are automatically converted to an AttributeFilter. Finally, providing two or more filters as separate parameters automatically puts them in an AndFilter.

But then, what happens to another common scenario: filtering files based on two name filters? Passing "*.txt" as the fourth parameter, and "*.doc" as the fifth would generate an AndFilter around them, thus only matching files that match the ".txt" and the ".doc" extensions... Oups!

We support yet another exception: any string filter can contain a pipe character (|) for providing multiple name filters that will be grouped in an OrFilter, like this:

  sourceFolder.CopyFilesTo( destFolder, true, true, "*.txt|*.doc" );

This will automatically be translated to:

  sourceFolder.CopyFilesTo( destFolder, true, true, 
    new OrFilter( new NameFilter( "*.txt" ), new NameFilter( "*.doc" ) ) );

By the way, most operator-like filters' constructors will accept strings and FileAttributes too, doing the translation to NameFilter and AttributeFilter instances for you.

The final "hidden" feature relates to case sensitivity. By default, the FileSystem is case insensitive, as is the Windows platform. But since archives like zip files may come from other planets like Linux or Mac OS X, we wanted to support case-sensitive file matching. If you prepend your string with the "greater than" character (<), the resulting NameFilter will be case-sensitive. The following code will only match files which have their extension in upper-case:

  sourceFolder.CopyFilesTo( destFolder, true, true, ">*.TXT" );

Since Windows does remember the casing of filenames, this can be very useful even on the Windows platform. Furthermore, since we released the library, the Mono project came to life, and our library can now be used on other platforms.

Extending filters

You can easily create custom filters by deriving from the Xceed.FileSystem.Filter class and overriding the IsItemMatching method. A SearchFilter class, which searches for a particular text within files could look like this:

  class SearchFilter : Filter
  {
    public SearchFilter( string text )
      : base( FilterScope.File )
    {
      if( text == null )
        throw new ArgumentNullException( "text" );

      if( text.Length == 0 )
        throw new ArgumentException( "The text cannot be empty.", "text" );

      m_text = text;
    }


    public override bool IsItemMatching( FileSystemItem item )
    {
      AbstractFile file = item as AbstractFile;


      if( file == null )
        return false;


      try
      {
        int bufferSize = ( file.Size < 0x1000000 )
          ? unchecked( ( int )file.Size ) 
          : 0x1000000;


        byte[] search = System.Text.Encoding.Default.GetBytes( m_text );


        if( search.Length <= bufferSize )
        {
          byte[] buffer = new byte[ bufferSize ];
          int found = 0;


          using( BinaryReader reader = new BinaryReader( file.OpenRead( FileShare.ReadWrite ) ) )
          {
            int read = 0;


            while( ( read = reader.Read( buffer, 0, bufferSize ) ) > 0 )
            {
              found = FindBuffer( buffer, 0, read, search, found );


              if( found == search.Length )
                return true;
            }
          }
        }      
      }
      catch {}


      return false;
    }


    private int FindBuffer( 
      byte[] source, 
      int sourceStart, 
      int sourceCount, 
      byte[] search, 
      int searchIndex )
    {
      // TODO: Param check!


      for( int i=0; i<sourceCount; i++ )
      {
        if( source[ sourceStart + i ] == search[ searchIndex ] )
        {
          if( ++searchIndex == search.Length )
            return searchIndex;
        }
        else
        {
          searchIndex = 0;
        }
      }


      return searchIndex;
    }


    private string m_text; // = null
  }

Using this custom filter, you can now copy only files that contain a particular text:

  sourceFolder.CopyFilesTo( destFolder, true, true, new SearchFilter( "allo" ) );

Conditionally recursing

One missing feature we had with the filtering will be addressed with today's release. There were no way to control which subfolders to recurse into or not when calling methods accepting filters (CopyFilesTo, MoveFilesTo, GetFiles, GetFolders). The FilterScope.Folder value wasn't preventing recursing into subfolders. It was only meant to include or exclude folder entries from being copied. But passing "true" or "false" as the "recurse" parameter was an all or nothing deal.

Today, we introduce a new scope: FilterScope.Recurse. It does not interfere with the File or Folder socpe, and only determines if we should continue matching files into each subfolder. Its number one use is for excluding subfolders:

  sourceFolder.CopyFilesTo( destFolder, true, true, 
    "*.txt", new NotFilter( new NameFilter( "Bar", FilterScope.Recurse ) ) );

The way you combine "Recurse" filters and other filters is irrelevant. When deciding to copy files or folders, the library ignores any filters with the Recurse scope. When deciding to call itself recursively, the library ignores any filters with the File or Folder scope.

We plan on providing new types of filters. Suggestions welcomed!

Just in case my previous post on the subject did not ring a bell, the release of version 2.1 of Xceed FTP for .NET means you can directly unzip from a zip file located on an FTP server, without downloading the file first! Look at the following code:

  using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )
  {
    FtpFile source = new FtpFile( connection, @"/images/Flowers/Backup/Flowers.zip" );
    DiskFolder dest = new DiskFolder( @"d:\temp\flowers" );

    ZipArchive zip = new ZipArchive( source );
    zip.CopyFilesTo( dest, true, true );
  }

The secret behind this code is the kind of stream "FtpFile.OpenRead" returns. Though we are dealing with a network connection, this stream is fully seekable! The FtpFile takes advantage of the "REST" FTP command, which tells the FTP server we wish to start the transfer at a specific offset. Thus, when the ZipArchive needs to seek at the end of the file to locate the ending header, a proper "REST" command is issued to avoid having to read all the zip file first. And the same happens when reading the central directory, or unzipping specific files.

I have put forward my incredible talent with ASCII art (sic) and updated my wonderful "FileSystem.txt" file, which describes classes available with the Xceed FileSystem for .NET.

                             ==============
                             FileSystemItem
                             ==============
                                    |
                     +--------------+------------------+
                     |                                 |
              ==============                     ============
              AbstractFolder                     AbstractFile
              ==============                     ============
                     |                                 |
         +---+---+---+---+---+---+     +---+---+---+---+---+---+---+
         |   |   |   |   |   |   |     |   |   |   |   |   |   |   |
 ==========  |   |   |   |   |   |     |   |   |   |   |   |   |  ========
 DiskFolder  |   |   |   |   |   |     |   |   |   |   |   |   |  DiskFile
 --============  |   |   |   |   |     |   |   |   |   |   |  ==========--
   MemoryFolder  |   |   |   |   |     |   |   |   |   |   |  MemoryFile
   --==============  |   |   |   |     |   |   |   |   |  ============--
     IsolatedFolder  |   |   |   |     |   |   |   |   |  IsolatedFile
     ------============  |   |   |     |   |   |   |  ============----
           ZippedFolder  |   |   |     |   |   |   |  ZippedFile
           -------=========  |   |     |   |   |  =======-------
                  FtpFolder  |   |     |   |   |  FtpFile
                  -============  |     |   |  ==========-
                   TarredFolder  |     |   |  TarredFile
                   ---=============    |  ===========---
                      GZippedFolder    |  GZippedFile
                      -------------   ==========-----
                                      StreamFile
                                      ----------

Can you guess what I'm working on?

I'm glad to announce that Xceed FTP for .NET 2.1 is now available for download. I've been working on this release for the past few months, and I'm very excited to finally see the FTP FileSystem come to life.

For those not familiar with the Xceed FileSystem, which comes with Xceed Zip for .NET, here is some code that sheds some light on what you can do with it. Consider these variables of the same base type:

// A file on disk

AbstractFile first = new DiskFile( @"d:\FileSystem.txt" );

// Another file on disk

AbstractFile second = new DiskFile( @"c:\temp\AnotherFileSystem.txt" );

// A file compressed in a zip file on disk

AbstractFile third = new ZippedFile( 

  new DiskFile( @"c:\temp\data.zip" ), "FileSystemInAZip.txt" );;

// A file in the isolated storage

AbstractFile fourth = new IsolatedFile( "Isolated.txt" );

// A file in memory (random name)

AbstractFile fifth = new MemoryFile();

// A file compressed in a zip file in memory

AbstractFile sixth = new ZippedFile( 

  new MemoryFile(), "VolatileFileSystem.txt" );

You can copy files around very easily:

// Copying the first file anywhere else is always the same!

first.CopyTo( second, true );

first.CopyTo( third, true );

first.CopyTo( fourth, true );

first.CopyTo( fifth, true );

first.CopyTo( sixth, true );

And accessing the contents of any file is always the same:

private void DisplayTextFile( AbstractFile file )

{

  Console.WriteLine( "Displaying the contents of {0}, which is a {1}.", 

    file.FullName,

    file.GetType().Name );

  using( StreamReader reader = new StreamReader( file.OpenRead() ) )

  {

    string line;

    while( ( line = reader.ReadLine() ) != null )

    {

      Console.WriteLine( line );

    }

  }

  Console.WriteLine();

}

// Displaying the contents of those files is always the same!

DisplayTextFile( first );

DisplayTextFile( second );

DisplayTextFile( third );

DisplayTextFile( fourth );

DisplayTextFile( fifth );

DisplayTextFile( sixth );

And why not finish this demonstration by deleting the files we just created.

// And finally, deleting files is the same!

second.Delete();

third.Delete();

fourth.Delete();

fifth.Delete();

sixth.Delete();

Any kind of file is an AbstractFile. Any kind of folder is an AbstractFolder. This way, a DiskFile, an IsolatedFile, a ZippedFile and a MemoryFile share a common set of properties and methods for accessing their metadata and reading/writing their actual data. And a DiskFolder, an IsolatedFolder, a ZippedFolder and a MemoryFolder share a common set of methods for discovering child items.

How does the FTP FileSystem come into play? Simply by offering the same abstraction over files and folders stored on an FTP server. We could simply add this code to the above sample, and everything works just as expected!

AbstractFile seventh = new FtpFile( 

  new FtpConnection( "localhost" ), @"\RemoteFileSystem.txt" );

first.CopyTo( seventh, true );

DisplayTextFile( seventh );

seventh.Delete();

Let's dive a little bit into this implementation of an AbstractFile and AbstractFolder. Each constructor requires an FtpConnection instance, which contains information on how to connect to the target FTP server. Though it looks like a simple information storage class, it does much more. Each time an FtpFile or an FtpFolder requires information, or an incoming or outgoing stream on a file's data, it asks the FtpConnection for an active command channel connection to the server. This way, a unique command channel is generally required for accessing many files on the server.

using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )

{

  connection.TraceWriter = Console.Out;

  FtpFolder root = new FtpFolder( connection );

  foreach( AbstractFile file in root.GetFiles( false, "*.txt" ) )

  {

    DisplayTextFile( file );

  }

}

If we comment out the "Console.WriteLine( line );" line in "DisplayTextFile", we can see the FTP conversation that occurred for the above code:

Connected to 66.46.177.250:21 on 9/27/2005 @ 2:24:54 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
> CWD /
< 250 Directory changed to /
> CWD /
< 250 Directory changed to /
> CWD /
< 250 Directory changed to /
> TYPE A
< 200 Type set to A.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,238)
> LIST
Data connection established with 66.46.177.250:1774 on 9/27/2005 @ 2:24:56 PM
< 150 Opening ASCII mode data connection for /bin/ls.
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
Displaying the contents of \FileSystem.txt, which is a FtpFile.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,240)
> RETR FileSystem.txt
Data connection established with 66.46.177.250:1776 on 9/27/2005 @ 2:24:57 PM
< 150 Opening BINARY mode data connection for FileSystem.txt (1198 Bytes).
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.

Displaying the contents of \appnote.txt, which is a FtpFile.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,246)
> RETR appnote.txt
Data connection established with 66.46.177.250:1782 on 9/27/2005 @ 2:24:58 PM
< 150 Opening BINARY mode data connection for appnote.txt (109785 Bytes).
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.

> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:25:04 PM

Each FtpFolder and FtpFile instance shared the same FtpConnection, and since no two operations were done at the same time, a single connection was required, as the log indicates. The FtpConnection object implements the IDisposable interface, since it keeps any active connection available until disposed (or finalized).

Now what happens if I try to open two files at the same time, like this?

using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )

{

  connection.TraceWriter = Console.Out;

  AbstractFile first = new FtpFile( connection, @"\FileSystem.txt" );

  AbstractFile second = new FtpFile( connection, @"\appnote.txt" );

  using( Stream firstStream = first.OpenRead() )

  {

    using( Stream secondStream = second.OpenRead() )

    {

      // In an FTP conversation with an FTP server, only one command

      // at a time can be pending. Here, we clearly have two files

      // open at the same time on the same FTP server. How? Each file

      // has its own connection to the FTP server!

    }

  }

}

The FtpConnection object will create extra command channel connections as required. The output shows two command channel connections were made:

Connected to 66.46.177.250:21 on 9/27/2005 @ 2:38:06 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> CWD /
< 250 Directory changed to /
> TYPE A
< 200 Type set to A.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,53)
> LIST
Data connection established with 66.46.177.250:2613 on 9/27/2005 @ 2:38:07 PM
< 150 Opening ASCII mode data connection for /bin/ls.
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,54)
> RETR FileSystem.txt
Data connection established with 66.46.177.250:2614 on 9/27/2005 @ 2:38:08 PM
< 150 Opening BINARY mode data connection for FileSystem.txt (1198 Bytes).
Connected to 66.46.177.250:21 on 9/27/2005 @ 2:38:08 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,55)
> RETR appnote.txt
Data connection established with 66.46.177.250:2615 on 9/27/2005 @ 2:38:09 PM
< 150 Opening BINARY mode data connection for appnote.txt (109785 Bytes).
< 426-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 426 Data connection closed, file transfer appnote.txt aborted.
> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:38:13 PM
> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:38:13 PM

And the great part about all this is that you don't have to worry about this while coding. You're just manipulating yet another kind of AbstractFile or AbstractFolder.

If we get back to the Zip implementation of the Xceed FileSystem, you can see that a ZippedFile or ZippedFolder (or ZipArchive, the root ZippedFolder) constructor needs to know which AbstractFile is holding the actual zip file that should contain this file or folder. "AbstractFile" truly means "any file", as long as there is an AbstractFile-derived class somewhere to expose this file. It means that zipping directly onto an FTP server is no more difficult than zipping in a regular file on disk.

AbstractFolder source = new DiskFolder( @"d:\Data" );

AbstractFolder localDest = new ZipArchive( 

  new DiskFile( @"d:\temp\local.zip" ) );

AbstractFolder remoteDest = new ZipArchive(

  new FtpFile( new FtpConnection( "localhost" ), @"remote.zip" ) );

// Copying is the same, no matter what is the destination

// file or folder.

source.CopyTo( localDest, true );

source.CopyTo( remoteDest, true );

Code for zipping in "D:\temp\local.zip" is no different than code for zipping in "ftp://localhost/remote.zip". And obviously, reading or unzipping from any zip file is the same.

AbstractFolder localSource = new ZipArchive( 

  new DiskFile( @"d:\temp\local.zip" ) );

AbstractFolder remoteSource = new ZipArchive(

  new FtpFile( new FtpConnection( "localhost" ), @"remote.zip" ) );

AbstractFolder dest = new DiskFolder( @"d:\restored" );

// Unzipping text files from any source is the same!

localSource.CopyFilesTo( dest, true, true, "*.txt" );

remoteSource.CopyFilesTo( dest, true, true, "*.txt" );

I really hope this new addition to the Xceed FileSystem will generate the same enthusiasm we had inventing and developping it. I'm very interested in hearing your opinions!

I'm currently working hard on the FileSystem support for Xceed FTP for .NET, and I had to share with you a screenshot of one of my tests. My XceedBox.NET sample is now fully supporting FtpFile and FtpFolder mapping.

As you can see, my working folder is a zip file located in a folder on an FTP server. The "type" command above has no idea what kind of file it is displaying. It receives an AbstractFile, gets a Stream on it via OpenRead, reads and displays the contents. That AbstractFile is actually a ZippedFile found in a ZippedFolder constructed around an FtpFile found in an FtpFolder.

In the new package v1.2.5309 which will be available for download next week resides a new feature you won't see much emphasis about, but which I was very eager to complete. You can now create a ZipArchive instance around an AbstractFile that does not support reading from.

(drum roll) ... (looking around) ... Nobody's applauding? That's because you probably don't know yet how useful this can be.

Most ASP.NET applications that wish to create zip files on the fly and send them in the response are either stuck with creating those zip files on disk in a temporary filename, or create them in a MemoryFile, then copy that MemoryFile in the response stream.

However, the StreamFile class was created for such purposes of exposing any existing Stream as an AbstractFile. You already could create a StreamFile around the Response's OutputStream. But passing that StreamFile to the ZipArchive's constructor would fail, because it can't read from it. Instead of assuming an empty zip file, it miserably failed. Shame.

No more... Since version 2.2.5302, it will assume the zip file is empty. So code like this works perfectly:

    public void ProcessRequest(HttpContext context)

    {

      context.Response.ContentType = "application/zip";

      context.Response.AddHeader( "Content-Disposition", "attachment; filename=images.bmp" );

      ZipArchive archive = new ZipArchive( new StreamFile( context.Response.OutputStream ) );

      DiskFolder source = new DiskFolder( context.Request.MapPath( "." ) );

      source.CopyFilesTo( archive, false, false, "*.bmp" );

    }

The same problem appeared when trying to combine Xceed Zip for .NET with Xceed FTP for .NET, to upload zip files directly on the FTP server. Though the FtpClient class exposes a very useful GetUploadStream method to get a direct stream on the data connection, code like this previously failed.

          using( Stream upload = client.GetUploadStream( "images.zip" ) )

          {

            ZipArchive archive = new ZipArchive( new StreamFile( upload ) );

            DiskFolder source = new DiskFolder( @"d:\images\" );

            source.CopyFilesTo( archive, false, false, "*.bmp" );

          }

Talk about short and sweet uploads of zip files!

Warning: Do not try this at home!

A few days ago, Pierre-Luc at support asked me if Xceed Zip for .NET was thread safe. I knew from his look that he was expecting a "yes" or a "no". At least, that's what the client who asked him the same question expected.

My first answer was more in nuances: Though the library was made to be safely accessible from multiple threads at the same time, by the nature of the sequential format of the zip file, it is not possible to work on the same zip file from multiple threads.

He nodded with approbation, confirming me his client wasn't trying such crazy action, but simply dealing with a multi-threaded application where each thread may be zipping in its own private file. I gave him my benediction: In that case, yes, Xceed Zip for .NET is thread safe.

Pierre-Luc wasn't two feet away when I was illuminated by an idea. It wouldn't be that crazy to try zipping into the same zip file from multiple threads. How neat would it be to benefit from multi processor or hyperthreading machines for zipping a single file? Guess what... you can! You shouldn't... but you can! Don't ask us to support this scenario... but you can!

Here's the deal. Any ZipArchive you modify gets updated when the last modify operation occurs. If you know you're about to make more than one modification to a single zip file, you should first call BeginUpdate, do all modifications, and finally call EndUpdate. The zip file will only get rebuilt on that final call. The files you copy into the zip file before EndUpdate will be compressed and stored in temp files within the ZipArchive's TempFolder.

That means any copy operation you perform within a BeginUpdate/EndUpdate block are atomic, and only involve compressing the sources into independant temp files. You see where I'm heading? How about spawning threads within that block, each thread copying its own source, and waiting for all threads to finish before calling EndUpdate?

I had to try it. I started with a class implementing IAsyncResult, which would be managing the copy operation on a separate thread:

But remember: Don't try this! I didn't tell you it was possible.

I had to take a look at an issue with a spanned zip file. The client graciously sent us the set of zip files he had on floppies. He copied each floppy in subfolders named "Disk1", "Disk2", "Disk3" and "Disk4", zipped those folders and sent us the resulting single huge zip file. Pretty standard.

I was about to copy each part back on floppies when I realized I was passing by a huge feature Xceed Zip for .NET offers regarding spanning and splitting: the ability to specify any AbstractFile, wherever it's located, when a new "disk" is required.

Before I show you how I skipped the unpleasant task of copying each part on separate floppies, let's first take a look at how you can support traditional spanning using Xceed Zip for .NET. The code below is what I call a minimum spanning implementation:

private static void Zip( AbstractFile zipFile, AbstractFolder sourceFolder )

{

  // Prepare ZipEvents object that will be passed to every method call.

  ZipEvents events = new ZipEvents();

  events.DiskRequired += new DiskRequiredEventHandler( ZipEvents_DiskRequired );

  // Create the target ZipArchive, and prepare for batch modifications.

  ZipArchive zip = new ZipArchive( events, null, zipFile );

  zip.BeginUpdate( events, null );

  try

  {

    // Allow this zip file to span.

    zip.AllowSpanning = true;

    // Zip this folder's contents

    sourceFolder.CopyFilesTo( events, null, zip, true, true );

  }

  finally

  {

    // Complete the batch modification of this zip file.

    zip.EndUpdate( events, null );

    events.DiskRequired -= new DiskRequiredEventHandler( ZipEvents_DiskRequired );

  }

}

private static void Unzip( AbstractFile zipFile, AbstractFolder destFolder )

{

  // Prepare ZipEvents object that will be passed to every method call.

  ZipEvents events = new ZipEvents();

  events.DiskRequired += new DiskRequiredEventHandler( ZipEvents_DiskRequired );

  // Create the source ZipArchive.

  ZipArchive zip = new ZipArchive( events, null, zipFile );

  try

  {

    // Unzip to destination folder.

    zip.CopyFilesTo( events, null, destFolder, true, true );

  }

  finally

  {

    events.DiskRequired -= new DiskRequiredEventHandler( ZipEvents_DiskRequired );

  }

}

private static void ZipEvents_DiskRequired(object sender, DiskRequiredEventArgs e)

{

  // Let the user know we need that disk and wait for feedback.

  Console.WriteLine( "Please insert disk #{0}, then press .", 
    e.DiskNumber.ToString() );

  Console.ReadLine();

  e.Action = DiskRequiredAction.Continue;

}

It's pretty straightforward. You simply give the user the time to insert the required disk. Don't forget that when unzipping, you must make sure the last zip file is the one available before creating your ZipArchive around it. There is no "insert last disk" event with Xceed Zip for .NET.

Now, back to my task. If you take a look at the DiskRequiredEventArgs parameter of the DiskRequired event, you see it exposes a "ZipFile" property of type AbstractFile. That's the AbstractFile for the part it's trying to locate. The above implementation requires a pause, to give time to the user to insert the correct disk. But what if the correct zip part is already available somewhere else? How about this implementation:

private static void ZipEvents_DiskRequired(object sender, DiskRequiredEventArgs e)

{

  // Let's check if the current zip file is located in a "DiskN" subfolder

  AbstractFolder subfolder = e.ZipFile.ParentFolder;

  if(  ( !subfolder.IsRoot )

    && ( subfolder.Name.ToUpper().StartsWith( "DISK" ) ) )

  {

    subfolder = subfolder.ParentFolder.GetFolder( 
      "Disk" + e.DiskNumber.ToString() );

    if( subfolder.Exists )

    {

      AbstractFile newZipFile = subfolder.GetFile( e.ZipFile.Name );

      if( newZipFile.Exists )

      {

        // No need to ask the user for the correct zip part, we found it!

        e.ZipFile = newZipFile;

        e.Action = DiskRequiredAction.Continue;

      }

    }

  }

  if( e.Action != DiskRequiredAction.Continue )

  {

    // Let the user know we need that disk and wait for feedback.

    Console.WriteLine( "Please insert disk #{0}, then press .", 
      e.DiskNumber.ToString() );

    Console.ReadLine();

    e.Action = DiskRequiredAction.Continue;

  }

}

As you can see, I have full control on what AbstractFile I provide to the library as the N^th zip file part. Since I have unzipped my client's zip file parts in my "D:\temp" folder, I can now call my Unzip method like this:

Unzip( 

  new DiskFile( @"d:\temp\Disk4\test.zip" ), 

  new DiskFolder( @"d:\temp\Unzipped" ) );

Then, the obvious striked me! Why did I unzip his "Floppies.zip" zip file in my "D:\temp" folder??? It's even simpler than I thought:

Unzip( 

  new ZippedFile( new DiskFile( @"d:\temp\floppies.zip" ), @"\Disk4\test.zip" ), 

  new DiskFolder( @"d:\temp\Unzipped" ) );

I'm unzipping zip file parts stored in a single zip file, without the need to have those parts really on floppies or even on disk. Wow! Files are files, folders are folders, no matter where they reside. Are you starting to get the idea behind the FileSystem?

A new Mono release today! As usual, I uninstall the previous release, install the new one, and test a few Xceed Zip for .NET console samples. Works perfectly!

But today I decided to try my Xceed FileSystem-based Command Prompt under Mono. Wow! Everything works almost perfectly. Zip, Ftp, Ram drive. Only a glitch with the Isolated Storage. This sample uses Mentalis.org's ConsoleAttributes library to customize the look and feel of the console. It works perfectly well with Mono (on Windows).

Any Mono users out there using Xceed products?

I've been working part time (translation: I should be working on something else) on a new sample: my own Command Prompt. I know, I'm reinventing the wheel, not to count that Microsoft will launch a new one called msh (codename Monad). But it was more a concept or proof around exposing AbstractFolder and AbstractFile within a command prompt.

E:\>dir

  Directory of E:\

      DATE     TIME     SIZE or TYPE NAME
27/12/2004  4:23 PM         [FOLDER] Backup
03/11/2004 10:33 AM         [FOLDER] Chart30
24/11/2004 10:09 AM         [FOLDER] CLR Profiler
11/01/2005  5:24 PM         [FOLDER] Config.Msi
24/11/2004 10:10 AM         [FOLDER] My Music
12/01/2005  4:13 PM         [FOLDER] My Pictures
10/09/2004  1:52 PM         [FOLDER] RECYCLER
30/09/2004  8:40 PM         [FOLDER] System Volume Information
02/02/2005  2:49 PM         [FOLDER] temp
03/11/2004 10:33 AM         [FOLDER] XceedProjectsNET
25/01/2005  7:06 AM              143 toto.txt

  Files: 1  Folders: 14  Total file size: 143

E:\>copy toto.txt temp
 100%
E:\>cd temp
E:\temp\>

As you can see, I can list the contents of folders, copy files, and change the working folder. The application simply manages a working "AbstractFolder", and enables commands to act on that folder (or an AbstractFolder obtained from an absolute path).

The sample quicky evolved into a prototype for upcoming features. Among other things, I needed a way to recognize a path like "E:\temp\test.zip\images" as a ZippedFolder within a zip file. Let's stop the talking, and show some traces:

E:\temp\>md test.zip
E:\temp\>md test.zip\images
E:\temp\>copy "..\My Pictures\Chalet\*" test.zip\images
 100%
E:\temp\>

What have I done here? Create a folder named "test.zip"? Well, the "md" command recognized the ".zip" extension as a request to create a new empty zip file. The second "md" command actually created a new folder within the zip file. And the paths can freely use the zip filename as a folder part for any command, as shown with the copy example. If we display the contents of "E:\temp", we see the two expected files:

E:\temp\>dir

  Directory of E:\temp\

      DATE     TIME     SIZE or TYPE NAME
02/02/2005  3:00 PM         40068736 test.zip
25/01/2005  7:06 AM              143 toto.txt

  Files: 2  Folders: 0  Total file size: 40068879

E:\temp\>

As you can see, "test.zip" is really a file (DiskFile) within "E:\temp" (DiskFolder). What happens if I try changing the current folder into that zip file?

E:\temp\>cd test.zip
E:\temp\test.zip\>dir

  Directory of E:\temp\test.zip\

      DATE     TIME     SIZE or TYPE NAME
02/02/2005  3:00 PM         [FOLDER] images

  Files: 0  Folders: 1  Total file size: 0

E:\temp\test.zip\>cd images
E:\temp\test.zip\images\>dir

  Directory of E:\temp\test.zip\images\

      DATE     TIME     SIZE or TYPE NAME
06/08/2000  4:40 PM          6400006 Chaises.bmp
06/08/2000  4:35 PM          6348550 Chute.bmp
06/08/2000  4:29 PM          6337678 Ciel1.bmp
06/08/2000  4:30 PM          6396226 Ciel2.bmp
06/08/2000  4:33 PM          6414418 Ciel3.bmp
06/08/2000  4:38 PM          6524278 Couple.bmp
06/08/2000  4:37 PM          6388054 Martine.bmp
06/08/2000  4:32 PM          6405478 Ombre.bmp
06/08/2000  4:41 PM          6359254 Rochers.bmp

  Files: 9  Folders: 0  Total file size: 57573942

E:\temp\test.zip\images\>

The zip file is exposed as a folder, because the path "E:\temp\test.zip" was recognized and mapped to a ZippedFolder around a DiskFile. And "images" is nothing more than a subfolder within that root ZippedFolder, actually something like:

new ZippedFolder( new DiskFile( @"E:\temp\test.zip" ), @"\images" );

Ok, let's get into serious things:

E:\temp\test.zip\images\>cd ..\..
E:\temp\>copy *.zip RAM:\
 100%
E:\temp\>cd RAM:\test.zip\images
RAM:\test.zip\images\>dir m*.bmp

  Directory of RAM:\test.zip\images\

      DATE     TIME     SIZE or TYPE NAME
06/08/2000  4:37 PM          6388054 Martine.bmp

  Files: 1  Folders: 0  Total file size: 6388054

RAM:\test.zip\images\>

My Command Prompt exposes a root MemoryFolder called "RAM:\", which I can freely use. The commands act the same, no matter if I'm deeling with a ZippedFolder around a DiskFile or a MemoryFile. Want more?

RAM:\test.zip\images\>cd ftp://vermouth
ftp://vermouth\>dir

  Directory of ftp://vermouth\

      DATE     TIME     SIZE or TYPE NAME

  Files: 0  Folders: 0  Total file size: 0

ftp://vermouth\>md foobar.zip
ftp://vermouth\>copy "E:\My Music\WMA\Mes Aieux" foobar.zip
 100%
ftp://vermouth\>dir

  Directory of ftp://vermouth\

      DATE     TIME     SIZE or TYPE NAME
02/02/2005  3:20 PM         62936759 foobar.zip

  Files: 1  Folders: 0  Total file size: 62936759

ftp://vermouth\>dir c:\inetpub\ftproot

  Directory of c:\inetpub\ftproot\

      DATE     TIME     SIZE or TYPE NAME
02/02/2005  3:20 PM         62936759 foobar.zip

  Files: 1  Folders: 0  Total file size: 62936759

ftp://vermouth\>

FTP servers are threated as any other kind of AbstractFolder. The application simply recognize the "FTP:" prefix as a signature for a root FtpFolder, as it did with "RAM:" exposed as a MemoryFolder. The command implementations don't care what kind of AbstractFolder or AbstractFile they are dealing with.

The engine behind this involves FileSystemMapper-derived classes. They mainly get asked two kinds of questions:

Question 1: Do you recognize this path as a root?

If so, they remove the part of the path they could recognize as a root folder, and return the matching AbstractFolder.

Examples of mappers and their responsability:

• DiskMapper : Drive letters and UNC paths (yes, you can "cd" into a UNC path!)
• FtpMapper : The "FTP:" prefix with server name, and optional username and password (e.g. ftp://user:pass@vermouth:9999)
• IsolatedStorageMapper : A custom prefix name like "STORE:" (that's the one my sample app supports).
• MemoryMapper : A custom prefix used to create the initial root MemoryFolder, like "RAM:" (that's the one my app supports). You can create more than one MemoryMapper to have more than one ram drive.

Question 2: Can you represent this AbstractFile as an AbstractFolder?

If so, they simply return the matching AbstractFolder.

An example of such a mapper:

• ZipFileMapper : It simply checks if the provided AbstractFile exists, then tries to create a ZipArchive around that AbstractFile in a try/catch. If it succeeds, it returns this ZipArchive (which derives from ZippedFolder).

Curiously, today I came across a post on our forums asking how to detect if a file is really a zip file. I gave this man the "new ZipArchive within a try/catch" solution, and he came back, as I feared, with concerns with the time wasted catching an exception for all those non-zip files. It's actually one of the bottlenecks of my Command Prompt sample. A lot of time is wasted throwing an exception for all non-zip files my app comes across. Well, I guess I'll have to work sooner than later on a new "ZipArchive.IsZipFile" method!

Now, you have to convince my boss I should put more time on this sample and these new FileSystem features! Does mapping absolute paths like shown above to their proper AbstractFolder or AbstractFile something that could be usefull for you?

Scott Hanselman just posted about a case-{in}sensitivity problem he just went through. That reminded me I wanted to talk to you about case-sensitivity in Xceed Zip for .NET and the FileSystem. I remember back in the design days, we debated long and hard on if the FileSystem should be case-sensitive or not. Once we decided to support both, the debate continued about what should be the default behavior.

The conclusions were simple:

1. Immitate by default.
2. Uniformity within single product.
3. Know the differences.

Since System.IO was case-insensitive (and moreover the whole Windows operating system), we had to be case-insensitive by default. Thus, if you have files "first.txt" and "second.TXT" in a folder, the code below will return you two files:

DiskFolder disk = new DiskFolder( @"t:\" );

AbstractFile[] files = disk.GetFiles( false, "*.txt" );

The same way, if you have both files in a zip file, the following code will return both:

ZipArchive zip = new ZipArchive( new DiskFile( @"t:\texts.zip" ) );

AbstractFile[] files = zip.GetFiles( false, "*.txt" );

Now, where it's getting tricky is that you will never have a folder on disk containing both "second.TXT" and "second.txt". The system won't let you create the second one. Thus the following code returns an existing file who's FullName is all lower case, even if the real file has an upper-case extension:

DiskFolder disk = new DiskFolder( @"t:\" );

AbstractFile file1 = disk.GetFile( "second.txt" );

You asked the "Disk" world for file "second.txt", and this world has recognized "second.TXT" as matching your request.

To obey to rule #2, the following code does exactly the same, even though the file stored in the zip file has its extension all upper case:

ZipArchive zip = new ZipArchive( new DiskFile( @"t:\texts.zip" ) );

AbstractFile file2 = zip.GetFile( "second.txt" );

But in a zip file, which can come from a different operating system, you potentially could end up with a zip file containing both. What would happen? I've created such a zip file for our tests, by adding "second.TXT" and "foobar.txt" to a zip file, and hex-editing "foobar" to "second":

second.zip (.23 KB)

When opening this file in WinZip, I can see both files. But when unzipping, it will unzip the first, then try to unzip the second over the first. You just can't unzip both in two separate files. Furthermore, trying to unzip any single one from within the classic view will always unzip both over the same file on disk.

How does Xceed Zip for .NET deal with such zip fles? Try the following code:

ZipArchive zip = new ZipArchive( new DiskFile( @"t:\second.zip" ) );

foreach( AbstractFile file in zip.GetFiles( false ) )

{

  Console.WriteLine( file.FullName );

}

The output is:

\second(1).txt
\second.TXT

Any file that case-insensitively matches another file gets appended a number. This is not a perfect solution, as there is never a perfect solution. To support rule #2, DiskFolder and ZippedFolder instances had to behave the same. This post and the documentation tries to address rule #3

Now, some of you want to always look for exact matches. You simply need to prepend the string mask with a ">", as in "I want a more precise match" ⁽¹⁾. The following code will match a single file:

DiskFolder disk = new DiskFolder( @"t:\" );

AbstractFile[] files = disk.GetFiles( false, ">*.txt" );

The idea with System.String filter parameters is that we replace them with a NameFilter, which is the one responsible for that ">" trick. It only works with methods accepting filters (GetFiles, GetFolders, CopyFilesTo, MoveFilesTo). Methods like GetFile can only return a single instance (actually always returns an instance which may exist or not). Those methods return the single and unique AbstractFile matching your string, based on the world this AbstractFolder belongs to.

⁽¹⁾: We actually debated between using "<" as in "match less items" or ">" as in "a more precise match". I think we ended up tossing a coin! 

I'm currently doing some tests on an alpha version of Xceed FTP for .NET, supporting the Xceed FileSystem. It's so wonderful to be able to manipulate files and folders no mather where they reside. Take this generic directory listing method:

private static void DisplayListing( AbstractFolder folder )

{

  if( !folder.Exists )

  {

    Console.WriteLine( "\n  Folder {0} does not exist.\n", folder.FullName );

  }

  else

  {

    FileSystemItem[] items = folder.GetItems( false );

    long totalSize = 0;

    int fileCount = 0;

    Console.WriteLine( "\n  Folder listing of {0}\n", folder.FullName );

    foreach( FileSystemItem item in items )

    {

      Console.Write( "{0} {1} ", 

        item.LastWriteDateTime.ToShortDateString(),

        item.LastWriteDateTime.ToShortTimeString() );

      AbstractFile file = item as AbstractFile;

      if( file == null )

      {

        Console.Write( "            " );

      }

      else

      {

        Console.Write( "{0,16} ", file.Size.ToString( "N0" ) );

        totalSize += file.Size;

        ++fileCount;

      }

      Console.WriteLine( item.Name );

    }

    int folderCount = items.Length - fileCount;

    Console.WriteLine( "\n  {0} file{1}, {2} folder{3}, {4} bytes\n",

      fileCount.ToString(),

      ( fileCount == 1 ) ? string.Empty : "s",

      folderCount.ToString(),

      ( folderCount == 1 ) ? string.Empty : "s",

      totalSize.ToString() );

  }

}

As you can see, the code does not need to know what exactly is that AbstractFolder. People familiar with Xceed Zip for .NET already know we could call the above method like this:

DiskFolder folder = new DiskFolder( @"C:\Program Files\Microsoft SDKs\WinFX" );

DisplayListing( folder );

And obtain this kind of output:

  Folder listing of C:\Program Files\Microsoft SDKs\WinFX\

 24/11/2004   9:29 AM   <DIR>          bin
 24/11/2004   9:27 AM   <DIR>          Help
 24/11/2004   9:27 AM   <DIR>          License
 24/11/2004   9:27 AM   <DIR>          misc
 24/11/2004   9:29 AM   <DIR>          Setup
 24/11/2004   9:29 AM   <DIR>          VS Install Directory
 11/11/2004   6:03 PM           16,001 ReleaseNotes.htm
 12/01/2005   2:58 PM           17,297 SetEnv.cmd

  2 files, 6 folders, 33298 bytes

Or call the same method like this:

ZippedFolder folder = new ZippedFolder( 
  new DiskFile( @"D:\sample.zip" ), "ContMenuExt" );

DisplayListing( folder );

To get this output:

  Folder listing of \ContMenuExt\

 08/01/2001  11:29 AM            8,091 ContextMenu.cpp
 02/01/2001   4:15 PM            1,005 ContextMenu.h
 08/01/2001  11:32 AM            7,820 ContextMenuExt.cpp
 28/11/2000  11:23 AM              225 ContextMenuExt.def
 02/01/2001   5:02 PM            5,301 ContextMenuExt.dsp
 28/11/2000  11:23 AM              551 ContextMenuExt.dsw
 28/11/2000  11:23 AM              742 ContextMenuExt.h
 02/01/2001   4:12 PM            1,440 ContextMenuExt.rc
 08/01/2001   3:29 PM                2 ReadMe.txt
 02/01/2001   4:12 PM            1,195 resource.h

  10 files, 0 folders, 26372 bytes

With the upcoming version of Xceed FTP for .NET, it won't be more difficult to display the contents of a folder located on an FTP server:

FtpConnectionInfo info = new FtpConnectionInfo( "ftp.microsoft.com" );

FtpFolder folder = new FtpFolder( info );

DisplayListing( folder );

  Folder listing of \

 25/11/2002  12:00 AM   <DIR>          bussys
 21/05/2001  12:00 AM   <DIR>          deskapps
 20/04/2001  12:00 AM   <DIR>          developr
 18/11/2002  12:00 AM   <DIR>          KBHelp
 02/07/2002  12:00 AM   <DIR>          MISC
 16/12/2002  12:00 AM   <DIR>          MISC1
 25/02/2000  12:00 AM   <DIR>          peropsys
 02/01/2001  12:00 AM   <DIR>          Products
 04/04/2003  12:00 AM   <DIR>          PSS
 21/09/2000  12:00 AM   <DIR>          ResKit
 25/02/2000  12:00 AM   <DIR>          Services
 25/02/2000  12:00 AM   <DIR>          Softlib

  0 files, 12 folders, 0 bytes

As a mather of fact, stuff like this already works fine on my machine:

FtpConnectionInfo info = new FtpConnectionInfo( "ftp.cam.org", "***", "***" );

FtpFile ftpFile = new FtpFile( info, @"\pub\Photos.zip" );

if( ftpFile.Exists )

  ftpFile.Delete();

DiskFolder sourceFolder = new DiskFolder( @"E:\My Pictures\Clément\Petites" );

ZipArchive destFolder = new ZipArchive( ftpFile );

sourceFolder.CopyFilesTo( destFolder, true, true );

DisplayListing( destFolder );

DisplayListing( ftpFile.ParentFolder );

This is the output:

  Folder listing of \

 16/09/2004   9:39 AM           11,820 Buzz.jpg
 16/09/2004   9:40 AM           11,143 Chalet - Bercé.jpg
 16/09/2004   9:42 AM           15,749 Clément et Michel.jpg
 16/09/2004   9:43 AM           18,473 Clément et Papa.jpg
 16/09/2004   9:42 AM           15,004 Clément et Valérie.jpg
 14/02/2003   4:17 PM            7,984 clément1.jpg
 14/02/2003   4:18 PM           11,288 clément2.jpg
 14/02/2003   4:18 PM           10,648 clément3.jpg
 11/08/2004   3:25 PM           18,499 Famille.jpg
 11/08/2004   3:23 PM           36,734 Fier.jpg
 16/09/2004   9:40 AM           14,959 Grande discussion.jpg
 16/09/2004   9:41 AM           13,426 Maman et Clément.jpg
 11/08/2004   3:25 PM           24,594 Piscine.jpg

  13 files, 0 folders, 210321 bytes

  Folder listing of \pub\

 27/12/2004  11:06 AM            7,986 Builds du 2004-09-27.htm
 12/01/2005   4:13 PM          208,439 Clement.zip
 27/12/2004  11:06 AM            1,068 FileSystem.txt
 18/01/2005   2:44 PM          208,423 Photos.zip
 27/12/2004  11:06 AM   <DIR>          Second
 27/12/2004  11:06 AM           11,723 VSSWarning.jpg
 27/12/2004  11:06 AM          117,695 appnote.txt
 27/12/2004  11:06 AM               96 vssver.scc

  7 files, 1 folder, 555430 bytes

Am I the only one to find this cool? q

Non-Transactional by defaut

The ZIP file format dictates by its nature that the creation of a zip file is a transactional operation. We cumulate a list of files to compress, with all metadata information, and we create the zip file in a single step, compressing each file sequentially, making sure to comply with a storage format that does not leave room for punctual updates. Imagine having to change the contents of a single file within a zip file. You have to rebuild the zip file from the beginning, by copying untouched files' compressed data to a new copy of the zip file, then append the modified file's compressed data, and complete the zip file with the new central directory and ending header.

On the opposite side, changing the contents of a file stored on your hard disk is simple. Each file is accessible randomly, and changing one's contents does not require moving or updating others. Take this for example:

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile file = new DiskFile( @"d:\mydata.txt" );

  if( !file.Exists )
    file.Create();

  using( Stream stream = file.OpenWrite( true ) )
  {
    stream.Write( mydata, 0, mydata.Length );
  }

The operation is atomic on the file. The Xceed FileSystem's goal is to mimic this random file access to any possible representation of a file. Thus, exposing compressed files stored in a zip file is no simple task. With the above code, if you replace new DiskFile(...) with new ZippedFile(...), it will work as expected. What you don't see is that only when the stream gets closed will the zip file get rebuilt. All data that you write to the stream is compressed and stored in a temp file, until the last "modify" operation is completed on that zip file. Another example:

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile file1 = new DiskFile( @"d:\mydata.txt" );
  AbstractFile file2 = new DiskFile( @"d:\mydatatoo.txt" );
  
  if( !file1.Exists )
    file1.Create();
    
  if( !file2.Exists )
    file2.Create();
    
  using( Stream stream1 = file1.OpenWrite( true ) )
  {
    using( Stream stream2 = file2.OpenWrite( true ) )
    {
      stream1.Write( mydata, 0, mydata.Length );
      stream2.Write( mydata, 0, mydata.Length );
    }
  }

In the atomic world of disk files, both files have no influence on the other. But again, replace DiskFile instances with ZippedFiles, and it's another story. The two files are stored in a zip file, which can only get rebuilt when the last "modify" operation completes, thus when "stream1.Close" is called. Will the above code work? Sure! But the zip file will be rebuilt three times. Try it!

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile zipFile = new DiskFile( @"d:\mydatafiles.zip" );
  AbstractFile file1 = new ZippedFile( zipFile, @"\mydata.txt" );
  AbstractFile file2 = new ZippedFile( zipFile, @"\mydatatoo.txt" );

  if( !file1.Exists )
    file1.Create();

  Console.WriteLine( "Check the zip file with WinZip!" );
  Console.WriteLine( "It should contain one empty file named 'mydata.txt'." );
  Console.ReadLine();

  if( !file2.Exists )
    file2.Create();

  Console.WriteLine( "Check the zip file with WinZip!" );
  Console.WriteLine( "It should contain two empty files now." );
  Console.ReadLine();

  using( Stream stream1 = file1.OpenWrite( true ) )
  {
    using( Stream stream2 = file2.OpenWrite( true ) )
    {
      stream1.Write( mydata, 0, mydata.Length );
      stream2.Write( mydata, 0, mydata.Length );
    }

    Console.WriteLine( "Check the zip file with WinZip!" );
    Console.WriteLine( "It still contains two empty files." );
    Console.ReadLine();
  }

  Console.WriteLine( "Check the zip file with WinZip!" );
  Console.WriteLine( "Now it contains both files with their data." );
  Console.ReadLine();

The first call to file1.Create increments the "modify" count to 1, then down to 0, so the zip file is built, containing an empty file. After the second call to Create, the zip file is again rebuilt, containing two empty files. When the first call to OpenWrite is made, the "modify" count gets up to 1. After the second call to OpenWrite, it's up to 2. Then stream2 is closed, and the count gets down to 1. Finally stream1 is closed, the count gets to 0, and the zip file is rebuilt, containing two files with compressed data.

In this simple example, the cost is not that much. Let's imagine worse:

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile zipFile = new DiskFile( @"d:\mydatafiles.zip" );

  if( zipFile.Exists )
    zipFile.Delete();

  for( int i=0; i<1000; i++ )
  {
    Console.WriteLine( "Loop {0}", i );
    AbstractFile file = new ZippedFile( zipFile, @"\data" + i.ToString() + ".txt" );

    if( !file.Exists ) 
      file.Create(); 

    using( Stream stream = file.OpenWrite( true ) ) 
    {
      stream.Write( mydata, 0, mydata.Length ); 
    }
  }

If you try this, you'll notice that each loop takes more time than the previous. Actually, when I tried this, I wasn't patient enough to wait until completion. The zip file would get rebuilt 2000 times, with more and more files already in the zip file. This is plainly unacceptable.

Transactional on demand

That's where the IBatchUpdateable interface comes to the rescue. It contains two simple methods: BeginUpdate and EndUpdate. Any AbstractFile or AbstractFolder's derived class can implement this interface, though you can limit this to the root folder. Once BeginUpdate is called, the implementor can hold any modifications to the underlying media until EndUpdate is called. ZipArchive, which represents the root ZippedFolder for a zip file, implement this interface. In short, BeginUpdate artificially increments the "modify" count to 1, and EndUpdate decrements it. If it gets to 0, the underlying zip file is rebuilt. You can call BeginUpdate and EndUpdate as many times as you want, but every call to BeginUpdate must be matched with a call to EndUpdate. The above code could now look like this:

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile zipFile = new DiskFile( @"d:\mydatafiles.zip" );
  ZipArchive zip = new ZipArchive( zipFile );

  if( zipFile.Exists )
    zipFile.Delete();

  zip.BeginUpdate();

  try
  {       
    for( int i=0; i<1000; i++ )
    {
      Console.WriteLine( "Loop {0}", i );
      AbstractFile file = new ZippedFile( zipFile, @"\data" + i.ToString() + ".txt" );

      if( !file.Exists ) 
        file.Create(); 

      using( Stream stream = file.OpenWrite( true ) ) 
      {
        stream.Write( mydata, 0, mydata.Length ); 
      }
    }
  }
  finally
  {
    zip.EndUpdate();
  }

Now that's better. On my machine, this takes a few seconds.

The FileSystem's main goal was to offer a unique and consistent interface for manipulating any kind of file or folder. That's why we decided that ZippedFile and ZippedFolder were to be non-transactional by default, even though in most cases, it will end-up producing less efficient code. It's the user's job to call BeginUpdate before modifying the zip file, and EndUpdate once completed, to achieve better performance.

By the way, for those who like the using( IDisposable ) pattern in C#, you can use the AutoBatchUpdate class like this:

  byte[] mydata = System.Text.Encoding.Default.GetBytes( "This is important!" );
  AbstractFile zipFile = new DiskFile( @"d:\mydatafiles.zip" );
  ZipArchive zip = new ZipArchive( zipFile );

  if( zipFile.Exists )
    zipFile.Delete();

  using( AutoBatchUpdate auto = new AutoBatchUpdate( zip ) )
  {
    for( int i=0; i<1000; i++ )
    {
      Console.WriteLine( "Loop {0}", i );
      AbstractFile file = new ZippedFile( zipFile, @"\data" + i.ToString() + ".txt" );

      if( !file.Exists ) 
        file.Create(); 

      using( Stream stream = file.OpenWrite( true ) ) 
      {
        stream.Write( mydata, 0, mydata.Length ); 
      }
    }
  }

The AutoBatchUpdate implements IDisposable, making sure to call BeginUpdate on the object at construction, and EndUpdate when disposed. What's even better is that you can pass any FileSystemItem: it will do nothing if the item's RootFolder does not implement IBatchUpdateable. Thus, you can use AutoBatchUpdate without having to know if the AbstractFile or AbstractFolder you're working with implements IBatchUpdateable or not.

Temp storage

Now, it's good to know that when using BeginUpdate and EndUpdate, the zip file is rebuilt only at the very last moment, but where goes the compressed data I'm writing to the streams? It must be stored somewhere, right? The ZipArchive class exposes two important properties: DefaultTempFolder (static) and TempFolder. By default, the first is equal to new DiskFolder( System.IO.Path.GetTempPath() ), the temp folder of the currently logged-in user. You can assign to it any AbstractFolder, as long as AbstractFile instances created in that folder yield seekable streams (ZippedFile.OpenWrite does not return a seekable stream).

Everytime you create the first instance of a ZipArchive for a given zip file, its TempFolder property is initialized to the value of DefaultTempFolder. Thus, if you assign a folder to the static DefaultTempFolder property, it will apply to all new instances of ZipArchive. If you assign a folder to the TempFolder property, it will only affect ZippedFile, ZippedFolder and ZipArchive instances dealing with that zip file.

If you run the above code while watching your temporary folder using Explorer (hit F5 a few times), you'll see appear and disapear filenames like "XFS330fe108-13b8-4ebb-2299-cace5fa0100a.tmp". Those files are holding the compressed data until the zip file gets rebuilt. Most serious zip libraries allow to use memory instead of a disk folder while zipping. For example, the Xceed Zip ActiveX exposes the UseTempFile property. When set to false, the library stores temp data in memory while building the zip file. With Xceed Zip for .NET, you achieve this by setting ZipArchive.DefaultTempFolder to new MemoryFolder(). Voilà! You are storing temporary data in memory. This is very useful for ASP.NET applications that cannot write on disk. And even better: it also works when updating existing zip files. But watch out! Don't zip gigabytes of files while using a MemoryFolder. There is a time for a MemoryFolder, and there is a time for a DiskFolder.

One of the gotchas of the Xceed FileSystem is its behaviour when it can't process a file a folder: it throws an exception right away. Call it lazy by default! Take this code for example:

AbstractFolder folder = new DiskFolder( @"c:\" );
AbstractFile[] gdiFiles = folder.GetFiles( true, "gdiplus.dll" );

foreach( AbstractFile file in gdiFiles )
{
  Console.WriteLine( "{0}: {1}", 
    file.LastWriteDateTime.ToShortDateString(), file.FullName );
}

You would expect this to list all gdiplus.dll files found on your C: drive. On most computers, it will throw an UnauthorizedAccessException, because the currently logged in user probably does not have read access to all folders, like "System Volume Information" or other users' profiles under "Documents and Settings".

Does that mean you're doomed? No. Every time a method that processes many items encounters a problem with an item, it will raise the ItemException event. If you handle this event, you have access to the current and target item (if applies) and the exception that is about to be thrown. More important, you can instruct the sender to try again on the same item, skip the faulty item and continue with the next one, or abort the current operation and throw the exception (default).

Before we take a closer look, you must understand how events work with the FileSystem. Since all files and folders are represented by their own instance, it would be unthinkable to advise for events on each and every one, or have delegates propagated on every child instance a folder creates for its processing.

Instead, every method that can raise events exposes pairs of overloads, one taking a FileSystemEvents and object as its first two parameters. You create yourself a single FileSystemEvents instance (or ZipEvents), advise for events you wish to handle on it, then pass it to every method which can accept one. The second object parameter can be anything you wish, and is passed back to the event handler. In French, we call this a "fourre-tout" parameter. Babelfish translates this to "hold-all". The library calls it "user data".

The above code would now look like this:

FileSystemEvents events = new FileSystemEvents();
events.ItemException += new ItemExceptionEventHandler( FileSystemEvents_ItemException );

AbstractFolder folder = new DiskFolder( @"c:\" );
AbstractFile[] gdiFiles = folder.GetFiles( events, null, true, "gdiplus.dll" );

foreach( AbstractFile file in gdiFiles )
{
  Console.WriteLine( "{0}: {1}", 
    file.LastWriteDateTime.ToShortDateString(), file.FullName );
}

With the event handler doing the simplest thing:

private static void FileSystemEvents_ItemException( object sender, ItemExceptionEventArgs e )
{
  System.Diagnostics.Debug.WriteLine( e.Exception.Message );
  e.Action = ItemExceptionAction.Ignore;
}

Good. This should work. You run it. You see a blank screen. Nothing seems to be running. Sure, you ear your hard drive rumbling, but your application looks dead. That's bad. The ScanningFolder, ItemProgression and ByteProgression events come to the rescue. The first one is fired everytime the contents of a folder is retrieved. The second one is fired when an item is about to be copied or moved. The last one is fired every 64k of copied or moved data between files. In our case, we will advise for the ScanningFolder event, by adding this line:

events.ScanningFolder += new ScanningFolderEventHandler( FileSystemEvents_ScanningFolder );

And implementing this event handler:

private static char[] mg_animationChars = new char[] { '|', '/', '-', '\\' };
private static int mg_animationIndex = 0;

private static void FileSystemEvents_ScanningFolder( object sender, ScanningFolderEventArgs e )
{
  Console.Write( mg_animationChars[ mg_animationIndex++ ] );
  Console.Write( '\b' );
  mg_animationIndex %= mg_animationChars.Length;
}

Wow! DOS-style animation, my favourites! (Hey, it was my school). For the remaining events, it's yours to discover. Feel free to post your questions on our forums. Though we do not guarantee a fast answer, I personally try to monitor my own products' groups as often as possible.

Before I dive into the Xceed FileSystem (the heart of Xceed Zip for .NET), a quick tour of the object model is required. At the heart of the Xceed FileSystem are three abstract classes: the base FileSystemItem class and the two derived AbstractFile and AbstractFolder classes. They expose the basic operations we can do on files and folders, like copy, move, create, delete, read from or write to, browse contents, get or change attributes and dates. You're already familiar with all this, either using Windows Explorer, a third party shell explorer, or the good old command prompt.

Files and folders are not only found on disk drives, they can be located in memory, on an FTP or HTTP server, on your Windows CE device, in the isolated storage, in your favorite source control system, in an email as an attachment, name it. Sometimes, the device is smart enough to expose itself as a real drive. For example, ram drives are usually accessible through normal file paths. They acquire a drive letter and trick the OS in thinking they are a legitimate disk drive. Another good example is Daemon Tools, which lets you expose a CD ISO image located on your hard disk as if it was a CD drive. In both cases, normal IO APIs can access them with the proper file paths.

But most often, integration stops at the shell or application level. For example, ActiveSync will let you access files and folders on your Windows CE device through Windows Explorer, as if they were normal files and folders. But don't try using classes under System.IO to access them. It won't work. You have to revert to RAPI, and manage the bridge between "worlds" yourself.

Another example is the zip file support in Windows XP (and originally in Windows 98 with the Plus! pack). From Windows Explorer, it lets you see zip files as if they were folders. What is a folder? A "place" where files are stored. What is a zip file? A "place" where files are stored. Makes sense! Neet. But again, something like this won't work:

System.IO.Directory.GetFiles( @"d:\data.zip", "*.bmp" );

For classes under System.IO, "d:\data.zip" is an unknown binary file, it's not a folder. You need third party libraries to read from that zip file, using a proprietary API which makes sense only for zip files. And most often, those libraries will support files accessible with a path or streams. Beyond that, you'll need to code it yourself.

The Xceed FileSystem removes those barriers between devices, virtual file systems, remote data, etc. A file is a file. A folder is a folder. Who cares how it works underneath. Let's says I want to get image files located in a particular folder of a zip file located on an FTP server, and copy them into a project folder located in a VSS database located on a server accessible through a UNC path.

AbstractFile ftpFile = new FtpFile( "ftp.xceedsoft.com", @"\pub\resources.zip" );
AbstractFolder zipFolder = new ZippedFolder( ftpFile, @"\Images" );

AbstractFile vssIniFile = new DiskFile( @"\\Repository\WebDepot\srcsafe.ini" );
AbstractFolder vssFolder = new VssFolder( vssIniFile, @"$\Intro\Logos" );

foreach( AbstractFile imageFile in zipFolder.GetFiles( false, "*.bmp" ) )
{
  imageFile.CopyTo( vssFolder, true );
}

An FtpFile is a file. A ZippedFolder is a folder. A DiskFile is a file. A VssFolder is a folder. Once you have the instances, working with them hides their implementation. You can copy files. You can browse folder contents.

Another example. We can read from files. We can write to files. Something like this makes perfect sense:

AbstractFile logFile = new DiskFile( @"d:\myapp.log" );

if( !logFile.Exists )
  logFile.Create();

string line = string.Empty;

using( StreamWriter writer = new StreamWriter( logFile.OpenWrite( false ) ) )
{
  while( ( line = Console.ReadLine() ).Length > 0 )
  {
    writer.WriteLine( line );
  }
}

using( StreamReader reader = new StreamReader( logFile.OpenRead() ) )
{
  while( ( line = reader.ReadLine() ) != null )
  {
    Console.WriteLine( line );
  }
}

A file is a file, right? Then changing the first line with this should not surprise you:

AbstractFile zipFile = new DiskFile( @"d:\myappdata.zip" );
AbstractFile logFile = new ZippedFile( zipFile, @"\logs\today.log" );

It still works. A zip file named "myappdata.zip" is created in the "D:\" folder, and contains a file named "logs\today.log" which contains what you typed.

Xceed Zip for .NET comes with the following classes:

DiskFile and DiskFolder are your regular disk file and folder accessible with a file path (drive letter or UNC path). MemoryFile and MemoryFolder virtually let you create ram drives for your process. IsolatedFile and IsolatedFolder give you access to the user isolated storage (System.IO.IsolatedStorage). Finally, StreamFile is a special implementation of an AbstractFile that lets you expose any stream as if it was a file.

You wont find the FtpFile and FtpFolder classes (yet). Though they are "feature complete", they will be available in Xceed FTP for .NET 2.1 (hopefully this year, or early next year). As for VssFile and VssFolder they are incomplete academic implementations someone here did for a project. Because that's the fun part: nothing's sealed. AbstractFile and AbstractFolder are waiting for you to derive from them. As a matter of fact, MemoryFile and MemoryFolder were first exposed as a sample on how to derive from AbstractFile and AbstractFolder. But they are so usefull that they are now first class citizens. Do you have other ideas of file and folder implementations that should be available in the Xceed FileSystem?

One of the software pieces I’ve made in my career which I’m the most proud of is the Xceed FileSystem. Ever since I’ve started programming in object oriented languages, I was convinced that files and folders were a base representation and starting implementation, no matter where they resided, how they were encoded, or how they could be accessed.

Back in my COM days, I’ve tried to accommodate my object-oriented vision of files and folders to the feature-oriented paradigm of COM. I remember a design meeting for Xceed FTP ActiveX where I was pushing for IFile and IFolder interfaces, with coclasses like “FtpFile”, “FtpFolder”, “DiskFile” or “DiskFolder”. But COM wasn’t ready for this… and moreover, neither were our customers.

I will always remember my first encounter with System.IO.FileInfo and System.IO.DirectoryInfo. It was Xceed’s early contact with that new .NET Framework platform Microsoft was planning to launch a year later; a fully object-oriented API! At last! Oh what a joy it was to see them displayed in front of me, waiting for me to derive from them and implement numerous new worlds. Sure, I was a little disappointed to see the base classes implementing disk I/O, as opposed to having derived classes do the job, but it was an early beta, and I convinced myself I could influence Microsoft to correct this minor mistake. Right?

Well… Those were my first impressions. I was dreaming of an object-oriented API, and the .NET Framework would slowly drift away from this idealist goal. I was dreaming of a Microsoft able to change stuff in 15 minutes just because they were suddenly more accessible.

A few weeks later, a colleague and I were at Redmond to get a peek at the next beta version of the .NET Framework, ready to convince whom it may concern that System.IO.FileSystemEntry et al were not quite perfect… and kaboom! It hit us: they had sealed FileInfo and DirectoryInfo in this most recent build. How could they not see the obvious? How could they not see “my” object-oriented file system?

That week, we did meet the System.IO guys, and they listened. They agreed the idea was cool. But the conclusion was just more evident: the .NET Framework’s goal was not to provide purely object-oriented framework. It was to provide a language-neutral, easy to understand framework. Being well-suited for object-oriented languages was not enough. And Kit George gave us the shoulder tap we needed. He told us our object-oriented file system sounded pretty cool. Why wait for the base classes coming from MS?

And that week, in building 20, Pascal Bourque and I designed the major part of Xceed Zip for .NET, with its object-oriented file system. It was a very special week. The kind you miss…