First, I wish all my readers health and happyness for the new year.

Now, let's jump into the subject of the day: Scott Hanselman's HanselMinutes. I'm currently listening to his first podcast. I've never been a real fan of podcasts, but since Scott Hanselman is about my number 1 blogger, I could not miss this event.

Hmmm, how can I express my feelings about podcasting without hurting Scott's feelings? Is it me, or are computer subjects not fit for audio? I want links! I want screenshots! I want examples! I want immediate access to extended information upon my needs! With a podcast, I'm stuck listening to all the stream. Sure, I can fast forward, but you end-up playing the "find that show you recorded" game you play with your VCR. Worse, you don't know what you're looking for. You are at the mercy of the podcaster. You can't filter, you can't opt in or out of a subject.

Maybe I'm not listening podcasts at the proper moment? Maybe I'm trying to use podcasts as if they were audio blogs, which they are not? I tried listening to a podcast in my car on the way to work, just to discover I was sad missing the local news and forecasts I usually listen to in the morning. I tried listening to a podcast at home in the evening when I push my computer geekness to its limits by moving back to a computer, but I generally need to disconnect from work, and I prefer playing Guild Wars! I tried listening to a podcast in bed before getting to sleep, just to find out I prefer doing other things in bed... like sleeping... and... ok, you get the picture!

The funny part is that I've been approached by the Visual Studio Talk Show for a 45 minutes podcast-style interview in French, and I've said yes. But it's only a one time deal. Even though this show is mostly accessible as a podcast, I see this as an interview, and no way I could maintain a weekly podcast.

So I'll conclude with Scott's own words: podcasting sucks. It wastes my precious time. I would have liked it very much if Carl Franklin would have asked Scott about his background, his developer path, about himself. I want to know more about Scott. For links, I'll continue reading his blog.

Oh, and one more thing: the damn advertising is barely tolerable.

I must admit, I'm no database wizz. It's been a looooong time since I've played with SQL, and I never really digged into System.Data. The rare moments I required a data connection, the design-time experience was enough, oh, and that Fill call on that adapter! q

Recently, a customer explained he was using an SqlDataReader to fetch its data on demand, to avoid loading too much. One of the fields was a byte array containing a GZip file. He was using the static method GZipCompressedStream.Decompress to get the data in that GZip file. Unfortunately, the data was sometimes quite large, and this technique prevented him from using SqlDataReader.GetBytes (exposed in the IDataRecord interface), which allows to read only chunks of a field at a time.

Bummer... My "Streams everywhere" modo was challenged. You see, Xceed Streaming Compression for .NET allows you to either decompress a single byte array in one operation (stateless), or wrap a GZipCompressedStream around your source stream and read from it to decompress data on the fly. But in this case, no streams. Nada. Or if there is one, I didn't find it.

I was not going to get defeated by that mere absence. The "Streaming" in "Xceed Streaming Compression for .NET" is exactly about that scenario. It turns out it was quite easy to overcome this little problem. I created myself a DataRecordStream class, which derives from System.IO.Stream.

Apart from the usual overrides required when deriving from System.IO.Stream, I expose a constructor requiring an IDataRecord parameter and the index of the field to expose as a stream.

    public DataRecordStream( IDataRecord record, int fieldIndex )
    {
      if( record == null )
        throw new ArgumentNullException( "record" );

      if( ( fieldIndex < 0 ) || ( fieldIndex >= record.FieldCount ) )
        throw new ArgumentOutOfRangeException( "fieldIndex", fieldIndex, "Invalid field index." );

      m_record = record;
      m_field = fieldIndex;
    }

Then, when Read is called, I simply turn to my IDataRecord's GetBytes method to fill that buffer.

    public override int Read( byte[] buffer, int offset, int count )
    {
      long read = m_record.GetBytes( m_field, m_position, buffer, offset, count );

      m_position += read;

      if( ( m_length != -1 ) && ( m_position > m_length ) )
      {
        // The reported length was smaller than the actual size.
        // We update the length dynamically.
        m_length = m_position;
      }

      return unchecked( ( int )read );
    }

The rest is just glue for managing the position and allowing seeking. The good part about this new class is that you can now wrap any pass-thru stream around it, for example a GZipCompressedStream. My customer can now read text compressed in a GZip file stored in one of its database fields quite easily, without consuming too much memory.

      SqlConnection connection = new SqlConnection( 
        "integrated security=SSPI;data source=xxx;initial catalog=GZipTest" );

      connection.Open();

      try
      {
        using( SqlCommand command = new SqlCommand( "SELECT * FROM GZipTestTable", connection ) )
        {
          using( SqlDataReader dataReader = command.ExecuteReader() )
          {
            while( dataReader.Read() )
            {
              using( StreamReader textReader = new StreamReader(
                       new GZipCompressedStream(
                         new DataRecordStream( 
                           dataReader, dataReader.GetOrdinal( "GZipField" ) ) ) ) )
              {
                string line;

                while( ( line = textReader.ReadLine() ) != null )
                {
                  Console.WriteLine( line );
                }
              }
            }
          }
        }
      }
      finally
      {
        connection.Close();
      }

I have made a VB.NET version of the class too. Enjoy!

DataRecordStream.zip (3.34 KB)

I previously gave a glimpse of how to zip into an HttpResponse's OutputStream, but it wasn't explaining all aspects of zipping from ASP.NET. So I'll get in more details here.

First, I have used my fantastic talent in UI designs to create this web page:

Yup, three checkboxes and a button is enough gadgets for me!

The first piece of code involves Application_Start. Since I know I won't be zipping gazillions of bytes, I want my web page to use memory as a temporary location for compressed data. How you do this with Xceed Zip for .NET is simple: You create a RAM drive! Oh the good old days of RAM drives...

    protected void Application_Start(Object sender, EventArgs e)
    {
      Xceed.Zip.Licenser.LicenseKey = "ZIN23-#####-#####-####";
      ZipArchive.DefaultTempFolder = new MemoryFolder();
    }

This new MemoryFolder is acting exactly like a per-process RAM drive. It's an AbstractFolder like any other AbstractFolder. The TempFolder of all new ZipArchive instances will be initialized to that value. Application_Start is also a great place where to set your license key, before anything else.

We're now ready for the button's click event. Again, I want to avoid write access on the hard drive, and wish to zip directly in the response stream. But the idea behind the Xceed FileSystem is to copy source files and folders to destination files and folders. How can I zip into a Stream? The StreamFile class comes to the rescue. It lets you expose a Stream as if it were an AbstractFile. Then, you can pass this StreamFile to the ZipArchive's constructor, to tell that new instance to write into that Stream. The rest is glue code for my wonderful ASP.NET application to zip the correct files.

    private void Button1_Click(object sender, System.EventArgs e)
    {
      if( !CheckBox1.Checked && !CheckBox2.Checked && !CheckBox3.Checked )
      {
        // Redirect to error page...
        return;
      }

      // The "MACHINE\ASP_NET" user must have read access to that folder.
      DiskFolder source = new DiskFolder( @"d:\" );

      // We want the client-side to recognize the upcoming file as a zip file.
      this.Response.ContentType = "application/zip";
      this.Response.AddHeader( "Content-Disposition", "attachment; filename=YourFiles.zip" );

      // We will zip directly in the response stream. The temporary compressed
      // data will be written to the ZipArchive's TempFolder, thus the MemoryFolder 
      // we set in Application_Start.
      ZipArchive destination = new ZipArchive( new StreamFile( this.Response.OutputStream ) );

      // And finally we zip in a single operation. If we had to zip more than
      // one source, we could have used ZipArchive.BeginUpdate/EndUpdate.
      ArrayList nameFilters = new ArrayList();

      if( CheckBox1.Checked )
        nameFilters.Add( new NameFilter( "*.txt" ) );

      if( CheckBox2.Checked )
        nameFilters.Add( new NameFilter( "*.jpg" ) );

      if( CheckBox3.Checked )
        nameFilters.Add( new NameFilter( "*.exe|*.dll" ) );

      // Passing more than one filter to CopyFilesTo does an "AndFilter"
      // by default.
      Filter mainFilter = ( nameFilters.Count == 1 )
        ? nameFilters[ 0 ] as Filter
        : new OrFilter( nameFilters.ToArray( typeof( NameFilter ) ) );

      source.CopyFilesTo( destination, false, true, mainFilter );

      this.Response.End();
    }

We now have an ASP.NET application which only requires read access to the source files and folders to zip. Everything else is done in memory, without drifting away from the logic of the Xceed FileSystem; manipulating files and folders.

Just in case my previous post on the subject did not ring a bell, the release of version 2.1 of Xceed FTP for .NET means you can directly unzip from a zip file located on an FTP server, without downloading the file first! Look at the following code:

  using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )
  {
    FtpFile source = new FtpFile( connection, @"/images/Flowers/Backup/Flowers.zip" );
    DiskFolder dest = new DiskFolder( @"d:\temp\flowers" );

    ZipArchive zip = new ZipArchive( source );
    zip.CopyFilesTo( dest, true, true );
  }

The secret behind this code is the kind of stream "FtpFile.OpenRead" returns. Though we are dealing with a network connection, this stream is fully seekable! The FtpFile takes advantage of the "REST" FTP command, which tells the FTP server we wish to start the transfer at a specific offset. Thus, when the ZipArchive needs to seek at the end of the file to locate the ending header, a proper "REST" command is issued to avoid having to read all the zip file first. And the same happens when reading the central directory, or unzipping specific files.

I have put forward my incredible talent with ASCII art (sic) and updated my wonderful "FileSystem.txt" file, which describes classes available with the Xceed FileSystem for .NET.

                             ==============
                             FileSystemItem
                             ==============
                                    |
                     +--------------+------------------+
                     |                                 |
              ==============                     ============
              AbstractFolder                     AbstractFile
              ==============                     ============
                     |                                 |
         +---+---+---+---+---+---+     +---+---+---+---+---+---+---+
         |   |   |   |   |   |   |     |   |   |   |   |   |   |   |
 ==========  |   |   |   |   |   |     |   |   |   |   |   |   |  ========
 DiskFolder  |   |   |   |   |   |     |   |   |   |   |   |   |  DiskFile
 --============  |   |   |   |   |     |   |   |   |   |   |  ==========--
   MemoryFolder  |   |   |   |   |     |   |   |   |   |   |  MemoryFile
   --==============  |   |   |   |     |   |   |   |   |  ============--
     IsolatedFolder  |   |   |   |     |   |   |   |   |  IsolatedFile
     ------============  |   |   |     |   |   |   |  ============----
           ZippedFolder  |   |   |     |   |   |   |  ZippedFile
           -------=========  |   |     |   |   |  =======-------
                  FtpFolder  |   |     |   |   |  FtpFile
                  -============  |     |   |  ==========-
                   TarredFolder  |     |   |  TarredFile
                   ---=============    |  ===========---
                      GZippedFolder    |  GZippedFile
                      -------------   ==========-----
                                      StreamFile
                                      ----------

Can you guess what I'm working on?

Have you ever thought of changing career? How about hockey? Here's your chance! Become one of the lucky graduates of the NHL Academy.

(Thanks Jeff, you made my day.)

I'm glad to announce that Xceed FTP for .NET 2.1 is now available for download. I've been working on this release for the past few months, and I'm very excited to finally see the FTP FileSystem come to life.

For those not familiar with the Xceed FileSystem, which comes with Xceed Zip for .NET, here is some code that sheds some light on what you can do with it. Consider these variables of the same base type:

// A file on disk

AbstractFile first = new DiskFile( @"d:\FileSystem.txt" );

// Another file on disk

AbstractFile second = new DiskFile( @"c:\temp\AnotherFileSystem.txt" );

// A file compressed in a zip file on disk

AbstractFile third = new ZippedFile( 

  new DiskFile( @"c:\temp\data.zip" ), "FileSystemInAZip.txt" );;

// A file in the isolated storage

AbstractFile fourth = new IsolatedFile( "Isolated.txt" );

// A file in memory (random name)

AbstractFile fifth = new MemoryFile();

// A file compressed in a zip file in memory

AbstractFile sixth = new ZippedFile( 

  new MemoryFile(), "VolatileFileSystem.txt" );

You can copy files around very easily:

// Copying the first file anywhere else is always the same!

first.CopyTo( second, true );

first.CopyTo( third, true );

first.CopyTo( fourth, true );

first.CopyTo( fifth, true );

first.CopyTo( sixth, true );

And accessing the contents of any file is always the same:

private void DisplayTextFile( AbstractFile file )

{

  Console.WriteLine( "Displaying the contents of {0}, which is a {1}.", 

    file.FullName,

    file.GetType().Name );

  using( StreamReader reader = new StreamReader( file.OpenRead() ) )

  {

    string line;

    while( ( line = reader.ReadLine() ) != null )

    {

      Console.WriteLine( line );

    }

  }

  Console.WriteLine();

}

// Displaying the contents of those files is always the same!

DisplayTextFile( first );

DisplayTextFile( second );

DisplayTextFile( third );

DisplayTextFile( fourth );

DisplayTextFile( fifth );

DisplayTextFile( sixth );

And why not finish this demonstration by deleting the files we just created.

// And finally, deleting files is the same!

second.Delete();

third.Delete();

fourth.Delete();

fifth.Delete();

sixth.Delete();

Any kind of file is an AbstractFile. Any kind of folder is an AbstractFolder. This way, a DiskFile, an IsolatedFile, a ZippedFile and a MemoryFile share a common set of properties and methods for accessing their metadata and reading/writing their actual data. And a DiskFolder, an IsolatedFolder, a ZippedFolder and a MemoryFolder share a common set of methods for discovering child items.

How does the FTP FileSystem come into play? Simply by offering the same abstraction over files and folders stored on an FTP server. We could simply add this code to the above sample, and everything works just as expected!

AbstractFile seventh = new FtpFile( 

  new FtpConnection( "localhost" ), @"\RemoteFileSystem.txt" );

first.CopyTo( seventh, true );

DisplayTextFile( seventh );

seventh.Delete();

Let's dive a little bit into this implementation of an AbstractFile and AbstractFolder. Each constructor requires an FtpConnection instance, which contains information on how to connect to the target FTP server. Though it looks like a simple information storage class, it does much more. Each time an FtpFile or an FtpFolder requires information, or an incoming or outgoing stream on a file's data, it asks the FtpConnection for an active command channel connection to the server. This way, a unique command channel is generally required for accessing many files on the server.

using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )

{

  connection.TraceWriter = Console.Out;

  FtpFolder root = new FtpFolder( connection );

  foreach( AbstractFile file in root.GetFiles( false, "*.txt" ) )

  {

    DisplayTextFile( file );

  }

}

If we comment out the "Console.WriteLine( line );" line in "DisplayTextFile", we can see the FTP conversation that occurred for the above code:

Connected to 66.46.177.250:21 on 9/27/2005 @ 2:24:54 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
> CWD /
< 250 Directory changed to /
> CWD /
< 250 Directory changed to /
> CWD /
< 250 Directory changed to /
> TYPE A
< 200 Type set to A.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,238)
> LIST
Data connection established with 66.46.177.250:1774 on 9/27/2005 @ 2:24:56 PM
< 150 Opening ASCII mode data connection for /bin/ls.
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
Displaying the contents of \FileSystem.txt, which is a FtpFile.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,240)
> RETR FileSystem.txt
Data connection established with 66.46.177.250:1776 on 9/27/2005 @ 2:24:57 PM
< 150 Opening BINARY mode data connection for FileSystem.txt (1198 Bytes).
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.

Displaying the contents of \appnote.txt, which is a FtpFile.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,6,246)
> RETR appnote.txt
Data connection established with 66.46.177.250:1782 on 9/27/2005 @ 2:24:58 PM
< 150 Opening BINARY mode data connection for appnote.txt (109785 Bytes).
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.

> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:25:04 PM

Each FtpFolder and FtpFile instance shared the same FtpConnection, and since no two operations were done at the same time, a single connection was required, as the log indicates. The FtpConnection object implements the IDisposable interface, since it keeps any active connection available until disposed (or finalized).

Now what happens if I try to open two files at the same time, like this?

using( FtpConnection connection = new FtpConnection( "ftp.xceed.com" ) )

{

  connection.TraceWriter = Console.Out;

  AbstractFile first = new FtpFile( connection, @"\FileSystem.txt" );

  AbstractFile second = new FtpFile( connection, @"\appnote.txt" );

  using( Stream firstStream = first.OpenRead() )

  {

    using( Stream secondStream = second.OpenRead() )

    {

      // In an FTP conversation with an FTP server, only one command

      // at a time can be pending. Here, we clearly have two files

      // open at the same time on the same FTP server. How? Each file

      // has its own connection to the FTP server!

    }

  }

}

The FtpConnection object will create extra command channel connections as required. The output shows two command channel connections were made:

Connected to 66.46.177.250:21 on 9/27/2005 @ 2:38:06 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> CWD /
< 250 Directory changed to /
> TYPE A
< 200 Type set to A.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,53)
> LIST
Data connection established with 66.46.177.250:2613 on 9/27/2005 @ 2:38:07 PM
< 150 Opening ASCII mode data connection for /bin/ls.
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,54)
> RETR FileSystem.txt
Data connection established with 66.46.177.250:2614 on 9/27/2005 @ 2:38:08 PM
< 150 Opening BINARY mode data connection for FileSystem.txt (1198 Bytes).
Connected to 66.46.177.250:21 on 9/27/2005 @ 2:38:08 PM
< 220 Serv-U FTP Server v6.0 for WinSock ready...
> USER anonymous
< 226-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 226 Transfer complete.
< 331 User name okay, please send complete E-mail address as password.
> PASS *****
< 230 User logged in, proceed.
> CWD /
< 250 Directory changed to /
> TYPE I
< 200 Type set to I.
> PASV
< 227 Entering Passive Mode (66,46,177,250,10,55)
> RETR appnote.txt
Data connection established with 66.46.177.250:2615 on 9/27/2005 @ 2:38:09 PM
< 150 Opening BINARY mode data connection for appnote.txt (109785 Bytes).
< 426-Maximum disk quota limited to Unlimited kBytes
<     Used disk quota 0 kBytes, available Unlimited kBytes
< 426 Data connection closed, file transfer appnote.txt aborted.
> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:38:13 PM
> QUIT
Disconnected from 66.46.177.250:21 on 9/27/2005 @ 2:38:13 PM

And the great part about all this is that you don't have to worry about this while coding. You're just manipulating yet another kind of AbstractFile or AbstractFolder.

If we get back to the Zip implementation of the Xceed FileSystem, you can see that a ZippedFile or ZippedFolder (or ZipArchive, the root ZippedFolder) constructor needs to know which AbstractFile is holding the actual zip file that should contain this file or folder. "AbstractFile" truly means "any file", as long as there is an AbstractFile-derived class somewhere to expose this file. It means that zipping directly onto an FTP server is no more difficult than zipping in a regular file on disk.

AbstractFolder source = new DiskFolder( @"d:\Data" );

AbstractFolder localDest = new ZipArchive( 

  new DiskFile( @"d:\temp\local.zip" ) );

AbstractFolder remoteDest = new ZipArchive(

  new FtpFile( new FtpConnection( "localhost" ), @"remote.zip" ) );

// Copying is the same, no matter what is the destination

// file or folder.

source.CopyTo( localDest, true );

source.CopyTo( remoteDest, true );

Code for zipping in "D:\temp\local.zip" is no different than code for zipping in "ftp://localhost/remote.zip". And obviously, reading or unzipping from any zip file is the same.

AbstractFolder localSource = new ZipArchive( 

  new DiskFile( @"d:\temp\local.zip" ) );

AbstractFolder remoteSource = new ZipArchive(

  new FtpFile( new FtpConnection( "localhost" ), @"remote.zip" ) );

AbstractFolder dest = new DiskFolder( @"d:\restored" );

// Unzipping text files from any source is the same!

localSource.CopyFilesTo( dest, true, true, "*.txt" );

remoteSource.CopyFilesTo( dest, true, true, "*.txt" );

I really hope this new addition to the Xceed FileSystem will generate the same enthusiasm we had inventing and developping it. I'm very interested in hearing your opinions!

I just discovered this amasing little gem through Scott. It's called PureText, and what it does is convert the clipboard's formatted text to unformatted text before pasting it to the current window.

By default, it binds to the Win-V key sequence. For example, you select some text on a web page, then press Win-V in a new email, and it will paste the unformatted text, without the web page's formatting. I already can see great use when pasting code from VS.NET! It will save me extra steps.