Wednesday, 21 November 2007

ASP.NET 3.5 has arrived

It has been less than a year, since I started working with ASP.NET, started with 1.1, using VB.NET and gradually getting hold of 2.0 using C#. I havent yet fully understood and implemented a single site in the so called technology, and here it comes!

On November 19, 2007 Microsoft officially released the ASP.NET version 3.5, and ushered the developers into the .NET 3.5 era. Enhancements have been released across the platform, which would now mean I have to be quick enough to embark on the technology or be left out.

As many of us would expect the changes or features in the new version are additive. Nothing has been taken off from the 2.0 version, and nothing has been drastically modified. Like people coming from 1.1 to 2.0, it was a complete shock. Quite a few things have been taken off, many of things modified, and very little if not, new things added. But this time, it seems Microsoft has been considerate with the developers. In short, 3.5 does'nt take away or change anythingor even break any functionality, concepts, or code present in 2.0 - it just adds new types and features and capabilities to the framework.

As you could have guessed it Visual Studio 2008 the Microsoft IDE(Integrated Developement Environment) is the recommended tool for developing the applications. Unlike the previous flavours of Visual Studio, this one can be used to develop applications using .net 1.1, 2.0, 3.0 as well as 3.5. So if you are not thinking of developing any new system, or upgrading your code at this moment of time, it would be wise decision to purchase Visual studio 2008, as it would support your older version of codes as well. It also includes some extra features such as improved Designer experience, JavaScript Debugging (Finally Microsoft got it in their heads!), and intellisense features; and the developers have the ability to view and even step into the core .NET framework code during debugging.

So in short, it would be really wonderful coding and even better experience in debugging the applications using the new system. I have got hold of a beta version of the Visual studio 2008. Will be playing with it in some times from now, and will keep you all posted.

Powered by ScribeFire.

The Internal Structure of .pdb Files

Whenever you are faced with an unknown data format, the first thing to do is to run some instances of it through a hex dump viewer. The w2k_dump.exe utility does a good job in this respect. Examining the hex dump of a Windows 2000 PDB file like ntoskrnl.pdb or ntfs.pdb reveals some interesting properties:
  • The file seems to be divided into blocks of fixed size—typically 0[ts]400 bytes.

  • Some blocks consist of long runs of 1-bits, occasionally interrupted by shorter sequences of 0-bits.

  • The information in the file is not necessarily contiguous. Sometimes, the data ends abruptly at a block boundary, but continues somewhere else in the file.

  • Some data blocks appear repeatedly within the file.

It took me some time until I finally realized that these are typical properties of a compound file. A compound file is a small file system packaged into a single file. The "file system" metaphor readily explains some of the above observations:

  • A file system subdivides a disk into sectors of fixed size, and groups the sectors into files of variable size. The sectors representing a file can be located anywhere on the disk and don't need to be contiguous—the file/sector assignments are defined in a file directory.

  • A compound file subdivides a raw disk file into pages of fixed size, and groups the pages into streams of variable size. The pages representing a file can be located anywhere in the raw disk file and don't need to be contiguous—the stream/page assignments are defined in a stream directory.

Obviously, almost any assertions about file systems can be mapped to compound files by simply replacing "sector" by "page", and "file" by "stream". The file system metaphor explains why a PDB file is organized in fixed-size blocks. It also explains why the blocks are not necessarily contiguous. What about the pages with the masses of 1-bits? Actually, this type of data is something very common in file systems. To keep track of used and unused sectors on the disk, many file systems maintain an allocation bit array that provides one bit for each sector (or sector cluster). If a sector is unused, its bit is set. Whenever the file system allocates space for a file, it searches for unused sectors by scanning the allocation bits. After adding a sector to a file, its allocation bit is set to zero. The same procedure is applied to the pages and streams of a compound file. The long runs of 1-bits represent unused pages, while the 0-bits are assigned to existing streams.

The only thing that is left now is the observation that some data blocks reoccur within a PDB file. The same thing happens with sectors on a disk. When a file in a file system is rewritten a couple of times, each write operation might use different sectors to store the data. Thus, it can happen that the disk contains free sectors with older duplicates of the file information. This doesn't constitute a problem for the file system. If the sector is marked free in the allocation bit array, it is unimportant what data it contains. As soon as the sector is reclaimed for another file, the data will be overwritten, anyway. Applying the file system metaphor once more to compound files, this means that the observed duplicate pages are usually left over from earlier versions of a stream that has been rewritten to different pages in the compound file. They can be safely ignored—all we have to care for are the pages that are referred to by the stream directory. The remaining unassigned pages should be regarded as garbage.

With the basic paradigm of PDB files being introduced now, we can step to the more interesting task of examining their basic building blocks. Listing 1 shows the layout of the PDB header. The PDB_HEADER starts with a lengthy signature that specifies the PDB version as a text string. The text is terminated with an end-of-file (EOF) character (ASCII code 0[ts]1A) and supplemented with the magic number 0[ts]0000474A, or "JG\0\0" if interpreted as a string. Maybe these are the initials of the designer of the PDB format. The embedded EOF character has the nice effect that an ignorant user can issue a command like type ntoskrnl.pdb in a console window without getting any garbage on the screen. The only thing that will be displayed is the message "Microsoft C/C++ program database 2.00\r\n". All Windows 2000 symbol files are shipped as PDB 2.00 files. Apparently, a PDB 1.00 format exists as well, but it seems to be structured quite differently.

Listing 1 The PDB File Header

#define PDB_SIGNATURE_200 \
"Microsoft C/C++ program database 2.00\r\n\x1AJG\0"


// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

typedef struct _PDB_SIGNATURE


// -----------------------------------------------------------------

#define PDB_STREAM_FREE -1

// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

typedef struct _PDB_STREAM
DWORD dStreamSize; // in bytes, -1 = free stream
PWORD pwStreamPages; // array of page numbers

#define PDB_STREAM_ sizeof (PDB_STREAM)

// -----------------------------------------------------------------

#define PDB_PAGE_SIZE_1K 0x0400 // bytes per page
#define PDB_PAGE_SIZE_2K 0x0800
#define PDB_PAGE_SIZE_4K 0x1000

#define PDB_PAGE_SHIFT_1K 10 // log2 (PDB_PAGE_SIZE_*)
#define PDB_PAGE_SHIFT_2K 11
#define PDB_PAGE_SHIFT_4K 12

#define PDB_PAGE_COUNT_1K 0xFFFF // page number < PDB_PAGE_COUNT_*
#define PDB_PAGE_COUNT_4K 0x7FFF

// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

typedef struct _PDB_HEADER
DWORD dPageSize; // 0x0400, 0x0800, 0x1000
WORD wStartPage; // 0x0009, 0x0005, 0x0002
WORD wFilePages; // file size / dPageSize
PDB_STREAM RootStream; // stream directory
WORD awRootPages []; // pages containing PDB_ROOT

#define PDB_HEADER_ sizeof (PDB_HEADER)

Following the signature at offset 0[ts]2C is a DWORD named dPageSize that specifies the size of the compound file pages in bytes. Legal values are 0[ts]0400 (1KB), 0[ts]0800 (2KB), and 0[ts]1000 (4KB). The wFilePages member reflects the total number of pages used by the PDB file image. Multiplying this value by the page size should always exactly match the file size in bytes. wStartPage is a zero-based page number that points to the first data page. The byte offset of this page can be computed by multiplying the page number by the page size. Typical values are 9 for 1KB pages (byte offset 0[ts]2400), 5 for 2KB pages (byte offset 0[ts]2800), or 2 for 4KB pages (byte offset 0[ts]2000). The pages between the PDB_HEADER and the first data page are reserved for the allocation bit array of the compound file, always starting at the beginning of the second page. This means that the PDB file maintains 0[ts]2000 bytes with 0[ts]10000 allocation bits if the page size is 1 or 2KB, and 0[ts]1000 bytes with 0[ts]8000 allocation bits if the page size is 4KB. In turn, this implies that the maximum amount of data a PDB file can manage is 64MB in 1KB page mode, and 128MB in 2KB or 4KB page mode.

The RootStream and awRootPages[] members concluding the PDB_HEADER describe the location of the stream directory within the PDB file. As already noted, the PDB file is conceptually a collection of variable-length streams that carry the actual data. The locations and compositions of the streams are managed in a single stream directory. Funny as it might seem, the stream directory itself is stored in a stream. I have called this very special stream the "root stream". The root stream holding the stream directory can be located anywhere in the PDB file. Its location and size are supplied by the RootStream and awRootPages[] members of the PDB_HEADER. The dStreamSize member of the PDB_STREAM substructure specifies the number of pages occupied by the stream directory, and the entries in the awRootPages[] array point to the pages containing the data.

The stream directory is composed of two sections: A header part in the form of a PDB_ROOT structure, as defined in Listing 2, and a data part consisting of an array of 16-bit page numbers. The wCount member of the PDB_ROOT section specifies the number of streams stored in the PDB compound file. The aStreams[] array contains a PDB_STREAM entry (see Listing 1) for each stream, and the page number slots follow immediately after the last aStreams[] entry.

Listing 2 The PDB Stream Directory

#define PDB_STREAM_PDB 1

// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

typedef struct _PDB_ROOT
WORD wReserved; // 0
PDB_STREAM aStreams []; // stream #0 reserved for stream table

#define PDB_ROOT_ sizeof (PDB_ROOT)

Finding the page number block associated to a given stream is somewhat tricky because the page directory doesn't provide any cues except the stream size. If you are interested in stream #3, you have to compute the number of pages occupied by streams #1 and #2 to get the desired start index within the page number array. Once the stream's page number list is located, reading the stream data is simple. Just walk through the list and multiply each page number by the page size to yield the file offset, and read pages from the computed offsets until the end of the stream is reached. Isn't it funny: On first sight, parsing a PDB file seemed rather tough. Now it turns out that it is actually quite simple—probably much simpler than parsing a .dbg file. The compound-file nature of the PDB format with its clear-cut random access to stream pages reduces the task of reading a stream to a mere concatenation of fixed-sized pages. I'm really amazed at this elegant data access mechanism!

An even greater benefit of the PDB format becomes apparent if it comes to updating an existing PDB file. Inserting data into a file with a sequential structure usually means reshuffling large portions of the contents. The PDB file's random-access structure borrowed from file systems allows addition and deletion of data with minimal effort, just like files can be modified with ease on a file system media. Only the stream directory has to be reshuffled at times when a stream grows or shrinks across a page boundary. This important property facilitates incremental updating of PDB files. As Microsoft puts it in a Knowledge Base article titled "INFO: PDB and DBG Files—What They Are and How They Work":

"The .PDB extension stands for 'program database.' It holds the new format for storing debugging information that was introduced in Visual C++ version 1.0. In the future, the .PDB file will also hold other project state information. One of the most important motivations for the change in format was to allow incremental linking of debug versions of programs, a change first introduced in Visual C++ version 2.0." (Microsoft Knowledge Base, article Q121366)

Now that the internal format of PDB files is clear, the next problem is to identify the contents of their streams. After examining various PDB files, I have come to the conclusion that each stream number serves a predefined purpose. For example, the first stream seems to always contain a stream directory, and the second one contains information about the PDB file that can be used to verify that the file matches an associated .dbg file. For example, the latter stream contains dSignature and dAge members that should have the same values as the corresponding members of an NB10 CodeView section. The eighth stream is most interesting in the context of this chapter because it hosts the CodeView symbol information we have been searching for. The meaning of the other streams is still unclear to me and constitutes another vast area for future research.

I am not going to include PDB reader sample code here because this would exceed the scope of this article without being rather interesting. You already know this program—it is the w2k_dump.exe utility that I have used to create some of the hex dump examples above. This simple console-mode utility provides a +p command line option that enables PDB stream decomposition. If the specified file is not a valid PDB file, the program falls back to sequential hex dump mode.

Powered by ScribeFire.

Tuesday, 13 November 2007

Useful Tool for browsing class definitions

Reflector is a class browser for .NET components. It is difficult if not impossible, to scan the Microsoft Documentation for the class definition, when you want to use a particular, function, but do not know about its parent class. Ofcourse, google is the ultimate key, but sometimes, even google screws up, by providing multiple options for the same results.

Having a class browser in such cases is very handy. Guess what, it comes from the person directly responsible for developing Microsoft projects, so can be trusted for atleast the basic definitions. Okay jokes apart, the following is the link to the page where you can download the product.
Lutz Roeder
And while you are at the page, do also browse through the plugins and add-ons for the reflector, even they are a cool utility!

Powered by ScribeFire.

Monday, 12 November 2007

C# ?? Operator

The ?? operator also termed as null Coalescing operator is one of the latest cool stuff in the .net 2.0. This operator returns the left hand operand if it is not null, or else it returns the right operand.

So using the statemtent in ways like

int y = x ?? -1;

so if the value of integer x is not null, then y is assigned the value of x, or in case if x is null, then y is assigned the value of -1. The point worth noticing is the return value if x is null, is -1, so the system knows that the assignement has not been successful.

I am not aware of a mission critical requirement for this operator, but there have been various examples, wherein the non-nullable data field, has been received as null. For eg. if you are displaying a table with user age, it might be possible, that the user has not entered the date, or date of birth. It might be useful to use this operator to display a message in such cases, instead of displaying a "0" or a random number in the report field.

Powered by ScribeFire.

Monday, 5 November 2007

Validating Email IDs in a user form

Validation is a key step in User registration. Though not intentional, many users tend to enter correct email address, in a form. Also many users just tend to enter a fictitious email to avoid spams from a public domain forum. But that is again a nightmare for webmasters, as they tend to collect a huge pile of unuseful data for users.
To avoid this, many websites, do send out a test email, which has a random link, which the user has to click, in order to activate their accounts. But then, not all public domain email servers are fast enough to send or receive the validation email. This might result into a lost visitor for the website.
So what should be done to optimise the process. I came across, a three step email validation process today, which does seem to promise a better validation. Though not completely tested by me, but still would recommend atleast a visit and try if you need.

<a href="" target="_blank">Click here</a> to visit Vasudevan's blog for further details.

Powered by ScribeFire.