cfix 1.5.1 released

A new version of cfix, the unit testing framework for C and C++ on Windows, is now available on Sourceforge. Despite fixing several minor issues, the new version resolves the following two issues that were reported by users:

• Definiting multiple WinUnit fixtures with setup/teardown routines in a single .cpp file leads to a compilation error
• A thread handle is leaked during execution of a test (#2889511)

Updated binaries and source code are available for download on Sourceforge.

Btw, in case you use cfix for kernel mode testing and are using WDK 7600, please have a look at my previous post: LTCG issues with the WIN7/amd64 environment of WDK 7600

LTCG issues with the WIN7/amd64 environment of WDK 7600

Now that Windows 7 is out, we all sooner or later have to upgrade to WDK 7600. I am still reluctant to move away from WDK 6000/6001 because of the dropped W2K support, but this is a different issue.

However, as one cfix user who has obviously already adopted WDK 7600 kindly pointed out to me, linking a kernel mode unit test against cfix using WDK 7600 and the WIN7/amd64 environment fails reproducibly with the following error message:

error fatal error C1047: The object or library file ‘…\lib\amd64\cfixkdrv.lib’ was created with an older compiler than other objects; rebuild old objects and libraries

In contrast, building the same driver for WIN7/x86 works fine.

As the documentation for C1047 indicates, this error is usually related to inconsistent usage of Link Time Code Generation (LTCG): As soon as you use LTCG, all objects and libraries must be compiled with /GL — this normally is not a big deal, but as this WDK page rightfully explains, it means that libraries built this way are not suitable for redistribution because of their dependency on a specific compiler/linker version. But of couse, it also means that a library not built using /GL cannot be used easily when you build your program using LTCG.

Before Windows 7, all WDK build environment configurations I am aware of did not use LTCG. Neither did cfix, so everything worked fine even if your compiler/linker versions did not match the ones used for building cfix.

With WDK 7600, this situation changes: While WLH and other downlevel build environments still do not seem to use LTCG, the WIN7 environment, at least for amd64, enables LTCG by default.

What this means is that as soon as you link against a library which is not part of the WDK and therefore likely to be built using a different compiler version, you’ll get C1047. It is thus no surprise that attempting to link against cfixkdrv.lib, which is the library all kernel mode unit tests have to link against and which itself has been built using WDK 6000, leads to the error quoted above.

However, once you have figured this out, the workaround for this issue is trivial: Disable LTCG for your test driver by adding the following line to your SOURCES file:


USER_C_FLAGS=/GL-


Link time code generation is a very powerful optimization technique and I encourage everybody to make use of it if possible. However, for the compatibility reasons outlined above, I consider it a rather stupid idea to have the WDK enable LTCG by default. Rather, I had much preferred to see an opt-in switch for LTCG.

Anyway, for the next version, I will consider shipping both, a 7600-compatible LTCG-enabled and a non-LTCG-enabled version of cfixkdrv.lib. Until then, the workaround described above will do the trick.

Launched ntrace.org

Having given my presentation on NTrace today at the WCRE in Lille/France, I have also opened ntrace.org to the public. NTrace, in case you have missed my previous posts, is a dynamic function boundary tracing system for Windows/x86 I initially developed as part of my Master’s thesis that is capable of performing DTrace-like tracing of both user and kernel mode components.

On the NTrace page, you will now find the paper itself as being published as part of the WCRE proceedings (mind the copyright notice, please) along with two screencasts: One showing how NTrace can be used to trace kernel mode components such as NTFS, and one demonstrating NTrace for user mode tracing.

If you have questions about NTrace or are interested in more details, please feel free to write me an email — my address is jpassing at acm org.

Mixing 32 and 64-bit components in a single MSI

Definetely one my pet peeves about Windows Installer is how it deals with instruction set architectures (ISAs). Looking at Windows NT history, supported ISAs have come (amd64, IA-64) and gone (Alpha, PowerPC, MIPS) — yet most of the time, there was more than one ISA being officially supported. Having to ship binaries for multiple ISAs therefore always has been on the agenda for many ISVs.

Needless to say, supporting multiple ISAs requires special consideration when developing setup packages and providing separate packages — one for each ISA — has become common practice to approach this. This approach makes perfect sense: Given the incompatibility of most ISAs, nobody needs Alpha binaries on a MIPS system or amd64 binaries on a IA-64 machine, so there seems little reason to mix ISAs within a single package.

Unsurprisingly, Windows Installer, which was created somewhere around 2000, also goes this route and encourages developers to provide separate packages for each ISA.

However, with the advent of amd64/x64/IA-32e/Intel 64/whateveryoucallit, the situation has changed: Because i386 and amd64 are so closely related and compatible, there are now plenty of situations where combining binaries of differing ISAs (i.e. amd64 and i386) in a single installer package makes perfect sense. Examples for this include:

• A package comprises a shell extension as well as a standalone App. For certain reasons (maybe the use of VB6), there only is a 32 bit version of the App. The shell extension, in contrast, is available for both, i386 and amd64. Whether you put everything into one package or provide separate packages for each ISA, one of them will comprise a mixture of ISAs.
• SDKs for unmanaged code usually include .lib and .dll files for multiple architectures. Shipping separate packages for i386 and amd64 (containing different binaries but the same headers, docs, etc.) may please the Windows Installer gods, but seems redundant, a waste of disk space, and user-unfriendly.

Thanks to the msidbComponentAttributes64bit flag, mixing architectures in a single MSI package is technically possible: You mark the package as being 32 bit and set said flag for all 64-bit components. Rather than splitting your setup into multiple packages, you can conveniently combine everything into one.

When reading the documentation (and ICE requirements, more on this later) carefully though, it turns out that this is not quite what the Windows Installer team invented this flag for. Anyway, it works fine, problem solved.

Hmmm…

If only there was not ICE80.

ICE80, alas, is critical if you intend to conform to the Requirements for the Windows Vista Logo Program for Software:

Applications must use the Windows Installer (MSI) or ClickOnce for installation. Windows Installation packages must not receive any errors from the Internal Consistency Evaluators (ICEs) listed here:

1-24, 27-31, 33-36, 38, 40-57, 59, 61-63, 65, 67-72, 74-84, 86-87, 89-94, 96-99

ICE80 mainly states that (1) you should not install 64 bit components to 32 bit directories (e.g. Program Files vs. Program Files (x86)) and (2) you should not use 64 bit components in a 32 bit package.

(1) is fair enough, although it raises the question where you should install your software to without splitting it in two or violating other ICE rules. Worse yet, (2) effectively means that said way to create multi-ISA packages, creating 32 bit packages with some components marked with msidbComponentAttributes64bit, is illegal alltogether.

So to be logo’ed, there seems to be no other way than providing separate packages, maybe along with (urgh!) a meta-package that installs the other two.

If there are more important things on your schedule than getting a Vista logo, ICE80 seems like something that can safely be ignored. Indeed, this is what I have done several times, including in case of the cfix installer.

Anyway, let’s ignore ICE80 once more and hold on to the plan of building a 32-bit package containing both, 32-bit and 64-bit components.

Urgh…

For an SDK that is installed on 64-bit Windows, it will usually make sense to install both, 32 and 64 bit .lib and .dll files etc. On 32-bit Windows, installing 64-bit components may seem odd, but due to the existence of amd64 compilers for i386, it still makes sense to install them or at least offer them as optional feature.

So far, so good. Things get interesting, though, when COM registration comes into play. Naturally, a 32 bit installer package sees the system like any other 32 bit application does. Most importantly, this means that Registry Reflection and File System Redirection applies.

Now consider a package that contains both a 32-bit and a 64-bit version of some COM server, each installed to a separate directory. COM Registration either be performed through the Class or the Registry table. Provided that the msidbComponentAttributes64bit flag has been used properly, such a package will work great on 64 bit systems thanks to Registry Reflection: The regsitry entries will be written to the proper (reflected) locations and both COM servers will work properly.

Now think what happens on 32-bit Windows: (1) There is no Registry Reflection and (2) Windows Installer silently ignores msidbComponentAttributes64bit flags. Result: The installation will run just as smooth as on the 64-bit system. However, while installing the files continues to works flawlessly, the registry will be left in a less-than-optimal state: Due to the nonexistence of Registry Reflection, the registration entries of both COM servers will have been written to the same location!

Needless to say, the server whose registration entries were written first will now be unusable.

In a way, Windows Installer has taken its revenge for breaking the rules.

Bottom line: Mixing 32 and 64-bit components in a single MSI works fine in many cases, but is against the MSI rules and can lead to further problems. And while I am still convinced that providing separate, ISA-specific packages is wrong or at least inconvenient in certain situations, it is definitely the safer and “right” way to go.

(Note: Windows Installer 4.5 introduced multi-package transactions, which allow reliable and transactional multi-package setups to be built so that splitting a setup into multiple packages can be implemented without much pain. However, very few users already have Windows Installer 4.5 installed and Windows 2000 is not even supported by this release. For many of us, relying on this feature therefore is not really an option.)

I’ll be at WCRE 2009 presenting NTrace

Next week, the 16th Working Conference on Reverse Engineering (WCRE) will be held in Lille, France. I will be there presenting NTrace: Function Boundary Tracing for Windows on IA-32.

NTrace is a dynamic function boundary tracing toolkit for IA-32/x86 that can be used to trace both kernel and user mode Windows components — examples for components that can be traced include the kernel itself (ntoskrnl), drivers like NTFS as well as user mode components such as kernel32, shell32 or even explorer.exe.

NTrace implements a novel approach to instrumenting IA-32 machine code and integrating with the Structured Exception Handling facility of Windows. Using this approach, NTrace is not only capable of tracing nearly the entire Windows kernel and system libraries, it is also faster than Solaris DTrace FBT on IA-32!

Details on how exactly NTrace works will be publiched in the paper, which will be made available soon. I will also publish more details on NTrace both here and on a dedicated NTrace website.

The work, by the way, is basically the result of my Master’s thesis I wrote back in 2008.

Visual Assert Beta 2 Released

The Beta 2 release of Visual Assert (formerly named cfix studio) is now available for download. The release marks a major step in the development of Visual Assert for that it not only comprises a number of bugfixes but also introduces major new features. The two most important certainly are support for EXE targets and Wizard assistance.

Support for EXE Targets

As announced in a previous post and also discussed in the post about the cfix 1.5 release, Visual Assert now fully supports unit tets emedded in EXE modules.

Previous releases required all unit tests to be compiled and linked into DLLs. In fact, the usage of DLLs has many advantages and therefore remains the recommended practice. However, for certain projects, such a requirement can turn out to be a true obstacle: Whenever the code to be tested is not exported from a DLL or part of a static library (LIB), accessing this code from within such a test DLL could become quite a challenge.

The fact that Visual Assert Beta 2 now fully supports EXE modules means the following: You can now place your unit tests wherever you think they fit best. Whether they are part of a DLL or an EXE, whether you create separate “unit test” projects or intermingle your test code with other code — it is now all up to you. Wherever you placed your tests, Visual Assert will find them and will provide a consistent user experience.

And the best part of the support for EXE modules is that it is totally non-intrusive: You do not have to change your main/WinMain function, much less any other code or build settings. And when run “outside” Visual Assert, i.e. launched directly or in the Visual Studio Debugger, the application will behave as normal.

Needless to say, the cfix 1.5 command line test runners, cfix32.exe and cfix64.exe also have been updated to properly deal with EXE modules.

The Wizard

Although neither the WinUnit API nor the cfix C and C++ API require much boilerplate code to be written, there still is some amount of code that more or less all unit tests share. Thanks to the new Wizard, you can now have Visual Assert generate this code for you. This really helps creating new fixtures more quickly!

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you are a full time tester or a unit testing sceptic, download the new release and try it by yourself. And of course, your feedback, both positive and negative, is always welcome and can be posted here.

Download Visual Assert Beta 2

cfix 1.5 released; adds support for EXE-embedded tests and kernel mode multi-threading

A new version of the cfix unit testing framework is now ready for download.

Unlike the previous release, which was mainly a maintenance release, cfix 1.5 adds major new features: kernel mode multithreading and EXE-based unit tests.

EXE based unit tests

As I discussed in a previous post in the context of Visual Assert/cfix studio, cfix’ restriction to DLL based unit tests has turned out to be quite a limitation for certain kinds of projects.

In cfix 1.5, this restriction has been removed: cfix now supports both, unit tests compiled and linked into DLL modules and unit tests embedded into EXE modules.

Now, when I say embeded into an EXE module, I do not mean that you may merely leave out the /LD compiler switch — it means that you may compile and link unit tests into the actual application EXE module without impacting this application’s behavior or having to change the application’s main() routine: When you run the application direcly or through the debugger, it will behave as normal (e.g. launch GUI). Once you launch it via cfix32 -exe app.exe (or cfix64 -exe app.exe, respectively), however, the application’s main() routine will not execute and instead, your embedded unit tests will run.

This may seem awkward at first — but it offers a tremendous advantage: All of the application’s code now immediately becomes testable (i.e. accessible by unit test code) without having to do complex reorganization of your build process or source tree layout (although it is still a very good idea to enhance the build process s.t. the unit tests are stripped in the RTM builds and to make cfix.dll a delayload DLL). For many projects, relying on this feature will therefore make adopting cfix and maybe even unit testing in general much easier.

In almost all regards, EXE-embedded unit tests behave the same as their DLL counterparts. They are, however, slightly less handy when it comes to debugging (unless you use Visual Assert, which will shield you from this). For this reason, and their greater flexibility in general, it should be noted though that DLL-based unit tests will remain the preferred choice.

Kernel mode multithreading

Since its first release, cfix has featured support for multi-threaded test cases. Multi-threaded test cases are tests which spawn child threads and — and this is the important point — both, the “main” thread and the child threads may trigger assertions (i.e. use CFIX_ASSERT and friends). Regardless of the thread an assertion occurs on, it will be recognized by the framework and will lead to the test case being marked as having failed. In case this sounds trivial to you, be informed that not even JUnit properly supports this :)

And while this feature has been supported for user mode tests ever since, the infrastructure for kernel mode unit tests, which was added in version 1.1, has lacked support for this feature: There was no kernel mode counterpart of CfixCreateThread and thus, only single-threaded kernel mode tests were supported.

cfix 1.5 now finally introduces CfixCreateSystemThread: CfixCreateSystemThread is basically a wrapper for PsCreateSystemThread with the added functionality of registering the child thread with cfix. Thus, all threads spawned using CfixCreateSystemThread (rather than using PsCreateSystemThread directly) are allowed to make use of assertions.

Kudos to Matt Oh for beta-testing this feature.

Minor enhacements

Another enhancement related to multi threaded tests is Auto-joining of child threads: After a test routine completes, the framework will now automatically check whether any child threads were created. If this is the case and any of these threads are still running, they will be waited on — not before all threads have terminated will the test run resume. This feature both makes writing multi-threaded tests more convenient (you do not have to wait by yourself) and safer (No more runaway child threads).

Finally, another feature worth noting is that by specifying the -td command line switch, cfix can be directed to not capture a stack backtrace when an assertion fails. Although stack backtraces are usually very helpful, the symbol loading can make their creation quite expensive. Using this switch can therefore speed up the usage of cfix.

Download/Upgrade Now

You see, there are many good reasons to upgrade your cfix installation or — in case you are not using cfix yet — to give cfix a try.

Better yet, check out the Visual Assert AddIn for Visual C++ — it is based on the new cfix 1.5 release and seamlessly integrates with Visual Studio.

Download cfix 1.5.0 Installer
Download cfix 1.5.0 source code

cfix studio renamed to Visual Assert

Back when I began thinking about creating a Visual Studio Add-In for cfix, I needed a code name for the project. After tentatively using the name cfix+ for a while, I quickly settled on cfix studio — given that the project revolved around cfix and Visual Studio, this name pretty much suggested itself.

Soon after going into Beta, however, I had to realize that this name was not without its problems. Most importantly, it makes it hard for users to properly differentiate between cfix and cfix studio. This obviously led to situations where people were not quite sure whether cfix studio is a supplement to, replacement of, or just new version of cfix.

I would not care too much about this ambiguity if the two projects were not very different in terms of licensing: While using the Add-In will require a license to be purchased once it leaves the beta status, the underlying cfix framework is, and will always remain open source and be licensed under the quite permissive LGPL.

As all APIs and libraries unit tests link against are part of the cfix framework, and the cfix framework itself is self-contained, this means that despite the Add-In being closed source, you still get the benefits of open source: Most importantly, there is no lock-in — you can stop using the Add-In and switch back to the command line tools at any time and be all-open source again. You are even free to create a fork of cfix at any time — there really is nothing other than convenience that binds you to using the Add-In.

Given that a growing number of people indeed tends to object to using closed source APIs and fears such vendor lock-in, I consider it important to stress this open source nature of the cfix framework underlying the Add-In.

However, at this point it should also become clear that a name that blurs the distinction between the two projects is counterproductive.

Based on this insight, I opted for dropping the cfix studio name and replacing it by something different: Visual Assert. This name should emphasise that the Add-In may be based on, but really is separate from the cfix framework.

The new name will be used beginning with the upcoming Beta 2 release.

Writing Data-Driven Custom Actions

Whenever Windows Installer’s built-in actions do not suffice to perform a specific task, a Custom Action needs to be written. Needless to say, Custom Actions, can be a bit tricky — not only can they be laborious to write and cumbersome to debug, they also run the risk of interfering with Windows Installer’s declarative, transactional way of performing installs.

It is not really surprising that Windows Installer therefore more or less discourages the use of Custom Actions unless it is absolutely necessary. Moreover, as a result of its declarative nature, it is understadable that Windows Installer prefers Custom Actions to be data-driven.

What this means in practice is that a Custom Action should not perform a hard-coded task — rather, it should query one or more (custom) tables containing the necessary information (in a declarative manner) about what is to be performed and should act accordingly.

Using WiX, creating custom tables turns out to be pretty easy. Let’s assume we create a Custom Action that, based on some condition, does something with a specific file. An appropriate table could look like this:

<CustomTable Id="MyCustomTable">
  <Column Id="Id" Type="string" PrimaryKey="yes"/>
  <Column Id="Path" Type="string"/>
  <Column Id="Condition" Type="string"/>

  <Row>
    <Data Column="Id">Foo</Data>
    <Data Column="Path">[INSTALLLOCATION]foo.txt</Data>
    <Data Column="Condition"><![CDATA[ &FeatureFoo=3 ]]></Data>
  </Row>
  <Row>
    <Data Column="Id">Bar</Data>
    <Data Column="Path">[INSTALLLOCATION]bar.txt</Data>
    <Data Column="Condition"><![CDATA[ &FeatureBar=3 ]]></Data>
  </Row>
</CustomTable>

To query this table, we have to open a view and fetch the records one by one:

PMSIHANDLE Database = MsiGetActiveDatabase( InstallHandle );
ASSERT( Database != NULL );

PMSIHANDLE View;
UINT Result = MsiDatabaseOpenView(
  Database,
  L"SELECT `Condition`, `Path`, FROM `MyCustomTable`",
  &View );
if ( ERROR_SUCCESS != Result )
{
  ...
}

Result = MsiViewExecute( View, NULL );
if ( ERROR_SUCCESS != Result )
{
  ...
}

for ( ;; )
{
  PMSIHANDLE Record;
  Result = MsiViewFetch( View, &Record );
  if ( Result == ERROR_NO_MORE_ITEMS  )
  {
    break;
  }
  else if ( ERROR_SUCCESS != Result )
  {
    ...
  }

  //
  // Read condition.
  //
  // N.B. Do not format -- this is done by
  // MsiEvaluateCondition itself.
  //

  WCHAR Condition[ 256 ];
  DWORD Length = _countof( Condition );
  Result = MsiRecordGetString(
    Record,
    1,
    Condition,
    &Length );
  if ( ERROR_SUCCESS != Result )
  {
    ...
  }

  if ( MSICONDITION_TRUE != MsiEvaluateCondition(
    InstallHandle,
    Condition ) )
  {
    //
    // This record can be skipped.
    //
    continue;
  }

  //
  // Read remaing fields.
  //

  WCHAR Path[ MAX_PATH ];
  Length = _countof( VszPath );
  Result = GetFormattedRecord(
    InstallHandle,
    Record,
    2,
    Path,
    &Length );
  if ( ERROR_SUCCESS != Result )
  {
    ...
  }

  ...
}

With GetFormattedRecord being the following utility routine:

static UINT GetFormattedRecord(
  __in MSIHANDLE InstallHandle,
  __in MSIHANDLE Record,
  __in UINT Field,
  __out PWSTR Value,
  __inout PDWORD Length
  )
{
  DWORD RecLength = *Length;
  UINT Result = MsiRecordGetString(
    Record,
    Field,
    Value,
    &RecLength );
  if ( ERROR_SUCCESS != Result )
  {
    *Length = RecLength;
    return Result;
  }

  PMSIHANDLE FormattingRecord = MsiCreateRecord( 1 );

  Result = MsiRecordSetString( FormattingRecord, 0, Value );
  if ( ERROR_SUCCESS != Result )
  {
    return Result;
  }

  return MsiFormatRecord(
    InstallHandle,
    FormattingRecord,
    Value,
    Length );
}

Some things are worth noting:

• I use PMSIHANDLE, which, as you probably already know, is not a typedef for MSIHANDLE* but rather a smart-pointer like class that automatically closes the handle when it goes out of scope.
• The use of backticks in the query.
• It must have been a Visual Basic programmer implementing MsiRecordGetString: Field Indexes start with 1, not 0. To make matters worse, reading from index 0 does not fail but returns arbitrary data. Finally, to confuse people further, indexes are 0-based for MsiRecordSetString.
• If the field contains formatted data, you have to MsiFormatRecord it yourself. For conditions, however, MsiEvaluateCondition handles that for you.

So far, so good. There is, however, one thing to notice: To access the installer database, the custom action must be a nondeferred action:

You cannot access the current installer session or all property data from a deferred execution custom action

The problem with nondeferred actions, however, is that they execute in user context — in contrast to deferred actions, which execute in system context. On pre-Vista platforms, a per-machine installer package can be expected to always be launched by an administrator (otherwise it will fail anyway) — in this case, the differences between user and system context may not be important — both, for example, have r/w access to files in %ProgramFiles%. On Vista and later OS, however, it is common to have a regular user launch an installation which causes an elevation prompt once it reaches the main install phase. In this case, the user context is significantly less privileged than system context.

For a hypothetical custom action that is intended to edit a file installed to %ProgramFiles%, this means that (disregarding rollback considerations and assuming proper scheduling) performing this action from within the nondeferred custom action will work fine on pre-Vista OS. When run on Vista, though, it is likely to fail due to lack of write access to %ProgramFiles%. In practice, this means that all system-changing tasks usually have to be performed by a deferred action.

To sum up: To be data-driven, you have to use nondeferred actions. To be able to perform any serious, system state-changing tasks, however, you have to use deferred actions.

Great.

As it turns out, however, there is a way to escape this catch-22, and it is carefully buried in the Windows Installer documentation:

[...] Actions that update the system, such as the InstallFiles and WriteRegistryValues actions, cannot be run by calling MsiDoAction. The exception to this rule is if MsiDoAction is called from a custom action that is scheduled in the InstallExecuteSequence table between the InstallInitialize and InstallFinalize actions. [...]

[From the Remarks section of MsiDoAction]

In fact, the way I came across this solution was by looking at the source code of the WiX XmlFile action, which I knew manages to both be data-driven (uses a custom table) and alter system state (edits XML files). The way it does this, and the point where the above remark comes into play, is as follows: In the nondeferred action, you do not perform any actions changing system state. Rather, you collect the information from the installer tables and stuff it (yuck) into the CustomActionData property. Then, leveraging MsiDoAction and passing said CustomActionData, you schedule another custom action — this time a deferred one — which parses the CustomActioData (yuck) and, based on this data, finally performs the actual modifications — in system context.

It really could not be easier and more intuitive, right?

Overview on Designing High-Performance Windows Applications

Back in 2008, the Windows Server Performance Team Blog, which I came across recently, ran a series of posts on Designing Applications for High Performance:

• Designing Applications for High Performance – Part I
• Designing Applications for High Performance – Part II
• Designing Applications for High Performance – Part III

If you are interested in developing server side applications for Windows, these articles are definitely worth reading.