One of the neat things you can do in BizTalk is to create a map that takes one kind of XML message and transforms it into another. They basically are a wrapper over XSLT and its extension infrastructure. The tool you use to create maps is the BizTalk mapper. In this tool you load in a source and destination schema - which get displayed in treeviews and then you drag elements from the source to the destination to describe how to build the destination message.

Now, in WF - especially when using Silver (the WF/WCF integration layer introduced in .NET 3.5) - you tend to spend alot of time taking data from one object and putting it into another as different objects are used to serialize and deserialize messages received and sent via WCF. This sounds alot like what the BizTalk Mapper does, but with objects rather than XML messages.

I was recently teaching Guerrilla Connected Systems (which has now been renamed Guerrilla Enterprise.NET) and Christian Weyer was doing a guest talk on REST and BizTalk Services. Afterwards, Christian and me were bemoaning the amount of mundane code you get forced to write in Silver and how sad it was that WF doesn't have a mapper. So we decided to build one. Unfortunately neither myself nor Christian are particularly talented when it comes to UI - but we knew someone who was: Jörg Neumann. So with Jörg doing the UI, myself writing the engine and Christian acting as our user/tester we set to work.

The result is the MapperActivity. The MapperActivity requires you to specify the SourceType and DestinationType - we used the Type Browser from CodeProject for this. Then you can edit the Mappings which brings up the mapping tool

You drag members (properties or fields) from the left hand side and drop them on type compatible members (properties or fields) on the right hand side.

Finally you need to databind the SourceObject and DestinationObject to tell the mapper where the data comes from and where the data is going. You can optionally get the MapperActivity to create the destination object tree if it doesn't exist using the CanCreateDestination property. Here's the full property grid:

And thanks to Jon Flanders for letting me crib his custom PropertyDescriptor implementation (why on earth don't we have a public one in the framework?). You need one of these when creating extender properties (the SourceObject and DestinationObject properties change type based on the SourceType and DestinationType respectively).

Here is the activity - I'll clean up the code before posting that.

MapperActivity.zip (37.78 KB)

Feedback always appreciated

In my last post I talked about the WCF async programming model. On the service side you have to create a custom implementation of IAsyncResult. Fortunately this turns out to be pretty much boilerplate code so here is a general purpose one

class UniversalAsyncResult : IAsyncResult, IDisposable
{
  // To store the callback and state passed
  AsyncCallback callback;
  object state;
 
  // For the implementation of IAsyncResult.AsyncCallback
  volatile ManualResetEvent waithandle;

  // Guard object for thread sync
  object guard = new object();

  // complete flag that is set to true on completion (supports IAsyncResult.IsCompleted)
  volatile bool complete;

  // Store the callback and state passed
  public UniversalAsyncResult(AsyncCallback callback, object state)
  {
    this.callback = callback;
    this.state = state;
  }

  // Called by consumer to set the call object to a completed state
  public void Complete()
  {
    // flag as complete
    complete = true;

    // signal the event object
    if (waithandle != null)
    {
      waithandle.Set();
    }

    // Fire callback to signal the call is complete
    if (callback != null)
    {
      callback(this);
    }
  }

  #region IAsyncResult Members
  public object AsyncState
  {
    get { return state; }
  }
 
  // Use double check locking for efficient lazy allocation of the event object
  public WaitHandle AsyncWaitHandle
  {
    get
    {
      if (waithandle == null)
      {
        lock (guard)
        {
          if (waithandle == null)
          {
            waithandle = new ManualResetEvent(false);
            // guard against the race condition that the waithandle is
            // allocated during or after complete
            if( complete )
            {
              waithandle.Set();
            }
          }
        }
      }
      return waithandle;
    }
  }

  public bool CompletedSynchronously
  {
    get { return false; }
  }

  public bool IsCompleted
  {
    get { return complete; }
  }
  #endregion

  #region IDisposable Members
  // Clean up the event if it got allocated
  public void Dispose()
  {
    if (waithandle != null)
    {
      waithandle.Close();
    }
  }
  #endregion
}

You create an instance of UniversalAsyncResult passing in an AsyncCallback and state object if available. When you have finished your async processing just call the Complete method on the UniversalAsyncResult object.

Comments welcome

WCF supports asynchronous programming. On the client side you can generate an async API for invoking the service using svcutil

svcutil /async <rest of command line here> 

In VS 2005 you had to use the command line. In VS2008 the Add Service Reference dialog now has the Advanced button which allows you to generate the async operations

However, the fact that you can do asynchronous programming on the service side is not so obvious. You need to create the service contract according to the standard .NET async pattern of Beginxxx / Endxxx. You also need to annotate the Beginxxx method with an OperationContract that says this is part of an async pair. Note you do not annotate the Endxxx method with [OperationContract]

[ServiceContract]
interface IGetData
{
    // This pair of methods are bound together as WCF is aware of the async pattern
    [OperationContract(AsyncPattern=true)]
    IAsyncResult BeginGetData(AsyncCallback cb, object state);
    string[] EndGetData(IAsyncResult iar);
}

Now, its fairly obvious that when the request message arrives the Beginxxx bethod is called by WCF. The question is how is the Endxxx method called? WCF passes the Beginxxx method an AsyncCallback that must somehow be cached and invoked when the operation is complete. This callback triggers the call to Endxxx. Another question is "what am I meant to return from the Beginxxx method?" - where does this IAsyncResult come from? And that, unfortunately, is up to you - you have to supply one, which is a good place to cache that AsyncCallback that WCF passed you.

Here is a simple sample shows how the service side works

Async.zip (219.99 KB)

Comments, as always, welcome

In my last post I talked about the base class I created for creating custom composites that you want control over the execution logic. Firstly Josh pointed out that I ought to state its limitations - it only executes (and therefore cancels) one child activity at a time so it would not be useful for building, for example, the ParallelActivity.

With that limitation in mind though, there are a number of useful applications. I had a think about a different execution model and came up with prioritized execution. So I built this on top of my base class which was an interesting exercise as I could use an attached property for the Priority which meant I didn't limit the children to a special child activity type. I also had a think about the design time and realised that the sequential and parallel models for composite design time wasn't appropriate so I stole^H^H^H^H^H leveraged the designer that the Conditional Activity Group uses which seemed to work well.

I packaged up the code and told Josh about it

"Oh yeah, thats an interesting one - Dharma Shukla and Bob Schmidt used that in their book"

...

It turns out that pretty much half the world has decided to build a prioritised execution composite.

Heres the code anyway

PrioritizedActivitySample.zip (76.36 KB)

But then I got to thinking - actually there is a problem with this approach - elegant as it may be. If you cannot change the priorities at runtime then all you are building is an elaborate sequence. So it turns out that attached properties cannot be data bound in WF - you have to use a full DependencyProperty created with DependencyProperty.Register rather than DependencyProperty.RegisterAttached. So I reworked the example, creating a special child activity DynamicPrioritizedSequenceActivity that has a priority that is a bindable dependency property. I also had to create a validator to make sure only this type of child could be added to the composite.

This neat thing now is you have a prioritized execution composite whose children can dynamically change their priority using data binding. The code for this is here

DynamicPrioritySample.zip (78.83 KB)

So between these two samples we have prioritized execution, designer support, attached property support and validator support

As usual all comments welcome

There are two types of custom composites you may end up writing:

1. a form of UserControl with a set of activities whose composite functionality you want to reuse. This is what you get when you add a new activity to a workflow project using the menu item. You end up with a class that derives from SequenceActivity and is the one occasion with a custom activity where you don;t really have to override Execute
2. a parent that wants to take control over the execution model of its children (think of the execution model of the IfElseActivity, WhileActivity and the ParallelActivity)

To me the second of these is more interesting (although no doubt less common) and involves writing a bit of plumbing. In general, though, there is a common pattern to the code and so I have created a base class that wraps this plumbing.

CustomCompositeBase.zip (55.6 KB)

There are two fundemental overridable methods:

LastActivityReached: returns whether there are more child activities to execute
GetNextActivity: returns the next activity to be scheduled

Inside the zip there is also a test project that implements a random execution composite (highly useful I know).

Feedback, as always, very welcome.

Yesterday I posted a base class for Async Activities that buried all the plumbing. However, looking at the zip I only had the binary for the base class and the test harness. Here is the zipped code.

AsyncActivityBaseClass12.zip (69.71 KB)

In addition Rod gave me some feedback (for example making the base class abstract messed up the design time view for the derived activity). In the light of that and other ideas I have made some changes:

• The base class is now concrete and so designer view now works for derived classes
• If you don't supply a queue name then the queue name defaults to the QualifiedName of the activity
• Cancellation support is now embedded in the base class.
• There is a new override called DoAsyncTeardown to allow you to deallocate any resources you allocated in DoAsyncSetup

I hope someone finds this useful

A while back Joe Duffy announced that they were changing the default max number of worker threads in the CLR threadpool in CLR 2.0 SP1. The change was to increase the limit 10 fold from 25 threads per processor per process to 250 per processor per process.

The basis of the change is that it is possible to deadlock the threadpool if you spawn and then wait for threadpool work from a threadpool thread (e.g. in processing an ASP.NET request). Increasing the size makes this occurance less likely (although doesn't remove the risk).

Although its not obvious, CLR 2.0 SP1 ships with .NET 3.5. So I was teaching Guerrilla .NET a couple of months ago and thought I'd demo the change by running  a project built under VS2005 then rebuild it under VS2008. What I hadn't expected (although on reflection its not that surprising) is that when I ran under 2005 on a machine with 2008 *installed* then 2005 also has the new behavior. In other words it patches the 2.0 CLR rather than being something that only affects new applications.

Why should we care? Well there are two main reasons we use threadpools. Firstly to remove the repeated cost of creating and destroying threads by repeatedly using a set of already created threads; secondly, it makes sure that we don't drown the machine in spawned threads when the application comes under load. Extra threads consume resources: the scheduler has to give them time and they default to 1Mb stack space.

And here is where I think the problem is. If you create a server side application, on 32 bit Windows, deploy it on an 8-way machine and put in under load, before this change you will cap out at 200 threads by default which is a reasonable (maybe a little high) number for IO bound processing. However, after the change this will cap out by default at 2000 threads. 2000 threads with 1Mb stack space means 2Gb of memory consumed by the threads. On 32 bit windows this is the total amount of addressable memory address space ... ouch! Say hello to Mr OutOfMemoryException.

So for existing applications this is a potentially breaking change. Your applications will have to take control of the max number of worker threads to ensure correct processing. ASP.NET already does this and you can adjust it via the <processModel> configuration element. In a windows service based application you have to make a call to ThreadPool.SetMaxThreads to control the threadpool.

To his credit Joe doen't pretend this change was the ideal resolution to a real-world problem but it is a change that will affect existing applcations as it is a patch to CLR 2.0 rather than a feature of a new version of the CLR

One of the most powerful aspects of Windows Workflow is that it can manage a workflow instance's state when it has no work to do. Obviously to do this it needs to know that your activity is waiting for something to happen. A number of activities in the Standard Activity Library run asynchronously like this -e.g. the DelayActivity and the new ReceiveActivity. However, most real WF projects require you to write your own custom activities. To perform long running operations or wait for external events to occur, these too should be able to run asynchronously.

To be usable in all situations, an async activity needs to do more than just derive from Activity and return ActivityExecutionStatus.Executing from the Execute override. It also needs to implement IActivityEventListener<QueueEventArgs> and IEventActivity (the latter allows it to be recognised as an event driven activity). We end up with 3 interesting methods

Execute
IEventActivity.Subscribe
IEventActivity.Unsubscribe

Depending where your activity is used these get called in a different order

Sequential Workflow: Execute
State Machine: Subscribe, Execute, Unsubscribe
ListenActivity: Subscribe, Unsubscribe, Execute

So getting the behavior correct in where you create the workflow queue you are going to listen on, set up the event handler on the queue, process the results from the queue and close the activity can be a little tricky. So I have created a base class for async activities that buries the complexity of the plumbing and leaves you with three responsiblities:

1. Tell the base class the name of the queue you wish to wait on
2. Optionally override a method to set up any infrastructure you need before you start waiting on the queue
3. Override a method to process the data that was passed on to the queue and decide whether you are still waiting for more data or you are finished

You can download the code here AsyncActivityBaseClass.zip (38.36 KB). Any feedback gratefully received

Finally, thanks to Josh and Christian for sanity checking the code

Really as a note to myself as I always forget how to do this. To turn off JIT debugging which takes an age to try to attach the debugger under the covers, you set this value in the registry

HKLM\SOFTWARE\Microsoft\.NETFramework\DbgJITDebugLaunchSetting=1

At least I now know where to go to find that information

The release of BizTalk 2006 has brought a lot of love to the messaging engine in terms of handling failures. Probably the biggest change was messaging errors like “no subscription” are now resumable. However, there are other changes such as being able to subscribe to failed messages using the ErrorReport context properties and enabling routing for failed messages on a receive or send port.

Another change was added in the Xml and Flat File disassemblers – that of recoverable interchange processing. When a disassemble is breaking apart a composite message either in the form of a flat file or an envelope schema what happens if something is wrong about one of the child messages? In BizTalk 2004 the entire batch would fail. In BizTalk 2006 this is still the default behaviour; however, you can now set RecoverableInterchange to true on the disassembler which will cause all successful messages to be processed and only the incorrect ones to fail.

This seems like a great idea but in this article I want to look at some of the subtleties in the XmlDisassembler that make this technology less useful than may first appear.

First, here is the setup. I have two schemas: an envelope that is simply a container for repeated records; a person schema with child elements firstName and lastName as part of a sequence group.

This is the person schema

<xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003"    

           xmlns="http://dotnetconslt.co.uk/schema/person"

           targetNamespace=http://dotnetconslt.co.uk/schema/person

           xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="person">

    <xs:complexType>

      <xs:sequence>

        <xs:element name="firstName" type="xs:string" />

        <xs:element name="lastName" type="xs:string" />

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

And here is the envelope

<xs:schema xmlns:ns0="http://dotnetconslt.co.uk/schema/person"

           xmlns:b="http://schemas.microsoft.com/BizTalk/2003"

           xmlns="http://dotnetconsult.co.uk/schema/envelope"

     targetNamespace="http://dotnetconsult.co.uk/schema/envelope"

           xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:import schemaLocation=".\person.xsd"

            namespace="http://dotnetconslt.co.uk/schema/person" />

  <xs:annotation>

    <xs:appinfo>

      <b:schemaInfo is_envelope="yes"                 

         xmlns:b=http://schemas.microsoft.com/BizTalk/2003 />

      <b:references>

        <b:reference

           targetNamespace="http://dotnetconslt.co.uk/schema/person" />

      </b:references>

    </xs:appinfo>

  </xs:annotation>

  <xs:element name="envelope">

    <xs:annotation>

      <xs:appinfo>

        <b:recordInfo body_xpath="/*[local-name()='envelope' and

namespace-uri()='http://dotnetconsult.co.uk/schema/envelope']" />

      </xs:appinfo>

    </xs:annotation>

    <xs:complexType>

      <xs:sequence>

        <xs:element minOccurs="1"

                    maxOccurs="unbounded"

                    ref="ns0:person" />

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

Notice the emboldened pieces that the XmlDisassembler uses to break apart the envelope.

Now, I deploy these two schemas into Biztalk, and then create a Receive Port with a Receive Location using the XmlReceive pipeline. I make sure that error reporting is turned on.

Next I create two send ports: one with a filter set to the BTS.ReceivePortName of the receive port; one using a filter set for error reporting like this

So with the infrastructure in place we can try submitting a message. Let’s start with one that works just to ensure that the envelope is working correctly. Here is the good message:

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Niels</firstName>

    <lastName>Bergland</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Jason</firstName>

    <lastName>Whittington</lastName>

  </ns1:person>

</ns0:envelope>

And as expected, dropping this into the receive folder results in three messages in the non-error send folder.

Now let’s use this message

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:personbad xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Niels</firstName>

    <lastName>Bergland</lastName>