I've been looking at the routing infrastructure Microsoft are releasing in .NET 3.5 SP1. This is the infrastructure that allows me to bind an HTTP Hander to a URI rather than simply using webforms. It is used in the ASP.NET MVC framework that is in development. The infrastructure is pretty clean in design, First you add a reference to System.Web.Routing. Then you simply create Route objects binding a URI to an implementation of IRouteHandler. Finally you add it to a RouteTable static class. Global.asax is the ideal spot for this code.

void Application_Start(object sender, EventArgs e) 
{
        Route r = new Route("books", new BooksRouteHandler());
        RouteTable.Routes.Add(r);
        r = new Route("books/isbn/{isbn}", new BooksRouteHandler());
        RouteTable.Routes.Add(r);
        r = new Route("search/price", new SearchRouteHandler());
        RouteTable.Routes.Add(r);
}

Here BooksRouteHandler and SearchRouteHandler implement IRouteHandler

public interface IRouteHandler
{
    IHttpHandler GetHttpHandler(RequestContext requestContext);
}

So for example the BooksRouteHandler looks like this

public class BooksRouteHandler : IRouteHandler
{
    public IHttpHandler GetHttpHandler(RequestContext requestContext)
    {
        if (requestContext.RouteData.Values.ContainsKey("isbn"))
        {
            string isbn = (string)requestContext.RouteData.Values["isbn"];
            return new ISBNHandler(isbn);
        }
        else
        {
            return new BooksHandler();
        }
    }
}

Where ISBNHandler and BooksHandler both implement IHttpHandler

This is all pretty straightforward. The one thing that had me puzzling for a while is who looks at the RouteTable. Reflector to the rescue! There is a module in the System.Web.Routing assembly called UrlRoutingModule. If you add this in to your web.config the routing starts working. The config piece looks like this

<httpModules>
      <add name="Routing" 
           type="System.Web.Routing.UrlRoutingModule, System.Web.Routing, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
</httpModules>

As promised, here are the demos from the precon myself and Dave Wheeler (get a blog Dave) did at Software Architect 2008. It was a fun day talking about security, WCF, WF, Windows Forms, WPF, Silverlight, Ajax, ASP.NET MVC, LINQ and Oslo

DotNetForArchitects.zip (791.24 KB)

There is a text file in the demos directory in the zip that explains the role of each of the projects in the solution

Edit: Updated the download link so hopefully the problems people have been experiencing will be resolved

I've just got back from Software Architect 2008. Its a great conference to speak at and an interesting change from speaking at hard core developer conferences like DevWeek. Thanks to everyone who attended my sessions - the slides and demos are below

SOA with WCF and WF - SOA.zip (368.43 KB)

Volta - Volta.zip (506.21 KB)

The slides and demos from the pre conference workshop on .NET 3.5 for architects that Dave Wheeler and me presented will be posted early next week. We have realised we really need a guide to what all the projects are and how they relate - so we'll add this documentation and post them

The Workflow Mapper Activity for copying data from one object to another was an interesting project for myself, Jörg and Christian. However, none of us has the cycles to turn it into the hugely valuable activity I think it could be. therefore, we have decided to publish the code on codeplex so hopefully we can get community involvement to polish the functionality.

You can find the project at

http://www.codeplex.com/WFMapperActivity

Take a look and let us know if you want to get involved in the project

I use my machine for development and research as well as teaching. Originally I only installed VS2005 and SQL Express. I later installed VS 2008. Finally I installed BizTalk (which required a full-blown SQL Server install) and thought nothing more of it.

More recently I was checking into how LINQ to SQL was doing some things and realised I couldn't find SQL Server Profiler on my machine - in fact I couldn't find any of the SQL Server tools! I googled this a bit and found this was a well known issue with installing SQL Express before SQL Server. So I resigned myself to sorting this out on the next repave.

I was talking to Kev Jones about this issue at DevWeek last week and yesterday he pops up and tells me he's solved it. Kev had the cunning idea of actually reading the SQL Server installation warning messages - which apparently tell you exactly what to do.

You can read Kev's post about it here

The demos from my DevWeek 2008 Postcon A Day of Connected Systems with VS 2008 are now here:

DayOfCS.zip (1.48 MB)

Thanks for attending the session

The demos for my DevWeek 2008 talk on a Developers Guide to Workflow are now available here:

DevGuideToWF.zip (31.23 KB)

Thanks a lot for attending the session

The demos from my DevWeek 2008 Silver talk about integrating WCF and WF are available here:

Silver.zip (107.74 KB)

Thanks for coming to the talk

The demos from my DevWeek 2008 Robust Long Running Workflow talk are now available here:

RobustLongRunning.zip (28.84 KB)

Thanks for coming to the talk.

Recently I wrote an article on Volta for the DevelopMentor newsletter DevelopMents. I concentrated on the core of what Volta is doing, IL rewriting, rather than highlight the IL to JavaScript functionality that has caught most attention. You can read the article here.

Recently I discussed creating robust multi-host workflow architectures with WF using the SqlWorkflowPersistenceService. I talked about an issue with the way workflow ownership is implemented and  showed some code that would fix the issue but that I also said was a hack. Ayende rightly pointed out that for high volume systems it would be a disaster. The point of that post was to highlight the problem.

Ideally the workflow team will fix the persistence service to recover from abandoned workflows more robustly - in the meantime the following stored procedure will unlock the abandoned workflows and make then runnable. The idea is to make this a scheduled job in the database running every few seconds - this is essentially what BizTalk does.

CREATE PROCEDURE dbo.ClearUnlockedWorkflows
AS
BEGIN
  SET NOCOUNT ON

  UPDATE InstanceState Set ownerID=null, 
                           ownedUntil=null, 
                           unlocked=1,
                           nextTimer=getdate()
  WHERE NOT ownerID is NULL and 
            ownedUntil < getdate()

  RETURN
END

The only oddity in the code is the setting of the nextTimer to the current time. The issue is that straight persistence (as opposed to unloading on a delay) sets this value to 31st Dec 9999 which is obviously not going to be reached for some time. Unfortunately the workflow will only be scheduled due to an expired timer so I have to reset the timer such that it will expire immediately. I can't think of any issues this would cause but if there's a scenario I haven't thought of all comments are welcome.

For the third year running I'm going to be speaking at BearPark's DevWeek conference. This year I actually managed to submit some breakout session titles on time too.

Wednesday 12th March

11:30 A developer’s guide to Windows Workflow Foundation
There are many challenges to writing software. Not least of these are lack of transparency of code and creating software that can execute correctly in the face of process or machine restart. Windows Workflow Foundation (WF) introduces a new way of writing software that solves these problems and more. This session explains what WF brings to your applications and explains how it works. Along the way we will see the major features of WF that make it a very powerful tool in your toolkit, removing the need for you to write a lot of complex plumbing.

14:00 Creating robust, long-running Workflows
Long-running processes have unique requirements in that they need to maintain state over process restart; Windows Workflow Foundation (WF) enables this with its persistence infrastructure. However, there are issues around hosting and activity development that require attention for long running workflows to be robust. This session looks at the design of the workflow persistence service; issues around hosting and creating full featured asynchronous activities. This session assumes some familiarity with WF.

16:00 Cross my palm with Silver – creating workflow-based WCF services
There are very good reasons for using a workflow to implement a WCF service: workflows can provide a clear platform for service composition (using a number of building block services to generate functionally richer service); workflows can manage long running stateful services without having to write your own plumbing to achieve this. This session introduces the new Visual Studio 2008 Workflow Services. This technology, previously known as “Silver”, provides a relatively seamless integration between WF and WCF, enabling the service developer to concentrate on the application functionality rather than the plumbing. This session assumes some familiarity with WF and WCF.

Friday 14th March - Postcon

A day of connected systems with Visual Studio 2008
Most businesses find themselves building applications that use two or more machines working together to produce their functionality. One of the challenges in this world, apart from the actual business logic being implemented, is connecting the different parts of the application in a way that best fits the environment the machines are places – are there firewalls in place? Are some parts of the application written on different platforms such as Java? Do the different parts of the application have to maintain their state over machine restart? Late 2006 saw Microsoft release WCF and WF to tackle some of these challenges. However, parts of the story were left untold – especially the integration between the two.
Visual Studio 2008 introduces a number of new features for writing service based software. Its features build on the libraries released as part of .NET 3.0, providing an integration layer between the two. In this pre/post conference session we start at the basics of how WCF and WF work and then look at the various integration technologies introduced in Visual Studio 2008.

So if you're attending DevWeek I hope to see you there

One of the most powerful features of the Windows Workflow Foundation is its ability to automate state management of long running processes using a persistence service. Microsoft supply a persistence service out of the box - the SqlWorkflowPersistenceService. Unsurprisingly this persists state in SQL Server (or SQL Express).

Once you have a persistence service more possibilities open up: different applications can progress the same workflow instance over time; multiple hosts can process workflows in a scale out model. This second feature needs a little investigation - there is a gotcha hiding there you need to be aware of.

The issue is how to stop more than one host picking up the same instance of a workflow and processing it at the same time - imagine the workflow transfered $10,000,000 from one account to another, you'd hardly want this happening twice. So if the possiblity exists for multiple hosts to see the same persistence store, the persistence service must be able to ensure only one host is executing a workflow at any one time.

The SqlWorkflowPersistenceService handles multiple concurrent hosts by using the concept of workflow ownership - the guid of a host (created randomly by the persistence service constructor) is stamped against a workflow that it is actively executing (not in an idle or completed state). Now the question comes "what if the host dies while executing a workflow?". This is what the ownership timeout is for. You set the ownership timeout in the constructor of the SqlWorkflowPersistenceService.

SqlWorkflowPersistenceService sql = 
            new SqlWorkflowPersistenceService(CONN, 
                                              true, 
                                              TimeSpan.FromSeconds(10), 
                                              TimeSpan.FromSeconds(5));

workflowRuntime.AddService(sql);

Here the third parameter specifies how long a host is allowed to run a workflow before persistence occurs. If the host takes longer than this then it will get an error when it atempts to persist. The fourth parameter is the polling interval for how often the persistence service will check for expired timers.

Now the idea is that if a host dies then it's lock will timeout and another host can pick up the work. There is a problem, however, in the implementation. The persistence service only looks for expired ownership locks when it first starts - not when it polls for expired timers. Therefore, for a workflow instance whose host has died mid-processing, it will only recover if a new host instance starts after the timeout has occurred.

So how can you make this more robust? Well we need a way to explicitly load the workflows that have had their ownership expire - unfortunately there is no exposed method to do this on the SqlWorkflowPersistenceService. Instead we have to get all the workflows, catch the exception if we load a locked one and unload any that aren't ready to run. Here is an example:

We, at DevelopMentor, are very excited to be running our SilverLight 2.0 course, Essential SilverLight 2.0 in the UK on 17th March - all the new bits straight from Mix'08! Course author, Dave Wheeler, will be delivering the course at our London facility so if you are interested in the next generation of Rich Internet Applications register now.

Christian noticed a bug in my Async Activity base class that I talked about here and here. The latest code is here

When working with WF I always find it useful having a BizTalk background. Issues that are reasonably well known in BizTalk are not immediatedly apparent in Workflow unless you know they are part and parcel of that style of programming.

One issue in BizTalk is where you are waiting for a number of messages to start off some processing and you don't know in which order they are going to arrive. In this situation you use a Parallel Shape and put activating receives in the branches initializing the same correlation set. The orchestration engine understands what you are trying to do and creates a convoy for the remaining messages when the first one arrives. This is known as a Concurrent Parallel Receive. You don't leave the parallel shape until the all the messages have arrived and the convoy ensures that the remaining messages are routed to the same orchestration instance.

There is, however, an inherent race condition in this architecture in that: if two messages arrive simultaneously, due to the asynchronous nature of BizTalk, both messages could be processed before the orchestration engine has a chance to set up the convoy infrastructure. We will end up with two instances of the orchestration both waiting for messages that will never arrive. All you can do is put timeouts in place to ensure your orchestrations can recover from that situation and flag the fact that the messages require resubmitting.

With Workflow Services we essentially have the same issue waiting for us. Lets set up the workflow ...

If we call this from a client as follows:

PingClient proxy = new PingClient();
proxy.Ping1();
proxy.Ping2();

then everything works ok - in fact it works irrespective of the order the operations are called in, that's the nature of this pattern. It works because by the time we make the second call, the first has completed and the context is now cached on the proxy.

But lets make the client a bit more complex:

static IDictionary<string, string> ctx = null;
static void Main(string[] args)
{
  PingClient proxy = new PingClient();
  IContextManager mgr = ((IChannel)proxy.InnerChannel).GetProperty<IContextManager>();

  Thread t = new Thread(DoIt);
  t.Start();

  if (ctx != null)
  {
    mgr.SetContext(ctx);
  }

  proxy.Ping1();
  ctx = mgr.GetContext();

  Console.WriteLine("press enter to exit");
  Console.ReadLine();
}

static void DoIt()
{
  PingClient proxy = new PingClient();
  IContextManager mgr = ((IChannel)proxy.InnerChannel).GetProperty<IContextManager>();

  if (ctx != null)
  {
    mgr.SetContext(ctx);
  }
  proxy.Ping2();
  ctx = mgr.GetContext();
}

Here we make the two calls on different proxies on different threads. A successful call stores the context in the static ctx field. Now a proxy will use the context if it has already been set, otherwise it assumes that it is the first call. So here the race condition has made its way all the way back to the client. There are things we could do about this in the client code (taking a lock out while we're making the call so we complete one and store the context before the other checks to see if the context is null), however, that really isn't the point. The messages may come from two separate client applications which both check a database for the context. Again we have an inherent race condition that we need to put architecture in place to detect and recover from. It would be nice to put the ParallelActivity in a ListenActivity with a timeout DelayActivity. However you can't do this because a ParallelActivity does not implement IEventActivity and so the ListenActivity validation will fail (the first activity in a branch must implement IEventActivity). We therefore have to put each receive in its own ListenActivity and time out the waits individually.

So is the situation pretty much the same for both WF and BizTalk? Well not really. The BizTalk window of failure is much smaller than the WF one as the race condition is isolated server side. Because WF ropes the client into message correlation the client has to receive the context and put it somewhere visible to other interested parties before the race condition is resolved.

Hopefully Oslo will bring data orientated correlation to the WF world

I've just been reading the latest team blog entry from the Volta team. This entry is meant to address the issue that I discussed here about the dangers of taking arbitrary objects and remoting them.

They say that they are abstracting away "unnecessary details" only leaving the necessary ones. Firstly, every abstraction leaks (just look at WCF for an example of that) so no matter how hard you try to abstract away the plumbing it will come through the abstraction in unexpected ways. Secondly the remote boundary is not an unnecessary detail. Its a fundemental part of the design of an application.

Unless Volta has heuristics inside it to generate a remote facade during execution, the interface into a remote object is of huge importance to how an application will perform and scale. Volta should at least give you a warning if you apply the [RunAtOrigin] attribute to a class with properties on it.

Does this mean that the whole idea is broken? Not at all - it just means that applications have to be designed with Volta in mind. Decisions have to be made in the design about which layers *may* get remoted and the interface into that layer should be designed accordingly. Then the exact decision about which layers to *actually* remote can be deferred and tuned according to profiling the application.

Ian pointed out that there was a race condition in the UniversalAsyncResult that I posted a few days ago. I have amended the original post rather than repost the code.

The changes are to make the complete flag and the waithandle field volatile and to check the complete flag after allocating the waithandle in case Complete was called while the waithandle was being allocated.

Isn't multithreaded programming fun

On 5th December, Microsoft Live Labs announced Volta. The idea of Volta is to be able to write your application in a .NET language of your choice and then the Volta infrastructure takes care of targetting the right platform (JavaScript or SilverLight, IE or FireFox). This side of things is pretty neat. The other thing it allows you to do is to write your application monolithically and then decide on distribution later on based on profiling, etc. This sounds quite neat but I have some reservations.

I went through the DCOM wave or "COM with a longer wire" as it was labelled at the time and learned some hard lessons. I then went through the .NET Remoting experience and learned  ... well ... the same hard lessons. The reality is unless you design something to be remote then when you remote it its going to suck.

Lets look at a Volta example. First some basics:

1. Download and install the CTP
2. Create a Volta application project

Now lets build our Volta application. You create the HTML page for the UI (heres the important snippet)

<body>
  <p>Press the button to do chatty stuff</p>
  <p><button id="chat">Chat!</button></p>
  <div id="output" />
</body>

You create the Volta code to wire up your code to the HTML

public partial class VoltaPage1 : Page
{
  Button chat;
  Div output;

  public VoltaPage1()
  {
    InitializeComponent();
  }

  partial void InitializeComponent()
  {
    chat = Document.GetById<Button>("chat");
    output = Document.GetById<Div>("output");
  }
}

Now create a class that has a chatty interface

If you double click on this server icon you get a monitor window that allows you to trace interaction with the server. Bring that up and check the Log Requests checkbox.

Now click on the Chat! button in the browser. Notice you get a bunch of assembly loads in the log window. Clear the log and press the Chat! button again. Now you see there are no requests going across the wire as the entire application is running in the browser.

Now go to the project properties and select the Volta tab. Enable the <queue dramatic music> Tiersplitter. Go to the HoldStatement class, right click on it and select Refactor -> Tier Split to Run at Origin. Notice we now get a [RunAtOrigin] attribute on the class.

Rebuild and press F5 again. Bring up the Log window again and select to log messages. Now every time you press the Chat! button it makes a request to the server. We have distributed our application with just one attribute! - how cool is that?! But hold on ... how many requests are going to the server? Well one for each property access.

Latency is the death knell of many distributed applications. You have to design remote services to be chunky (lots of data in one round trip) otherwise the network roundtripping will kill your application performance. So saying you can defer architecting your solution in terms of distribution then simply annotate classes with an attribute is a dangerous route to go down. We have already learned the hard way that taking a component and just putting it on the other end of a wire does not make for a good distributed application - Volta appears to be pushing us in that direction again.

Yesterday I announced the release of the MapperActivity. I said I'd clean up the source code and post it.

Two things are worth noting:

1. The dependency on the CodeProject Type Browser has gone
2. There is a known issue where you can map more than one source element to the same destination - in this case the last mapping you create will win. We will fix this in a subsequent release.

Here's the latest version with the code

MapperActivityLibrary.zip (933.62 KB)

One of the neat things you can do in BizTalk is to create a map that takes one kind of XML message and transforms it into another. They basically are a wrapper over XSLT and its extension infrastructure. The tool you use to create maps is the BizTalk mapper. In this tool you load in a source and destination schema - which get displayed in treeviews and then you drag elements from the source to the destination to describe how to build the destination message.

Now, in WF - especially when using Silver (the WF/WCF integration layer introduced in .NET 3.5) - you tend to spend alot of time taking data from one object and putting it into another as different objects are used to serialize and deserialize messages received and sent via WCF. This sounds alot like what the BizTalk Mapper does, but with objects rather than XML messages.

I was recently teaching Guerrilla Connected Systems (which has now been renamed Guerrilla Enterprise.NET) and Christian Weyer was doing a guest talk on REST and BizTalk Services. Afterwards, Christian and me were bemoaning the amount of mundane code you get forced to write in Silver and how sad it was that WF doesn't have a mapper. So we decided to build one. Unfortunately neither myself nor Christian are particularly talented when it comes to UI - but we knew someone who was: Jörg Neumann. So with Jörg doing the UI, myself writing the engine and Christian acting as our user/tester we set to work.

The result is the MapperActivity. The MapperActivity requires you to specify the SourceType and DestinationType - we used the Type Browser from CodeProject for this. Then you can edit the Mappings which brings up the mapping tool

You drag members (properties or fields) from the left hand side and drop them on type compatible members (properties or fields) on the right hand side.

Finally you need to databind the SourceObject and DestinationObject to tell the mapper where the data comes from and where the data is going. You can optionally get the MapperActivity to create the destination object tree if it doesn't exist using the CanCreateDestination property. Here's the full property grid:

And thanks to Jon Flanders for letting me crib his custom PropertyDescriptor implementation (why on earth don't we have a public one in the framework?). You need one of these when creating extender properties (the SourceObject and DestinationObject properties change type based on the SourceType and DestinationType respectively).

Here is the activity - I'll clean up the code before posting that.

MapperActivity.zip (37.78 KB)

Feedback always appreciated

In my last post I talked about the WCF async programming model. On the service side you have to create a custom implementation of IAsyncResult. Fortunately this turns out to be pretty much boilerplate code so here is a general purpose one

class UniversalAsyncResult : IAsyncResult, IDisposable
{
  // To store the callback and state passed
  AsyncCallback callback;
  object state;
 
  // For the implementation of IAsyncResult.AsyncCallback
  volatile ManualResetEvent waithandle;

  // Guard object for thread sync
  object guard = new object();

  // complete flag that is set to true on completion (supports IAsyncResult.IsCompleted)
  volatile bool complete;

  // Store the callback and state passed
  public UniversalAsyncResult(AsyncCallback callback, object state)
  {
    this.callback = callback;
    this.state = state;
  }

  // Called by consumer to set the call object to a completed state
  public void Complete()
  {
    // flag as complete
    complete = true;

    // signal the event object
    if (waithandle != null)
    {
      waithandle.Set();
    }

    // Fire callback to signal the call is complete
    if (callback != null)
    {
      callback(this);
    }
  }

  #region IAsyncResult Members
  public object AsyncState
  {
    get { return state; }
  }
 
  // Use double check locking for efficient lazy allocation of the event object
  public WaitHandle AsyncWaitHandle
  {
    get
    {
      if (waithandle == null)
      {
        lock (guard)
        {
          if (waithandle == null)
          {
            waithandle = new ManualResetEvent(false);
            // guard against the race condition that the waithandle is
            // allocated during or after complete
            if( complete )
            {
              waithandle.Set();
            }
          }