# Monday, 18 June 2007

The release of BizTalk 2006 has brought a lot of love to the messaging engine in terms of handling failures. Probably the biggest change was messaging errors like “no subscription” are now resumable. However, there are other changes such as being able to subscribe to failed messages using the ErrorReport context properties and enabling routing for failed messages on a receive or send port.

Another change was added in the Xml and Flat File disassemblers – that of recoverable interchange processing. When a disassemble is breaking apart a composite message either in the form of a flat file or an envelope schema what happens if something is wrong about one of the child messages? In BizTalk 2004 the entire batch would fail. In BizTalk 2006 this is still the default behaviour; however, you can now set RecoverableInterchange to true on the disassembler which will cause all successful messages to be processed and only the incorrect ones to fail.

This seems like a great idea but in this article I want to look at some of the subtleties in the XmlDisassembler that make this technology less useful than may first appear.

First, here is the setup. I have two schemas: an envelope that is simply a container for repeated records; a person schema with child elements firstName and lastName as part of a sequence group.

This is the person schema

<xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003"    

           xmlns="http://dotnetconslt.co.uk/schema/person"

           targetNamespace=http://dotnetconslt.co.uk/schema/person

           xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="person">

    <xs:complexType>

      <xs:sequence>

        <xs:element name="firstName" type="xs:string" />

        <xs:element name="lastName" type="xs:string" />

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

And here is the envelope

<xs:schema xmlns:ns0="http://dotnetconslt.co.uk/schema/person"

           xmlns:b="http://schemas.microsoft.com/BizTalk/2003"

           xmlns="http://dotnetconsult.co.uk/schema/envelope"

     targetNamespace="http://dotnetconsult.co.uk/schema/envelope"

           xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:import schemaLocation=".\person.xsd"

            namespace="http://dotnetconslt.co.uk/schema/person" />

  <xs:annotation>

    <xs:appinfo>

      <b:schemaInfo is_envelope="yes"                 

         xmlns:b=http://schemas.microsoft.com/BizTalk/2003 />

      <b:references>

        <b:reference

           targetNamespace="http://dotnetconslt.co.uk/schema/person" />

      </b:references>

    </xs:appinfo>

  </xs:annotation>

  <xs:element name="envelope">

    <xs:annotation>

      <xs:appinfo>

        <b:recordInfo body_xpath="/*[local-name()='envelope' and

namespace-uri()='http://dotnetconsult.co.uk/schema/envelope']" />

      </xs:appinfo>

    </xs:annotation>

    <xs:complexType>

      <xs:sequence>

        <xs:element minOccurs="1"

                    maxOccurs="unbounded"

                    ref="ns0:person" />

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

Notice the emboldened pieces that the XmlDisassembler uses to break apart the envelope.

Now, I deploy these two schemas into Biztalk, and then create a Receive Port with a Receive Location using the XmlReceive pipeline. I make sure that error reporting is turned on.

Next I create two send ports: one with a filter set to the BTS.ReceivePortName of the receive port; one using a filter set for error reporting like this

So with the infrastructure in place we can try submitting a message. Let’s start with one that works just to ensure that the envelope is working correctly. Here is the good message:

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Niels</firstName>

    <lastName>Bergland</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Jason</firstName>

    <lastName>Whittington</lastName>

  </ns1:person>

</ns0:envelope>

And as expected, dropping this into the receive folder results in three messages in the non-error send folder.

Now let’s use this message

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:personbad xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Niels</firstName>

    <lastName>Bergland</lastName>

  </ns1:personbad>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Jason</firstName>

    <lastName>Whittington</lastName>

  </ns1:person>

</ns0:envelope>

The problem here is that we don’t have a schema deployed describing the personbad element. We drop this into the receive folder and this time we get the default behaviour that the whole batch gets abandoned and the single envelope message ends up in the error send folder.

OK let’s turn on recoverable interchange now. We could deploy a new pipeline with the XmlDisassembler configured differently. However in BizTalk 2006 there is a UI for the per-instance pipeline data so we can actually reconfigure this receive location’s XmlReceive pipeline. The button to get to it is shown here

Now we can change the Recoverable Interchange setting to true

Now when we deploy the last message we get two child messages into the non-error send folder and one into the error send folder – the one that could not be processed.

Well that seems to work pretty well for an incorrect well-formed message – lets try a non-well-formed message:

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:personbad xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Niels</firstName>

    <lastName>Bergland</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Jason</firstName>

    <lastName>Whittington</lastName>

  </ns1:person>

</ns0:envelope>

Notice that the second person record the start and end element names do not match. If we drop this into the receive folder we end up with the single envelope in the error send folder. This is because the XmlDisassembler requires well-formed XML to work. OK so that’s not going to work – what we need is another type of error to test. Lets make one of the child person records schema invalid.

<ns0:envelope xmlns:ns0="http://dotnetconsult.co.uk/schema/envelope">

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Rich</firstName>

    <lastName>Blewett</lastName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <lastName>Bergland</lastName>

    <firstName>Niels</firstName>

  </ns1:person>

  <ns1:person xmlns:ns1="http://dotnetconslt.co.uk/schema/person">

    <firstName>Jason</firstName>

    <lastName>Whittington</lastName>

  </ns1:person>

</ns0:envelope>

This time the second person record has the first and last name element transposed. As this is part of a sequence group this is not valid according to the schema.

We drop this message into the receive folder and we get three messages in the non-error send folder ... what?! It turns out that by default BizTalk with the XmlReceive pipeline does not perform validation.

However, we can force the XmlDisassembler to validate messages – once again lets use the per-instance pipeline data to achieve this

Now we drop the last message into the receive folder again and we get the single envelope message in the error send folder. Hmmm, so apparently the XmlDisassembler validates the entire envelope which is not what we want at all. The envelope validation is failing before we ever take the envelope apart.

All is not lost. Let’s validate the messages after disassembly. To do this we can use the XmlValidator component. But this component is not in the XmlReceive pipeline so we have to deploy a new pipeline with the XmlDisassembler in the disassemble stage and the XmlValidator in the validation stage. We’ll configure the XmlValidator to use the person schema to validate the message as this will be the message type after the disassember has run. Remember the XmlDisassembler will need to use recoverable interchange but not perform validation. So its pipeline data will look something like this:

So now we drop our envelope, with the schema invalid child, into the receive folder and out the other end we get ... the envelope message in the error send folder. The problem here is that Recoverable Interchange is only valid during the disassemble stage and so again the validation error causes the whole batch to fail and we are back to square one.

OK, so validation will have to take place in the disassemble stage. Unfortunately even if we make the envelop “dumb” with the child elements weakly typed as xs:any, the actual message validation uses the deployed schema and before the disassembly takes place – so the whole message is validated, not the ones resulting from disassembly.

So it would appear that as long as the envelope message has a matching pair on invalid child root node elements then Recoverable Interchange adds value. However, the likelihood that this is the only sort of invalid message you will get is, to me, pretty unlikely so based on that I’d say that Recoverable Interchange is of limited value with the XmlDisassembler. The frustrating thing is all it would take to make it a very powerful tool is to allow validation to only run on the disassembled messages and not on the envelope as a hole. In that case the failures would be within the scope of recoverable interchange and, therefore, the good messages would get through and the bad ones be caught.

Monday, 18 June 2007 16:21:50 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 
# Sunday, 17 June 2007

So I was teaching a cut down version of our Essential BizTalk course last week and one thing I was talking about struck me as making a good blog post (to follow shortly). So I cranked up my BizTalk virtual machine and realised I hadn't run it on this computer before as it discovered lots of new hardware.

I noticed that I also hadn't yet installed the VMWare Tools so I installed them and rebooted. At that point I remembered why i hadn't installed the VMWare Tools. this was a VPC image that I had converted that had the VPC Additions installed and the two don't play nice together.

I found myself at the CTRL-ALT-DEL screen with the image refusing to recognize any input such as CTRL-ALT-INS or using the menu to send the CTRL-ALT-DEL to the virtual machine.

I decided to reboot into safe mode and uninstall the VMWare Tools - no dice - still no input accepted. At this point I was faced with having to rebuild my image from scratch (stupidly I'd never taken a snapshot so I had nothing to recover to).

Then I remembered that the VMWare conversion was non-destructive, so I deleted all the VMWare bits of the image and left myself with the .vmc and .vhd files. This time when I re-added the image it re-converted and I got back to my starting point ... I immediately took a snapshot this time.

I keep on being reminded why I like VMWare so much.

Sunday, 17 June 2007 16:08:48 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 
# Saturday, 09 June 2007

Sometimes you have to wonder at the stupidity of organizations.

There were thunderstorms over Miami where my flight home to the UK from TechEd so amid chaotic scenes at gate 15 my flight got canceled. Unfortunately when they made the announcement I was on the phone so didn't initially realize. So when I finally worked out what was going on the gate was clearing. I was told to go reclaim my bags in the baggage hall then reschedule my flight at ticketing. And here I have to state my amazement of how massively inefficient the baggage handing appears to be at Orlando - I waited 1:15 on my way in and it took about an hour for my bag to come out after they canceled the flight even though they hadn't actually taken it on to a plane - most bizarre.

So I eventually get my bag and head up to ticketing only to find the other 200 people on my flight in front of me. Now the problem is that the line was so long it now went outside the terminal - ahh Orlando in the summer and the joy of non-A/C environments. The queue was moving very slowly but it turned out that people had been given a 1-800 number to call to rebook their flights. So I got that number off someone and got my flight rebooked for tomorrow. However, the airline were putting people up at a hotel and were giving out vouchers which you had to stay in the queue to receive. 

So here is what I think is stupid: a whole bunch of people in that queue had already got different flights and were simply waiting to get their hotel voucher - why on earth didn't they have someone walking down the queue giving out the vouchers to people who had already resolved their flight?

In the end i got sick of waiting and went and booked my own hotel over the Internet ... hold on, maybe I just answered my own question.

Saturday, 09 June 2007 04:09:13 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 
# Thursday, 07 June 2007

Something seems broken in the CLR ThreadPool. I was teaching Effective .NET recently and I noticed very strange behavior. It was the first time I had taught the course off my laptop which is a Core 2 Duo. Normally I teach the course in our training centre and the instructor machine is hyper-threaded.

I ran the following code:

static void Main(string[] args)
{
 
Console.ReadLine();

  ThreadPool.SetMinThreads(50, 1000);
 
ThreadPool.SetMaxThreads(5, 1000);

  int wtcount;
 
int iocount;

  ThreadPool.GetMaxThreads(out wtcount, out iocount);

  Console.WriteLine("{0} : {1}", wtcount, iocount);
}

Here I am calling SetMinThreads and SetMaxThreads. SetMinThreads "warms up" the thread pool so the default 0.5 second delay in spawning a new thread into the thread pool is avoided. SetMaxThreads changes the cap on the maximum number of threads available in the thread pool.

I had run this code several times before when talking about controlling the ThreadPool and expected the following output

5 : 1000

In other words the capping of the thread pool takes precedence over warming it up - which I would have thought is the correct behavior. However, ran I ran it this time I saw

50 : 1000

Which basically meant that SetMinThreads was overriding SetMaxThreads. But I had seen the expected behavior before on a hyper-threaded machine. So I did one more test (which is why the Console.ReadLine is at the top of the program). I started the program and then before pressing Enter went to task manager and set the process affinity to only execute on one processor. Lo and Behold I get the output

5 : 1000

So the behavior with respect to SetMinThreads and SetMaxThreads is different if the process is running with two (or more?) processors or not (note there is no difference between standard single processor and a hyper-threaded processor). This to me is totally busted. I know what I think the behavior should be but irrespective of that - it should definitely be consistent across the number of processors.

Edit: Dan points out to me that SetMinThreads(50, 1000) is obviously going to fail on a single proc machine as there are only 25 threads in the thread pool - Doh! So what was happening was on a single proc test the call to SetMinThreads failed and so the call to SetMaxThreads succeeded. As Dan pointed out - if you use 25 instead of 50 for the number of threads in SetMinThreads then the behavior is consistent in that the two APIs both fail if they try to do inconsistent things. Thanks Dan.

However, I do find it odd that I have asked the CLR to do something for me and it has failed - I would kind of expect an exception. Now I know I didn't check the return value so it is partially my own fault, but surely return values should only be used for expected behavior.

Thursday, 07 June 2007 22:45:22 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 

In the previous article I talked about hosting options for WCF. Here I'll discuss hosting for WF. However, to understand the options fully we need to digress briefly and talk about some of the features of WF.

Asynchronous Execution

The execution model of WF is inherently asynchronous. Activities are queued up for execution and the runtime asks the scheduler service to make the calls to Activity.Execute on to threads. The DefaultWorkflowSchedulerService (which you get if you take no special actions) maps Execute calls on to thread pool worker threads. The ManualWorkflowSchedulerService (which also comes out of the box but you have to install specifically install using WorkflowRuntime.AddService) schedules Execute on the thread that calls ManualWorkflowSchedulerService.RunWorkflow.

Persistence

By default the workflow runtime does not persist workflows. Only when an object of a type that derives from WorkflowPersistenceService is added as a runtime service does the workflow runtime start persisting workflows both in the form of checkpoints when significant things have happened in the workflow's execution and to unload a workflow when it goes idle. When something happens that should be directed to an unloaded workflow the host (or a service) calls WorkflowRuntime.GetWorkflow(instanceId) and the runtime can reload the workflow from persistent storage.

One issue here is that of timers. If a workflow has gone idle waiting for a timer to expire (say via the DelayActivity) then no external event is going to occur to trigger a call WorkflowRuntime.GetWorkflow(instanceId)to allow the workflow to continue execution. In this case the persistence service must have functionality to discover when a timer has expired in a persisted workflow. The out of the box implementation, the SqlWorkflowPersistenceService polls the database periodically (how often is controlled by a parameter passed to its constructor). Obviously for this to occur the workflow runtime has to be loaded and running (via WorkflowRuntime.StartRuntime).

Hosting Options

The above two issues affect your WF hosting decisions and implementation. As with WCF you can host in any .NET process but in many ways the responsibilities of a WF host are more far reaching than for a WCF host. Essentially a WCF host has to start listening on the endpoints for the services it is hosting. A WF host not only has to configure and start the workflow runtime but also decide when new workflow instances should be created and broker communication from the outside world into the workflow. So lets look at the various options.

WF Console Hosting

When you install the Visual Studio 2005 Extensions for Windows Workflow Foundation package you get new project types which include console projects for both sequential and state machine workflows. These project types are useful for doing research and demos but thats about it.

WF Smart Client Hosting

There are a number of WF features that make it desirable to host WF in a smart client. Among these are the ability to control complex page navigation and the WF rules engine for controlling complex cross field navigation. The Acropolis team are planning to provide WF based navigation in their framework.

WF ASP.NET Hosting

One of the common use cases of WF hosting is controlling page flow in ASP.NET applications. This isn't by any means the sum total of application of WF in ASP.NET; for example, the website may be the gateway into starting and monitoring long running business processes. There are two main issues with ASP.NET hosting:

  1. ASP.NET runs executes on thread pool worker threads as does the DefaultWorkflowSchedulerService. ASP.NET also stores the HttpContext in thread local storage. So if the workflow invokes functionality that requires access to HttpContext or the ASP.NET request needs to wait for the workflow then you should use the ManualWorkflowSchedulerService as the scheduler. This also means that the workflow runtime and ASP.Net will not be competing for threads.
  2. For long running workflows the workflow will probably be persisted and so the persistence service needs to recognise when timeouts have occurred. However, to do this the workflow runtime needs to be executing. So why is this an issue? Well in ASP.NET hosting the lifetime of the workflow runtime is generally tied to that of the Application object and therefore to the lifetime of the hosting AppDomain. But ASP.NET does AppDomain and worker process recycling - the AppDomain not being recreated until the next request. Therefore, if no requests come in (maybe the site has bursts of activity then periods of inactivity) the AppDomain may be recycled and the workflow runtime stop executing. At this point the persistence service can no longer recognise when timers have expired.

As long as your workflows do not require persistence or you have (or can create via a sentinel process) a regular stream of requests then ASP.NET is an excellent workflow host - especially if IIS is being used to host WCF endpoints that hand off to workflows. just remember it is not suitable for all applications.

WF Windows Service Hosting

A Windows Service is the most flexible host for WF as the workflow runtime lifetime is generally tied to the lifetime of the process. Obviously, you will probably have to write a communication layer to allow the workflows to be contacted by the outside world. In places where ASP.NET isn't a suitable host it can delegate to a Windows Service based host. Just remember there is an overhead to hand off to the windows service from another process (say via WCF using the netNamedPipeBinding) so make sure ASP.NET really isn't suitable before you go down this path.

WF is a powerful tool for many applications and the host plays a crucial role in providing the right environment for WF execution. Hopefully this article has given some insight into use cases and issues with the various hosting options.

 

.NET | WCF | WF
Thursday, 07 June 2007 13:39:04 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 

I have just uploaded the code from mine and Jon's precon on WCF and WF. You can download it here.

.NET | WCF | WF
Thursday, 07 June 2007 04:26:21 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 
# Wednesday, 06 June 2007

WCF and WF are both frameworks. Specifically this means they are not full blown products but rather blocks of utility code with which to build products. For both of these frameworks the hosting process is undefined and it is up to you to provide this. As .NET based frameworks the host can be any .NET process so what are the advantages and disadvantages of each? In this the first of two articles I'll look at options and issues hosting WCF then in the next article I'll do the same for WF.

WCF Console Hosting

Console apps are useful for testing things out and can be useful for mocking up a service if you are testing a client. However, they are not suitable for production systems as they have no resilience associated with them.

WCF Smart Client Hosting

Useful for peer to peer applications.

WCF Windows Service Hosting

This is a tangible host for  production services. Whether this is the best host depends on a number of questions:

  • What operation system am I running on?
  • What network protocols do I need to support?
  • Do I need to execute any code before the first request comes in?

Assuming you want to run this on a server OS and one that is currently released you will have to use Windows Server 2003 (.NET 3.0 isn't supported on Windows 2000 or earlier). If you want to use any protocol other HTTP or you need to execute code before the first request comes in you must use a Windows Service to host your WCF service.

If you are happy to run on Vista or Windows Server 2008 Beta (formally known as Longhorn Server) then you should only use a Windows Service if you need to execute code before the first request arrives.

WCF IIS Hosting

IIS should be your host of choice for WCF services. However, there are two issues to be aware of:

  1. IIS6 and earlier can only host services that use HTTP as a transport. IIS7 introduces the Windows Process Activation Service that allows arbitrary protocols (such as TCP and MSMQ) to have the same activation facilities as HTTP.
  2. IIS provides activation on the first request. If you need to execute code before this (for example if you have a service that records trend data over time for some external data source and your service queries this trend data) then you cannot use IIS as a host without adding out of band infrastructure to bootstrap the application by performing a request.

So the options for WCF hosting are pretty straightforward. In the next article we'll see that the hosting options for WF have some issues that may not at first be obvious.

.NET | WCF | WF
Wednesday, 06 June 2007 22:36:24 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 

Just sitting in on a session on extending the WF rules infrastructure given by Moustafa Ahmed. Some interesting ideas. I especially liked the infrastructure for decoupling the rules from the workflow assembly using a repository, custom workflow service and a custom policy activity. He also showed a custom rules editor that was business user friendly (which the OOB one isn't) called inrule which looked pretty good.

.NET | WF
Wednesday, 06 June 2007 20:05:31 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 

So I'm here at TechEd 2007 enjoying the humidity of Orlando in June. I delivered a Precon on WCF and WF on Sunday with Jon Flanders which was a blast. Mark Smith who writes our WPF course asked me to check out what I could find out about Acropolis - a new framework for creating WPF applications. The Acropolis website went live on Monday and so I wandered over to the boothe to see what I could find out. It came as quite a surprise that Dave Hill, who i used to work with in the UK about 8 years ago, was on the stall. Dave is the architect on Acropolis - its a small world. I had lunch with Dave and it looks like they have some interesting ideas for helping to enforce separation between the presentation layer and business funtionality in an application and it is all expressed declaratively in XAML. If you're building smart clients in WPF its definitely worth keeping an eye on.

.NET | WPF
Wednesday, 06 June 2007 16:15:23 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   | 
# Tuesday, 05 June 2007

After suffering the pain of using BlogX as a blog engine for some time, I have finally moved my blog on to a current engine - namely dasBlog. Hopefully a more accessible engine will get me blogging more regularly

Tuesday, 05 June 2007 15:53:35 (GMT Daylight Time, UTC+01:00)  #    Disclaimer  |   |