It's been awhile!

Yes... I know, I haven't blogged to my loyal following of 2 readers but times have been busy.

Busy doing what? Well, on December 5, 2008 my wife and I welcomed our two sons into the world at 12:01, and 12:02pm. Evan Christopher Taylor and Ethan Bradley Taylor, and yes, they are identical.

To anyone that has had twins, or any kind of multiples for that matter, you know that first few months there isn't time for anything, much less blogging. As part of having twins and the new found responsibilities of parenting, I've been retraining myself in the way I work?

Why would you do that?

Well, back in the day I used to be able to have 3-4 hours of non-stop quiet to complete my coding tasks. Now, that time is shorted to anywhere from 15-45 minutes without distraction, so my ability to stare at code and trace deeply into a process is much more difficult.

As a result I've learned to think about code constantly. Of course, I do that anyways, but instead of daydreaming of perfect transcendent systems with 4GLs, I think about the method I need to write for a problem. Taking things at a very granular level, and planning it all in my head before I ever execute.

I'm sure this is the way we are "supposed" to do software development, but thats not nearly as fun as attaching a debugger, clicking something and following it down the path of where it goes. Alas no more.

Now that the boys are 4 months, I seem to have more time on my hands to actually write, perhaps I'll actually get to do it now!

Based on some new happenings at work, I'll be shifting my focus to ASP.NET MVC + Team Foundation Server (and a little bit of workflow, but not as much, at least not until .NET4).

Deploying Workflow Foundation Part 1.5 - Thread Safety and ExternalDataExchange

*Sigh* another late night debugging a CI build that has been failing all day. And just as I fix the portion I know is wrong, I find that another delivery not related to anything I do broke another part of the build. And, it's going to take some digging to find the problem.

So, before I get into that beast, I wanted to at least share a note on Thread Safety and using ExternalDataExchange interfaces.

The first important thing to note is ExternalDataExchange services live as singletons in the workflow runtime. However, unlike our powerful new toy WCF, you are not guarunteed a singleton with proper thread blocking.

What does that mean?

If you haven't heard of Free Threading, that's pretty much what this is. Think about when you add an ExternalDataExchange service to the workflow runtime. When you do it in code, it would look something to the effect of.

using (WorkflowRuntime runtime = new WorkflowRuntime("WorkflowRuntimeConfigurationPT"))
{

  ExternalDataExchangeService dataExchangeService = new ExternalDataExchangeService();
  MyWorkflowService service1 = new MyWorkflowService();
  runtime.addService(dataExchangeService)
  dataExchangeService.AddService(service1);

}

The important thing to note above is the construction of our Workflow service that we will use to perform whatever function we wish. From our previous example it could be to update the database with information from the workflow state about an order, such as adding or removing an item.

We construct an instance of that service and let the runtime and dataexchange take that over. But keep in mind, we constructed the instance (instead of referencing a type in which case the runtime would create instances as needed).

Often times we find ourselves being a bit lazy and not thinking about what exactly is going to be accessing instances of our classes. Especially when we are not necessarily thinking about multi-threaded programming (I think WCF spoiled this for us). After all, the application that I wrote doesn't use the ThreadPool or anything else touching System.Threading.

But Workflow does.

What can happen is two workflows may reach a state in which they call the same method on the same interface. Because you can only have only 1 instance of any interface loaded into a single runtime this can cause both threads from each workflow instance to "cross paths". When debugging, some may notice this as "code jumping", where your running F10, and then all of a sudden your 3 lines before you were and not paying attention to your thread monitor window. ;)

What can happen is the same thing in any free threaded application. Always remember the basics of computers, a processor can only process a single instruction at a time. And a line of code does not necessarily equal a single instruction (no where close ;)) so, the Thread Scheduler deep inside the OS will swap out different sequences of instructions from different processes at different times, not necessarily completing a logical set of instructions as we humans see it.

Sometimes this doesn't matter, however in most instances you may be using a class level or even method level variable, in which case if both threads are accessing the method at the same time, thread A sets a variable to 1 and thread B sets it to 2. When thread A gets scheduled its next instruction, values that it originally had may not be there, leading to "weird" or "random" results and or exceptions. Also, the most common example is that both thread A and B create a race condition where neither can exit a loop because of what the other one is doing.

I will not go into a huge speech of the importance of thread safety and how to do it, there are much better examples out there than what I will give you. The key to take from this post is always think when developing your workflow services, "two workflow may access this code at the exact same time, what have I done to protect myself". The easy thing to do is use of the lock statement. This will save you hours of headache later.

HTH

Deploying Workflow Foundation Part 1 - Introduction and Solution Setup

It's been a long time since I've gotten a chance to write. I certainly had a number of topics to discuss, just no time to sit down and put them into words. Well, words that made sense.

I've decided to write about a topic that I thought needed some attention. I've seen a number of posts of people struggling with deploying and upgrading Workflow Foundation as well as dealing with the same issue in my professional life. Before that though...

Over the past 8 weeks, my life has taken considerable changes both personally and professionally. No, I didn't leave my job, however, if you remember in previous posts, I work for a large financial institution, and my first few months were really nothing documentation. In my opinion not necessarily the most useful documentation, but I digress. That all changed about 12 weeks ago with me being the onshore lead for an offshore project at work.

Back to that in a moment...

I wouldn't normally share something as personal on a technical blog, but for this, a small snippet is acceptable. Just over a year ago my wife and I lost our son Ryan at birth (he was 38 weeks). Of course, a tragic time and difficult for both of us. However in April we found out my wife was pregnant again. We were incredibly excited and as we made it past the first trimester we were anxiously awaiting the day to find out the gender. Of course, like any parents we started shopping for names, we quickly selected a girls name without any question (just as we had selected Ryan's). But we were stuck between two boys names. Well, that day came and looks like there was a reason we couldn't decide... We need 'em both. ;)

Back to our regularly scheduled programming...

A little background on this project. We had 4 workflows, each being very simliar in soliciting some responses from various stakeholders in an operation and modifying data accordingly. Nothing too crazy. Now I obviously cannot share those workflows, so I've created a beginning to end + iterations example of using Windows Workflow in *any* environment which I will be sharing over the next few posts.

Ok let us take a process we all know. An online shopping cart system. To start off, I've created the most simple cart system that I can. It contains no data other than an ID (and that Id will be provided by the workflow engine). Below is what my state machine looks like.

The workflow has 3 states. ShoppingCartState, OrderPlacedState, and OrderShippedState. The workflow responds to events from ExternalDataExchange decorated interfaces.

But before we get into all that. I want to talk about the solution layout and a very important thing about workflow that is overlooked. That my friends, is versioning, and how to properly version Workflow.

Microsoft's Provided Workflow Templates for VS2005 & VS2008

I'm a huge Microsoft fan, so this isn't a complaint, just a clarification. The Workflow project templates that are included with the Workflow Extensions for VS2005 and VS2008 are not meant to be used in an almost *any* production environment other than one where either

• The workflow never changes
• If upgraded, all existing data is either destroyed or migrated (sometimes always painfully).

This doesn't sound like a very good situation in a large application or small. Truthfully, I've found that following the BizTalk solution layout seems to be the best way to easily change and upgrade workflows. And what exactly is that?

• Interfaces Project - Version # should never change (next post I will cover how it can). This represents interfaces Workflow uses to communicate with External Services using the ExternalDataExchangeService.
• NOTE: Your interfaces project should also include enum's and inherited ExternalDataEventArgs. If you don't, you will need to verify and possibly modify
  
• Workflow Project - Version # should change on nearly every deployment
• Services Implementation Project - Version # should change on an as needed basis.
  

Or, a picture is worth 1000 words.



I have three projects of interest represented here. Now I have actually broken one of my own rules in which case I have joined the services implementation with my Workflow project. Though it isn't necessarily a best practice, the version # on the services is not nearly as important as the interface version # for this example. So we can cheat a little in this example.

Also notice I've added a Strong Name key to my Workflow Project as well as the interface project. I have to add it to the interface project because it is referenced by the workflow project. The workflow project is strong named because I'm going to install it into the GAC. I could install the interface library in the GAC as well.

Why should the interface version never change?

This seems to be the biggest "gotcha" when developing Workflow enabled applications. It isn't a bug, but something very much meant by design. Under the hood WF has it's own runtime that handles events through it's own messaging infrastructure. If you look deep enough, you'll find the eventing model uses something similar to MSMQ to deliver an event message to a specified workflow. It may actually be MSMQ, but I cannot say for sure at this point. This way, the runtime handles the message delivery of an event without us having to worry about if the workflow is in the proper state to receive the event. This post will not go into the details of guarunteed asynchronous message delivery, but it's a fun topic if you have the time.


Why install the Workflow DLL in the GAC?

Simple, in most cases (even something like this shopping cart) we will have long running transactions. The process of getting the order into our warehouse may be fast, however, if we are shipping a physical item there may be a few days between shipping and updating it. Workflow offers persistence services for keeping these transactions alive through reboots, shutdowns, and whatever else may happen.

Lets say we add a new step to the process, such as AtVendorState where we ship part of our order from a 3rd party vendor. When we deploy this, we still want our existing workflow to finish running the way they started, that way we don't have to come up with some sort of upgrade program or a completly separate deployment of the services and workflows.

In order to allow both the old workflows and new workflows run in the same runtime with independent versions, we must install both workflow assemblies in the GAC and allow the runtime to determine which one it wants at any particular time.

We all admit, the workflow's have to change, not only that, but long running processes subscribe to one verison of the workflow and are intended to finish that way. After all, the runtime cannot determine how to properly transition version 1.0 of your workflow to version 1.1.

But you know what doesn't change? Interfaces. Specifically I'm talking about the interfaces that are used to raise events into the workflow instance, and for the workflow instance to contact external systems (I know you can use WCF with 3.5, but we are only allowed to use up to 3.0 at work, and thankful for that). This was where I found the most "problems" when deploying workflow.

However, it wasn't as if the deployment was hard, it was realizing the exceptions we were getting even though we never changed anything but a version were because of strong rules in place and not problems other than that which the code is delivered.

Workflow is a lot like BizTalk, in the way that it is *very* version dependent. However, BizTalk provides some niceties for binding between versions of send ports, receive ports and orchastrations. If you want that in workflow, you are doing it yourself.

Look forward to part 2 where I talk more about ExternalDataExchange services and Thread Safety. This is a huge topic and very easy to glaze over, but will start causing you more problems as the # of workflows (especially long running) continues to grow.

YakShaver.NET = UCM+ for TFS

I've been quiet recently. Mainly because I've been flipping my house. No, not to sell, but because I want to enjoy it that way. We moved in just about 2 years ago, and never really touched a thing. After August I had little motivation until recently, in which case I had more than I could imagine (perhaps quitting smoking helped).

That isn't what this entry is about however (I'll post pics at a later time). Well sort of. My reason for taking the break was both because of the house and because I wasn't exactly sure what I wanted of YakShaver.

I should rephrase that, I knew what I wanted, however I didn't really have a method to my madness. It was going in a number of different directions with no solid foundation to any of it. Or for that matter, what was my starting point?

At work, we use Rational ClearCase UCM (Unified Change Management). This is slightly different than Base ClearCase (which is really just a glorified version of SubVersion from what some have told me) in the sense that it provides an overlay to the SCM tool. Unfortunately at work, our ClearCase admins do not necessarily explain all the intricacies of what UCM and how its terminology applies to more commonly known terms (such as Stream == Branch, View == Workspace, Deliver == Copy + Merge).

Because there is very little explanation, most people seem to have a bad taste in their mouth about CC UCM, when in fact, it's rather brilliant. ClearCase is still slow and kludgy and I'm more of a fan of TFS with the direct work item integration (rather than Rational's other products such as RequisitePro and ClearQuest which integrate into CC+UCM).

Brilliant? IBM product? Isn't this blasphemy from a .NET programmer and MS lover? C'mon, we're all coders, whether that is Java or .NET, Ruby or Python. IBM has good idea, MS has good ideas, Avanade comes up with good ideas and gives them to MS, the cycle goes on and on. But I digress.

So what exactly is the 50,000 foot view of UCM? Let's look at what I consider to be the fundamental part, which is UCM's branching strategy. Again, UCM isn't anything special other than the fact it does some tasks for you. So when you create a Stream, your creating a branch that can be isolated for the developer, they can check in their work without affecting anyone elses code.

We are all *supposed* to follow a branching strategy, however many times we get lazy and forget, UCM kind of prevents this, creating your private development stream and those changes get merged into an Integration Stream (aka Trunk. though I should be careful, it doesn't necessarily mean trunk, especially if you have feature integration streams).

When the developer has completed their activities (another term in UCM world, aka Work Items) and all works well, they deliver this to the integration stream, where all developers deliver work and do merging between elements to get a finalized product.

The nice thing about Rational is they have a decent Merge/Diff tool that can handle most merging on it's own. TFS offers the same feature, though I haven't truly seen how well its merging capabilities are. My previous experience on TFS was in a 3 person team that rarely every collided. My current environment has 75+ developers, all doing their own thing and colliding quite often, UCM helps filter some of that.

Now that I've said all that, to the point of the blog. As I started to understand UCM more and more (especially during a CC upgrade recently) I realized this was exactly what I wanted out of YakShaver.NET. A UCM like tool (that provided automatic branching strategy and merging, while taking advantage of other TFS tools), but provides a bit more. This is where the idea of "Sandboxing" comes into play.

If I could take an integration stream/branch and deploy that version to a sandbox environment (isolated much like my development environment is) then I can accomplish the goal of working with Testers/QA/Clients in an isolated environment and keeping congruency between all of it using a friendly application. This is where the UCM+ comes in (UCM plus a little more).

I'm not sure if I can use this name or not, I'll have to find out, but it is the goal to turn YakShaver.NET into a UCM type front end for TFS.

I'm glad I can finally put that in a single sentence. Now that I know what I want to build, time to actually do it. ;)

Another Awesome Tool for Remote Development

I happened to catch this little gem inside Visual Studio at work. I've been reading more by Scott Hanselman recently. He had a great link to this wicked little tool called Microsoft SharedView.

http://connect.microsoft.com/site/sitehome.aspx?SiteID=94

Wow... Be anywhere the world and get a code review. Sick...

CC.NET + MS Test 2008 + New Template

In a bit of YakShaving today I found the need to modify the MSTestSummary.xsl file for CC.NET so that it worked properly with the output from mstest.exe 2008. I haven't seen if 2005 uses the same format or not.

Anyways, hope some find this useful. You can download it from my CodePlex site here. I'll see if ThoughtWorks wants to put it up on their site.

Interesting Distinction Between BizTalk 2006 and Workflow

I've been working with Workflow Foundation quite a bit lately and even had the opportunity to be teaching it to my design team at work about it's intricacies (which has been a lot of fun as well).

Coming from a BizTalk background also makes it exciting to see the differences between it and WF and how they are similar. Something neat about BizTalk is its use of Send Ports and Send Groups, which allows multiple services to be informed of an event within an orchestration (the equivalent to a sequential workflow in WF). With a send port, I could theoretically have multiple instances of the same type of service with small differences between the two and use correlation to deal with events back from multiple data sources.

However in WF, it isn't quite the same. Given everything is interface driven that is what the WF Runtime is looking at, so if I try to add multiple services of the same interface to the DataExchangeService an exception is thrown with the following message

"An instance of ExternalDataExchangeService of type TestWorkflowApp.IApproverService already exists in the runtime container."

The code that I used to do this is pretty simple. I created another class (although it would appear I could just instantiate another instance of my ApproverService class) which was the exact same implementation (just a different name). Then tried to add both to the Data Exchange Service as so

dataExchangeService.AddService(approverService);
dataExchangeService.AddService(secondApproverService);

Exception thrown.

I suppose there should be some major distinctions in capability between a free product that is part of the framework like Workflow vs. a 25k+ product like BizTalk. Certainly BizTalk offers much more than a workflow runtime (far beyond the scope of this blog) but thought it was important to point out some differences.