Workflow Delay Activity Does Not Continue

One of my colleague came to me and showed me that her idle workflow instance which is currently waiting for delay activity to time up to continue the work, is actually not working. The workflow instance is not activated and continue the process automatically. I am here to share my troubleshooting experience.

It is a workflow service hosted in IIS 7.5, and involving AppFabric 1.1. Here is the story, the following is a sample how the workflow design look like:

The workflow allows user to supply a future datetime value to the service, and then the workflow instance will sleep until the time has come and then wake up and continue the process. So, as you see above, the Receive activity will accept the StartDateTime parameter, and then the Assign activity there will calculate the time span for how long the instance need to delay. After the Delay activity, will do a WriteLine activity.

The problem is the WriteLine activity never fire, the instance is still stuck at Delay activity. If you happen to encounter the similar problem now, you may want to check whether there is a missing net.pipe protocol enabled at IIS application level. The reason is workflow instance normally will turn to Idle state after some time, then the AppFabric Workflow Management Service (WMS) rely on net.pipe to activate workflow instances when the time comes.

Check at Web Site level:

Check at Web Application level:

Next, you may want to check your AppFabric persistence database. From the web server, is it connectable to the persistence database. Make sure you are able to ping the database IP address or hostname, and also confirm the firewall clearance. Note: SQL Server default TCP port is 1433.

When everything are confirmed correct, check the server time in web server and database server. My actual root cause is the server time are not the same in between web and database servers. The database server had 3 minutes time later than web server.

If you execute the following query from the AppFabric persistence database:

SELECT * FROM [System.Activities.DurableInstancing].[InstancesTable]

You will notice that your workflow instance has the PendingTimer column value is a past date compare to the web server date. Note: PendingTimer is in UTC time, you need to offset the hours manually base on your location. It is the time when your workflow instance should be re-activated by WMS. With the date time value discrepancy between the web and database server, I guess WMS ignore the instance activation since it is a past date.

Web Server time: 2014-04-10 21:00:00
Database Server time: 2014-04-10 20:57:00

The problem was solved after making both servers time in sync.

If you wonder what can still be done to those existing idle instances, what can you do to "wake" it up? The workaround is to suspend those idle instances, then resume them by using the AppFabric Dashboard in IIS.

AppFabric Caching

I have been given a task to research and find out any replacement for ASP.NET Cache. And, whether the cache can be used in any other application type besides web. I have done some research and comparison in AppFabric Caching, Enterprise Library Caching Application Block and System.Runtime.Caching. Today topic cover about AppFabric Caching, next post will cover about System.Runtime.Caching.

Scenario

Normally we are puzzled with which caching technology should we use and what are the differences between AppFabric cache compare to the other cache technology.

The caching feature that AppFabric provided is distributed cache, which mean it makes the cache available and accessible in one or many servers but they are unified into one entity for your application. When you are using ASP.NET cache or any caching facility, you notice that the cache is only available to your own application domain.

Image credit &source from : MSDN

Features

Comparing AppFabric cache vs ASP.NET cache, here are the differences and special features.

1. The cache can be distributed hosting in one or multiple server.
2. The cache can be hosted in a different standalone server, separate from Application server.
3. The cache cluster promote high availability. If one of the cache server is down for some reason, the cache will still be available.
4. The cache automatically sync data between cache hosts.

Configuration

After you have installed and start configuring the AppFabric, you would come across the following window in the wizard.

There are 2 Caching Service configuration provider, one is SQL Server and another is XML. If you wonder which provider to pick, it is depending on your environment. For my server, it is not joined with Active Directory (AD). Therefore, XML is my only option. If you forcefully pick SQL Server, you would get the following notification:

The provider difference is just the location of keeping cache cluster configuration. All the information that you enter in the "Next" page will be stored in SQL Server or a XML file.

Since I am using XML configuration provider, I have to create a new workgroup account to run the Caching Service. You are not allowed to use built-in account such as NT AUTHORITY\NETWORK SERVICE. The same workgroup account needs to be created in the other cache server, and applying same configuration.

Now, you need to pick one of your cache server to be a lead host, then create a network shared folder in that server. The network shared folder must be accessible by other hosts and giving the Cache Service account read/write privilege.

Once you are done with it, configure the File Share in AppFabric Configuration Wizard. Then, set the cluster size. If you are configuring the lead host, choose the New Cluster. For other cache host, choose the Join Cluster.

Next, the default TCP ports which will be used by AppFabric Cache Service are shown in the spinner box. Note them down and create firewall rule to unblock those ports. If you find any of these ports conflicting or is being used by any of you applicantion, you can change the port here.

That's it for AppFabric Cache configuration.

Usage

The cache usage is similar to ASP.NET cache. Open up Visual Studio, create any new project, then add the following assemblies.

C:\Program Files\AppFabric 1.1 for Windows Server\Microsoft.ApplicationServer.Caching.Core.dll

C:\Program Files\AppFabric 1.1 for Windows Server\Microsoft.ApplicationServer.Caching.Client.dll

Note: You need to manually browse and locate the above assemblies. If you add the references from the Assemblies Extension page, they are actually coming from Windows Azure SDK.

The following is the error that you will encounter if you added wrong assemblies:

ErrorCode<ERRCA0019>:SubStatus<ES0001>:Check the client version. It should be within the allowed version range on the server. If necessary, upgrade the client to the allowed version.

The following code need to be run once only to initialize the cache for your application:
Code credit to Scott Hanselman.

private static DataCacheFactory _factory = null;

private static DataCache _cache = null;

public static DataCache GetCache()

{

    if (_cache != null)

        return _cache;

    //List cache host server

    //Note: 22233 is the cache port

    List<DataCacheServerEndpoint> servers = new List<DataCacheServerEndpoint>();

    servers.Add(new DataCacheServerEndpoint("MY-SYLVESTER-01", 22233));

    servers.Add(new DataCacheServerEndpoint("MY-SYLVESTER-02", 22233));

    servers.Add(new DataCacheServerEndpoint("MY-SYLVESTER-03", 22233));

    servers.Add(new DataCacheServerEndpoint("MY-SYLVESTER-04", 22233));

    servers.Add(new DataCacheServerEndpoint("MY-SYLVESTER-05", 22233));

    //Set the cache hosts

    DataCacheFactoryConfiguration configuration = new DataCacheFactoryConfiguration();

    configuration.Servers = servers;

    configuration.LocalCacheProperties = new DataCacheLocalCacheProperties();

    //Set this if you want to get notified if any changes to your cache

    configuration.NotificationProperties = new DataCacheNotificationProperties(10000, new System.TimeSpan(0, 0, 3));

    //Default value for queue length: 10000

    //Default value for poll interval: 300 seconds

    //Disable tracing to avoid informational/verbose messages on the web page

    DataCacheClientLogManager.ChangeLogLevel(System.Diagnostics.TraceLevel.Off);

    //Pass configuration settings to cacheFactory constructor

    _factory = new DataCacheFactory(configuration);

    //Get reference to named cache called "default"

    _cache = _factory.GetCache("default");

    return _cache;

}

In order to insert data into the cache, just the following one liner:

public void SetData(string key, object data)

{

    _cache.Put(key, data);

}

In order to get the data from the cache, one liner as follow:

public object GetData(string key)

{

    return _cache.Get(key);

}

There are more complicated DataCache methods supported, check it out from MSDN.

Cache Notification

If you need the AppFabric Cache Service to notify you any changes to the cache, there are a few callback event can be subscribed. Before you can use this feature, you need to enable it during the cache initialization by setting the DataCacheNotificationProperties in DataCacheFactoryConfiguration.

Also, you need to enable notification by using this PowerShell command: 

Set-CacheConfig -CacheName <Cache Name> -NotificationsEnabled true

You also can change the AppFabric Cache configuration XML file in the shared folder.

Then, restart the cache cluster by restarting the AppFabric Caching Service windows service or executing the PowerShell command: Restart-CacheCluster

All the AppFabric Caching PowerShell command can be found HERE.

Now, I want to get notified when someone replace one of the cache item. The following is the code use to subscribe the callback method:

private static DataCache _cache;

       

static AppFabricCacheService ()

{

    _cache = CacheUtil.GetCache();

    _cache.AddItemLevelCallback(

        "myItem",

        DataCacheOperations.ReplaceItem,

        OnDataCacheItemChanged);

}

private static void OnDataCacheItemChanged(string cacheName, string regionName, string key, DataCacheItemVersion version, DataCacheOperations cacheOperation, DataCacheNotificationDescriptor nd)

{

    Debug.WriteLine("Someone changed my cache item!");

    Debug.WriteLine("Cache Name: " + cacheName);

    Debug.WriteLine("Cache Item Key: " + key);

}

If you want to get notified for other action to the cache, you can use different DataCacheOperations filter. More Info

You can use the bitwise OR to add more filter to your callback event subscription. For example, the following code will raise the OnDataCacheItemChanged callback method if someone add or replace or remove your cache item.

_cache.AddItemLevelCallback(

    "myItem",

    DataCacheOperations.AddItem |

    DataCacheOperations.ReplaceItem |

    DataCacheOperations.RemoveItem,

    OnDataCacheItemChanged);

AppFabric cache notification not just support cache item level notification, it also support for:

• Cache level in bulk
• Cache level
• Item level
• Region level

More Info

Note: These callback events are not supported in Windows Azure Shared Caching.

There are a few more things about AppFabric Caching, but it is too much to cover in one topic. The common and important thing that we need to know about caching such as security, concurrency, expiration and eviction, all can be found in this MSDN article.

Summary

The AppFabric caching is awesome, however, IMHO, if you are needing a simple cache that replace ASP.NET cache, AppFabric caching is overkill. Unless, you really need a caching service that support high availability, scalability, this is what you want.

AppFabric caching is meant to be hosted in different server from your application server. Therefore, hardware and software licensing needs to be considered. Otherwise, you can still choose to ignore the recommendation from Microsoft by hosting the AppFabric cache in the same server with your application server. Then, the server hardware resource could become the concern.

I notice that the AppFabric cache memory consumption is rather high compare to other normal caching facility. By default, the AppFabric caching will utilize 50% of your server memory. And, it is going to reserve quite some amount of memory even though you never have that much of data keeping in the cache or not. However, don't worry, the memory reserve can be tuned down by setting the high / low watermark percentage of memory in the cluster host configuration. See Windows Server AppFabric Memory Consumption Behavior for more info.

If you are looking for smaller caching facility and a replacement for ASP.NET cache for non web application, please come back and check out the next coming blog post.

Workflow Management Service (WMS) - High Memory Usage

The WorkflowManagementService.exe process is using a lot of memory until you get the OutOfMemoryException error when you are hosting your application. Looking at this situation, you may be suspecting the process is having memory leak problem. Restarting the AppFabric Workflow Management Service (WMS) windows service can temporarily free up the memory, however, the WMS process memory usage will still keep growing gradually until your server is out of memory again.

How to troubleshoot?

Open up Event Viewer, go to the Applications and Services Logs, expand the Microsoft folder, then expand Application Server-System Services folder, and look into the Admin log. You should see a lot of error which are related to Workflow Management Service. Let’s take a look at the logs with the source of Application Server-System Services Workflow Management Service only.

Following are the errors which you would see from the log.

Sample Error 1:

Failed to invoke service management endpoint at 'net.pipe://<server name>//ServiceManagement.svc' to activate service '/<service name>.svc'.\rException: 'The message with To 'net.pipe://<server name>//ServiceManagement.svc' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher.  Check that the sender and receiver's EndpointAddresses agree.'

Sample Error 2:

Could not find net.pipe base address for site name 'Default Web Site' application name '/<your application name>. Check that the site and application have net.pipe enabled, and that the application still exists.

Sample Error 3:

Throwing an exception. Exception Could not locate binding information for the site Default Web Site and application /<your application name>.

Steps to rectify the problem:

WMS is trying to invoke the ServiceManagement service endpoint. Check whether the WMS is invoking a valid service endpoint. Open up IIS, go to the Website level, and check whether you have net.pipe binding information created?

If yes, go to the Application level, check whether you have net.pipe protocol enabled.

If problem persist, filter the event log, look for the error similar to Sample Error 3. The website name and application name mentioned in the log most likely does not exist in IIS. When WMS failed to activate the service due to the missing application in IIS, it keeps retrying and logging the error to the event log. And, because of this never ending retrying and logging, it flooded the event logs every few seconds, and took away some processing power and memory.

Base on the AppFabric system architecture, WMS actually loads the service configuration base on the data from the AppFabric Persistence Store database. More Info

When you have a new service deployed to the IIS, WMS will register your service into the AppFabric persistence store, in the System.Activities.DurableInstancing.ServiceDeploymentsTable. Querying this table will show you all the services which are under WMS monitoring. Identify and confirm the services in this table are all exist and hosted in your IIS.

By right, you should have one application server, one AppFabric service and one instance store. If you come to this post, that's mean you had setup the AppFabric architecture wrongly like me. I suppose you have multiple application servers, multiple AppFabric services, and they are sharing one instance store. If you are referring to the architecture from the MSDN blog, and questioning why this architecture setup is wrong, may be this diagram below confuse us?

I suppose the diagram is meant for load balanced environment, one same application in multiple servers and one database and one persistence store.

Anyway, back to the problem. In order to fix this up, we have to have one application server, one AppFabric service and one persistence store. And, following is the steps of how to clean up the data in the persistence store.

Identify all the unwanted services from the ServiceDeploymentTable, mark down the services ID first. We have to clean up all the instances related to the service before deleting the service. Perform the following query to check your instances.

SELECT *

FROM [System.Activities.DurableInstancing].[InstancesTable] t1

LEFT JOIN [System.Activities.DurableInstancing].[ServiceDeploymentsTable] t2

ON t1.ServiceDeploymentId = t2.Id

WHERE t2.Id IN (<your sIds>)

Then, delete all the instances.

DELETE

FROM [System.Activities.DurableInstancing].[InstancesTable]

WHERE ServiceDeploymentId IN (<your sIds>)

Then, delete the services.

DELETE

FROM [System.Activities.DurableInstancing].[ServiceDeploymentsTable]

WHERE Id IN (<your sIds>)

Finally, restart the WMS windows service. Then, monitor your Event Viewer to see any error still occur.

Credits:

Images and information source from MSDN: http://blogs.iis.net/appfabric/archive/2010/03/24/windows-server-appfabric-architecture.aspx