Workflow Management Service (WMS) - High Memory Usage

The WorkflowManagementService.exe process is using a lot of memory until you get the OutOfMemoryException error when you are hosting your application. Looking at this situation, you may be suspecting the process is having memory leak problem. Restarting the AppFabric Workflow Management Service (WMS) windows service can temporarily free up the memory, however, the WMS process memory usage will still keep growing gradually until your server is out of memory again.

How to troubleshoot?

Open up Event Viewer, go to the Applications and Services Logs, expand the Microsoft folder, then expand Application Server-System Services folder, and look into the Admin log. You should see a lot of error which are related to Workflow Management Service. Let’s take a look at the logs with the source of Application Server-System Services Workflow Management Service only.

Following are the errors which you would see from the log.

Sample Error 1:

Failed to invoke service management endpoint at 'net.pipe://<server name>//ServiceManagement.svc' to activate service '/<service name>.svc'.\rException: 'The message with To 'net.pipe://<server name>//ServiceManagement.svc' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher.  Check that the sender and receiver's EndpointAddresses agree.'

Sample Error 2:

Could not find net.pipe base address for site name 'Default Web Site' application name '/<your application name>. Check that the site and application have net.pipe enabled, and that the application still exists.

Sample Error 3:

Throwing an exception. Exception Could not locate binding information for the site Default Web Site and application /<your application name>.

Steps to rectify the problem:

WMS is trying to invoke the ServiceManagement service endpoint. Check whether the WMS is invoking a valid service endpoint. Open up IIS, go to the Website level, and check whether you have net.pipe binding information created?

If yes, go to the Application level, check whether you have net.pipe protocol enabled.

If problem persist, filter the event log, look for the error similar to Sample Error 3. The website name and application name mentioned in the log most likely does not exist in IIS. When WMS failed to activate the service due to the missing application in IIS, it keeps retrying and logging the error to the event log. And, because of this never ending retrying and logging, it flooded the event logs every few seconds, and took away some processing power and memory.

Base on the AppFabric system architecture, WMS actually loads the service configuration base on the data from the AppFabric Persistence Store database. More Info

When you have a new service deployed to the IIS, WMS will register your service into the AppFabric persistence store, in the System.Activities.DurableInstancing.ServiceDeploymentsTable. Querying this table will show you all the services which are under WMS monitoring. Identify and confirm the services in this table are all exist and hosted in your IIS.

By right, you should have one application server, one AppFabric service and one instance store. If you come to this post, that's mean you had setup the AppFabric architecture wrongly like me. I suppose you have multiple application servers, multiple AppFabric services, and they are sharing one instance store. If you are referring to the architecture from the MSDN blog, and questioning why this architecture setup is wrong, may be this diagram below confuse us?

I suppose the diagram is meant for load balanced environment, one same application in multiple servers and one database and one persistence store.

Anyway, back to the problem. In order to fix this up, we have to have one application server, one AppFabric service and one persistence store. And, following is the steps of how to clean up the data in the persistence store.

Identify all the unwanted services from the ServiceDeploymentTable, mark down the services ID first. We have to clean up all the instances related to the service before deleting the service. Perform the following query to check your instances.

SELECT *

FROM [System.Activities.DurableInstancing].[InstancesTable] t1

LEFT JOIN [System.Activities.DurableInstancing].[ServiceDeploymentsTable] t2

ON t1.ServiceDeploymentId = t2.Id

WHERE t2.Id IN (<your sIds>)

Then, delete all the instances.

DELETE

FROM [System.Activities.DurableInstancing].[InstancesTable]

WHERE ServiceDeploymentId IN (<your sIds>)

Then, delete the services.

DELETE

FROM [System.Activities.DurableInstancing].[ServiceDeploymentsTable]

WHERE Id IN (<your sIds>)

Finally, restart the WMS windows service. Then, monitor your Event Viewer to see any error still occur.

Credits:

Images and information source from MSDN: http://blogs.iis.net/appfabric/archive/2010/03/24/windows-server-appfabric-architecture.aspx