The WorkflowManagementService.exe process is using a lot of
memory until you get the OutOfMemoryException
error when you are hosting your application. Looking at this situation, you may
be suspecting the process is having memory leak problem. Restarting the AppFabric
Workflow Management Service (WMS) windows service can temporarily free up the
memory, however, the WMS process memory usage will still keep growing gradually
until your server is out of memory again.
How to troubleshoot?
Open up Event Viewer, go to the Applications and Services
Logs, expand the Microsoft folder, then expand Application Server-System
Services folder, and look into the Admin log. You should see a lot of error which
are related to Workflow Management Service. Let’s take a look at the logs with the source
of Application Server-System Services
Workflow Management Service only.
Following are the errors which you would see from the log.
Sample Error 1:
Failed to invoke service management endpoint at 'net.pipe://<server
name>//ServiceManagement.svc' to activate service '/<service name>.svc'.\rException:
'The message with To 'net.pipe://<server name>//ServiceManagement.svc'
cannot be processed at the receiver, due to an AddressFilter mismatch at the
EndpointDispatcher. Check that the sender and receiver's EndpointAddresses
agree.'
Sample Error 2:
Could not find net.pipe base address for site name
'Default Web Site' application name '/<your application name>. Check that
the site and application have net.pipe enabled, and that the application still
exists.
Sample Error 3:
Throwing an exception. Exception Could not locate
binding information for the site Default Web Site and application /<your application
name>.
Steps to rectify the problem:
WMS is trying to invoke the ServiceManagement service
endpoint. Check whether the WMS is invoking a valid service endpoint. Open up
IIS, go to the Website level, and
check whether you have net.pipe
binding information created?
If yes, go to the Application
level, check whether you have net.pipe
protocol enabled.
If problem persist, filter the event log, look for the error similar to Sample Error 3. The website name and application name mentioned in the log
most likely does not exist in IIS. When WMS failed to activate the service due to the missing application in IIS, it keeps retrying and logging the error to the event log. And, because of this never ending retrying and logging, it flooded the event logs every few seconds, and took away some processing power and memory.
Base on the AppFabric system architecture, WMS actually loads the service configuration base on the data from the AppFabric Persistence Store database. More Info
When you have a new service deployed to the IIS, WMS will register your service into the AppFabric persistence store, in the System.Activities.DurableInstancing.ServiceDeploymentsTable. Querying this table will show you all the services which are under WMS monitoring. Identify and confirm the services in this table are all exist and hosted in your IIS.
By right, you should have one application server, one AppFabric service and one instance store. If you come to this post, that's mean you had setup the AppFabric architecture wrongly like me. I suppose you have multiple application servers, multiple AppFabric services, and they are sharing one instance store. If you are referring to the architecture from the MSDN blog, and questioning why this architecture setup is wrong, may be this diagram below confuse us?
I suppose the diagram is meant for load balanced environment, one same application in multiple servers and one database and one persistence store.
Anyway, back to the problem. In order to fix this up, we have to have one application server, one AppFabric service and one persistence store. And, following is the steps of how to clean up the data in the persistence store.
Identify all the unwanted services from the ServiceDeploymentTable, mark down the services ID first. We have to clean up all the instances related to the service before deleting the service. Perform the following query to check your instances.
SELECT *
FROM [System.Activities.DurableInstancing].[InstancesTable] t1
LEFT JOIN
[System.Activities.DurableInstancing].[ServiceDeploymentsTable]
t2
ON t1.ServiceDeploymentId = t2.Id
WHERE t2.Id IN (<your sIds>)
Then, delete all the instances.
DELETE
FROM [System.Activities.DurableInstancing].[InstancesTable]
WHERE ServiceDeploymentId IN (<your sIds>)
Then, delete the services.
DELETE
FROM [System.Activities.DurableInstancing].[ServiceDeploymentsTable]
WHERE Id
IN (<your sIds>)
Finally, restart the WMS windows service. Then, monitor your Event Viewer to see any error still occur.
Credits:
Images and information source from MSDN: http://blogs.iis.net/appfabric/archive/2010/03/24/windows-server-appfabric-architecture.aspx
