Monday, December 16, 2013

File System Watcher Integrate with Windows Workflow Foundation

Today I want to share how to integrate file system watcher in workflow. There are a few ways to do it. Option 1 (simple way) is whenever there is a file is dropped into a folder, the file system watcher will kick in and spawn a workflow instance to process your file. Option 2 (hard way) is to integrate file system watcher into the workflow which mean you need just 1 workflow instance to wait for file watcher event fire then process the file. I want to share the hard way.

Concept

Here is the flow of the concept. Picture is worth a thousand words.


Challenges

In order to implement the above flow, I need a custom workflow activity to handle the logic. I chose to use NativeActivity is because I need to use the bookmarking feature. Bookmarking can induce idle, I want to keep my workflow instance idle state whenever there is no file exists in the folder.

Next challenge is whenever an instance has gone idle, the activity context will be different when the file system watcher event kick in to wake the instance to resume the process. Therefore, I need a custom workflow instance extension to handle the bookmark resume by implementing IWorkflowInstanceExtension.

Implementation

First, prepare the custom workflow instance extension which can support the resume bookmark functionality.

public class FileReceiveExtension : IWorkflowInstanceExtension
{
    private WorkflowInstanceProxy _proxy;

    public IEnumerable<object> GetAdditionalExtensions()
    {
        return null;
    }

    public void SetInstance(WorkflowInstanceProxy instance)
    {
        _proxy = instance;
    }

    public void ResumeBookmark(Bookmark bookmark, FileInfo file)
    {
        IAsyncResult result = _proxy.BeginResumeBookmark(bookmark, file,
            (asyncResult) =>
            {
                    
            }, null);
    }
}

Next, create a custom NativeActivity then add the custom workflow instance extension into the activity cache metadata.

protected override void CacheMetadata(NativeActivityMetadata metadata)
{
    metadata.AddDefaultExtensionProvider<FileReceiveExtension>(() => new FileReceiveExtension());
    base.CacheMetadata(metadata);
}

Set the CanInduceIdle property value to true.

protected override bool CanInduceIdle
{
    get
    {
        return true;
    }
}


Write the custom activity implementation to create file system watcher and register the necessary event handler.

protected override void Execute(NativeActivityContext context)
{
    //Get the folder path from the activity parameter
    string monitoringPath = context.GetValue(this.MonitoringPath);

    FileSystemWatcher watcher = new FileSystemWatcher();
    watcher.Path = monitoringPath;
    watcher.NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite;
    watcher.Created += watcher_Created;
    watcher.Error += watcher_Error;
    watcher.EnableRaisingEvents = true;

    _extension = context.GetExtension<FileReceiveExtension>();
    _bookmark = context.CreateBookmark(BookmarkResumed);
}

Write the implementation of file system watcher created event.

private void watcher_Created(object sender, FileSystemEventArgs e)
{
    //Resume the bookmark and pass the FileInfo object as activity parameter
    _extension.ResumeBookmark(_bookmark, new FileInfo(e.FullPath));
}

Write the implementation of bookmark resume event.

private void BookmarkResumed(NativeActivityContext context, Bookmark bookmark, object value)
{
    FileInfo file = (FileInfo)value;

    //Call your own file processor method
    //Note: I am using async for better performance (Optional)
    Task task = _fileProcessor.ProcessAsync(file);

    task.ContinueWith((e) =>
    {
        if (e.IsFaulted)
        {
            Console.WriteLine(e.Exception);
        }
    });

    //Create a new bookmark immediately after a file is being processed
    _bookmark = context.CreateBookmark(BookmarkResumed);
}

Draw the workflow with the new custom activity. The Start and Stop receive activities are used to control my file system watcher process by making service call.



Start hosting the workflow service, then call the Start web method to test it out. You only need to call the method once to spawn one workflow instance only. Since I had created the custom file receive activity that accept monitoring path, you can spawn another workflow instance to have another instance of file system watcher which monitor different file path.


If you are interested with the source code, feel free to download it from HERE.



Saturday, December 7, 2013

How to Limit Transaction Per Second (tps) in WCF?

Service throttling is one of the very useful features in WCF. You can simply configure your service with a custom service behavior to limit the number of service call, session and instance by using the following configuration:

      <serviceBehaviors>
        <behavior name="MyBehavior">
          <serviceThrottling 
            maxConcurrentCalls="1" 
            maxConcurrentSessions="1" 
            maxConcurrentInstances="1"
          />
        </behavior>
      </serviceBehaviors>

However, the WCF service throttling is affecting service level only. What if you have a requirement that your service should be limited to accept only 3 requests per second? You cannot precisely do the restriction and controlling with service throttling. Therefore, we have to control the transaction limit at the code level instead, the idea comes from Serena Yeoh.

First, we have to create a WCF service. My service setup is as follow:

[ServiceContract]
public interface ISampleService
{
    [OperationContract]
    string GetData(string value);

}

The GetData is one very simple and dumb operation. It just accept whatever string value and then return the same value back to client.

Next, your service has to be singleton. Why? Because I can only control the limit with 1 instance only.

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Single)]

public class SampleService : ISampleService


The following is the method I used to control the transaction limit:

//Lock an object to prevent multiple instance access / execute the following code at the same time
lock (countLock)
{
    //StopWatch timer started in constructor
    //Check if the timer elapsed time is less than 1 second
    if (watch.ElapsedMilliseconds <= 1000)
    {
        //If there are 3 hits in less than 1 second
        if (counter >= 3)
        {
            //Prepare to sleep or wait until the next second time up
            //Calculate the sleep time by using 1 second time minus the elapsed time
            int sleepTime = (int)(1000 - watch.ElapsedMilliseconds);
            counter = 1; //reset back the hit counter

            Debug.WriteLine("Sleep : " + sleepTime + " ms");

            if (sleepTime >= 0)
                Thread.Sleep(sleepTime);

            watch.Restart();
        }
        else
        {
            //The hit is still within the limit, let it pass
            counter++;
        }
    }
    else
    {
        //The elapsed time has exceed 1 seconds
        //Reset the counter and timer
        //Proceed to process
        counter = 1;
        watch.Restart();
    }

}


In order to test my service, I have created a console program to call my WCF service.

class Program
{
    static void Main(string[] args)
    {
        //Parallelly or concurrently hitting my service
        //to test whether the service process more than 3 transactions per second
        Parallel.For(0, 100, i =>
        {
            SampleServiceClient proxy = new SampleServiceClient();
            Console.WriteLine(string.Format(
                "{0} Service call {1} : {2}",
                DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss.fff"),
                i,
                proxy.GetData("Test")));
        });

        Console.WriteLine("Press any key to continue...");
        Console.ReadKey();
    }
}

This is the result that I got, notice that there are only 3 transaction per second only even though I used parallel for loop to call my WCF service.



If you are interested with my source code, feel to download it from HERE.


Sunday, November 24, 2013

Data Binding in Web Test (Web Service Test)

Today I would like to share how to perform parametrized functional test on WCF web service by using Visual Studio 2012.

Note: Only for Visual Studio Test Professional or Ultimate Edition.

First, create a new Web Performance and Load Test Project.


Then, add a new web service request in your web test.


Open up the properties window for the newly created web service request. Enter your WCF service URL in the Url property.



Add a new HTTP header for your web service request. 



Open the properties window for your newly added header. Select the name as SOAPAction.


Then enter the value for the SOAP action. The value is depending on which service operation that you would like to test. The information can be obtained from WSDL.



Open the properties window for String Body. Select the content type as text/xml. Then, enter the SOAP message into the String Body property. If you do not know how to form a proper SOAP message, here is a shortcut tip for you.



Open up WCF Test Client. It is located at C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\wcftestclient.exe. Or, you can fire up Developer Command Prompt for VS2012 from your all programs list, then enter wcftestclient in the command prompt.



At the My Service Projects, right click it then select Add Service. Enter your WCF service Url, then hit the OK button.

Double click the service operation that you want to test. Fill in all the required values. Then hit the Invoke button.



Click the XML tab at the bottom of the window.



Copy the XML from the Request field. That is the SOAP message that you need to put into the String Body in the web test in Visual Studio.



Go back to Visual Studio, paste the copied XML from the WCF Test Client into the String Body property field. Remove the whole <Action> element from the SOAP Header if the WCF service endpoint is configured with basicHttpBinding which is using SOAP version 1.1.



Hit the OK button, then click the Run Test button to test whether the SOAP message is accepted by the WCF service. You should expect the return is HTTP200 status and expected SOAP message response.


Now, prepare test data for parametrization. Open Microsoft Excel or Notepad to create a comma delimited text file (*.csv). The first row always is the column name, the subsequent rows are the test data.



Go back to Visual Studio, right click the Data Sources folder, click Add Data Source. Enter a data source name, select CSV file, then click Next button. Click the Browse button and choose the CSV file which you had just created with test data.



Open up the String Body property field editor. Replace the value that you want to parametrize with {{DataSourceName.TableName.ColumnName}}.



Open up the Local.testings from the Solution Explorer.



Go to Web Test, choose the radio button "One run per data source row". Click the Apply button, then close the window.



In the web test window, click the Run Test button. You should expect to see multiple run with different request SOAP message.









Saturday, November 16, 2013

Exception: Network Access for Distributed Transaction Manager (MSDTC) Has Been Disabled

Quite some time ago, some of the developers complained about an exception had been thrown from the TransactionScope. And, I attempted to troubleshoot the problem with one of the developers together.

Here is the story. The following is the exception detail:

Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.

If you google around, the given solution normally is asking you to make sure the Distribute Transaction Coordinator (DTC) is configured properly and firewall is cleared, but I still get the same exception.

During the troubleshooting process, I found that there were different connection strings being used, which is like this:

<connectionStrings>
  <add name="db1"
       connectionString="server=myServer1;database=myDataBase; uid=myUser;password=myPassword;"
       providerName="System.Data.SqlClient" />
  <add name="db2"
       connectionString="Data Source=myServer2;Initial Catalog=myDataBase;User ID=myUser;Password=myPassword;"
       providerName="System.Data.SqlClient" />
</connectionStrings>

The developer went on to change one of the connection string to make them using the same format. The result is no more exception had been thrown.

This is weird and went on to search for the difference between these 2 connection string. Base on the article from MSDN - http://msdn.microsoft.com/en-us/library/jj653752(v=vs.110).aspx, they are synonymous with each other. But, DTC does not treat them the same. I do not know why and how TransactionScope internal work. Perhaps, you tell me why if you come across this page and know the answer.

Conclusion, the above exception message is rather useless and sound like a common error message for anything which is related to DTC. Do not just check your DTC configuration, but also check your connection strings.

UPDATE ADD-ON:

There are some people asking more about how to troubleshoot this error. Besides the connection string issue stated as above, there are actually more thing to check:

1. Firewall

Please ensure that you have unblock the firewall for DTC in all related web server and database server.

Inbound Rule:


Outbound Rule:



2. DTC Windows Service

Make sure the Distributed Transaction Coordinator service is running in all related web server and database server.


3. Hostname Resolve

Make sure all the server hostname are able to be resolved in all related web server and database server. Either update the DNS server and then flush the DNS cache by executing ipconfig.exe /flushdns command or add in hostname in the host file manually which is located at C:\Windows\System32\drivers\etc\hosts.

For instance, you have a web server and two database server. Web server must be able to ping both database servers with server name instead of IP address. The database server must be able to ping the web server and the other database server too by pinging both servers hostname.

4. Configure MSDTC Security

Run the command dcomcnfg, then expand the Component Services all the way down to Distributed Transaction Coordinator. Right click the Local DTC, then click the Properties.


Enable the Network DTC Access and remain the default configuration.



Send Transactional SMS with API

This post cover how to send transactional SMS using the Alibaba Cloud Short Message Service API. Transactional SMS usually come with One Tim...