WCF/WF Service Deployment Automation with PowerShell

Today I want to cover about how to automate the WCF/WF service deployment and configuration in IIS by using PowerShell. During a normal manual deployment process, we copy the compiled assemblies to a folder in the web server, and then configure the IIS by setting up an application pool, website then application. Therefore, in order to automate the deployment process, first we need the following information:

1. Copy assemblies source location
2. Copy assemblies destination location
3. Application pool name
4. Website name
5. Application name
6. Create app pool? (optional)
7. Is Workflow? (optional)
8. Is 32-bit platform? (optional)

File Copy

For the copy assemblies part, we can rely on robust file copy tool (robocopy) to do the job. It is reliable, configurable, support auto resume, etc. Robocopy is available or come with windows resource toolkit. I am going to split the script into 2 parts. First part only cover the file copy, second part only cover the IIS configuration. So that the file copy is not tightly coupled with IIS configuration because sometime we only want the file copy but excluding IIS configuration, or skip the file copy and only want to configure IIS.

Script #1:

param (

      [string]$source = $(throw "-source is required."),

      [string]$destination = $(throw "-destination is required.")

)

write-output "`r`n"

write-output "Source is $($source)"

write-output "Destination is $($destination)"

write-output "`r`n"

#copy all the files including all level subfolders from source to destination.

#folder will be automatically created if not exist in the destination.

write-output "Copying file from source to destination..."

robocopy $source $destination "/v" "/e"

write-output "`r`n"

write-output "File copy operation has been completed."

This is how it look like after running the script:

IIS Configuration

Next, the IIS configuration part, we have to support new and existing application deployment. So, there are some checking required in the script to check for application pool and application existence. In the following script, it will prompt to user to ask whether want to overwrite existing application pool configuration and also another prompt for application configuration.

In order to use PowerShell to configure IIS, we need to import WebAdministration module into our script by using the Import-Module "WebAdministration" command. This module allow us to configure almost everything in IIS. In my following script, I use the module to create application pool, create application and change application setting. For more information about PowerShell with IIS, visit HERE.

In order to setup an application, basically we need the information of the website name, application name, application pool name, and physical files path. They are needed to be part of my script parameters. And then, I have 3 parameters with switch data type:
$createAppPool is the flag used to create application pool.
$isWorkflow is the flag used to configure the application to enable net.pipe protocol for Workflow Management Service to activate workflow instances. And also setting the autoStart mode to true and using AlwaysRunning start-mode to make the IIS worker process automatically start and running all the time.
$is32bit is the flag used to set the application mode to run as 32-bit process in WOW64 mode.

Script #2:

param (

      [string]$websiteName = $(throw "-websiteName is required."),

      [string]$appName = $(throw "-appName is required."),

      [string]$appPoolName = $(throw "-appPoolName is required."),

      [string]$physicalPath = $(throw "-physicalPath is required."),

      [switch]$createAppPool,

      [switch]$isWorkflow,

      [switch]$is32bit

)

#import the IIS administration module.

Import-Module "WebAdministration"

write-output "`r`n"

write-output "Website Name is $($websiteName)"

write-output "Application Name is $($appName)"

write-output "App Pool Name is $($appPoolName)"

write-output "Physical Path is $($physicalPath)"

write-output "Create App Pool is $($createAppPool)"

write-output "Workflow is $($isWorkflow)"

write-output "32-bit is $($is32bit)"

write-output "`r`n"

#reusable function to configure app pool.

function Configure-AppPool

{

      #when isWorkflow flag is supplied, set the autoStart, startMode and app pool recycling time.

      if ($isWorkflow.isPresent)

      {

            write-output "Configuring app pool $($appPoolName)...`r`n"

            Set-ItemProperty IIS:\AppPools\$appPoolName -name autoStart -value True

            Set-ItemProperty IIS:\AppPools\$appPoolName -name startMode -value AlwaysRunning

            Set-ItemProperty IIS:\AppPools\$appPoolName -name recycling.periodicRestart.time -value "00:00:00"

      }

     

      #when is32bit flag is supplied, set the app pool to run in 32-bit platform mode.

      if ($is32bit.isPresent)

      {

            write-output "Setting app pool $($appPoolName) run on 32-bit...`r`n"

            Set-ItemProperty IIS:\AppPools\$appPoolName -name enable32BitAppOnWin64 -value True

      }

}

#reusable function to enable net.pipe protocol for application

function Set-Application-EnabledProtocols

{

      #when isWorkflow flag is supplied, enable net.pipe protocol for the application.

      if ($isWorkflow.isPresent)

      {

            write-output "Enable net.pipe protocol for $($websiteName)\$($appName)...`r`n"

            Set-ItemProperty IIS:\Sites\$websiteName\$appName -name enabledProtocols -Value "http,net.pipe"

      }

}

write-output "Configuring IIS...`r`n"

#when createAppPool flag is supplied, create a new app pool with the provided name.

if ($createAppPool.isPresent)

{

      write-output "Creating app pool $($appPoolName)...`r`n"

     

      #check if the app pool is already exist.

      if (Test-Path IIS:\AppPools\$appPoolName)

      {

            #ask to overwrite the app pool setting?

            write-output "App pool $($appPoolName) already exists.`r`n"

            $overwriteAppPool = read-host "Do you want to overwrite the app pool setting? [Y]es [N]o"

           

            if ($overwriteAppPool -eq "Y" -or $overwriteAppPool -eq "y")

            {

                  #overwrite the app pool setting.

                  write-output "`r`n"

                  write-output "Overwrite App pool $($appPoolName) settings...`r`n"

                  Configure-AppPool

            }

      }

      else

      {

            #directly create new app pool and then configure it.

            New-Item IIS:\AppPools\$appPoolName

            write-output "`r`n"

            Configure-AppPool

      }    

}

write-output "Creating application $($websiteName)\$($appName)...`r`n"

#check if the application already exist.

$detectedApp = Get-WebApplication -name $appName -site $websiteName

if ($detectedApp -ne $null)

{

      #ask to overwrite the application setting?

      write-output "Application $($websiteName)\$($appName) already exists.`r`n"

      $overwriteApp = read-host "Do you want to overwrite the application setting? [Y]es [N]o"

     

      if ($overwriteApp -eq "Y" -or $overwriteApp -eq "y")

      {    

            #overwrite the application setting.

            write-output "`r`n"

            write-output "Overwrite application settings...`r`n"

            write-output "Assigning app pool $($appPoolName) to application $($websiteName)\$($appName)...`r`n"

            #Set-ItemProperty IIS:\Sites\$websiteName\$appName -name applicationPool -value $appPoolName

           

            Set-Application-EnabledProtocols

      }

}

else

{

      #directly create new application.

      New-Item IIS:\Sites\$websiteName\$appName -physicalPath $physicalPath -applicationPool $appPoolName -type Application

      write-output "`r`n"

     

      Set-Application-EnabledProtocols

}

write-output "IIS configuration has been completed.`r`n"

For the new setup, the output screen look like this:

If there is an existing application pool and application in the IIS, the screen look like this:

Lastly, if you want to automate in running both scripts together, you can combine both PowerShell scripts into one like below. The condition is script #1 and #2 must be sitting together in one same folder.

Script #1 + Script #2

param (

      [string]$source = $(throw "-source is required."),

      [string]$destination = $(throw "-destination is required."),

      [string]$websiteName = $(throw "-websiteName is required."),

      [string]$appName = $(throw "-appName is required."),

      [string]$appPoolName = $(throw "-appPoolName is required."),

      [switch]$createAppPool,

      [switch]$isWorkflow,

      [switch]$is32bit

)

write-output "`r`n"

#check if the app-deploy.ps1 file exist in the same folder with this script.

if (Test-Path .\app-deploy.ps1)

{

      #invoke app-deploy script with the provided arguments.

      Invoke-Expression -Command ".\app-deploy.ps1 -source $($source) -destination $($destination)"

}

else

{

      #terminate the script when app-deploy.ps1 not found.

      write-output "ERROR: Could not locate app-deploy.ps1 file in the same folder.`r`n"

      EXIT

}

#dynamically construct the optional command arguments before calling service-prepare.ps1 script.

$commandArguments = "-websiteName '$($websiteName)' -appName '$($appName)' -appPoolName '$($appPoolName)' -physicalPath '$($destination)'"

if ($createAppPool.isPresent)

{

      $commandArguments = $commandArguments + " -createAppPool"

}

if ($isWorkflow.isPresent)

{

      $commandArguments = $commandArguments + " -isWorkflow"

}

if ($is32bit.isPresent)

{

      $commandArguments = $commandArguments + " -is32bit"

}

if (Test-Path .\service-prepare.ps1)

{

      #invoke service-prepare script with the provided arguments.

      Invoke-Expression -Command ".\service-prepare.ps1 $($commandArguments)"

}

else

{

      #show error when service-prepare.ps1 is not exist in the same folder with this script.

      write-output "ERROR: Could not locate service-prepare.ps1 file in the same folder.`r`n"

}

Lastly verify the application pool and application creation and its setting in IIS. Enjoy power-shelling~

Round Robin File Processor

Problem

I wonder if you ever encounter a given system requirement that need you to process files immediately which are dropped into one specific folder? And then, when one file is big, it takes a lot of time to complete the process, but it causes the other files which were dropped later are being processed late. In order to fasten the process, you use multithreading to process all the files at the same time. Later, you will realize multithreading would stress the database and you will encounter command timeout or resource deadlock related error.

Solution

Since all the files in one folder are required to be processed immediately, but you do not want to stress up the database,  how about we process every file at the same time but bit by bit and turn by turn like round robin concept? This idea comes from Serena Yeoh. Later you will find out how I implement the concept in this post. This concept will create an illusion or user experience that all the files "look like" are currently being processed. In fact, all the files are not completely processed yet. Also, it solve the problem when one big file is being processed, the other smaller files that come in later no longer need to queue.

Above image is to illustrate the round robin file processor behavior. There are 6 files waiting to be processed. File 1 is going to be processed first, it will be processed for 3 seconds only, then pause. Next file will be processed for the same 3 seconds duration then pause. Same treatment for the rest of the files. Until the last file is being processed for 3 seconds, the next turn will go back to File 1 and resume its process.

Implementation

First, create a file system watcher that will pick up a file to process from a specific folder.

FileSystemWatcher watcher = new FileSystemWatcher();

watcher.Path = Path.Combine(Environment.CurrentDirectory, "TestData");

watcher.Filter = "*.*";

watcher.NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite;

watcher.IncludeSubdirectories = false;

watcher.Created += watcher_Created;

watcher.Error += watcher_Error;

watcher.EnableRaisingEvents = true;

The following is how I process the file when the Created event is fired. The most important thing that make round robin behavior is the ManualResetEvent reference is kept in a List. The ManualResetEvent will be used in another long running thread that manage the thread to pause/resume process.

private static void watcher_Created(object sender, FileSystemEventArgs e)

{

    ProcessFile(e.FullPath);

}

private static void ProcessFile(string filename)

{

    //Create a manual reset event with false signal initial state

    ManualResetEvent resetEvent = new ManualResetEvent(false);

    //Create a new thread to process the file

    //So that it wont jam the main thread

    Task task = Task.Run(() =>

    {

        int retry = 0;

        while (retry < 3)

        {

            try

            {

                string filenameOnly = Path.GetFileName(filename);

                using (FileStream stream = File.OpenRead(filename))

                using (StreamReader reader = new StreamReader(stream))

                {

                    string data = null;

                    long line = 1;

                    while ((data = reader.ReadLine()) != null) //check if the file is empty or end of line

                    {

                        resetEvent.WaitOne(); //always pause, wait for signal to resume

                        Console.WriteLine("Processing " + filenameOnly + " - Line " + line.ToString());

                        line++;

                        Thread.Sleep(1000); //just to slow down the process to see the round robin behavior

                        //when the Task is cancelled, stop the processing

                        if (_cts.IsCancellationRequested)

                            break;

                    }

                }

                break;

            }

            catch (IOException)

            {

                //file may be locked by other process while copying file

                //retry reading file after 1 sec

                Thread.Sleep(1000);

                retry++;

            }

        }

    }, _cts.Token);

    //store the ManualResetEvent reference into a list

    _signalList.Add(resetEvent);

}

The following is the long running Task that manage threads to pause or resume. What it does is every 3 seconds, it will pause all the threads, then keep track every thread's turn and resume only the thread that reach its turn.

//Long running task

Task.Run(() =>

{

    int counter = 0;

    while (!_cts.IsCancellationRequested)

    {

        try

        {

            //Pause all

            foreach (var signal in _signalList)

                signal.Reset();

            if (_signalList != null && _signalList.Count > 0)

            {

                //Resume one

                var resetEvent = _signalList[counter];

                resetEvent.Set();

                counter++;

                //Resume chunk

                //Code example below resume 4 file processing at the same time

                //var resetEvents = _signalList.Skip(counter).Take(4);

                //foreach (var resetEvent in resetEvents)

                //    resetEvent.Set();

                //counter += 4;

                if (counter == _signalList.Count)

                    counter = 0;

            }

            //Every 3 seconds, change turn

            Thread.Sleep(3000);

        }

        catch(Exception ex)

        {

            Console.WriteLine(ex);

        }

    }

}, _cts.Token);

The counter variable is used as a enumerator to track turn and trigger the Set() or Reset() method from the ManualResetEvent reference from the _signalList.

This is how my console look like when I place 3 files into the folder for testing:

While these 3 files are still being processed, if I drop the 4th file in, this is what going to be happen:

And then, it will automatically resume the first file process.

Lastly, if you are interested with my complete source code, feel free to download it from HERE.