Build Automation using PowerShell
Now that we know why build automation is needed, in this detailed post I will cover how to approach build automation using PowerShell. Since this is a technical post I have included the list of acronyms and some useful links towards the end for reader’s benefits.
Background & Requirement
We use Microsoft Visual studio to develop our .Net projects. For a modestly sized client project with both web services and windows services, we were initially relying on manual approach to create daily builds and deploy it on QA, UAT and Production servers. On an average, this was taking around 2 hours of a DevOps Team member and more if some issues crop up. QA was getting delayed every morning getting the link for QA server with new build. We needed to automate this workflow as much as possible.
I was asked to undertake the task. Being a newbie to this, I did some research and found that doing Continuous integration (CI) means to use different tools and tool sets. Tools that I could use were Team City, Jenkins, Team Foundation Server, Bamboo, Circle CI and so on. Most were excellent tools supporting entire stages of CI but these also cost money in terms of licenses, either per user or per seat. Besides cost, there were also learning curves and their own limitations.
I have worked with PowerShell since its alpha release in 2005 and grew fond of its power and versatility over time. I use it for most of my automation needs, be it personal or work related. A relatively new windows command shell, it is integrated with C# which makes it much more powerful than traditional windows CMD shell. Much loved and rarely rebuked, it has very vibrant user community in System management and automation space. Just do a Net search for common System Management tasks through PowerShell. Through Powershell Core , it is also making its way in Linux world. PowerShell is released with many inbuilt cmdlets to do most of day to day work and a lot of windows components & other vendors have provided their own cmdlets to manage their environments through PowerShell command line.
Justification & Business Case
Faced with licensing costs & other constraints of traditional CI tools and my own comfort with PowerShell, I decided to script the whole Build Automation workflow using PowerShell. It was a wonderful decision. Automation has been working very well since 2 years and saving minimum 2 hours per day for a small project with 5 web services and 3 windows services and code stored in SVN repository. Later on, we extended it to work with much bigger project with 11 Visual Studio (VS) solutions and 71 VS projects with code stored in TFS (Team Foundation Server). It saved 3 hours per day on the second project. We could deploy this on any windows machine where Windows Management Framework (WMF) 4.0 can be installed. Everyone on development team was aware of the code build status and DevOps could use output of this process to deploy it inside client VPN.
Creating Build Management Framework
For our Build Management workflow, I used the below tools, cmdlets and techniques
- Retrieve code from source control using SVN command line and TFS PowerShell
- Modify Solution files and Project files which are plain XML and easily manipulated from PowerShell.
- Use exe to download on-demand packages.
- Use MSBUILD and MSDEPLOY to build and deploy to desired location.
- Use IIS PowerShell cmdlets to manage Web services and deploy them.
- Use Out of the box PowerShell cmdlets to manage Windows services.
- Validate deployments and do BVT using Invoke-WebRequest, Invoke-RestMethod and other out of the box cmdlets.
- Invoke our test suites written in TestNG or nUnit directly.
- Use Send-MailMessage to send mails with results and the attachments. Securely save Sender password in system through PowerShell Secure Capabilities.
If I had been automating build created in other environment like Java, Maven/Ant & Git, I could have easily installed these on my build machine and use the corresponding command line to get code and build it.
Major Benefit of the PowerShell based approach
Major benefits of using PowerShell to do Build Automation as per me are as follows
- Easy availability and No Licensing cost – All things mentioned here are available free of cost on any windows computer. There is no ongoing cost beyond initial development cost.
- Easy deployability – This setup can be easily deployed anywhere on a windows computer and it can start working without much hitch.
- Easy maintenance – Once used to tweaking the configuration files required for it, even a novice DevOps engineer can maintain this. If DevOps team is well versed with PowerShell, they can even debug and update the scripts as and when needed.
- Modularity – I divided my solution in 4 parts, SVN fetch, Solution Modification, Build, Deploy. Most of the code in this is easily reusable across various .net projects.
- Longevity – Microsoft is committed to enhance PowerShell experience across the board, as also many vendors. Investment in this is not going to be wasted down the line.
- Existing knowledge – Most DevOps engineers have some exposure to PowerShell and can utilize their knowledge very easily in this rather than learning a new UI tool.
- Full control – PowerShell cmdlets and other command line tools usually expose more information and allow more fine-tuned control than any UI based tools which always need some scripting support.
With PowerShell, immense power is available with a DevOps engineer to tweak things as needed. Why run around and struggle with various evolving tools when everything is available easily to do most of the build automation tasks through out of the box tools?
Added advantage of a build automation project undertaken through PowerShell would be major knowledge upgrade for the DevOps team. They will be exposed to many tools & concepts and become more nimble & productive with effective usage of PowerShell in their day to day life.
Cmdlets available from others and coming with new software installation
As I noted earlier, many vendors have extended support for PowerShell through their own PowerShell cmdlet packages. I will list out few now which do not constitute an exhaustive list by any standards but gives a glimpse of what can be done through PowerShell from a DevOps architect perspective. If your workflow constitutes more steps than mentioned earlier, you may need to use one of the below.
- AWS tools for Windows PowerShell – Manage AWS services from the Windows PowerShell scripting environment.
- Azure PowerShell
- SQL Server PowerShell – SQL Server 2017 supports Windows PowerShell. PowerShell supports more complex logic than Transact-SQL scripts, giving SQL Server administrators the ability to build robust administration scripts.
- Oracle Cluster and PowerShell
- Net and Data Access – Connecting to Oracle Database from PowerShell.
- PowerShell for Docker – Under construction now, under open source project but very promising.
- Manage VPN Connections through PowerShell – If needed; connect to VPN before code download or deployment.
- Manage Windows Clusters through PowerShell
- Microsoft Office PowerShell cmdlets – Automate editing of Office Files.
- PowerShell in Jenkins – Use PowerShell scripts in Jenkins.
Links, Acronyms & Further readings
- PowerShell learning from Microsoft Virtual Academy
- Continuous integration – A development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
- PowerShell upgrade
- List of Build Automation tools
- Some CI tools – Team City , Jenkins , Team Foundation Server , Bamboo , Circle CI
- SVN – Apache Subversion
- TestNG – TestNG is a testing framework inspired from jUnit and nUnit but introducing some new functionality that make it more powerful and easier to use.
- nUnit – NUnit is a unit-testing framework for all .Net languages
- QA – Quality Assurance
- UAT – User acceptance testing
- Cmdlets – a lightweight command that is used in the Windows PowerShell environment. The Windows PowerShell runtime invokes these cmdlets within the context of automation scripts that are provided at the command line.