Posts tagged Team City

Automated Production Testing and Monitoring

With almost two years under my belt of having an app in production on Microsoft’s Azure I’ve learned some hard, hard lessons. One of the biggest is you need to have active testing and monitoring of your production environment from both inside and outside the environment. It sounds like a no-brainer, but it’s so far down the priority list for most developers and startups that it takes a long time to get to, if ever.


Some history is in order; Resgrid is a system designed to provide logistics and management capabilities for first responder organizations like volunteer and career fire departments, EMS, public safety, search and rescue, HAZMAT and more.

Resgrid was founded by myself and a partner staxmanade. It started as a simple website and a couple of mobile apps with one page and some big buttons. A year later it’s now a complete end to end management and logistics system that runs on Windows Azure.

Because our market is first responders we need to ensure uptime and that information is relayed quickly, having something down for even a short period of time can impact our customers and in our market there is no good time to go down.

Create Test Accounts/Data

One of the first hard lessons what not creating seed data that we could use for testing and verification on the production system. We use Entity Framework as our backing repository mechanism and it’s very easy to add data into the Seed method of the Configuration for migrations:

protected override void Seed(Contexts.DataContext context)
            //  This method will be called after migrating to the latest version.

            //  You can use the DbSet<T>.AddOrUpdate() helper extension method 
            //  to avoid creating duplicate seed data. E.g.
                  p => p.FullName,
                  new Person { FullName = "Andrew Peters" },
                  new Person { FullName = "Brice Lambson" },
                  new Person { FullName = "Rowan Miller" }

Use this method and seed your database! What you going to seed you may ask yourself? Anything that you may need to log into the system and perform actions, customer records, login records, etc. Even if you never plan to do production testing, create the data, and you’ll have it just in case.

Do no try and use the Seed method after you have data in the system already, it starts to not turn out well. We had an instance where we tried creating a test department after we had customers in the system already and it created over 300 test departments.

Monitor/Test Externally

Finding our your system is down from your customers is not good practice. You should know that there is an issue, well before your customers let you know. A critical part of this, is testing your site from a place on the Internet that isn’t on the same network or backbone as your system. Azure recently introduced Endpoint Status Monitoring that helps with this, but it’s just high level check.

If your using Team City as your CI server you can setup configurations that run on a schedule that can test your system with complex code, scripts, calls and much more. Have your CI server from another provider, like if your using Azure, have your CI server on Amazon, etc. Also I recommend using a backup service like Pingdom to provide backup monitoring and uptime analytics.

Fully Test Critical Processes

In Resgrid, we send out dispatches (calls) via email, text messages and push notifications. This is a mission critical process for us as our customers need these systems to work. Although we cannot guarantee delivery of the message we can say that if it never gets sent, they will never receive it. So we have jobs that run that perform user actions that would generate those messages and we monitor to ensure we receive them. We test the full flow in production at least once an hour.

Test Multiple Paths

If you have a website and an API site living on different systems, test them both. Don’t just rely on testing say the website path to ensure the system is working end to end.

You don’t always have to automate

You don’t have to automate everything, if it takes you 5 minutes to check something and it takes you 8 hours to develop an automated solution you would have to test something 96 times to break even. As a small/micro business you need to use your time where it’s most effective and that will bring in new customers and keep existing ones happy. If a manual test works and is not high friction, that might be your best bet.

Tools and Services to Get Started

Before starting Resgrid I started collecting a number of tools and services that would help launch a company. Everything from DNS providers to legal documents. Most of these services are free or very low cost, as I like to bootstrap my businesses to keep the costs down and keep flexibility up.


I’ve previously written about using Microsoft’s Azure for providing  cloud services and using the BizSpark program for getting $150 of Azure credit a month for 3 years. So I won’t include this in here, but remember that Azure does support more then .Net so even if your not in the .Net stack you can use Azure.

Below are the products and services that Resgrid uses or used, which is the company that I founded with my partner staxmanade in late 2012. Resgrid is a cloud based service for first responder organizations (volunteer or career fire departments, EMS, public safety, search and rescue, etc) to manage personnel, units, departments and logistics.

Below are the free or low cost products and services that Resgrid uses and we’ve used in the past to run companies and help reduce friction and increase automation and productivity. The only other service we won’t cover in this article is the payment processor that we will cover in it’s own post.

Team City by JetBrains (
What can I say about Team City, except it’s the most versatile, complete and comprehensive CI (Continuous Integration) and build server out there. Every company I’ve worked for since I discovered Team City in 2008/2009 I have championed it’s use. For Resgrid it automated our CI builds, our unit and integration testing and we now use it to deploy to staging and production environments.

CloudFlare (
I first heard about CloudFlare when some people were using it to mitigate DDoS attacks against their websites. But CloudFlare is way more then just security production. It’s a free (and paid if you want SSL) service that hosts your DNS for you, provides front end proxy’s and caching, optimizations, CDN (Content Delivery Network) security and so much more. There is also a rich 3rd party ecosystem of apps you can use and I’ve started to trust CloudFlare’s analytics more then Goggle’s or anyone else’s that uses JS because I always use Ghostery and I’m sure software like it is getting more popular.

BitBucket by Atlassian (
Atlassian is one of my favorite companies, just behind JetBrains. When they launched BitBucket as an alternative to GitHub I was a little skeptical, but no longer. Compared to GitHub BitBucket doesn’t have all the features or the super slick interface but it just get the job done. Why BitBucket? Unlimited Private Repo’s and a team of up to 5 people that’s why! If your code is OSS I would go with GitHub, but if your code’s private then I would go with BitBucket, nice and easy.

UserVoice (
My go to HelpDesk, Knowledgebase and customer support software has been UserVoice for many years now. Resgrid uses UserVoice very successfully for over a year now with integration with our mobile applications iOS, Android and PhoneGap. UserVoice is free for one agent, which is perfect for bootstrapped companies or micro enterprises.

MailChimp by Rocket Science Group (
I was first turned off of MailChimp just because of it’s name, but I’m very glad I pushed past that. MailChimp is an amazing email/newsletter service, with a WYSIWYG editor, easy subscriber management and so much more. In under 15 minutes you can have a professional looking newsletter and have it sent out to all of your contacts. MailChimp is free up to 2,000 subscribers and you can send 12,000 emails a month.

SurveyMoz (
Finding a good, low cost, survey tool was actually very hard, I had to dig deep to find SurveyMoz. Although SurveyMoz has a free plan, it’s limited to 20 questions per survey and 250 responses per month. But it’s paid plan, only $16 per month is amazing when compared to everyone else. Very easy to use tools, powerful capabilities and flexibility makes SurveyMoz a great deal at $16 a month and if your just starting out and will have under 250 responses then free is amazing.

LawDepot (
You will have to draw up legal documents, and keep them updated, generate new ones and much more. LawDepot isn’t free, but it’s pretty low cost, under $100 a year for unlimited legal document generation or you could pay per document, but if your going to buy a couple or more, just pay for the year. The interface isn’t the best, but it works well and it guides you through creation of the documents and all the options. From formation to NDA’s to ELUA’s and much more.

Grasshopper (
Grasshopper is a virtual phone system for your business. I started with Grasshopper back when it was called GotVMail and it’s plan was I believe 9 a month. But for 12 a month Grasshopper is a great deal. Customers still love seeing a phone number, and it’s a great way for them to get in contact with you. I was happy with GotVMail back in the day and I’m pleased with Grasshopper now.

Skype by Microsoft (
Here at Resgrid we use Skype for everything, virtual team meetings, IM’ing back and forth, file transfers and contacting our customers, all for just under $10 a month. The Skype software itself is pretty horrid, there’s no way I can talk that up. For the longest time I could not get Skype video to work with Windows 8 desktop mode, not until I upgraded to Windows 8.1. Plus they constant nagging after every call is just plain annoying. But for quick calls, video conferences and IM’s, it’s an all in one platform.

Trello by Fog Creek (
It took a long, long time to find a free/low cost agile project management solution that I liked. I’ve used a lot of them, from AgileZen, to TFS, and pretty much everything in between but Trello is my solution for a simple, easy to use Agile board from here in out. First I love that you can have multiple boards and public boards, it’s something that I like combining with UserVoice that allows customers to see that your working on issues. So you may have a private board for those super secret things and a public board for your customers to see. It’s interface is simple, easy to use and keeps out of your way.

So those are my go-to services and products for spinning up a new company or brining into an established company to reduce friction. As a developer run bootstrapped business our time can either be spent adding business value, working with/acquiring our customers or being a plumber. Personally I would rather focus on business value and customers then deployments, builds, and maintaining custom built solutions to problems that there are free/low cost solutions for.

In my next article I’ll cover some payment processors and then accounting software.

Azure Deployments with Team City and Powershell

My constant gripe with Azure is the quality of the error messages that it emits. Over the last week I burned days trying to solve a certificate issues that did not exist. Those horrid error messages bit me again while trying to setup automated deployments from Team City to Azure.

I found some great resources to getting Azure deployments working:

Between those 3 blogs I was able to get Team City up and running with automated deployments to Azure. But for one reason the script didn’t work for me. So I’ve created a Gist with the script I’m using for the Azure deployment

Another problem I ran into, circling back to horrid error messages is my build was getting this error:

 The certificate with thumbprint was not found.


Ignore that warning! Look deeper, I got caught with trying to fix that error, as it was the first one, but the real error message is:

ResourceNotFound. Message: No deployments were found.

For me, I had the paths wrong for my cspkg for the service I was trying to deploy. So multiple errors, a path issue and a config variable issue. Once I fixed those errors it worked great.

Happy Deployments!

Go to Top