Over the last year I have been building a cloud services company using Microsoft Azure as our platform. Why Azure? Well the first reason was that mainly develop on the Microsoft platform, although now that’s becoming less and less of true. The second is the BizSpark program that gives startups access to great Microsoft software and hardware for 3 years. Lastly I think Microsoft has built a compelling cloud platform, the list of features are amazing and the flexibility you get sometimes can be unreal. But when things go wrong they really go wrong.
I’ve had some history with Azure. I was at the PDC where it was announced and went to a lot of presentations regarding it over the next few years. Finally I decided last year to found a company that would provide a cloud based logistics system for first responder organizations (like fire departments, ems, police, etc). Around October of 2012 my system, in very early beta, went live on Azure and has been running there ever since, with over 99% availability.
Spinning up the app and deploying it took no time at all. Unfortunately up until August of 2013 what you got for free on Azure was limited, for example I couldn’t use Azure Websites like I wanted, because it wasn’t part of my BizSpark allotment, I had to use Cloud Services. which is the elder statesman of Azure being the original feature next to blob storage.
So here are some of the issues I’ve run into in my first year so hopefully you don’t have to. The thing I’ve noticed about Azure is that there is very little best practice information out there and why things need to be done a certain way. Hopefully I can help a little.
SQL Azure is Unreliable
This really should not be a surprise, as Microsoft has said it many times before. If you contact MS Support about SQL Azure database issues you will be directed to these three sites http://msdn.microsoft.com/en-us/library/windowsazure/dn338083.aspx, http://social.technet.microsoft.com/wiki/contents/articles/3507.windows-azure-sql-database-performance-and-elasticity-guide.aspx and http://social.technet.microsoft.com/wiki/contents/articles/4235.retry-logic-for-transient-failures-in-windows-azure-sql-database.aspx. So you might as well read them now. Microsoft Pattern and Practices has put out the Transient Fault Handling application block a must when working with Azure resources like SQL Azure. You also cannot use the default ASP.Net membership and authentication that we get in MVC/ASP.Net apps unless you build retry logic on top of it. SQL Azure will go down on you for a period of time, sometimes a few seconds, sometimes a few minutes and sometimes for hours. Plan on it and mitigate it. Via my logs I have a SQL Azure issue every few days on a very low traffic website (mainly due to my use of ASP.Net auth provider). SQL Azure only provides fault tolerance, but you can still loose all your data. RedGate has a service that I use to backup my SQL Azure db to geo-redundant blob storage.
Always Create a Virtual Network
On man did this one bite me hard two weeks ago. Once you create anything in Azure you cannot add it to a virtual network after the fact, Cloud Services, VM’s, etc. Even if you never plan on using backend VM’s or private/internal systems create a Virtual Network and an Affinity group and add everything to it. Cloud Services cannot loop back to VM via Virtual IP’s only via Internal Virtual Network IP’s. Save yourself a lot of trouble and get this rolling as your very first Azure activity.
Virtual IP Addresses Change
Yea, this sucks. But every time you do a full new deployment to Azure for your cloud services it’s Virtual IP Address will change. If your using the Azure project in Visual Studio it automatically checks the Deployment Upgrade checkbox in the Advanced Settings tab on the Publish screen. Deployment Upgrades don’t do a full recycle of the environment, they just update changed code. This is ok for 90% of the time, but sometimes you need to make an Azure configuration change and modify those cscfg files, that will require a full update, which means unchecking that check box and loosing your Virtual IP. I highly recommend using CloudFlare in front of Azure. If you do you can handle VIP changes far easier then with just CNAME DNS.
Azure Tools Suck
Between PowerShell, CLI, Azure Projects in VS and the Management Portal each tool has it’s own nuances and each gives you strange, confusing and contradictory error messages. One recommendation I have is if your having issues publishing from Visual Studio’s Azure projects just create a package, and use the Management Portal, it gives slightly better error messages.
Monitor Your Azure Expenses
You need to keep an eye on your Azure Expenses. Once you remove that spending limit you cannot set another one. Why they don’t let you set alerts or a non zero cap is beyond me. Lately Microsoft has been making some horrible decisions and those extend into Azure as well. Somehow one of my instances went from an Extra Small to an Extra Large. Microsoft only refunded 1 month, as it ran that way for 2 months. Getting Microsoft to refund money is very difficult. You have a very limited time window when you get your invoice to dispute the change, past that then good luck. CloudExpress is one way of trying to monitor Azure’s costs like CloudAbility but I haven’t gotten my Azure account working in CloudExpress and CloudAbility doesn’t support Azure.
Cloud Service Deployments
Don’t get too attached to your cloud service deployments. I’ve been told by a few times by Microsoft support that some changes cannot be made once a cloud service is deployed. Not even a full update will accommodate those changes and you will have to completely remove and deploy a brand new cloud service to implement the change.
Information is out of date and scarce
The vast majority of information you find regarding Azure out on the Internet is out of date. I can’t tell you how many times I’ve pulled up an article and found that it was the old Azure interface. Finding answers to solutions, like Virtual Network problems is also very difficult. Take for example this Azure SQL Database Performance Guide that Microsoft support sent me a number of the links in it don’t work. Get used to it. You also have to pay for Support on Azure, thankfully they have a low cost developer plan, but you get ‘best effort’ response and it costs around $10 dollars a month.
So that’s some of the issues I’ve run into in year one. I feel it’s getting better but Microsoft isn’t the best at providing clear and easy to understand guidance or documentation. Also if Azure keeps evolving as quickly as it is there will be more and more out of data info on the Internet. The best advice I can give you for working with Azure is have a plan B for everything you can, chances are within a year you will need to use some of them at least once.
Recently I started experimenting with TypeScript, as we started using it at my day job and I wanted to use it for Resgrid. I really like the concept of what TypeScript is and what it brings to the table and for non-JS developers like myself. But recently I have not had a lot of luck getting TypeScript working in my projects.
This was using the latest version of TypeScript 0.9.1.1 and the latest version of WebEssentials (the one with TypeScript removed).
But I still couldn’t get JS files generated from the TypeScript files. No errors in the compiler and only a few warnings. Finally I found this blog post from almost a year ago.
I edited my MVC Apps’s web project file (.csproj) and added this BeforeBuild target:
<Target Name="BeforeBuild"> <Exec Command=""$(PROGRAMFILES)\Microsoft SDKs\TypeScript\tsc" @(TypeScriptCompile ->'"%(fullpath)"', ' ')" /> </Target>
So it’s still not generating on saving of the TypeScript file, but at least it is now generating them on a build.
As developers and technologists we always seem to have great ideas for new products or services. Some times we white board them, and think of how cool they are. Other times we even start spiking a prototype. Rarer still we even put something out on the Internet for all to see, we might even advertise to the world of it’s existence outside of our friends and family.
I’ve done more of the above then I can count. I have a whole product pipeline with all the tools to squeeze out a deliverable, that’s how many times I’ve done this. Over the last 5 years I’ve worked on at least a dozen products and services. Some making it to the light of day only to get abandoned, others being turned into Open Source projects that I maintain with increasing rarity.
But every once and a while an idea comes along that’s so pure, so simple that it just has to work, and as developers and technologists we are almost predestined to attempt. For me that project is Resgrid, a cloud based logistics system for first responders. Your will be something different, how will you know when you got that idea? I don’t think there is a definite answer, the heavens might open up and harps will play, or there could be pin drop science.
Bounce the idea off someone familiar with the problem domain. I can’t tell you how many times I talked with technologists and non-tech people alike and nodding heads and affirmations that sent me in the wrong direction. Your idea is like your baby, you want to protect it and keep it from harm.
But really your idea is more like a battle bot. You’ve got to shove it into the octagon and get beat up a few times to iron out the issues and perfect it. The adage is “Fail Fast” and failing just past inception is pretty fast, and that’s exactly what you want. Failing does not imply giving up, although knowing when to give up is an art form, it just means getting your ideas worked out as quickly as possible.
So bring your idea to people who know the problem domain, and chat with them, search them out in person or on the Internet. Don’t force an NDA on them, just talk with them, get their feedback and have them gage how useful it is, do they have any suggestions? Try getting feedback from a few people, I think I talked with 3 to 5 people at length about mine before I really got the whole picture and my idea had some new armor.
Focus it down, be selective and minimalist. This one is hard for me and I think is probably hard for a lot of developers. We want to throw the kitchen sink at our projects, leaving flexibility for every conceivable (and unconceivable) future need. It’s hard, but focusing in and dialing it down to a handful of features will allow you to focus your time on those features that directly solve the problem or need.
Chances are you’ll be working on this in your spare time. You’ll be developing one of two things, features that directly impact the users or facets of the overall architecture that do not. I’ve spent many nights hacking PowerShell build scripts, getting team city humming along just how I wanted, and all along I could have delivering functionality. I am by no ways saying that you should have good build scripts, a CI server, unit and integration tests. But those can wait.
Start with a spike or prototype that has the features you need and start refining from there. Build out and refactor from that core, get your system solving the problem or providing the service at the prototype level without the noise so you can fail fast and fail cheaply.
Do it for more then just money. I know there might someday be a VC or investor reading this that might balk at that statement, but I can tell you there will be many more that will agree with it. The money you get from your product or service should be a byproduct of the process. I’m not saying you shouldn’t make any money, or that you shouldn’t have a good business plan. But If you whole goal of the product and service is just money your probably going to lack the passion and perseverance needed to continue when times get tough, and they will.
Your passion needs to be solving the problem or providing the service. Not the technology, or implementation or event the money, it’s being customer centric and devoted to providing that product or service.
I have by no means made it by any stretch of the imagination. I’m not even making money off my idea yet. But I feel that we learn far more from our mistakes and failures then our successes and boy have I made my fair share. Having been though those experiences I’m doing my best to avoid the traps that got me in the past and lead to a successful outcome.
My next number of a blog entries will be about my experiences, failures, ideas, thoughts and products and services that have gotten me to this point. Stay tuned and stay frosty.