.Net’s Stagnation and Brain Drain

0

Recently Justin Angel posted an essay he’d written titled “The Collapse of the .Net Ecosystem, V3”. It’s an interesting read and very well could be flawed in many meaningful, statically and logical ways. But if you haven’t popped your head up in a while there are noticeably less .Net jobs then there were in 2006-2010. This could be relative or absolute, but when I was looking for a  job a few years ago the C#/.Net listing were few and far between.

brain-drain

The jobs that are out there aren’t at startups, they mostly aren’t working on solving interesting problems, they are Line of Business applications.  Head over to Angel.co and see how many of their 8,400+ startups are using C#, hint it’s less then 2.5%.

There is nothing wrong with Line of Business applications or enterprise app development. You can make a very good living doing this and get to work on some cool stuff, maybe. But without new people, new ideas, different points of view and fresh perspectives the .Net Community becomes an echo chamber. As a whole it will lag behind other more cutting edge development techniques and methodologies because we aren’t exposed to them, as people aren’t flocking to be .Net developers.

You may be saying to yourself, “That’s fine I don’t want to work for a startup anyways” and that’s a perfectly valid opinion. But it’s undeniable that startups attract the younger developers, who can take the risk and the commitment to the startup pace. These then become the next generation of enterprise developers. If there are only a handful of .Net startups what does that say about the possible new input into the .Net Development collective? How many developers are leaving? I’d bet you we have a net loss in people leaving the .Net community verses new minds coming in. As I sit here the path that is currently laid out for .Net developers is the track languages like COBOL followed.

For the last few years Stack Overflow has done a developer survey. You can head over to SO and check out the results. The section I want to focus on is “Most Popular Technologies”. Here you can easily compare 2013, 2014 and 2015 trends.

2015-06-15_8-02-02

Here is the breakdown for C# for the last 3 years:

2015 2014 2013
C# 31.6% 37.6% 44.7%

In 2 years C#’s popularity on SO has dropped 13.1%. To be fair JavaScript dropped 4.5% as well from 2014, but they call out Node.js (13.3% and AngularJS 13.3%) and Angular didn’t have any numbers for the 2014 survey. The trend may not be alarming to all, but at a minimum it should let you take notice and stock of what the future looks like.

.Net will be around for a long time, but in 5 years how much new development will be done in it compared to how much maintenance? What about in 10? I love the C# language and the .Net Framework, it’s elegant, complete and very capable. But would I want to just fix bugs on a 10 year old legacy system day in and out with it? Sure you will be able to find C# work, but is it going to provide the challenges that keeps us developers engaged and provide job satisfaction?

I don’t blame Microsoft for this (ok maybe just a little), they are responding to what the market wants. You can’t continue to just push you language/framework as the only way to do things. Hell even Apple announced that they are open sourcing Swift. But times have changed, in the past you needed to do .Net for Windows development. Web application weren’t very good and to build any decent Windows Apps you used .Net. Eventually those Windows Apps got ported to ASP.Net, then ported to MVC. We went from Windows where really only C++ and .Net mattered to the Web where, well lots of things matter. Now to build a “Metro” app you can JavaScript or C++, no .Net required and there are tools to utilize “Metro” apps in the normal desktop.

I think with Microsoft Open Sourcing a good chunk of their ASP.Net stack they can compete better with Java, companies that don’t run Windows can use straight .Net now on their Linux/Unix boxes. Yes there was always Mono but that never really took off as I had hopped. So there is some light on the end of the end of the tunnel, hopefully it’s not a “No Exit” sign.

Resgrid is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).

Trying Meteor on Azure Web Apps\Websites

0

I’ve been starting to play around with NodeJS, the MEAN stack and now recently Meteor. As a developer, I’d like another tool in my belt. I’ve got the .Net Framework stack down, but what if I had to use something else. I’m not a huge fan of Java, although Java 8/9 looks interesting with LINQ support.

nfhgSo I recently started playing around with Meteor, which is a RAILS’esque superset for NodeJS with some cool features (sharing client and server state seamlessly between the web browser and server) plus lots of plumbing abstraction.

I’m also a huge fan of Azure, but after some quick Googling l found out that Azure Websites, now Azure Web Apps, does not support Meteor. You can spin it up with a Linux VM and/or a Docker container on top of that, but that’s not the route I wanted to go through, yet.

Meteor using it’s own packaging and deployment engine. But under the hood it’s just a specialized NodeJS app. Some more Googling and I found out about Demeteorizer, which unpacks a meteor app back into it’s own NodeJS parts.

Inside your meteor app you can run the command “demeteorizer” to decompile the meteor app.

2015-06-07_12-17-54

Now were not out of the woods yet. Meteor utilizes native bcrypt which requires native (Visual Studio 2010) build tools to compile. This does not work on Azure Web apps. The workaround is to run node install and compile locally then include the node_modules directory in your upload. Note that if your using the Free or Shared version of Azure Web Apps those are 32bit only, so you need to compile under a 32bit architecture..

Meteor 1.1.0.2 uses Node 0.10.36, but as of this post Azure only supports up to 0.10.32. To get around this you need to create a bin directory in the .demeteorized directory and download the correct version of node. You can follow this guide for more detailed information. Ensure you get the correct bit’ness (the default folder is 32bit and there is a folder in the npm dist directory for 64bit).

Create a file in the .demeteorized directory called IISNode.yml with the following line in it:

nodeProcessCommandLine: "D:\home\site\wwwroot\bin\node.exe"

Edit the package.json and change the version of bcrypt from 0.7.8 to 0.8.3. I had to do this to be able to build bcrypt on my system. You may need to add “–msvs_version=2013” params when building on your system if the default build chain doesn’t work. Also if your on a 64bit system and deploying to a 32bit Azure Web App instance (this is the default for free/shared) you need to add “–arch=ia32” to your npm install commands.

After you’ve manually updated the package.json run the following command in the demeteorized directory:

npm install

Next got into the programs/server directory and edit the package.json file there. You want to change fibers from a tarball to 1.0.5 so it should look like this:

"fibers": "1.0.5"

Now run the follow command in .demeteorized\programs\server:

npm install

That should get us setup to deploy our Meteor app to Azure Web Apps. Go into Azure and create your new web app. Once it’s created go into it and click the configure tab. Scroll down to the “app settings” section and supply the following variables

MONGO_URL
ROOT_URL
MAIL_URL

2015-06-07_12-39-32

If you don’t have a MongoDB you can add one from MongoLabs as an App Service.

2015-06-07_12-41-46

They have a free 500MB version you can play with. Once it’s create and in your account, click on it, then click “Connection Info” in the bottom to get your connection string. Utilizing your deployment method of choice, deploy you app to Azure.

After all this the Meteor app still isn’t running on Azure. I’m getting the following error:

Tue Jun 09 2015 15:12:51 GMT+0000 (Coordinated Universal Time): Unaught exception: RangeError: Out of memory
    at Object.<anonymous> (D:\home\site\wwwroot\programs\server\boot.js:263:4)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (D:\home\site\wwwroot\main.js:9:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
Application has thrown an uncaught exception and is terminated:
RangeError: Out of memory
    at Object.<anonymous> (D:\home\site\wwwroot\programs\server\boot.js:263:4)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (D:\home\site\wwwroot\main.js:9:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)

A tweet form Jeremiah Billmann points to an issue with Node-Fibers and Azure (or IIS).

So possibly the issue here is Node-Fibers not playing nice with IISNode or IIS. Hopefully I can find a solution soon as I love to zero friction deployment and management of Azure Web Apps.

Resgrid is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).

Azure CPU Linear Growth/Ramp Up Issue

Recently Resgrid experienced an issue with our backend cloud service web role instances. After a deployment our CPU, in all instances, would ramp up from a normal of around 10% to over 90% stay there for a while then ‘reset’ back to 10% and begin the process again.

2015-05-08_13-50-47

Above is a screen shot of the monitoring tab in the Azure portal for the visual inclined. Sometimes it would peak at over 90% and stay there, and dramatically affect our API performance for an extended period of time. This is a huge deal as most of Resgrid users interact with it via the API at some level (calls going out, emails being imported and of course our mobile apps).

We were at a complete loss why this was happening. It seemed to start a while ago, but was exacerbated by new work. But only on our API stack, our Web stack was following a normal CPU pattern, hovering around 20%. Both projects shared the exact same service, database, caching and other provider code. The API layer itself was pretty thin. So what in the world was going on?

After a lot of debugging and redeployments we were at a loss. Time to call in the big guns, Microsoft. The good thing about the Azure Support plans, if you sign up and pay the $29 bucks you get support. So no need to be paying that all the time.

We sent around 10GB’s worth of dumps and PerfView traces to MS to review. So what did they discover:

From majority of the callstack, we see StackExchange_Redis!StackExchange.Redis.SocketManager calls either for reading or writing to the queues. Now there are 2153 active threads in the process! This seems to be too high and it would be interesting to see if you arerunning into connection problems mentioned here: https://social.msdn.microsoft.com/Forums/en-US/5e075053-802a-4a46-9fea-a0e859e9a7a9/redis-cache-sudden-100-cpu-and-crash?forum=azurecache

Now there are 1065 StackExchange.Redis.ConnectionMultiplexer objects in the managed heap! The dump shows that most of these ConnectionMultiplexer objects has connection failure message “UnableToResolvePhysicalConnection on PING”. So it seems there are lot of connection issues happening.

I would highly recommend customer to update StackExchange.Redis from v1.0.333 to v1.0.450 (https://www.nuget.org/packages/StackExchange.Redis/1.0.450 ). The older version might have had such 100% high CPU issues. Also are you creating multiple multiplexer object? Redis cache recommend to have one object and reuse it.

We started using Azure’s Redis cache a while back to cache our Geocoding results. We’ve been doing more and more work with that recently as it’s critical information to first responders. Getting a cached value from Redis is far faster and cost effective then contacting Google, Yahoo or Bing.

When we set it up we installed version 1.0.333, which was supposed to fix the CPU issue and may have in some cases but not ours. We use Ninject to control the lifecycle of our objects and had our RedisProvider in a singleton scope, but that may been part of the issue as well.

We upgraded to the latest StackEchange.Redis (v1.0.450) and marked out ConnectionMultiplexer as static and that fixed the issue. So if your seeing a CPU ramp up and using Redis, check your packages/dll’s and ensure your ConnectionMultiplexer is static.

Lessons Learned:

  • Always enable Remote Desktop for the roles, web or worker. This was amazingly helpful when Microsoft needed us to install software on the machine.
  • Pick an instance to let fail and cycle the other instances. This keeps your service up and running while allowing you to test. The Azure load balance seems to be a round robin, so your high CPU instance will still get traffic.
  • In Cloud Service deployments turning off Update Deployments does not issue you a fresh VM. If you install anything on the VM and a deploy without a Update Deployment method selected (Incremental or Simultaneous) is safe.
  • To get a fresh VM you need to “Reimage” from the Azure Management Portal, Instances section. Deployment and Reimages will keep the same machine name in case you have something else that keys off machine name.
  • Azure CPU metrics are based on averages over a 5 minute period. Just because Azure is reporting 90% CPU utilization if you log into the VM you won’t see the CPU pegged at 90%.
  • Have another tool to monitor performance. I’ll be reviewing NewRelic in a latter blog post.
  • Do not rely on Profiling, Intellitrace or Remote Debugging. In both VS2013 and VS2015RC we were unable to get those to work correctly.
  • When using the Debug Diagnostics Collection tool, the HTTP Response time trigger did nothing. Although the application was slow, the way the IIS Server was determining if it was ‘slow’ didn’t work. Performance Counters worked best.

Resgrid is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).

Go to Top