Posts tagged Microsoft

SignalR, Azure and Scaleout

Recently Resgrid started utilizing SignalR to allow real-time updates to clients. If your unfamiliar with SignalR is a server/client Web Socket framework giving an easy API for Microsoft backend server solutions (like MVC, ASP.Net, etc) to utilize Web Sockets. Resgrid has a fairly simple infrastructure: Dual Azure Web App instances for web and api, backend SQL database and workers.

signalrBecause of our multi web server configuration we needed to use a Scaleout system, to allow for clients connected to one web or api instances to receive/send to all instances. If your running multiple web servers and your users trigger SignalR events on one instances, without scaleout only users connected to that same instance will get the event.

SignalR currently supports three methods for scaling out, Redis, Azure Service Bus and SQL Server. Unfortunately with the current version of SignalR (2.2.0) Redis and Azure Service Bus options are in some stages or broken.

This is an unfortunate situation, SignalR’s last release was December 11th 2014, that’s well over a year ago for a pretty critical technology on the Microsoft stack. The last commits on the repository’s master branch (at this time are over 4 months old). If this Github was a physical place, I’d expect to see boarded up windows and tumble weeds. There is light on the maintenance front, SignalR-Server (a different repo) appears to be pretty active.

Why is this an issue? Well because the SignalR repo is over a year old it’s out of sync with the Azure Service Bus assemblies. As of this writing the latest Service Bus Nuget packages greater then 3.0 are incompatible with SignalR Azure Service Bus Scaleout. The error your likely to receive will be something like:

Could not load type ‘Microsoft.ServiceBus.Messaging.MessageClientEntity’ from assembly ‘Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’

If you can stay in lock-step with SignalR’s Azure Service Bus version (2.1.2) then this is a viable option for scaleout, but for most projects this won’t work. Especially if your utilizing features like Azure Web Jobs, which require a version of Service Bus >3 to use some of it’s features.

Another option is to use a Redis server for scaleout. But it seems that Redis is currently in just as bad (or maybe worse) state. Using Redis for scaleout you may not be able to keep an open connection between SignalR and Redis for any period of time, or at all.

Errors you may receive may be:

SignalR.ScaleoutMessageBus Error: 0 : Stream(0) – Send failed: System.AggregateException: One or more errors occurred. —> System.ObjectDisposedException: Cannot access a disposed object.

System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task) at offset 38
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 40
at Exceptionless.Api.Hubs.MessageBusConnection.<OnConnected>d__2.MoveNext() at offset 322 in C:\projects\exceptionless\Source\Api\Hubs\MessageBusConnection.cs:line 23:col 17

The connection has been closed (Error); no new messages can be delivered; the last command was sent xxxxms ago

In testing on our Azure Web Apps instances talking to the Azure Redis server we found that Redis scaleout is completely unusable. At this point the only scaleout option that seems to work is SQL Server. Although we don’t use this option (we have a hacked version using the latest Service Bus version). Our reasoning is we didn’t want to put any additional load on our SQL server, which might not a concern for your needs.

It’s a sad state of affairs for SignalR. Although not totally abandoned there doesn’t seem to much, if any, resources put behind it. Microsoft has pushed this technology hard in the .Net space and, at a minimum, open and honest communication would be nice.

My guess is that like Entity Framework 6.x they will only release bug fix updates and minor versions, while they spin up a SignalR Core style version, matching the pattern of EF6 and EF Core.

If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.

Excessive Memory Size using the Microsoft Service Bus

During periods of high activity on the Resgrid system NewRelic would send out a large amount of warnings due to excessive memory usage. This was great information but I didn’t know why we would have excessive memory usage without corresponding CPU to match.

MemoryLeak_thumb

I like to design my API’s as stateless as possible and the Resgrid API’s are no different. They do a fair amount of business logic and work, but all the heavy stuff is all offloaded to backend workers, databases or Azure itself via the Service Bus. So this slow progressive rise in memory was troubling.

With work on moving Resgrid to App Services from Cloud Services I decided to run some JMeter tests against the API to see what was going on. After a bunch of read based operations tests, no memory leak. Then I analyzed what was occurring during the NewRelic alerts, there were a lot of sets.

I altered the JMeter tests and found this:

2015-12-17 16_18_15-Resgrid - Microsoft Visual Studio (Administrator)

That’s around 500MB in memory allocated to just the Microsoft.ServiceBus operations, particularly the Microsoft.ServiceBus.Common.IOThreadTimer. Some quick goggling and I found this issue on GitHub, so it seems at least as of end of 2014 this was a known issue.

I checked and the version of Microsoft.ServiceBus I had installed from Nuget was 2.4.1 which was from July 2014. Which was before the GitHub issue was posted (in late 2014). The current version of Microsoft.ServiceBus is 3.0.9 from November 2015. I updated and re-ran my JMeter tests and this was the result:

2015-12-17 17_59_49-Resgrid (Running) - Microsoft Visual Studio (Administrator)

After the update, no more memory leak, and an inspection of the objects in memory reveals that no large service bus objects in the heap. So if your using the Azure Service Bus and experiencing memory leaks I highly recommend looking at your Microsoft.ServiceBus version.

If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.

Microsoft BizSpark Retrospective

In October of this year Resgrid graduated form the Microsoft BizSpark program after 3 years. In those 3 years I’ve had the chance to experience a number of things the program offered and want to give a retrospective on the program as a whole.

startup-pirates-119-876fb79e0905b58bf21f-For the uninitiated the BizSpark program is a way startups can leverage Microsoft technologies or services at zero (or low cost) for 3 years. There program also features outreach support, partners with Incubators and Accelerators and much more.

 

It’s a WIN-WIN for Microsoft and for Startups. The startup gets access to Microsoft tech, products and services for free or at reasonable rates and Microsoft has a chance to enable a long term relationship.

The Good

The good parts of the BizSpark program are the access to Microsoft products and services. With BizSpark you get an MSDN Ultimate license, which gives you licenses to a large number of products. Plus you get $150/mo Azure credit, perfect for testing and development.

Microsoft helps your startups visibility as well. They have a Featured Startup section that is shown to a broad audience. There were other options put forward in front of us, some mention in blog articles and a podcast (although that never materialized).

The Bad

If your startup is hardware dependent (i.e. Surface devices) there is limited help from Microsoft in that area, see the section below. Support and/or offers for your company could be based on the current marketing priorities for Microsoft. For example when Resgrid was going through the BizSpark program the priority was Open Source, if your software or service was built on no open source during that time frame, no additional support or offers for you.

The Ugly

I only had a point of contact with Microsoft for about a week and he was based out of the East Coast. I live about a 45 minute drive from MSLI’s offices, yet there was no one in my area apparently. I’m also unclear if he was an MVP, Evangelist, Advisor or what. Past that I had no contact with Microsoft and received no help in sourcing hardware (trying to rent or buy Surface devices for development of our software in Police, Fire, EMS vehicles).

Also if your not in a major metro area or one of the coasts you can almost forget about BizSpark Plus. Incubators I looked at would not accept distant startups and some of them didn’t exist or their online presence didn’t exist. This is a shame, as there are entrepreneurs and startups all over the country and the world, but if your not lucky to live in the Bay Area, Seattle, NYC, well no plus for you.

Conclusion

Microsoft’s BizSpark is a great program and I recommend it to everyone I talk to thinking about starting up. But to get the full benefit you’re startup needs to be located in certain areas and be aligned with Microsoft’s current marking priorities. It also doesn’t hurt if your startup is targeting other businesses (B2B software) as they will receive a fair amount of the BizSpark program communications that you may be a part of.

It would be nice for Microsoft to have more Reps or Advisors around to help guide startups through the program and make the most of it. I would also hope that in the future Microsoft finds a way to open up BizSpark Plus to more areas either through expanding it’s partners or in this age using the Internet.

Resgrid is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).

Go to Top