Kicking the can down the road


Being a developer is an interesting profession. On one hand there are an engineering feel to it but on the other hand it’s far more like art. For me calling myself a “Software Engineer” is more for marketing then anything else. But lets face it, little of what developers/programmers do is actual engineering.

kicking-the-canEngineers are certified go through an apprenticeship process then design and create things in the real world that, for the most part, have to stand the test of time. To me Electrical, Mechanical, Structural, etc are the real engineers and us developers haven’t yet earned that right to use that title in the way we do.

How often in a development meeting do we sit there, talk about a design/architecture/code flaw that could cause issues down the road but decide to not even begin to address it? It happens so often we have an industry term for it “Tech Debt”. Can you imagine some structural engineer’s having the same discussion?

Tom: “So I cobbled together this bridge design from a bunch of other actual bridge designs and sample designs and we look ready to go.”

Mike: “Nice, but looking at these designs if we have the amount of traffic we estimate, and we fully expect the bridge to be popular in the near future, it will start leaning to one side”

Tom: “Yea, but we have a deadline, we can always go back and add more supports latter”

Structural Engineers can’t go back and refactor the bridge after it’s been built as easily as developers can with code. There are also lots of other differences, but it really boils down to is that engineers can’t kick the can down the road as easily as developers can, and so we do.

We talk about “performance as a feature”, but what about “ease of maintenance as a feature”, “scalable architecture as a feature”, “security as a feature” or “testing as a feature”? Every time we kick something down the road, label it as “Tech Debt” and put it in the backlog if were being truthful with ourselves we know full well that unless it catches fire we will almost never be back to address it.

Eventually all that Tech Debt will catch up to you and when it does, it can cost you business, alienate customers, hurt your image or even cause your company to fail. I’ve talked about this problem before and at Resgrid I try and follow a 60/40 approach. 60% new features/customer facing bugs, 40% tech debt, testing, automation and tooling.

Here are some guidelines I feel will help stop us from kicking cans down the road and start turning developers into engineers.

  1. Design/Code for the developers around you.
    One thing I’ve always admired about the military is the “Do it for the person next to you” attitude most service members have. The same can be said for the fire service. Sure you start off wanting a thrill, but after the first few times your going in because it’s your duty and because your doing it for your crew. Developers need to have the same mentality, don’t code for yourself or for your company, craft code for the developers on your team. Do it for the person in the cube next to you.
  2. Balance Architecture & Implementation
    I’ve seen my fair share of architecture astronauts in my day and they can do more harm then good. The architecture implemented needs to match the problem domain and how it’s going to be implemented. The architecture you use for a internal LoB app will be different then one that will be deployed on the cloud. No architecture is perfect and it never will be. Ideally you need to design your Architecture for the worst case scenario for 5 years out? How many users do expect in 5 years? Double that, then how are you going to handle that? How many web servers, databases will you need? Is that the scale you should have been using eventing, CQRS, etc?
  3. Code for now and for the future
    If your response to something new is “well that’s the way we’ve always done it” or some variation of it your coding in the past. Yes it’s painful to constantly keep up to date with the latest trends and best practices, but your hurting yourself and the entity you working for by not incorporating the current best practices. You will find it increasingly hard to find developers that want to work, or even know how to work, in your ‘brand new’ code base. I’ve been interviewing a lot of developers lately and almost none of them with less then 7 years experience have ever worked on an WebForms app. Starting a new project? Using WebForms? You will find it very difficult to find developers that know that technology in the near future. This same principal goes for patterns & practices.
  4. Don’t Investing in Tooling/Automation too Early
    When you first starting out a project get the minimum amount of tooling/automation you need. This is a train of thought used by startups. Time spent on setting up elaborate tooling, complex automation is time not spent properly architecting your project, implementing features or fixing bugs. Because tooling/automation lives out-of-band of your core project you can cycle on it more quickly and once you’ve done things by hand a while you know exactly where the pain points are. When you first start off, you’ll just be guessing. A lot of tooling/automation is coupled to your environment as well, so you may start off deploying locally, but then move to Azure, your tooling will have to change at that point. Start working on automation/tooling once your project is established, is maintaining good velocity and your target environments are well known.
  5. 60/40 Every Sprint, In Every SDLC Phase
    Whether your just starting your app or are in maintenance mode balance the work between features/bugs (the 60%) and tech debt, testing, automation and tooling (the 40%). Break down complex technical debt items into smaller pieces and work on them every sprint. I call the 40% bucket “Preventative Code Maintenance” and it should be the #3 priority on your backlog at all times.
  6. Document your culture and live it
    Have coding contracts and guidelines before you start any project. Your codebase, especially early on, should be unified and feel cohesive. If I got into Mikes code it should feel like Sally’s. Utilize tools like Style/FX Cop and ReSharper to nudge people in the right direction. Utilize Pull Requests, Peer Reviews or Pair Programming to keep your codebase like a well run HOA. No one developer should ‘own’ code, go in clean up code and fix broken windows. Practice Scout Coding at all times. Not everyone on the team needs to be in complete agreement on something, but once the team commits to it everyone needs to be on board. You can either succeed as a team or fail individually.
  7. “Whatever” as a feature
    When your documenting what your application or system should do. Right there should be “It should be performant, it should scale, it should be maintainable and it should be secure”. Even for internal only applications if your app goes down or is slow, it’s costing you money in wasted employee time. You shouldn’t, on a whim, sacrifice performance or security to push out a new feature without the business knowing the full extent of the tradeoff.
  8. Automate Deployments with Smoke Tests
    At first glance this may seem contradictory to #4, but that item is based on timing. When you first start working on the project you won’t be deploying to a production or pre-prod environment. Much latter down the road (hence the timing aspect) you should completely automate your deployments. In the day of Containers, Slot Deployments, CI servers you should never be manually modifying your production environment. One day you will mess up and it could cost you. Yes, the “rm –rf” guy was a hoax, but it’s also a cautionary tale.

We have to balance getting features out or getting the product out with all of the ‘back of the house’ concerns. But we have to remember that we spend our days in the ‘back of the house’. If the architecture it’s good, code isn’t well formed or meets standards, patterns and practices aren’t being followed it’s only going to slow development down, cause bugs and arguments.

If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.

SignalR, Azure and Scaleout


Recently Resgrid started utilizing SignalR to allow real-time updates to clients. If your unfamiliar with SignalR is a server/client Web Socket framework giving an easy API for Microsoft backend server solutions (like MVC, ASP.Net, etc) to utilize Web Sockets. Resgrid has a fairly simple infrastructure: Dual Azure Web App instances for web and api, backend SQL database and workers.

signalrBecause of our multi web server configuration we needed to use a Scaleout system, to allow for clients connected to one web or api instances to receive/send to all instances. If your running multiple web servers and your users trigger SignalR events on one instances, without scaleout only users connected to that same instance will get the event.

SignalR currently supports three methods for scaling out, Redis, Azure Service Bus and SQL Server. Unfortunately with the current version of SignalR (2.2.0) Redis and Azure Service Bus options are in some stages or broken.

This is an unfortunate situation, SignalR’s last release was December 11th 2014, that’s well over a year ago for a pretty critical technology on the Microsoft stack. The last commits on the repository’s master branch (at this time are over 4 months old). If this Github was a physical place, I’d expect to see boarded up windows and tumble weeds. There is light on the maintenance front, SignalR-Server (a different repo) appears to be pretty active.

Why is this an issue? Well because the SignalR repo is over a year old it’s out of sync with the Azure Service Bus assemblies. As of this writing the latest Service Bus Nuget packages greater then 3.0 are incompatible with SignalR Azure Service Bus Scaleout. The error your likely to receive will be something like:

Could not load type ‘Microsoft.ServiceBus.Messaging.MessageClientEntity’ from assembly ‘Microsoft.ServiceBus, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35’

If you can stay in lock-step with SignalR’s Azure Service Bus version (2.1.2) then this is a viable option for scaleout, but for most projects this won’t work. Especially if your utilizing features like Azure Web Jobs, which require a version of Service Bus >3 to use some of it’s features.

Another option is to use a Redis server for scaleout. But it seems that Redis is currently in just as bad (or maybe worse) state. Using Redis for scaleout you may not be able to keep an open connection between SignalR and Redis for any period of time, or at all.

Errors you may receive may be:

SignalR.ScaleoutMessageBus Error: 0 : Stream(0) – Send failed: System.AggregateException: One or more errors occurred. —> System.ObjectDisposedException: Cannot access a disposed object.

System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task) at offset 38
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 40
at Exceptionless.Api.Hubs.MessageBusConnection.<OnConnected>d__2.MoveNext() at offset 322 in C:\projects\exceptionless\Source\Api\Hubs\MessageBusConnection.cs:line 23:col 17

The connection has been closed (Error); no new messages can be delivered; the last command was sent xxxxms ago

In testing on our Azure Web Apps instances talking to the Azure Redis server we found that Redis scaleout is completely unusable. At this point the only scaleout option that seems to work is SQL Server. Although we don’t use this option (we have a hacked version using the latest Service Bus version). Our reasoning is we didn’t want to put any additional load on our SQL server, which might not a concern for your needs.

It’s a sad state of affairs for SignalR. Although not totally abandoned there doesn’t seem to much, if any, resources put behind it. Microsoft has pushed this technology hard in the .Net space and, at a minimum, open and honest communication would be nice.

My guess is that like Entity Framework 6.x they will only release bug fix updates and minor versions, while they spin up a SignalR Core style version, matching the pattern of EF6 and EF Core.

If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.

4 ways to fix YouTube’s Copyright system

I am not a content creator on social media, but I am a consumer. I have been a software developer for quite a while and have had exposure to Patent Infringement claims. As a developer and a business man, I do run a very small startup, I understand both sides of the issue, but our current trajectory is untenable. From the absolute mockery of public domain to aggressive and downright hostile copyright abuse that exists today a backlash is coming and the sooner companies like Google realize it the better.

Word cloud for Copyright infringementYouTube has two major issues that as a consumer I can spot. First is the Content ID system. Content ID is an automated system for cataloging copyrighted audio and video works and assigning them an ID in a database. All videos uploaded to YouTube are scanned and compared against the Content ID database, leading to automated takedowns. The second major issue is DMCA claims that can be filed against a video by a copyright holder. This is a manual submission (Content ID is fully automatic after you upload and catalog your works) but all other parts of the process are automated.



Why is this broken?

YouTube’s copyright system is getting close to 10 years old. This is an eternity in terms of the Internet. That long ago there weren’t people Let’s Playing, live streaming or making a living giving commentary. Ten years ago no one was sure how people could really make money off of the Internet like this and piracy was rampant. Companies like Nintendo, Sega, Sony, Viacom, etc thought that YouTube would be the downfall of their entire business models, through pirating or people watching others use their content. It’s under these conditions that the system as we have today with very little changes was enacted. YouTube was also not getting sued by the people creating content for it’s platform, thus it’s easy of use was heavily skewed toward the companies trying to protect their content.

But as with all things, if it doesn’t evolve it and it tilted in favor of one party over another it will start to be abused.

What are we seeing?

What we are seeing today is the YouTube’s Copyright system is being abused. Companies use the system to claim copyright over Fair Use videos and steal monetization (the ability for a creator to make money off of their videos), put content creators out of business and even more chilling silence critics. This happens because there are no penalties (even if the reporting page says there are) to report false claims. Instantly when you submit a claim or a Content ID match occurs for the next 30 or 60 days, you get the revenue from the video. Even if your claim is false and you withdraw it at the end. Because a channel (the storefront for all the content creators videos) can only have a max of 3 strikes against it at one time this system can easily be used to silence critics and stifle someone’s free speech for a time period.

YouTube is the #2 search engine on the Internet. As a platform it’s given the ability for a lot of people to make a living creating content. These content creators do it all legally, a lot of time utilizing the Fair Use provision to adding commentary or insights. But now they are under a constant risk. Imagine some, or all, of your income for 30, or 60 days could be stolen by a 3rd party. YouTube is popular because of these creators, not because of large companies.

Some ideas for fix the problem

As a developer I can see 4 easy ways to balance YouTube’s Copyright system. These aren’t end-all-be-all fixes, but intended to be things that can be added to the system to make it more fair and curb abuse.

1.) Strike System against False Claims

     There is a 3 strike system if you’re a content creator on the platform. If you get even one strike that can adversely affect your ability to create an upload videos, make money or even have a channel at all. But there is very little issues if you’re creating false claims. Companies should have to file claims on their own YouTube account. Sending out false claims should result in degradation of the claimants channel, unable to upload videos, unable to monetize, etc.

2.) Pending disputes don’t count against the system.

     When a claim is filed on a channel now it’s pretty much an instant strike. This is completely backwards and YouTube is automatically assuming guilt. Which could have been fine in 2005 but now is punishing creators (who are the major reason YouTube is popular) right out of the gate. Any claims pending against the channel (The ones within the 30,60 day response window) should not count against a channel.

3.) Monetization Should Be Held In Escrow (existing or added)

     Right now YouTube is an active participant in the Internets version of wage theft. As the system stands today if you have a monetized video on YouTube and someone files a claim they can draw the process out to the limit then at the very end withdraw the claim and keep the money. The same goes for Content ID, you have a monetized video and YouTube automatically detects content in the database, you money is now diverted to them. All money collected during the process should be held by YouTube (i.e. in Escrow) until the process is complete. This includes all existing and monetization that could be added during the process by either party.

4.) Claimant should pay for monetization stolen

     Content creators time is just as valuable as anyone’s. But the burden of proof is completely on them to submit documentation and information on these claims and thus end up spending a lot of time on this process. So this one is pretty simple, if you submit a claim against a monetized video you should have to pony up what the max spend over the time frame would be. If the max length for any claim is 60 days, then you should have to put into escrow 2 months with of estimated monetization. If your claim is false or invalid you loose the money up to that point. So if it takes 45 days, then you loose 45 days worth of that estimated monetization. That money would then go to the creator who had their time wasted and may split some off to fund a legal defense fund or go to charity.

Go to Top