Posts tagged Architecture

Kicking the can down the road

Being a developer is an interesting profession. On one hand there are an engineering feel to it but on the other hand it’s far more like art. For me calling myself a “Software Engineer” is more for marketing then anything else. But lets face it, little of what developers/programmers do is actual engineering.

kicking-the-canEngineers are certified go through an apprenticeship process then design and create things in the real world that, for the most part, have to stand the test of time. To me Electrical, Mechanical, Structural, etc are the real engineers and us developers haven’t yet earned that right to use that title in the way we do.

How often in a development meeting do we sit there, talk about a design/architecture/code flaw that could cause issues down the road but decide to not even begin to address it? It happens so often we have an industry term for it “Tech Debt”. Can you imagine some structural engineer’s having the same discussion?

Tom: “So I cobbled together this bridge design from a bunch of other actual bridge designs and sample designs and we look ready to go.”

Mike: “Nice, but looking at these designs if we have the amount of traffic we estimate, and we fully expect the bridge to be popular in the near future, it will start leaning to one side”

Tom: “Yea, but we have a deadline, we can always go back and add more supports latter”

Structural Engineers can’t go back and refactor the bridge after it’s been built as easily as developers can with code. There are also lots of other differences, but it really boils down to is that engineers can’t kick the can down the road as easily as developers can, and so we do.

We talk about “performance as a feature”, but what about “ease of maintenance as a feature”, “scalable architecture as a feature”, “security as a feature” or “testing as a feature”? Every time we kick something down the road, label it as “Tech Debt” and put it in the backlog if were being truthful with ourselves we know full well that unless it catches fire we will almost never be back to address it.

Eventually all that Tech Debt will catch up to you and when it does, it can cost you business, alienate customers, hurt your image or even cause your company to fail. I’ve talked about this problem before and at Resgrid I try and follow a 60/40 approach. 60% new features/customer facing bugs, 40% tech debt, testing, automation and tooling.

Here are some guidelines I feel will help stop us from kicking cans down the road and start turning developers into engineers.

  1. Design/Code for the developers around you.
    One thing I’ve always admired about the military is the “Do it for the person next to you” attitude most service members have. The same can be said for the fire service. Sure you start off wanting a thrill, but after the first few times your going in because it’s your duty and because your doing it for your crew. Developers need to have the same mentality, don’t code for yourself or for your company, craft code for the developers on your team. Do it for the person in the cube next to you.
  2. Balance Architecture & Implementation
    I’ve seen my fair share of architecture astronauts in my day and they can do more harm then good. The architecture implemented needs to match the problem domain and how it’s going to be implemented. The architecture you use for a internal LoB app will be different then one that will be deployed on the cloud. No architecture is perfect and it never will be. Ideally you need to design your Architecture for the worst case scenario for 5 years out? How many users do expect in 5 years? Double that, then how are you going to handle that? How many web servers, databases will you need? Is that the scale you should have been using eventing, CQRS, etc?
  3. Code for now and for the future
    If your response to something new is “well that’s the way we’ve always done it” or some variation of it your coding in the past. Yes it’s painful to constantly keep up to date with the latest trends and best practices, but your hurting yourself and the entity you working for by not incorporating the current best practices. You will find it increasingly hard to find developers that want to work, or even know how to work, in your ‘brand new’ code base. I’ve been interviewing a lot of developers lately and almost none of them with less then 7 years experience have ever worked on an WebForms app. Starting a new project? Using WebForms? You will find it very difficult to find developers that know that technology in the near future. This same principal goes for patterns & practices.
  4. Don’t Investing in Tooling/Automation too Early
    When you first starting out a project get the minimum amount of tooling/automation you need. This is a train of thought used by startups. Time spent on setting up elaborate tooling, complex automation is time not spent properly architecting your project, implementing features or fixing bugs. Because tooling/automation lives out-of-band of your core project you can cycle on it more quickly and once you’ve done things by hand a while you know exactly where the pain points are. When you first start off, you’ll just be guessing. A lot of tooling/automation is coupled to your environment as well, so you may start off deploying locally, but then move to Azure, your tooling will have to change at that point. Start working on automation/tooling once your project is established, is maintaining good velocity and your target environments are well known.
  5. 60/40 Every Sprint, In Every SDLC Phase
    Whether your just starting your app or are in maintenance mode balance the work between features/bugs (the 60%) and tech debt, testing, automation and tooling (the 40%). Break down complex technical debt items into smaller pieces and work on them every sprint. I call the 40% bucket “Preventative Code Maintenance” and it should be the #3 priority on your backlog at all times.
  6. Document your culture and live it
    Have coding contracts and guidelines before you start any project. Your codebase, especially early on, should be unified and feel cohesive. If I got into Mikes code it should feel like Sally’s. Utilize tools like Style/FX Cop and ReSharper to nudge people in the right direction. Utilize Pull Requests, Peer Reviews or Pair Programming to keep your codebase like a well run HOA. No one developer should ‘own’ code, go in clean up code and fix broken windows. Practice Scout Coding at all times. Not everyone on the team needs to be in complete agreement on something, but once the team commits to it everyone needs to be on board. You can either succeed as a team or fail individually.
  7. “Whatever” as a feature
    When your documenting what your application or system should do. Right there should be “It should be performant, it should scale, it should be maintainable and it should be secure”. Even for internal only applications if your app goes down or is slow, it’s costing you money in wasted employee time. You shouldn’t, on a whim, sacrifice performance or security to push out a new feature without the business knowing the full extent of the tradeoff.
  8. Automate Deployments with Smoke Tests
    At first glance this may seem contradictory to #4, but that item is based on timing. When you first start working on the project you won’t be deploying to a production or pre-prod environment. Much latter down the road (hence the timing aspect) you should completely automate your deployments. In the day of Containers, Slot Deployments, CI servers you should never be manually modifying your production environment. One day you will mess up and it could cost you. Yes, the “rm –rf” guy was a hoax, but it’s also a cautionary tale.

We have to balance getting features out or getting the product out with all of the ‘back of the house’ concerns. But we have to remember that we spend our days in the ‘back of the house’. If the architecture it’s good, code isn’t well formed or meets standards, patterns and practices aren’t being followed it’s only going to slow development down, cause bugs and arguments.

If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.

To The Cloud!

I’m increasingly becoming the “Cloud Guy” or more specifically the “Azure Guy” in my development community and workplace. There’s nothing wrong with that, I actually love it. I attended PDC 2008 where Microsoft launched Azure and have been playing and using it ever since.

hr-cloud-storageWhen I have conversations with people and companies asking about cloud options usually the conversation is around virtual machines and networks.

The discussions usually center around using the cloud to replace their infrastructure, servers, storage arrays, network equipment, etc. Cloud providers offer excellent Infrastructure to augment or replace your own.

Should you replace or augment your own infrastructure with a cloud providers? Ask yourself a simple question.

“Are you in the business of selling your infrastructure to customers or does it provide a strategic advantage for your core product or service?”

If you answer no to the above going to the cloud is a really good option for you. I was a ‘hardware guy’ for the first six years of my career, at the height I helped build and run multi-million dollar network’s. You have to deal with hardware refresh cycles, endless patching, updates and upgrades so so much more for what is the IT world equivalent of your building’s water and power service.

For the vast majority of companies out there it doesn’t make sense to build and manage that. But if you plan is to replace your physical in-house servers with cloud based VM’s you may be going about it all wrong.

The next question I usually ask is:

“Do you have custom built software is that vital to you business that you are looking to put on the cloud?”

If the answer is no then usually a IaaS solution is best or choosing a completely managed service like Office 365, which Resgrid has been using for over a year to great success.

If your answer is yes then the benefit of the cloud to your software stack isn’t on the IaaS (Infrastructure as a Service) side but the PaaS (Platform as a Service) side.

When you transition to the cloud with an IaaS mindset, you will probably mimic your existing hardware with cloud VM’s. For example a couple VM’s for your website (behind a load balancer that needs to be configured) then some application server VM’s and finally a database VM. In addition there is storage (for your data and your VM’s disks) and probably VLAN configurations to setup. So for a normal n-Tier application you have at least 5 pieces to setup and maintain. Your cloud VM’s still need updates and maintenance but the hardware maintenance is abstracted for you.

But if you go a level higher and migrate your applications utilizing a PaaS mindset you forget about the underlying VM’s. Instead you can leverage the agnostic nature of the cloud and take advantage of automatic load balancing and elastic scaling. Your management also becomes much simpler, I can scale a PaaS app with a couple a mouse clicks, but an IaaS app requires more effort.

Software designed against an in-house network and infrastructure are usually poorly suited to take advantage of PaaS offerings and even to an extent IaaS solutions. The concerns for software development are different in the the cloud then in the LAN.

Transient Fault Handling, Async and Eventing are just a small sampling of things you may need to implement into your application’s code. Even your application’s architecture will change when you design for the cloud first.

Moving to the cloud when you have custom software should be a lot less about the hardware, VM’s, networks and more about application architecture and patterns to best leverage the PaaS offerings that a cloud provider like Azure offers you.

Resgrid is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).

Complex Software Architecture and your Startup

Every day I get exposed to different architectures and designs for systems. It’s a great contrast to see how a large company tends to have pretty complex software architectures compared to software I’ve designed outside the large company environment. There are many reasons for this, some of it is developers trying to show off, justifying their jobs, create job security, trying to be too clever or trying to be too helpful. Maybe the use cases or requirements were over engineered planning for every “What if” scenario under the sun.

The vast majority of the time it’s not malicious or has even a hint of bad intent. But most of the time on the developer side it’s science experiments that turned into production code or developers trying to make something too “pit of success’ie” without contemplating the drawbacks.

image59Complexity carries a heavy burden that most developers don’t want to think about and instead look at architecture as the solution to every problem. But too often when simple problems are over engineered it causes more problems down the line. For a large company, this is ok, they can take the extra time, throw the extra money at it or add manpower. For a startup, especially a bootstrapped startup our best feature is agility, and if our software is too complex to be agile we are trying to win a marathon with our legs tied together.

What are some problems that arise from the dark art of architecture and complex system design and why are they bad for startups?

1. Complexity make debugging more difficult and tracing issues more problematic. The more complicated your system is, the more layers it has is more you will have to debug at 3:42 AM on a Tuesday morning when the production system seizes up. When you’re in a startup environment there usually isn’t a DevOps team to deal with those issues for you at odd hours.

2. Complexity creates friction to change. The more complexity the more you need to refactor or recode when you change directions. It’s also more time, and effort you loose when you throw away that system. You will be throwing away code, and lots of it over the life of your startup. When you go in add a new feature or recode something you can then go in and refactor.

3. Complexity systems take more time to code. In a startup environment the easiest, quickest, hacky, ugliest solution is almost always the right choice. The less time you spend on things the quicker you get stuff done and to your customers getting feedback. Delivering is the most important thing. This isn’t to say you should push out totally buggy unusable systems, they need to work.

4. Architecture is not a deliverable. In the vast majority of cases this is true, but there are cases where this may be false. If your designing a system to say show people how to cook, your users don’t care how may layers you have, how you got CQRS and async full stack with DI and 100% coverage. Are those cool things? Yea, but they aren’t as important as us developers like to think they are.

5. It’s hard to bring new developers into complex system. In a startup environment development is probably one of the least valuable things you can do. As a founder you probably need to spend more time on business development, customer acquisition, planning features, marketing, etc. So you’ve reached that point and you want to bring in a technical founder, consultant or employee, the more complex your architecture the harder it will be for the new people to learn and longer it will be before they are up and running efficiently.

I’m not ‘anti-architecture’ or ‘anti-patterns’, instead I’m of the mindset that there is a time and a place for science experiments, architecture and complexity but rarely are those good for your early stage startup. I’m still trying to fight my inner architect when I develop for my startup, it’s difficult, but next time I’m up at 3:42 AM debugging an issue I’ll be a little more happy.

Go to Top