Unless you are either very new or have no connection to the software delivery profession and industry (in which case I’d be curious to hear how you found this blog post), you have probably heard of “DevOps”. DevOps is “the union of people, processes, and products, to enable continuous delivery of value to our end users.” What this actually means for developers, delivery teams, and organizations is often not very clear. But everyone says you have to do it, so it has got to be good, right?

Below are 10 reasons that implementing a DevOps transformation might not be a good idea for you:

1. It’s okay; you know what you’re doing

It says it right there in the formal definition of DevOps: DevOps is about enabling delivery of value to our end users. That might be great for some teams, but you don’t need that kind of distraction. You know exactly what you need to build – the project plan is very clear about it. Besides, your end users will use whatever you build for them. They are users in the end; that’s what they do: use.

2. With DevOps you must continuously deliver

Again, it says it right there in the definition: “…enable continuous delivery of value…” You have enough trouble as it is with your semi-annual releases; imagine how tough it would be to have to do that multiple times a day! Sure, if you release every few weeks, let alone every day, your releases would be simpler and smaller, but don’t they always say that size doesn’t matter, and that it is the thought that counts? Well, thinking about continuous delivery is scary, and that’s that.

3. DevOps destabilizes corporate structure

From a practical standpoint, DevOps is about having software developers collaborate with operations engineers, and that just won’t do. Your organization is set up in silos; you even have developers and ops folks in different buildings, just to make sure that they don’t start giving ideas to each other. Give them an inch, and they will start conspiring together – operations engineers will make requests for making the product more operable, and developers might get operations to help them deliver more frequently, and you just established how scary that is!

4. Automation upsets the workforce

A core aspect of DevOps is the extensive use of automation to build, deploy, and test the software. If you allow development teams to set up automated build pipelines, what are the build masters going to do? If you shift all of your tests from manual to automated unit tests what are you going to do with your QA engineers? If you automate your deployments, let alone the creation of test and production environments, what are your operations engineers going to do?!?

Sure, automated tests cannot replace manual tests completely, and there will always be important exploratory tests for QA engineers to execute. Of course your operations engineers have a lot of work to do, just keeping the production site up and running, but you cannot take the chance, can you?

5. DevOps makes you deliver faster

One of the outcomes of adopting DevOps practices is that teams can deliver software to production faster than before, or in business terms: faster time to market. Not so fast! Haste makes waste, after all, and you don’t want to create waste, do you? Besides, if you somehow manage to deliver faster than the competition, you might be forced to innovate, rather than follow other organizations down a well-known path. Besides, you might not even be a consumer facing organization, and there might not be any competition. Your users will just have to accept that this is how you roll.

It’s not like if your leadership decides that you are not delivering fast enough, they might decide to outsource your work to a software firm that can move faster. Right..?

6. Constant feedback can be confusing

The early and continuous feedback that is a big part of DevOps deliveries can be overwhelming. Whether it is from stakeholders at a sprint review, or input from your product’s telemetry, that your developers enabled the operations engineers to gather and analyze, the feedback might surprise you as to what your users are doing with the software you delivered, and might indicate that you should invest more time further developing features that you thought were fine as they are, or worse, might suggest that you cease efforts in developing a feature that you have planned to develop for another three or four weeks. Either way, this feedback might disagree with the plan that you spent months putting together before a single line of code was written. That can be very upsetting! What should you do in that case? Just ignore the plan?!? Not plan so much in the first place?!? What???

7. Production telemetry helps detect defects

As if it is not bad enough that operations engineers use production telemetry to try to influence what the product does by suggesting changes to the plan, they also use it to detect defects! This completely violates the DRY principle (Don’t Repeat Yourself)! You already have testers and a QA team that are supposed to find all of the defects in the product. If telemetry uncovers any problems that you didn’t already know about, it could make you look bad, right?

Sure, if you have build, test, and deployment automation in place, you could possibly fix the problem before the users begin to complain, but you already decided that automation is not a good fit for your organization.

8. Autonomous engineering is scary

DevOps teams take responsibility for the entire software delivery process, from development to production. DevOps teams operate with a high level of autonomy, and that is scary! How can you know what the developers are doing if they are not going through all of the bureaucratic tape that your organization put in place to inhibit excessive productivity and effectiveness?

Of course, modern tooling can ensure that engineers are not doing things that they are not supposed to do, but you cannot trust computer systems to consistently apply your policies. Besides, if the teams can simply do whatever is needed to deliver software effectively, without your say so, what will your job become?!?

9. DevOps promotes engagement

DevOps teams are known to be happier, more engaged, and better motivated than their less autonomous counterparts, and we cannot have that. You pay your engineering teams to deliver features, not to be happy!

10. DevOps teams are more effective

DevOps practices enable teams to be more efficient and reduce costs associated with organizational waste… Hmm… perhaps that means that you can lay off some people and do more with less..?

Grade your self…

Count how many of these arguments seem like a good reason to avoid implementing a DevOps transformation in your organization. If you scored more than zero, you might want to think about the value proposition – enabling continuous delivery of value to you end users. If customer satisfaction is important to your business, it might be worth the effort.

If you think otherwise, say so in the comments – let’s have a chat!

Summary

Despite the sarcastic tone of this post, every single one of these points describes one of the virtues of DevOps, and how adopting its principles and practices can help your organization. I hope you see the value.

Cheers,

Assaf

One thing that sets high-performing individuals and teams apart from lower performing teams is experience. Experience is a shortcut. It saves time. It helps you skip the time and effort required to research a particular problem. Experience lets you cut directly to the solution.

Experience allows you to skip the questions.

Unfortunately, experience can also give you a false sense of security. You feel secure that you know all of the answers to the relevant questions, that you know the complete context of the problem. You assume that nothing has changed since you have last evaluated the problem space.

That assumption can be wrong, and detrimental to your work.

A Short Story

So, once upon a time, about a month or so ago, I was running a hackathon with a team of developers. They were a great team, coming together for the first time in the organization’s history to solve a problem agilely, that is to say, all hands on deck – developers, testers, operations engineers, and the product owner sitting together – working together towards a shared common goal, to the exclusion of all other unrelated tasks.

At some point we realized that we have a dilemma regarding the form of the output of our feature. The default way, the way that their primary customer has established with them a long time ago, and the way that they have been working ever since, was to provide a full “baseline” dataset, in the form of a CSV file, and then provide daily “modification files”, i.e. files that add new records or remove the old ones.

The problem was that we now have to support not just the primary customer but many other customers, some so small that they have little in the way of an IT department capable of writing a process to take monthly baselines and apply daily changes automatically. The solution that some of the team came up with was to send them full daily datasets. Each day they were to use the new CSV as the sole source of truth. These small customers could consume the full daily files with Excel, with no additional IT work needed.

Of course, a lively discussion ensued. Part of the team said that we cannot change how things work because our primary customer relies on it. The other half argued that the many smaller new customers simply cannot deal with the “old way”.

After letting them discuss it for a while I proposed we call the customer representative. The team leader did. After a short introduction, referring to the new feature (which they’ve previously discussed), the team leader, a proponent of the old solution asked “do you want us to send you the files in the same way we did before with the other features, that is monthly baseline with daily changes?” The customer representative said that “yes, it would be fine”.

I thought about it for a moment. Our many new smaller customers needed a different solution. We needed to reframe the question. I asked the team leader to relay the following question to the customer: “Would it be acceptable if instead of using the old method we’ve had before, for this feature we will send full daily datasets?”

The customer representative’s response was, and I believe I am quoting her verbatim, as follows:

“Oh my god, yes! That would be so much simpler for us to implement on our end!”

High-fives and fist bumps were had across the entire room. Team spirits went up dramatically.

The Moral

The team leader was very experienced with the system we were modifying, and very familiar with the customer. They talked with one another regularly. This blinded him from the simple truth that while the current solution was acceptable, a better solution would be greatly appreciated.

The rule I drew from this experience is this – if you want to change something, never immediately assume that current conditions are immutable. If you’ve never discussed the possibility to change, you cannot know whether others – your partners, customers, teammates, or managers – would resist, or the reverse – enthusiastically gush and support the change. Perhaps they feel the same pain that you do, the pain driving your desire for change, and merely lack the initiative to stand up and do something about it.

In short, don’t hesitate to ask. Phrase the question in a way that benefits you, and highlights the change you wish to make. It cannot hurt. You might be pleasantly surprised with the results!

So you’ve heard about DevOps. That is great. You’ve decided that your organization could really benefit from a “DevOps transformation”. Even better. You’ve even gone above and beyond, and memorized the following definition for DevOps:

DevOps is the union of people, process and technology to enable continuous delivery of value to customers.

Awesome.

But what should you do next? How do you deal with the fact that the developers in your organization cannot or will not own the process of delivering into production? How do you deal with the fact that your operations engineers cannot or do not fully understand the software to which they have been given stewardship?

How do you get your “Dev” and “Ops” to work together? How do you become a “DevOps” organization?!?

Method #1: You Build It – You Operate It!

If your developers are already, among other things, for whichever reasons, tasked with deploying software, monitoring it in production, and dealing with live site incidents (LSIs), then congratulations, you are already half way there. More than half, really. The organizational structure, tearing down the walls between “Dev” and “Ops” is as difficult as it was about a decade ago, when the Agile movement introduced the notion of tearing down the walls between testers and programmers (a.k.a. “Dev” and “QA”), as well as that between developers and business (i.e. Scrum teams consisting both developers and business people). It is even more difficult in some organizations, where developers report to the VP of R&D, while operation engineers report to the VP of Operations; the first common manager who connects both departments is the CEO. Not the person who should be burdened with getting these two realms to cooperate.

Other organizations are not so lucky. But do not despair; the following methods describe how to enable an organization’s DevOps efforts through collaboration of separate “Development” and “Operations” departments/teams.

Method #2: Embedded Ops

In an organization where development and operations exist in two separate hierarchies (a.k.a. “silos”), if you have enough operations engineers, you should assign – embed - one in each development team. This engineer would be responsible to field all of their development team’s questions and requests, and would be responsible to help the development team deliver their software into production.

This setup has the benefit of being able to create self-managing teams who ultimately own the responsibility of deploying software to production, and making their product “ops-friendly” (e.g. automatically testable, deployable, include monitoring hooks, etc.), while maintaining a governed center of excellence (COE) that defines how software should be built, deployed, tested, and monitored.

If you have more operations engineers than development teams (yes, I know how unlikely that is…), the remaining engineers can focus on cross-cutting concerns such as monitoring overall systemwide health, and creating tools to be used by the teams for their operational efforts.

The embedded Ops engineers can, instead of merely handling the operational workload for their respective teams, help the developers cross-train so that they can become more independent, allowing such teams to move towards the first method. This can free up the embedded engineer to rejoin the main Ops team, or to move on to helping a newly formed team, or an existing team that requires extra attention.

Method #3: Liaison Ops

Many organizations, like those described in the previous section have separate “dev” and “ops” departments. Unlike the aforementioned group, however, these organizations do not have enough operations engineers to dedicate to each development team. In this case, we want to put each Ops engineer in charge of serving several teams. This engineer will have to help each team, and balance their workload with the needs of all of the teams with whom they are liaising. The liaising operations engineer will have the same responsibilities and roles as do the embedded engineers; the only difference is in the number of teams they support.

Sooner or later, liaising Ops engineers become bottlenecks, or at least find scheduling to be difficult. For this reason, if the embedded Ops engineers have the option to train their teams to operational independence, the liaising ops must make training their developers for independence a high priority.

A question that invariably comes up is “how many teams can a liaising operations engineer support?” The answer is, of course, “it depends”. It depends on both the engineer and the teams.

Method #4: The Operational Service Team

Sooner or later, organizations with embedded or liaising operations engineers want to – or must – gravitate towards the servicing model. In this model, the operations team does not support individual development teams directly, except in extreme cases. If an engineer from this operations team has to help a development team deploy their code or debug it in production, then there was a procedural failure at some point.

The role of the service team is to create tools, skills, and knowledge that enable development teams to deploy and own their code in production. This may include creating scripts and templates that automate the build and release processes, setting up automated security and quality checks as gates for continuous delivery, setting up dashboards, system health checks, etc. If it is unique to a team’s product, the team owns it. If it is a cross-cutting concern, the ops team owns it. The ops team provisions environments, or enables teams to provision environments within guidelines and constraints; the developers consume the services.

The Operational Service Team may also be responsible to create DevOps-related knowledge for the teams to consume. This may come in the form of any thing from wiki pages to providing formal or ad-hoc training.

Note that this method is very similar to the first method. In both cases, the development teams are responsible for their code all the way in to production. The primary difference, is that in the first method, there is no operations team; in the fourth (this) method, the operations team serves as a center of excellence.

Summary

Hopefully, this post will help you determine where your organization is in its DevOps journey, and help you figure out where you want to be.

Good luck, and safe journeys,

Assaf

In the previous post we discussed what WIP (Work In Process/Progress) is, and how to track it. In this post I want to discuss why WIP limits are so important and how they contribute to improving the team's effectiveness and throughput.

So There's This Little Law…

Little's Law describes the relationship between the average throughput in a system, the average process time (a.k.a. Cycle Time) for an item in the system, and the amount of items being processed at the same time (your WIP). For the mathematically inclined reader, this relationship is described in the following formula: L=λW, or in other words, the long-term average WIP (L) is equal to the long-term average throughput (λ) multiplied by the average wait/process/cycle time (W).

This theorem is both simple and profound; while the formula is intuitively easy to grasp, it also means that it is unaffected by other factors such as the arrival process distribution, service distribution, order of process, or for all intents and purposes – anything else!

This law therefore holds true for both simple and complex systems of any nature.

Okay. So this is interesting (humor me here – let's assume that if you're reading this post you are finding this interesting) but what does this mean?

Applying Little's Law to Product Development

If we play around with the formula, we will arrive at a way to determine the process time – or in product development terms, the amount of time to develop a feature – by dividing the WIP by the system throughput, or W=L/λ. This means that we can reduce the amount of time it will take us to implement features by limiting the amount of items we develop concurrently!

This too should be intuitive. Even if we completely ignore all of the disadvantages of multitasking, context shifting, resource allocation, deadlocking, and conflicting priorities, it is simple to see that if we could, for example, complete only 80% of the planned work, it would be more valuable to have 80% of our items 100% done and delivered, than to have started work on all 100% of our items, but each only 80% complete, and in no shape to ship. Yes in the described scenario, as a manager, you could find how to blame your developers for not completing 100% of the planned work, but this was preventable! That's on you!

However, there is another subtle application that is implied by the modified theorem. WIP predicts process time!

Predicting Process Time based on WIP

Predicting process time is not new. We just like to call it estimating. Estimation is defined as making an extremely hopeful guess about the amount of time it will take to develop or (and?) deliver something, based on personal or shared past experiences (or more often, an arbitrary demand, dictated by someone in a higher tax bracket than the people doing the work). Okay, so perhaps it is not defined quite like this, but in all honesty it really should be.

Regardless of what technique (if any) you use to come up with your estimates, all of the data that feeds your estimates are based on the past. In other words, they are trailing indicators. This means that the numbers come after the fact – they could be used to explain the previous items' process time, and we statistically assume (read: hope) that they will hold true for our upcoming work.

By contrast, your WIP is a leading indicator! This means that the WIP predicts (affects) the process time for items entering the system, I.e. not yet developed.

Let's look at the following Cumulative Flow Diagram from a VSTS project:

Let's try to reduce the work in progress. We will do this by focusing our efforts on closing items that are already active instead of beginning work on new items. This will be represented by taking items out of the Active state, and putting them back in the New state, and in return, moving items from the Resolved state into the Closed state. Doing so will result in the following modified cumulative flow diagram:

Note that as our WIP is reduced, so does our process time! We have reduced the time to deliver an item – any item – doing nothing more than limiting the amount of work the team processes concurrently! And this doesn't even account for the extra benefits that knowledge work such as software development derives from focusing on a single small unit of work!

Conclusion

In this blog post we've discussed Little's Law and its implications for product management. Likewise, we demonstrated the importance of WIP limits as leading indicators for process time, and how reducing the amount of work we have in process concurrently reduces the amount of time to process items through the system. Armed with this knowledge I hope you will be able to make use of this because all you need to do is not take on a new job before you are done with (most of) the ones you've already begun.

Stay lean because you can(ban),

Assaf