Category Archives: Architecture

You are not faster with an 80% solution

When the schedule is tight for a software development project, one of the first things non-technical people contemplate, is reducing quality and/or scope and go for an 80% solution. This can work but only under particular circumstances.

A recent discussion I had with some folks was about such a situation. A very tight deadline had been issued by senior management and multiple departments had to collaborate on a solution that spawned multiple parts of the organizations and their core IT systems. Those systems (think about something like ERP and CRM) have a multitude of connections, grown over many years without governance, but not for the data that were required now.

The people were all quite supportive and a constructive discussion took off. Not surprisingly, finding a common language took some time. Especially so because only after a while did we realize that in several cases the same term had a different meaning. In the end there was a particular detail where no obvious solution was available.

I was responsible for the technical implementation on one of the two systems involved. My counterpart on the business side and I discussed various options and and eventually he came up with the idea to just safe time by going for the 80% solution. We talked about a number of aspects and it soon became clear to me that this would actually require more time than to just “do it properly”.

This sounds, at the very least, counter-intuitive for most people. So let me illustrate my point with a non-technical equivalent. Many years ago I had to prepare for my then-boss a list with press clippings every day. It basically meant to go through a long list, that was created automatically, and remove everything he was not interested in. This was quite time-consuming and he was genuinely surprised when he found out.

His statement then was “Why do you need so much time, it is only 20 of the 200 articles that are relevant”. That was of course correct. But what he had not thought about was that I had to go through the list of all 200 articles and check some in more detail than just looking at the headline. So it did not really make a difference whether I cut the list down to 100 or 60 or 20 articles, the time required stayed more or less the same. He agreed on the point and quite soon decided there was better use for my time.

It is exactly the same with taking shortcuts during a technical implementation. The easy part is to decide that you want to cut 20% off the next release. But things get tricky when it comes to deciding which 20% to skip without any undesired side-effects. In order to be able to do that, you need to understand in depth what the consequences would be. So quite often you spend more time trying to understand potential ramifications for various scenarios, than it would take to simply do it properly right from the start.

And that is only the beginning of the problem. You need to document the results of your analysis really carefully, if you want to be sure that nothing is missed when the first code change is required. Even a seemingly minor bug fix has an increased potential to break something if the current overall solution is built on a set of assumptions. And adding enhancements or new features will be even more “fun”. Because people have to read through all the documentation and also understand it. Together this is a time-consuming task with considerable risk to still overlook something (or hit an edge-case that was not documented).

But in reality this documentation does not exist anyway because you wanted to save time in the first place. And detailed documentation is not exactly supportive here. So people throw in bits and pieces and you end up with a document that is typically even worse than most project documentation that is out there. So whoever has the pleasure to work on the code without having all the details in mind is really screwed. That includes you, by the way, because you will have forgotten most after only a few weeks.

The solution is usually a re-implementation because it is faster than to fix the broken code. Of course that re-implementation often needs additional logic to handle data migration for transactions that were completed with the 80% solution and now ruin your data quality. But hang on, that might just be 20% effort you can safe right now ….

On Micro Services

Just a couple of thoughts on micro services, triggered by a German blog post on heise Developer. So let’s go …

I have read quite a few articles on micro services, and literally all of them were overly optimistic and just scratching the, of course, positive surface. The examples are usually just slightly more complex than “Hello World” and also completely focused on technical aspects. For a non-deep dive technical exercise these things are ok. But nobody should fool him- or herself: the real world is much more complex.

There is, by the way, a real déjà vu here for me. Many things sound remarkable similar to what was said ten years ago in the context of SOA (Service-Oriented Architecture). Then, also, things like modularity and increased agility were cited all the time. (Re-use was the other standard one, but that did not work in most cases.) So what is really different now? I honestly fail to see anything new.

Instead there are arguments, again remarkably similar to SOA, in favor of faster possibility to adopt, easier deployment of change, and smaller and less complex code. Well, they are wrong. The key here is complexity and what its consequences are. The core complexity comes from the business requirements and not the chosen technology or architecture. And no matter how you slice, dice, and re-arrange things – the business complexity simply stays. Your only chance for survival is to find a clever way to layer things in a manner that makes sense for your business.

You will find that there are things that lend themselves well to (micro) services and of course that is how they should be approached. But you do not really have to have a separate deployment unit for every Mickey Mouse service. Just apply common sense and also keep in mind that things are in constant flux anyway.

So instead of spending much time on micro services, I would rather recommend to think about ways, which ensure that your system/architecture can easily adjust. And from a business perspective that is, of course.

Re-Writing Software from Scratch

It is not uncommon that an existing piece of non-trivial software gets re-written from scratch. Personally, I find this preference for a greenfield approach interesting, because it is not something I share. In fact, I believe that it is a fundamentally flawed approach in many  cases, so why are people still doing this today?

Probably because they just like it more and also it is perceived, at least when things get started, as more glorious. But when you dig deeper the picture changes dramatically.  Disclaimer: Of course there are situations when re-starting is the only option. But it typically involves aspects like required special hardware not being available any more.

When I started writing this post with making some notes, it all started with technical arguments. They are of course relevant, but the business side is much more critical. Just re-phrase the initial statement about re-writing to something like

Instead of gradually improving the existing software and learn along the way, we spend an enormous amount of money on something that delivers no additional value compared to what we have now. For this period, which we currently estimate to be 2 years (it is very likely to be longer), apart from very minor additions, the business will not get anything new, even if the market requires it; so long-term we risk the existence of the organization. And yes, we may actually loose some key members of staff along the way. Also, it is not certain that the new software ever works as expected. But should it really do, the market has changed so much, that the software is not very useful for doing business anyway and we can start all over again.

Is there anyone who still thinks that re-writing is generally a good idea?

Let us now change perspective and look at it from a software vendor’s point of view. Because the scenario above was written with an in-house application in mind. What is different, when we look at a company that develops and sells enterprise software? For this text the main properties of enterprise software are that it is used in large organizations to support critical business processes. How keen will customers be to bet their own future on something new, i.e. not tested? But even if they waited long enough for things to stabilize, there would be the migration effort. And if that effort comes towards them anyway, they may just as well look at the market for alternatives. So you would actively encourage your customer base to turn to the competition. Brilliant idea, right?

What becomes clear looking at things like that, is what the core value proposition of enterprise software is: investment protection. That is why we will, for the foreseeable future, continue to have mainframes with decades-old software running on them. Yes, these machines are expensive. But the alternatives are more expensive and in addition pose enormous risk.

In the context of commercial software vendors one argument for a re-write is that of additional revenue. It is often seen as easier to get a given amount of money for a new product than an improved version of something already out there. But that is the one-off view. What you typically want as a software vendor is a happy customer that pays maintenance every year and, whenever they need something new, first turns to you rather than the competition for options. Also, such a happy customer is by far the best marketing you can get. It may not look as sexy as getting new customers all the time, but it certainly drives the financial bottom line.

Switching over to the technical side, there are a few arguments that are typically made in favor of a restart. My favorite is the better architecture of the new solution, which will be the basis for future flexibility, growth, etc. I believe that most people are sincere here and think they can come up with something better. But the truth is that unless someone has done something similar successfully in the past, there is a big chance that the effort is hugely underestimated. Yes, technology has advanced and we have better languages and frameworks. But on the other hand the requirements have also grown dramatically. Think about high availability, scalability, performance and all the others. Even more important, though, is the business side. With something brand new people will have greatly increased expectations. So giving them something like-for-like will probably not be seen as success.

The not-invented-here syndrome is also relevant in this context and particularly with more junior teams. I have seen a case when an established framework used in more than 9,000 business-critical (i.e. direct impact on revenue) installations was dismissed in favor of something the team wanted to develop themselves. And I can tell you that the latter was a really bad implementation. Whether it was a misguided sense of pride or a fundamental lack of knowledge I cannot say. But while certainly being the most extreme incarnation I have seen so far, it was certainly not the only one.

So far my thoughts on the subject. Please let me know what you think about this topic.

Revisiting Software Architecture

Quite recently I heard a statement similar to

“The application works, so there is no need to consider changing the architecture.”

I was a bit surprised and must admit that in this situation had no proper response for someone who obviously had a view so different from everything I believe in. But when you think about it, there is obviously a number of reasons why this statement was a bit premature. Let’s have a look at this in more detail.

There are several assumptions and implicit connotations, which in our case did not hold true. The very first is that the application actually works, and at the time that was not entirely clear. We had just gone through a rather bumpy go-live and there had not yet been a single work item processed by the system from start to finish, let alone all the edge cases covered. (We had done a sanity test with a limited set of data, but that had been executed by folks long on the project and not real end users.) So with all the issues that had surfaced during the project, nobody really knew how well the application would work in the real world.

The second assumption is that the chosen architecture is a good fit for the requirements. From a communication theory point of view this actually means “a good fit for what I understand the requirements to be”. So you could turn the statement in question around and say “You have not learned anything new about the requirements since you started the implementation?”. Because that is what it really means: I never look back and challenge my own thoughts or decisions. Rather dumb, isn’t it?

Interestingly, the statement was made in the context of a discussion about additional requirements. So there is a new situation and of course I should re-evaluate my options. It might indeed be tempting to just continue “the old way” until you really hit a wall. But if that happens you have consciously increased sunk costs. And even if you can “avoid the wall”, there is still a chance that a fresh look at things could have fostered a better result. So apart from the saved effort (and that is only the analysis, not a code change yet) you can only loose.

The next reason are difficulties with the original approach and of that there had been plenty in our case. Of course people are happy that things finally sort-of work. But the more difficulties there have been along the way, the bigger the risk that the current implementation is either fragile or still has some hidden issues.

And last but not least there are new tools that have become available in the meantime. Whether they have an architectural impact obviously depends on the specific circumstances. And it is a fine line, because there is always temptation to go for the new, cool thing. But does it provide enough added value to accept the risks that come with such a switch? Moving from a relational database to one that is graph-based, is one example that lends itself quite well to this discussion. When your use-case is about “objects” and their relationships with one another (social networks are the standard example here), the change away from a relational database is probably a serious option. If you deal with financial transactions, things look a bit different.

So in a nutshell here are the situations when you should explicitly re-evaluate your application’s architecture:

  • Improved understanding of the original requirements (e.g. after the first release has gone live)
  • New requirements
  • Difficulties faced with the initial approach
  • New alternatives available

So even if you are not such a big fan of re-factoring in the context of architecture, I could hopefully show you some reasons why it is usually the way to go.

Martin Kleppmann – Conflict Resolution for Eventual Consistency

Here is a rather interesting video from Martin Kleppmann where he talks about dealing with concurrent changes to data. While the title may sound theoretical to some, it is a topic that probably every developer has come across. And here is also the link to the paper with the algorithm presented. If you are interested in an implementation, check this Github project.