Category Archives: Architecture

Performance and Thread Count

I have seen many mails and forum posts, which sort-of imply that increasing the number of threads working on a task will speed things up overall. This is a dangerous path to follow in many cases, because it has the major drawback of adding overhead.

In some cases there is a range where the increase of parallelism indeed improves the situation. But that is only because of very particular performance characteristics. Basically what needs to fit is the ration of in-system execution time vs. wait time for other systems. And the critical “side” condition for the other systems is that there is no resource congestion/conflict there: I.e. if you decide to add more load (by increasing the number of threads on your side) to an external database that is already maxed out, how do you expect your code to run faster?

Lastly, there is neither a one-size-fits-all number of threads to recommend, nor can one always say that adding threads will make things better at all. The opposite can happen quite easily and it always needs proper testing with full load. Putting the latter onto the system is harder than most people think, unfortunately. But the benefit will very often be a profoundly improved understanding about the application and especially its TRUE bottlenecks. Parallelism will almost always yield results very different from a single-threaded execution (e.g. when run from Designer).

Let me close by saying that in a surprisingly high number of scenarios, and again there is a number of pre-conditions associated, a single-threaded execution will give the highest performance (which we have not defined as a term at all here, by the way). This is because the overhead mentioned at the beginning kicks in. If you manage to have your code run such that you reduce or even eliminate cache misses on the CPU, you can see improvements by a factor of 10,000+ (and that is really not percent but a factor). But in general the effort to achieve that outweighs the usefulness; unless we talk about things like high-frequency trading of course.

webMethods Integration Server: How to Structure Packages

For the webMethods Integration Server (IS) packages are “containers” that bundle together various assets and get deployed as sort-of atomic units; this makes them more or less comparable to web apps in a servlet container.  So if you are given a certain task to solve, you need to figure out a clever way to organize the various parts of the overall implementation. Unless we talk about something trivial and short-lived (ever seen?), this is more complicated than it might appear at a first glance.

I did my first project on Integration Server 4.6 in early 2002 and can now openly confess, that the result, while working fine from a business perspective, was not something I would still be proud of today. To a small extent, the hardware equipment can be blamed. I was working with a notebook that had 256 MB of RAM, a single-core 800 MHz mobile Pentium III, a 20 GB hard disk with a whopping 4200 RPM, and a 14 inch display with a 1024×768 resolution.

You may now ask, how this can affect code quality. Most important in my case was performance. If pressing the save button literally sends the machine into a 2+ minutes period of frantic disk activity, you think more than twice before doing this. (Eventually, I got a memory upgrade to 512 MB.) And this is only the tip of the iceberg. In summary, refactoring is very hard if your concentration is constantly drawn away, because you have to wait for your machine to be ready again. Luckily we are past those issues today.

For the sake of this article let’s work with the following scenario, which is close to something I worked on about two years ago:

The business requirement is to automatically perform modifications to certain sales opportunities in the CRM system and report the result to the account manager.

Of course you can put everything in one package. There will be no need to think about dependency management and much less work for setting up the DEV environment, the CI server, deployment scripts (Chef in my case), etc. And in fact this is how I started. Pretty soon I moved some really generic utility services like data type conversion or date calculation into a separate utilities package. But other than that things stayed quite monolithic.

It didn’t last long, though. When I was half-way through with the initial implementation, the next, more or less separate business scenario that involved the CRM system showed up. Obviously, there was no point in having connection details, services, and other configurations related to the latter multiple times. So I moved all CRM-related stuff to yet another package.

To be clear: I was not taken by surprise when this happened. Instead it was a deliberate decision to be very careful with upfront framework building. This has been one of my personal rules for a while and it has the great advantage of being efficient as well as effective. The efficiency comes from the fact that no time is spent on things that later turn out not to be needed a second time. More important, however, is the effectiveness. It means that the outcome is useful. If you ever came across an API that is cumbersome to use, you know what I’m talking about.

Coming to an end, here is the abstract description of how I structure my IS packages. There are certainly other ways, but this works really well for me and not once in many years has caused issues or created that gut-feeling, which tells you that something is not right, although you cannot say what it is.

  • Business logic: Here is where specifics of the business side are handled. What are the criteria to select an opportunity for processing? What changes shall be applied? Technical details of the CRM system are not in scope here.
  • System details: How to perform a query against the CRM system using the criteria coming from the business logic level above? What are the connection details for a given environment (DEV, TEST, PROD)? Not in scope is e.g. the implementation of auditing.
  • Common topics: How to send notifications (e.g. where is your mail server)? Auditing, dealing with data structures, date manipulation, logging, etc.

In terms of naming conventions I suggest the following

<COMPANY_INITIALS>_<TYPE>_<DESCRIPTION>

Here are a few examples for the famous Acme Corp.

  • AC_APP_CrmOppCheck   : Application that performs checks on sales opportunities
  • AC_CMN_Audit                     : Auditing
  • AC_CMN_Util                         : Utilities
  • AC_SYS_CRM                         : Connectivity and services for CRM system
  • AC_SYS_ERP                          : Connectivity and services for ERP system

If you have questions, please do not hesitate to ask in the comments.

It’s Not Beautifying but Reduction of Complexity

I am writing this on a Saturday afternoon after having spent several hours going over Java code that is about a year old,  needed only occasionally, and has not caused issues for quite a while. Still I spent personal time to improve or, shame on me, even add the Javdocs. Also, some additional error checks were added and the code structure simplified.

None of this had been asked for by anyone else. And probably quite a few people would call it a terrible waste of time, given the various new features still needed so badly. So why did I do it?

It was an experiment, a successful one  from where I stand. In the beginning I merely wanted to see in how many ways the code could be improved with sufficient time gone by after the initial writing. I ended up doing the following things:

  • Renamed class ServiceWrapper to ServiceWrapperManager, because the class is not an actual wrapper but creates and deletes them.
  • Renamed variable adapterServiceNode to node, because via a wrong user input it could point to a node object of a different type and not an adapter service. (The same was done in two other cases.)
  • Added multiple checks that would throw an IllegalArgumentException, if the user had provided wrong input. So rather than having to wonder why the creation of a wrapper had failed, I would be “shouted” at early in the process.
  • Added and improved multiple Javadoc entries. This, in turn, made me look at the code more closely where I discovered multiple things to improve or even fix.

When I look at the code now, it is cleaner, more robust, and easier to read. And the latter is the most important aspect on my case, because it reduces technical debt. So here you have your argument.

And for business people this means that I “keep my head above water level”. Or in other words: It is an absolute prerequisite for fast time-to-market that my code base is tidy and well-structured. Only then can I react quickly and change things without having to think hours what might break in other places, should I now add this or that new feature.

Job Interview Question: What is Your Favorite Algorithm?

While watching the video below, I was intrigued, when this interview questions was discussed (timestamp 56:30-57:35).

It made me think about my response, had I ever been asked the same. And it did not take too long before the answer was clear: the B-Tree. There is a very good section on it in Martin Kleppmann‘s book “Designing Data-Intensive Applications“. And I highly recommend this book anyway.

But as a starting point on B-Trees the following video is also quite helpful:

Structure of Development Teams

I just came across another interesting statement about the size of development teams:

The good systems that are presently working were written by small groups. More than twenty programmers working on a project is usually disastrous.
 

What makes this so interesting is its origin. It was said in 1968 during the famous NATO Conference on Software Engineering. Fifty years later I think this is still remarkably true. But why?

In my post on the size of development teams I was aiming at complexity and scope as drivers. The interesting question, though, is whether there are additional points to consider. Having worked pretty closely with some other projects in the meantime, I came across another influencing factor: availability of qualified staff.

You can look at this in several ways. The first hurdle is obviously to have enough developers with the required experience at your disposal. The second is how to allocate their capacity to the portfolio of activities you run. Or put differently: How do you deal with the shortage of know-how you face? Because, let’s be honest, there is always more demand for highly skilled developers than can be supplied. (I am having a déjà vu with my “beloved” macro economics lecture 😉 back in 1996.)

If your organization is above a certain size, chances are that overall a sufficient numbers of good people work there. Does that put you in a better position than some poor fellow working for a small company? Not really. Because your project competes for these folks with all the other work within the organization. And what is the difference compared to competition with an external market? Yes, if you have really good connections to higher management, you can escalate things and possibly get additional people. But you will also burn bridges in the process, which is usually too high a price.

So instead you probably end up with the usual mix of folks: Very few rock stars, many middle-of-the-road folks, some promising newbies, and the occasional looser that nobody wants. Can you deliver something really great with such a mix? It will be difficult in a typical corporate setup where you have to somehow involve everybody. And this involvement of the less qualified half of people will slow down the “upper” half.

It is a very delicate subject and there are many fine lines, some from a legal perspective and many more from decency-to-others point of view. Also, the organization needs to think about tomorrow and therefore must have a “funnel” of to-be-rockstars, which need the best training they can get. And the latter is always working on a difficult project with experts and learn from them. But what few organizations do, is look at competence levels in detail and factor them in.

In other words: You can learn a lot from someone who is one, two, or perhaps three levels above you. But if someone is ten years ahead of what you currently know, the difference is just too big. You will only grasp a small fraction of what they teach you, and even that with a considerable risk of misunderstandings. And they honestly cannot understand why you are not following their great advice. Mutual frustration and dislike are usually the result.

Whether you take competence levels into account or not, considerable effort needs to be spent on non-development activities. If you have a taste for management and leading people that will be a great opportunity for you. But if your primary concern is getting something great and possibly visionary delivered, you should seriously consider a totally different approach: an underground project.

Flying under the radar can really be liberating. This must not be confused with idling around or working on some obscure pet project. It is truly about delivering what the organization needs but cannot accommodate in its own structure. Of course you should be certain that your boss will not fire you, if he or she finds out. But as you probably already guessed, driving an underground project does not mean that you are freed from politics, lobbying etc. On the contrary! You must prepare upfront quite carefully how you counter resistance or outright attacks. And yes, you are running a personal risk. But there is no reward without risk.

You are not faster with an 80% solution

When the schedule is tight for a software development project, one of the first things non-technical people contemplate, is reducing quality and/or scope and go for an 80% solution. This can work but only under particular circumstances.

A recent discussion I had with some folks was about such a situation. A very tight deadline had been issued by senior management and multiple departments had to collaborate on a solution that spawned multiple parts of the organizations and their core IT systems. Those systems (think about something like ERP and CRM) have a multitude of connections, grown over many years without governance, but not for the data that were required now.

The people were all quite supportive and a constructive discussion took off. Not surprisingly, finding a common language took some time. Especially so because only after a while did we realize that in several cases the same term had a different meaning. In the end there was a particular detail where no obvious solution was available.

I was responsible for the technical implementation on one of the two systems involved. My counterpart on the business side and I discussed various options and and eventually he came up with the idea to just safe time by going for the 80% solution. We talked about a number of aspects and it soon became clear to me that this would actually require more time than to just “do it properly”.

This sounds, at the very least, counter-intuitive for most people. So let me illustrate my point with a non-technical equivalent. Many years ago I had to prepare for my then-boss a list with press clippings every day. It basically meant to go through a long list, that was created automatically, and remove everything he was not interested in. This was quite time-consuming and he was genuinely surprised when he found out.

His statement then was “Why do you need so much time, it is only 20 of the 200 articles that are relevant”. That was of course correct. But what he had not thought about was that I had to go through the list of all 200 articles and check some in more detail than just looking at the headline. So it did not really make a difference whether I cut the list down to 100 or 60 or 20 articles, the time required stayed more or less the same. He agreed on the point and quite soon decided there was better use for my time.

It is exactly the same with taking shortcuts during a technical implementation. The easy part is to decide that you want to cut 20% off the next release. But things get tricky when it comes to deciding which 20% to skip without any undesired side-effects. In order to be able to do that, you need to understand in depth what the consequences would be. So quite often you spend more time trying to understand potential ramifications for various scenarios, than it would take to simply do it properly right from the start.

And that is only the beginning of the problem. You need to document the results of your analysis really carefully, if you want to be sure that nothing is missed when the first code change is required. Even a seemingly minor bug fix has an increased potential to break something if the current overall solution is built on a set of assumptions. And adding enhancements or new features will be even more “fun”. Because people have to read through all the documentation and also understand it. Together this is a time-consuming task with considerable risk to still overlook something (or hit an edge-case that was not documented).

But in reality this documentation does not exist anyway because you wanted to save time in the first place. And detailed documentation is not exactly supportive here. So people throw in bits and pieces and you end up with a document that is typically even worse than most project documentation that is out there. So whoever has the pleasure to work on the code without having all the details in mind is really screwed. That includes you, by the way, because you will have forgotten most after only a few weeks.

The solution is usually a re-implementation because it is faster than to fix the broken code. Of course that re-implementation often needs additional logic to handle data migration for transactions that were completed with the 80% solution and now ruin your data quality. But hang on, that might just be 20% effort you can safe right now ….

On Micro Services

Just a couple of thoughts on micro services, triggered by a German blog post on heise Developer. So let’s go …

I have read quite a few articles on micro services, and literally all of them were overly optimistic and just scratching the, of course, positive surface. The examples are usually just slightly more complex than “Hello World” and also completely focused on technical aspects. For a non-deep dive technical exercise these things are ok. But nobody should fool him- or herself: the real world is much more complex.

There is, by the way, a real déjà vu here for me. Many things sound remarkable similar to what was said ten years ago in the context of SOA (Service-Oriented Architecture). Then, also, things like modularity and increased agility were cited all the time. (Re-use was the other standard one, but that did not work in most cases.) So what is really different now? I honestly fail to see anything new.

Instead there are arguments, again remarkably similar to SOA, in favor of faster possibility to adopt, easier deployment of change, and smaller and less complex code. Well, they are wrong. The key here is complexity and what its consequences are. The core complexity comes from the business requirements and not the chosen technology or architecture. And no matter how you slice, dice, and re-arrange things – the business complexity simply stays. Your only chance for survival is to find a clever way to layer things in a manner that makes sense for your business.

You will find that there are things that lend themselves well to (micro) services and of course that is how they should be approached. But you do not really have to have a separate deployment unit for every Mickey Mouse service. Just apply common sense and also keep in mind that things are in constant flux anyway.

So instead of spending much time on micro services, I would rather recommend to think about ways, which ensure that your system/architecture can easily adjust. And from a business perspective that is, of course.

Re-Writing Software from Scratch

It is not uncommon that an existing piece of non-trivial software gets re-written from scratch. Personally, I find this preference for a greenfield approach interesting, because it is not something I share. In fact, I believe that it is a fundamentally flawed approach in many  cases, so why are people still doing this today?

Probably because they just like it more and also it is perceived, at least when things get started, as more glorious. But when you dig deeper the picture changes dramatically.  Disclaimer: Of course there are situations when re-starting is the only option. But it typically involves aspects like required special hardware not being available any more.

When I started writing this post with making some notes, it all started with technical arguments. They are of course relevant, but the business side is much more critical. Just re-phrase the initial statement about re-writing to something like

Instead of gradually improving the existing software and learn along the way, we spend an enormous amount of money on something that delivers no additional value compared to what we have now. For this period, which we currently estimate to be 2 years (it is very likely to be longer), apart from very minor additions, the business will not get anything new, even if the market requires it; so long-term we risk the existence of the organization. And yes, we may actually loose some key members of staff along the way. Also, it is not certain that the new software ever works as expected. But should it really do, the market has changed so much, that the software is not very useful for doing business anyway and we can start all over again.

Is there anyone who still thinks that re-writing is generally a good idea?

Let us now change perspective and look at it from a software vendor’s point of view. Because the scenario above was written with an in-house application in mind. What is different, when we look at a company that develops and sells enterprise software? For this text the main properties of enterprise software are that it is used in large organizations to support critical business processes. How keen will customers be to bet their own future on something new, i.e. not tested? But even if they waited long enough for things to stabilize, there would be the migration effort. And if that effort comes towards them anyway, they may just as well look at the market for alternatives. So you would actively encourage your customer base to turn to the competition. Brilliant idea, right?

What becomes clear looking at things like that, is what the core value proposition of enterprise software is: investment protection. That is why we will, for the foreseeable future, continue to have mainframes with decades-old software running on them. Yes, these machines are expensive. But the alternatives are more expensive and in addition pose enormous risk.

In the context of commercial software vendors one argument for a re-write is that of additional revenue. It is often seen as easier to get a given amount of money for a new product than an improved version of something already out there. But that is the one-off view. What you typically want as a software vendor is a happy customer that pays maintenance every year and, whenever they need something new, first turns to you rather than the competition for options. Also, such a happy customer is by far the best marketing you can get. It may not look as sexy as getting new customers all the time, but it certainly drives the financial bottom line.

Switching over to the technical side, there are a few arguments that are typically made in favor of a restart. My favorite is the better architecture of the new solution, which will be the basis for future flexibility, growth, etc. I believe that most people are sincere here and think they can come up with something better. But the truth is that unless someone has done something similar successfully in the past, there is a big chance that the effort is hugely underestimated. Yes, technology has advanced and we have better languages and frameworks. But on the other hand the requirements have also grown dramatically. Think about high availability, scalability, performance and all the others. Even more important, though, is the business side. With something brand new people will have greatly increased expectations. So giving them something like-for-like will probably not be seen as success.

The not-invented-here syndrome is also relevant in this context and particularly with more junior teams. I have seen a case when an established framework used in more than 9,000 business-critical (i.e. direct impact on revenue) installations was dismissed in favor of something the team wanted to develop themselves. And I can tell you that the latter was a really bad implementation. Whether it was a misguided sense of pride or a fundamental lack of knowledge I cannot say. But while certainly being the most extreme incarnation I have seen so far, it was certainly not the only one.

So far my thoughts on the subject. Please let me know what you think about this topic.