Growing applications handled by micro services approach

Let's have a look on characteristics of systems we are building. In general, clients spend only about 10% of their money on building application. The rest 90% of their money is the cost of ownership. Our systems are getting less maintainable. They grow in size and from time to time we are witnesses of IT failures, like 'the biggest IT failure ever seen'. Also, for some reason, there is a tendency to rewrite the entire application every 2 to 5 years.

Above observations and concerns are leading us to one of the great bugaboos of software applications over the years, which is their size. The problem is that we simply tend to build too big systems! Although above statements seem to be quite obvious, we still do same mistakes all over again. 

Let's think for a while and ask ourselves a question: what we do, when a given class is doing too many things? We simply go and refactor it, so that it sticks to SRP principle. Second question: what we do when our system is doing too much? Basically, we keep adding new features. It is extremely counter intuitive, but that is what we do. In fact, I am not even surprised, as the most common way of teaching TDD is to apply it together with SOLID principles on a unit level. Of course it is true, but it might be extended to all levels of abstraction. However, having a tangled application, it is our duty to apply SRP principle on a system level. Such an approach may lead us, but is not limited, to micro services architecture.

Micro services characteristics
There is no official definition of micro services, but they do have their own, specific characteristics. Micro services might be treated as a result of applying refactoring, with an SRP principle in mind, on a system level. They enable easier testing and re-usage of services. Also, you may treat them as fine-grained SOA approach. Moreover, micro services create real options, as you may adapt different (right) tools for different jobs (Unix system analogy). Exploiting micro services style we are getting independent deployment and scale, easier test and a change (side effect of small systems!) and very important option - flexibility.

Architect and his job
Let's step back for a second and describe what an architect does? An architect is a person who is the 'chief builder' of a system. His role is to satisfy clients' needs. In addition, he should do 'just enough architecture and design upfront'. 

In terms of architect's work, it might be compared to city planning. All major requirements have to be envisaged in the plan and embraced accordingly. For instance, there has to be a place for industrial and housing zones, possibly some sub-zones for light and heavy industry, too. I am sure, people would also appreciate nice city centre with greenery and leisure bits. On the flip side, it would not be a good idea to build a playground for children in the middle of industrial zone, instead of housing area. What is more, such a city should have shared utilities like: electricity, waterworks and gas pipelines. 

All these decisions and many others must be done upfront. However, they cannot restrain or impose too tight constraints on zones themselves. That is why, an architect should do 'just enough design upfront'.

Getting maximum flexibility with upfront design
'Just enough' is in fact very fuzzy quantifier of 'upfront design', so in real life we need more concrete guidelines. We should be starting from asking a question, how we can provide just enough upfront design in order to provide flexibility in our architecture? 

Having city planning example in mind, it may lead us to a conclusion that such a high level plan is actually something what business people call a capability map. In simple terms, it is a diagram, which maps business needs and requirements into a set of services talking to each other, where services might be zones and interfaces between services might be utilities, in our city planning example.

In order to get a flexible design, apparently contradictory, but in fact complementary concepts are used i.e. evolutionary architecture and emergent design.

In a nutshell, evolutionary architecture is 'all the stuff that is hard to change later' ;) - usually these are interfaces and contracts between components/systems. That is why one of the core rules of evolutionary architecture suggests to defer decisions as late as possible (aka keep your options open).
On the other hand, there is a concept of emergent design. It is an idea of deriving a system design in the continuous process of adding incremental changes.
Here comes another, tricky question, how we should use these techniques? Luckily, the rule itself is fairly simple. We apply emergent design on a level of bounded contexts/services (zones) and evolutionary architecture on a level of gaps between bounded contexts i.e. interfaces.

Hexagonal architecture
Also, while designing the general architecture and communication between components/services, it is very important not to obstruct the natural, creative flow of the thoughts by distorted diagrams.

We should be using right tools from the very beginning - hexagonal architecture approach is one of them. It gives us extremely clear view on what should happen in the core of a given application and how that application should communicate with external systems. Apart from that hexagonal architecture introduces more generic concept of looking at a system. In contrary to common approach, we are able to freely express more than two integration points in our model (i.e. UI and DB integrations). 

Incidentally, if you were imaging a system that is described in terms of hexagonal architecture and sticking to SRP principle, instantly you would get a handy tool for depicting micro services and making SRP fully visible on a system level! Apart from that such a micro service also conforms to Bovine's conjecture: "objects should be no bigger than one's head", which in our case applies to the system.

Integration
Assuming we have micro services in place, we can start introducing well-defined interfaces (contracts) between them. It will give us a power of communication with external world e.g. micro services.

There are two, very important attributes of decent software:

• high cohesion
• loose coupling

High cohesion relates to a service. Basically it means service should have a single reason to change - that is where SRP comes from. In terms of loose coupling, it might be treated as description of components dependencies to other components. In this case it should be as loose, as possible.

Over the years, in software design we went through many integration patterns and anti-patterns, as well. It would be worth going through majority of them and pointing out theirs characteristics.

Data oriented integration (dependency on DB level):

• no loose coupling
• hard to reason about
• brittle approach
• difficult to maintain and change

Procedure oriented integration (RCP, CORBA, WSDL-binding, JAXB, Java Serialization):

• method calls across boundaries
• sharing serialized object (tight dependency)
• adding methods with different input parameters doing similar things
• above leads to 'God classes' syndrome 
• above leads to very difficult change
• above leads to 'objects explosion'
• above leads to SRP and interface segregation principle (ISP) violation

Document oriented integration (JMS, messaging in general):

• asynchronous (decoupled)
• allows additive changes without breaking existing clients
• adding a field in document without break
• renaming a field in document, by adding new field - no breaks
• requires middleware

Resource oriented integration (REST, not HTTP):

• language agnostic 
• expose state
• noun oriented, not verb oriented like HTTP

When we talk about integration, it would be hard to forget about Postel's law: "Be liberal in what you do, conservative in what you expect". It might be translated into slightly more practical hint: "only bind to what you need, to reduce breaking service consumption".

As we see, there are many integration styles. However, RESTful approach looks like the most promising one. It is a fairly light concept, providing desired components' properties like loose coupling and it does not repeat errors from the past.

Summary
I believe I have outlined main issues in software design, prodding us to better understanding of the nature of the problem and finding out some better solutions. Searching for a right design is a continuous process and starts from the very beginning. Collecting and defining clients' requirements, the time architects spend on thinking about all use cases and satisfying client defined criteria, developers correctly modelling and implementing ideas can never be neglected. All these actions are crucial in the process of building anti-fragile and well-designed software.

Wrestling with Singleton (Logger) - round 3

Singletons have many faces. One of them is logging mechanism. Before we start having a deeper look into entire case, let's go straight away to a simple example, to set some context.

package com.korczak.oskar.refactoring.singleton.logger.before;

import com.korczak.oskar.refactoring.singleton.logger.Engine;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import static org.mockito.Mockito.verify;
import static org.mockito.MockitoAnnotations.initMocks;

@RunWith(MockitoJUnitRunner.class)
public class CarTest {

 @Mock private Engine engine;
 @InjectMocks private Car car;

 @BeforeMethod
 public void setUp() {
  initMocks(this);
 }

 @Test
 public void shouldRunEngine() {
  car.start();

  verify(engine).run();
 }

 @Test
 public void shouldStopEngine() {
  car.stop();

  verify(engine).stop();
 }
}

package com.korczak.oskar.refactoring.singleton.logger.before;

import com.korczak.oskar.refactoring.singleton.logger.Engine;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class Car {

 private Logger carLogger = LoggerFactory.getLogger(Car.class);

 private Engine engine;

 public Car(Engine engine){
  this.engine = engine;
 }

 public void start(){
  engine.run();
  carLogger.info("Engine is running.");
 }

 public void stop(){
  engine.stop();
  carLogger.info("Engine is stopped.");
 }
}

We can also assume that above snippet is actually a part of, so called, fairly decent application, which is logging everything everywhere, so to speak.

What can you tell about such an application?
Is there anything particular, which is stemming out from our imaginary application?
It may look like to somewhat silly task, but unfortunately it has very deep implications, in terms of design, solving production issues, sometimes performance, overall usage and approach. It is quite a few things tangled to logging, as such.

First of all, let's decompose above snippet with attention to some basic facts:

• Static logger is a singleton
• Breach of unit test definition i.e. system under test is writing to file/console
• Lack of readability to some extent
• A bit of noise in business logic

Now, let's discuss above points a bit further and infer some more, useful information.

Logger is a Singleton

Logger is created via static call and a Singleton is returned. However, it is not that bad, as we are only writing to Singleton and never reading from it. It means we are not exploiting a global state introduced in our JVM by Singleton. On the other hand, it is worth bearing in mind that static call means no seams. There are two direct implications of that: 

• untested production code, as there are no seams
• message will be logged anyway during running tests

Unit test definition breach
That is how we came to the second fact related to unit tests.
Michael Feathers defines unit tests as: 

A test is not a unit test if:

It talks to the database

It communicates across the network

It touches the file system

It can't run at the same time as any of your other unit tests

You have to do special things to your environment (such as editing config files) to run it.

I suppose we can agree that above test is an integration one, rather then a unit sort of thing. Feathers's definition sticks to what people understand under unit test term. 

Why don't we finally admit that so called unit tests, which are logging information to a file/console whatever, are in fact a decent integration tests. Full stop.

What is the business related reason?

Going further one can focus on a business reason in that whole tangled situation and answer a question: is there any business related reason to log particular information in particular place?
The most common answer would be that by logging we mean sending a message from application to whoever is browsing logs. Usually, it would be two main groups of recipients: L1/L2 and L3/testes/developers teams. 
L1/L2 guys would like to see only nasty things happening in a given application e.g. errors, alerts and perhaps warnings. On the other hand, L3/testers/developers would like to see a bit more i.e. states generated by application during its processing etc. It is worth reiterating that none of them is a business person.

When we dig a bit deeper and ask what is the underlying principle of logging, we will probably come to below two points:

• log a state of processed unit
• say 'I was here' or 'I have chosen that path in you application logic'

Above things tell us where we are and are often called diagnostic reasons/purposes. They share a common feature that none of them is driven by any business requirement. Let's be honest, business people are not reading logs. They can only read spreadsheets, but you probably know that already, I suppose.

Readability and noise
Unfortunately, logging affects a bit negatively the overall readability of business logic. It adds additional noise and clutters codebase, when it is overused.

How it might be done?
The easiest might be to imagine an event driven system, where an event comes in, it is being processed and some final response is sent. As it was pointed, logging would be used for diagnostic purposes. If yes, I am questioning the logger usage, almost at all. Why don't we change our applications design so that it can monitor, diagnose and track the state of all, incoming events itself? 

I already can hear answers: 'it is too much effort', 'we don't need it' or better one 'our project is different'. ;)

Partially, it might be true and I agree it is pointless, in some trivial cases, to build errors aware applications for your younger sister's diary application, although still it is worth at least considering it in some larger projects. 
Also, don't get me wrong, if there is a clear business drive for having something logged, just do it. The only thing I am asking to remember is fact that logging is a sort of contract between application and whoever is reading logs. If there is a business need for having some log entry, simply test it. 

When you have self aware system in place and an issue occurs, you may ask your application for some useful 'diagnostic information' (including states) on demand via say REST API or whatever. 

Implementing that sort of approach is in fact a win-win situation for all project stakeholders. Not only L3 and developers are able to deal with errors in processing, but also testers, L1, L2 teams are able to do that. Even BAs start analyzing and solving some burning questions themselves. 

In a long run, it is less work for busy developers, everybody can leverage and use diagnostic information. What's even more important, the knowledge and the responsibility is shared across entire team/stakeholders. It only requires two things: 

1. Stop logging everything in the log files
2. Add sort of issues memory or events state memory to your system

The first step here is to start testing logging in your applications. Try to make logger a collaborator of your class, not a static call and use it where it is really necessary. 
Testing logging enforces thinking about every single message, which might be potentially logged. Additional effort spent on writing test for logger, should actually drive aware developer through all above concerns, helping to understand the business reason behind particular piece of code.

package com.korczak.oskar.refactoring.singleton.logger.after;

import com.korczak.oskar.refactoring.singleton.logger.Engine;
import org.junit.runner.RunWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Captor;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
import org.slf4j.Logger;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import static org.mockito.Mockito.verify;
import static org.mockito.MockitoAnnotations.initMocks;
import static org.testng.AssertJUnit.assertEquals;

@RunWith(MockitoJUnitRunner.class)
public class DecentCarTest {

 @Mock private Engine engine;
 @Mock private Logger decentCarLogger;
 @InjectMocks private DecentCar decentCar;

 @Captor private ArgumentCaptor argument;

 @BeforeMethod
 public void initializeMocks() {
  initMocks(this);
  argument = ArgumentCaptor.forClass(String.class);
 }

 @Test
 public void shouldRunEngineAndLogMessage() {
  decentCar.start();

  verify(engine).run();
  verify(decentCarLogger).info(argument.capture());
  assertEquals("Engine is running.", argument.getValue());
 }

 @Test
 public void shouldStopEngineAndLogMessage() {
  decentCar.stop();

  verify(engine).stop();
  verify(decentCarLogger).info(argument.capture());
  assertEquals("Engine is stopped.", argument.getValue());
 }
}

package com.korczak.oskar.refactoring.singleton.logger.after;

import com.korczak.oskar.refactoring.singleton.logger.Engine;
import org.slf4j.Logger;

public class DecentCar {

 private Engine engine;
 private Logger decentCarLogger;

 public DecentCar(Engine engine, Logger decentCarLogger){
  this.engine = engine;
  this.decentCarLogger = decentCarLogger;
 }

 public void start(){
  engine.run();
  decentCarLogger.info("Engine is running.");
 }

 public void stop(){
  engine.stop();
  decentCarLogger.info("Engine is stopped.");
 }
}

You can find entire example on my GitHub account.

Round three is finished.

Which game are you playing ?

Few days back, I spoke with my friend about pretty "funny" situation. He told me that developers at his project are playing a game, which my friend called Pair Programming vs. Code Review. If developers like each other, they play Pair Programming. If they do not like each other, then they play Code Review.

As you can see those developers found a problem, where it was not even existing. That's a rare and undesired skill, which often manifest itself in pseudo pragmatic teams, who are probably not sticking to 5th Agile manifesto principle: 
"Build projects around motivated individuals. 
Give them the environment and support they need, and trust them to get the job done."
Generally, it is called over-engineering! But not only that, they are loosing time for guerrilla warfare, rather than focusing on helping each other. They also seem not to have any respect to each other. 

The question is: should it be this way? Of course, no!
I would like to be clear about Pair Programming and Code Review, so let's start from some definitions.

Scratching Tiger, Hidden Dragon - Pair Programming

One developer types in code at the keyboard. He works at the tactical level. The other one looks for mistakes and thinks strategically about whether the code is being written right and whether the right code is being written. 

You can also describe pair programming in a more holistic and tao oriented way, as a yin-yang or a unity and harmony arousing from diversity. There are many approaches explaining principles of pair programming like coder and observer, driver and navigator, however I am always keen on scratching tiger and hidden dragon parallel. 

In other words, it is a conjunction of opposite features like active and passive, linear and non-linear or specific and general in a collective action such as solving a problem - programming in this case.

To be fairly honest, one can extend and generalize this approach to any sort of task you have got to do. 

The overall idea is very easy to grasp and simple. Let's look at pair programming from two, different perspectives: tiger and dragon.

Scratching Tiger:

1. Active person writing the code, almost scratching it
2. Keeps on focusing on syntax and details
3. Uses his linear part of brain, logical (left) hemisphere to solve arousing problems

Hidden Dragon:

1. Passive person pays attention to code quality and acceptance criteria fulfillment, almost like dragon, flying in the sky and observing what tiger does
2. Can see whole solution, has an overall idea of subsequent tasks and steps
3. Uses his non-linear part of brain, abstract (right) hemisphere to imagine a flow and grasp the bigger picture

You would probably agree that above description is somewhat static and simply a bit boring. Frankly, you are right, but the key here is movement and synergy you can get from it.

How can we get the movement ?

Again, this is simple. You need a proper space. Comfortable desk, two chars and two keyboards. These things are really important, so do not neglect any of it, especially two keyboards. These prerequisites enable higher level of collaboration between developers. Apart from that they are not loosing time and ideas, while swapping keyboards.

And now comes the most important part. Developers are creating a collective mind, which is buzzing and boiling due to new, fresh ideas. The worst thing you can do, is not switching between Tiger and Dragon role. You have to do it frequently, without any hesitation and freely. You cannot be afraid of your partner saying leave that keyboard, etc. When you see Tiger did a typo, initialized a loop in a wrong way or did not do refactoring, which might be done - simply take over and use your own keyboard to fix it. You need to change perspectives very often and switch your brain from linear to non-linear mode and vice versa. That is the only way you can solve difficult problems. 

Why pairing on solving tasks?

Mainly because of four reasons:

• Readability - two people take care about readable code, both of them have to understand what was written

• Code quality - percentage of defects found 40% - 60% (Steve McConnell "Code Complete")

• Bus factor - knowledge sharing and mitigating the risk of single point of failure

• Pressure - group shields developers from external pressure much better than single developer can protect himself. What's more, in stressful situations people tend to cluster to face difficult situation.

Okay, so when it is worth to pair? 

It is definitely more expedient to pair on crucial and strategically important tasks. It is also useful to pair on both very hard, non-trivial implementations and medium level difficulty tasks. There is no use of pairing on relatively easy tasks, as they do not add so much value, comparing to effort.

Always remember about rotating pairs, not giving an opportunity to get used to your partner. The efficiency is decreasing then. Moreover, it is also good not to pair newbies, as there is not master-apprentice relation, which negatively affects pairing process. Definitely people need to like each other.

Code Review - No junk in a trunk

It is a sort of semi formal walkthrough the code aiming to:

1. share knowledge (mitigate bus factor)
2. check design from high level perspective
3. check at least SOLID rules appliance
4. check definition of done (DoD)
5. increase resilience before holidays

When can you perform code review?

Of course after development and testing, but before merging to master or trunk, so that there is no junk in a trunk.

The programmer or pair, who developed a solution, walks some other team member through new code. They start from telling the story and acceptance criteria, then they are explaining applied design and showing working, green, CI  system tests. There is also a place for Q&A and potentially some code review remarks or changes. Code review is able to discover between 20% to 40% of defects (Steve McConnell "Code Complete").

Summary

Code Review and Pair Programming are two, complementary techniques improving the general quality of the provided solution. Apart from that they mitigate the risk associated with bus factor, by knowledge sharing.

Both of them create a useful unity in development cycle. 

You may even use scratching tiger, hidden dragon parallel, in this case, as well. Pair Programming would be a tiger and code review a dragon looking at the solution from the distance. 

It is much cheaper (sic!) to let developers be tigers and dragons, rather than giving them a wigging that they loose time for crap.

Rarely you have to choose between these two approaches. Then do it wisely and choose pairing. If your management is trying to be clumsy lean oriented and tells you should not do pairing, as it is a pure waste, take at least code review. Remember, half pairing is still better than not paring at all. It is always good to discuss design, start doing the most tricky bit together and then split and finish what you started separately, if time is so valuable. 

However, in fact it does not hold true. You are writing robust and clean solutions faster while paring and reviewing, than not. 

Show Me the Value

Do you recall a well-known phrase uttered by characters in the 1996 film 'Jerry Maguire'?

If one would like to sum up Agile practices in a neatly tailored way: 
'Show Me the Value' is one of the best approaches I have ever heard about. 

Early and continuous customer's satisfaction in cyberspace

Let's consider first agile manifesto principle, i.e.: 

"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."

What it basically says is that there is nothing more important for us, than satisfying our business client through:

• early delivery
• continuous delivery
• delivery of valuable software

Ideally, we ought to start to deliver something valuable for our customer from the very first day, when the project starts. Apart from that we have to remember about delivering these valuable bits constantly.

Hmm ... one may say, in reality, it is quite hard to commence delivering valuable software from the beginning of the project. 

To my mind, an important question seems to be: what is the definition of valuable software from client's perspective? The thing that only matters for business is something that adds value for them. They are 100% right, however in my humble opinion, majority of business people have slightly twisted perception of that definition. In other words, they seem to consider value in places, which directly maps to money. Whereas, a deeper look from risk management angle reveals business value in operational readiness or continuity of deliveries. This also means money, but this time not direct one. The root cause might be lack of risk management skills and not being able to see the clear, bigger picture of what is going on. They seem not to appreciate value hidden in infrastructure almost at all. In contrary, in any army, especially during the war, infrastructure like robust roads and railways or fast and reliable communication channels etc. are fundamental factors. The Second World War proved that rail and roads might be a key to transfer military units between places in few hours. 

Infrastructure, especially CI infrastructure, is not a direct business value itself. Although in IT it is something invisible (all in all everything is a software), the quality of it is a clue and creates a reliable scaffold, where business values can be spread. For instance painters always prime their canvas before painting. They do it in a fairly detailed and decent way, as they know it is a "scaffold for their vision". Infrastructure in IT pays dividend when the project lasts and even after that when it operates. So the question remais: what is a business value? Is infrastructure an added value for business? Is being operational BAU 24/7 a fair point? Aren't these two adjectives (early and continuous) a deliberate game of words in agile's first principle? Apparently contradictory, but in fact complementary statements: early and continuous delivery. In economy, they tend to say the the best business is the one, which is operating all the time. Wait a second, isn't it what we just said that infrastructure is something allowing business to be run and produce value 24/7? Yes that is exactly what we were in mind!

From my perspective, every project starts to deliver a value for business, from the very first day, either in pure business stories taken from backlog or in terms of infrastructure being a skeleton for whole business we cater. 

Now, let's focus on cybernetics, an extremely relevant and transdisciplinary science, rooted in ancient Greece and even earlier. Basically, it is an approach for exploring and examining regulatory systems in terms of their structure, constraints and possibilites. Cybernetics is only applicable when the system being analysed is involved in a closed signal loop. It means that actions done by the system causes some change in its environment and that change is fed to the system via information (feedback) that enables the system to change its behavior. Above description has two, valid points. First of all, cybernetics is a transdisciplinary science, which connotes a research strategy crossing many disciplinary boundaries, to create a holistic approach. This might be compared to generalization rule in OO languages. Secondly, definition focuses on feedback loop, as a main source of information for changing its behavior. Every scientist and discoverer may and should ask, if there is any principle or rule we can take from Cybernetics and apply it in Software Engineering?

Having all above knowledge in mind and treating it as a prerequisites let's do some cybernetic analysis and try to answer above question. Given every member of agile team is a distinct cybernetic system, equipped with some sort of feedback loop. Let's further assume that environment, we are deriving information from, is code base. Then the only unknown remains feedback loop. A source control management (SCM) tool might be perceived as a sort of feedback loop. 

Let's put under examination two very well known SCMs: SVN and Git. 

Probably all of us know what 'painful merges' are. We are exposed to them as we are breaking at least one of below rules (I was discussing that topic here):

• Baby steps
• Small scope of changes
• Short living branches 

A rule of thumb is that SVN merges are not the most pleasant things we may do, in terms of development process. Because they are so, we do not tend to do them as frequent as we should. As a matter of fact, we end up with, say one merge per day, which is done in the morning, or even not that often. We are obviously breaking at least one of above rules and we might end up with something called a 'merge hell'!

On the other side, Git as an alternative SCM solution is having incorporated merging process (or rebasing if you like) into your day to day development cycle. Complying to rules allowing you to avoid painful merges and having Git on your side, enables you to commit more often, comparing to SVN. 

Let's come back to our cybernetic experiment and SCM as a parallel of feedback loop. How do you think, which SCM is better? Is it SVN with its clumsy funny merging process and rare synchronisation with remote repository or Git, which is pushing you to rebase frequently as a part of your development cycle? It is your choice, but remember about everyday pain of all people using strategy you have chosen!