160   medium.com

Refreshing Comments...

Am I the only architect that hates microservices with gusto?

I've only seen bad implementations out there (tens of products), and have yet to encounter one that looks reasonable. If your average developer has trouble implementing something sane when the only problem is splitting data and functions locally into different components, why in the world would you assume that putting in a network layer, exponentially increasing the failure rates, loosing good stack traces, making refactorings that have to move data from one part of the architecture to another a pain in the butt (because requirements never change right?), etc... would be a good thing?

The horrors I've seen... oh boy, the horrors.

The people I've heard praising microservice architectures so far fall into two categories: (1) Ex developers that moved into management and read a few articles/bought a book on the subject and think it's a great idea (but have no actual practical experience) (2) developers that have just started working on a new one and are still in the honeymoon phase (mind you, not developers that have started working on an __existing__ one that is a few years old, those figure out the mess they are in pretty quickly).

You can screw up not-so-micro services too, but it's far easier to screw up microservice architectures and they are far, far harder to fix down the line when you figure out how you messed up (or re-architect when that thing that the PM told you was never going to be a use case is going to become a use case (after they tell you they are sorry of course)).

Totally agree - I'm not against services but I say as few services as humanly possible!

The issues with micro services are just so unbelievably large that it'd take too long to enumerate them here, but for me the largest issue is you are putting arbirary network partitions into your application and you CANNOT possibly know when these network partitions will suddenly make an operation you need to be atomic impossible (or at best eventually consistent).

In general it is excruciatingly difficult to iterate on micro services as well.

Then there is deployment, service mesh, k8s etc. etc. that you have to wrangle with, all because people refused to make bounded contexts in their application without making them into different services. It is much simpler to just design your monolith into separate contexts than it is to introduce all the error handling you need for network partitions.

I personally don't understand why Elixir isn't more loved by these micro services zealots - it allows you to get extremely far without adding network partitions and then allows you to switch to message passing when you need and then networked message passing all extremely smoothly as it makes sense for your app.

The best network layer is the memory controller of the CPU. Never breaks, no partitioning problems. If you really wnat for whatever reason uservices run it on one machine,
Architecture driven by engineering AKA security, authorization, performance, processes and timing, data flow, user/system interaction, hardware devices etc. can inform/define what the subsystems are.

The architecture in the article here seems to be driven by a certain type of OO data/interface-modelling:

In school/courses/tutorials/books we are initially taught a type of object oriented data-modelling, which is at the level of: objects in the world are objects in our programs. A person is a Person class, a shopping cart is a ShoppingCart class. You can put stuff into the ShoppingCart, it belongs to a Person and so on.

I call this "Kindergarten-OO": It puts labels on things in the world and then models software interactions 1:1 based on that, as if software is a simulation of the world.

This conflates all kinds of things and the actual engineering and user-interaction requirements get smeared all over your codebase.

Now this kind of Microservice design looks kind of like that but lifted up to a network of computers.

You end up with suggestions like: "Design applications around eventual consistency" (article). Why? Because you didn't design your architecture around timing processes, data-structures/-flow and interactions, but around this made up simulation with ~N^2 communication channels.

A process that takes a longish time and maybe doesn't need to be real-time consistent: Analytics. Let that be eventually consistent, close the feedback loop each Friday and enrich your data which then informs the rendering (ordering, emphasis of products and so on) on your front-end.

An architecture around interactions, flow, data-processing, security/auth and so on looks much more like a decision tree, state-machine or something like that, rather than like a zettelkasten graph.

The only application that I encountered that fit the microservice idiom well (obviously, in my opinion) was postfix (some diagrams at http://www.postfix.org/OVERVIEW.html). Note it is usually using unix pipes for communication instead of networking, so some of the complaints about the networking disappear.

Even there, the "clear" separation of concerns means that it becomes difficult to do some things that seem like straightforward business goals. You'd like to bounce emails to a user's aliases if they are over quota but allow messages to that user from support. Unfortunately the milter and the alias resolution hooks are different services and don't get to communicate, unless you use some other functionality... I'm a bit hazy on the specifics, but I can remember needing to bend over backwards sometimes to satisfy things because you couldn't chain additional information through. This ordinarily comes up quite a bit when people compare it to exim, an SMTP monolith where you are more easily able to use information from one service in another.

I think postfix is well built, but the architectural choice has consequences and it isn't a silver bullet.

> Am I the only architect that hates microservices with gusto?

I only have a problem with the label "microservice". A service is an independent product, that's all. "micro" says absolutely nothing about the size of that product. It's like "serverless", it doesn't even explain what it actually is thus quickly become a marketing buzzword.

I find this opinion to be rather odd. FANG and e-commerce firms like eBay solved scale by partitioning functionality into services for scalability.

What is key is to have a scalable time series db to monitor the interactions of the components and services (and microservices) that comprise your platform. Measure the KPIs around hardware, OS, software and services interactions. Monitor and alert on them, via real-time dashboards.

I get where you are coming from. But we certainly did a decent implementation using Microservices. The project is basically complete and in production and it's the backbone of a major company, running a few million requests a day. Not that we didn't make mistakes but I would say it's definitely possible...
I have done a lot of work with microservices, and I agree they come with a ton of cognitive overhead and new ways for things to fail in unexpected and painful ways.

In particular it now requires software engineers to understand principles and design patterns for distributed systems that when I was first starting out were relegated to theory and university papers. Can you guarantee ordering? Can you guarantee exactly once? How do you get transactional semantics? How do you handle cascading failures? It's all a bit overwhelming.

I have also seen however that monoliths at some point lock up an organization. The key measures of software delivery performance are deploy frequency, lead time from commit to running in production, mean time to recovery and change failure rate. Monoliths impact all of these when you engineering org starts scaling past a handful of teams. A monolith also becomes a single point of failure. One person in one team checks in bad code, and the whole system falls down. Microservices can give you resilience. You can also scale microservices independently - a CPU intensive service can get a bunch of pods, a memory intensive service gets a beefier memory allocation, and other services can run idle most of the time or even be deployed as FAAS functions that almost never run.

So I agree it's not a technical solution as much as it is about scaling an organization. At WeWork the team shrank so much that we began consolidating small services back into a single service because the operational overhead of having separate services didn't make sense any more when a single team was responsible for all of them.

I do think choosing to split out a new service is a big decision to make. You have to really think through the cost and benefits. You also need to make sure you have the right tools in place so you can detect and recover quickly.

Things like log aggregation, SLOs defined, measured and alerted on, canary deploys, progressive delivery, etc. Without that you're running completely blind.

I also think you want to be thoughtful about keeping your coupling low. If each service depends on twenty other services then in my mind you now just have a distributed monolith.

> Can you guarantee ordering? Can you guarantee exactly once? How do you get transactional semantics? How do you handle cascading failures? It's all a bit overwhelming.

I don’t advocate DDD/microservices but if you absolutely must go down this road, Kafka Streams really neatly answers all these questions. Ben Stopford’s book[0] and blog posts[1] are really great if you want to know more.

[0] http://www.benstopford.com/2018/04/27/book-designing-event-d...

[1] https://www.confluent.io/blog/building-a-microservices-ecosy...

Domain Driven Microservices looks good on paper, but its a very very terrible idea.

1. You can't run join queries across domains

2. You can't migrate your schema across domains

3. You will need to sync and map a db with domain dbs for reporting

4. You need to write api/client server for every domain and maintain this with db schema changes

It seems like database schema and domain api are very coupled. They should have nothing to do with each other. You should be able to change your database schema without changing your api....

If you want to expose extra features or change the api design, you version your api.

Reporting is a problem, but most large business have some kind of datalake/datawarehouse where all of it comes together anyway these days.

> If you want to expose extra features or change the api design, you version your api.

But you are still coupled. If you have to version your API that means it is relied on by something else and in order to use that extra feature still requires both (or more) services to all be on the same page.

Microservices let you deploy and maintain at different times (if that is appealing to you) but in order to realize a new business feature you are still coupled.

Now you have x tests for x versions as well, make sure the old version remains old and the new feature works in the new version.

I mean yes, if you want a use a new feature using the new api. Then your going to have to move onto the new api. But that would happen regardless of using microservices.

If you version it though, services that don't care can cary on regardless.

I think that many people look at Microservices from a purely technical standpoint. How much compute power do I get per dollar spent. What about the monitoring complexities and the logging problems?

Microservices are actually often driven by business requirements. According to the Myhtical Man Month communication channels follow n(n − 1) / 2. So a team of 4 developers has 6 possible communication channels, a team of 5 already 10. This eincreases sharply as team sizes increase and is one of the reasons big enterprise projects suffer compare to startups.

The business benefit of Microservices (done right) is that you build islands of communication, that communicate using only one outside channel. Now communication increases less sharply as teams are separated by their boundaries.

According to the guardian the team size and structure is one of the secret for Amazons success. https://www.theguardian.com/technology/2018/apr/24/the-two-p...

For smaller teams the communication benefit will often not outweigh the additional complexity.

Ex Amazon SDE here. Amazon uses a service based architecture where one team (e.g. 4 to 8 engineers) maintains one service.

The main benefit is organizational and lies in having a small and very measurable interface between services.

Crucially, the team the develops the service runs it and provides 24/7 on-call for it. This also include taking decisions on how much to spend on hardware VS optimization as the team is responsible for the overall performance.

More importantly, the team is now responsible for service outages.

This is very different from having a team write many microservices. It's especially bad when a team writes a bunch of microservices (the so-called distributed monolith) and somebody else has to run them.

Sadly the article does not focus on this aspect.

As a side note, once developers realize they get paged out of bed by their own code... it's amusing to see how the cool framework of the month is not cool anymore.

At my previous company this was introduced as well, but the incentives were very warped. There was no advantage to actually being all that stable, since then you missed out on "firefighting glory". I never saw praise or promotions for maintaining high uptime. I'd be very interested to learn if Amazon managed to overcome this and if so, how.
I cannot talk for the whole company but I don't think I witnessed a culture of "firefighting glory", rather the opposite.

There is a formal process to investigate and correct errors after a non-trivial incident that requires collecting evidence and then discuss what happen, when, why and who did what with the whole team.

Then people ask the question "why was this not prevented or forseen?" and keep going backward until you exit the technical realm and look into people's choices.

The root causes could be, for example, that it was a management decision to prioritize something else over availability concerns... and the engineers are off the hook.

Sometimes one engineer was overworked and tired and did an error in good faith that is too difficult to prevent and avoid, and that's also acceptable.

Sometimes that the whole team ignored an availability issue in the product architecture, and that's bad.

And so on... Needless to say the outcome can impact people's performance review.

My first microservices were written using Sun RPC, the company whose motto was "The network is the computer".

Since then we have seen plenty of microservices solutions come and go, what stays is the complexity of dealing with distributed programming.

Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else.

> Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else.

This is the most important bit. To me microservices is just another way to arrange your system into module, not much different in concept to modules and packages. Regardless of whether you’re dealing with monolithic or distributed system, if your abstractions are poor, your work and your system will suffer.

> Regardless of whether you’re dealing with monolithic or distributed system

Agreeing somewhat, but not totally ; monoliths are much easier to test and stabilise. All bigger microservices projects I know (personally, not what I read from Uber etc) have tremendous overheads compared to their monolithic counterparts. Some of them are really nicely done, but there still is a lot of overhead in support (usually the services are written by different people (and possible different languages/environments, for instance .net core, .net framework (for Windows specific services) + Typescript/node), so other people need quite a large ramp up to get into them), networking, monitoring, etc. As far as I have seen, it needs a larger team.

And it is very different from having you modular built monolith in your IDE for a spin, than doing a little (local/dev) testrun of 25 microservices + the one you are working on. I can feel the benefits but I have yet to encounter them in real life; so far I have seen (smaller; ~100m/year rev) companies going back to monolith and most of the stories I read are from very large companies with massive development+deployment teams where you actually can have 3+ people per microservice.

If you are working with 3-4 people managing 20+ (changing, forming a complex system), I don't really seen it happen (but would love to see practical examples of that) while monoliths of the same complexity/functionality (literally; I worked on a few 'just because scaling!' rebuilds from monoliths->MS(->back)) have no issues at all with that.

I’m not really arguing which one is easier to run, maintain, or anything; I was just making a point about abstractions and system modularisation.

Hypothetically, I should be able to take a well-designed monolith and “deconstruct” it into a distributed system by deploying modules individually and replacing function calls with network calls. Going the opposite direction, I should be able to do the reverse with a well-structured distributed system.

This pattern of modularisation exists across the software stack; it goes as far as the digital logic level with lumped-element abstraction. In general, we trade-off some benefits with a more complex messaging mechanism, akin to threads sharing the same memory space VS processes communicating through sockets or files. In general these are all technical details that help us accomplish what we need our system to do, but pinning the right abstractions helps us actually reason about our system.

> Anyone that isn't able to write modular code using the language features is going to write network spaghetti code,

Languages (libraries) should support this more because people are doing it. I don't want to be bothered by this complexity etc for most of the project; I will architect it properly but the language features should help me in this case.

Some new languages/envs are exploring this, like [0] and Erlang ofcourse is an example of naturally working (or having to work) like that.

In many other environments we work it's mostly annoying; very possible but a chore instead of the language/environment helping me. We roll out quite large deployments anyway; given the choice I rather would see a nicely architected monolith.

[0] https://www.unisonweb.org/

Apparently Ada and Modula-2 aren't fashionable, given how long ago languages and libraries support this.

I could pick more recent examples, just wanted to make a point how old the concept goes.

Good explanation. But I have two questions:

1) So which "team" then takes ownership of the entire katamari ball of sprawling microservices and understands the interconnectnedness and nuances of overlaps and competition between each microservice?

2) If the concept of technical debt at yesterday's monolithic architectures is biting us now, what does the future technical debt of microservices hold?

1. No one has ownership over all the services, that's shared ownership. A team might own the platform and be responsible for up time, but individual teams own their services. Somewhere higher in the engineering org there's explicit ownership of end to end product and its usually tied with that product team or again the platform team.

2. As you scale, sometimes its easier to create a new service than it is to go modify that old service. When a company scales fast or employees churn its either rewrite that service or leave it where it is and build something new. This can lead to something that looks a lot more organic than computational. I have only seen the likes of Google handle deprecation well, to the dislike of most people external but survival of the fittest is harsh at best.

in both 1 & 2, I only see money pits... This sounds like so much overhead... Par for the course for enterprises, but I see too many startups now deploying 50+ microservices and suffering with the overhead (hosting, maintenance, employee-count/churn/training, monitoring etc).
> in both 1 & 2, I only see money pits... This sounds like so much overhead...

That's the point. The overhead is high but the other options to employ extra developers bring even more overhead. And the company wants extra developers because the faster (or more parallel) development still has a good ROI.

If you want low overhead, a small team is the way to go.

But what happens is you end up with a ton of them, fulfilling the same prophecy you were trying to avoid, or you keep trying to cram more “volume” behind the same surface area, until you eventually tear a hole in the fabric of space time and/or your business.

The way to succeed is to do fewer things but make them count. Except that looks awful on your annual review, so the real way to succeed is to Defect. Be seen doing as much as you can and get out before they figure out you consumed all of the oxygen in the room.

Good points and it feels like this is what most people miss. They take microservices from a purely technical perspective and then rant about its complexity. You don't need a large scale distributed systems platform when you're a small team or have no scaling requirements. Microservices is largely driven through that organisational need of teams having to ship products/features independently and its just easier when the communication is via a service API vs humans or explicitly when walking over each others code.
If communication friction and overhead is the primary thrust for microservices (where you obviate said overhead by keeping to small teams), how do you address and manage the need for communication between teams?

Obviating communication costs within teams is one thing, but what of the overhead and costs of communication for the whole system? Eg. how do you avoid the cost of deploying a new team that ends up deploying a duplicate microservice?

I would say that, if you're at a point where teams are deploying duplicate services with no idea of each other, you're already so large with so many teams, that you might desire that as the correct outcome. You now have hundreds of teams running in parallel, and there's no point in coordinating with everyone, just coordinate with the ten that you are adjacent to. So, yes, the company is not maximally efficient, but it can take advantage of parallelism, and that's ultimately much more important for speed of progress. I could see an argument that it'd take months to get all the teams to all meet and agree and coordinate on what this shared service should be, when you could have built two variants simultaneously in a couple of weeks.
These things are rarely binary. In that there's no one size fits all solution. It comes down to the culture of the company and how they communicate and then how the technology evolved based on their needs. So if you have an engineering org which practices high bandwidth communication there might be a drive to keep that up by coordinating a meeting once a week by nominating one person from the team to go provide updates and learn about what else is happening.

You may also explicitly establish an architecture team that has oversight and tries to ensure everyone's aligned on direction. But mostly if you at least have some shared place where you explicitly communicate the creation of new services or design development then it allows you to avoid some of the issues you're talking about. The other thing is having a way to discover APIs and services so you can go check if something exists before building it. In many cases teams do still build a variant themselves because they need something slightly different and thats ok if they're willing to support it.

Communication between teams is handled by team leads talking to each other - where I’m at we have standups each day within teams, and then every other day those are immediately followed by team leads having another standup where the focus is on what teams are doing, rather than what individuals are doing.

It’s not perfect, but it’s a lot more efficient than the previous state of every individual developer needing awareness of what every other individual developer is working on at any given moment.

You're making a point that Microservices help teams to solve the problem of increasing communication needs for larger teams. While your statement makes a lot of sense I'd like to ask why this problem cannot be solved with packages or classes within a monolith? A team can own a Microservice and expose an api to others the same way they could expose an api on a class or package, no? Microservices are a way of deploying small parts of software. It's not about communication across teams. It can help with communication but this is not only true for Microservices.
I believe it could be done via classes or packages, but only if the providers have the discipline to design it like a rest service, and the users then have the discipline to only use it like a rest service. Each package needs a small, consistent interface, with no way to interact except through that interface, with that same style of consistency across all libraries in the same way. The code must be isolated at runtime, so it can't steal resources from other pieces of code in the monolith or cause anything to crash. It can only respond to requests along the interface, either successfully or unsucessfully. So, theoretically possible, but in practice difficult once you add humans to the mix.
So you are in principle of course correct. But keep in mind that Microservices can be deployed independently of each other. With Monoliths there is a single deployment that causes a lot of synchronization issues. Every team needs to deploy in lockstep.

This is all theoretical to some degree, because I have seen teams deploy Microservices in banks during the deployment windows and I have had people explclaim they have ONE Microservice.

So you can definitely do it, but Microservices have other advantages, as in how easy can you then scale the components, how resource hungry are individual services and so on.

> Every team needs to deploy in lockstep.

Team A can finish their part of joint effort before Team B and get deployed on Monday, then Team B finishes their effort and gets deployed on Tuesday: this looks the same in both microservices or monoliths. You can have blue green deployments with monoliths with zero downtime.

Why does it look the same? Because you have to be careful with how you modify the interfaces in any case. If team A is requiring a new field from Team B to be not null immediately, then the Monday deployment breaks in either case. If team A uses a feature toggle or just does the old thing when the new field is null, that has nothing to do with microservices vs monoliths. If you are using a monorepo in both cases the synchronization story is the same: master needs to build and run without error.

The real distinction is what the Monday and Tuesday deployments look like in either case. With monoliths, the Monday deployment and the Tuesday deployment look identical: the whole monolith goes up and down. With Microservices, team A can be responsible for their deployment on Monday and it can be completely distinct from what team B does on Tuesday. If team A is team B, then the gain was a minor allowance in the deployments being different which might be a net loss.

Actually, when team A and team B are in lock step, then the monolith allows changes to be deployed that would otherwise be considered breaking if deployed in a microservice architecture. However, a lot of monoliths will still be clustered and might talk to each other across the network, killing this advantage.

The reason that I've seen (therefore anecdotal) is even if you attempt to cater for handoffs with packages/libraries/classes/APIs, eventually a point comes where some of these don't exist in a vacuum. Or rather, they exist in a vacuum until they don't.

For example a change in one library might cause an unexpected degradation in an unrelated application. The communication at that point is strained as there are now multiple teams firefighting and playing the blame game. Or a slight change in one library caused deployments to fail for everyone. In a perfect world with perfect engineering practices this wouldn't happen but as many have pointed out, development is a reflection of the business, and the business is not perfect.

Having mini-services, so even just taking a monolith and separating it out into manageable chunks, reduces the communication problem immensely. Doesn't do away with it completely, but goes a long long way!

As if the blame game doesn't take place with microservices as well.

It is specially "funny" if the microservices are developed by several contractors.

And they often are! Because that's mainly one of the advantages... You can simply outsource a spec of the endpoints and that's it => flop over a ourservice.swagger.yml and get a working system back. Possibly in a language/framework you team has no experience with, but it was cheap.

Recent fun; a bunch of microservices written by the same contractor were killing the shared db (the 'state' of all microservices); so while most were running fine, some were hitting the db hard and costing the company quite a lot of money until it was fixed. No-one felt it was their responsibility to test & fix because it was an external company and they 'delivered' according to spec. There are many ways to do this better, but it happens quite a lot (also with monoliths ofcourse but there I find it faster/easier to diagnose + fix (depending on the size; I guess there is a point somewhere in LoC where microservices start to shine).

Yes. For example, a service starts sending bad data due to a bug. Now other services have acted on it, stored it, etc. It becomes a problem for many people.
This assumes the teams are using compatible frameworks, build systems, or even languages. A web API is often the easiest common denominator.

Then you get into deployment orchestration. Not every org can handle pure CI/CD. Even if they could they'd have to all agree to it.

It's much more difficult to evolve with technology or run experiments and you'd better make sure you pick the right frameworks and tech stacks upfront.
It is possible to keep teams "on islands" with a modular monolith. Each team builds a module. Logging, monitoring, and other shared infrastructure is there for all teams.
Until you come to the deployment phase where all teams need to communicate and deploy in sync. This can be a significant factor for slow down.

Also Microservices don't sync on the datastore, these are (in theory) not shared.

So yes Microservices can be emulated in Monolithic architectures, as can Monolithic Microservices be built. Usually you end up with the worst of both worlds. The complexity of Microservices with the speed of Monoliths.

Organisation and communication between teams is improved by having good APIs.

You can have that with or without microservices. You can screw this up with or without microservices.

The main factors for going for microservices are technical but, as always, must be understood.

you can do islands of communications in a monolith as well, with the added advantage of significantly lower deployment complexity as well lower incidental complexity
I view a working microservices application as a monolith with unreliable network connections between components. So overall adding a lot more complexity for the promise of lowers costs and scale-ability.

One advantage that I do see is the ability to deploy components individually, but I see more downsides than benefits overall. I hope that consensus is becoming "start with a (modular, well designed) monolith and scale out the parts as needed".

Bleah. I hate articles like this. They regurgitate all the well known talking points about enterprise software development but none of the realities of micro services. It sounds like they haven’t actually worked in the trenches supporting and coding real microservices at scale.

The realities are that going from monolith to microservices is a huge investment, and only large engineering teams should do this migration.

The reason is that once you go microservices You instantly require an immediate investment in devops which is something you didn’t need as much before with a monolith. Oncall rotations and pager duty schedules become imperative because services will go down. Also monitoring is another investment that you immediately need otherwise you’re completely blind.

The article talks about things that are obvious, like designing the apis etc, but those are the same regardless of if it’s a module within a monolith or a microservice. All the characteristics they list at the beginning is no different than what a team that owns a module in a monolith would require as well. It feels like this article isn’t very useful because it regurgitates concepts from regular enterprise development and packages as new for microservices when it’s not.

What the article lacks is the difference in architecting for a real microservice environment.

One of the biggest missed issues is never mentioning dependencies. Sure it’s great to talk about how to model services with eventual consistency blah blah. But if you already have a large engineering team, which you must have otherwise why have microservices, then there will be a dependency between services and to believe otherwise is crazy. You will probably have one or two services that when they go down, your entire company is down. It’s imperative that every service is architected with that in mind and what happens when it’s down. Hand waving “eventual consistency” won’t cut it in the real world, it’s the biggest problem that each team needs to focus on early otherwise it’s a ticking time bomb.

> Bleah. I hate articles like this. They regurgitate all the well known talking points about enterprise software development but none of the realities of micro services.

I disagree. The post is a good summary of all the key features and main concepts of a microservices architecture developed with domain-driven design. It does what it says on the tin, and you don't have to read three books to get there.

> It sounds like they haven’t actually worked in the trenches supporting and coding real microservices at scale.

I have no idea where you got that idea. It certainly wasn't from the blog post, though.

> The realities are that going from monolith to microservices is a huge investment, and only large engineering teams should do this migration.

Your rant makes no sense at all. You're just mindlessly repeating cliches about issues that have absolutely nothing to do with the subject. I mean, the blog post is about microservices software architecture with domain-driven design and clean code, but here you are whining about how you believe refactoring projects designed following an entirely different architectural style is hard.

My point is everything they talk about is common to any type of enterprise software development. It’s not new. And then they slap on “microservices” at the end so they can claim to have written something new.

If they want to write something new then talk about the realities of how microservice development is different than regular enterprise software development. Not regurgitating the same old concepts and pretending it’s eye opening and new.

Um, it may not be new to you but it is new and eye opening to me and there might be hundreds or thousands of people who haven't read about this. From a newbie's POV, I quite like how the author explains the concepts with the example of an e-commerce company. Sure there will be advanced concepts but the author makes no claims that he is addressing that audience :) .
> My point is everything they talk about is common to any type of enterprise software development. It’s not new. And then they slap on “microservices” at the end so they can claim to have written something new.

Again, your rant makes no sense at all. The blog post talks about a specific distributed system architecture style whose services are designed following a specific architecture style.

And here you are, angry at someone because... Because you believe the author has ulterior motivations?

> If they want to write something new then talk about

Sorry, but you have no authority to tell others what they should or should not write.

The author of this post contributed something to the public that's interesting and has value. Meanwhile what have you done?

It’s inherent in this comment that you really don’t understand DDD and why adhering to its precepts is a critical improvement in designing microservices.

Plenty of books to read, videos to watch. Learn/listen before you speak.

You have no idea what you're talking about. Monitoring and logging is just as important with a monolith, how else are you gonna notice when something is going wrong? In fact it is much more likely that your monolith will go down and lead to a significant outage because the monolith is bigger and thus has more failure points. Furthermore, a monolith that crashes will take _everything_ else with it whereas a microservice that is unavailable only impacts a part of your business. You can do some disaster recovery in your monolith to a certain degree but you will only be reimplementing things that K8S can already do.

> You will probably have one or two services that when they go down, your entire company is down

If you have microservices like that then your macro architecture is bad because one of the key points of an SOA is to limit the impact of failures. Probably you are using a lot of synchronous communication like REST because it was "easier" and now you have to pay the price. You chose consistency (which can be a valid choice) and now you complain about lack of availability when any undergrad knows you can not possibly have both (at a scale that requires partitioning). Using synchronous communication everywhere between microservices leads to a "distributed monolith" where you have to pay most of the costs of a SOA but get few of the benefits.

The one correct thing you said is that a microservice architecture is indeed a huge investment that requires sophisticated devops processes in order not to drown in operational tasks. Microservices are a good choice when you need to scale either in amount of users or developers beyond the capabilities of a monolith (i.e. you need an application that is partition tolerant). At that point you can chose between consistency + easier development or availability. No matter what choice you make, it will be painful but it can't be avoided.

> Hide implementation detail and expose functionality through intention-revealing interfaces.

Except "REST APIs" (what microservices expose almost always) don't reveal anything most of the time, wanting to hide internal state behind a beautiful and uniform facade essentially representing a two-tier model. For example, orders and order line items can't be added/deleted at will, but depending on whether the order has been processed or not, should expose a "cancel" action (with partial refund/with penalty for cancelling), etc. Sure you can hide that behind a DELETE method, or "reify" a "cancelled" status, but why?

> Services are resilient to failures.

In the case of "REST APIs", this means manually and labouriously attempting to reconcile state across multiple services in the client.

Compare this with state changes though ordinary RPC, which can expose a meaningful state model, give contextual diagnostics/exceptions, and can have transaction context flowing with RPC calls.

I've been interested and confused about DDD, hexagonal architecture, clean architecture etc. for a couple of years now but unfortunately have not had time to dedicate towards reading up on all of them. Has anyone actually implemented any of this as part of their jobs? I guess it all depends what you do. I mainly do web stuff, and Django/DRF lets us go fast. Out of the box, I think this goes against the aforementioned architecture as the ORM couples the domain model with the data model. Besides, gathering requirements (at least at my company) is unfortunately an ongoing thing which is just the nature of a startup I suppose.
I've been through a couple monolith to microservice transitions. I will say this... if you're monolith is slow because of it's design limitations, and your plan is to propagate those limitations across N microservices w/ all the overhead that comes with microservices, you're microservices will only be good at scaling the billable hours of the consultants who sold you on the idea in the first place.
> Out of the box, I think this goes against the aforementioned architecture as the ORM couples the domain model with the data model.

It doesn't. You're confusing the persistence model with the domain model. Your persistence model is dictated by your need to persist your data while meeting the requirements of your domain model. Your domain model does not care about your persistence requirements. Your persistence model is bounded to your persistence service and should not leak out, while your domain model is ubiquitous to your system.

Let's put it this way: with a Clean Architecture you are able to add/replace any persistence service you wish, and that has no impact on any other service.

Can you provide a concrete example? I would assume if you wanted to decouple the persistence model from the domain model then you would need to write a separate domain model in Django since the domain and persistence model are tightly coupled.
I'm not very familiar with Django thus my ability to provide relevant examples using this particular framework is limited.

However I direct your attention to the fact that Django docs make it very clear that to them in general each model maps to a single database table. That's quite Spartan even for an ORM, and obviously the point of a domain model is to model a problem domain, not to model the database.

Following the DDD practice at my current workplace. Hard to get into initially but it forces you to write modular code. Ramp up has been slow but nowadays, unlike previously, I'm able to write whole services with tests before even running them. And feel comfortable that they'll work. But that might not be solely down to DDD.
Yes, we did all that a few years ago and it's still in use. Clean code, hexagonal architecture, DDD, event sourcing.

Main lesson is that you should not use it for everything (yes, the books say it, but it's easy to overdo it in the beginning). Don't do DDD for the CRUD parts of your app. It works extremely well for business use cases, but it's very bad at CRUD. A mix of both really hits the sweep spot though.

No microservices though, just a monolith. Imo most businesses shouldn't need microservices, only the ones that really need to scale. We definitely don't have anything even close to that.

I wish hexagonal architecture picked used less overloaded words than ports and adapters. And while we're at it, I wish it used hexagons, or picked a different name. It's super obtuse to learn & teach.
I've used hexagonal architecture extensively. All of the typically claimed benefits are true. It's great for enforcing separation of concerns, and for permitting easy swapping of components. Great for mocking components in tests. Great for long-life software.

Most materials present hexagonal architecture in a very one-sided way, however. They don't discuss the downsides.

For example, every connection point between components requires an abstract API layer. This is a significant overhead compared to just having components call into eachother. In the long run, this is worth the effort. In the short term, there is no benefit.

Additionally, while it is true that the APIs can be flexibly hooked up to any implementation, there is still plenty of room for bad assumptions to creep into those APIs. When you have many adapters for a given port, those assumptions are entrenched, and are difficult to change.

I've been on a few projects now that attempted to implement most of these patterns wholesale. The idea is simple to grasp and implement with a small team, but it's been nearly impossible maintain any cohesion for larger organizations. Most enterprise programmers are at least familiar enough with MVC that it's not an issue. But very few in my experience have actually worked on a project that adopted DDD. Some have never even heard of it.

Personally, even following all the guidelines laid out by the famous book and other ancillary documentation and articles, such as this one, the whole thing can fall apart quickly when you have a business requirement that doesn't match your technical architecture. You will at some point have to cross-contexts and communicate between domains. In Martin Fowler's article, this is literally hand-waved. I think there's maybe a few sentences on it?

Here, the suggestion is to use 'event storming' to make sure you design your services correctly from the start. Okay, what about if your business pivots your initial assumptions are wrong. Our team has adopted an action approach, but it goes by different names (channels, user actions, business actions) which model a single action and compose sub-actions from different domains, usually in a single transaction. This seems to work, but now we're operating in some non-aggregate space that couples all the domains.

So basically, the most difficult to manage and complex scenarios, where you model your cross-context flows, are never described beyond, use async (not always possible) and re-organize your services (also, not always possible). Would love to hear if anyone else has any experience here, or maybe I'm missing some fundamental part of DDD.

I’ve been using DDD with Django/DRF and as a mid level, I’m confused as to why we aren’t leaning on native Django more and writing less boilerplate. I don’t know if this is a common complaint though.
Do you have any examples how you accomplish this? Do you have separate domain models which are decoupled from Django's models? Do you use the repository pattern?
I’m hesitant to give examples unfortunately but yes to the above and lots of what we do is bespoke whereas we could leverage Django for a lot of it. How are you putting it together? Independent Python classes that exist outside of most of Django core?
You don't need micro services to do DDD - you can do DDD within a Django app. I'm not a Django man myself, but a quick google shows several projects and blogs discussing it.

In essence, DDD forces you to have conversations about separation of concerns, and then being careful to stick to them. Microservices force that, but you can get the same effect with well-thought out namespaces in a monolith.

This talks about the thing that makes microservices fail most often, getting the bounded contexts wrong. It doesn't go into enough detail to show you how to do it right but does tell you what to watch for when you're doing it wrong.

The biggest, most common mistake is when a bounded context is called "<noun> service". This should be a huge red flag. Unfortunately, most books (and this post) always use this bad naming starting everyone down the bad path. At the very minimum call it a "<noun> management service" if it is the primary/general create/update activity. e.g you might have a checkout service that creates most orders and a order management service that can make drafts or adjustments. Order management (being the decoupled source of truth) might even be downstream from checkout service. Other services that need this information should get it's own copy asynchronously, i.e. decoupled.

The best way to learn how to do bounded contexts right is to study ones that worked well. Most writing tries to describe this in an abstract way using placeholder widget service things that helps no one.

The BFF pattern is good but doesn't solve everything. If you're accessing multiple services for a synchronous mutation, it's likely not highly reliable but can get by for an interactive service that can tolerate some errors. A messaging service is not one that should, they should always accept, store and forward with the lowest possible error rates.

I might be a bit cynical / neckbeard, but the tech industry tends to go in circles with good engineering principles sold as the newest fads / hype. Before all the microservices craze it was REST and APIs, before that web services, SOA and ESBs, and before that XML-RPC / CORBA, EAI, and enterprise Java beans. And before that we had, we someone we had RPC over the network (‘open systems’).

Similarly DDD promotes separation of concerns and test driven development. That is exactly what eXtreme Programming told us in late 90ies when OOP and unit testing was in. Before that it was structured programming and software design specs, 4GL etc.

Bottom line is I wish people would start calling a spade a spade and indulge in less fads. Let’s just call it structured programming over the network with clear separation of concerns and clean interfaces. That’s good engineering. Test early and often. Write good documentation.

I'm completely on the same page with you on that one.

I understand though the marketing needs. IT has this steady influx of beginners that don't have the insight yet. Since there are so many of them, at any given time, it's reasonable (or maybe it actually works) to use standard advertising techniques at them. "Secret sauce", "The missing puzzle piece", "10 things that make you a better dev", "Dev done correctly".

Developers are kind of a dream target group: lots of disposable income, technically adept (i.e. at the cost of social skills), excited to experience a latest unicorn.

I'm seeing this whole "hype cycle" thing as a competition among more senior, adept in marketing, developers. Since there is such a high demand, the supply is also rich.

Could/should there be a 'wikipedia for software development practices'?
What about https://wiki.c2.com
Genuine question: whats wrong with wikipedia itself?
I have often wondered this.

For some reason, important information isn't making it there. There's no page for Chain Replication. Maybe nobody has tried to add it? Chain Replication has been around for so long that it seems unlikely nobody would have tried to add it.

Wikipedia also seems to fall apart at the seams. It's great at giving you the answer to questions for which there is an unambiguous answer, but it doesn't do a good job of answering all the other questions you might have. Why is Prolog not very popular, what seems to be holding it back? Any answer to this is mostly speculation, something which is not allowed on Wikipedia.

The problem is that way many aren't able to sponsor conferences, do talks about "how we rewrote X in Y", write blogs about best practices and sell books.

Yes I am being cynical as well, but have a quite filled up bookshelf to prove my point.

> DDD promotes (...) test driven development.

It does not, nor does the blog post say so. The two are completely orthogonal, and while TDD is a rather questionable development method, DDD is a widely accepted design principle.

> TDD is a rather questionable development method, DDD is a widely accepted design principle

These claims are orthogonal, so the "while" construction is misleading.

TDD and DDD are both arguably questionable and both undoubtedly widely accepted.

> I might be a bit cynical / neckbeard, but the tech industry tends to go in circles with good engineering principles sold as the newest fads / hype.

Not cynical at all — this is well observed and I wish more people saw it. I think software engineering is more susceptible (permissive cause) to fads because so much of the work is more craft than science and the actual catalyst (proximate cause) for the faddishness is the economic incentives driving the posturing for "thought leadership".

Consultancies need to compensate for generally not having a long-term stake in the projects they work on in order to overcome skepticism from clients so they have to invest in content marketing via thought leadership. The principles of marketing don't change -- convince people they have a problem, sell them the solution.

In a similar fashion every developer has various incentives to either consume and propagate or create this kind of content as well.

The only check on faddishness is well-grounded technical leadership.

I think you're missing the point. They're not fads. They solved a problem for certain people in a certain generation. Technologies and trends rarely make it beyond 10 years and every 10 years we come up with new technologies, new abstractions, for new generations.
I agree with some of the other older hands in here, much of technology 'changes' are rarely anything new, just new names and tweaked implementations of much the same core elements. New generations like new names, worded in terms they become familiar with. Doesn't make them less valid, but often not as new as perceived. Much like the music business, new bands put out covers of old music, new audience thinks it's a great new sound, but underneath it may be very much the same old song.
This is really not true. While every industry, and especially Software Engineering, is prone to hype, the reason things become popular is that they deliver real value.

REST API's and microservices are a design that originated when software teams started growing in size and didn't want to lose the agility they observed when having control of their complete stack. It was enabled by a fundamental shift in infrastructure: previously, everything was managed by dedicated infra teams, but Containers and Container Orchestrators gave development teams control of their deployments. The old model of monolithic services stood in the way of developers who had access to this new power, and hence microservices were invented to address that.

I could give you similar reasons for the rise of Big Data, Infra as Code etc. Every one of these changes are driven by changes in fundamental technologies. They aren't fads.

When I read the article and looked at the e-commerce diagram, all I could think was "hmm, this seems to be pretty much exactly how we implemented the early versions of amazon.com, except without any of the this technology or terminology".
Don't forget cohesion. Microservices achieve the highest cohesion usually since the microservice does just one thing. It's reasonable to combine services together, as long as cohesion is maintained.
Microservices are a good way to scale up development teams. So if you have 16 developers you can partition the work in two teams with one microservice each. If you have 80 devs you can have 10 teams with one microservice each team. Use Conwayś law to your advantage. Use DDD to decide where the boundaries are.

The advantage of having each team own a microservice is that each team can deploy their software independently from the rest of the teams. And it works as long as the APIs are honored. So Microservices are way to decouple the work of different teams allowing them to work independently from each other.

This only works for the simplest cases. If the two microservices must share data and direct calls (API calls) are not allowed (because they make your services highly coupled to each other to the point where if one of them ia down, then the other cannot work properly, or simply because your use case cannot tolerate the latency overhead) then each service must 1) replicate the data needed locally by listening to interesting events of the other service and b) emit interesting events so that the other service can update its local db. In other words, async communication.

While this let's you scale up teams, it's a huge overhead. So, unless you have over, let's say, 500 devs, it's not worth it.

> This only works for the simplest cases. If the two microservices must share data and direct calls (API calls) are not allowed

Actually, one assumption is that the microservices don't share data. Only API (Rest, SOAP, GraphQL, etc) calls are allowed. And the boundaries are created minimizing the coupling between microservices.

> While this let's you scale up teams, it's a huge overhead. So, unless you have over, let's say, 500 devs, it's not worth it.

As opposed to Monolith ? I think you will have problems collaborating in a Monolith at around 30 devs.

What I am opposing is the idea that you can have 30 devs and 50 microservices. Which I have seen in real life.

Why do you need Microservices to create boundaries in your code?

You trade the ability for teams to deploy independently for the ability to be able to make breaking changes. This is a good trade off in my experience and is better for team velocity.

The approach we have taken is to keep a single codebase and deploy. We have applied the ideas from DDD to segment the code into contexts and assign ownership of each of those contexts to a team. This allows us to get some of the benefits of Microservices without losing all the benefits of a monolith. Having these boundaries in place also gives us the flexibility to move to Microservices if and when we decide we need to.

Ultimately the point of separating code into microservices is to develop the interfaces between teams. They certainly do not make sense compared to a monolithic app until a certain scale.

When you have 10 teams working in 1 mono-service, who is responsible for which parts of the DB schema? Can I query a table in one part of the app that another part owns? What if another team changes a table that I relied on and introduces a bug? Who is responsible for fixing it?

Microservices solve the people problems more than they solve the tech problems, assigning organizational responsibility for functionality, infrastructure, strategic objectives, etc. It also provides clear interface points between these things that can make organizing work easier. You get to split your 1 org into 10 little vertical orgs. This has benefits beyond just code boundaries. Now you can have 10 different goals all being achieved at once

You can go the other way, of course, and split the stack horizontally (One team is responsible for infrastructure, one for DBA, one for Frontend, etc) but then no team is working towards any one goal. Who is responsible for making forward progress?

The realization I’ve had after scaling a dev org from a single team to 10+ is that you don’t need microservices to achieve any of that.

> When you have 10 teams working on 1 mono-service, who is responsible for which parts of the DB schema?

You can still develop interfaces between teams without microservices. There are a number of ways to do this but the simplest is just a folder structure that reflects the boundaries and enforce them by convention. If the model/repository/[your favourite DB abstraction] lives in `context1/user` you can’t access it from `context2/account` without going through `context1/publicInterface`.

It’s surprisingly easy, depending on your language, to write tools to automatically verify boundary violations if you need to.

This still allows you to solve all of the problems you listed .

> Ultimately the point of separating code into microservices is to develop the interfaces between teams. They certainly do not make sense compared to a monolithic app until a certain scale.

Exactly. This is the idea that I haven't been able to articulate. And it is rarely addressed in the literature.

bingo. having each team own he boundary of a service lets everyone work as fast as possible, making choices that are right for their service, while preserving the right level of abstraction between components.

that’s very different from a team of 8 people managing two dozen microservices, which is just insanity.

I see the writer is from "WalmartLabs", so I assume this is advice for "Walmart scale" applications. For the rest of us it's probably good to remember that "you are not Walmart."

> Now, we know that in any modern application, sacrificing availability is not a good idea either.

I feel like this will apply to less than 10% of applications. Normally you'd rather deal with the occasional downtime than dig through the fallout of broken attempts at distributed transactions. Ask yourself "What amount of $X,000 are we losing per minute of downtime" and see if it rounds down to zero.

> Another way to model this system is to separate, or group related models into separate microservices. In DDD, these models — Price, Priced Items, and Discounts — are called Aggregates.

And this must be the case for less than 1% of readers - it would take a massive system before I'd consider having prices and discounts in separate databases. You're taking on the overhead of writing an additional service for what could have been a database join.

Wow, microservices sound like a very painful trade-off to make. Is it really worth it?
Microservices are for when you have no other choice. If you truly cannot build a monolith, then build microservices. If you have merely chosen to adopt a microservice architecture when a monolith would have been suitable, then you have made a huge mistake.
It definitely can be, if it’s done for the “right reasons”. Mostly the reason would be to scale teams but still keep agility.

It really couples nicely with DDD and I’m in the middle of my second project where these principles are used.

I would not do it unless there’s a dedicated platform team that will take care of all the infra pieces you’ll require. This kind of points to the scale where it makes sense, I guess!

> Hide implementation detail and expose functionality through intention-revealing interfaces.

Great, then a new US comes in that changes the intention and you enjoy the pleasure of changing the interface. Enter versioning. You start crying.

> Great, then a new US comes in that changes the intention and you enjoy the pleasure of changing the interface.

That's not a microservices problem, not even a software architecture problem. That's at best a project management problem. In fact, pretty much every single engineering project there ever was would struggle to meet any form of neck-breaking change in design requirements.

But with microservices you have more supposed independent interfaces. With a more monolithic approach you might just change a system internally and deploy it. Internal interfaces are generally easy to adapt.
> But with microservices you have more supposed independent interfaces.

You don't. You have more services accessed over a network, and might not have control over all services. But that's already par for the course in nonoliths.

If you were to refactor your components by changing classes around (say, a major version update in a dependency) while not updating the components that used them then you would also have problems.

That's why versioning and service discovery and HATEOAS are a thing.

But the compiler will find that. It‘s very different from deploying services. (And I‘ve been doing both, for years.)
One of the problems with forcing strictly disparate domain/service is that each one introduces its own security context and authZ handling. That's a lot of places for mistakes.
My platform team supports devs with a proxy that handles authN & authZ. Service to service is secured using mTLS automatically.

I’ve stated it previously in the thread - you need to enforce conventions through tooling - which is where “devops” (ugh, I know, but really...) comes in to play.

We try to view the tooling platform as just another DDD domain.

You're just stating that distributed services need to have security.
I think they stated that distributed security is more work / harder than monolithic security. I would tend to agree.
In my experience, the reason it is more work is because you end up with something much more well defined and robust.

Each service having well scoped and defined RBAC or AuthZ for its focused set of features makes the whole architecture as a whole much easier to reason about from a security standpoint. I've done successful pen testing and auditing of some monoliths in my time where the critical security issues arose out of untested and unexpected execution paths that were only possible because the surface area is so large.

Maybe in theory a "well written" monolith would be superior but I'm only going but what I have seen in practice. I think the extra work is worth the trade off.

I'm really having a hard time finding the willpower to read articles that just even mention "microservices" in the title after working on such architectures (with Java) for a while.

Nice idea but most of the times horribly implemented, with no care about monitoring and logging as someone has already said.

It requires a degree of attention to details during design and development that few are willing to put in. It's easy to go for a prepackaged solution/framework thinking that you are building something that could be considered state of the art, when instead you end up with something that takes multiple tries to even boot correctly or 10 minutes or so to "stabilize" a garbled mess of inter-dependant components.

Designing these systems require a whole new set of skills that were not required when designing monoliths.

You really point out that you’re going to require a dedicated team to manage all the things you’ll need. I you’re not at this scale there will be a lot of overhead to manage for the dev teams.

I’m in a team like this, supporting a bunch of developers - we build a CLI, an API (in our domain! It’s just another domain with the devs as consumers) and juggle approx. 16 different tools (monitoring, metrics, tracing, service discovery, etc etc).

Doing it through custom tooling that enforces conventions is, IMHO, the way to do it successfully.

This is precisely what the team I lead is responsible for as well. We’ve got a set of standardised libraries, deployment conventions, and a CLI which encapsulate almost everything needed to have a service coexist within the wider platform in a consistent manner.

We’re also responsible for wider reaching services within the platform such as an API gateway, authentication, and event ingestion from IoT devices. As the team that has an on-call rota we also give the final go/no-go on services going into production.

There is a lot of very interesting work to be done in this area (amazing books coming out lately from O'Reilly about these topics) but I feel there is still some piece of the puzzle missing. We need more tools/practices for the developers to guide their development and understand how an architecture behave while they are still building it, to drive design decisions (and also check/correct them along the way).
I worked for a company that had tried to adopt microservices (without a good reason to) and failed miserably. Over time we were able to re-architect it into a "happy medium" of a few large components. It was a huge, costly effort... and I've never learned so much.

Beyond technical concerns, I learned that you can architect a looming disaster and then leave for a higher position at a big company. Nobody watching your conference talk will know.

Off topic but wanted to flag:

> [...] we intend to cover our experiences with Event Storming in a separate blog.

In case anyone is not familiar with this usage, these days "blog" is often used instead of "blog post." I was thoroughly confused by this when I first ran into it. I tried fighting against this usage by colleagues, but it seems to be widespread enough now that it cannot be fought... I guess language evolves, and I have to get with the times.

I suspect one or more of the following aspects might be at play: • general loss of identity (since lots of posts are on medium, and less on a personal and work blogs) • "blog" is short for "blog post" • desire to distance oneself from "post" which is more associated with facebook and instagram posts

I don't read any more into it than someone saying "put it on the USB" instead of spelling out USB stick/drive/etc. It's just a lazy abbreviation that caught on.
It was indeed confusing.I honestly thought that the author was using this article as a clickbait setup to drive traffic to another blog.

Blog: web log. A collection of entries that are used to log something and is made available to the internet.

Blog post: a specific entry posted in a blog.