One of the advantages of working for a company like Terem is that we get exposure to a wide array of teams, with different ways of working and preferred architecture patterns. Over the years, I’ve come to realise that while there are “good” and “bad” patterns, the more important consideration with architecture is problem/solution fit.
As Cloud computing has taken off and reduced the effort involved in infrastructure provisioning, the microservice pattern has picked up steam, becoming a standard pattern that teams default to using. While the microservice pattern is a great way to break up a large product and distribute responsibility among teams, it’s not always an appropriate pattern. Additionally, teams can inadvertently end up building a “distributed monolith” rather than a series of truly independent microservices.
Identifying that you have a Distributed Monolith can be one of the first challenges.
How Do I Know If I Have a Distributed Monolith?
The simple answer is that your services are a distributed monolith if they are tightly coupled and cannot be updated in isolation from each other. For more nuance and detail around this, read on for some diagnostic questions that will help you answer this question.
Identifying a Distributed Monolith
Here are some diagnostic questions to ask to identify if you have ended up with a Distributed Monolith. Green flags are things that point to your architecture having a healthy microservice architecture. Red flags are the symptoms that might make you want to rethink your approach.
1. Can you deploy and scale services independently?
The main value proposition of microservices is the ability to update services independently. Some applications are a poor fit for microservices as they require tight integration between features and data types. Other times, you may have a viable path for microservices but have inadvertently built tight coupling between them, nullifying potential benefits.
- Green flags:
- Deployments of services are completely independent
- Services can scale independently of each other
- Deploying one service has no impact on any other
- Red flags:
- Deployments need to be coordinated across multiple services
- Services cannot scale independently
- Frequent “all or nothing” deployments
2. How coupled is your data model across services?
A common pitfall with microservices is “all-or-nothing” thinking. We’ve seen teams go to great lengths to completely segregate their data into separate databases owned by separate services, then turn around and build a complex syncing layer between them all because they actually need access to each other’s data in real time. Instead, you might need to pair your microservice architecture with a shared data layer (using database roles to limit write access where appropriate).
- Green flags:
- Databases and schemas are shared where possible
- Transactions distributed across services are infrequent
- Services retain data and access ownership with well-defined contracts
- Red flags:
- Multiple services reading/writing to the same database tables
- Frequent use of distributed transactions for data consistency
- Lack of clear data ownership boundaries between services
3. What is the nature of inter-service communication?
The simple test here is whether or not any of your services become unavailable if another service is down. One of the key benefits of microservices is localising failures/downtime, but this benefit is often missed when microservices are built with the assumption that all other services are always available. Especially if there is no plan for when they’re not (graceful degradation is a good strategy here).
- Green flags
- Appropriate usage of synchronous vs. asynchronous communication patterns
- Shallow call chains between services
- No circular dependencies between services
- Red flags:
- Long chains of synchronous service calls
- Services failing due to other services being unavailable
- Circular dependencies between services
4. How does a change in one service impact others?
Services should be owned and operated independently, but there’s no way to avoid strong contracts and communication between teams. Service boundaries should be owned by the team managing a service, and contract changes should be backwards-compatible and communicated appropriately with relevant stakeholders.
- Green flags:
- Infrequent changes requiring updates in multiple services
- Low complexity of coordinating changes across services
- Existence of implicit contracts between services
- Red flags:
- Changes in one service frequently breaking others
- Need for extensive regression testing across all services for minor changes
- Difficulty in understanding the full impact of a change
5. How granular are your services, and do they align with business capabilities?
This is one of the key problems that arise with microservices. Where do you draw boundaries between services? How many services do you need? Your service architecture must pragmatically align with your business and program/project goals rather than being created in isolation with a focus on technology rather than business outcomes.
- Green flags:
- Size and scope of individual services align with business capabilities
- Alignment of services with business domains or capabilities rather than technical concerns
- Infrequent changes crossing service boundaries
- Red flags:
- Services are split along technical rather than business lines
- Overly fine-grained services that don’t represent meaningful business capabilities
- Frequent need to update multiple services for a single business change
If you read this section identifying more with the red flags than the green, go back to first principles and make sure your tech approach aligns with your business needs, and not someone else’s “best practice”.
Distributed Monolith Remediations
Unfortunately, there is no easy way to move away from microservices that are tightly coupled, and the specific remediation for your organisation will be quite individual…
…but that doesn’t mean all hope is lost!
Here are some measures that will help you remediate your situation if you’ve inadvertently ended up building a distributed monolith.
- Simplify and consolidate services that don’t need to be separated. A good rule of thumb is that you don’t need more than one service for each 5 to 10 engineers. The whole point of microservice architecture is to ensure many engineers can update the same application while minimising the occurrence of conflicting work. Your number of services should scale with team size.
- Define clear boundaries between services, and create strong contracts that define how they interact or share resources. Make sure services changes do not require updating any other service
- Ensure you’re not creating arbitrary rules or boundaries that adhere to what someone else defines as “best practices”, instead make sure you are making pragmatic decisions that are appropriate for your team size, maturity, and of course your business needs
It can be tempting to maintain the status quo rather than stating an unpopular opinion such as “we need to change our architecture” or “our architecture is too complicated for our team to sustain”. The only motivation to do this is recognising that your current situation is a source of technical debt, which means it will slow down future work until it’s resolved. While it’s difficult to align teams on technical approaches, when it comes to the language of time and money, a pragmatic plan that reduces ongoing time and costs associated with development will always have a great chance of success.