System Design: Distributed Systems | Disaster Recovery | Monolith vs Microservices Architecture
Vruttant Mankad
6021
Career

Hello Leetcoders!

In this post I will be discussing about disaster recovery in distributed systems. First I'll give you an overview on monolith vs microservice architecture and continue from there for better understanding.

I suggest people who are new to distributed systems design to check out my previous posts on the same to get a better idea on this. Though, I'll keep this article as beginner friendly as possible so no prerequisites are required.

Monolith Architecture :

  • A general misconception is that an architecture with just 1 physical unit as a server and 1 physical unit as a DB is a monolith architecture but as soon as we add more servers its not monolith anymore. This is incorrect.
  • An architecture where a set of servers perform the same set of functions and can be scaled horizontally to accumulate more load is also a monolith architecture. For Example :

image

  • The above architecture is also monolithic as both the servers and Databases perform the same functions (The code running on both servers is same and any type of request can be handled by any server).
  • A little background for those who haven't seen my previous posts:
    • The reverse proxy load balances the requests on to the 2 servers and the DB master(writes+reads) and replica(only reads). You would notice 2 instances of everything. That is because we want to avoid single point of failure.

Microservices Architecture

  • Here, Each seperate service runs on a different server completely. For Example, A server to serve payment requests, one server to perform authentication, etc.
  • The decision on which services to seperate and which to not, is the job of the systems architect and is very crucial as you wouldn't want to seperate services all the time due to extra resources required and increase in RPC (remote procedure calls) between the services.
  • This is a very simple explanation of the architecture , and you can read about it in detail online. I'm covering the simplest form here to demonstrate the Disaster Recovery Part.

image

  • The above architecture is the simplest form of a microservices architecture. Each Business unit has its own server and a DB.
    • Note - The different servers can have the application code from different programming languages. E.g. Payment service coded in JAVA and Auth service in Python. This is no issue as these services only communicate with each other via API calls.
    • Note - The Database Systems can also be of different providers and can be SQL and/or NOSQL . Its the application's job to form a consolidated output out of this.

Monolith vs Microservices

  • Monolith architectures are suitable for small teams and where you have resource constraints. But , at the same time , a new developer would find it difficult to understand the already setup monolith architecture as all functionalities are on one server and it can be tough to determine how each process communicates with one another.
  • Microservices decouple the system and one can work on one particular service without having to know the details of the other services. Hence, suitable for large organisations.
  • Microservice architecture also prevent the whole system going down in case of very heavy load. Only the affected service would go down and other services will work fine. Monolith servers would make the app completely go down even if one service receives very high load.

Disaster Recovery(DR)

  • Coming to the crux of the article, why would we need DR servers ?

    • What if the whole datacenter where your resources are kept is affected by natural or manmade disasters ?
    • What would your app fall back to? What about the data of your customers ?

  • We need DR servers to make sure the above problems' effect is mitigated as much as possible. And, as you might've gussed , we would put DR servers in a different Data center in a different location.

  • Question is how would we do this ? Well it is easy in a monolith architecture . For Example : DC1 is active and DC2 is for DR
    image

  • The only extra thing we need is to resolve the DNS of the application to point to the IP of the new data center and we are fine.

  • You might also notice that the DR server's DB replicates from the master of the active DC. This is to ensure that data is transferred to DC2.

  • Now lets see what would happen if DC1 is hit by a disaster:

    • The DC 1 services go down. Some monitoring software that we have used to monitor this detects this and the system admins resolve the DNS to point to DR's IP .
    • The DR's DB is promoted to act as master and it can now accept writes.
      • The running transactions would be lost (which is fine as data is not inconsistent). But, everything that the DC1 master committed, will be available at the DR's database as the replication is generally asynchronous.

  • Clients would now connect to the DR server and functions are working fine. (Simplest scenario here for better understanding). In reality system architects would have to check for replication lag and network lag before bringing everything up.

BUT, How would we do this in a microservices architecture ? Some things that would create problems

  • We would need many resources as we cant say that we can replicate 5-10 servuice's data in one DB as I mentioned that DBs of each service could be different.
  • So, we have replication setup from each DB instance to each replicate instance in DR server. But, consider this:
    • You placed an order and made a payment (order and payment are different services). OrderDB's data was replicated but Payment happened just a second ago and it wasn't replicated to the DR server and the disaster occured.
    • What now ? you payed but the data shows you didn't pay according to the DR's DB. BIG PROBLEM Inconsisent data!
  • Now we have a problem here . When backing(B) up an entire microservices architecture, it is not possible to achieve both availability(A) and consistency(C) - BAC Theorm.
    • We obviously need the backup. To achieve availability, we would want that each microservice asynchronously replicates its data. But, as mentioned, this creates inconsistent data.
    • To achieve consistency, we would want that the microservices replication happens synchronously and all DBs should replicate together. This hinders the availability as updates cannot happen when synchronous replication is going on.

Lets see the architecture that would help us solve this :
image

  • The monitor worker is a worker which will send state reports of the services and database to the master monitor from each DC.

    • Here, each service needs to keep a timestamp to the changes it makes and the monitor worker periodically sends the state updates to master monitor.

  • Hence, the above architecture shows you both the solutions. If you want consistency, have synchronous replication setup. If you want availability, setup monitor workers.

    • Now in case of a disaster, its the operations team's job to replay the updates received to master monitor from DC1 to DC2. As DC2 is inconsistent here. They can also choose to bring it to the previous consistent state and scratch the recent writes completely.

Summary

  • DR datacenters help in mitigating loss of data and operation in case of disasters.
  • Easy to implement in monolith architecture.
  • BAC theorm states we can't guarantee availability and consistency in microservices DR.

This was a very ideal and high level view of the Disaster Recovery process in monolith and microservices architecture.
Do share your thoughts and correct me if I'm wrong anywhere. Thanks for reading!

Comments (6)