Tips on System design from a 20+ YOE Engineer

Question by @user1287o
What would you recommend to improve this skill?

Answer
This is a great question. I believe (and this is from my interviewing experience) if you are not working in building distributed scalable system expecting you to ace the design interview is a very bad expectation. Your design question should be suited to match your skills. Having said that you can learn it though it will be difficult. Let me try if I can divide it:

• Take a system and draw various components (for example Amazon ordering system or it can be anything). Clarify requirements enough and go deep in few critical areas and cover breadth. If your interviewer wants details about specific they will ask. Some designs have a specific solution and you should know them like building auto-complete, building a rate limiting system etc. Some can be open ended. Try to get those signs.
  Do some calculation. How many request? How much you are storing per request? This will give you an idea regarding scaling needs. One way to think about it is you are launching a new feature, identify your daily active users and see what percentage of them will start using your service. Typically not everyone will use your feature on Day 1.
• Important components and common to most of the distributed systems are LB (Load Balancer), your application/service (hosts on which your code is running, I have mostly worked on service which allows communication to them via ReST endpoint and so knowing ReST is critical like ReST verbs GET, POST, PUT, DELETE etc. You do not need to go too deep but think of API versioning, API abuse (throttling, authentication (mTLS, OAUTH, SSO),security, compliance etc. No one expects you to give details of all of these. But mentioning them is good. And always mention things which you can go deep. Do not throw buzzwords or things that you do not know), cache (leaving on the same host or has it's own fleet), your data store (database, object store etc. SQL/NoSQL tradeoffs, CAP theorem, does your use case needs consistent data or eventual consistency is okay?). And you build redundancy/fault tolerance across all of these components or have a plan in case one of them fail. Always avoid single point of failure and talk about how you deal with failures.
• Now some important concepts/questions to think about. Do you need to process the request synchronously or just accept it and process it in an async way. This changes your design. If your request doesn't need to be processed in a synchronous manner, you can use some kind of distributed queue to spread the load across your host, spin more workers to process it fast or add more host etc. Now think what is important latency, availability etc. You need to define your goals. If latency is critical you may need to cache. But caching can be tricky. Known different kinds of caching strategy (write through, read only with TTL etc. it depends on your use case and how much staleness you can tolerate in your design).
• Define your API signature, request body, response etc. Once you have this you can start defining your data store schema. You can do it other way round too.
  At this time you have enough thing to draw a decent high level architecture or sequence diagram. Now start identifying bottlenecks. Think what can break and how. Too many request, data store not able to handle amount of data (do we need to partition data/sharding? some database does this automatically) etc. identify bottlenecks and think how can you handle them. Again service breaks and we have outage. The point is how easy is to restore the service back and can you identify these early enough. Talk various trade offs ton minimize the impact. Again depends on lot of factor and you just want to mention few and write their pros/cons and move on. Is regional service is better than global ones? Do you have fault tolerance built as part of availability domains? Again no right/wrong answer here.
• One of the most critical things that we need to build in any service is monitoring. Again there are standard ways of emitting metrics and monitoring it via dashboard and set alarms on them. but mention them. If things go wrong do you have enough logs to troubleshoot? Again what are your logging, too much log can have other problems.
• Analytics/ML - if your use case has a need for it
• Testing - you should know how to test these (again a beast in itself but knowing it is good)

There are lot of materials on the web which discusses these in detail, but having a real world experience makes it easy. Ideally:

• LB (various policies, I have mostly seen weighted round robin)
• Data base (SQl/NoSQL). Different kind of NoSQL. DynamoDB is key value pair. Also know about Cassandra, some column based database etc. MySQL and Oracle is still being used. How do you scale your database? How do you store relationship in a DB?
• Persistent store (Object store) - mostly for storing blob/media elements like Amazon S3, Oracle Casper etc.
• Distributed queue - Amazon SQS, Apache Kafka etc.
• Cache - Memcache, Reddis
• For some design question you will need to know push vs. poll. Again XMLHttpRequest vs. web socket etc. These are mostly for building timeline and WhatsApp kind of feature. See what kind of pub/sub model are available. Amazon SQS with SNS, triggering a lambda function on an event, know about some streaming service.

I am not sure if I answered your question or confused you more. But in these interviews this will be my strategy.

• Gather requirements (do not assume anything). Make a bulleted list of requirements. Not more then 2-3 minutes.
• Define Goals and non goals. Non goals doesn't mean they are not critical. Non goals means you will not talk about these unless interviewer wants to talk about question. Example, messaging system needs encryption to be discussed but building a game leader board might not. Again good to say that you know these things are critical but you will not cover these unless asked. 2-3 minutes.
• Back of envelop calculation - data size, request count etc. 2-3 minutes
• API design - 5 minutes (define API, authentication etc.)
• Data store - pros/cons tradeoff 5 minutes
• Overall architecture - draw it - 2-3 minutes
• Start identifying bottlenecks and propose solution, your architecture diagram will change. for example if you are adding caching, queue etc. - 10 - 15 minutes. This is where you are diving deep and showing your knowledge and expertize.
• Summarize what you covered and ask interviewer if we need to cover something else.

Collected from comment section and thought useful, upvote so that everyone can see!