Folks, I got interviewed at MoEngage for SSE role.
YOE: 2.5+

This is going to be a long read.

I got this asked: design a notification sending system which sends notifs to users on the basis of their preferences. e.g., in slack you can specify the kinds of notifs you wanna receive(just mentions, all notifs, just teams, only channel tags and all).

I approached this like a hybrid model: design + simple (basic) search:

1. design: firing an event (in queue, like Kafka / pub-sub) as soon as message is published. message looked like below
2. basic search: fetching all users (just their ids) given the channel_id.

message_posted event:

{
  "content": string,
  "channel_id": int, 
  "type": enum (regular | channel | here | mention | team),
  "createdAt": unix epoch
  "createdBy": int (user id)
}

I specified the data type of content, type only in the interview while designing.

I was asked how I'll efficiently figure out the number of users associated to a channel. At this point I introduced a db and in turn a table / clxn that'll keep the user - channel mapping. Now I was asked the entity schema. I had 2 paths both of which that I jotted down on the excali board:

1. a user clxn where I keep a field channels (a list of channel ids) in each doc. I build an index on channel_id field and perform search
2. a channel_user bridge collection where I create 1 to many mapping of channel to users (10 users in 1 channel are represented by 10 entries)

I got prompted here about how I'll perform the search in the list. Which I said that I don't want that either but going with #2 I can have table explosion (as 10k users in 100 channels would mean 1M records), hence hesitant. He gave a suggestion / hint which basically meant that high runtime compute times are worse than spending money on a disk. I said exact thing to him acking what he said, and chose #2.

Now, he asked about how I'll perform the search in table if I have 1M users in a channel. I took this under consideration along with that if there are multiple channels like this and messages keep getting posted in them then we'll need to partition our table on channel_id basis, using which our search space would reduce and now we'll perform deep pagination via performing batched searches on indexed field (channel_id) while applying sort every time on id, so that I know exactly where I left off and where to start the next query from (batches of 500 size, with sortable ids(ints for now), e.g.,
1,500 in first batch, for second query: id > 500 and id < (500) + (500)), instead deadly limit / offset combo (not sure if he listened it, he rushed to next question).

The next question: what if this query takes 100ms?
I asked: Is 100ms for marshalling the entire row or just hte user_ids? Because if the entire row, then I'll marshal only the select fields (id and user_id).
Him: Ignore the network latency, 100ms is search time solely at db end.

Upon calculating total time: it was 200s and the SLA was dispatching notifs within 90s since message was posted.

At this point I was out of choices and had no option but to introduce a cache(redis-esque) on channel_id basis, e.g., channel_id: [ user_id ] (list of user_id and nothing else)

I was going to complete it, but got the interruption about maintaining consistency b/w it and the db.
I backed it with when channel table gets update then we fire a rety backed FNF event to update the redis (wanted to back this with scheduler(daily frequency) but forgot and was being bombarded w/ questions).

He asked and I gave the same solution for retrieving the user's notif prefs.

By this point, he said that it's fine we would be able to comply w/ SLA.

At this point he made me do some BOTEC which I was going to do earlier but he stopped me:
Task: estimating the ephemeral storage that 1 instance of my service would need.
Given:
1. size of singe user id: 10 bytes
2. messages posting throughput: 50k per sec
3. avg channel size(assume all of them to be unique): 2k

Calc:
total size just of uids: (2k * 50k * 10) bytes = (100M * 10) bytes = 1B bytes ≈ 1000MB ≈ 1GB.

Now it moved to system like discord where we have sale events where messages are posted in channel where a small number of people are present + a really big channel (like announcements / gen).

• 10 messages per channel + announcements channel(1M users) = 10M notifs

• 50k messages w/ 2k users per channel
• 1:1 DMs (small number)

≈11 M notifs (at most)

How I would modify the existing system for this or would the current system be capable of serving this much load. I couldn't grab the question fully at once and I clarified it.

Interviewer made some comments that translated to queue becoming a bottleneck.

I intended to solve it via partitioning the queue, sending messages to the dedicated partitions with load being distributed evenly in the queue. As for the partition key, I sad the userId and I was asked the number of partitions needed. I took reference from my previous calc and said that :

1 partition complies w/ SLA for 50k notifs per sec
I just used unitary method to eval the same for 11M notifs per sec: that came around 1k.
I wrote down: partitionKey = (userId % 1000).

I mentioned it. This number was kinda big for me hence I wanted to shard it further on the basis of some type using which I could avoid hot partition problem, and hence started checking it's feasibility via performing a BOTEC again via assuming a message's average length, but interviewer stopped me and said that this is workable, but you should keep in mind that if you are using kafka then there is no going back after increasing partitions, to which I agreed.