Let's say there is a client which needs to monitor how many active users are there on his website at any point of time. He wants to monitor this and for that he uses our monitoring service. So, how would we design such a service?

Assume there is an agent on every client node/machine which submits data to your analytics service.

Functional Requirements

• The service would receive different types of events from clients and should be able to process each differently, and finally show it in separate graphs on the UI.
• The service should store data for the last 6 months, at least.

Non-Functional Requirements

• The service should be highly available.
• There are 10000 nodes with 500M DAU. This will give you an idea of the volume of event data.

Note:

• Do not concentrate on the UI side. Assume you have the UI (web client) where the user can select a particular metric and the type of graph to show the metric.
• The example of event data described here is just one example. There can various other kinds of data and can be considered out of scope of this question.
• A detailed design describing the different libraries or frameworks to use like kafka, cassandra, hadoop, etc. would be great.