I was asked to design a system where an IOT sensor is constantly sending data to the server (1 kB per second) and server's backend has to forward this data set to a machine learning service to make sense out of that data. The machine learning service needs data of exactly 2 seconds in order to determine the best results.

All the historic data has to be retained/persisted for future use case.

For example:

1. The sensor sends data set 1 but server cannot send it to the ML service as it does not have enough data yet.
2. The sensor sends data set 2 so now server can send dataset 1+2 to the ML service. Now, dataset 1 is not going to be used so it can be send to some archival storage.
3. The sensor The sensor sends data set 3 so now server can send dataset 2+3 to the ML service. Now, dataset 2 is not going to be used so it can be send to some archival storage.

I proposed using Redis as a message queue to store the mapping of data set and the sensor ID. I tried to explain him that there will be an ingestion service that will store data set into an in-memory database and that can be used a message queue. After inserting the data into

Proposed Architecture:
steps from ingestion service:

1. Insert/update data to redis
2. Invoke consumer service API
3. Consumer service reads data from redis based on the sensor_id passed during the API.

																			
sensors [1,2.....n] ==> server : <ingestion_service> -(1)--->  Redis
                                                   |         /
												  (2)    (3)
												   |     /
												   |    /
											<consumer _service>

sensors [1,2.....n] ==> server : <ingestion_service> --> Insert the recieved records into redis cache. {sensor_id: [[data_set_1], [data_set_2]]} . Then use the sensor_id and invoke a consumer service API that will retrieve data set 1 and 2 from the redis cache and will send it to the ML service. It can remove data_set1 or whatever the first member of the corresponding list is and send to the archival storage.

Needless to say, the interviewer didn't seem happy and ended the interview 20 minutes early. Any tips, help would be highly appreciated.

This was from a no-name startup.