Was asked this during an Interview for a Cloud Engineer position at IBM.

I was asked to design a system which takes a large amount of data, cleans it, adds specified properties to it and sends the results so that the other services can filter data with the newly added properties. Details follow.
The input are Hundreds of files per hour, where each line of the file is object formated in JSONL. Each file can have between 1 and 10000 objects. Each object has the same x ammount of properties. The most crutial property is the video_url property, which contains a URL to a video stored in some server. The purpose of the system is to take that url and tag's it appropriately and store's the results.

• The service which tags the videos is already implemented and has a API which you can use.
• The stakeholders would like to see some statistics about the processed results. E.g. how many videos have the same tag, etc.

Questions

• How would you implement the system? Tech stack, etc..
• Which errors could occur?
• What if the URL to a video is not valid? What if 100 URL are not valid?

My answer
Since this is a cloud position I thought about doing systems design with the help of piplines. So I thought about a service that parses the input, takes out the important parts and send it to kafka. Once the data is in kafka there would be another service that reads from the topic and processes each message. It takes the URL and does and requests the tag of the video by doing a API request. The results are send then to another kafka topic ready to be consumed by other services that need the information. For example, there could be a service listening to this last topic and sotring the results in a no SQL DB.

What do you think? I did not consider the error part because of the time limitations. However I think it was an important part since we would like to avoid sending errors or invalid URL's to kafka because the everyone can consume it and that mean that every service would have then invalid data.