Functional Requirements (Given by panel) :

1. The firm has a black box, where one can feed in some demographic details (mobile, age, salary, credit score, etc) and the black box would return whether user is approved for credit card or not.
2. Design an application, where you use this black box to ensure users are approved or rejected for credit card or loan.

Non Functional Requirements (I came up with this):

1. Fault Tolerant
2. Scalable - Even if million users use our app, it should work as BAU
3. Low Latency

My Approach:
As you can see, question is very very abstract - the point of this question is to see if candidates can think of multiple solutions/requirements/issues and expand on the original requirement.
My thinking was:

• The input can be from anywhere - if lets say we(company) ran a marketing ad, then end users will be regular people/customers, for whom client can be iOS/Android devices or a web app UI page.
• If there is a 3rd party who gather data from users and sell/redirect data to our company, in this case users will be an intermediary and often such 3rd parties will have bulk user list, so another channel of communication for bulk can be FTP/SFTP.
• Now coming to our core logic, we will have below APIs:
  • POST /create/user - create a new user. Specifically using POST API here because in banking, client data is confidential and in event of hacking, if we use GET API, data can be extracted by malicious people. Hence passing payload using Req Body via POST API
  • GET /users/all - Get list of all active credit card users
  • POST /create/card - API to create new card type. Card types can be based on customer valuabilitiy and salary/loyalty.
  • POST /check-user-approvability - This will send a req (via REST/GRAPHQL) to our black box (black box will take user demography input and return if user is approved/rejected for credit card and if approved, what is the card type and credit limit, and if rejected, a rejection reason). Again using POST here for client data security, even though use case resonates more with GET API.
  • POST /user-application-audit - An endpoint which will log all applications, along with status (approved/rejected) - This will serve as audit. We will also have a similar GET API to fetch audit history.
• Now, coming to DBMS, I will use RDBMS here, just because of clear relationships between different entities. One is free to use NoSQL here as well.
  One can see schema below. I did not complete schema, because focus was more on API and non functional requirements. We can easily add more demographic fields to the tables.
  

Architecture Diagram:

Important Considerations/Trade Offs vis-a-vis architecture:

• Entry point will be a LB/API Rate Limiter. This will redirect requests to our API service, which is responsible for communicating to and fro Back Box. Lets call our core service as Recommendation Service.
• This service is of course made redundant with multiple nodes. LB redirects requests to all nodes using either a weighted round robin or consistent hashing mechanism.
• Service Nodes need to be checked if they are alive or not - this can be done by sending hearbeats after every 5 seconds. And if 2 or more consecutive heartbeats fail, we can declare that node dead and spin up a new node. One can also use Apache Zookeeper, as a centralised store to keep track of nodes, heartbeat, failover, etc.
• Similar logic with DB - multiple nodes will be there, they will be made fault tolerant and redundant.
• Since there are multiple DB nodes, to ensure consistency/no race condition, we will use a Single Leader Replication mechanism so data in DB is replicated asynchronously. Again, their config can be maintained in Zookeeper along with heartbeats.
• There is a risk with Single leader replication as well - if the leader goes down, there could be short term chaos, and leader election will take some time to elect a new leader. So one can use Multi Leader replication or Quorum/handoff based replication as well. To keep things simple, I will stick with Single Leader Repl.
• 80% of recommendations of creedit card have similar data - i.e. same salary band of users, similar age/credit rating band etc. Therefore we can leverage use of an in memory cache (memcached/redis) to save these trends/rules. The app will first hit cache and if its a cache miss, then hit the actual black box.
• Caching eviction mechanism can be LRU.
• Also, users will be geographically distributed, so one can also use Edge Locations/CDNs in caching.
• In case of multiple load/requests, we can use auto scaling to perform horizontal scaling. Even then if its still very slow, we can park all requests in an MQ and let them be processed at their own speed. We can display a message to user saying your request is being processed and let them know once its done. This way our system is not overwhelmed. This is done because for us Availability is of prime importance, even over consistency (CAP Theorem). No matter what, app should be accessible and should be servicing our requests, even if its slow due to high load.

Nice to have features:

1. Audit service will be fed into a ML service. This ML service can be used for demographic research, fraud detection, etc.
2. If we have known fraudsters, we can have a blacklist between LB/Rate Limiter and our service to filter out such fraudsters/spamsters. This will also avoid DDOS.
3. Also, apply rate limiters on all APIs to avoid DDOS attacks. Even if genuine users without DDOS intention rapidly call our APIs, it can get overwhelmed, hence better to apply rate limiters.
4. We will also have an MQ like Kafka, this will be connected to all components - this will be like a Dead Letter Queue - i.e. all errors will be passed via this MQ and can be connected to Splunk engine, for easier debugging.
5. Since our data is confidential (customer data), so we can also encrypt data, both in REST and in transit, to ensure data sanity. We can use something like MD5 or SHA1.

This is what I could come up with in 40 mins. There was lots of questioning by interviewers and they seemed happy. How do you think about this?
How can you design it or improve my design? Any constructive feedback?