Greetings fellow JobSeekers, today I am sharing my recent interview experience with Tekion for Senior Data Engineering Role. I read on reddit that they are hosting a hiring drive in next 1-2 days, hope it helps someone.

Background:

• College: Tier 1
• Total Exp: 6
• Last Firm: Fintech(Data Platform team, ~4years)
• Last Designation: Senior Software Engineer
• Stack: Backend + Data Engineer(Python,java,postgres,mongoDB,aws)
• Currently Unemployed

Process:

• I applied for SSE role on multiple platform like - Instahyre, Naukri, Weekday etc.. not sure from where I was shortlisted
• Got a recruiter's call after ~ 2-3 weeks of continuous applying, Round 1 was scheduled(virtual).
• Panel was also SSE(7+), started with basic intro and convo about previous project. My past experience was very similar to what they are currently building so we had a lot to talk.
• Round 1 Question: Given 2 HUGE tables(100M+ records) with continous data coming to them. We need to device a way to efficiently MERGE these tables and store it in NoSQL DB.
  • I started with a one time SNAPSHOT load at a fixed watermark(useful for later use)
  • Then a CDC pipeline to sync the incremental changes from last synced watermark
  • Given the scale, we can't expect to fire a single query to move 100M+ records at once
  • Then we discussed how and where to join these records. Started with merging them at CDC itself - but that would fail as we might not always get all table changes at same time(DUAL CDC + QUEUE)
  • Solution: Convert the CDC into events(Kafka or other event streaming arch.), instead of whole records, only PK needs to there in event
  • Have multiple brokers in kafka cluster, for inter-related tables we need a way for quick lookup(INDEX maybe, but not always true for HUGE loads)
  • Workers then consume these events, get the related tables PK also then DO UPSERT in merged table with lookup on source table

Note: We also need to take care of reconsilation and refill etc..

• Cleared this round with positive response

• Next round rescheduled multiple times due to panel availablity issue, got disturbed by other employee's more than once during interview, panel went away.

• Finally second round after ~ 1+week, basic intro, project discussion

• Round 2 Question: Quite similar to First but more emphasis on resilient reconsilation and backfill. Multiple HUGE tables, being updated at different interval, design an ETL pipeline to effiecienly load them to destination.

  • Shared watermark approach, event driven arch. to avoid long polling on source tables(when they get updated)
  • Miss on my side - miscommunication with panel, hence missed the emphasis on watermark detection logic which the panel was expecting(I did mention it at the start, but explored EDA in much depth)

Received rejection mail after 7+ days. Yes, their feedback loop is quite slow, which I also read a lot on other forums also + culture, But I can't be choosy for now.

This was my first Data Engineering interview ever.

My 2 cent Tips: ALWAYS, always get a clear answer from panel first about what they are expecting and not fall for their silence assuming you are going in correct direction.