I am trying to find best resources online to learn how Zookeeper does failure detection in a cluster of nodes, leader election if master goes down, heart beating with all the nodes in a cluster and service discovery.
I recently had an interview where the discussion was about how nodes in a leaderless cluster could be constantly monitored so that when any of them goes down, their jobs are assigned to other nodes in the cluster. I said we could use Zookeeper to do heartbeat with the nodes but the interviewer wanted to know more about how that can be done and what that process looks like.
I had no answers for that.
So to save my face if such thing happens again, can some one please suggest good resources (which have examples on failure detection) on such coordination systems ?
Thanks