Design a distributed web crawler to crawl 1 billion url\'s as a hacker avoiding detection\nRestrictions/questions/comments\n- For crawling a hacker has about 10,000 personal computers available all over the world\n- You can deploy a software to each computer to run the system\n- For each url, download and save html content\n- Desing should be such that traffic between particular nodes should be minimal to avoid detection\n- It should start with a root url and eventually crawl all 1 billion url\'s\n- Load should be distributed evenly across all computers\n- How would you prevent crawling of duplicate url\'s?\n- How you would store data?\n- How would you distribute url\'s across all remote computers\n- There should not be a central authority which all remote computers connect to as that will go against the rule of avoiding detection

Design a distributed web crawler to crawl 1 billion url\'s as a hacker avoiding detection\nRestrictions/questions/comments\n- For crawling a hacker has about 10