Addressing a public rally at Deoghar, Prime Minister Narendra Modi said that the biggest worry in the state was that of the Bangladeshi infiltrators and the need to protect the tribal families was at the top.