"Contest in Baramati is within the family," Ajit Pawar told reporters here on Thursday when asked why the PM would not be holding a rally in his constituency.