The brothers made the switch from their family home in Ohio to Los Angeles in 2014, but their father admitted he had some concerns ahead of the move