Police removed six people who had taken over a vacant flat over concerns about drug use.