Removal of anti-immigration posters must not be left to the police to deal with alone, the Justice Minister has said.