The Social Democrats have narrowly won an election in the eastern German state of Brandenburg, just edging out the far-right Alternative for Germany party.