Vote caps fraught debate over growing influence of Alternative for Germany party in Europe’s largest country