Soaring church spires, the 1,000-year-old town centre unblemished by second world war bombing or graffiti, snow-capped Alps in the middle distance – Kaufbeuren, in Bavaria, can count many blessings.

Unemployment is in the low single digits, the Luftwaffe backed away from plans to move its training school for Eurofighter and Tornado jet technicians elsewhere and crime is at a historic low.

However, as voters prepare to elect a new European parliament next month, deep-seated fears have gripped a significant share of the electorate in one of the most affluent pockets of Europe’s top economy and delivered it to the far-right Alternative für Deutschland (AfD).

The bond between the party and its voters appears unshaken even by a cascade of recent scandals. The AfD’s lead candidate for the election, Maximilian Krah, was forced by his party leadership on Wednesday to resign from its board and stop campaigning after he told Italy’s La Repubblica that the SS, the Nazi paramilitary force which ran the death camps, were not all criminals and could only be judged on the basis of “individual guilt”.

Previous rhetoric from Krah, an admirer of Donald TrumpThe SS remark proved to be the bridge too far, leading ledFrance’s far-right National Rally (RN) to announce it would sever ties with the AfD in the EU assembly after the June election. The populist Identity and Democracy political group in the European parliament then moved to expel the AfD delegation “with immediate effect”.

Krah had already been in the spotlight for suspected Chinese and Russian ties after one of his aides was arrested on charges of spying for China. On the eve of his resignation, he told the Observer the allegations were “merely an attempt to distract from our political arguments” and threatened legal action against his accusers.

Polls show the AfD going into the EU election with 14-18% of the vote, well off its 23% high in October but up from its 11% score in 2019. Despite the rap on the knuckles, Krah is virtually guaranteed a seat in the next European parliament.

As he arrived for the Kaufbeuren rally last week wearing a blue suit and his trademark pocket handkerchief, Krah posed for selfies with dozens of fans including young men with short haircuts wearing lederhosen.

Surprisingly for many, the AfD continues to make inroads in Germany’s prosperous south and west, beyond its heartland in the poorer ex-communist east, as it embraces more extreme views on immigration, the war in Ukraine and national atonement for the Holocaust. But the right’s ascendancy has also given rise to a lively local pro-democracy movement that seeks to draw lessons from the town’s grim Nazi past.

In Kaufbeuren, nearly one in five gave the AfD their vote at the last state election in October. Pelted by cold rain, about 200 people gathered in a quaint old town square for the AfD rally waving large German flags. A similar number of peaceful counter-protesters rallied 300m away, led by the Omas gegen Rechts (Grannies Against the Right) movement and joined by mayor Stefan Bosse and a brass band.

Krah’s supporters said they were disgusted by chancellor Olaf Scholz’s centre-left-led coalition and anxious about their children’s future.

“It’s a catastrophe – the worst government we ever had,” said civil servant Manuela, 55, who was from a neighbouring town and, like most of the AfD supporters, declined to give her surname. She brought her teenage daughter to the rally. Despite the low rates of violent crime, she said her family no longer felt safe on the streets due to “Islamists”.

Anti-AfD activists booed and whistled from the sidelines as Krah addressed the rally. Manuela said: “They call us Nazis just because we’re patriots. The world laughs at us because no country is as dumb as Germany, with our exaggerated tolerance and diversity. They’ve been telling us for decades we should carry this guilt, and so we should rescue the whole world and be its dole office.”

A bombshell report in January revealed that senior AfD members had attended a meeting at a lakeside villa where they discussed a scheme for the mass deportation of German citizens with immigrant backgrounds. The revelation sparked anti-extremism demonstrations nationwide, including in Kaufbeuren.

Doreen, 53, who works in hospital catering, said the “remigration” plan appealed to her. “I have friends and acquaintances who migrated here and they agree: those who want to integrate should be able to stay, and those who just want to exploit the social welfare system should be told to go home.”

Elke, a 54-year-old nurse, said she liked the AfD’s opposition to what she called overreach in Berlin and Brussels. “I want to drive a car with an internal combustion engine, I want gas heating and I don’t want a war against Russia,” she said. “It would have been over long ago if all that money hadn’t flowed to Ukraine.”

Several AfD voters said they opposed Germany’s military support for Ukraine and worried about a Europe-wide conflagration triggered by standing up to Vladimir Putin.

As Krah took the stage, local party officials released eight white doves of peace into the overcast sky to applause from the crowd.

A father of eight, Krah spoke of his fear that his 21-year-old son could become “cannon fodder on the eastern front” if Germany brings back conscription, a proposal floated in limited form by defence minister Boris Pistorius to address looming security threats.

Krah warned that Kaufbeuren’s surrounding region of Swabia, “one of the richest parts of Germany”, was under threat from politicians who would take away its security and prosperity with climate protection measures and “mass migration”. He presented the AfD as the defender of “traditional families” while its opponents believed there were “53 genders”.

Bosse, the mayor, told the Observer that a large part of the AfD’s base in Kaufbeuren comprises ethnic Germans from the former Soviet Union who arrived in the 1990s, many of whom harbour pro-Putin views. In the pandemic, the region also had a strong anti-vaxxer movement, which now opposes Nato aid for Ukraine.

Bosse has led the town for 20 years. In his wood-panelled office overlooking the town’s immaculate high street, he said that he was “ashamed” that the far right had gained a foothold again, despite efforts across the political spectrum to blunt its appeal.

“I also feel shame toward the British and French and all those who suffered so much under the Third Reich that in Germany you have a political force establishing itself that is trying to just wipe away these horrible crimes,” he said.

Bosse, from the conservative Christian Social Union, said he is haunted by a particular chapter of the Nazi past in Kaufbeuren, which under Adolf Hitler hosted a dynamite factory employing forced labourers, satellite concentration camps belonging to Dachau and a psychiatric hospital that orchestrated the extermination of more than 1,500 men, women and children under the .

Even after the town’s “liberation” by the US on 27 April 1945, the hospital continued to operate on the town’s outskirts under its Nazi administrators, Bosse said. Another 100 helpless patients were murdered before GIs finally took control of the facility on 2 July.

“It is incredible to me that, even after the second world war was over, no one in Kaufbeuren told the Americans what was happening and said: ‘You have to go in there and set them free,’” he said.

Speaking at the counter-demonstration, Bosse noted that 2 July is now marked by the town each year with a memorial ceremony for the victims as a reminder of the need for civil courage. “We have got to stand up for this democracy together,” he said.