The Conservative party has been urged to decline or return a reported further £5m donation made by Frank Hester, whose remarks about Diane Abbott have been widely condemned as racist and misogynistic.

It comes as Rishi Sunak faces increasing pressure over the £10m previously given by the millionaire businessman, with Douglas Ross, the leader of the Scottish Conservatives, saying on Thursday that the donations should be looked at by the party.

Conservative party spokespeople have not responded to a report by Tortoise that Hester, who told a 2019 meeting that seeing Abbott on TV made “you just want to hate all black women”, had given more money.

Hester also said that Abbott, the UK’s longest-serving black MP, who sits as an independent after losing the Labour whip last year, “should be shot”.

Tortoise cited an unnamed Conservative source as saying Hester had given another £5m, taking his total donations to £15m, and that the party was “sitting on” the new money before it was confirmed in the next Electoral Commission update of donations, due in early June.

Anneliese Dodds, the Labour party chair, said: “Frank Hester’s remarks were clearly racist, misogynistic and have no place in our politics. There is absolutely no excuse for the Conservatives accepting additional money from Frank Hester. They should pay this back before it hits the coffers.

“Rishi Sunak needs to pay back every penny, cut ties with Frank Hester and apologise unequivocally to Diane Abbott.”

Wendy Chamberlain, the Liberal Democrat chief whip, said: “The Conservative party must urgently confirm whether these reports are true and if so hand this tainted money back.

“The Conservative party has dragged its feet in condemning these racist remarks for what they are. If that was in any way linked to this £5m donation it would show this scandal is even worse than we thought.”

Ross, speaking to journalists after first minister’s questions, echoed a call from the Scottish Conservatives for the UK party to “carefully review” the donations it received from Hester.

He said the comments were “racist, unacceptable and wrong”, adding: “I think the party at a UK level needs to look at that donation.”

Pressed on whether the money should be returned, he said: “I think this should be looked at. I think most people would think it’s fair to allow a review to review the situation and look into it.”

Others in the party who have called for the money to be returned include Andy Street, the West Midlands mayor; Sayeeda Warsi, a Tory peer and former party chair; and Chris Patten, also a peer and a cabinet minister under John Major and Margaret Thatcher.

In a statement on Monday, Hester’s company said he “accepts that he was rude about Diane Abbott in a private meeting several years ago but his criticism had nothing to do with her gender nor colour of skin”.

The statement said Hester “abhors racism” and that “he rang Diane Abbott twice today to try to apologise directly for the hurt he has caused her, and is deeply sorry for his remarks. He wishes to make it clear that he regards racism as a poison which has no place in public life.”

Hester has donated £10m as an individial and via his company, the Phoenix Partnership, a software firm he formed and solely owns, and which has won more than £400m of NHS and prison contracts in the last eight years. It is so far unclear if the extra £5m may have come from him individually or via his company.

Hester has been contacted for moment.