Donald Trump’s Madison Square Garden rally was meant to be a triumphant return to the city that made his name. Instead, the event has reheated simmering tensions on the island territory of Puerto Rico, where locals say they feel like second-class citizens, days out from a historic US election.

Comments from a comedian at Trump’s rally this week, describing Puerto Rico as a “floating island of garbage”, might have fired up local resentment against the Republican candidate, but those who live there will be unable to show their anger through the ballot box.

Residents of Puerto Rico are among the more than 3 million Americans whose votes won’t count, when the country elects its president next week.

Most residents of the US territories – Puerto Rico, Guam, American Samoa, the US Virgin Islands and the Northern Marianas – are citizens and pay federal taxes. Many have family on the mainland.

But the territories have no votes in the electoral college that ultimately decides the president.

“They’re just basically captive to whatever Congress and the president want to do to them,” says Luis Fuentes-Rohwer, a law professor at Indiana University. “To be a citizen means to be a member of the political community. Not in the US.”

It’s one of the few things uniting the disparate territories, which range from Puerto Rico, home to 3 million citizens in the Caribbean, to American Samoa, closer to Australia than America with a population of just 50,000.

Residents have no say over Washington’s decisions that control their everyday lives, according to Karina Claudio Betancourt, a student in Puerto Rico’s capital city, San Juan.

“Elections impact us because the president decides what to do or not to do with Puerto Rico right now,” they say.

Betancourt points to Donald Trump’s response to Hurricane Maria, which devastated the island, and the Obama administration’s imposition of a board controlling government funding.

“Basic transportation and health services and things that impact us day-to-day are decided by the Congress of the United States and by this fiscal oversight board, but we don’t get to vote for any of that,” they say. “People are fed up.”

Guam resident Phillip Gilbert says it’s unfair to leave territory voters disfranchised.

“I do feel robbed not being able to vote,” he says. “As a service member, I especially feel that this is not fair after serving my country.”

The 47-year-old could register as a Washington State resident and vote absentee while stationed in Guam with the navy, but Gilbert lost the right to vote for president once he had retired in the territory.

Charles Ala’ilima is on the board of Right to Democracy, one of a growing number of organisations working to get territory residents – 98% of whom are people of colour – more influence.

“There is this othering going on of people of colour, people of the territories, right?” he says. “Unless [you] actually have citizenship and the right to vote, you are always going to be considered other, not a full part of the United States.”

While his home of American Samoa may be too small to become a state on its own, representation could just start with Congress asking the population whether they want to be a permanent part of the US, Ala’ilima says.

“If they put that in front of the people, I expect 99% would say we’d be a permanent part of the United States … [and] the United States then has to make adjustments in their laws to allow the territory to become a full fledged part of the political process.”

But territories could also be given a say in the presidency by amending the constitution to grant them electoral college votes, as happened for Washington DC.

Many Puerto Ricans who have moved to the mainland will be able to make their voices heard on 5 November, with large communities in Pennsylvania and Florida being targted by both campaigns.

Those who remain in the territory meanwhile, have made numerous efforts to become a state in their own right. The island has more people than the five smallest states combined and statehood is supported by many, but the final decision lies with Congress – and recent efforts have stalled.

Those delays and persistent disfranchisement means some Puerto Ricans are eager to leave the US orbit altogether.

On Tuesday, alongside elections for local politicians and an informal straw poll for their favoured presidential candidate, they will vote in their seventh non-binding referendum on status – this time putting full and partial independence up against statehood.

“It’s not fair that that all of these decisions are being made as if we were savages, as if we were people who can’t have self determination,” Betancourt says. “I don’t want to vote for the United States – I want to vote for the president of Puerto Rico.”

Additional reporting by Mar-Vic Cagurangan in Guam