With more than one million votes counted, 77 per cent supported remaining in Canada, while 23 per cent backed beginning a process towards separation, according to unofficial results from Elections Alberta.
The referendum was not a direct vote on independence. Approval would have set Alberta on a path towards a future vote on separation from Canada.
"Albertans made a clear choice," Carney, who was raised in the province, said early on Tuesday.
"They chose unity, to build together, and to take care of each other. Above all, Albertans chose Canada."
Carney said the result would allow Canadians to move forward together and accelerate efforts to diversify trade, develop major infrastructure projects and make Canada a global energy powerhouse, with Alberta at its centre.
Alberta's separatists, however, saw a silver lining.
"We don't have the final results. I think the numbers are going to fluctuate, but it is fair to say that at least a quarter of Albertans feel that Alberta needs to be treated better," said Keith Wilson, a leader of the separatist movement.
Alberta Premier Danielle Smith welcomed the early results, saying they showed widespread support for remaining in Canada.
"Albertans have voted for a stronger Alberta in a united Canada," Smith posted on social media.
Smith, who called the referendum amid pressure from separatists within her governing United Conservative Party, or UCP, despite opposing independence, said she would address the province on Tuesday evening about the results and Alberta's future.
"We are a solid country that's united from coast to coast to coast," Thomas Lukaszuk, a former Alberta deputy premier who led the Forever Canadian campaign against separation, told supporters.
Results were delayed after late openings forced more than 100 polling stations to extend voting hours on Monday.
Alberta, home to the world's fourth-largest proven oil reserves, is cowboy country and a conservative stronghold whose political culture has long had more in common with the American West than much of Canada.
Resentment towards Canada's federal government in Ottawa runs deep, but has not traditionally translated into widespread support for independence.
Despite the proposal's apparent failure, separatist leaders have said they turned a fringe cause into a movement that will not disappear.
Separatist parties traditionally attracted only one per cent or two per cent of the vote, former premier Jason Kenney said.
Anger over COVID-19 restrictions helped mobilise activists who later embraced the separatist movement, he said.
Trump's trade war and threats to make Canada the 51st US state have helped Carney rally support in Alberta, where Liberal prime ministers have historically been unpopular.
