Prime Minister Anthony Albanese revealed an OpenAI agent scaled a digital "fence" to breach a Medicare platform and access non-public information at a press conference in New York.
Staff had only directed it to research public medicine funding on the internet.
The timing of this revelation was "pretty tactical" according to Olivia Shen, a Director at the United States Studies Centre at the University of Sydney.
"The prime minister's over there at the UN talking about the need for global good governance on AI, and we just happen to have had this incident occur that they were notified about last week," Ms Shen told AAP.
The disclosure of the incident came as Mr Albanese used his appearance at the UN to urge foreign leaders to follow Australia's lead in regulating AI safeguards.
No matter the timing, Ms Shen said it was further proof of the need to act and enforce regulation.
"This isn't theoretical anymore ... it's urgent, and we can't keep dismissing these risks as hypothetical or the stuff of science fiction. They're literally happening now."
The breach occurred on June 18, however, OpenAI did not notify the government about the hack until September 10, after discovering it in August.
Reporting and reaction times should be much quicker than the three month delay in this case Ms Shen said, backing requirements similar to cyber security incidents which impact critical infrastructure.
She said the race to develop and roll out AI to assert global dominance has also been blown out of proportion.Â
"There's this vision of AI as this technology that is going to be fundamentally transformative, and it probably will be but the idea that it translates directly into geopolitical power and supremacy, I think needs a lot more testing."
RMIT Innovative Interactive Technologies expert Dana McKay expressed her disappointment at OpenAI's attempt to remedy the situation by sending an email to a public Services Australia address.
"I'd be expecting them to say, 'right, this is how (the data) might have been incorporated into our models. This is what we're going to do to cleanse it from our models. This is what we're going to do to make sure it doesn't happen again,'" she told AAP.
"'Oh, and by the way, we are extremely sorry.'"
Associate Professor McKay said she was optimistic laws could be put in place to combat unsupervised agents making unsafe and illegal decisions.
"All this is happening in a climate where there is a willingness to legislate ...we might see some changes happen before it is too late."
But putting all eyes on artificial intelligence means we risk missing out on other technology breakthroughs.
"Other technologies we should potentially be looking at that are geopolitically significant. Quantum, space, synthetic biology, and biotech would be my top picks," Ms Shen said.
