The camera-equipped technology products are being quickly adopted across Australia.
Cheap versions sold by retailers such as Kmart for $90 have increased worries about privacy and people being filmed covertly without their consent.
The attorney-general has ruled out an import ban for the devices.
"We're not having a regulatory environment that's playing whack-a-mole every time a device or a technology is developed that could be used in harmful ways," Michelle Rowland told the ABC.
She acknowledged the importance of the smart glasses as a tool for people with a disability.
"We're neglecting the very legitimate reasons for which some people in our society are using these devices," she told the ABC.
"They include people who are vision impaired, for whom this has been a real game changer."
The products have made it much easier for people with sight impairments to participate in daily life, Vision Australia's chief mission officer Chris Edwards told AAP.
Mr Edwards, who is totally blind, described a previously laborious process of unlocking his phone, typing in codes and opening the correct app with the help of accessibility features to do something as simple as reading a menu.
Now, he uses a couple of hands-free steps on his glasses.
He says accessibility for people like him outweighs any potential risks when considering a blanket ban.
"There have been people that can use phones irresponsibly and do the wrong thing," he said.
A short ban to give the government and disability community time to talk properly and consider options should be considered, University of Technology Sydney policy specialist Sarah Sacher said.
"Right now, we know these glasses aren't safe. There could be a temporary measure to keep them off the market until we've got the right protections in place," she said.
Ms Sacher was particularly concerned about cheap dupes of smart glasses, which could open a "Pandora's box" of misuse.
"There are different kinds of data usage and safeguards - or even no safeguards - in various dupes that make it harder and harder to keep a handle on (how they are being used)," she said.
Smart glasses are not mentioned at length in the draft legislation, which targets AI and increased control of personal identity information, but supporting documents suggest crackdowns on misuse should occur at a local level.
"We need to make sure that we strike that right balance whilst keeping our privacy laws technology neutral and also principles based to account for other inventions that we will see in the future," Ms Rowland said.
