Mr Herzog concluded his final closely guarded four-day visit in Melbourne on Thursday, where he met senior politicians and community leaders.
Despite the president declaring there was "hope in the air" in his final speech, about 10,000 people gathered on the steps of Flinders Street Station in defiance, saying he was not welcome.
The demonstrators remained largely peaceful, despite a small scuffle breaking out as Palestinian organiser Tasnim Sammak addressed the crowd.
Victoria Police said special powers granted for the event were not used.
Among the speakers was Palestinian activist Muayad Ali, who criticised the prime minister's decision to host the president in Australia.
"I never swear, but this is not diplomacy," he said.
Activist Bella described the strong turnout as heartwarming, while condemning leaders for rolling out the red carpet for the president.
"As people with conscience we have to say, war criminals are not welcome here and so we're out in our thousands protesting," she told AAP.
The protest was a far cry from the scenes witnessed in Sydney, where at least nine people were charged and 27 arrested after protesters were punched and pushed by police when an anti-Herzog rally turned violent.
Not far away from the rally, and inside a heavily secured building in the inner-city suburb of Southbank, surrounded by a large police presence and neighbouring blocks locked down, the president addressed more than 500 Jewish and Victorian leaders and senior school students.
Mr Herzog said his visit to Melbourne had ignited a renewed sense of hope.
"I must tell you that we return to Israel feeling empowered because we have seen first hand the beauty and resilience of this community," he said.
Mr Herzog said discussions with the Australian leadership to address "vile rhetoric, misinformation and shameful anti-Semitism" were conducted with candour, open-mindedness and a great deal of mutual respect.
My Herzog said the large security detail outside the secret venue on Thursday to protect Jews from harassment and disturbance was "obscure and odd".
"I say to all those protesters outside, go protest in front of the Iranian embassy or whichever embassy they have," Mr Herzog said to a cheering crowd, saying the "empire of evil" had killed tens of thousands of their own people.
Mr Herzog had earlier met with Victorian Governor Margaret Gardner and Premier Jacinta Allan at Government House.
Outside, a large contingent of police and security formed as about two dozen protesters turned out to protest his arrival.
Donning black hoods and masks, police costumes and keffiyeh scarves, the group heckled as the Israeli president arrived in a large motorcade.
Police ordered the group back before the protesters unfurled banners which read, "stop committing genocide in Gaza" and "all colonies will fall" and chanted "from the river to the sea".
Melbourne University's Parkville campus was sprayed with graffiti saying "Death to Herzog" on Thursday morning.
Ms Allan said the graffiti was "absolutely contemptible" and the reason for the president's visit was to comfort the Jewish community after the Bondi terrorist attack.
A university spokesperson said the graffiti was removed soon after staff became aware of it and the university stood firmly against anti-Semitism, racism, hatred and violence.
Mr Herzog was invited to Australia by the federal government following the Bondi attack but has received a hostile reception from protesters and some politicians over Israel's offensive in the Palestinian enclave of Gaza.
Mr Herzog met survivors of the Bondi attack in Sydney, before being hosted by Prime Minister Anthony Albanese in Canberra.
The Israeli president has described his four-day trip to Australia as very emotional, saying it was important to visit the community and express condolences.
14°C