Mr Frydenberg made the comment at a Sydney Writers' Festival event on Tuesday evening - the day of the release of his memoir A Grateful Son.
Speaking to veteran ABC journalist Leigh Sales in front of a crowd of about 300 people, which included his parents Erika and Harry, the former member for Kooyong said he left Canberra "remaining an optimist".
"I did leave ... remaining a believer in the integrity of our system, and most importantly, the ability of politics and a political career to make a difference," he said.
This was despite the events that lead to his losing the seat to independent MP Monique Ryan at the 2022 election, including a number of scandals which plagued the then-prime minister Scott Morrison.
Mr Frydenberg described Mr Morrison as "extremely ambitious" and saying his conduct at the time he took over the leadership of the party as "transactional".
He referenced an anecdote in the book where Mr Morrison asked him to run on a joint leadership ticket, an offer Mr Frydenberg declined in favour of running independently.
He said Mr Morrison then publicly presented the decision to run separately as a mutual one.
"It just gives people ... I hope, an interesting insight into some of the dynamics at play, particularly around leadership," Mr Frydenberg said.
"Which happens in both parties, but can tear parties apart."
The release of the book has fuelled speculation he could return to politics, which Mr Frydenberg has said he remained open to.
He held the seat of Kooyong for 12 years from 2010 and was treasurer in the Morrison government, including through the COVID-19 pandemic.
On Tuesday night, Mr Frydenberg also defended the $88.9 billion JobKeeper program he oversaw during this time.
He said those who had criticised the eye-watering spending were "in the cheap seats".
"Those people were not there with the elected responsibility to save the economy and to keep people in work," he said.
Mr Frydenberg has been vocal in his criticism of the opposition's current leader Angus Taylor, saying Mr Taylor's handling of the Liberal Party had contributed to its polling being at an "all-time low".
But the opposition leader said he would welcome the political return of Mr Frydenberg, despite the attack on his leadership.
"I welcome every Liberal who cares about the future of our nation - and I know Josh does - to put shoulder to the wheel in the coming months to get rid of a rotten Labor government and make the change that Australia needs happen," he told reporters in Canberra on Monday.
7°C