The government on Thursday confirmed Mandelson - who was subsequently fired after Starmer said he had lied about the strength of his relationship with sex offender Jeffrey Epstein - had failed his security vetting before taking up the role.
The government said Starmer was unaware officials in the Foreign Office had overruled the vetting recommendation and a source said the most senior official at the foreign ministry, Olly Robbins, would leave his role after losing Starmer's confidence.
"I don't think the prime minister can get out of his responsibility by sacking Olly Robbins. I think the buck has to stop with Mr Starmer," Liberal Democrat leader Ed Davey told BBC Radio on Friday.
"I think the evidence suggests that he misled the Commons (parliament) and misled the public. That's against all the rules, and that's why we've called for him to go."
Starmer has previously apologised for the appointment but defended his own actions, accusing Mandelson of creating a "litany of deceit" about his Epstein ties and promising to release documents on how he was appointed.
Senior minister Darren Jones told LBC Starmer was "furious" at not being told Mandelson had failed the security vetting and would update parliament on Monday.
He said Starmer had not misled parliament and the process around it was followed, but was flawed.
"I don't think it brings the prime minister's future into question," Jones said.
Mandelson is under police investigation for allegedly leaking government documents to the sex offender Jeffrey Epstein.
He has not commented publicly on allegations he leaked documents, and a lawyer for Mandelson did not provide a comment on Thursday about the vetting process.
