Rubén Vargas scored in the 84th minute for the Swiss shortly after Tarik Muharemovic was sent off for a dangerous tackle, leaving Bosnia with 10 men.
Manzambi scored again in the 90th on a pass from Vargas, and captain Granit Xhaka converted from the penalty spot with the final kick of the game in stoppage time.
Veteran Xhaka had voiced sharp criticism on the team after the Qatar match which did not go down well everywhere, but he said he had no regrets.
"There is always a lot of unrest around me and a lot is written. Maybe I need these provocations. All that counts for me is that I am backed by the coach and the other players," the Sunderland player told Germany's MagentaTV.
"We played good football," he added and went on to praise the substitutes.
"We need every player. We got a push with them. The whole team profits. You need this mentality and hunger if you don't start."
Ermin Mahmic scored in the fourth minute of second-half injury time for Bosnia, who still have a legitimate chance of reaching the knockout stage in only their second World Cup after going toe-to-toe with higher-ranked Switzerland deep into the second half.
Switzerland followed up their disappointing 1-1 draw with Qatar last week by largely dominating play against the momentum-laden Dragons, who were unbeaten in their last nine competitive matches.
But the Swiss couldn't break through until a moment of brilliance from Manzambi, a Geneva native who plays for German club Freiburg - and then Muharemovic's red card changed the match.
Manzambi scored three minutes after coming on for standout winger Dan Ndoye. Manzambi jumped and connected perfectly on Amar Memic's attempted clearing header, setting off a wild celebration in the nervous Swiss section of SoFi Stadium.
"It is like a dream scoring two goals, I will definitely not sleep," 20-year-old man-of-the-match Manzambi said. "We wanted to win. We knew we wanted to do better than against Qatar."
Bosnia were backed by a raucous cheering section with tens of thousands of fans in the crowd of 70,026 celebrating their nation's second World Cup appearance.
The Dragons had less possession and fewer chances, yet every good moment was greeted by roars.
with DPA
