Ipswich then virtually sealed the points inside 15 minutes with goals from Owen Garvan - his first of the season - and top-scorer Alan Lee's 16th of the campaign.
Plymouth rallied impressively after the break and looked the more likely to score before Town substitute Danny Haynes sealed the win in the final minute with a fine individual goal.
The game's key moment arrived straight from the kick-off as Walters went down as he clashed with Doumbe as they chased a long ball.
Referee Jonathan Moss waved play on but dismissed Doumbe after consulting with his linesman who had his flag up.
It looked a harsh decision and was an odd one on the referee's part, having initially felt there was minimal contact.
Town went ahead with their first opening on nine minutes when Gary Roberts superbly brought down a deep David Wright cross-field ball and found Garvan, who turned Marcel Seip before curling the ball home.
Garvan was involved again as the hosts doubled their advantage on 15 minutes as he pulled the ball back for Lee to tap home from six yards after good work from Walters in the build-up.
The rest of the half was a procession of Ipswich chances, the best of them seeing Walters chip Luke McCormick only for Paul Connolly to clear and Walters taking what would have been an open goal off Lee when he could only glance on a Jason De Vos header back.
Lee also had the ball in the net when Walters slipped him in but was flagged offside in a borderline decision.
The second half was poor entertainment with Ipswich happy to sit back protecting their lead and allowing Plymouth plenty of time on the ball which they used it to good effect.
Sylvain Ebanks-Blake ran through to score on 53 minutes, but another decision went against Plymouth with the officials deciding he had handled as he took the ball away from De Vos.
Lewis Price denied Seip from a header, held an Ebanks-Blake shot and pulled off a good block to deny Barry Hayles as Town's slackness almost saw them punished.
Alex Bruce pulled off a goal-saving block to deny Lilian Nalis with ten minutes left, but then as the visitors tired, Haynes outpaced Peter Halmosi down the right and fired in the third off the post.
Haynes then set up fellow substitute Billy Clarke for what should have been a tap-in in stoppage time but the youngster skied hopelessly over the bar.