Lee sent Willy Gueret the wrong way to net his 12th goal of the season after Richard Duffy had tugged back Jaime Peters six yards from goal on 64 minutes.
Swansea began on the front foot and inside six minutes both Darren Pratley and Ade Akinfenwa had fired in long-range shots.
In between, Dan Harding's header into the penalty area dropped to Lee near the penalty spot, but his attempt to loft his header over Gueret saw the ball soar over the crossbar.
Swansea carved out a chance midway through the half when lone forward Akinfenwa laid the ball back for Thomas Butler, but his 20-yarder was tame and easily gathered by Welsh international Lewis Price.
Billy Clarke shot wide ambitiously from 30 yards after Izzy Iriekpen's clearance came straight to him with Gueret off his line, before Peters got to the byline and pulled the back for Sylvain Legwinski to curl narrowly wide from the edge of the box.
Butler curled way over before Akinfenwa was left unmarked to reach a Tom Williams cross, but his header was too weak to trouble Price.
Lee headed over from a Gary Roberts cross before Clarke wasted a good chance, volleying wide from 15 yards after Sito Castro's lofted ball dropped to him in the box.
There was controversy just before the interval as Clarke, waved back on after leaving the field for treatment, robbed the unaware Duffy, swapped passes with Lee, burst past Dennis Lawrence and fired wide.
Ipswich began the second half brightly, with Sito weaving his way past two men and crossing towards Clarke on the penalty spot, only for Duffy to arrive just in time to nod behind.
Andy Robinson was booked for bringing down Roberts from behind and the Ipswich winger took the free-kick himself, curling wide from almost 30 yards.
Town went ahead on 64 minutes when Gavin Williams' free-kick to the far post looked as if it would drop for Peters to volley home from six yards, only for Tom Williams to pull the Canadian's shirt.
Referee Nigel Miller spotted the infringement, with Lee stepping forward to send Gueret the wrong way from the spot.
Lee Trundle was introduced two minutes later and within three minutes Swansea carved out a good chance, with Robinson playing in Leon Britton whose fierce low drive was well blocked by the legs of Price.
Trundle almost created an equaliser on 78 minutes when he got on the end of an Akinfenwa flick, held off Harding and Fabian Wilnis and fired right across the face of goal with Butler throwing himself at the ball a couple of yards out, but unable to get a touch.
A long ball launched by sub Marcos Painter then went over Trundle's head, but fellow sub Pawel Abbott stole in behind him, but Price was in the right place to block his flicked header with his chest and the loose ball was scrambled away.
Sito clumsily fouled Butler to concede a free-kick in a dangerous area, and was rightly booked, but again set-piece specialist Robinson disappointed by lofting the ball well over.
Ipswich managed to hang on through the rest of the stoppage time to end Swansea's FA Cup dream for another year, while Town, who played at Wembley in 2000's play-off final, just before its demolition, keep their hopes alive of returning there in the year it reopens.
Photos from the Swansea Match added to the gallery - click here to view