Striker Pablo Counago was the biggest culprit for the home side, missing three gilt-edged opportunities, the worst being a tame 69th minute penalty which Nicky Weaver palmed away easily.
In a tame first half there was little to warm the fans on a bitterly cold evening, with neither side hitting the target in the opening half hour.
Garvan put Town ahead with his fifth goal of the season when his curling free-kick beat everybody including Weaver to nestle in the far corner.
Weaver made a fine save to palm away a rasping Velice Sumulikoski drive from 25 yards before ex-Charlton man Kevin Lisbie curled an effort narrowly wide.
Town should have sealed the points midway through the second half when Kelly Youga needlessly handled to concede a penalty, but sub Counago's sidefoot effort was comfortably saved by Weaver diving to his left.
Bailey bravely cleared off his own line from a corner, being injured in the process, before Counago twice wastefully shot wide from great positions.
It had the feeling of a game where the visitors might snatch a point, despite being outplayed, and so it proved.
Sub Hameur Bouazza crossed from the right and Bailey collected, turned and rifled past Richard Wright from eight yards to stun the home fans.
The hosts couldn't believe it, but try as they might, they were unable to find a way through the Charlton rearguard again and left the field to a chorus of boos.