Reliable Pseudo-Labels and Verified Reasoning for Label-Efficient Segmentation

Event Description

Abstract: Semantic segmentation needs a label for every pixel; reasoning segmentation, which segments what a user instruction describes, also needs the instructions. One Cityscapes image took over 1.5 hours to annotate; medical, satellite, and drone images need experts. Semi-supervised methods train on few labeled and many unlabeled images, with the model's predictions as pseudo-labels. Two weaknesses remain in such data-constrained scenarios: the pixel-level loss carries no context, and pseudo-labels are noisy; early in training, about one in five above a 0.95 threshold is wrong, causing error propagation.
This dissertation addresses the problem at its source: a prediction on an unlabeled image is used, as a pseudo-label for training or as the final mask at inference, according to a reliability estimate independent of the model's confidence score. The four methods compute it with a patch-level multi-label classifier, an object detector, consistency across paraphrased instructions, and a frozen multimodal large language model (MLLM) over foundation-model tools.
First, MPMC, a multi-scale patch-based multi-label classifier for any teacher-student framework, down-weights pseudo-labels contradicting its per-patch class predictions. Second, pseudo-label weighting scores each pixel by agreement with a detector trained on the same labels and by the overlap of its top feature indices with its class prototype. Both improve four frameworks on all splits of Pascal VOC and Cityscapes, the latter by 1.5 to 3.7 mIoU with 1/16 of the labels. Third, we formulate semi-supervised reasoning segmentation, where an MLLM learns from a few labeled images with instructions and many unlabeled images; CORA writes relational instructions from the masks, weights pseudo-labels by consistency across paraphrases, and aligns object tokens of labeled and unlabeled images, outperforming prior methods by 2.3 cIoU with 100 labeled Cityscapes and 2.4 with 180 PanNuke images. Finally, VisChain needs no target-domain labels: a frozen copy of the MLLM verifies each step of its reasoning chain against the image, retries failed stages with other tools, and learns tool choice from stored results. Trained only on natural images, it reaches 75.4 gIoU on ReasonSeg and leads every method not trained on them by 3.3 to 16.6 gIoU on satellite, drone, and object-part benchmarks.

Speaker: PhD student Prantik Howlader

Location: New Computer Science 109

Zoom Link: https://stonybrook.zoom.us/j/99512003190?pwd=UHWaJ3gmzgFPF8StokYo4rvwBwlh2X.1
Meeting ID: 995 1200 3190
Passcode: 709906

Date Start

Date End