FuzzyCD: Fuzzy Contrastive Decoding to Alleviate Object Hallucination in Large Vision-Language Models

motivation

The distorted input does not always hallucinate

Contrastive decoding subtracts the logits of a distorted input x′ from the original logits. This assumes x′ amplifies hallucination. When it does not, the subtraction creates new hallucinations instead of removing them.

In this example, a filtered image makes the model more confident in the correct color. Subtracting its logits pushes a wrong color to the top. FuzzyCD first checks how confident the model is on each input, then decides how much to subtract.

Q: What color is this?

Logits from Figure 5 of the paper. Step 3 applies Eq. 2 with α = 1.

abstract

Fuzzy rules decide how much to contrast

Large vision-language models (LVLMs) often exhibit object hallucination, a phenomenon where models generate descriptions of non-existent objects within images. Prior methods have sought to mitigate this issue by adjusting model logits to reduce linguistic bias, but they often lack precise control over visual uncertainty, sometimes exacerbating hallucinations instead of mitigating them. To address this limitation, we propose a novel decoding strategy called fuzzy contrastive decoding (FuzzyCD) that uses Takagi-Sugeno fuzzy inference to refine hallucination control. FuzzyCD adaptively assigns weights to high-hallucination logits while mitigating unnecessary linguistic bias. Specifically, it transforms the log-probabilities of top-1 tokens from both standard and hallucination logits into a confidence linguistic fuzzy set. Through Takagi-Sugeno fuzzy inference, it dynamically adjusts hallucination logits to prevent the model from over-relying on spurious linguistic patterns. Experimental results on object hallucination datasets demonstrate that hallucination is mitigated by 11%p compared to conventional LVLMs. In-depth analyses highlight the effectiveness of FuzzyCD in enhancing the reliability of vision-language models.

method

From confidence to contrast, step by step

01 Input

02 Fuzzify

03 Rule weights

04 Rule outputs

05 Aggregate

06 Contrast

Why log-probability

To choose the confidence score, a simple classifier trained on VizWiz detects hallucination in LLaVA-1.5 from five logit statistics. Probability and log-probability reach the highest AUC in every setting.

	VizWiz	POPE
Criterion	val	random	popular	adversarial
Logits-mean	0.58	0.24	0.26	0.31
Logits-min	0.56	0.25	0.27	0.31
Logits-max	0.70	0.69	0.72	0.72
Probability	0.75	0.79	0.78	0.76
Log-probability	0.75	0.79	0.78	0.76

Hallucination detection AUC, LLaVA-1.5. Highlighted: the score FuzzyCD uses.

Density plots of logits mean, logits max, probability and log-probability for correct and incorrect answers.

results

Best accuracy in every POPE setting

Evaluated on POPE, MME and ROPE against VCD, ICD, VDD and OPERA, with LLaVA-1.5 and InstructBLIP (Vicuna-7B).

6 / 6

POPE settings with the highest accuracy

+5.43

avg. POPE accuracy gain, LLaVA-1.5

641.66

MME total, highest among methods

POPE (MSCOCO)

	LLaVA-1.5				InstructBLIP
Method	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1

Bold: best per column. Highlighted: FuzzyCD.

MME, LLaVA-1.5

Largest gains on object-level tasks: Existence +19.33, Count +56.66.

	Object		Attribute
Method	Existence	Count	Position	Color	Total
Default	175.67	106.67	114.00	160.00	556.34
+ VCD	184.66	138.33	128.67	153.00	604.66
+ VDD	190.00	138.33	126.67	165.00	620.00
+ FuzzyCD	195.00	163.33	123.33	160.00	641.66

ROPE, LLaVA-1.5

Best on all six splits. Single-object Hom. 31.88 → 57.88.

	Multi-object			Single-object
Method	Wild	Hom.	Het.	Wild	Hom.	Het.
LLaVA-1.5	13.96	31.88	3.98	13.96	31.88	3.98
+ VCD	13.57	28.68	6.37	24.76	48.33	9.84
+ ICD	14.35	32.63	5.69	22.92	45.47	10.24
+ VCC	17.80	40.77	7.40	27.49	52.12	11.06
+ OPERA	13.20	37.14	3.82	13.20	37.14	3.82
+ FuzzyCD	21.12	46.44	7.45	29.01	57.88	11.30

Across model families and scales

POPE accuracy improves on all five LVLMs from 7B to 72B, most on 7B to 13B models.

Accuracy of base decoding and FuzzyCD across five LVLMs: LLaVA 1.5 7B +5.12, InstructBLIP 7B +5.41, LLaVA-NeXT 13B +4.46, QwenVL 32B +0.72, QwenVL 72B +1.05.

Choice of image filter

POPE accuracy stays within 86.5 to 89.3 across five filters. Sobel is best.

Filter	Acc. (%)
Bilateral	86.86
Median	86.53
Sobel	89.33
Sharpen (default)	88.76
Gaussian	87.83

Qualitative example

Asked “What is this a picture of?”, the model is confident on both the original and the filtered image (High 0.99 and 0.91). Rule 9 gets the highest weight, which signals no hallucination, so the filtered logits are not amplified and FuzzyCD answers sugar. Standard contrastive decoding suppresses Sugar and Domino, the top candidates on both inputs, and answers k.

Rule weights for the Domino Sugar example: Rule 9 is highest; FuzzyCD multiplies the amplified logits by zero and answers sugar, while visual contrastive decoding answers k.

cite

BibTeX

@inproceedings{kim2025fuzzycd,
  title     = {Fuzzy Contrastive Decoding to Alleviate Object Hallucination
               in Large Vision-Language Models},
  author    = {Kim, Jieun and Kim, Jinmyeong and Kim, Yoonji and Cho, Sung-Bae},
  booktitle = {Proceedings of the IEEE/CVF International Conference
               on Computer Vision (ICCV)},
  pages     = {20572--20581},
  year      = {2025},
  doi       = {10.1109/ICCV51701.2025.01913}
}