ViLA: Visual–Linguistic Abductive Reasoning with LLMs for Knowledge-based Visual Question Answering

motivation

Perception and reasoning run as separate steps

LLM-based multimodal reasoning either aligns image features with the language space or turns the image into text for the LLM. Both keep visual perception and language reasoning apart. Chain-of-thought drifts away from the question's intent. Direct prediction skips the knowledge the question needs.

People solve such questions by abduction. They guess candidate answers from what they see, justify each one with visual cues and background knowledge, and keep the explanation that fits best. ViLA models this process across the VLM and the LLM.

For 'What is required to be on the ground to do this sport?', chain-of-thought answers 'suitable environment' and direct prediction answers 'Skis', both wrong; the abductive chain scores premises for Skis and Snow and answers Snow.

abstract

Hypothesize, justify, then select

Recent attempts to leverage large language models (LLMs) for reasoning and pre-trained knowledge in multi-modal reasoning focus on two main approaches: aligning image features with linguistic space, and converting images into textual cues to exploit the implicit reasoning capabilities of LLMs. Although they integrate visual information into the reasoning pipeline, they often treat visual perception and language reasoning as separate processes, limiting the potential for fully unified multi-modal reasoning. In this paper, we propose a novel method, Visual–Linguistic Abductive Reasoning (ViLA), inspired by human abductive reasoning processes. ViLA hypothesizes a plausible answer, generates the corresponding visual and textual premises, and employs fuzzy scoring to select the most coherent combination, thus deriving the final inference. This process integrates visual and linguistic modalities into interpretable abductive reasoning chains, enabling unified multi-modal reasoning. Without fine-tuning LLMs or retrieving external knowledge, ViLA improves performance by 2.31% on AOKVQA, 1.7% on OKVQA, and 1.7% on GQA over previous state-of-the-art models, while also improving interpretability and stability.

method

An abductive chain across the VLM and the LLM

Two cats lying on a desk, one of them on a keyboard.

QWhere are the cats sitting on top of?

o₁desk

Cats are on a desk.
S_a 0.9S_r 0.50.86kept
Desk is a common place for cats to sit.
S_a 0.9S_r 0.10.82kept

H₁ ∧ H₂ ⊨ desk

o₂keyboard

Cats are on keyboard.
S_a 0.9S_r 0.50.86kept
Cats are on computer keyboard.
S_a 0.9S_r 0.50.86kept

H₃ ∧ H₄ ⊨ keyboard

S_a adequacy to the image (VLM)S_r relevance to the question (LLM)bar: validity μ_V, line at t = 0.7

VLM with H_final → keyboard ✓ LLaVA → desk ✗ CoT → desk ✗

Example and scores from Figure 6 of the paper. Validity is computed with w_v = 0.9 and w_l = 0.1.

Why keep several candidates

The VLM's first beam is right 58% of the time, but the correct answer is within the top 2 beams 72% of the time and within the top 10, 89%. The model often finds the right answer and then discards it at the final selection. ViLA keeps the top-k beam outputs as answer candidates for abductive reasoning.

Top-k accuracy rises from 58% at k=1 to 72% at k=2 and 89% at k=10, while Hit@k falls from 58 to 2.

Fuzzy scoring by prompt

Both models score every premise with the same multiple-choice prompt. The VLM judges whether it is reasonable to the image (adequacy μ_A), and the LLM whether it is relevant to the question (relevance μ_R). The options map to very low 0.1, low 0.5, medium 0.7, high 0.9 and very high 0.95. The value 0.5 counts as low because both models give moderate scores even for weak entailment.

Validity μ_V = w_v·μ_A + w_l·μ_R, a convex combination with w_v + w_l = 1. Premises with μ_V ≥ t become H_final.

Choose the most appropriate score to indicate
whether the information provided is {reasonable
to the image / relevant to the question}.
Return only the letter corresponding to your
choice from the options below:
(a) 0.1  (b) 0.5  (c) 0.7  (d) 0.9  (e) 0.95
(f) unknown

results

Higher accuracy without fine-tuning or retrieval

Evaluated on OK-VQA, A-OKVQA and GQA. Unless noted, the VLM is LLaVA-NeXT (Mistral-7B) and the LLM is Mistral-7B, with 2 answer candidates, 2 premises, t = 0.7 and w_v = 0.9.

69.98

A-OKVQA accuracy, best in the comparison

68.91

OK-VQA accuracy, best in the comparison

+4.85

GQA accuracy over LLaVA-NeXT

OK-VQA and A-OKVQA

Method	OK-VQA	A-OKVQA
Direct answering
MAVEx	41.37	–
UnifER	42.13	–
ViLBERT	35.20	30.60
LXMERT	36.91	30.70
KRISP	38.40	33.70
PICa	48.00	43.37
Prophet	61.10	58.20
LION	57.33	60.87
QACap	68.20	66.30
Structured reasoning
ViperGPT	51.90	39.50
VisProg*	41.84	52.12
VCTP	56.20	53.20
Base VLMs
BLIP2	32.32	38.94
InstructBLIP	61.16	62.23
LLaVA-NeXT	55.61	66.48
+ CoT (7B)	13.34	34.10
+ CoT (GPT-4)	30.56	52.71
Base VLMs + ViLA
BLIP2 + ViLA (7B)	34.05	39.37
InstructBLIP + ViLA (7B)	61.51	62.31
LLaVA-NeXT + ViLA (7B)	57.92	68.18
LLaVA-NeXT-13B + ViLA (GPT-4)	68.91	69.98

Accuracy (%). GPT-4 refers to gpt-4o-mini.

GQA

Method	Knowledge source	GQA
LXMERT	LXMERT	60.0
BLIP2	Vicuna-13B	41.0
InstructBLIP	Vicuna-7B	49.2
InstructBLIP	Vicuna-13B	49.5
ViperGPT	GPT-3	48.1
VisProg	CLIP, GPT-3	50.5
LLaVA-NeXT	Mistral-7B	58.74
+ CoT	Mistral-7B	32.1
+ ViLA	Mistral-7B	63.59

Accuracy (%) on GQA test-dev.

Turning the image into text for chain-of-thought hurts: LLaVA-NeXT drops from 58.74 to 32.1 on GQA with CoT. ViLA keeps each modality as its own premises and gains 4.85 instead.

Reasoning depth and question type

On GQA, ViLA beats LLaVA at every reasoning depth, most at one step (+18.1). It improves both binary and open questions. Its predicted answer distribution is also closer to the ground truth, with a lower chi-square distance (Dist).

GQA accuracy of LLaVA and ViLA by reasoning step (gains +18.1, +7.9, +0.9, +2.3, +0.4) and by question type (Binary +1.9, Open +7.3, Acc +4.9, Dist -0.1).

GQA question categories

ViLA improves every structural and semantic category. The largest gains are in Query among structural types and in Global and Category among semantic types.

	Structural reasoning
Model	Cho	Comp	Log	Qry	Ver	Avg
LLaVA	84.15	67.23	77.93	71.22	81.35	76.38
+ ViLA	85.56	68.25	79.87	78.55	83.75	79.20

	Semantic reasoning
Model	Attr	Cat	Glob	Obj	Rel	Avg
LLaVA	68.01	37.95	21.66	84.32	51.53	52.69
+ ViLA	71.33	51.72	64.33	86.89	54.97	65.85

Rationale quality

On the A-OKVQA validation set, ViLA's rationales are closer to the ground-truth rationales than chain-of-thought rationales from the same LLM and from GPT-4o-mini. Similarity is the average cosine similarity of CLIP (ViT-B/16) text embeddings. CoT* generates the rationale before the answer.

Method	LLM	Sentence similarity
CoT	Mistral-7B	38.7
CoT*	Mistral-7B	40.9
CoT	GPT-4o-mini	39.02
ViLA	Mistral-7B	44.8

Qualitative examples

For each answer candidate, ViLA builds premises and scores them with adequacy S_a and relevance S_r; the premises shown passed the fuzzy evaluation. With these premises, ViLA answers keyboard and water, while LLaVA and CoT answer desk and pond.

Two examples. Cats: premises for desk and keyboard with their scores; LLaVA and CoT answer desk, ViLA answers keyboard. Zebras: premises for pond and water; LLaVA and CoT answer pond, ViLA answers water.

Ablations

Accuracy stays at 66.41 for VLM weights 0.1 to 0.8 and jumps to 68.18 at w_v = 0.9. Visual adequacy matters more than textual relevance when judging premises.

Accuracy is flat at about 66.4% for VLM weight 0.1 to 0.8 and rises to about 68.2% at 0.9.

On A-OKVQA validation, 3 premises with 1 candidate or 2 of each work best. Too many premises or candidates add ambiguous evidence and lower accuracy.

Accuracy heatmap over number of answer candidates (1 to 5) and premises (1 to 5); the best cells are 68.18 at 1 candidate and 3 premises and 68.03 at 2 and 2.

Premises from each model help in different ways. Using only the LLM's or only the VLM's premises gives 65.33; combining both gives 68.18, a gain of 2.85 points.

LLM premises	VLM premises	Acc. (%)
✓		65.33
	✓	65.33
✓	✓	68.18

cite

BibTeX

@inproceedings{kim-etal-2026-visual,
  title     = {Visual{--}Linguistic Abductive Reasoning with {LLM}s for
               Knowledge-based Visual Question Answering},
  author    = {Kim, Jieun and Jeong, Yujin and Cho, Sung-Bae},
  booktitle = {Findings of the Association for Computational Linguistics:
               EACL 2026},
  year      = {2026},
  url       = {https://aclanthology.org/2026.findings-eacl.343/}
}