Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) limited controllability, since users can only modify input images while training VLMs is often out of reach; and (ii) lack of explainability, since it is difficult to trace VLM errors to specific causes. We propose TextFlow, addressing these issues with two stages: (i) a Vision Textualizer, which generates textual representations from flowchart images, and (ii) a Textual Reasoner, which performs question-answering based on the text representations. TextFlow offers three key advantages: users can select the type of text representation (e.g., Graphviz, Mermaid, PlantUML) or convert it into an executable graph object to call tools; it improves explainability by attributing errors more clearly to visual or textual processing components; and it promotes modularization, allowing advanced LLMs to be used in the reasoner stage when VLMs underperform end-to-end. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance and robustness.

Code: github.com/JunyiYe/TextFlow

Recommended citation: Ye, J., Dash, A., Yin, W., & Wang, G. (2025). "Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding." Proceedings of NAACL 2025.
Download Paper