OSG-Q3: Robust OpenSCENARIO Generation with Advanced Strategies on Large Language Models

Advanced Driver Assistance System (ADAS) testing is a critical process for validating driving functions before deployment. This process requires practitioners to transform natural language scenario descriptions into standardized formats for simulation. However, manual creation of executable scenarios is often laborious, complex, and time-consuming. Direct Large Language Model (LLM) application frequently fails due to syntax errors and hallucinations. In this work, we propose OSG-Q3, a simulator-aware LLM-based framework that generates a structured JSON representation through context-acquisition tools and then deterministically converts it into standardized scenarios. To ensure rigorous evaluation, we introduce a criteria-based benchmark of 162 collected ADAS test cases written in natural language, covering multiple ADAS functionalities. Experiments demonstrate that OSG-Q3 substantially improves scenario generation performance, increasing the success rate from 36.4% with direct Sonnet 4.6 inference to 62.3% under the OSG-Q3 framework. Furthermore, when deployed with a fine-tuned Qwen3-4B model, OSG-Q3 achieves the strongest performance among all evaluated configurations, reaching 76.5% scenario success rate, and outperforming Sonnet 4.6 by 14.2% under the same setting. These results indicate that the key components of OSG-Q3, including the intermediate representation and the fine-tuned model, each contribute to improving overall scenario generation performance.

OSG-Q3 is a simulator-aware framework for converting natural-language ADAS requirements into executable OpenSCENARIO XML files. The key design principle is to separate scenario reasoning, simulator-specific context acquisition, and XML synthesis: the LLM generates scenario semantics in a structured JSON representation, simulator-aware tools provide map and catalog information, and the converter deterministically produces OpenSCENARIO XML. Given an input requirement, an LLM-based agent may call simulator-aware tools during generation to obtain information about available maps, road topology, lane configurations, and catalog assets. The agent then produces the structured JSON representation. This JSON output is parsed, validated, converted into OpenSCENARIO XML, executed in Esmini, and evaluated using the checklist-based protocol.

OSG-Q3 evaluates generated scenarios as executable artifacts rather than text outputs. The generated OpenSCENARIO XML file is executed in Esmini, and the resulting trace is converted into an observed checklist. This observed checklist captures what actually happens during simulation, including spawn state, lane behavior, speed profile, and interactions with surrounding entities.

BibTeX

@misc{tri2026osgq3,
  title={OSG-Q3: Robust OpenSCENARIO Generation with Advanced Strategies on Large Language Models},
  author={Tran Cao Tri Doan and Huy Tran and Dac Hoang Phu Nguyen and Duc Dung Nguyen},
  year={2026},
  institution={AITech Lab, Ho Chi Minh City University of Technology, VNUHCM, Vietnam},
  url={https://github.com/ADML4U/OSG-Q3}
}

OSG-Q3: Robust OpenSCENARIO Generation with Advanced Strategies on Large Language Models

Abstract

Overview of OSG-Q3 Framework

Generated Scenarios in Esmini

BibTeX