@inproceedings{smith-crabb-etal-2025-pushing,
    title = "Pushing the (Generative) Envelope: Measuring the Effect of Prompt Technique and Temperature on the Generation of Model-based Systems Engineering Artifacts",
    author = "Smith Crabb, Erin  and
      Bernard, Cedric  and
      Jones, Matthew  and
      Dakota, Daniel",
    editor = "Angelova, Galia  and
      Kunilovskaya, Maria  and
      Escribe, Marie  and
      Mitkov, Ruslan",
    booktitle = "Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era",
    month = sep,
    year = "2025",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://preview.aclanthology.org/corrections-2026-01/2025.ranlp-1.137/",
    pages = "1188--1194",
    abstract = "System engineers use Model-based systems engineering (MBSE) approaches to help design and model system requirements. This manually intensive process requires expertise in both the domain of artifact creation (e.g., the requirements for a vacuum), and how to encode that information in a machine readable form (e.g., SysML). We investigated leveraging local LLMs to generate initial draft artifacts using a variety of prompt techniques and temperatures. Our experiments showed promise for generating certain types of artifacts, suggesting that even smaller, local models possesses enough MBSE knowledge to support system engineers. We observed however that while scores for artifacts remain stable across different temperature settings, this is potentially misleading as significantly different, though semantically equivalent, generations can be produced."
}