@inproceedings{shang-etal-2024-traveler,
    title = "{T}rave{LER}: A Modular Multi-{LMM} Agent Framework for Video Question-Answering",
    author = "Shang, Chuyi  and
      You, Amos  and
      Subramanian, Sanjay  and
      Darrell, Trevor  and
      Herzig, Roei",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/declare-journal/2024.emnlp-main.544/",
    doi = "10.18653/v1/2024.emnlp-main.544",
    pages = "9740--9766",
    abstract = "Recently, image-based Large Multimodal Models (LMMs) have made significant progress in video question-answering (VideoQA) using a frame-wise approach by leveraging large-scale pretraining in a zero-shot manner. Nevertheless, these models need to be capable of finding relevant information, extracting it, and answering the question simultaneously. Currently, existing methods perform all of these steps in a single pass without being able to adapt if insufficient or incorrect information is collected. To overcome this, we introduce a modular multi-LMM agent framework based on several agents with different roles, instructed by a Planner agent that updates its instructions using shared feedback from the other agents. Specifically, we propose TraveLER, a method that can create a plan to ``\textbf{Trave}rse'' through the video, ask questions about individual frames to ``\textbf{L}ocate'' and store key information, and then ``\textbf{E}valuate'' if there is enough information to answer the question. Finally, if there is not enough information, our method is able to ``\textbf{R}eplan'' based on its collected knowledge. Through extensive experiments, we find that the proposed TraveLER approach improves performance on several VideoQA benchmarks without the need to fine-tune on specific datasets. Our code is available at \url{https://github.com/traveler-framework/TraveLER}."
}