Abstract
Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.- Anthology ID:
- D17-1103
- Volume:
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Month:
- September
- Year:
- 2017
- Address:
- Copenhagen, Denmark
- Editors:
- Martha Palmer, Rebecca Hwa, Sebastian Riedel
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 979–985
- Language:
- URL:
- https://aclanthology.org/D17-1103
- DOI:
- 10.18653/v1/D17-1103
- Cite (ACL):
- Ramakanth Pasunuru and Mohit Bansal. 2017. Reinforced Video Captioning with Entailment Rewards. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 979–985, Copenhagen, Denmark. Association for Computational Linguistics.
- Cite (Informal):
- Reinforced Video Captioning with Entailment Rewards (Pasunuru & Bansal, EMNLP 2017)
- PDF:
- https://preview.aclanthology.org/fix-dup-bibkey/D17-1103.pdf
- Data
- MSR-VTT