Michiaki Tatsubori


2022

pdf
DiffG-RL: Leveraging Difference between Environment State and Common Sense
Tsunehiko Tanaka | Daiki Kimura | Michiaki Tatsubori
Findings of the Association for Computational Linguistics: EMNLP 2022

Taking into account background knowledge as the context has always been an important part of solving tasks that involve natural language. One representative example of such tasks is text-based games, where players need to make decisions based on both description text previously shown in the game, and their own background knowledge about the language and common sense. In this work, we investigate not simply giving common sense, as can be seen in prior research, but also its effective usage. We assume that a part of the environment states different from common sense should constitute one of the grounds for action selection. We propose a novel agent, DiffG-RL, which constructs a Difference Graph that organizes the environment states and common sense by means of interactive objects with a dedicated graph encoder. DiffG-RL also contains a framework for extracting the appropriate amount and representation of common sense from the source to support the construction of the graph. We validate DiffG-RL in experiments with text-based games that require common sense and show that it outperforms baselines by 17% of scores. We will make our code publicly available.

2021

pdf
Neuro-Symbolic Approaches for Text-Based Policy Learning
Subhajit Chaudhury | Prithviraj Sen | Masaki Ono | Daiki Kimura | Michiaki Tatsubori | Asim Munawar
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Text-Based Games (TBGs) have emerged as important testbeds for reinforcement learning (RL) in the natural language domain. Previous methods using LSTM-based action policies are uninterpretable and often overfit the training games showing poor performance to unseen test games. We present SymboLic Action policy for Textual Environments (SLATE), that learns interpretable action policy rules from symbolic abstractions of textual observations for improved generalization. We outline a method for end-to-end differentiable symbolic rule learning and show that such symbolic policies outperform previous state-of-the-art methods in text-based RL for the coin collector environment from 5-10x fewer training games. Additionally, our method provides human-understandable policy rules that can be readily verified for their logical consistency and can be easily debugged.

pdf
Neuro-Symbolic Reinforcement Learning with First-Order Logic
Daiki Kimura | Masaki Ono | Subhajit Chaudhury | Ryosuke Kohita | Akifumi Wachi | Don Joven Agravante | Michiaki Tatsubori | Asim Munawar | Alexander Gray
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Deep reinforcement learning (RL) methods often require many trials before convergence, and no direct interpretability of trained policies is provided. In order to achieve fast convergence and interpretability for the policy in RL, we propose a novel RL method for text-based games with a recent neuro-symbolic framework called Logical Neural Network, which can learn symbolic and interpretable rules in their differentiable network. The method is first to extract first-order logical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly interpretable logical operators. Our experimental results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark.

pdf
Language-based General Action Template for Reinforcement Learning Agents
Ryosuke Kohita | Akifumi Wachi | Daiki Kimura | Subhajit Chaudhury | Michiaki Tatsubori | Asim Munawar
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

pdf
LOA: Logical Optimal Actions for Text-based Interaction Games
Daiki Kimura | Subhajit Chaudhury | Masaki Ono | Michiaki Tatsubori | Don Joven Agravante | Asim Munawar | Akifumi Wachi | Ryosuke Kohita | Alexander Gray
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations

We present Logical Optimal Actions (LOA), an action decision architecture of reinforcement learning applications with a neuro-symbolic framework which is a combination of neural network and symbolic knowledge acquisition approach for natural language interaction games. The demonstration for LOA experiments consists of a web-based interactive platform for text-based games and visualization for acquired knowledge for improving interpretability for trained rules. This demonstration also provides a comparison module with other neuro-symbolic approaches as well as non-symbolic state-of-the-art agent models on the same text-based games. Our LOA also provides open-sourced implementation in Python for the reinforcement learning environment to facilitate an experiment for studying neuro-symbolic agents. Demo site: https://ibm.biz/acl21-loa, Code: https://github.com/ibm/loa

2020

pdf
Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
Subhajit Chaudhury | Daiki Kimura | Kartik Talamadupula | Michiaki Tatsubori | Asim Munawar | Ryuki Tachibana
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

We show that Reinforcement Learning (RL) methods for solving Text-Based Games (TBGs) often fail to generalize on unseen games, especially in small data regimes. To address this issue, we propose Context Relevant Episodic State Truncation (CREST) for irrelevant token removal in observation text for improved generalization. Our method first trains a base model using Q-learning, which typically overfits the training games. The base model’s action token distribution is used to perform observation pruning that removes irrelevant tokens. A second bootstrapped model is then retrained on the pruned observation text. Our bootstrapped agent shows improved generalization in solving unseen TextWorld games, using 10x-20x fewer training games compared to previous state-of-the-art (SOTA) methods despite requiring fewer number of training episodes.