Engineered AI Still Matters for Question Answering

Many question-answering systems rely on a significant amount of engineering effort. They often require both knowledge bases and rules, which can be very expensive to create. Even when these systems rely heavily on statistical machine learning, there is also an enormous amount of effort spent on identifying what features are useful for learning and implementing capabilities (often using knowledge bases and rules) to assign values to those features. However, in recent years an alternative approach has been growing in popularity: single-strategy systems in which one statistical model is used to address the entire task. These models take in questions and content and produce answers from that content. Of course, configuring a statistical learning system (e.g., deciding on the structure of a neural network) requires some engineering work, but it can be vastly less. As a result, people building practical question-answering systems for commercial use have an incentive to prefer this kind of technology: it is cheaper to develop. Furthermore, there is mounting evidence in the field of reading comprehension to suggest that single-strategy statistical models are extremely effective at answering questions. However, we believe that this evidence is a misleading artifact of the way that the reading comprehension task has been formalized. We explore two types of data: for conventional reading comprehension and for end-to-end factoid question answering. We show that the former is extraordinarily well suited to pure statistical methods and there is little additional benefit from engineered knowledge and rules. However, we also show that this result does not generalize to factoid question answering, where engineered knowledge and rules have substantially more impact. We conclude that the dramatic success of purely statistical methods on conventional reading comprehension reflect the artificially constrained nature of the problem and that engineered knowledge and rules remain useful for more realistic and open-ended tasks.