Abstract
This paper argues that integration is an essential approach to advancing the state of the art in cognitive systems. Integration is the path to aggregating requirements for cognition, as well as the solutions to those requirements. However, evaluation of integrated cognitive systems has proven difficult and expensive, which has hindered scientific advances. We suggest approaches to evaluating integrated cognitive systems by introducing the notion of practical evaluation, as well as the imposition of requirements that can only be satisfied with cognitive systems solutions. Practical evaluation involves not only measuring a system’s task competence but also the properties of adaptivity, directability, understandability, and trustworthiness.
