In the world of AI, training agents to interact well with humans is crucial. But measuring progress in such complex interactions is not easy. That’s where the Standardised Test Suite (STS) comes in. Developed for evaluating agents in multi-modal interactions, STS places agents in real human interaction scenarios. These scenarios are then replayed to the agents, who receive instructions and continue the interaction offline. Their performance is then rated by human raters, ranking the agents based on their success rate.
Traditional methods of training AI, like reinforcement learning, don’t work well for human interactions. This is because many human behaviors are hard to put into words or formalize. For example, answering the question “What are you looking at?” depends on various factors like the context and speaker’s intent. Interactive evaluation by humans can be noisy and expensive, making it difficult to control instructions given to agents. Previous evaluation methods, like scripted probe tasks, also have limitations.
The STS methodology aims to change this by providing a controlled and rapid evaluation of agent performance in human-agent interactions. Similar to human-annotated datasets in machine learning, STS requires human annotations for now, but automation of these annotations may be possible in the future. This innovative method has the potential to accelerate research in human-agent interaction.