Developing Instruction-Following Agents in Grounded Environments
Developing instruction-following agents in grounded environments poses challenges like sample efficiency and generalizability. These agents need to learn from a few demonstrations and perform well in new environments with novel instructions. Techniques like reinforcement learning and imitation learning are common but can be costly due to their reliance on trial and error or expert guidance.
Language-Grounded Instruction Following with LLMs
In language-grounded instruction following, agents receive instructions and partial observations in the environment and take actions accordingly. Recent studies show that Large Language Models (LLMs) display sample-efficient learning when pretrained, across various tasks including robotic control. However, existing methods for instruction following depend on LLMs online during inference, which can be impractical and costly.
Introducing Language Feedback Models (LFMs)
Researchers from Microsoft Research and the University of Waterloo have proposed Language Feedback Models (LFMs) for policy improvement in instruction following. LFMs leverage LLMs to provide feedback on agent behavior in grounded environments, helping identify desirable actions. This technique enables sample-efficient and cost-effective policy improvement without continuous reliance on LLMs, offering interpretability for human validation of imitation data.
LFMs improve policy learning by leveraging LLMs to identify productive behavior and enhance policies without constant LLM interactions. They excel in identifying desirable behavior for imitation learning, outperform baseline methods, and offer cost-effective policy improvements. LFMs generalize well to new environments and provide detailed, human-interpretable feedback, fostering trust in imitation data.
In conclusion, Language Feedback Models significantly enhance policy performance in grounded instruction following, showing efficacy in identifying desirable behavior and improving policies without continual LLM usage.