OpenAI releases Jack of All Trades (JAT) — multi-purpose Transformer agent

AI Impact Summary

OpenAI has released Jack of All Trades (JAT), a multi-purpose Transformer agent designed for generalist agent training. This project leverages a Gato-inspired approach, training multi-modal Transformer models on a newly created dataset containing expert trajectories across diverse environments like Atari, BabyAI, Meta-World, and MuJoCo. The JAT model itself is based on EleutherAI’s GPT-Neo implementation and incorporates a novel embedding mechanism to handle sequential decision tasks, utilizing CNNs for image data and linear layers for continuous values.