Introducing Croissant: A New Metadata Format for ML-Ready Datasets
Machine learning (ML) practitioners often struggle with understanding and organizing datasets when training ML models. This slows down progress in the field of ML because of the wide variety of data representations. ML datasets can include text, structured data, images, audio, and video, each with their own unique organization and format.
To address this challenge, a community effort led to the development of Croissant, a new metadata format for ML-ready datasets. Croissant doesn’t change how data is represented, but provides a standard way to describe and organize it. It builds upon existing standards like schema.org and includes metadata specific to ML needs.
Why Croissant is important for the ML community
The majority of ML work involves handling data. Datasets are crucial in training models, and the lack of a common format can make data work more challenging. Croissant aims to simplify the ML development process by making datasets easier to find, clean, and analyze. It also enables ML frameworks to train and test models more efficiently.
How Croissant works
Users can now search for Croissant datasets on platforms like Kaggle, Hugging Face, and OpenML. They can also create, inspect, and modify Croissant metadata using the Croissant editor. Dataset authors can easily publish their datasets with Croissant metadata, making their data more discoverable and valuable.
Future directions for Croissant
To fully benefit from Croissant, the ML community needs to support and adopt this new format. All dataset creators are encouraged to provide Croissant metadata, and platforms hosting datasets should embed Croissant files for easy access. Tools that work with ML datasets should also consider supporting Croissant datasets.
Join us in contributing to the effort to reduce the data development burden and enhance the ML research and development ecosystem with Croissant.
Acknowledgements
Croissant was developed by a collaborative community effort, including teams from Dataset Search, Kaggle, and TensorFlow Datasets at Google. Other contributors come from organizations such as Bayer, Hugging Face, NASA, and Harvard, among others.