Music Caption Generation Using Large Language Models

Introduction

Music caption generation is a process of retrieving information from music tracks by generating natural language descriptions in the form of sentences. Unlike music tagging, which involves labeling music with specific tags, music captioning focuses on generating textual descriptions. This field has gained significant attention in recent years.

The Challenge of Dataset Collection

Researchers studying music caption generation face challenges due to the expensive and time-consuming task of collecting datasets. Moreover, the limited availability of music-language datasets makes training models for music captioning difficult. Large language models (LLMs) with billions of parameters have shown promise in handling tasks with minimal examples. These models are trained on vast amounts of text data from various sources, enabling them to understand and interpret words in different contexts.

The LP-MusicCaps Dataset

A team of researchers from South Korea developed a method called LP-MusicCaps (Large language-based Pseudo music caption dataset) to address the dataset scarcity. They created a music captioning dataset by applying LLMs to existing tagging datasets. The dataset consists of approximately 2.2 million captions paired with 0.5 million audio clips. The researchers proposed an LLM-based approach to generate the dataset and developed a systematic evaluation scheme for the music captions. They demonstrated that models trained on LP-MusicCaps perform well in both zero-shot and transfer learning scenarios.

Training and Evaluation

The researchers used the GPT-3.5 Turbo language model, known for its exceptional performance in various tasks, to perform music caption generation. The model underwent a training process with a large corpus of data and fine-tuning using reinforcement learning with human feedback. They compared the LLM-based caption generator with template-based methods and K2C augmentation. The template-based approach showed improved performance due to its utilization of musical context in the template.

Evaluating Caption Diversity

The diversity of the generated captions was evaluated using the BERT-Score metric. The LLM-based caption generator produced captions with higher BERT-Score values, indicating a wider range of language expressions and variations. This makes the generated captions more engaging and contextually rich.

As researchers continue to refine their approach, they aim to leverage the power of language models to advance music caption generation and contribute to music information retrieval.

Source link