SKY-Piano: A Multimodal Piano Performance Dataset
The paper presents SKY-Piano, a multimodal piano performance dataset with 11 hours of recordings, motion data, video, audio, MIDI, and MusicXML scores.
Presented a large multimodal piano performance dataset with various types of data and tools for annotating and generating MIDI from motion data.
Before reading this…
Applications
- →Music information retrieval
- →Piano performance analysis
To understand this paper, make sure you know these concepts first:
- Understanding of music information retrievalfind papers →
- Basic knowledge of piano performancefind papers →
Abstract
More Like ThisMusic information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.