Poems Set to Music: A Multimodal Analysis of Schubert’s Song Cycle Winterreise

A Kunstlied (German for art song) or simply Lied is a poem set to music by a composer. This genre, which flourished in the 19th century, constitutes a unique connection between text and music. In this paper, we take the first steps towards a computational, multimodal analysis of this close relationship between poems and art songs. Our aim is to explore the relations between emotions expressed in poetry and musical devices chosen by the composers. Our strategy involves annotating text and music for semantic properties. Based on the musical time scale, we then measure the extent to which these properties overlap and show potential correlations between them. As a case study, we apply our methods to the Schubert Winterreise Dataset, a multimodal dataset of 24 songs by Franz Schubert including text, score, and audio data along with various annotations. By examining local keys and chords as musical properties, we find that, overall, Schubert used more major keys to express positive emotions such as joy and love relative to the whole song cycle. For negative emotions, we find that Schubert used chords that exert more tension and less stability. We conclude that our strategy is able to unveil cross-modal relationships and reflect on necessary steps towards a large-scale corpus analysis.