The AI revolution, which has begun to transform our lives over the past three years, is built on a fundamental linguistic principle that lies at the base of large language models such as ChatGPT. Words in a natural language are not strung together in random patterns; rather, there is a statistical structure that allows the model to guess the next word based on what came before. Yet these models overlook a crucial dimension of human communication: content that is not conveyed by words. In a new study in Proceedings of the National Academy of Sciences, USA (PNAS), researchers from Prof. 鈥檚 lab at the Weizmann Institute of Science reveal that the melody of speech in spontaneous conversations in English functions as a distinct language, with a 鈥渧ocabulary鈥� of hundreds of basic melodies and even rules of syntax that can be used to predict the next melody in the sequence. The study lays the foundation for an artificial intelligence that will understand language beyond words.
The melody, or music, of speech, referred to by the linguistic term 鈥減rosody,鈥� encompasses variations in pitch (intonation), loudness (for example, for emphasis), tempo and sound quality (such as a whisper or creaky voice). This form of expression predates words in evolution: Recent studies reveal that both chimpanzees and whales incorporate complex prosodic structures in their communication. In human communication, prosody adds a nuanced layer of meaning beyond words. A brief pause, much like a comma, can change the meaning of a sentence (鈥淟et鈥檚 eat Grandma鈥�) and the tempo of spoken text can generate suspense. Linguists specializing in prosody have traditionally studied literary texts and ways in which prosody reflects historical changes. This meant that, despite prosody鈥檚 critical importance for the understanding of human language, its study remained a niche field for years, devoid of applications and filled with conflicting ideas about prosody鈥檚 structure and significance.
“”Our study lays the foundation for an automated system to compile a 鈥榙ictionary鈥� of prosody for every human language and for different speaker populations”
Prosody, however, is an inherent part of every conversation. It assigns linguistic function to words 鈥� for instance, whether they pose a question or state a fact 鈥� and reveals the speakers鈥� attitude toward what they say. In the new study, led by linguist Dr. Nadav Matalon and neuroscientist Dr. Eyal Weinreb from Moses鈥檚 lab in Weizmann鈥檚 Physics of Complex Systems Department, the researchers analyzed prosody as an unfamiliar language, aiming to deliver a data-driven explanation for the linguistic mystery of prosody鈥檚 structure and meaning. Rather than relying on literature, they used two massive collections of audio recordings of spontaneous conversations, one of telephone conversations between two participants and the other of face-to-face conversations in various locations, such as a kitchen or classroom.
The first task for the research team was to compile a dictionary of the short melodies that function as 鈥渨ords鈥� in English-language prosody and to assign each of them a function and a meaning. 鈥淭o understand why there is no prosodic dictionary yet, it鈥檚 worth remembering that there wasn鈥檛 even a comprehensive English dictionary until the nineteenth century,鈥� Moses says. 鈥淲hen the University of Oxford was tasked with compiling one, it asked the public to help with the workload by sending quotes showing the historical changes in the meaning of words. One of the main contributors was a prisoner who spent more than 20 years reading books and sending quotes. In our study, instead of collecting information by ourselves over the course of decades, we analyzed massive collections of audio recordings, using AI.鈥�
The melody of each person鈥檚 speech is unique, but the AI model found several hundred basic patterns that recur, with slight variations, in all spontaneous English conversations. While written words are sequences of letters, a prosodic 鈥渨ord鈥� is a short melody, that is, a short sequence of sounds with varying pitch, lasting about a second on average. To work out the meaning of these 鈥渨ords,鈥� Matalon sampled 20 basic melodic patterns and then listened to the recordings again. 鈥淲e discovered that each pattern has several linguistic functions,鈥� he explains. 鈥淔or example, depending on the context, a pattern can define whether someone is asking a question or making a statement. However, each pattern typically conveys one specific attitude of the speaker 鈥� such as curiosity, surprise or confusion 鈥� toward what鈥檚 being said. One common prosodic 鈥榳ord鈥� is a sharp rise of the pitch followed by a quick drop. This pattern signals enthusiasm and, depending on the context, can express strong agreement or acknowledgment of receiving important new information.鈥�
“”Oxford鈥檚 first full English dictionary came out in the 19th century, with the public helping manage the workload 鈥� including a prisoner who contributed for 20 years”
Next, the researchers tried to identify syntactic rules governing the order of these prosodic patterns, which can potentially allow future language learning models to understand and use prosody. 鈥淲e noticed that there are patterns that tend to appear next to each other, in pairs, in spontaneous speech,鈥� Weinreb explains. 鈥淚t鈥檚 a simple statistical system, in which the correct choice of the next unit in a sequence depends solely on the previous one. This system works well for spontaneous conversation because it requires planning only a few seconds ahead, which is just as long as short-term memory lasts.鈥� These pattern pairs, the researchers discovered, act as simple sentences, expressing 鈥渙ne new idea,鈥� so that each pair relates to a specific topic, adding a single piece of information about it 鈥� for example, referring to a fact mentioned in the conversation and providing positive feedback.
鈥淥ur study lays the foundation for the development of an automated system that will compile a 鈥榙ictionary鈥� of prosody and identify its syntactic rules for every human language and for different speaker populations,鈥� Moses says.
鈥淧rosody can vary depending on social status, historic events and the age of the speakers, and these variations can even manifest themselves in literary works that carefully reflect spontaneous speech,鈥� Matalon adds. 鈥淲e analyzed audiobooks as part of the study and discovered that prosodic patterns are longer in scripted speech and that the simple paired syntax of spontaneous conversation has disappeared. There are other differences, too. It鈥檚 safe to assume that the aging process and the acquisition of language in childhood are also accompanied by quantifiable prosodic changes. Moreover, there is evidence that prosody is important in internal speech 鈥� the language of thought 鈥� and that we can deepen our understanding of the existing prosody of robotic voices that are produced by speech-generating devices. The model we created promises to close the gaps that emerged over the centuries in research into expression beyond words.鈥�
A major future application of an automated prosodic dictionary might be the development of AI capable of understanding and conveying messages through the melody of speech rather than words alone. 鈥淚magine if Siri could understand from the melody of your voice how you feel about a certain subject, what鈥檚 important to you or whether you think you know better than her,鈥� Weinreb adds, 鈥渁nd that she could adapt her response to make it sound enthusiastic or sad. We already have brain implants that convert neural activity into speech for people who can鈥檛 speak. If we can teach prosody to a computer model, we鈥檒l be adding a significant layer of human expression that robotic systems currently lack.鈥�