Aleksandra Ćwiek

Aleksandra Ćwiek

Phonetician

Publications

Is gesture-speech physics at work in rhythmic pointing? Evidence from Polish counting-out rhymes

Published in Proceedings of the 20th International Congress of Phonetic Sciences, 2023

Gesture-speech physics’ refers to a possible biomechanical coupling between manual gesture and speech. According to this thesis, rapid gesturing leaves a direct imprint on acoustics (intensity, F0), as gesture accelerations/decelerations increase expiratory forces and therefore subglottal pressure, leading to higher amplitude envelope peaks and higher F0 values. This acoustic effect has been reported in lab experiments, spontaneous speech, clinical studies, and professional vocal performers. The current study investigates this phenomenon in Polish counting-out rhymes, using motion capture data and acoustic recordings from 11 native Polish speakers. Following the gesture-speech physics thesis, we expect acceleration/deceleration peaks to be correlated with speech intensity/F0. Through Bayesian analyses, we obtained a weak but reliable coupling of deceleration of the pointing hand and the nearest peak in the smoothed amplitude envelope.

Recommended citation: Kadavá, Š., Ćwiek, A., Stoltmann, K., Fuchs, S., & Pouw, W. (2023). Is gesture-speech physics at work in rhythmic pointing? Evidence from Polish counting-out rhymes. Proceedings of the 20th International Congress of Phonetic Sciences. International Congress of Phonetic Sciences (ICPhS 2023), Prague, Czech Republic. https://doi.org/10.31219/osf.io/67fzc

A Roadmap for Technological Innovation in Multimodal Communication Research

Published in Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, 2023

This is a collaborative paper written by the ViCom consortium

Recommended citation: Gregori, A., Amici, F., Brilmayer, I., Ćwiek, A., Fritzsche, L., Fuchs, S., Henlein, A., Herbort, O., Kügler, F., Lemanski, J., Liebal, K., Lücking, A., Mehler, A., Nguyen, K. T., Pouw, W., Prieto, P., Rohrer, P. L., Sánchez-Ramón, P. G., Schulte-Rüther, M., … von Eiff, C. I. (2023). A Roadmap for Technological Innovation in Multimodal Communication Research. In V. G. Duffy (Ed.), Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management (pp. 402–438). Springer Nature Switzerland. http://olacwiek.github.io/files/Gregori_et_al._2023_submitted_version.pdf

Random forests, sound symbolism and Pokémon evolution

Published in PLOS One, 2023

This study constructs machine learning algorithms that are trained to classify samples using sound symbolism, and then it reports on an experiment designed to measure their understanding against human participants. Random forests are trained using the names of Pokémon, which are fictional video game characters, and their evolutionary status. Pokémon undergo evolution when certain in-game conditions are met. Evolution changes the appearance, abilities, and names of Pokémon. In the first experiment, we train three random forests using the sounds that make up the names of Japanese, Chinese, and Korean Pokémon to classify Pokémon into pre-evolution and post-evolution categories. We then train a fourth random forest using the results of an elicitation experiment whereby Japanese participants named previously unseen Pokémon. In Experiment 2, we reproduce those random forests with name length as a feature and compare the performance of the random forests against humans in a classification experiment whereby Japanese participants classified the names elicited in Experiment 1 into pre-and post-evolution categories. Experiment 2 reveals an issue pertaining to overfitting in Experiment 1 which we resolve using a novel cross-validation method. The results show that the random forests are efficient learners of systematic sound-meaning correspondence patterns and can classify samples with greater accuracy than the human participants.

Recommended citation: Kilpatrick, A. J.*, Ćwiek, A.*, & Kawahara, S. (2023). Random forests, sound symbolism and Pokémon evolution. PLOS ONE, 18(1), e0279350. https://doi.org/10.1371/journal.pone.0279350

A cross-linguistic, sound symbolic relationship between labial consonants, voiced plosives, and Pokémon friendship

Published in Frontiers in Psychology, 2023

This paper presents a cross-linguistic study of sound symbolism, analysing a six-language corpus of all Pokémon names available as of January 2022. It tests the effects of labial consonants and voiced plosives on a Pokémon attribute known as friendship. Friendship is a mechanic in the core series of Pokémon video games that arguably reflects how friendly each Pokémon is. Poisson regression is used to examine the relationship between the friendship mechanic and the number of times /p/, /b/, /d/, /m/, /g/, and /w/ occur in the names of English, Japanese, Korean, Chinese, German, and French Pokémon. Bilabial plosives, /p/ and /b/, typically represent high friendship values in Pokémon names while /m/, /d/, and /g/ typically represent low friendship values. No association is found for /w/ in any language. Many of the previously known cases of cross-linguistic sound symbolic patterns can be explained by the relationship between how sounds in words are articulated and the physical qualities of the referents. This study, however, builds upon the underexplored relationship between sound symbolism and abstract qualities.

Recommended citation: Kilpatrick, A., Ćwiek, A., Lewis, E., & Kawahara, S. (2023). A cross-linguistic, sound symbolic relationship between labial consonants, voiced plosives, and Pokémon friendship. Frontiers in Psychology, 14. https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1113143

Lautsymbolische Größencodierung bei der Benennung von Hunden

Published in 18. Phonetik und Phonologie im deutschsprachigen Raum, 2022

This abstract was first-authored by the Nadine Christoph, an MA student from my course on sound symbolism in the summer semester 2022 at the Humboldt-Universität zu Berlin. She presented it at the 18th P&P in Bielefeld.

Recommended citation: Christoph, N., & Ćwiek, A. (2022). Lautsymbolische Größencodierung bei der Benennung von Hunden. Tagungsband der 18. Tagung Phonetik und Phonologie im deutschsprachigen Raum. Phonetik und Phonologie im deutschsprachigen Raum, Bielefeld, Germany. http://olacwiek.github.io/files/PP_2022_paper_3865.pdf

The bouba/kiki effect is robust across cultures and writing systems

Published in Philosophical Transactions of the Royal Society B: Biological Sciences, 2022

Linguistic communication requires speakers to mutually agree on the meanings of words, but how does such a system first get off the ground? One solution is to rely on iconic gestures: visual signs whose form directly resembles or otherwise cues their meaning without any previously established correspondence. However, it is debated whether vocalizations could have played a similar role. We report the first extensive cross-cultural study investigating whether people from diverse linguistic backgrounds can understand novel vocalizations for a range of meanings. In two comprehension experiments, we tested whether vocalizations produced by English speakers could be understood by listeners from 28 languages from 12 language families. Listeners from each language were more accurate than chance at guessing the intended referent of the vocalizations for each of the meanings tested. Our findings challenge the often-cited idea that vocalizations have limited potential for iconic representation, demonstrating that in the absence of words people can use vocalizations to communicate a variety of meanings.

Recommended citation: Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., Kawahara, S., Koutalidis, S., Krifka, M., Lippus, P., Lupyan, G., Oh, G. E., Paul, J., Petrone, C., Ridouane, R., Reiter, S., Schümchen, N., Szalontai, Á., Ünal-Logacev, Ö., … Winter, B. (2022). The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1841), 20200390. https://royalsocietypublishing.org/doi/10.1098/rstb.2020.0390

Novel vocalizations are understood across cultures

Published in Scientific Reports, 2021

Linguistic communication requires speakers to mutually agree on the meanings of words, but how does such a system first get off the ground? One solution is to rely on iconic gestures: visual signs whose form directly resembles or otherwise cues their meaning without any previously established correspondence. However, it is debated whether vocalizations could have played a similar role. We report the first extensive cross-cultural study investigating whether people from diverse linguistic backgrounds can understand novel vocalizations for a range of meanings. In two comprehension experiments, we tested whether vocalizations produced by English speakers could be understood by listeners from 28 languages from 12 language families. Listeners from each language were more accurate than chance at guessing the intended referent of the vocalizations for each of the meanings tested. Our findings challenge the often-cited idea that vocalizations have limited potential for iconic representation, demonstrating that in the absence of words people can use vocalizations to communicate a variety of meanings.

Recommended citation: Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., Kawahara, S., Koutalidis, S., Krifka, M., Lippus, P., Lupyan, G., Oh, G. E., Paul, J., Petrone, C., Ridouane, R., Reiter, S., Schümchen, N., Szalontai, Á., Ünal-Logacev, Ö., … Perlman, M. (2021). Novel vocalizations are understood across cultures. Scientific Reports, 11(1), 10108. https://www.nature.com/articles/s41598-021-89445-4

Hand-Mouth Coordination in a Pointing Task Requiring Manual Precision

Published in Proceedings of the 12th ISSP, 2020

In daily life, articulatory movements and pointing gestures are tightly coupled. Nevertheless, the two motor systems governing the movements of the articulators and hands differ in their dynamics: the articulators are fast and much lighter than the limbs, which are slower due to their mass. We investigated the timely coordination of those motor systems in a pointing task requiring manual precision. In our experiment, the initial segment was always [p], allowing the participants for early articulatory preparation. Most importantly, we found that the hand gesture onset precedes the onset of the articulatory gesture. We also found that some speakers begin the articulatory movement only after reaching the hand gesture target. Overall, our data reveal that when the articulatory movement is not audible, as it is the case of [p], speakers are very flexible in the coordination between hand and mouth.

Recommended citation: Ćwiek, A., & Fuchs, S. (2020). Hand-Mouth Coordination in a Pointing Task Requiring Manual Precision. Proceedings of the 12th ISSP. 12th International Seminar on Speech Production, New Haven, CT: Haskins Laboratories. http://olacwiek.github.io/files/ISSP_Cwiek_Fuchs.pdf

Iconic prosody is rooted in sensori-motor properties: Fundamental frequency and the vertical space

Published in CogSci 2019, 2019

The iconic cross-modal correspondence between fundamental frequency and location in vertical space (“high is up”) has long been described in the literature. However, an explanation for this relationship has not been proposed. We conducted an experiment in which participants shot at cans projected on the wall in different vertical positions. We found that mean fundamental frequency was significantly influenced by vertical head position. Moving the head upwards changes the position of the larynx, which pulls on the cricothyroid muscle and changes the fundamental frequency. We thus propose that the iconic relationship between fundamental frequency and vertical space is grounded in the body.

Recommended citation: Ćwiek, A., & Fuchs, S. (2019). Iconic prosody is rooted in sensori-motor properties: Fundamental frequency and the vertical space. Proceedings of the Annual Meeting of the Cognitive Science Society, 41, 1572–1578. http://olacwiek.github.io/files/0282.pdf

Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies

Published in Journal of Phonetics, 2019

In this paper, we explore the possibility to gather perceptual impressions of prosodic prominence by exploiting the strong prosody-gesture link, i.e., by having listeners transform a perceptual impression into a motor movement, namely drumming, for two domains of prominence: word-level and syllable-level. A feasibility study reveals that such a procedure is indeed easily and speedily mastered by naïve listeners, but more difficult for word-level prominences. We furthermore examine whether “drummed” annotations are comparable to those gathered with more established annotation protocols based on cumulative naïve impressions and fine-grained expert ratings. These comparisons reveal high correspondences across all prominence annotation protocols, thus corroborating the general usefulness of the gestural approach. The analyses also reveal that all annotation protocols are strongly driven by structural linguistic considerations. We then use Random Forest Models to investigate the relative impact of signal and structural cues to prominence annotations. We find that expert ratings of prosodic prominence are guided comparatively more by structural concerns than those of naïve annotators, that word-level annotations are influenced more by structural linguistic cues than syllable-level ones, and that “drummed” annotations are driven least by structural cues. Lastly, we isolate two main listener strategies among our group of “drummers”, namely those integrating structural and signal cues to prominence, and those being guided predominantly by signal cues.

Recommended citation: Wagner, P., Ćwiek, A., & Samlowski, B. (2019). Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies. Journal of Phonetics, 76, 100911. http://olacwiek.github.io/files/wagner_2019_exploiting.pdf

The Acoustic Realization of Prosodic Prominence in Polish: Word-level Stress and Phrase-level Accent

Published in 9th International Conference on Speech Prosody 2018, 2018

This publication is based on my MA thesis, supervised by Petra Wagner and Zofia Malisz

Recommended citation: Ćwiek, A., & Wagner, P. (2018). The Acoustic Realization of Prosodic Prominence in Polish: Word-level Stress and Phrase-level Accent. 9th International Conference on Speech Prosody 2018, 922–926. http://olacwiek.github.io/files/cwiek_2018_acoustic.pdf

Acoustics and discourse function of two types of breathing signals

Published in Nordic Prosody: Proceedings of the XIIth Conference, Trondheim 2016, 2017

This publication is based on my BA thesis, supervised by Petra Wagner and Mattias Heldner

Recommended citation: Ćwiek, A., Włodarczak, M., Heldner, M., & Wagner, P. (2017). Acoustics and discourse function of two types of breathing signals. Nordic Prosody: Proceedings of the XIIth Conference, Trondheim 2016, 83–91. https://su.diva-portal.org/smash/get/diva2:1128916/FULLTEXT02.pdf

Beat it! – Gesture-based Prominence Annotation as a Window to Individual Prosody Processing Strategies

Published in 12. Phonetik und Phonologie im deutschsprachigen Raum, 2016

Recommended citation: Wagner, P., Ćwiek, A., & Samlowski, B. (2016). Beat it! – Gesture-based Prominence Annotation as a Window to Individual Prosody Processing Strategies. Tagungsband der 12. Tagung Phonetik und Phonologie im deutschsprachigen Raum, 211-214. https://pub.uni-bielefeld.de/download/2905407/2905410/wagner_cwiek_samlowski2016.pdf