Project outline
Modelling Human Behaviour, Attention and Listening Effort in Conversation.
Real conversation is more than a sequence of speech signals. When listening becomes difficult, people adapt: they may turn towards a talker, shift their visual attention, change when they respond, ask for repetition, participate less, or rely more strongly on context.
This PhD will investigate whether these behaviours can tell us not only what was heard, but whether a conversation is actually working for the listener. Using realistic multi-person conversational data, controlled experiments, and computational modelling, the project will study how acoustic conditions affect attention, listening effort and conversational behaviour.
The long-term aim is to develop models of the human behavioural layer that future intelligent audio systems could use to understand when a listener is coping, struggling or losing track.
Further project information
Modern speech technologies are increasingly good at enhancing, separating and recognising speech, but they are usually evaluated at the signal or word level. These measures tell us relatively little about whether a listener is able to follow a conversation, remain engaged and participate successfully. This project addresses that gap by asking whether human behaviour can provide richer evidence about when communication is working for the listener.
The PhD will investigate how people adapt their listening behaviour as acoustic conditions become more difficult.
Candidate behaviours include head movement and orientation, gaze and visual attention, response timing, turn-taking, participation and requests for repetition. The goal will be to develop computational models linking these behaviours to factors such as acoustic difficulty, listener attention, listening effort and conversational success.
The project will begin with realistic multi-person conversational datasets, particularly ECHI [1], which contains natural four-person conversations recorded in cafeteria-like noise with wearable and close-talk audio, head tracking, known spatial layouts and speaker-activity information. This provides an unusually rich platform for studying interactions between the acoustic scene, speaker activity, listener orientation and conversational dynamics. Follow-on ECHI-2 data and other suitable multimodal datasets may also be used.
Statistical and machine-learning methods will be used to discover and model behavioural patterns associated with listening difficulty, attention and effort. A key question will be whether behavioural signals provide information beyond acoustic measures alone, and whether models generalise across talkers, rooms, spatial layouts and listener profiles.
The strongest findings will then be tested in controlled experiments. Conversational scenes may be re-rendered using spatial audio, loudspeakers, screens or VR/AR-style presentation, allowing factors such as acoustic difficulty, spatial layout, visual access, enhancement strategy and listener hearing profile to be manipulated systematically. Behavioural measures such as head motion, gaze, response timing and turn-following can then be related to comprehension, task success, subjective effort or confidence.
The project is a collaboration with Mitsubishi Electric Research Laboratories (MERL). It is intended as fundamental, publishable research rather than development of a particular speech-enhancement product. Its longer-term relevance is to intelligent audio systems that can respond not only to the acoustic scene, but also to evidence about whether communication is working for the listener.
- [1] https://huggingface.co/datasets/CHiME9-ECHI/CHiME9-ECHI
Subject Areas
- Speech and Audio Processing
- Machine Learning
- Acoustics
- Hearing and Listening
- Multimodal Interaction
- Behavioural Modelling
Required technical skills/academic qualifications/experience
Applicants should have a 2.1 or higher in an undergraduate degree or a master’s degree in a relevant quantitative discipline such as computer science, engineering, acoustics, psychology, cognitive science or a related area.
Useful experience would include some of the following:
- programming for data analysis, ideally in Python or a similar language
- statistics, machine learning or quantitative experimental methods
- signal processing, speech/audio technology, or analysis of time-series data
- working with behavioural, multimodal or human-participant data
Experience in all of these areas is not expected. The project is interdisciplinary and will provide opportunities to develop new skills during the PhD. We particularly value curiosity, analytical thinking, and a willingness to work across disciplinary boundaries.
We particularly encourage applications from women and other groups under‑represented in research. All appointments will be made on merit.
Mode of attendance
This project is only available on a full-time basis.
Supervisors
Project Partners
This project is hosted at the University of Sheffield and supported by Mitsubishi Electric Research Laboratories (MERL).
