The agreement of phonetic transcriptions between paediatric speech and language therapists transcribing a disordered speech sample

Background

Speech and language therapists (SLTs) regularly use phonetic transcription to record and analyse typical and disordered speech. Phonetic transcription is highly demanding of auditory perceptual skills so researchers are sceptical about its accuracy and reliability. The literature describes how phonetic transcription is prone to auditory illusions and biases, such as a preference to transcribe speech sounds from the transcriber's own language. Few empirical research studies have calculated agreement amongst transcribers where a range of agreement scores have been reported (51%–97%). There is a consensus that agreement rates decrease as phonetic detail increases. Vowels and consonants are characterised by different perceptual features within the International Phonetic Alphabet (IPA) so they may differ in agreement rates, and thus far there is contradictory evidence as to whether vowels or consonants are more agreed upon. Transcription agreement studies to date have most commonly recruited phoneticians rather than SLTs so further research is warranted to determine transcription agreement amongst SLTs and its impact on clinical practice.

Aims

The study's primary aim was to calculate agreement scores from a group of English-speaking SLTs who transcribed disordered speech samples in an ecologically valid setting. The study also sought to contribute to the pre-existing contradictory evidence base regarding whether vowels or consonants may be more agreed upon by comparing their agreement scores. The researcher aimed to comment on SLTs’ use of diacritics and non-native speech symbols (symbols not included in the English phonetic inventory) in terms of their frequency and agreement of use. By analysing transcriptions, the study aimed to discuss the impact transcription variability has on speech sound error patterns and thus its impact on clinical decision-making such as diagnosis, choice of intervention and therapy targets.

Methods

Twelve paediatric SLTs were recruited via a convenience sample at two National Health Service trusts, two of whom were specialists in Speech Sound Disorders (SSDs). Participants transcribed 16 words from a video of a boy with disordered speech completing the Diagnostic Evaluation of Articulation and Phonology (DEAP, Dodd et al., 2006) from a telehealth appointment. The use of diacritics and non-English IPA symbols were manually analysed. A classic agreement score was calculated for each target word using a Python script, and then for vowels and consonants in isolation. Datasets were manually analysed to determine whether differences in transcription resulted in the identification of different speech sound error patterns. The researcher considered the implications this had within clinical practice.

Results

The average classic agreement score was 56.3%. Consonants were more agreed upon than vowels with agreement scores of 62.8% and 48.6%, respectively. Nine participants (75%) used diacritics (most commonly length marks) and eight participants (67%) transcribed non-native IPA symbols at least once in their datasets, but generally with low frequencies and agreements amongst their use. Transcriptions captured the occurrence of typical and atypical error patterns but only three error patterns, out of the 20 identified, were present in all 12 participants’ transcriptions.

Conclusions

The agreement score of 56.3% questions the accuracy and reliability of transcription amongst SLTs which is an essential skill of the profession. The findings highlight SLTs should be more cautious of interpreting vowels than consonants given lower agreement rates. The frequency of use of non-native symbols and diacritics was relatively low which could reflect a low accuracy of their use or reduced confidence in transcribing these. The study discussed how variations in transcriptions can impact phonological and phonetic analysis, which in turn can influence clinical decision-making such as diagnosing SSDs, selecting further diagnostic assessments and choosing therapy targets and interventions. The Royal College of Speech and Language Therapists-endorsed transcription guidelines (Child Speech Disorder Research Network, 2017) could be revised to convey realistic expectations of SLTs’ transcription skills, or SLTs should be offered more training to improve transcription skills to meet current expectations. Other suggestions to improve transcription accuracy are discussed such as via instrumental methods, yet these come with their own limitations such as practicality, costs and need for specialist training.

WHAT THIS PAPER ADDS

What is already known on this subject

• Phonetic transcription is highly demanding of human perceptual skills, and researchers are sceptical about its reliability. There are few empirical research studies calculating agreement amongst transcribers, and a range of agreement scores have been reported (51%−97%) dependent on the research conditions. Research mostly involves experienced transcribers (e.g., phoneticians) rather than speech and language therapists (SLTs), a profession expected to regularly use phonetic transcription to record and analyse typical and disordered speech.

What this study adds to existing knowledge

• A range of transcription agreement scores have been reported in previous studies, mainly comparing pairs or small groups of specialist transcribers rather than SLTs. This study provides an agreement score of 56.3% when a group of 12 SLTs transcribed a disordered speech sample in an ecologically valid setting (where speech samples were taken from a real-life speech sound assessment over a telehealth appointment using the Diagnostic Evaluation of Articulation and Phonology). The study found consonants are more agreed upon than vowels, adding to the contradictory evidence base. Unlike other studies, the researcher analysed transcriptions to identify error patterns to examine the impact that transcription variation has on clinical decision-making.

What are the potential or actual clinical implications of this work?

• The researcher questions whether SLTs are meeting the expectations of ‘accurate transcription’ as listed by the Royal College of Speech and Language Therapists (RCSLT) endorsed transcription guidelines (Child Speech Disorder Research Network, 2017) given the relatively low agreement score of 56.3%. The study also questions the reliability of the use of diacritics and non-English International Phonetic Alphabet (IPA) symbols and whether SLTs should be expected to use these due to perceptual limitations. Twenty phonological processes emerged from the datasets, only three of which were agreed upon across all 12 participants. The researcher therefore discusses how transcription variations could result in different diagnoses, therapy targets and interventions choices. The paper suggests more training is required to enhance transcription accuracy, and also considers the appropriateness of utilising instrumental methods whilst recognising its limitations such as feasibility, costs and specialist training needs.