Progress in research on orthorexia nervosa is still hampered by the use of the ORTO–15 and its modifications

Abstract

More than 20 years ago, it has been suggested that some people are so obsessed with eating healthily that this may even be considered a new type of disordered eating, so-called orthorexia nervosa (ON; Bratman, 1997; Bratman & Knight, 2000). Features of ON that most experts in the field agree upon are that persons (1) have a subjective definition of healthy or “pure” eating based on a dietary theory or set of beliefs; (2) have a strong preoccupation with their eating behavior and self-imposed rigid rules which include spending an excessive amount of time for planning, obtaining, preparing, or eating the food; and (3) that these behaviors lead to a nutritionally unbalanced diet that negatively affects physical and mental health (Donini et al., 2022).

Although this agreement exists, the concept of ON is still controversially discussed among scientists and clinicians. Specifically, while there are persons who are noticeably obsessed with eating healthily, it is unclear whether this can also result in a condition that is clinically relevant, that is, needs to be treated medically or psychotherapeutically, and if there is such a condition, if it is a condition that is distinct from established eating disorders. For example, it has been argued that such cases can rarely been found and that most cases who show clinically relevant orthorexic tendencies would also fulfill the diagnostic criteria for anorexia nervosa (Bhattacharya et al., 2022; Meule & Voderholzer, 2021). Because of this, it is all the more important that there are standardized and psychometrically sound assessment methods for orthorexic symptomatology to learn more about the prevalence and correlates of ON and its overlaps with and distinctiveness from other eating disorders.

Bratman and Knight (2000) described 10 tentative criteria for ON, which were intended as a self-test, that is, readers should decide for themselves whether each criterion applied to them or not. Although not developed as standardized, scientific measures, it has since been used in studies on ON as a self-report questionnaire with some researchers denoting it as Bratman's Orthorexia Test (BOT; Missbach et al., 2017). Yet, Missbach et al. (2017) also advised against using the BOT because of unknown psychometric properties and inconsistent use (e.g., different response scales and scoring) across studies.

The first standardized questionnaire that was developed for research purposes was the ORTO–15 (Donini et al., 2005). The ORTO–15 has since been used in numerous studies and has been translated in numerous languages (Oberle & Noebel, 2023). However, most of these studies consistently showed that the ORTO–15 is an unreliable and invalid measure. That is, internal reliability usually is unacceptable and one-factor models of all 15 items usually have poor model fit (e.g., Rogoza, 2019), indicating that no total score of all 15 items should be calculated, let alone interpreted. One reason for this may be its scoring procedure, which is quite peculiar: while all items are answered on a four-point scale with 1 = always, 2 = often, 3 = sometimes, 4 = never, and some items are then recoded inversely (i.e., 1 = 4, 2 = 3, 3 = 2, 4 = 1), there are two items that are recoded as 2 = always, 4 = often, 3 = sometimes, 1 = never. Indeed, one study suggested that the psychometric properties of the ORTO–15 are better when the scale is scored differently (i.e., when only two of all items are recoded; Meule et al., 2020). Another reason that has been discussed in the literature is that some items are worded unclearly and do not seem to be generic for orthorexic symptomatology (e.g., Missbach et al., 2015).

To circumvent these issues, researchers have removed items that had poor psychometric properties. However, this lead to a plethora of different modified versions of the ORTO–15 consisting of 7–12 items and, most recently, the six-item ORTO–R (Rogoza & Donini, 2021). Unfortunately, this lead to a confusing literature on ON, findings of which are not comparable. For example, Toti et al. (2022) used the ORTO–15 and additionally analyzed a 12-, 11-, 9-, 7-, and 6-item version all in one study, obviously being unsure which version should be chosen. Lakritz et al. (2022) used the ORTO–15 but did not use the original scoring procedure proposed by Donini et al. (2005) but scored items based on the factor analysis reported by Meule et al. (2020). These inconsistencies are particularly crucial when studies provide prevalence estimates of ON, which have been found to be unrealistically high and very different across studies that used the ORTO–15 and its derivatives (e.g., Aiello et al., 2022; Bergonzi & Massarollo, 2022; Kujawowicz et al., 2022). Thus, even when using short forms of the ORTO–15 or different scoring procedures leads to better psychometric properties in terms of internal reliability and factorial validity, this does not mean that this leads to an improved, growing knowledge about ON as findings cannot be accumulated across different studies.

Revisions of the ORTO–15 also do not solve the problem of some items lacking content validity. For example, even the most recent (and briefest) revision of the ORTO–15—the ORTO–R (Rogoza & Donini, 2021)—still includes an item “Do you believe that strict consuming only of healthy food may improve your appearance?” for which higher endorsement is interpreted as higher orthorexic symptomatology. However, in a recent study among 47 experts in the field, 93% agreed with the statement that—in contrast to anorexia nervosa—appearance concerns are not central in ON (Donini et al., 2022), so endorsing this item actually speaks against the presence of ON.

Because of all these issues, Missbach et al. (2017) have argued years ago “that translating and adapting the already existing measures (BOT, ORTO–15) are not contributing to our knowledge of ON producing yet additional findings that are of mediocre quality” (p. 523) and that new tools to assess ON should be developed. This has been done indeed as there are several alternatives to the ORTO–15 such as the Eating Habits Questionnaire (Gleaves et al., 2013), the Düsseldorf Orthorexia Scale (Barthels et al., 2015), the Teruel Orthorexia Scale (Barrada & Roncero, 2018), the Orthorexia Nervosa Inventory (Oberle et al., 2021), or the French Orthorexia Scale (Dajon et al., 2021). While the ORTO–15 is the oldest of all questionnaires, it is—understandably—the questionnaire that has been used most often overall (cf. Oberle & Noebel, 2023). However, based on the recommendations against using the ORTO–15 that have been articulated clearly for years (Meule et al., 2020; Missbach et al., 2015, 2017), it may be that the use of other questionnaires have now superseded the use of the ORTO–15. Unfortunately, this does not seem to be the case. For example, in 71 studies that were published in 2022 in Web of Science-listed journals (Figure 1), the ORTO–15 or its modifications were used in more than half of these studies (Figure 2; the list of all studies that this analysis is based on can be accessed at https://osf.io/mfud6).

The ORTO–15 is an unreliable and invalid measure for orthorexic symptomatology. Revisions of this scale by removing items or changing scoring procedures do not solve these problems and do not contribute to consistent literature that increases knowledge about ON. As the ORTO–15 and its modifications are still the most often used measures in studies on ON, it seems necessary to reiterate the recommendations made earlier by others: neither the ORTO–15 nor its other versions should be used in studies on ON. Instead, alternative measures that have been developed in the past 10 years should be preferred.

The author declares no conflict of interest.