(PDF) A Preregistered Validation Study of Methods Measuring Analytic and Holistic Cognitive Styles: What do We Actually Measure

VALIDATION OF METHODS MEASURING COGNITIVE STYLES A Preregistered Validation Study of Methods Measuring Analytic and Holistic Cognitive Styles: What do We Actually Measure and How Well? Lacko, D.,1* Prošek, T.,1 Čeněk, J.,2, 3 Helísková, M.,1† Ugwitz, P.,3† Svoboda, V.,1& Počaji, P.,1& Vais, M.,1& Halířová, H.,1& Juřík, V.,1 & Šašinka, Č.3 1 Department of Psychology, Faculty of Arts, Masaryk University, Brno, Czech Republic 2 Department of Social Studies, Faculty of Regional Development and International Studies, Mendel University in Brno, Brno, Czech Republic 3 Department of Information and Library Studies, Faculty of Arts, Masaryk University, Brno, Czech Republic * : Corresponding author: Mgr. David Lacko,

† : These authors equally contributed to this work & : These authors also contributed equally to this work Acknowledgments This publication was supported by Masaryk University (MUNI/A/1323/2020: “Validation of the methods for analytic/holistic cognitive style”) and by the Czech Science Foundation (GC19- 09265J: “The Influence of Socio-Cultural Factors and Writing Systems on the Perception and Cognition of Complex Visual Stimuli”). We would like to thank the experimental humanities laboratory at Masaryk University (HUMELab) for providing us with the Hypothesis software, Dr Elizabeth R. Peterson for providing us with a licence and materials for E-CSA-WA, Dr Dylan Molenaar for helping us solve an issue with a function in the diffIRT R package, and Bc. Bianka Masariková, Bc. Kamila Vlčková and Mgr. Nicol Dostálová for data collection during pilot testing. Declarations Funding: This publication was supported by Masaryk University (MUNI/A/1323/2020) and by the Czech Science Foundation (GC19-09265J). Conflicts of interest/Competing interests: The authors have no financial or proprietary interests in any material discussed in this article. 1 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Ethics approval: The study has been approved by the Masaryk University Ethical Board (EKV-2020-118). Consent to participate: All participants signed online informed consent. Authors' contributions: • DL: research design, methods design, methods translation, data analysis, paper writing, pre-registration, project administration • TP: research design, data analysis, pre-registration • JČ: methods design, method translation, paper revision • MH: methods construction • PU: methods construction • VS: data collection, method translation, sampling • PP: data collection, method translation, pilot testing • MV: data collection, method translation, sampling • HH: data collection, method translation, pilot testing • VJ: project administration, paper revision • ČŠ: technical support, methods construction, paper revision Open Practices: The data, materials and R syntaxes are available at https://osf.io/7ezax/ and the study was preregistered (see https://osf.io/w483c). URL links in the manuscript were anonymized. The study has been published before peer-review as a preprint (https://osf.io/w483c). 2 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Abstract Cognitive styles are commonly studied constructs in cognitive psychology. It can be argued that measurement of these styles in the past had significant shortcomings in validity and reliability. The theory of analytic and holistic cognitive styles followed from traditional research of cognitive styles and attempted to overcome these shortcomings. Unfortunately, the psychometric properties of its measurement methods in many cases were debatable or not reported. New statistical approaches, such as analysis of reaction times, have been reported in the recent literature but remain overlooked by current research on analytic and holistic cognitive styles. The aim of this pre-registered study was to verify the psychometric properties (i.e., factor structure, split-half reliability, test-retest reliability, discriminant validity with intelligence and personality, and divergent, concurrent and predictive validity) of several methods routinely applied in the field. We developed/adapted six methods, and selected several types frequently applied in cognitive style research: self-report questionnaires, methods based on rod-and-frame test principles, embedded figures, and methods based on hierarchical figures. The analysis was conducted on 392 Czech participants, with two data collection waves. The results indicate that the use of self-report questionnaires and methods based on the rod-and- frame principle may be unreliable, demonstrating unsatisfactory factor structure and no absence of association with intelligence. The use of embedded and hierarchical figures is recommended. Because the concurrent and divergent validity of the methods did not correspond with the original two-dimensional theory and predictive validity (cross-cultural differences) was not examined, additional research is needed for better understanding of the construct. Keywords: cognitive styles, analytic and holistic cognitive style, global and local processing, reaction times, validity, reliability 3 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Introduction In the past, the term cognitive style often referred to the individual approach of processing and acquiring information (Ausburn & Ausburn, 1978). More precisely, it refers to stable attitudes, preferences, or habitual strategies which determine an individual’s mode of perceiving, remembering, thinking, learning, and problem-solving (Messick, 1976; Witkin et al., 1977) and allow adaptation to the external world developing through interaction with the surrounding environment (Kozhevnikov et al., 2014; Kozhevnikov, 2007). It also represents the missing link between cognition and personality (Ridding & Rayner, 1998; Sternberg & Grigorenko, 1997). The main characteristics of the cognitive style are therefore the relative stability of the construct (at least from the short-time perspective) and the absence of associations with cognitive abilities and personality (Ausburn & Ausburn, 1978; Messick, 1976; 1984; Witkin et al., 1977). The first examination of cognitive styles can be traced to the works of the early pioneers of psychological science, such as James (1890), Galton (1883) and Jung (1923). From the first use of the term cognitive style 85 years ago by Allport (1937), dozens of models of cognitive style with hundreds of dimensions were created up to the 1970s (for a review, see Armstrong et al., 2011; Ausburn & Ausburn, 1978; Cassidy, 2004; Hayes & Allinson, 1994; Kozhevnikov et al., 2014; Kozhevnikov, 2007; Riding, 1997; Riding & Cheema, 1991; Sternberg & Grigorenko, 1997; Tiedemann, 1989; Messick, 1974; Kogan & Saarni, 1990; Zhang et al., 2012). In fact, such a huge ambivalence in the terminology1 and consequent frivolous application of the concept led to a crisis of scientific confidence in the final decades of the twentieth century (Kogan & Saarni, 1990; Kozhevnikov et al., 2014; Kozhevnikov, 2007; Zhang et al., 2012). Several scholars attempted to unify the spectrum of models (e.g., Allinson & Hayes, 1996; Curry, 1983; Hayes & Allinson, 1994; Riding, 1991; Riding & Cheema, 1991), create new multidimensional models of cognitive style (e.g., Stenberg, 1997; Sternberg & Grigorenko, 1997; Sternberg & Zhang, 2005;), incorporate cognitive metastyles, i.e., styles operating on a metacognitive level into existing models, such as mobility-fixity or reflective- impulsive metastyles (e.g., Niaz, 1983; Keller & Ripoll, 2001) or proposed complex multilevel hierarchical models of cognitive style (e.g., Miller, 1987; 1991; Nosal, 1990; Kozhevnikov et al., 2014). 1 Cognitive styles are often used as a synonym for thinking, intellectual, processing or perceptual styles. Although they should slightly differ from the learning styles (used in education), decision-making styles (used in business) and personal styles (used in psychotherapy), many scholars use cognitive and learning styles in the same manner. 4 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Perhaps surprisingly, the majority of these works coincide in that one of the fundamental and superordinate orthogonal dimensions is the wholistic-analytic dimension we investigate in this article. According to these models, persons with a stronger preference for a wholistic cognitive style tend to process information as an integrated whole, whereas persons with strong preference for an analytic cognitive style proceed in discrete parts of that whole (Graff, 2003). In the prior research, these two types of cognitive styles were usually perceived as two polar ends of one continuum (i.e,. unidimensional structure). Old controversy The most representative and frequently applied theory from the wholistic-analytic cognitive style family is the field independence-dependence cognitive style (FDI), based on Witkin’s theory of psychological differentiation (Witkin et al., 1977) and cognitive restructuring (Witkin & Goodenough, 1981). However, similarly to other theories of cognitive style, FDI has often received criticism (for a review, see Messick, 1984; Tiedemann, 1989; Moran, 1985; Álvarez- Montero et al., 2018; Coffield et al., 2004; Curry et al., 1990; Cassidy, 2004; Kozhevnikov, 2007). Besides problems stemming from vague conceptualization, other theoretical shortcomings and a resulting range of frequently unrelated and unconnected dimensions (Kozhevnikov et al., 2014; Riding & Cheema, 1991; Sternberg & Grigorenko, 1997), the theory’s main shortcoming is in insufficient evidence for its convergent validity. Throughout the theory’s history, two generations of instruments for measuring cognitive style have been introduced: maximum performance tests (1st generation) and self- report questionnaires (2nd generation; for a review of various methods, see Armstrong et al., 2011; Álvarez-Montero et al., 2018; Coffield et al., 2004; Cools et al., 2013; Peterson et al., 2009; Kozhevnikov, 2007). The validity of the first-generation methods was questionable due to undesirable associations with cognitive abilities, whereas the second-generation methods were hampered by associations with personality (Cueno et al., 2018). According to some critics, the first generation of methods assessed cognitive ability rather than cognitive style (e.g., Evans et al., 1013; Rittschof, 2010; Sternberg & Grigorenko, 1997; Tiedemann, 1989; Kozhevnikov, 2007) since unsuitably high associations of cognitive style with general intelligence (e.g., Cooperman, 1980; Cuneo et al., 2018; Flexer & Roberge, 1980; McKenna, 1984; Riding & Pearson, 1994; Widiger et al., 1980; Weisz et al., 1975; Tinajero & Páramo, 1997; Vernon, 1972; Rémy & Gilles, 2014), spatial ability (e.g., Boccia et al., 2016; MacLeod et al., 1986; Zhang et al, 2004), working memory (e.g., Bahar & Hansell, 5 VALIDATION OF METHODS MEASURING COGNITIVE STYLES 2000; Miyake et al., 2011), attention (e.g., Fitzgibbons et al., 1965; Guisande et al., 2007), and academic achievement (Páramo & Tinajero, 1990; Tinajero & Páramo, 1997) have been found in previous studies. Concerning second generation methods, some authors concluded weak or moderate associations with personality yet did not interpret this as a violation of discriminant validity (e.g., Busato et al., 1999; Zhang, 2002; 2006). Others discovered high associations with personality traits questioning the validity (e.g., Cueno et al., 2018; Furnham, 1992; Von Wittich & Antonakis, 2011). The discriminant validity of wholistic-analytic based cognitive styles is therefore unclear. Self-report methods are also often unrelated to performance-based measures of cognitive style, an observation which challenges their convergent validity (e.g., Bergman & Engelbrektson, 1973; Cueno et al., 2018; Zhang, 2004). Another crucial aspect of the validity of cognitive styles is stability of the construct. Cognitive styles may shape themselves throughout a person’s life (Goodenough & Witkin, 1977), be affected by various socio-cultural factors (Allinson & Hayes, 1996; Hayes & Allinson, 1998; Kozhevnikov et al., 2014; Witkin & Berry, 1975) and be generally considered dynamic (Zhang, 2013) and task dependent (Kozhevnikov et al., 2014). Nevertheless, they should remain relatively stable in the short to medium-term. The contemporary body of knowledge, however, remains inconclusive since some studies have found that cognitive styles are stable (e.g., Lis & Powers, 1979; Kepner & Neimark, 1984; Moran, 1983) and others have found they may change significantly, for instance after specific training (Goldstein & Chance, 1965; Ludwig & Lachnit, 2004). Finally, some past studies criticized the generally poor or unknown psychometric properties of instruments which measure cognitive style (e.g., Álvarez-Montero et al., 2018; Coffield et al., 2004; Curry, 1990; Tiedermann, 1989) and highlighted the absence of combinations of mixed-methods applied in cognitive style assessments (Bendall et al., 2016). Current perspective Over the last century, the term cognitive style was one of the most intensively studied constructs in cognitive sciences, and despite some research receiving much criticism, it has remained the dominant theory in several psychological research fields until this day (Evans & Waring, 2012). Specifically, the wholistic-analytic dimension has lately gained a dominant position in cross-cultural research (Choi et al., 2007). Strongly inspired by the cross-cultural 6 VALIDATION OF METHODS MEASURING COGNITIVE STYLES differences between “modern” and “traditional” cultures found in Witkin’s FDI (see Witkin, 1979; Witkin & Berry, 1975) and between Western and Eastern countries at similar level of technological development (see Nisbett, 2003), Nisbett et al. proposed the new and currently predominant theory of analytic and holistic cognitive style2 twenty years ago (AH; for a review, see Ishii, 2012; Nisbett et al., 2001; Nisbett & Masuda, 2003; Nisbett & Miyamoto, 2005). Many researchers across the globe found that Eastern countries differ from their Western counterparts in their perception and cognition (Easterners tend to be more holistic, Westerners more analytic), for instance in the cognitive processes of object categorization (e.g., Chiu, 1972; Ji et a., 2004; Norenzayan et al., 2002), causal attribution (e.g., Choi & Nisbett, 1988; Choi et al., 1999; Masuda & Kitayama, 2004; Miyamoto & Kitayama, 2002; Morris & Peng, 1994), reasoning about contradictions (e.g., Peng & Nisbett, 1999), visual attention to focal objects and background (e.g., Masuda & Nisbett, 2001; Nisbet & Masuda, 2003, Chua, Boland et al., 2005), object-background differentiation (e.g., Kühnen et al., 2001), change detection (e.g., Masuda & Nisbett, 2006; Masuda et al., 2016), scene memory (e.g., Mickley Steinmetz et al., 2017), dependence on visual external reference frameworks (e.g., Ji et al., 2000; Kitayama et al., 2003), or global and local distribution of attention (e.g., McKone et al., 2010), etc. To analyse (not only) cross-cultural differences in such a diverse spectrum of cognitive processes, multiple tasks were introduced ranging from self-report inventories and performance-based measures based on older wholistic-analytic cognitive style methods, through new computer-based instruments, to the usage of modern technologies such as virtual reality or eye-tracking. In contrast to the older methods, these methods usually incorporate specific tasks for both analytic and holistic cognition and thus demonstrate the desirable shift from unidimensional to a two-dimensional structure. We identified four main clusters of methods which attempt to eliminate the shortcomings of measurement summarized in the previous section. 2 Unfortunately, confusing terminology can be observed even with these terms. The term wholistic is used more broadly and refers to holistic cognition in general, whereas holistic is term used purely in a Nisbett’s cross-cultural theory. In this article, we keep this distinction. Concerning the term analytic, it is necessary to note that similar terminology is also currently in use in research of intuitive–analytic cognitive styles assessed, for example, through the cognitive reflection test. This analytic cognitive style, however, does not correspond to the analytic cognitive style described above. 7 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Self-report questionnaires Despite self-report questionnaires being relatively common in the assessment of other cognitive styles, their use specifically for AH measurement remains relatively scarce. Yet, the analytic and holistic styles are at least partially represented, for example, in the analytic-intuitive dimension of the Cognitive Styles Index (CSI; Allinson & Hayes, 1996), the global-local dimensions of the Thinking Style Inventory (TSI; Sternberg & Wagner 1991) and its revisions (Sternberg et al., 2003; 2007), the analytic-holistic (originally left-right brain dominance) dimensions of the Styles of Learning and Thinking (SOLAT; Torrance et al., 1998), in knowing, planning, and creating the dimensions of the Cognitive Style Indicator (CoSI; Cools & Van den Broeck, 2007), the linear and nonlinear decision-making dimensions of Linear-Nonlinear Thinking Style Profile (LNTSP; Vance et al., 2007), or the global bias dimension of the Sussex Cognitive Styles Questionnaire (SCSQ; Mealor et al., 2016). Nevertheless, only a few inventories have been created exclusively for the measurement of AH as defined above. The best examples are the two-dimensional Holism Scale (HS; Choi et al., 2003), four-dimensional Analysis-Holism Scale (AHS; Choi et al., 2007; Koo et al., 2018) and the recently published four-dimensional Holistic Cognition Scale (HCS; Lux et al., 2021). Only the AHS will be described in more detail here since it is the most commonly used scale in the field. It contains four subscales (locus of attention, causal theory, perception of change and attitude toward contradictions) and twenty-four 7-point Likert items (1 = strongly disagree, 7 = strongly agree) with six items per subscale. This instrument showed discriminant validity with individualism/collectivism and independent/interdependent self-construal scales and concurrent validity in terms of weak associations with the categorization task proposed by Norenzayan et al. (2002, Study 2). In the above-mentioned study, the confirmatory factor analysis (CFA) also suggested a relatively good model fit of a four-factor structure, χ2 (249) = 1108.54, χ2/df = 4.45, GFI = .83. However, its internal reliability was not entirely satisfactory in all subscales (Study 1: αcausality = .71, αcontradictions = .69, αchange = .58 and αattention = .56; Study 2: αcausality = .76, αcontradictions = .71, αchange = .71 and αattention = .67), which challenges further research. 8 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Performance-based measures based on the wholistic-analytic family This cluster of methods stems from the early experiments of Witkin and Asch on space orientation (Asch & Witkin, 1948a; 1948b; Witkin, 1949; Witkin & Asch, 1948a; 1948b) and the FDI theory (i.e., psychological differentiation and cognitive restructuring). Based on their previous work, they formulated the Rod and Frame Test (RFT; Witkin & Goodenough, 1981; Witkin & Goodenough, 1976) and its modifications Body Adjustment Test (BAT) and Room Adjustment Test (RAT). In the RFT method, participants are asked to set a rod, embedded in a square, to the subjective vertical position, regardless of the surrounding frame. The method is partially based on Wertheimer’s (1912) tilted-mirror experiment and is derived from the little-known Gestalt principle called frame of reference (Duncker, 1929; Rock, 1992). Through recent technological progress, it is now also possible to administer this test in virtual reality (e.g., Reger et al., 2003) which greatly simplifies the entire testing procedure. The Framed-Lined Test (FLT; Kitayama et al., 2003) is a pen-and-paper method based on the RFT. Compared to the RFT, where the ideal solution strategy is always field independence, the FLT takes into account both the absolute and relative fields of reference in two individual subtests. In applying this measure, participants are exposed to the original stimulus (square with a vertical line). Their goal is to draw a line of exactly the same absolute length as the original, regardless of the size of the square (analytic task) or line that proportionally corresponds to the proportion of the line and side of the square of the original stimulus (holistic task). Despite its 18-year history, the psychometric properties of the FLT are unknown. Only two studies have verified its reliability via internal consistency, which is an inappropriate reliability indicator for performance-based measures (the αs < .70; cf. Kitayama et al., 2009; Na et al., 2019). The FDI is commonly assessed via the Embedded Figures Test (EFT; Witkin & Goodenough, 1976) and its modified versions the Group Embedded Figures Test (GEFT) and Children’s Embedded Figures Test (CEFT). These methods are based on Gottschaldt’s embedded figures (Gottschaldt, 1926), which are complex figures composed of simple figures. Participants are instructed to spot a simple form within a more complex figure. The psychological principle beyond these figures lies in the Gestalt principles of figure - ground organization, especially the laws of proximity, similarity, good continuation, closure and mirror symmetry (Koffka, 1935; Wagemans et al., 2012; Wertheimer, 1923). Cognitive Style Analysis (CSA; Riding, 1991; Riding & Cheema, 1991) is a commonly used instrument based on the 9 VALIDATION OF METHODS MEASURING COGNITIVE STYLES EFT family of methods. The CSA contains verbal-imagery and wholistic-analytic cognitive style dimensions, but only the latter is described here in more detail. Similarly to the FLT, the CSA enhances the reductionist unidimensional approach of the EFT and incorporates two subtests of AH cognitive style measurement. The CSA (unlike EFT) also does not correlate with intelligence (Riding & Pearson, 1994), personality (Riding & Wigley, 1997) or academic achievement (Peterson & Meissel, 2015), although some serious issues with its test-retest reliability have been revealed (e.g., Cook, 20018; Parkinson et al., 2004; Peterson et al., 2003; Rezaei & Katz, 2004). Hence, the Extended Cognitive Style Analysis – Wholistic/Analytic (E-CSA-W/A) was proposed by Peterson et al. (2003; 2005). It contains 80 items (40 analytic and 40 holistic). In a holistic subtest, participants are presented with two complex figures and their goal is to identify whether these figures are identical. In an analytic subtest, participants are exposed to one simple and one complex figure, and their goal is to reveal whether the complex figure contains the simple figure. Participants respond by pressing one of two keyboard buttons. E- CSA-W/A showed sufficient split-half reliability, parallel forms reliability and test-retest reliability (Peterson et al., 2003; Aslan et al., 2018) and a lack of association with mathematical performance (Pitta-Pantazi & Christou, 2009), intelligence and personality (Peterson et al., 2005). Performance-based measures based on Global/Local families The following group of methods is based on global and local processing of Navon’s hierarchical figures (Navon, 1977; 1981), i.e., a large figure (global level) composed of small figures (local level). These figures were created with respect to the Gestalt principles of grouping, especially in proximity and continuity (Wertheimer, 1923). Generally, human perception demonstrates a significant global advantage (faster identification of the global figure than the local; for a review, see Kimchi, 1992; Navon, 2003). Compared to the embedded figures used in E-CSA-WA, hierarchical figures are self-contained and not nested in the surrounding context. These methods were not originally proposed for AH measurement. In fact, probably the first use of Navon figures in the context of cognitive styles can be traced to 2006 (Peterson & Deary, 2006) and 2008 and 2010, when the first cross-cultural comparisons were conducted (Davidoff et al., 2008; McKone et al., 2010). From the perspective of analytic and holistic 10 VALIDATION OF METHODS MEASURING COGNITIVE STYLES cognitive styles, local processing corresponds to the analytic cognitive style and global processing to the holistic cognitive style (Peterson & Deary, 2006). Today, different versions of Navon figures are frequently used to estimate a person’s global and local processing, for example, Wholistic Analytic Inspection Time (WA-IT; Peterson & Deary, 2006). These modifications differ at two levels: 1) the type of figure and 2) the aim of the task. The type of figure can be either verbal or non-verbal. Previous studies have included verbal figures such as instance numbers (e.g., Shedden & Reid, 2001; Van Fleet et al., 2011), logograms (e.g., Kiyokawa et al., 2012) and Latin script (e.g., McKone et al., 2010; Navon, 1977; Van Fleet et al., 2011). Examples of non-verbal figure types are geometric shapes (Davidoff et al., 2008; De Fockert & Cooper, 2014; Oishi et al., 2014; Kimchi & Palmer, 1982), abstract drawings (Poirel et al., 2006), fuzzy objects (Van Lier, 1999) and specific objects such as faces (Rozin et al., 2016). Both verbal and non-verbal types can be mutually combined (e.g., a large letter composed of smaller geometric shapes). Global and local features can also be combined; each figure may be congruent (local features are the same as global features) or incongruent (local features differ from global features; Navon, 1977). Distinction according to the aim of the task is less complicated since the task’s aim might be to find the correct answer (i.e., Navon Search Task) or to select an answer which is as similar as possible to the original figure (Navon Similarity Matching Task; Caparos et al., 2015). The literature usually describes the following effects (Gerlach & Starrfelt, 2018): global preference effect (people identify global features more quickly than local features), interference effect (people identify congruent stimuli more quickly than incongruent stimuli) and inter-level interference effect (the interference effect is emphasized during identification of local features). Evidence of the psychometric properties of Navon figures is rather mixed. Some studies suggested that these methods have satisfactory test-retest reliability (Dale & Arnell, 2014; Dale & Arnell, 2013) and split-half reliability (Gerlach & Poirel, 2018; Gerlach & Starrfelt, 2018), while others directly questioned their validity and reliability (Chamberlain et al., 2017; Hedge et al., 2017). Hierarchical figures are also not associated with general intelligence (with the exception of local accuracy, Mine & Szczerbinski, 2009). Other methods Methods other than those mentioned above (i.e., self-report questionnaires, wholistic-analytic and global-local families of methods) have also been used within the AH paradigm. The 11 VALIDATION OF METHODS MEASURING COGNITIVE STYLES literature contains several studies which focus on context sensitivity. The theoretical baseline is the assumption that cultures vary in their context sensitivity, in which holistic perceivers focus relatively less on the focal object (visually salient part of the stimulus) and relatively more on the background than their analytic counterparts. Quite often, these studies applied complex visual stimuli of artificially created or natural visual scenes as stimulus material. The scenes were presented to participants for a certain time (several seconds), and the researcher was then interested in the relative proportion of attention focused on one of the two basic segments of the scene: the focal object(s) or the background (Masuda & Nisbett, 2001, Study 2). These studies commonly employed eye- tracking methodology: the focus of attention was derived from dwell time and the number of fixations on the two specific regions, defined as ROIs (Regions of Interest). Typically, the participants (freely) viewed each scene while their eye-movements were measured (Rayner et al., 2009). The free-viewing task was sometimes combined with recording the participant’s reaction to the picture (Čeněk et al., 2020) or the participant’s recognition of the focal object with background (previously seen or unseen) manipulation (Chua, Boland et al., 2005; Duan et al., 2016; Evans et al., 2009). Some studies employed similar methodology using dynamic scenes instead (Masuda & Nisbett, 2006, Study 1). Other studies combined the perception of scenes with recording and analysis of the participant’s narratives (Chua, Leu et al., 2005; Senzaki et al. 2013; Waxman et al., 2016) or visual searches (Alotaibi et al., 2017). Several studies used a combination of multiple methods (e.g., Jurkat et al., 2020; Mavridis et al., 2020; Rayner et al., 2007). A similar group of methods stems from the change blindness task (Masuda & Nisbett, 2006), which is based on the flicker paradigm (Rensink et al., 1999). Again, scenes are partitioned into focal object and background segments. In each trial a change in the scene occurs in one of the segments. The speed of detection of the change is measured. Holistic perceivers are expected to detect changes in the background relatively more quickly than analytic perceivers (Boduroglu et al., 2009; Choi et al., 2015; Masuda & Nisbett, 2006; Masuda et al., 2016). Another set of tasks is based on the categorization (triad) task (Chiu, 1972; Norenzayan et al., 2002), which tests whether participants categorize objects based on the thematic relationship or focal attributes. This principle was adapted, for example, to categorize multivariate symbols on cartographic maps (Lacko, Šašinka et al., 2020). Na et al. (2019) described several other tasks not as commonly employed as the methods described above, namely the inclusion task (Choi et al., 2003), proverb task (Peng & 12 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Nisbett, 1999), change task (Ji et al., 2001), outside-in task (Cohen & Gunz, 2002), causal attribution task (Kitayama et al., 2006) and twenty statement task (Kuhn & McPartland, 1954). Generally, these tasks do not represent any formally standardized methods, and scholars usually use their own ad-hoc stimuli and interpret the validity of tasks according to the patterns of results expected in certain cultures. Their psychometric properties are therefore unknown. The only exception is research by Na et al. (2019), who found that the internal consistency of these tasks varies significantly (αs ranged from .24 to .96) and that the test-retest reliability of four of these methods (outside-in task, change blindness task, inclusion task and twenty statement task) is moderate at best (rs ranged from .47 to .70). It should be noted that this review of methods is by far not exhaustive and neglects studies which are further away from the main focus of this article (e.g., studies on reasoning, neuropsychological studies). New Challenges Uncertain or unknown psychometric properties Despite the relatively large body of literature which describes various AH measurement methods, evidence of psychometric properties in most of them remains unknown or ambiguous. Studies which applied self-report questionnaires and FLT did not report any evidence of stability in construct, and Navon-based methods reported unclear results. Even studies which provided some evidence of validity or reliability of measures can be disputed (e.g., E-CSA-WA instrument), mainly because of statistical and methodological reasons (inappropriate analyses, sample size or composition). The most serious problem, however, is the lack of concurrent validity in AH methods. As with FDI, AH measurement shows inconsistent associations between methods. Specifically, the FLT only weakly associated with the change blindness (r = .19) and causal attribution (r = .22) tasks, and no associations were detected with the other nine AH measurement methods (Na et al., 2019; see also Na et al., 2010). Navon hierarchical figures also barely associated with Gottschaldt embedded figures (e.g., Chamberlain et al., 2017; Huygelier et al., 2018; Milne & Szczerbinski, 2009; Peterson & Deary, 2006), except for one study, which found a strong association (cf. Poirel et al., 2008). Some studies reported low or no correlations between various modifications of Navon figures (Dale, & Arnell, 2013), thereby also challenging the convergent validity of this method. 13 VALIDATION OF METHODS MEASURING COGNITIVE STYLES A lack of association was also observed between self-report questionnaires and performance- based measures of AH (e.g., β = .21 for Choi et al., 2007, study 5; r = .05 for Sadler-Smith et al., 2000). Hence, scholars in the field appear to be using methods which might not even be related to each other but are interpreting their findings as differences under the same attributes (i.e., analytic and holistic cognitive styles). The final important factor to consider concerns self-report questionnaires. Despite the validation studies of AHS and HSC using desirable structural equation modeling techniques to verify its factor structure, no self-reporting has provided any evidence of cross-cultural comparability, such as scalar measurement invariance (see Boer et al., 2018; Chen, 2008; Fischer & Karl, 2019; Milfont & Fischer, 2010). As discussed above, this criterion is sometimes not even met in well-established cross-cultural constructs such as the individualism- collectivism dimension (Lacko et al., 2022). Beyond the East-West dichotomy The AH represents one of the flagships of current research on cross-cultural differences in perception and cognition. It is therefore not surprising that scholars often deduce the validity of AH measurement based on obtained cross-cultural research evidence while assuming that Westerners are more analytic and less holistic than their Eastern counterparts (Nisbett et al., 2001). This approach in estimating criterion-related validity is highlighted by the statement that AH cross-cultural differences not being reducible to an individual level (see Na et al., 2010). This main assumption of difference between East and West, however, is not supported in several recent studies, which showed contradictory or ambiguous results or results with small effect sizes in various tasks and methods (e.g., Čeněk et al., 2020; Evans et al., 2009; Hakim et al., 2017; Lacko, Šašinka et al., 2020; Rayner et al., 2007; 2009; Stachoň et al., 2008a; 2008b; von Mühlenen et al., 2018). In this regard, it is also necessary to highlight that the main body of AH research tends to overlook countries beyond the East-West dichotomy (typically comparing populations from Japan or China and the USA or Western Europe). Since cross- cultural comparisons of countries in their level of AH are often the only evidence of the validity of AH instruments, the currently mixed results might be interpreted against the validity of the instruments which were used. However, unclear results, especially from countries outside the typical East-West dichotomy, indicate that the AH theory should perhaps be revisited to correspond to the empirical findings from a large number of different cultures. 14 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Distribution of reaction times and their relationship to cognitive processes Reaction times (RTs) have held a prominent position in experimental and cognitive psychological research of analytic and holistic cognitive styles. To analyse RTs, however, scientists have typically used the statistical techniques they were the most familiar with, such as analysis of variance on the sample mean (Van Zandt, 2002), although this was found unsuitable in many of its applications (Whelan, 2008; for exceptions see e.g., Haaf & Rouder, 2017; Schramm & Rouder, 2019) because RTs are usually not identically and independently distributed (iid) as a result of trial-by-trial sequential effects. More importantly, in a majority of cases, RTs are not normally distributed (Gaussian) but rise rapidly on the left and have a long positive tail on the right (Van Zandt, 2002; Whelan, 2008). This feature of RTs may have produced misleading or potentially contradictory results (Lo & Andrew, 2015). Even though it is common practice to ensure the normal distribution of RTs through the use of various transformations (mostly logarithmic, reciprocal or inverse Gaussian), these transformation techniques are not recommended under most circumstances since they change the unit of measurement (Rousselet & Wilcox, 2020) and statistically do not yield any core improvements (e.g., increase of statistical power; Schramm & Rouder, 2019). Similarly, even the use of robust or non-parametric statistical procedures or linear mixed-effect models with box-cox transformation should not be considered adequate (see Lo & Andrew, 2015; Schramm & Rouder, 2019; Whelan, 2008). One of the major challenges of current AH research is therefore the incorporation of suitable methods of statistical analysis of RTs. These methods would reflect not only the specific RT distribution but also its true intrinsic relationship to the cognitive construct of AH per se. Several procedures can be identified in the current literature (for a review, see De Boeck & Jeon, 2019; Kyllonen & Zu, 2016). In this section, we briefly describe the three most commonly applied approaches to RT analysis: distribution analysis, joint models and process models. The first approach attempts to overcome the traditional and rather reductionist method of simply calculating measures of central tendency such as mean and median and aims to capture the much greater amount of information present in the full RT distribution (Faulkenberry, 2017; Voss et al., 2013). For these purposes, several distributions have been proposed to fit chronometric data, namely ex-Gaussian, shifted log-normal, gamma, Wald, 15 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Weibull, Gumbel, Poisson and ex-Wald distributions, and also shifted distributions, such as shifted log-normal, shifted Wald or shifted Weibull, and many others (Anders et al., 2016; Lo & Andrew, 2015; Schramm, & Rouder, 2019; Van Zandt, 2000). Each distribution is composed of unique parameters which nevertheless share some similarities (Palmer et. al., 2011; Moscoso del Prado Martin, 2009). For example, ex-Gaussian, which is a combination of a Gaussian and an exponential distribution, consists of three parameters: µ (mean of the Gaussian component), σ (standard deviation of the Gaussian component) and τ (mean of the exponential component; Luce, 1986). Some of these distributions can be reliably captured via generalized linear mixed- effect models (Lo & Andrew, 2015), but these parameters can also be incorporated into traditional analyses instead of means or medians (e.g., t-test on the tau parameter). Joint models, the second named approach, jointly analyzes RT distribution and performance accuracy to increase the level of obtained information, as RTs and accuracy might interact under the so-called speed accuracy trade-off (Wagenmakers et al., 2007). The most commonly used technique for joint modeling of RTs and accuracy is probably the hierarchical item response theory (IRT) model (van der Linden, 2007). This model estimates two parameters: latent construct and working speed. However, IRT does not model the underlying psychological mechanism which links responses to the measured construct (Molenaar et al., 2015; van der Maas et al., 2011). This potential shortcoming is overcome in the third approach, named process models. This approach models an underlying cognitive construct which, unlike IRT, provides the connection between the response and the estimated construct (Molenaar et al., 2015; van der Maas et al., 2011). The typical process model is the drift diffusion model (Ratcliff et al., 2016). These models decompose RTs into several interpretable parameters (Voss et al., 2013), for example, drift rate (speed of information accumulation, indicating the quality of the underlying cognitive construct), separation boundary (quantification of the amount of information needed to make a decision) or non-decision time (processes such as stimuli encoding or motoric reaction). Linear ballistic accumulation (LBA) model is another example of process models. Unlike drift diffusion model this model assumes linear accumulation of evidence and can be applied to a task with 2 and more response choices. Within LBA, five parameters are usually estimated (drift rate for each accumulator, threshold, starting point, start point variability, non decision time; Brown & Heathcote, 2008). Diffusion-based item response theory models represent a combination of both drift diffusion models and IRT (Molenaar et al., 2015; van der Maas et al., 2011). The models can be applied to performance tasks (Q-diffusion IRT model) or personality-type data (D-diffusion 16 VALIDATION OF METHODS MEASURING COGNITIVE STYLES IRT model). Compared to a drift diffusion model, the diffusion IRT model disentangles drift rate and separation boundary into separate person and item parameters (Molenaar et al., 2015). However, when the tasks are easy to solve (which is often the case of AH measurement), the diffusion model might not fit such data well (Anders et al., 2016; Faulkenberry, 2017). The shifted Wald distribution was hence proposed to deal with such situations (Anders et al., 2016, Faulkenberry, 2017; Steingroever et al., 2021), representing a simplified process model since it utilizes only RT, not accuracy. Three parameters are typically estimated (drift rate, threshold and non-decision time; Anders et al., 2016), although Steingroever et al. (2021) proposed a fourth parameter which models drift rates across trial variability. Unfortunately, ninety-five percent of articles which analyse RT in the three leading experimental journals ignored the specific character of RT (Balota & Yap, 2011), and to the best of our knowledge, not a single article has yet used appropriate estimates of RT in AH measurements. It is therefore possible that all previous AH studies which compared RTs did not compare the real differences in AH cognitive traits, but rather the differences in psychomotor tempo, stimulus encoding, response carefulness (Molenaar et al., 2015) or working speed (Fox et al., 2021). Futility of derived indices Although AH is proposed and especially measured as a two-dimensional construct, meaning that some participants might show a tendency to reason or perceive analytically and others holistically (naturally, some might show high or low both styles), researchers often reduce the amount of information in their data by calculating a derived index (e.g., summary or ratio between mean or median of RTs) and use it as a single “relative” indicator of cognitive style. For example, the manual of E-CSA-WA suggests calculating the main index as a ratio of median reaction times between analytic and holistic subtests (Peterson, 2005), and Navon-based methods often use the difference between subtests as an indicator of the global precedence score (e.g., McKone et al., 2010). Even though the FLT results are generally analysed separately according to their analytic and holistic scores, even here derived indices are used (e.g., Istomin et al., 2014). This type of an approach, however, cannot be regarded as appropriate, because it is not only rather reductionist (i.e., because it ignores the reaction time distribution), in some cases it is clearly misleading. The derived indices would be biased, especially in comparing groups 17 VALIDATION OF METHODS MEASURING COGNITIVE STYLES with highly different reaction times. Most of these indices are also unreliable, and since they differ from each other, they can alter, for example, the significance of the results (Gerlach & Krumborg, 2014). Even though some authors have attempted to solve this issue, for example, by using a relative percentage difference (Lacko, 2018) or standardized mean difference (Cohen’s d) between subtests (Gerlach & Krumborg, 2014), because of the two-dimensional nature of the AH construct, scholars are not forced to use any derived indices and instead analyse scores from multiple subtests separately. In summary, it seems that the “old controversy” might have survived to this day, even after a face-lift in the form of analytic and holistic cognitive style theory. Many new challenges have also been raised recently. The results of cross-cultural comparisons of AH might be biased due to the lack of sound measures. Hence, the aim in the present article is to verify the psychometric properties of methods which measure AH. According to many researchers in this field, a step of this type in improving the validity and reliability of methods is necessary (Peterson et al., 2009). To do so, we implemented several important steps: 1) application of relatively recently created methods of measuring AH to overcome the issues of the old controversy (e.g., holistic style is not perceived as inferior to analytic style); 2) the use of multiple and different methods; 3) analysis of discriminant validity with personality and intelligence; 4) verification of the stability of the construct; 5) collection of a large sample of the general population (i.e., non- student); 6) no derived indices; 7) application of advanced statistical procedures for RT estimation. Methods Analytic plan The hypotheses, statistical analyses and data cleaning procedures were pre-registered before data collection (see https://osf.io/w483c/?view_only=5d0ed04160554599835155ff10fa8030 – anonymized link for review). The hypotheses were pre-registered as a validation process composed of five phases (Fig. 1). In the first phase, specific aspects of validity and reliability (e.g., factor structure, internal consistency, split-half reliability) are verified. This phase was designated Phase 0 since it is not relevant to all the methods. In Phase 1, all methods are tested for their stability in time (test-retest reliability). In Phase 2, the discriminant validity of the 18 VALIDATION OF METHODS MEASURING COGNITIVE STYLES methods is verified. In Phase 3, the concurrent and convergent validity of the methods is assessed. Finally, in Phase 4, predictive validity is estimated. Throughout the analysis, methods which clearly demonstrate insufficient quality do not progress to the next phase. The deviations from a pre-registered analytic plan are described in Appendix 1. The data, methods and R source codes are available online (see https://osf.io/7ezax/?view_only=ca6007dc96574cce87066113060b65d8 – anonymized link for review). The research was approved by the ethical committee of [anonymised] University (Ref. No. anonymised). Figure 1: Validation process. 19 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Measures Methods for measurement of AH Analysis-Holism Scale (AHS; Choi et al., 2007; Koo et al., 2018). For a description, see section Self-report questionnaires. Extended Cognitive Style Analysis – Wholistic/Analytic (E-CSA-WA; Peterson et al., 2003; 2005). E-CSA-WA is illustrated in Figure 2. For a detailed description, see section Performance-based measures based on the wholistic-analytic family. Figure 2: Example of E-CSA-WA stimuli. Absolute-relative test (ART). The ART is a computer-based adaptation of pen-and-paper FLT (Kitayama et al., 2003). In this measure, the goal is exactly the same as with FLT (see section Performance-based measures based on the wholistic-analytic family). The original stimulus is 20 VALIDATION OF METHODS MEASURING COGNITIVE STYLES presented for 5,000 ms, followed by a mask presented for 100 ms. It contains 12 items (6 for analytic and 6 for holistic subtest, see Fig 3). We used line lengths from a study by Hakim et al. (2017) and converted them into pixels (1 mm2 = 3.78 px); in some cases, they were slightly shortened (see Table 1). This step was necessary, because the FLT is almost always used in a pen-and-paper version (with a few exceptions: Hakim et al., 2017; Klein et al., 2010; Stachoň et al., 2018). The main index of ART is the absolute mean difference (|∆M|) among the correct and drawn absolute/relative lengths of lines (Kitayama et al., 2003; 2009). Figure 3: Example of ART stimuli. Table 1: Length of lines and sides of the squares in ART. Item Side of the original length of the Side of the Absolute line Relative line square original line experimental square 1 408 276 306 276 207 2 553 386 533 386 386 3 382 106 582 106 106 21 VALIDATION OF METHODS MEASURING COGNITIVE STYLES 4 612 113 306 113 57 5 408 83 612 83 125 6 306 257 612 257 514 * note: all lengths are in pixels. Compound Figure Test 1 (CFT1). CFT1 is a verbal (numbers) and incongruent form of the Navon Search task (Fig. 4). Participants have to choose the correct answer from four options and indicate it with a mouse-click. The method is composed of 32 figures (16 for local identification and 16 for global identification). Fixation crosses are presented before each trial (for 500ms) and the figure remains visible until the participant's response. Even though the number of items is relatively low, even such a number might be sufficient (Davidoff et al., 2008; von Mühlenen et al., 2018) and the CFT1 might, therefore, serve as an effective short screening tool. Previous use of the CFT1 (e.g., Čeněk et al., 2020; Lacko, Šašinka et al., 2020; Opach et al., 2018; Šašinka et al., 2018) showed that incorrect answers are relatively rare, and therefore the error rate usually serves as a control variable (multiple errors per subtest can suggest misunderstanding the instructions or careless responses). (A) CFT1 stimulus. (B) CFT1 procedure. Figure 4: Example of CFT1 stimulus. Compound Figure Test 2 (CFT2). CFT2 is an extended version of CFT1 (80 items instead of 32), with several modification whose purpose is to increase the difficulty of responding correctly, thereby obtaining greater variability in accuracy: 1) local features are larger, decreasing the advantage of the global features (Ahmed & De Fockert, 2012); 2) items are block-randomized (participants do not know whether they will identify a local or global feature 22 VALIDATION OF METHODS MEASURING COGNITIVE STYLES in the next stimulus); 3) the presentation time is shorter; original stimulus disappears (after 100, 150, 200 or 250 ms; the time is the same for each block of 20 items; see Fig. 5). All other settings are the same as with CFT1. This measure was created from previous research (e.g., Gerlach & Poirel, 2018; Jansari et al., 2015; McKone et al., 2010). Figure 5: Example of CFT2 stimulus. Compound Figure Test 3 (CFT3). CFT3 is a non-verbal (uses geometric shapes) and incongruent form of the Navon Similarity Matching Task, which was created from previous research (Caparos et al., 2015; Davidoff et al., 2008; Oishi et al., 2014). The participant does not choose the correct answer, but rather chooses a preferable answer from two options. CFT3 contains 20 items, with one sample stimulus and two options (the first shares its global feature with the original stimulus, the second shares the local features; see Fig. 6). The participant is instructed to choose the option which, according to his/her opinion, is more similar to the sample stimulus. The main score is usually calculated from the number of choices in which the participant preferred the global or local option; RTs are not analysed (e.g., Oishi et al., 2014), but in line with our previous criticism of derived indices, we recorded RTs which allowed us to get two independent scores for both analytic and holistic dimensions (using LBA) instead of one ipsative score. 23 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure 6: Example of CFT3 stimuli. Methods for discriminant validity Big-Five Inventory 2 (BFI-2). The BFI-2 (Soto & John, 2017) is an updated version of BFI (John et al., 2008) and measures a theoretically expected five-factor model of personality: extraversion, agreeableness, conscientiousness, negative emotionality and open-mindedness. Besides these five domain scales, BFI-2 contains 15 facet subscales. The BFI-2 is composed of 60 items with 5-point Likert scales (1 = disagree strongly, 5 = agree strongly). The original English version showed satisfactory psychometric properties for a hierarchical model (each domain scale was a second-order factor, with three facets as first-order factors) and with a control for acquiescence (CFIs ≥ .930, TLIs ≥ .907, RMSEAs ≤ .078). Its internal consistency was very good for domain scales (αs ≥ .83) and moderate for facet subscales (αs ≥ .66). The test-retest was moderate to good for both domain scales (rs ≥ .76) and facet subscales (rs ≥ .66). We administered the adapted and validated Czech version by Hřebíčková et al. (2020). The Czech version yielded similarly satisfactory results for internal consistency (domain 24 VALIDATION OF METHODS MEASURING COGNITIVE STYLES scales: αs ≥ .81, facet subscales: αs ≥ .56), test-retest reliability (domain scales: rs ≥ .81, facet subscales: rs ≥ .72) and factor structure (CFIs ≥ .92, TLIs ≥ .90, RMSEAs ≤ .09). International Cognitive Ability Resource (ICAR; Condon & Revelle, 2014; (Dworak et al., 2021). ICAR is an open-source set of methods measuring cognitive ability. It was created by the International Cognitive Ability Resource Team (2014). ICAR currently contains more than 1,000 items for 19 lower-level constructs (Revelle et al., 2020). From this huge pool of tasks, we selected matrix reasoning (11 items, similar principle to Raven’s progressive matrices; see Fig. 7), three-dimensional rotation (24 items; see Fig. 8) for estimating general intelligence, and computer-generated number series. Since the full subtest originally contained 49 number series, we shortened it and randomly selected one item from each item model (= cognitive operators and their combinations), resulting in 11 number series (items 1, 6, 11, 16, 20, 25, 31, 36, 39, 42 and 47). The ICAR generally showed satisfactory psychometric properties (e.g., Condon & Revelle, 2014; Loe et al., 2018; Young et al., 2019). 25 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure 7: Example of an ICAR matrix reasoning subtest. 26 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure 8: Example of an ICAR three-dimensional subtest. Procedure Method development procedure The methods which required translation were translated using a back-translation method by two independent translators in each phase, the results were then compared and discussed by the entire translation team to eliminate any potential shifts in meaning. The CFT1 (e.g., Čeněk et al., 2020; Lacko, Šašinka et al., 2020; Opach et al., 2018; Šašinka et al., 2018) and ART (e.g., Lacko, 2018; Stachoň et al., 2018) have been repeatedly verified in the past, and therefore the feedback from respondents about the clarity of instructions and technical functioning during computer testing was abundant. However, the remainder of AH measures have not yet been verified, and therefore we performed two quantitative pilot studies on 32 participants (see Lacko, Čeněk et al., 2020) and 21 participants (see Masarikova, 2021) and one qualitative study on 7 participants through cognitive interviewing (not published). From these three pilot studies, we clarified the instructions to reduce any potential misunderstandings. Most of the methods also contained a set of practice trials with feedback. Testing procedure In compliance with pre-registration, we initially created set of 48 different strata based on the combination of three criteria: gender (male; female), education level (no education & 27 VALIDATION OF METHODS MEASURING COGNITIVE STYLES elementary; high school; high school with a graduation; university) and age (18–24; 25–34; 35–44; 45–54; 55–64; 65+) which proportionally corresponded to their representation in the general Czech population (Czech Statistical Office, 2019). These target subgroups of the general population were then addressed in relevant groups in social network and thematic websites using the snowball method, resulting in a relatively balanced pool of participants of 600 volunteers. Forty randomly selected participants were given a reward of CZK 2,000 (approx. 40x €80). All participants were tested online in two data collection waves. Before the first wave, participants signed into the participant pool via an introductory questionnaire (to check the strata proportions). The first wave was conducted in early May, 2021, the second was collected from mid-June to mid-August, 2021. The difference in time between the test and retest in all cases was therefore at least one month. In both waves, all volunteers were asked to participate actively. During the first wave, participants completed all methods which focused on cognitive style assessment, three subtests of ICAR and a demographic questionnaire; during the second wave, they completed all methods for cognitive style and BFI-2 (in the mentioned order). The typical time of administration of the entire test battery was approximately 90 minutes for the first wave and 45 minutes for the second wave. Even though online testing can be criticized for a potentially lower internal validity because of lower experimental control, no other option was available during the ongoing COVID-19 pandemic in the Czech Republic. It must be noted that this is a limit of the current study. However, we attempted to unify the experimental conditions as much as possible by issuing the following instructions. Participants were asked to: 1) use a computer or laptop with a keyboard and computer mouse (i.e., no tablets or touchpad); 2) be connected to a stable internet connection (ideally through an ethernet cable); 3) use at least a monitor with 15″ diagonally, 720p resolution and 60Hz frequency; 4) answer in a quiet and undisturbed environment; 5) take a short break after each method; 6) not ingest any alcohol or other drugs which could affect their concentration. All methods were adapted for online testing in Hypothesis software (Šašinka et al., 2017; Popelka et al., 2016), which allows reliable capture of response times, forces all slides to be the same size on different monitors and are preloaded into the web-browser before answering commences (answers are also saved locally on the web-browser, and after the tests are completed, all response times are uploaded to the Hypothesis server). Several psychological measures were already validated on this platform (see Šašinka et al., 2021). 28 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Nevertheless, we had to deal with several technical problems arising from the participants’ use of monitors with unusual resolutions and from unexpected technical difficulties in the SQL database. The most severe technical problems were lack of randomization for the first and second subtests in E-CSA-WA during the first wave of data collection (i.e., only default pseudo-randomization was applied), loss of data from some participants in the number series subtest of ICAR, and incorrect construction of the three- dimensional rotation subtest of ICAR, which did not save answers for items 1, 6, 8, 10, 13, 15, 20 and 23. Participants We collected data from 392 participants in total (380 in the first wave, 217 in the second wave). Of that number, 116 participants passed all methods in both waves. Each method was answered at least by 196 participants. Eight additional participants were removed because of their high number of invalid answers (see section below); 384 participants were included in the analysis (see Table 2). Pairwise approach was used for dealing with missing values. An a priori power analysis suggested that 250 participants should be sufficient for reliable estimation of crucial statistical procedures (see pre-registration for more details, https://osf.io/w483c/?view_only=5d0ed04160554599835155ff10fa8030). We therefore consider this sample size sufficient. Table 2: Demographic characteristics of the participants. Age Range 18–90 Me (IQR) 26 (13) M (SD) 30.35 (12.15) Gender Female 191 (49.74%) Male 101 (26.30%) NA 92 (23.96%) Education Elementary 12 (3.13%) 29 VALIDATION OF METHODS MEASURING COGNITIVE STYLES High school 7 (1.83%) High school with 120 (31.25%) graduation University 153 (39.84%) NA 92 (23.96) SES Poor 7 (1.82%) Lower mid 50 (13.02%) Mid 198 (51.6%) Upper mid 36 (9.38%) NA 93 (24.2%) Marital status Single 142 (36.98%) In relationship 76 (19.8%) married 65 (16.93%) Divorced 7 (1.82%) Widow 2 (0.52%) Number of siblings Range 0–8 Me (IQR) 1 (1) * note: Me = median, IQR = interquartile range, M = mean, SD = standard deviation, NA = missing values. Results Data analysis The RTs from CFT1, CFT2, E-CSA-WA were modelled using several methods with various parameters to estimate the real level of AH: 1) ex-Gaussian distribution with mu, sigma and 30 VALIDATION OF METHODS MEASURING COGNITIVE STYLES tau parameters; 2) lognormal response time item response theory models (LNIRT) with theta and gamma parameters; 3) Q-diffusion item response theory models (diffIRT) with theta and gamma parameters; 4) shifted Wald distribution process model with drift, alpha and theta parameters; 5) Bayesian 4-parameter shifted Wald distribution process model with drift, alpha, theta and drift variability parameters. Only the parameters which represented a cognitive trait of AH were applied in the following analyses (i.e., tau parameter for ex-Gaussian distribution, theta parameters for LNIRT and diffIRT, and drift parameters for shifted Wald and Bayesian 4-parameter shifted Wald distributions). The Gibbs sampling method with three Markov Chain Monte Carlo (MCMC) chains and 10,000 iterations was used to estimate the Bayesian 4-parameter Shifted Wald model. The convergence of the MCMC chains was evaluated according to the R^ statistic, which yielded adequate values; R^ < 1.1. Use of the Bayesian framework requires specification of the prior distribution of parameters for estimation. To the best of our knowledge, no prior research on this topic of interest has been conducted using the statistical procedures applied in the present study. We therefore use information and set up the prior distributions from the results of pilot studies. (For prior distributions, see https://osf.io/7ezax/?view_only=ca6007dc96574cce87066113060b65d8). In setting up the prior distributions, the drift mean, alpha and theta parameters were taken from truncated normal distributions (Steingroever et al., 2021). Although pilot studies were conducted, the information obtained from them was limited, and we set up the prior distributions deliberately rather non-informatively. Each model of the five approaches was verified before main analysis. However, the only RT estimations which yielded satisfactory fit indices were the shifted Wald process model. This is not surprising since the diffusion IRT model and LNIRT were designed for methods which produce high variability in accuracy. Even so, the methods were rather easy for participants to answer (accuracy of CFT1 local: 96.8%, CFT1 global: 97.3%, CFT2 local: 86.9%, CFT2 global: 92.1%, E-CSA-WA analytic: 97.2%, E-CSA-WA holistic: 96.8%). Although ex-Gaussian distributions fit the data adequately, their fits were inferior compared to shifted Wald. Hence, the parameter estimates of the shifted Wald model may be considered the most reliable. Since the Bayesian 4-parameter shifted Wald model adds one extra parameter, its results should be more accurate (Steingroever et al., 2021). Its estimates also highly correlated with the estimates of the shifted Wald process model. Therefore, only estimates of this model are reported in the main analyses. The complete report of model fit evaluation is 31 VALIDATION OF METHODS MEASURING COGNITIVE STYLES shown in Appendix 2; the robustness check of results performed with all five approaches is shown in Appendix 3. The CFT3 was modelled via the hierarchical LBA that models, besides five LBA parameters, also priors for the group-level means for starting point, non decision time and treshold, and group level drift rates (Annis et al., 2017). Only the drift rate parameter which represents a cognitive trait, was inserted into final analyses. We adopted the R code from Annis et al. (2017). Hamiltonian Monte Carlo algorithm with 4 chains and 4 000 iterations (500 burn- in) was used to estimate the hierarchical LBA model. Having no information about the prior distribution of the LBA parameters beforehand, we used the prior distributions reported in Annis et al. (2017). Besides these estimations, thetas from ICAR subtests were estimated with Rasch models. The analysis of split-half reliability and the correlation analyses were bootstrapped with 100,000 iterations. Since the BFI-II had already been validated in Czech and also yielded satisfactory internal consistency in our sample (extraversion ω = .872, agreeableness ω = .869, conscientiousness ω = .907, negative emotionality ω = .914, open mindedness ω = .881), we applied the subscale scores in the analyses as observed variables calculated as arithmetic means. In most cases, the discriminant validity was verified via two-one-sided t-tests (TOST, Lakens, 2017), which examined practical significance rather than statistical significance and provided evidence to support null hypotheses. In accordance with pre-registration, we specified the upper and lower equivalence bounds based on the smallest effect size of interest (SESOI) to -.25 and .25 as equivalents to practically null (absent) effect sizes. The values within the equivalence range lack practical significance and are therefore equivalent to the null hypothesis (i.e., evidence for discriminant validity). The predictive validity of the methods was verified with a set of one-tailed Welch’ t-tests for independent samples with p-values corrected using the Holm-Bonferroni method. All analyses were performed in R (v4.1.1; R Core Team, 2021) with packages lavaan (Rosseel, 2012), semTools (Jorgensen et al., 2021), TOSTER (Lakens, 2017), psych (Revelle, 2021), diffIRT (Molenaar et al., 2015), LNIRT (Fox et al., 2021), eRm (Mair & Hatzinger, 2007), retimes (Massidda et a., 2013), irr (Gamer et al., 2019) and multicon (Sherman, 2015). 32 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Data cleaning Data cleaning was performed precisely as pre-registered. We adopted the following arbitrary criteria as an assessment of invalid responses: RTs ms < 200 ms (= answer before stimuli appearance) or RTs > 5,000 ms (= absurdly high RT) in CFT1, CFT2 and E-CSA-WA (solely in the case of CFT3, the criterium was set to RTs > 10,000s since instruction did not emphasise the reaction speed and the answering was based on higher cognitive processing) and deviation > 150 px (= absurdly high |∆M|) in ART. In all methods, the participants with missing values (i.e., invalid answers or unfinished tests) greater than 50% were removed from further analysis. No other criteria for participant or response removal (e.g., analysis of outliers) were applied since the mechanical removal of values is not recommended for analysis of RTs (Ratcliff, 1993). Some additional criteria were applied only for diffIRT and LNIRT estimation. In the case of diffIRT, participants were removed according to their estimates which were intractable. In the case of LNIRT, participants were removed according to person-fit statistics which revealed who had a posterior probability higher than 95% of aberrant response times or accuracy. The results of data cleaning are in Table 3: Table 3: Removed answers and participants. Measure RT < 200 ms RT > 5,000 ms |∆M| > 150 px NA > 50% diffIRT LNIRT CFT1 W1 46 (0.51%) 24 (0.26%) - 0 8 11 CFT1 W2 8 (0.13%) 8 (0.13%) - 1 (0.51%) 1 2 CFT2 W1 4 (0.02%) 586 (2.84%) - 9 (3.49%) 7 4 CFT2 W2 0 333 (2.26%) - 6 (3.16%) 0 1 CFT3 W1 17 (0.30%) 92 (1.92%) ** - 1 (0.35%) - - CFT3 W2 3 (0.08%) 38 (0.96%) ** - 0 - - E-CSA-WA W1 95 (0.42%) 848 (3.79%) - 27 (9.64%) 2 8 E-CSA-WA W2 45 (0.31%) 307 (2.12%) - 6 (3.31%) 0 2 ART W1 - - 149 (3.39%) 12 (3.28%) - - ART W2 - - 44 (1.71%) 3 (3.28%) - - AHS W1 - - - 1 (0.29%) - - AHS W2 - - - 0 - - BFI-2 - - - 0 - - 33 VALIDATION OF METHODS MEASURING COGNITIVE STYLES ICAR matrices - - - 0 - - ICAR rotation - - - 18 (5.68%) - - ICAR numbers - - - 140 (44.16%) - - * note: RT = reaction time, ms = milliseconds, |∆| = absolute mean difference, NA = missing values, W1 = first wave of data collection, W2 = second wave of data collection, ** = the criterium was set up for 10,000s. It is evident from Table 3 that a relatively small number of answers (ns ≤ 3.79%) and only a minority of participants (besides two exceptions ns ≤ 5.68%) were removed across all methods. Two measures, however, indicate a higher rate of removed participants. In the first wave of data collection of E-CSA-WA, we removed almost 10% of participants from further analysis. This is most likely due to the technical issue described above. We also removed 44% of participants from the ICAR number series subtest as a result of a large number of participants being unable to send their answers for this subtest. Since such a huge number of removed participants could bias the result, we omitted the entire subtest from further analysis. Phase 0: Specific evidence of psychometric qualities In the first step, the factor structure of AHS was verified with confirmatory factor analysis. Even though several alternative structures were proposed, none of them were even close to the pre-registered criteria of model evaluation (RMSEA < .80, SRMR < .80, CFI > .90, TLI > .90; see Table 4). The internal consistency of all scales was also highly insufficient (locus of attention ω = .159, causal theory ω = .254, perception of change ω = .234, attitude toward contradictions ω = .550). Since the AHS demonstrated significant shortcomings in its factor structure and internal consistency and no model adjustment helped overcome these issues, we omitted AHS from further analysis. Table 4: Confirmatory factor analysis of AHS. CFI TLI RMSEA [CI 90%] SRMR mod1 .461 .395 .120 [.114, .127] .116 mod2 .461 .385 .121 [.115, .128] .116 mod3 .683 .605 .097 [.091, .104] .083 mod4a .223 -.324 .233 [.209, .257] .127 mod4b .619 .365 .145 [.112, .182] .098 34 VALIDATION OF METHODS MEASURING COGNITIVE STYLES mod4c .728 .547 .095 [.063, .129] .061 mod4d .894 .823 .093 [.059, .129] .053 mod5 .415 .359 .124 [.118, .130] .117 mod6 .725 .622 .129 [.116, .143] .110 mod7 .778 .704 .164 [.145, .183] .092 *note: mod1 = original factor structure; mod2 = hierarchical factor structure (one second-order factor); mod3 = control for response bias; mod4a-d = each subscale separately; mod5 = 1 factor structure; mod6 = original structure without items with low factor scores or high cross-loadings; mod7 = 1 factor structure without items with low factor scores or high cross-loadings; RMSEA = Root mean square error of approximation; CI = Confidence intervals; CFI = Comparative fit index; TLI = Tucker-Lewis index; SRMR = Standardized root mean square residual. In the next step, we estimated empirical reliability from latent trait estimates and their corresponding standard errors (calculated from diffIRT), i.e., empirical reliability of maximum a posteriori estimates, showing sufficient (reliability ≥ .70) for both subscales of CFT1 (local = .958, global = .983), CFT2 (local = .974, global = .954) and E-CSA-WA (analytic = .945, holistic = .942). Split-half reliability was estimated only in ART since each item contains very different lengths of lines which might affect the response accuracy. Split-half reliability was calculated on random halves using Guttman’s λ2. The reliability was identical to the pre-registered threshold for sufficient evidence of reliability (analytic = .50, holistic = .50). Phase 1: Stability of the construct The stability of the construct, i.e., test-retest reliability, was estimated using intraclass correlation coefficients (ICC) with two-way mixed effects and absolute agreement (ICC ≥ .50 suggests moderate reliability, and ICC ≥ .75 indicates good reliability). Comprehensive results are listed in Tables 5 and 6. We found that ART indicated moderate reliability for the holistic subtest but insufficient reliability for the analytic subtest. CFT3 indicated good reliability. CFT1, CFT2 and E-CSA-WA indicated good reliability in median raw RTs and moderate reliability estimated according to the drift parameter from the Bayesian 4-parameter shifted Wald process model (except for the holistic subtest of E-CSA-WA, which, similarly to ART, showed ICC slightly below the pre-registered threshold). 35 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Table 5: Intraclass correlation coefficients of ART and CFT3. ART CFT 3 Analytic Holistic Analytic (hierarchical LBA drift) Analytic (hierarchical LBA drift) .444 [.259, .582] .529 [.376, .645] .791 [.702, .853] .701 [.578, .788] Table 6: Intraclass correlation coefficients of CFT1, CFT2 and E-CSA-WA. CFT 1 CFT 2 E-CSA-WA Local Global Local Global Analytic Holistic Raw RT: Median .900 .857 .871 .854 .795 .739 [.865, .926] [.807, .894] [.701, .932] [.962, .919] [.608, .880] [.349, .868] Bayesian 4- .505 .538 .570 .616 .573 .411 parameter shifted [.329, .671] [.373, .660] [.351, .708] [.416, .740] [.374, .704] [-.039, .634] Wald: Drift Phase 2: Discriminant validity with intelligence and personality The discriminant validity of E-CSA-WA and CFT2 with personality traits of extraversion, agreeableness, conscientiousness, negative emotionality and open mindedness measured by BFI-II was computed according to the heterotrait-monotrait ratio of correlations (HTMT). As pre-registered, the values were calculated specifically for this analysis on a subscale level (e.g., 4 global subtests in CFT2) and subsequently applied in HTMT as indicators of AH. Since none of the values was above .90, all methods showed satisfactory discriminant validity (Table 7). Table 7: Discriminant validity of CFT2 and E-CSA-WA with personality traits. Method Indicator extrave agreeable conscienti negative open- rsion ness ousness emotionality mindedness CFT2: Local Raw RT: Me .101 .137 .060 .119 .134 Bayesian 4-parameter .237 .177 .172 .191 .294 shifted Wald: Drift CFT2: Global Raw RT: Me .102 .126 .060 .095 .136 Bayesian 4-parameter .191 .159 .147 .183 .256 shifted Wald: Drift 36 VALIDATION OF METHODS MEASURING COGNITIVE STYLES E-CSA-WA: Raw RT: Me .144 .100 .085 .090 .122 Local Bayesian 4-parameter .106 .128 .122 .099 .151 shifted Wald: Drift E-CSA-WA: Raw RT: Me .121 .086 .087 .070 .118 Global Bayesian 4-parameter .163 .120 .074 .1112 .121 shifted Wald: Drift The discriminant validity of ART, CFT1 and CFT3 with personality was verified using the TOST procedure. All correlations with personality were lower than .20 for all methods, and none exceeded the pre-registered criteria for practical significance (Table 8). The only exception was the drift parameter from CFT3 holistic subtests which correlated with conscientiousness, but its correlation only neglictibly exceeded the pre-registered treshold (r = .205). We can therefore conclude that all methods indicated satisfactory discriminant validity with personality traits. Table 8: Discriminant validity of ART, CFT1 and CFT3 with personality traits. Method Indicator extraversio agreeabl conscientio negative open n eness usness emotionality mindedness ART: Analytic |∆M| .106 .075 .192 -.059 .052 ART: Holistic |∆M| .077 .079 .064 -.038 -.021 CFT1: Local Raw RT: Me -.033 .154 .101 -.055 -.022 Bayesian 4-parameter .117 -.074 -.108 .012 -.010 shifted Wald: Drift CFT1: Global Raw RT: Me -.056 .161 .106 -.048 -.050 Bayesian 4-parameter -.043 -.006 -.068 .023 -.006 shifted Wald: Drift CFT3: Hierarchical LBA: -.137 -.086 -.080 .067 -.134 Analytic Drift CFT3: Holistic Hierarchical LBA: .056 -.090 .205 * -.061 .063 Drift note: * = insignificant TOST results (i.e., practically significant association). The discriminant validity of all methods with intelligence (measured by ICAR subtest of matrix reasoning and rotation) was also verified using TOST. CFT1 indicated weak negative 37 VALIDATION OF METHODS MEASURING COGNITIVE STYLES correlations with matrix reasoning for both raw RTs and drift parameters (quicker participants in both subscales were more successful in matrix reasoning). These associations, however, were contained within the equivalence range, and therefore practical significance was not established. Associations between CFT2, CFT3 and E-CSA-WE were generally much lower and therefore also practically insignificant for both RTs and drift parameters (Table 9). Nevertheless, CFT2 indicated practically significant and potentially problematic associations with the rotation subtest according to the theta parameters of diffusion IRT models and lognormal RT IRT models (Appendix 3, Table S3D). The causes and consequences of these associations are elaborated in the section Discussion. Finally, the holistic subtest of ART indicated statistically and practically significant negative association with the rotation subset of ICAR. Participants who were more accurate in drawing relative lines in ART were also more successful in rotation. Since the discriminant criterion with intelligence is crucial, and ART also revealed some issues with stability, we omitted this method from the next phase of the validation process. Table 9: Discriminant validity with intelligence. Method Indicator Matrix Rotation ART: Analytic |∆M| -.228 -.248 ART: Holistic |∆M| -.247 -.360 * CFT1: Local Raw RT: Me -.248 -.175 Bayesian 4-parameter shifted .256 .161 Wald: Drift CFT1: Global Raw RT: Me -.266 -.168 Bayesian 4-parameter shifted .204 .128 Wald: Drift CFT2: Local Raw RT: Me .054 .115 Bayesian 4-parameter shifted .029 .047 Wald: Drift CFT2: Global Raw RT: Me .130 .157 Bayesian 4-parameter shifted -.042 -.064 Wald: Drift CFT3: Hierarchical LBA: Drift -.116 -.081 Analytic 38 VALIDATION OF METHODS MEASURING COGNITIVE STYLES CFT3: Holistic Hierarchical LBA: Drift -.048 -.025 E-CSA-WA: Raw RT: Me .075 .086 Local Bayesian 4-parameter shifted .057 .105 Wald: Drift E-CSA-WA: Raw RT: Me -.006 -.049 Global Bayesian 4-parameter shifted -.053 -.024 Wald: Drift note: * = insignificant TOST results (i.e., practically significant association). Phase 3: Concurrent and divergent validity The concurrent and divergent validity was estimated within a multi-trait multi-method matrix (MTMM). These results suggest that all three methods measure entirely different traits (i.e., the associations between related analytic/holistic subtests from various methods are not satisfactory). Similarly, low associations were found also for the same method-different trait values, (i.e., the association between analytic and holistic subtest within a single measure), and for different method-different trait values (i.e., associations between analytic/holistic subtest from different methods which should not be related). Hence, the associations between subtests from various methods which should not be related are actually similar to those which should be related (Table 10). Table 10: Multi-trait multi-method matrix for CFT1, CFT2 and E-CSA-WA. Indicator of RTs Same Trait-Different Same Method-Different Different Method- Method Trait Different Trait Bayesian 4-parameter .113 .075 .153 shifted Wald: Drift To analyse the specific association between methods in more detail, we performed a Spearman correlation analysis (Table 11). It is evident that CFT3 was not associated with any other measure since all its correlations were lower than .12. Although the associations between CFT1, CFT3 and E-CSA-WA at the RT level were higher than the pre-registered threshold of .30, these results were not replicated for drift parameters. Lack of association between the drift parameters of various methods indicates that CFT3 and CFT1 represent different aspects of AH. Even though CFT2 and E-CSA-WA showed weak associations with drift parameters, they 39 VALIDATION OF METHODS MEASURING COGNITIVE STYLES cannot be interpreted in terms of satisfactory concurrent validity, and therefore, most likely also measure different facets of AH. Table 11: Spearman’s rank correlation coefficients between related AH subtests. Method Indicator CFT2 CFT3 E-CSA-WA CFT1: Local Raw RT: Me .38 [.24, .48] *** - .31 [.15, .39] *** Bayesian 4-parameter shifted .09 [-.04, .24] .06 [-.05, .18] .18 [.05, .29] * Wald: Drift CFT1: Global Raw RT: Me .33 [.22, .45] *** - .41 [.30, .49] *** Bayesian 4-parameter shifted .08 [-.07, .23] .03 [-.08, .17] .06 [-.06, .18] Wald: Drift CFT2: Local Raw RT: Me - - .36 [.20, .47] *** Bayesian 4-parameter shifted - -.02 [-.19, .14] .26 [.13, .38] *** Wald: Drift CFT2: Global Raw RT: Me - - .32 [.14, .42] *** Bayesian 4-parameter shifted - -.04 [-.16, .13] .23 [.12, .35] *** Wald: Drift CFT3: Hierarchical LBA: Drift - - .12 [-.03, .20] Analytic CFT3: Hierarchical LBA: Drift - - .10 [-.04, .22] Holistic note: *** = p-value < .001, ** = p-value < .01, * = p-value < .05. Even thought the MTMM revealed the very small values in the same method-different trait values, we also report the associations between analytic and holistic subtests within each method separately because they provide more evidence about dimensionality of the construct (Table 12). CFT1, CFT2 and E-CSA-WA showed very strong associations between both subtests for RTs, and subtests of CFT2 and E-CSA-WA remained highly correlated, even for drift parameters. On the other hand, the CFT3 showed negative associations between subtests (this fact caused the low same method-different trait value since other subtests from other three methods were associated). This suggests that CFT1 might effectively distinguish between analytic and holistic dimensions, whereas the assumption of two dimensions might be violated for CFT2 and E-CSA-WA. As for CFT3, the negative associations means that participants who 40 VALIDATION OF METHODS MEASURING COGNITIVE STYLES score more on analytic subtest, score lower on holistic subtest (and vica versa) which suggests uni-dimensional structure of AH. Table 12: Spearman's rank correlation coefficients between subtests within one measure. Method Indicator Associations between subtests CFT1 Raw RT: Me .83 [.78, .87] *** Bayesian 4-parameter shifted .20 [.08, .33] ** Wald: Drift CFT2 Raw RT: Me .89 [.86, .92] *** Bayesian 4-parameter shifted .62 [.52, .68] *** Wald: Drift CFT3 Hierarchical LBA: Drift -.38 [-.51, -.26] *** E-CSA-WA Raw RT: Me .83 [.78, .86] *** Bayesian 4-parameter shifted .56 [.46, .66] *** Wald: Drift note: *** = p-value < .001, ** = p-value < .01, * = p-value < .05. Phase 4: Predictive validity The final phase of the validation process attempted to verify the predictive validity of methods. Since the theory of AH is built predominantly on cross-cultural comparisons, the feasibility of verifying predictive validity in single-group samples is rather limited. Nevertheless, according to some evidence, social class should affect AH similarly to other cultural influences (Grossmann & Varnum, 2011). Persons from lower social classes should be more holistic and less analytic than persons from higher social classes. For this purpose, we used a self-report variable for socioeconomic status (SES) and split it into two extremes: poor and lower mid SES (N = 57) on one side and upper mid SES (N = 36) on the other. The category of mid SES was also the most frequent and omitted from further comparison. This step decreased the statistical power of Welch's t-tests, leading to statistically insignificant results despite its potentially interesting effect sizes. Only the CFT1 local subtest met the pre-registered criteria of significant difference (pHolm < .05 & d > 0.2) at the RT level in the expected direction. However, the remainder of comparisons for both raw RT level and drift parameter were insignificant and thus barely 41 VALIDATION OF METHODS MEASURING COGNITIVE STYLES interpretable, especially since some of them were opposite to the expected results (see Table 13). Table 13: Differences between participants from lower and upper mid-socioeconomic status at their level of AH. Method Indicator Lower Upper mid t df p pHolm d mid M M CFT1: Raw RT: Me 1.139 0.996 4.13 81.95 < .001 .001 .823 Local Bayesian 4-parameter 4.134 4.327 -1.95 65.03 .028 .336 .439 shifted Wald: Drift CFT1: Raw RT: Me 1.012 0.874 3.28 81.80 .999 1 .643 Global Bayesian 4-parameter 3.893 3.900 -0.05 51.48 .521 1 .013 shifted Wald: Drift CFT2: Raw RT: Me 2.229 2.069 1.72 39.34 .047 .517 .475 Local Bayesian 4-parameter 2.455 2.525 -0.61 35.79 .272 1 .175 shifted Wald: Drift CFT2: Raw RT: Me 2.244 2.062 1.82 38.04 .962 1 .509 Global Bayesian 4-parameter 2.533 2.664 -1.17 36.49 .893 1 .358 shifted Wald: Drift CFT3: Hierarchical LBA: Drift 0.812 0.774 0.33 68.92 .630 1 .077 Analytic CFT3: Hierarchical LBA: Drift 2.285 2.372 -0.85 68.61 .200 1 .194 Holistic E-CSA- Raw RT: Me 1.533 1.328 2.27 75.33 .013 .169 .486 WA: Local Bayesian 4-parameter 2.932 3.099 -1.15 59.67 .128 1 .270 shifted Wald: Drift E-CSA- Raw RT: Me 1.293 1.195 1.57 75.95 .940 1 .331 WA: Global Bayesian 4-parameter 2.243 2.507 -1.85 61.36 .965 1 .431 shifted Wald: Drift note: Me = median; M = mean; t = t-statistics; df = degrees of freedom; p = one-tailed p-value; pHolm = p-value adjusted by the Holm-Bonferroni method; d = Cohen’s d. 42 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Discussion Psychometric properties of developed AH instruments The article presented six proposed methods for measuring AH. Once developed, these methods and their psychometric properties were assessed. Our findings indicate the failure of self-report AHS to pass the first criterion of the valid factor structure. Since repeated verification of factor structure on different samples from various cultural groups is considered necessary when establishing the validity of questionnaires which measure cross-cultural constructs (Boer et al., 2018; Chen, 2008; Fischer & Karl, 2019; Milfont & Fischer, 2010), its validity might be seriously compromised. More importantly, no other research on cognitive styles using self- report scales has provided evidence of scalar measurement invariance across cultures that would allow meaningful comparison of latent means across countries. Establishing the measurement invariance is an important criterion which is quite often not satisfied in other more cross-culturally rooted questionnaires (Lacko et al., 2022). For this reason, we are sceptical of using AH self-report questionnaires in future research. The FLT appeared to be a very promising successor of the Witkin’s rod-and-frame test. However, our computer-based adaptation demonstrated a problem with the stability of the construct in time and an undesirable correlation with general intelligence, namely with its spatial ability subtest. Since no previous study has provided sufficient evidence for the validity of this method, in the context of this study, we cannot recommend it for further use. It appears that the rod-and-frame principle might be an interesting indicator of spatial ability, but its informative value regarding analytic and holistic cognitive styles remains ambiguous. The E-CSA-WA, however, indicated moderate test-retest reliability, absence of association with personality and intelligence, and very weak concurrent validity with CFT1 and CFT2. Together with previous evidence of validity and reliability (Peterson et al., 2003; 2005) and a sufficient number of items per subscale for reliable RT estimation, we may, with certain reservations, consider it a valid method and recommend it (and the principle of embedded figures behind the instrument) for further use in assessing an individual’s cognitive style. However, the method’s moderate correlation between both subtests might suggest a certain dependence between analytic and holistic modes of processing information, which represents a certain limitation which should be further studied. Regarding Navon figures, three modifications of this test were used (CFT1, CFT2 and CFT3). All relevant psychometric properties of CFT1 were found to be satisfactory. Even 43 VALIDATION OF METHODS MEASURING COGNITIVE STYLES though this instrument indicated weak association with intelligence, it was not practically significant. Both subtests were only weakly associated with each other and hence we can recommend CFT1 for further use. However, since CFT1 contains a small number of items (although for traditional RT analysis, it might be considered satisfactory, see Davidoff et al., 2008; von Mühlenen et al., 2018), estimation of the drift parameters in the shifted Wald model might be unreliable, and adding more items per subtest is desirable. To reflect this in future research, at least fifty items per subtest are recommended for a reliable estimation of drift parameters within shifted Wald distribution (Anders et al., 2016). CFT2 also satisfied almost all validity and reliability criteria (with the exception of high correlation in its subtests; similarly to E-CSA-WA). CFT2, however, yielded some inconsistencies in the results which required investigation before the use of this instrument as an indicator of AH. The accuracy of CFT2 indicated higher variability in responses and was generally lower than in other instruments. In relation to this, the alternative RT estimations which take into account difficulty, for example, lognormal response time item response theory models and Q-diffusion item response theory models, also indicated a slight improvement in model fit (and therefore more reliable estimation). Even though we still cannot consider these estimations reliable, they indicate that the CFT2 has a practically significant association with intelligence (rotation subtest of ICAR, see Table S3D in Appendix 3). Is it therefore possible that when the difficulty of the tasks increases, solving them automatically overlaps with cognitive ability? If this is true, AH measurement must rely on simplier tasks to keep its discriminant validity with cognitive ability such as intelligence. We believe that the principle behind CFT2 needs further examination to verify whether the incorporation of more challenging tasks generates associations with general intelligence. The last method, CFT3, overall possessed good psychometric properties. Its stability in time was very high, probably because it is based on a different principle than the previous two Navon hierarchical figures (i.e., similarity matching task). It was not associated with intelligence and most of personality traits. Even thought it was slightly associated with conscientiousness (more holistic people were more conscientiousness), the association was very low and did not yeopardize its discriminat validity. Furthermore, with respect to the observed systematic cultural differences in the Big Five personality traits (McCrae et al., 2005), our finding appears to be logical. After a few modifications, even this instrument can be considered for further use. The crucial modification should lie in the increase of number of tasks, since LBA generally needs more items to be reliable (Brown & Heathcote, 2008). However, it is also necessary to find an adequate compromise between the recommended 44 VALIDATION OF METHODS MEASURING COGNITIVE STYLES number of trials for reliable estimation and added information value since these tasks are generally very simple and repetitive and participants tend to identify their own preferences very quickly. Common shortcomings of AH instruments and future research Although in the previous chapter we recommended three methods and the principles behind them for reliable AH measurement in the future (CFT1, CFT3, E-CSA-WA) and one method for deeper inspection of the link between difficulty in AH tasks and general intelligence (CFT2), we also identified some of their key limitations. These issues do not necessarily jeopardize the validity or reliability of the methods, because they may simply stem from the insufficiently substantiated theoretical background of AH research. These issues also might represent the future research in the AH field. The first limitation relates to the stability of the AH construct. CFT1, CFT2 and E- CSA-WA only tightly exceeded the pre-registered threshold of ICC > .50 for drift parameters estimated within Bayesian 4-parameter shifted Wald process models (the holistic subtest of the E-CSA-WA was slightly below this threshold). Good stability was shown only in the CFT3 that was not based on the Navon search task but on the Navon similarity matching task. These results were not surprising, as a considerable number of questions have been recently raised about the stability of the AH concept. For example, Zhang (2013) argued that cognitive styles are inherently a dynamic phenomenon, and Kozhevnikov et al. (2014) emphasized a more task- dependent character in cognitive styles. Many situational factors are also considered to potentially affect the scores in perceptual tasks. For instance, RT for Navon figures may be affected by the participant’s current mood (positive moods are more likely to elicit a global level of processing, whereas negative moods lead to a local level of processing; e.g., De Fockert & Cooper, 2014; Gasper & Clore, 2002; Ji et al., 2019). Our original results supported the theoretical models, suggesting a dynamic change in the AH level rather than traditional views on a cognitive style as an entirely stable trait. Further research manipulating with the length of test-retest measurement and examining the factors influencing the change in the level of AH (such as the effects of training or the emotional state) can enrich the current knowledge about the stability of analytic and holistic cognition. The second limitation relates to the dimensionality of AH. The correlation analysis revealed that the methods are not effective in distinguishing between their analytic and holistic subtests (with the exception of CFT3). It is possible that E-CSA-WA and CFT2 both measure 45 VALIDATION OF METHODS MEASURING COGNITIVE STYLES one-dimensional constructs with two slightly different tasks (or that both of these dimensions underline a single second-order factor). What seems probable is that analytic and holistic styles do not represent orthogonal dimensions but are at least to some extent associated with each other. It is a question whether this finding should be considered a limitation or an immanent feature of AH. Further research can attempt to distinguish analytic and holistic subtests more satisfactorily from each other. This distinction could be pursued with eye-tracking research measuring dwell time spent on background and dominant objects. From the findings, analytic components might emphasize a focus on detail (simple figures which must be identified in complex figures should differ only in small details) and holistic parts might incorporate even more complex and embedded backgrounds for figures. However, from the current theoretical approaches in combination with our empirical findings, it is impossible to decide whether analytic and holistic subtests should be (positively) correlated, as this association was beyond the scope of previous research and must be replicated (besides as a consequence of the usage of derived indices). The third limitation is in divergent validity. The MTMM showed that the methods do not effectively distinguish between subtests. Concerning concurrent validity, deeper inspection using correlation analyses revealed that CFT1 and CFT2 were associated only weakly (rs < .30) with E-CSA-WA and that CFT2 and CFT1 did not correlate at all. CFT3 also did not relate to any other methods. These results agree with previous research which revealed only weak associations between various AH instruments and between modified versions of an instrument (e.g., Na et al., 2019; see also Na et al., 2010; Chamberlain et al., 2017; Huygelier et al., 2018; Milne & Szczerbinski, 2009; Peterson & Deary, 2006; Dale & Arnell, 2013). These findings are strongly against a two-dimensional AH theoretical model, which we suggest should be potentially revised with respect to our present findings. This is also in line with some other research which already found u two-dimensional model of AH unsuitable and simplistic (e.g., Čeněk et al., 2020; Lacko, Šašinka et al., 2020; Na et al., 2010; Wong et al., 2021). Future research can be inspired by already described complex multilevel hierarchical models which were proposed to deal with a multitude of cognitive style models but not specifically to the AH dimension (e.g., Miller, 1987; 1991; Nosal, 1990; Kozhevnikov et al., 2014). For example, Kozhevnikov et al. (2014) described four main clusters of cognitive style, namely context dependence/independence, rule-based vs. intuitive processing, integration vs. compartmentalization, and internal vs. external locus of processing, which can manifest at four hierarchically sorted levels: perception, concept formation, higher-order cognitive processing and metacognitive processing. It is thus possible, that AH methods measure to some extent 46 VALIDATION OF METHODS MEASURING COGNITIVE STYLES independent facets of the AH construct or even entirely independent constructs which manifest similarly in cross-cultural comparisons but are not related at the individual level. Finally, the fourth limitation is in the lack of predictive validity. None of the methods were capable of detecting the differences between participants of low and high socioeconomic status, and some of the statistically significant differences even indicated opposite directions (i.e., participants with higher SES showed higher levels of both holistic and analytic cognitive style). One group being higher in both subtests than the other is one of the possible outcomes of group comparisons and does not necessarily mean that instruments measure ability rather than style or trait. For example, Lee et al. (2021) compared holistic and analytic thinkers (based on categorization/triad task) and found that holistic thinkers were quicker in both local and global subtests of hierarchical figures. The true reason behind these findings, however, is most likely the uncertain dimensionality of the construct and should be considered a topic for future research. Even though we did not find evidence of predictive validity, we do not interpret this as diminished evidence of the psychometric properties of the instruments used, for several reasons: 1) the SES relationship to AH is not yet well established, and the only study which has analysed this applied the variable “social class” composed of characteristics other than SES (cf. Grossmann & Varnum, 2011); 2), the Czech population is relatively homogenous in terms of participant SES, and the AH differences derived from cross-cultural comparisons might therefore not be manifested in a single-culture sample (Na et al., 2010); 3) the effect of social class can vary as a function of cultural contexts (Miyamoto et al., 2018; Na et al., 2016); 4) because we compared only extremes in SES, we obtained a small number of participants per group and thus also low statistical power; and 5) AH is a cross-cultural theory and its predictive validity therefore lies in cross-cultural comparison (which represents the only available choice of predictive validity) and not comparisons within a single-culture group. Hence, we must conclude that the predictive validity of the proposed instruments remains unknown and further robust cross-cultural comparisons are necessary. Without such comparisons, we cannot clearly tell whether methods indeed measure the analytic and holistic cognitive style or not. 47 VALIDATION OF METHODS MEASURING COGNITIVE STYLES References Ahmed, L., & De Fockert, J. W. (2012). Working memory load can both improve and impair selective attention: Evidence from the Navon paradigm. Attention, Perception, & Psychophysics, 74(7), 1397–1405. Allinson, J., & Hayes, C. (1996). The Cognitive Style Index, a measure of intuition-analysis for organizational research. Journal of Management Studies, 33, 119–135. Alotaibi, A., Underwood, G., & Smith, A. D. (2017). Cultural differences in attention: Eye movement evidence from a comparative visual search task. Consciousness and cognition, 55(2017), 254-265. Allport, G. W. (1937). Personality: A psychological interpretation. New York: Holt. Álvarez-Montero, F. J., Leyva-Cruz, M. G., & Moreno-Alcaraz, F. (2018). Learning Styles Inventories: an update of Coffield, Moseley, Hall, & Ecclestone’s Reliability and Validity Matrix. Electronic Journal of Research in Education Psychology, 16(46), 597. https://doi.org/10.25115/ejrep.v16i46.2237. Anders, R., Alario, F.-X., & Van Maanen, L. (2016). The Shifted Wald Distribution for Response Time Data Analysis. Psychological Methods, 21(3), 309–327. https://doi.org/10.1037/met0000066 Annis, J., Miller, B. J., & Palmeri, T. J. (2017). Bayesian inference with Stan: A tutorial on adding custom distributions. Behavior research methods, 49(3), 863–886. https://doi.org/10.3758/s13428-016-0746-9 Armstrong, S. J., Cools, E., & Sadler-Smith, E. (2011). Role of Cognitive Styles in Business and Management: Reviewing 40 Years of Research. International Journal of Management Reviews, 14(3), 238-262. Asch, S., & Witkin, H. (1948a). Studies in space orientation: I. Perception of the upright with displaced visual fields. Journal of experimental psychology, 38(3), 325-337. Asch, S., & Witkin, H. (1948b). Studies in space orientation: II. Perception of the upright with displaced visual fields and with body tilted. Journal of Experimental Psychology, 38(4), 455-477. 48 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Aslan, H., Aslan, A., Dinc, D., & Yunluel, D. (2018). Testing the Reliability of CSA Test on a Sample of Turkish Population. International Journal of Scientific and Technological Research, 4(9), 27-31 Ausburn, L., & Ausburn, F. (1978). Cognitive styles: Some information and implications for instructional design. Educational Communication and Technology, 26(4), 337– 354. Balota, D. A., & Yap, M. J. (2011). Moving Beyond the Mean in Studies of Mental Chronometry. Current Directions in Psychological Science, 20(3), 160–166. doi:10.1177/0963721411408885 Bahar, M., & Hansell, M. H. (2000). The Relationship Between Some Psychological Factors and their Effect on the Performance of Grid Questions and Word Association Tests. Educational Psychology, 20(3), 349–364. doi:10.1080/713663739. Bendall, R. C. A., Galpin, A., Marrow, L. P., & Cassidy, S. (2016). Cognitive Style: Time to Experiment. Frontiers in Psychology, 7. doi:10.3389/fpsyg.2016.01786. Bergman, H., & Engelbrektson, K. (1973). An examination of factor structure of Rod-and- frame Test and Embedded-figures Test. Perceptual and motor skills, 37(3), 939–947. https://doi.org/10.2466/pms.1973.37.3.939. Boccia, M., Piccardi, L., Di Marco, M., Pizzamiglio, L., & Guariglia, C. (2016). Does field independence predict visuo-spatial abilities underpinning human navigation? Behavioural evidence. Experimental brain research, 234(10), 2799–2807. https://doi.org/10.1007/s00221-016-4682-9. Boduroglu, A., Shah, P., & Nisbett, R. E. (2009). Cultural differences in allocation of attention in visual information processing. Journal of Cross-Cultural Psychology, 40(3), 349-360. Boer, D., Hanke, K., & He, J. (2018). On Detecting Systematic Measurement Error in Cross- Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests. Journal of Cross-Cultural Psychology, 49(5), 713–734. doi: 10.1177/0022022117749042. Brown, S. D., & Heathcote, A. (2008). The simplest complete model of choice response time: linear ballistic accumulation. Cognitive Psychology, 57(3), 153–178. https://doi.org/10.1016/j.cogpsych.2007.12.002 49 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Busato, V. V., Prins, F. J., Elshout, J. J., & Hamaker, C. (1999). The relation between learning styles, the Big Five personality traits and achievement motivation in higher education. Personality and Individual Differences, 26(1), 129–140. https://doi.org/10.1016/S0191-8869(98)00112-3. Caparos, S., Fortier-St-Pierre, S., Gosselin, J., Blanchette, I., & Brisson, B. (2015). The tree to the left, the forest to the right: Political attitude and perceptual bias. Cognition, 134, 155-164. Cassidy, S. (2004). Learning styles: An overview of theories, models, and measures. Educational Psychology, 24(4), 419–444. Čeněk, J., Tsai, J.-L., & Šašinka, Č. (2020). Cultural variations in global and local attention and eye-movement patterns during the perception of complex visual scenes: Comparison of Czech and Taiwanese university students. PLOS ONE, 15(11), e0242501. https://doi.org/10.1371/journal.pone.0242501. Chamberlain, R., Van der Hallen, R., Huygelier, H., Van de Cruys, S., & Wagemans, J. (2017). Local-global processing bias is not a unitary individual difference in visual processing. Vision Research, 141, 247-257. Chen, F. F. (2007). Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464– 504. doi: 10.1080/10705510701301834. Chiu, L-H. (1972). A cross-cultural comparison of cognitive styles in Chinese and American children. International Journal of Psychology, 7(4), 235-242. Choi, I., & Nisbett, R. (1998). Situational Salience and Cultural Differences in the Correspondence Bias and Actor-Observer Bias. Personality and Social Psychology Bulletin, 24(9). 949-960. Choi, H., Connor, C. B., Wason, S. E., & Kahan, T. A. (2016). The effects of interdependent and independent priming on Western participants’ ability to perceive changes in visual scenes. Journal of Cross-Cultural Psychology, 47(1), 97-108. Choi, I., Dalal, R., Kim-Prieto, C., & Park, H. (2003). Culture and judgement of causal relevance. Journal of Personality and Social Psychology, 84(1), 46–59. doi:10.1037/0022-3514.84.1.46. 50 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Choi, I., Koo, M., & Choi, J. (2007). Individual differences in analytic versus holistic thinking. Personality and Social Psychology Bulletin, 33(5), 691-705. Choi, I., Nisbett, R., & Norenzayan, A. (1999). Causal Attribution Across Cultures: Variation and Universality. Psychological Bulletin, 125(1), 47-63. Chua, H., Boland, J., & Nisbett, R. (2005). Cultural variation in eye movements during scene perception. Proceedings of the National Academy of Sciences of the United States of America, 102(35), 12629-12633. Chua, H., Leu, J., & Nisbett, R. E. (2005). Diverging views of social events. Personality and Social Psychology Bulletin, 31, 925–934. Coffield, F., Moseley, D., Hall, E., and Ecclestone, K. (2004). Learning Styles and Pedagogy in Post-16 Learning. London: Learning Skills Research Centre. Retrieved from: https://www.leerbeleving.nl/wp-content/uploads/2011/09/learning-styles.pdf. Cohen, D., & Gunz, A. (2002). As seen by the other … Perspectives on the self in the memories and emotional perceptions of Easterners and Westerners. Psychological Science, 13(1), 55–59. Condon, D. M., & Revelle, W. (2014). The international cognitive ability resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52– 64. https://doi.org/10.1016/j.intell.2014.01.004. Cook, D. A. (2008). Scores From Riding’s Cognitive Styles Analysis Have Poor Test–Retest Reliability. Teaching and Learning in Medicine, 20(3), 225–229. doi:10.1080/10401330802199492. Cools, E., & Van den Broeck, H. (2007). Development and Validation of the Cognitive Style Indicator. The Journal of Psychology, 141(4), 359–387. doi:10.3200/jrlp.141.4.359- 388. Cools, E., Armstrong, S. J., & Verbrigghe, J. (2013). Methodological practices in cognitive style research: Insights and recommendations from the field of business and psychology. European Journal of Work and Organizational Psychology, 23(4), 627– 641. https://doi.org/10.1080/1359432x.2013.788245. Cooperman, E. (1980). Field differentiation and intelligence. The Journal of Psychology, 105(1), 29–33. 51 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Cuneo, F., Antonietti, J. P., & Mohr, C. (2018). Unkept promises of cognitive styles: A new look at old measurements. PloS one, 13(8), e0203115. https://doi.org/10.1371/journal.pone.0203115 Curry, L. (1983). An organisation of learning styles theory and constructs. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec, Canada. Retrieved from: https://files.eric.ed.gov/fulltext/ED235185.pdf. Curry, L. (1990). A critique of the research on learning styles. Educational Leadership, 48, 50–52. Czech Statistical Office (2019). Population structure by sex, age and educational attainment. Retrieved from: https://www.czso.cz/documents/10180/120583268/300002200102.pdf/ef2fb63c-7a0f- 424f-b5f2-e5360ab32d57?version=1.1. Dale, G., & Arnell, K. M. (2013). Investigating the stability of and relationships among global/local processing measures. Attention, Perception, & Psychophysics, 75, 394– 406. Dale, G., & Arnell, K. M. (2014). Lost in the forest, stuck in the trees: dispositional global/local bias is resistant to exposure to high and low spatial frequencies. PloS One, 9(7), e98625. Davidoff, J., Fonteneau, E., & Fagot, J. (2008). Local and global processing: Observations from a remote culture. Cognition, 108(3), 702–709. doi:10.1016/j.cognition.2008.06.004. De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.00102 De Fockert, J. W., & Cooper, C. (2014) Higher levels of depression are associated with reduced global bias in visual processing. Cognition and Emotion, 28(3), 541-549. Duan, Z., Wang, F., & Hong, J. (2016). Culture shapes how we look: Comparison between Chinese and African university students. Journal of Eye Movement Research, 9(6), 1- 10. Duncker, K. (1929). Uber induzierte Bewegung [On induced motion]. Psychologische Forschung, 12, 180-259. 52 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Dworak, E. M., Revelle, W., Doebler, P., & Condon, D. M. (2021). Using the International Cognitive Ability Resource as an open source tool to explore individual differences in cognitive ability. Personality and Individual Differences, 169, 109906. https://doi.org/10.1016/j.paid.2020.109906. Evans, C., & Waring, M. (2012). Application of styles in educational instruction and assessment. In L. F. Zhang, R. J. Sternberg, & S. Rayner (Eds.), The handbook of intellectual styles (pp. 297–330). New York, NY: Springer. Evans, C., Richardson, J. T. E., & Waring, M. (2013). Field independence: Reviewing the evidence. British Journal of Educational Psychology, 83(2), 210–224. doi:10.1111/bjep.12015. Evans, K., Rotello, C. M., Li, X., & Rayner, K. (2009). Scene perception and memory revealed by eye movements and receiver-operating characteristic analyses: does a cultural difference truly exist?. Quarterly journal of experimental psychology (2006), 62(2), 276–285. https://doi.org/10.1080/17470210802373720 Faulkenberry, T. J. (2017). A Single-Boundary Accumulator Model of Response Times in an Addition Verification Task. Frontiers in Psychology, 8, 1225. https://doi.org/10.3389/fpsyg.2017.01225 Fischer, R., & Karl, J. A. (2019). A primer to (Cross-Cultural) multi-group invariance testing possibilities in R. Frontiers in psychology, 10, 1507. doi: 10.3389/fpsyg.2019.01507. Fitzgibbons, D., Goldberger, L., & Eagle, M. (1965). Field dependence and memory for incidental material. Perceptual and Motor Skills, 21(3), 743–749. https://doi.org/10.2466/pms.1965.21.3.743. Flexer, B. K., & Roberge, J. J. (1980). IQ, field dependence–independence, and the development of formal operational thought. Journal of General Psychology, 103(2), 191–201. https://doi.org/10.1080/00221309.1980.9920998. Fox, J.-P., Klotzke, K., & Simsek, A. S. (2021). LNIRT: An R Package for Joint Modeling of Response Accuracy and Times. ArXiv:2106.10144 [Stat]. http://arxiv.org/abs/2106.10144 Furnham, A. (1992). Personality and learning style: A study of three instruments. Personality and Individual Differences, 13(4), 429–438. doi:10.1016/0191-8869(92)90071-v. Galton, F. (1883). Inquiries into Human Faculty and Its Development. London: Macmillan. 53 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Gamer, M., Lemon, J., & Singh, I. F. P. (2019). irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.84.1. https://CRAN.R- project.org/package=irr. Gasper, K., & Clore, G. L. (2002). Attending to the Big Picture: Mood and Global Versus Local Processing of Visual Information. Psychological Science, 13(1), 34–40. doi:10.1111/1467-9280.00406 Gerlach, C., & Krumborg, J. R. (2014). Same, same — but different: On the use of Navon derived measures of global/local processing in studies of face processing. Acta Psychologica, 153, 28–38. doi:10.1016/j.actpsy.2014.09.004 Gerlach, Ch., & Poirel, N. (2018). Navon’s classical paradigm concerning local and global processing relates systematically to visual object classification performance. Scientific Reports, 8(1), 324. Gerlach, Ch., & Starrfelt, R. (2018). Global precedence effects account for individual differences in both face and object recognition performance. Psychonomic Bulletin & Review volume 25, 1365–1372. Goldstein, A. G., & Chance, J. E. (1965). Effects of practice on sex-related differences in performance on Embedded Figures. Psychonomic Science, 3(8), 361–362. https://doi.org/10.3758/BF03343180. Goodenough, R., & Witkin, H. (1977). Origins of field-dependent and field-independent cognitive styles. ETS Research Report Series, 1977(1), i-80. Gottschaldt, K. (1926). Über den Einfluss der Erfahrung auf die Wahrnehmung von Figuren [The influence of experience upon the perception of figures]. Psychologische Forschung, 8, 261–317. https://doi.org/10.1007/BF02411523 Graff, M. (2003). Learning from web-based instructional systems and cognitive style. British Journal Of Educational Technology, 34(4), 407-418. Grossmann, I., & Varnum, M. E. W. (2011). Social Class, Culture, and Cognition. Social Psychological and Personality Science, 2(1), 81–89. https://doi.org/10.1177/1948550610377119 Guisande, M.A., Páramo, M.F., Tinajero, C., & Almeida L.S. (2007). Field dependence- independence (FDI) cognitive style: An analysis of attentional functioning. Psicothema, 19(4), 572-577. 54 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Haaf, J. M., & Rouder, J. N. (2017). Developing constraint in Bayesian mixed models. Psychological Methods, 22(4), 779–798. https://doi.org/10.1037/met0000156 Hakim, N., Simons, D. J., Zhao, H., & Wan, X. (2017). Do easterners and westerners differ in visual cognition? A preregistered examination of three visual cognition tasks. Social Psychological and Personality Science, 8(2), 142–152. https://doi.org/10.1177/1948550616667613 Hayes, J., & Allinson., C. (1994). Cognitive style and its relevance for management practice. British Journal of Management, 5(1), 53-7 1. Hedge, C., Powell, G., & Sumner, P. (2017). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50, 1166–1186. Hřebíčková, M., Jelínek, M., Květon, P., Benkovič, A., Botek, M., Sudzina, F., Soto, C. J., & John, O. P. (2020). Big Five Inventory 2 (BFI-2): Hierarchický model s 15 subškálami. Československá psychologie, 64(4), 437-460. Huygelier, H., Van der Hallen, R., Wagemans, J., de-Wit, L., & Chamberlain, R. (2018). The Leuven Embedded Figures Test (L-EFT): Measuring perception, intelligence or executive function? PeerJ, 6, Article e4524. https://doi.org/10.7717/peerj.4524 Ishii, K. (2012). Culture and the mode of thought: A review. Asian Journal of Social Psychology, 16(2), 123–132. doi:10.1111/ajsp.12011 Istomin, K. V., Panáková, J., & Heady, P. (2014). Culture, perception, and artistic visualization: a comparative study of children's drawings in three Siberian cultural groups. Cognitive science, 38(1), 76–100. https://doi.org/10.1111/cogs.12051 James, W. (1890). The Principles of Psychology. London: Macmillan. Jansari, A., Miller, S., Pearce, L., Cobb, S., Sagiv, N., Williams, A. L., Tree, J. J., & Hanley, J. R. (2015). The man who mistook his neuropsychologist for a popstar: When configural processing fails in acquired prosopagnosia. Frontiers in Human Neuroscience, 9, 390. Jurkat, S., Köster, M., Yovsi, R., & Kärtner, J. (2020). The development of context-sensitive attention across cultures: the impact of stimulus familiarity. Frontiers in Psychology, 11, 1526. 55 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Ji, L., Peng, K., & Nisbett, R. (2000). Culture, control, and perception of relationships in the environment. Journal of Personality and Social psychology. 78(5), 943-955. Ji, L., Zhang, Z., & Nisbett, R. (2004). Is it culture or is it language? Examination of language effects in cross-cultural research on categorization. Journal of Personality and Social Psychology, 87(2), 57–65. Ji, L.-J., Nisbett, R. E., & Su, Y. (2001). Culture, change, and prediction. Psychological Science, 12(6), 450–456. https://doi. org/10.1111/1467-9280.00384. Ji, L.-J., Yap, S., Best, M. W., & McGeorge, K. (2019). Global Processing Makes People Happier Than Local Processing. In Frontiers in Psychology. Frontiers in Psychology, 10, 670. https://doi.org/10.3389/fpsyg.2019.00670 John, O. P., Naumann, L. P., & Soto, C. J. (2008). Paradigm Shift to the Integrative Big-Five Trait Taxonomy: History, Measurement, and Conceptual Issues. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (pp. 114-158). New York, NY: Guilford Press. Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2021). semTools: Useful tools for structural equation modeling. R package version 0.5-4. Retrieved from https://CRAN.R-project.org/package=semTools. Jung, C.G. (1923). Psychological Types. London: Routledge & Kegan Paul. Keller, J., & Ripoll, H. (2001). Reflective–impulsive style and conceptual tempo in a gross- motor task. Perceptual and Motor Skills, 92, 739 –749. Kepner, M. D., & Neimark, E. D. (1984). Test-retest reliability and differential patterns of score change on the group embedded figures test. Journal of personality and social psychology, 46(6), 1405–1413. https://doi.org/10.1037//0022-3514.46.6.1405. Kimchi, R., & Palmer, S. E. (1982). Form and texture in hierarchically constructed patterns. Journal of Experimental Psychology: Human Perception and Performance, 8(4), 521–535. Kimchi, R. (1992). Primacy of wholistic processing and global/local paradigm: A critical review. Psychological Bulletin, 112, 24–38. doi: 10.1037/0033-2909.112.1.24. 56 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Kitayama, S., Duffy, S., Kawamura, T., & Larsen, J. T. (2003). Perceiving an object and its context in different cultures: A cultural look at new look. Psychological Science, 14(3), 201-206. Kitayama, S., Ishii, K., Imada, T., Takemura, K., & Ramaswamy, J. (2006). Voluntary settlement and the spirit of independence: Evidence from Japan's ‘northern frontier’. Journal of Personality and Social Psychology, 91(3), 369–384. https://doi. org/10.1037/0022-3514.91.3.369 Kitayama, S., Park, H., Sevincer, A., Karasawa, M., & Uskul, A. (2009). A cultural task analysis of implicit independence: Comparing North America, Western Europe, and East Asia. Journal of Personality and Social Psychology, 97(2), 236-255. Kiyokawa, S., Dienes, Z., Tanaka, D., Yamada, A., & Crowe, L. (2012). Cross cultural differences in unconscious knowledge. Cognition, 124(1), 16-24. Klein, O., Ventura, P., Fernandes, T., Marques, L., Licata, L., & Semin, G. (2010), Effects of schooling and literacy on linguistic abstraction: The role of holistic vs. analytic processing styles. European Journal of Social Psychology, 40(7), 1095–1102. Koffka, K. (1935). Principles of Gestalt psychology. London: Routledge & Kegan Paul, Ltd. Kogan, N., & Saarni, C. (1990). Cognitive style in children: Some evolving trends. In O. N. Saracho (Ed.), Cognitive style and early education (pp. 3–31). New York, NY: Gordon & Breach. Koo, M., Choi, J. A., & Choi, I. (2018). Analytic versus holistic cognition: Constructs and measurement. In J. Spencer-Rodgers & K. Peng (Eds.), The psychological and cultural foundations of East Asian cognition: Contradiction, change, and holism (pp. 105–134). Oxford University Press. Kozhevnikov, M., Evans, C., & Kosslyn, S. (2014). Cognitive Style as Environmentally Sensitive Individual Differences in Cognition: A Modern Synthesis and Applications in Education, Business, and Management. Psychological Science in the Public Interest, 15(1), 3-33. Kozhevnikov, M. (2007). Cognitive styles in the context of modern psychology: toward an integrated framework of cognitive style. Psychological bulletin, 133(3), 464-481. Kuhn, M. H., & McPartland, T. S. (1954). An empirical investigation of self-attitudes. American Sociological Review, 19, 68–76. https://doi. org/10.2307/2088175 57 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Kühnen, U., Hannover, B., Roeder, U., Shah, A. A., Schubert, B., Upmeyer, A., & Zakaria, S. (2001). Cross-cultural variations in identifying embedded figures: Comparisons from the United States, Germany, Russia, and Malaysia. Journal of Cross-Cultural Psychology, 32(3), 365–371. https://doi.org/10.1177/0022022101032003007. Kyllonen, P. C., & Zu, J. (2016). Use of Response Time for Measuring Cognitive Ability. Journal of Intelligence, 4(4), 14. https://doi.org/10.3390/jintelligence4040014 Lacko, Čeněk, J., Šašinka, Č., Ugwitz, P., Šašinková, A., Lu, W., & Stachoň, Z. (2020). Souběžná validita metod měřících analytický a holistický kognitivní styl – předběžná analýza [Convergent validity of several methods measuring analytic and holistic cognitive style – preliminary analysis]. In E. Maierová, L. Viktorová, M. Dolejš, & T. Dominik (Eds.), PhD existence 10 "Člověk a čas": Česko-slovenská psychologická konference (nejen) pro doktorandy a o doktorandech (pp. 201-212). Olomouc: Palacký University. Lacko, D., Čeněk, J., Točík, J., Avsec, A., Đorđević, V., Genc, A., Haka, F., Šakotić- Kurbalija, J., Mohorić, T., Neziri, I., & Subotić, S. (2022). The Necessity of Testing Measurement Invariance in Cross-Cultural Research: Potential Bias in Cross-Cultural Comparisons With Individualism– Collectivism Self-Report Scales. Cross-Cultural Research, 56(2–3), 228–267. https://doi.org/10.1177/10693971211068971.Lacko, D., Šašinka, Č., Stachoň, Z., Lu, W., & Čeněk, J. (2020). Cross-Cultural Differences in Cognitive Style, Individualism/Collectivism and Map Reading between Central European and East Asian University Students. Studia Psychologica, 62(1). https://doi.org/10.31577/sp.2020.01.789. Lacko, D. (2018). Individuální a interkulturní rozdíly ve vnímání a myšlení [The individual and intercultural differences in perception and cognition]. Diploma thesis, Masaryk University. Lakens, D. (2017). Equivalence tests: A practical primer for t-tests, correlations, and meta- analyses. Social Psychological and Personality Science, 8(4), 355-362. doi: 10.1177/1948550617697177. Lee, L. Y., Talhelm, T., Zhang, X., Hu, B., & Lv, X. (2021). Holistic thinkers process divided-attention tasks faster: From the global/local perspective. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues. Advance online publication. https://doi.org/10.1007/s12144-021-01879-1 58 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Lis, D. J., & Powers, J. E. (1979). Reliability and validity of the Group Embedded Figures Test for a grade school sample. Perceptual and motor skills, 48(2), 660–662. https://doi.org/10.2466/pms.1979.48.2.660. Lo, S., & Andrews, S. (2015). To transform or not to transform: using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6, 1171. Loe, B. S., Sun, L., Simonfy, F., & Doebler, P. (2018). Evaluating an Automated Number Series Item Generator Using Linear Logistic Test Models. Journal of Intelligence, 6(2), 20. https://doi.org/10.3390/jintelligence6020020. Luce, R. D. (1986). Response Times. New York, NY: Oxford University Press. Ludwig, I., & Lachnit, H. (2004). Effects of practice and transfer in the detection of embedded figures. Psychological Research, 68,277-288. Lux, A. A., Grover, S. L., & Teo, S. T. T. (2021). Development and Validation of the Holistic Cognition Scale. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.551623. MacLeod, C., Jackson, R., & Palmer, J. (1986). On the relation between spatial ability and field dependence. Intelligence, 10(2), 141–151. Mair, P., &, Hatzinger, R. (2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20. https://www.jstatsoft.org/v20/i09. Masarikova, B. (2021). Vybrané metódy počítačového testovania kognitívneho štýlu [Selected methods of computer cognitive style testing]. Bachelor's thesis, Masaryk University. Massidda, D. (2013). retimes: Reaction time analysis [Computer software manual]. Retrieved from https://CRAN.R-project.org/package=retimes (R package version 0.1-2) Masuda, T., & Kitayama, S. (2004). Perceived-induced constraint and attitude attribution in Japan and in the US: A case for cultural dependence of the correspondence bias. Journal of Experimental Social Psychology, 40(3), 409–416. Masuda, T., & Nisbett, R. (2001). Attending holistically versus analytically: comparing the context sensitivity of Japanese and Americans. Journal of personality and social psychology, 81(5), 922-934. 59 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Masuda, T., & Nisbett, R. (2006). Culture and change blindness. Cognitive Science, 30(2), 381-399. Masuda, T., Ishii, K., & Kimura, J. (2016). When does the culturally dominant mode of attention appear or disappear? Comparing patterns of eye movement during the visual flicker task between European Canadians and Japanese. Journal of Cross-Cultural Psychology, 47(7), 997-1014. Matzke, D., & Wagenmakers, E.-J. (2009). Psychological interpretation of the ex-Gaussian and shifted Wald parameters: A diffusion model analysis. Psychonomic Bulletin & Review, 16(5), 798–817. https://doi.org/10.3758/PBR.16.5.798 Mavridis, P., Kärtner, J., Cavalcante, L. I. C., Resende, B., Schuhmacher, N., & Köster, M. (2020). the development of context-sensitive attention in urban and rural Brazil. Frontiers in psychology, 11, 1623.McKenna, F. P. (1984). Measures of field dependence: Cognitive style or cognitive ability? Journal of Personality and Social Psychology, 47(3), 593–603. doi:10.1037/0022-3514.47.3.593. McKone, E., Aimola Davies, A., Fernando, D., Aalders, R., Leung, H., Wickramariyaratne, T., & Platow, M. J. (2010). Asia has the global advantage: Race and visual attention. Vision Research, 50(16), 1540–1549. https://doi.org/10.1016/j.visres.2010.05.010¨ McCrae, R. R., Terracciano, A., & Personality Profiles of Cultures Project. (2005). Personality profiles of cultures: Aggregate personality traits. Journal of Personality and Social Psychology, 89(3), 407–425. https://doi.org/10.1037/0022-3514.89.3.407 Mealor, A. D., Simner, J., Rothen, N., Carmichael, D. A., & Ward, J. (2016). Different Dimensions of Cognitive Style in Typical and Atypical Cognition: New Evidence and a New Measurement Tool. PloS one, 11(5), e0155483. https://doi.org/10.1371/journal.pone.0155483. Messick, S. (1976). Personality consistencies in cognition and creativity. In Messick, S. (Ed.), Individuality in learning (pp. 4–23). San Francisco, CA: Jossey-Bass. Messick, S. (1984). The nature of cognitive styles: problems and promise in educational practice. Educational Psychologist, 19(2), 59-74. Mickley Steinmetz, K. R., Sturkie, C. M., Rochester, N. M., Liu, X., & Gutchess, A. H. (2017). Cross-cultural differences in item and background memory: examining the 60 VALIDATION OF METHODS MEASURING COGNITIVE STYLES influence of emotional intensity and scene congruency. Memory, 26(6), 751–758. doi:10.1080/09658211.2017.1406119 Miller, A. (1987). Cognitive styles: An integrated model. Educational Psychology, 7, 251– 268. Miller, A. (1991). Personality types, learning styles, and educational goals. Educational Psychology, 11, 217–238. Milfont, T. L., & Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of psychological research, 3(1), 111-130. doi: 10.21500/20112084.857. Milne, E., & Szczerbinski, M. (2009). Global and local perceptual style, field-independence, and central coherence: An attempt at concept validation. Advances in cognitive psychology, 5, 1–26. https://doi.org/10.2478/v10053-008-0062-8 Miyake, A., Witzki, A. H., & Emerson, M. J. (2001). Field dependence–independence from a working memory perspective: A dual-task investigation of the Hidden Figures Test. Memory, 9(4-6), 445–457. https://doi.org/10.1080/09658210143000029. Miyamoto, Y., & Kitayama, S. (2002). Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior. Journal of Personality and Social Psychology, 83(5), 1239–1248. Miyamoto, Y., Yoo, J., Levine, C. S., Park, J., Boylan, J. M., Sims, T., Markus, H. R., Kitayama, S., Kawakami, N., Karasawa, M., Coe, C. L., Love, G. D., & Ryff, C. D. (2018). Culture and social hierarchy: Self- and other-oriented correlates of socioeconomic status across cultures. Journal of Personality and Social Psychology, 115(3), 427–445. https://doi.org/10.1037/pspi0000133 Molenaar, D., Tuerlinckx, F., & van der Maas, H. L. J. (2015). Fitting Diffusion Item Response Theory Models for Responses and Response Times Using the R Package diffIRT. Journal of Statistical Software, 66(4), 1–34. https://doi.org/10.18637/jss.v066.i04. Moran, A. P. (1983). An Irish Psychometric Appraisal of the Group Embedded Figures Test. Perceptual and Motor Skills, 57(2), 647–648. doi:10.2466/pms.1983.57.2.647. Moran, A. P. (1985). Unresolved issues in research on field dependence-independence. Social Behavior and Personality: an international journal, 13(2), 119-124. 61 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Morris, M., & Peng, K. (1994). Culture and cause: American and Chinese attributions for social and physical events. Journal of Personality and Social Psychology, 67(6), 949- 971. Moscoso del Prado Martin, F. (2009). A Theory of Reaction Time Distributions. (Unpublished). Retrieved from: http://cogprints.org/6326/. Na, J., Grossmann, I., Varnum, M. E. W., Kitayama, S., Gonzalez, R., & Nisbett, R. E. (2010). Cultural differences are not always reducible to individual differences. Proceedings of the National Academy of Sciences, 107(14), 6192–6197. doi:10.1073/pnas.1001911107 Na, J., Grossmann, I., Varnum, M. E. W., Karasawa, M., Cho, Y., Kitayama, S., & Nisbett, R. E. (2019). Culture and personality revisited: Behavioral profiles and within-person stability in interdependent (vs. independent) social orientation and holistic (vs. analytic) cognitive style. Journal of Personality, 0(0), 1–17. Na, J., McDonough, I. M., Chan, M. Y., & Park D. C. (2016) Social-class differences in consumer choices: Working-class individuals are more sensitive to choices of others than middle-class individuals. Personality and Social Psychology Bulletin, 42(4), 430- 443. doi: 10.1177/0146167216634043 Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9(3), 353-383. Navon, D. (1981). The forest revisited: More on global precedence. Psychological Research, 43(1), 1-32. Navon, D. (2003). What does a compound letter tell the psychologist’s mind? Acta Psychologica, 114(3), 273-309. Niaz, M. (1987). Mobility-Fixity Dimension in Witkin's Theory of FieldDependence/Independence and its Implications for Problem Solving in Science. Perceptual and Motor Skills, 65(3), 755–764. Nisbett, R., & Masuda, T. (2003). Culture and point of view. Proceedings of the National Academy of Sciences, 100(19), 11163-11170. Nisbett, R., & Miyamoto, Y. (2005). The Influence of Culture: Holistic versus Analytic Perception. Trends in Cognitive Sciences, 9(10), 467-473. 62 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Nisbett, R. (2003). The geography of thought: How asians and westerners think differently… and why. New York: The Free Press. Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: holistic versus analytic cognition. Psychological review, 108(2), 291–310. https://doi.org/10.1037/0033-295x.108.2.291 Norenzayan, A., Smith, E., Kim, B., & Nisbett, R. (2002). Cultural preferences for formal versus intuitive reasoning. Cognitive Science, 26(5), 653-684. Nosal, C. S. (1990). Psychologiczne modele umyslu [Psychological models of mind]. Warsaw, Poland: PWN. Oishi, S., Jaswal, V. K., Lillard, A. S., Mizokawa, A., Hitokoto, H., & Tsutsui, Y. (2014). Cultural variations in global versus local processing: A developmental perspective. Developmental Psychology, 50(12), 2654–2665. Opach, T., Popelka, S., Doležalová, J., & Rod, J. K. (2018). Star and polyline glyphs in a grid plot and on a map display: Which perform better?. Cartography and Geographic Information Science, 45(5), 400-419.Palmer, E. M., Horowitz, T. S., Torralba, A., & Wolfe, J. M. (2011). What are the shapes of response time distributions in visual search? Journal of Experimental Psychology: Human Perception and Performance, 37(1), 58–71. Páramo, M. F., & Tinajero, C. (1990). Field dependence/independence and performance in school: an argument against neutrality of cognitive style. Perceptual and motor skills, 70(3 Pt 2), 1079–1087. https://doi.org/10.2466/pms.1990.70.3c.1079. Parkinson, A., Mullally, A. A. P., & Redmond, J. A. (2004). Test–retest reliability of Riding’s cognitive styles analysis test. Personality and Individual Differences, 37(6), 1273–1278. doi:10.1016/j.paid.2003.12.012 Peng, K., & Nisbett, R. E. (1999). Culture, dialectics, and reasoning about contradiction. American Psychologist, 54(9), 741–754. https://doi.org/10.1037/0003-066X.54.9.741 Peterson, E. R., & Deary, I. J. (2006). Examining wholistic–analytic style using preferences in early information processing. Personality and Individual Differences, 41(1), 3-14. Peterson, E. R., & Meissel, K. (2015). The effect of Cognitive Style Analysis (CSA) test on achievement: A meta-analytic review. Learning and Individual Differences, 38, 115– 122. https://doi.org/10.1016/j.lindif.2015.01.011 63 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Peterson, E. R., Deary, I. J., & Austin, E. J. (2003). The reliability of Riding’s Cognitive Style Analysis test. Personality and Individual Differences, 34(5), 881–891. https://doi.org/10.1016/s0191-8869(02)00116-2. Peterson, E. R., Deary, I. J., & Austin, E. J. (2005). Are intelligence and personality related to verbal-imagery and wholistic-analytic cognitive styles? Personality and Individual Differences, 39(1), 201-213. Peterson, E. R., Rayner, S. G., & Armstrong, S. J. (2009). Researching the psychology of cognitive style and learning style: Is there really a future? Learning and Individual Differences, 19(4), 518–523. https://doi.org/10.1016/j.lindif.2009.06.003. Peterson, E. R. (2005). Verbal Imagery Cognitive Styles Test & Extended Cognitive Style Analysis-Wholistic Analytic Test: Administration Guide. University of Edinburgh. Pitta-Pantazi, D., & Christou, C. (2009). Cognitive styles, task presentation mode and mathematical performance. Research in Mathematics Education, 11(2), 131-148. Poirel, N., Pineau, A., & Mellet, E. (2006). Implicit identification of irrelevant local objects interacts with global/local processing of hierarchical stimuli. Acta Psychologica, 122(3), 321-336. Poirel, N., Pineau, A., Jobard, G., & Mellet, E. (2008). Seeing the forest before the trees depends on individual field-dependency characteristics. Experimental Psychology, 55(5), 328–333. https://doi.org/10.1027/1618-3169.55.5.328 Popelka, S., Stachoň, Z., Šašinka, Č., & Doležalová, J. (2016). EyeTribe Tracker Data Accuracy Evaluation and Its Interconnection with Hypothesis Software for Cartographic Purposes. Computational intelligence and neuroscience, 2016, 9172506. https://doi.org/10.1155/2016/9172506. Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510–532. Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion Decision Model: Current Issues and History. Trends in Cognitive Sciences, 20(4), 260–281. https://doi.org/10.1016/j.tics.2016.01.007 Rayner, K., Castelhano, M. S., & Yang, J. (2009). Eye movements when looking at unusual/weird scenes: Are there cultural differences? Journal of Experimental 64 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Psychology: Learning, Memory, and Cognition, 35(1), 254–259. https://doi.org/10.1037/a0013508 Rayner, K., Li, X., Williams, C. C., Cave, K. R., & Well, A. D. (2007). Eye movements during information processing tasks: individual differences and cultural effects. Vision research, 47(21), 2714–2726. https://doi.org/10.1016/j.visres.2007.05.007 R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. Reger, G. M., McGee, J. S., van der Zaag, C., Thiebaux, M., Buckwalter, J. G., & Rizzo, A. A. (2003). A 3D Virtual Environment Rod and Frame Test: The Reliability and Validity of Four Traditional Scoring Methods for Older Adults. Journal of Clinical and Experimental Neuropsychology, 25(8), 1169–1177. https://doi.org/10.1076/jcen.25.8.1169.16733 Rémy, L., & Gilles, P.-Y. (2014). Relationship between field dependence-independence and the g factor: What can problem-solving strategies tell us? Revue Européenne de Psychologie Appliquée/European Review of Applied Psychology, 64(2), 77–82. doi:10.1016/j.erap.2014.02.001. Rensink, R. A., O’Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 368-373. Revelle, W., Dworak, E. M., & Condon, D. (2020). Cognitive Ability in Everyday Life: The Utility of Open-Source Measures. Current Directions in Psychological Science, 29(4), 358–363. https://doi.org/10.1177/0963721420922178. Revelle, W. (2021). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. R package version 2.1.3, https://CRAN.R-project.org/package=psych. Rezaei, A. R., & Katz, L. (2004). Evaluation of the reliability and validity of the cognitive styles analysis. Personality and Individual Differences, 36(6), 1317–1327. https://doi.org/10.1016/s0191-8869(03)00219-8 Ridding, R., & Rayner, S. (1998). Cognitive Styles and Learning Strategies Understanding Style Differences in Learning and Behavior. London: David Fulton Publishers. Ridding, R. & Cheema, I. (1991). Cognitive Styles: An Overview and Integration. Educational Psychology, 11(3/4), 193-216. 65 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Riding, R. (1991). Cognitive Style Analysis - CSA administration. Birmingham: Learning & Training and Technology. Riding, R. (1997). On the Nature of Cognitive Style. Educational Psychology, 17(1-2), 29– 49. doi:10.1080/0144341970170102 Riding, R. J., & Pearson, F. (1994). The relationship between cognitive style and intelligence. Educational Psychology, 14(4), 413–425. https://doi.org/10.1080/0144341940140404. Riding, R. J., & Wigley, S. (1997). The relationship between cognitive style and personality in further education students. Personality and Individual Differences, 23(3), 379–389. doi:10.1016/s0191-8869(97)80003-7 Rittschof, K. A. (2010). Field dependence–independence as visuospatial and executive functioning in working memory: implications for instructional systems design and research. Educational Technology Research and Development, 58(1),99-114. Rock, I. (1992). Comment on Asch and Witkin’s “Studies in space orientation II.” Journal of Experimental Psychology: General, 121(4), 404–406. doi:10.1037/0096- 3445.121.4.404 Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. https://www.jstatsoft.org/v48/i02/. Rouder, J. N., Province, J. M., Morey, R. D., Gomez, P., & Heathcote, A. (2015). The Lognormal Race: A Cognitive-Process Model of Choice and Latency with Desirable Psychometric Properties. Psychometrika, 80(2), 491–513. https://doi.org/10.1007/s11336-013-9396-3 Rousselet, G. A., & Wilcox, R. R. (2020). Reaction times and other skewed distributions: problems with the mean and the median. Meta-Psychology, 4. https://doi.org/10.15626/MP.2019.1630) Rozin, P., Moscovitch, M., & Imada, S. (2016). Right: Left:: East: West. Evidence that individuals from East Asian and South Asian cultures emphasize right hemisphere functions in comparison to Euro-American cultures. Neuropsychologia, 90, 3-11. Sadler-Smith, E., Spicer, D. P., & Tsang, F. (2000). Validity of the Cognitive Style Index: Replication and Extension. British Journal of Management, 11(2), 175–181. doi:10.1111/1467-8551.t01-1-00159 66 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Šašinka, Č., Lacko, D., Čeněk, J., Popelka, S., Ugwitz, P., Řádová, H., Fabianová, M., Šašinková, A., Brančík, J., & Jankovská, M. (2021). ImGo: A Novel Tool for Behavioral Impulsivity Assessment Based on Go/NoGo Tasks. Psychological reports, 332941211040431. Advance online publication. https://doi.org/10.1177/00332941211040431 Šašinka, Č., Morong, K., & Stachoň, Z. (2017). The Hypothesis platform: An Online tool for experimental research into work with maps and behavior in electronic environments. ISPRS International Journal of Geo-Information, 6(12), 1-22. Šašinka, Č., Stachoň, Z., Kubíček, P., Tamm, S., Matas, A., & Kukaňová, M. (2018). The Impact of global/local bias on task-solving in map-related tasks employing extrinsic and intrinsic visualization of risk uncertainty maps. The Cartographic Journal, 55(4), 1-17. Schramm, P., & Rouder, J. (2019). Are Reaction Time Transformations Really Beneficial? PsyArXiv. https://doi.org/10.31234/osf.io/9ksa6 Senzaki, S., Masuda, T., & Ishii, K. (2014). When is perception top‐down and when is it not? Culture, narrative, and attention. Cognitive Science, 38(7), 1493-1506. Shedden, J., & Reid, G. (2001). A variable mapping task produces symmetrical interference between global information and local information. Perception & Psychophysics, 63(2), 241-252. Sherman, R. A. (2015). multicon: Multivariate Constructs. R package version 1.6. https://CRAN.R-project.org/package=multicon Soto, C. J., & John, O. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of personality and social psychology, 113(1), 117–143. https://doi.org/10.1037/pspp0000096. Stachoň, Z., Šašinka, Č., Čeněk, J., Štěrba, Z., Angsuesser, S., Fabrikant, S. I., Štampach, R., & Morong, K. (2018). Cross-cultural differences in figure–ground perception of cartographic stimuli. Cartography and Geographic Information Science, 46(1), 82– 94. https://doi.org/10.1080/15230406.2018.1470575 67 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Steingroever, H., Wabersich, D., & Wagenmakers, E.-J. (2021). Modeling across-trial variability in the Wald drift rate parameter. Behavior Research Methods, 53(3), 1060– 1076. https://doi.org/10.3758/s13428-020-01448-7 Sternberg, R., & Grigorenko, E. (1997). Are cognitive styles still in style? American Psychologist, 52(7), 700-712. Sternberg, R., & Zhang, L. (2005). Styles of thinking as a basis of differentiated instruction. Theory into Practice, 44(3), 245-253. Sternberg, R. J., Wagner, R. K., & Zhang, L. F. (2003). Thinking Styles Inventory-Revised I. Unpublished test. Yale University. Sternberg, R. J., Wagner, R. K., & Zhang, L. F. (2007). Thinking Styles Inventory–Revised II (Unpublished test). Tufts University, Medford, MA. Sternberg, R. J. (1997). Thinking styles. New York: Cambridge University Press The International Cognitive Ability Resource Team (2014): http://icar-project.com/. Tiedemann, J. (1989). Measures of cognitive styles: A critical review. Educational Psychologist, 24(3), 261-275. Tinajero, C., & Páramo, M. F. (1997). Field dependence–independence and academic achievement: A re-examination of their relationship. British Journal of Educational Psychology, 67(2), 199–212. https://doi.org/10.1111/j.2044-8279.1997.tb01237.x. Torrance, E.P., McCarthy, B. & Kolesinski, M.T. (1988). Style of Learning and Thinking. Bensenville, IL: Scholastic Testing Service, Inc. Vance, C. M., Groves, K. S., Paik, Y., & Kindler, H. (2007). Understanding and Measuring Linear–NonLinear Thinking Style for Enhanced Management Education and Professional Practice. Academy of Management Learning & Education, 6(2), 167– 185. doi:10.5465/amle.2007.25223457 van der Maas, H. L. J., Molenaar, D., Maris, G., Kievit, R. A., & Borsboom, D. (2011). Cognitive psychology meets psychometric theory: on the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118(2), 339–356. https://doi.org/10.1037/a0022749 van der Linden, W. J. (2007). A Hierarchical Framework for Modeling Speed and Accuracy on Test Items. Psychometrika, 72(3), 287. https://doi.org/10.1007/s11336-006-1478-z 68 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Van Fleet, T., Hoang-duc, A., DeGutis, J., Robertson, L. (2011). Modulation of non-spatial attention and the global/local processing bias. Neuropsychologia, 49(3), 352-359. Van Lier, R. (1999). Investigating global effects in visual occlusion: from a partly occluded square to the back of a tree-trunk. Acta Psychologica, 102(2–3), 203-220. Van Zandt, T. (2000). How to fit a response time distribution. Psychonomic Bulletin & Review, 7, 424–465. Van Zandt, T. (2002). Analysis of Response Time Distributions. In Stevens' Handbook of Experimental Psychology, H. Pashler (Ed.). Vernon, P. E. (1972). The distinctiveness of field independence. Journal of Personality, 40(3), 366–391. https://doi.org/10.1111/j.1467-6494.1972.tb00068.x. von Mühlenen, A., Bellaera, L., Singh, A., & Srinivasan, N. (2018). The effect of sadness on global-local processing. Attention, perception & psychophysics, 80(5), 1072–1082. https://doi.org/10.3758/s13414-018-1534-7 Von Wittich, D., & Antonakis, J. (2011). The KAI cognitive style inventory: Was it personality all along? Personality and Individual Differences, 50(7), 1044–1049. doi:10.1016/j.paid.2011.01.022. Voss, A., Nagler, M., & Lerche, V. (2013). Diffusion Models in Experimental Psychology. Experimental Psychology, 60(6), 385–402. https://doi.org/10.1027/1618- 3169/a000218 Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh, M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure–ground organization. Psychological Bulletin, 138(6), 1172–1217. https://doi.org/10.1037/a0029333. Wagenmakers, E.-J., Van Der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ- diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14(1), 3–22. https://doi.org/10.3758/BF03194023. Waxman, S. R., Fu, X., Ferguson, B., Geraghty, K., Leddon, E., Liang, J., & Zhao, M. F. (2016). How early is infants' attention to objects and actions shaped by culture? New evidence from 24-month-olds raised in the US and China. Frontiers in Psychology, 7, 97. 69 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Weisz, J. R., O'Neill, P., & O'Neill, P. C. (1975). Field dependence-independence on the Children's Embedded Figures Test: Cognitive style or cognitive level? Developmental Psychology, 11(4), 539–540. https://doi.org/10.1037/h0076673. Wertheimer, M. (1922). Untersuchungen zur Lehre von der Gestalt. Psychologische Forschung, 1(1), 47–58. https://doi.org/10.1007/bf00410385. Whelan, R. (2008). Effective analysis of reaction time data. The Psychological Record, 58(3), 475–482. Widiger, T. A., Knudson, R. M., & Rorer, L. G. (1980). Convergent and discriminant validity of measures of cognitive styles and abilities. Journal of Personality and Social Psychology, 39(1), 116–129. https://doi.org/10.1037/0022-3514.39.1.116. Witkin, H., & Asch, S. (1948a). Studies in space orientation. III. Perception of the upright in the absence of a visual field. Journal of Experimental Psychology, 38(5), 603-614. Witkin, H., & Asch, S. (1948b). Studies in space orientation. IV. Further experiments on perception of the upright with displaced visual fields. Journal of Experimental Psychology, 38(6), 762-782. Witkin, H., & Berry, J. (1975). Psychological differentiation in cross-cultural perspective. ETS Research Bulletin Series, 1975(1), i-100. Witkin, H., & Goodenough, D. (1976). Field dependence and interpersonal behavior. ETS Research Report Series, 1976(1), i–78. Witkin, H., Moore, C., Goodenough, D., & Cox, P. (1977). Field dependent and field independent cognitive styles and their educational implications. Review of Educational Research, 47(1), 1–64. Witkin, H. (1949). Perception of body position and of the position of the visual field. Psychological Monographs: General and Applied, 63(7), i–46. https://doi.org/10.1037/h0093613. Witkin, H. (1979). Socialization, culture and ecology in the development of group and sex differences in cognitive style. Human Development, 22(5), 358–372. https://doi.org/10.1159/000272455. Wong, V. C., Wyer, R. S., Jr., Wyer, N. A., & Adaval, R. (2021). Dimensions of holistic thinking: Implications for nonsocial information processing across cultures. Journal 70 VALIDATION OF METHODS MEASURING COGNITIVE STYLES of Experimental Psychology: General, 150(12), 2636–2658. https://doi.org/10.1037/xge0001060 Young, S. R., Keith, T. Z., & Bond, M. A. (2019). Age and sex invariance of the International Cognitive Ability Resource (ICAR). Intelligence, 77, 101399. doi:10.1016/j.intell.2019.101399. Zhang, L. F., Sternberg, R. J., & Rayner, S. (Eds.). (2012). Handbook of intellectual styles: Preferences in cognition, learning, and thinking. New York, NY: Springer. Zhang, L.-F. (2002). Thinking Styles and the Big Five Personality Traits. Educational Psychology, 22(1), 17–31. https://doi.org/10.1080/01443410120101224. Zhang, L.-F. (2004). Field-dependence/independence: Cognitive style or perceptual ability?-- Validating against thinking styles and academic achievement. Personality and Individual Differences, 37(6), 1295–1311. https://doi.org/10.1016/j.paid.2003.12.015. Zhang, L.-F. (2006). Thinking styles and the big five personality traits revisited. Personality and Individual Differences, 40(6), 1177–1187. https://doi.org/10.1016/j.paid.2005.10.011. Zhang, L. F. (2013). The malleability of intellectual styles. New York, NY: Cambridge University Press. 71 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Appendix 1: Deviations from Pre-Registration Table S1A: Deviations from pre-registration Category Pre-registered Conducted Justification Sample size 500 participants. 392 participants. Although we were able to generate a pool of 600 participants exactly as pre-registered and motivated the participants with high financial reward, we were not able to gather data from all 500 participants, even after several reminders. However, 392 participants are considered still satisfactory in terms of statistical power and reliable estimation of parameters. Sample Partially represents Not representative. Even though our sample contained hard to characteristics Czech population. achieve cohorts and differed from traditionally used student populations, it cannot be considered even partially representative. Methods 24 items of ICAR Only 13 items of the As a result of a technical issue, one specific 3D rotation. 3D rotation subtest answer was not saved. This concerned items 1, were used. 6, 8, 10, 13, 15, 20 and 23. These were removed from analysis. Methods ICAR number series Omitted from As a result of a technical issue, many subtest. analysis. participants were not able to answer some number series. Because 44.16% of participants reported more than 50% of missing values, we removed this subtest from analysis. Data cleaning Removal of Person fit statistics We encountered several computational participants based on implemented in the problems during estimation of the Q-diffusion their person fits in LNIRT package, and IRT model. Respondents were extracted CFT1, CFT2, E- manual elimination according to the error line which specifies a CSA-WA. based on misfit case during estimation. When no indices computationally of the misfit case were given and the model problematic cases in was not estimated, we applied person fit estimating the statistics from the LNIRT package and diffusion IRT model. extracted participants on these indices. Then, we proceeded with estimation of the Q-model. Data cleaning Removal of Not conducted. This procedure was specified only for the AH multivariate outliers scale. Since the removal of multivariate outliers using Mahalanobis did not change the results of a very poor fit, we distance. did not include it in the article. Data cleaning Removal of Not conducted. This procedure was not suitable for most univariate outliers analyses. Because we performed a detailed pre- using the ±3*IQR registered data cleaning procedure, we removed rule. any other additional participants on the basis of this arbitrary rule. Data cleaning Removing RTs Removed RTs higher The pre-registered criterion was related only to higher than 5s than 10s in the case CFT1, CFT2 and E-CSA-WA and not to CFT3. of CFT3. Since we used LBA which requires RTs, we specified additional criteria purely for CFT3. We set this criterion to be higher (10s), because the instruction did not emphasise the reaction 72 VALIDATION OF METHODS MEASURING COGNITIVE STYLES speed and the answering was based on higher cognitive processing than in previous methods. RTs Ex-gaussian Application of Because our methods indicated a lack of estimations distributions, shifted Wald and variability in accuracy, we also applied hierarchical IRT Bayesian shifted additional RT estimations which we did not models and diffusion Wald models. know of during the process of pre-registration. IRT models. These new models fit the data well, and we therefore reported them in the main article. Data analysis Thurstonian IRT Hierarchical linear In questionnaires, Thurstonian IRT (TIRT) is model for the CFT3. ballistic accumulator applied in a forced-choice format. Because (LBA) simulation and empirical studies imply that at least five traits should be included to address the ipsativity of the score, we could not use this approach. However, the LBA was created to handle such situations and therefore we applied it. Data analysis Not specified. Bootstrapping with To obtain more accurate results and their 100,000 iterations. confidence intervals, bootstrapping where possible (i.e., split-half reliability and correlation analyses) was performed. Results Removal of methods ART and E-CSA- We kept these methods in additional analyses with ICC < .50 from WA continued to the since their stability was always only slightly further validation next validation below .50 for one subtest (ART analytic subtest phases due its low phases. ICC = .444, E-CSA-WA analytic subtest = stability in time. .433). Results Verification of Verification of We did not collect data on the educational level predictive validity predictive validity of the participants’ parents since this question through differences through differences was not specified during approval by the in social class in socioeconomic ethical committee. Socioeconomic status by (derived from the status. itself should be a significant predictor of AH. participant’s education, participant’s socioeconomic status and education level of parents. Results Verification of Analysis of the ANOVA was not used because we wanted to differences in social differences in simplify the results. We therefore reported only classes using socioeconomic status two directly compared groups of interest. ANOVA and using t-tests on two comparison of low extremely social class and high contrasting groups social class through (poor and lower mid post-hoc tests. SES vs. upper mid SES). 73 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Appendix 2: Evaluation of models fits Ex-gaussian distribution was evaluated according to a posterior predictive check. A comparison of observed and simulated data did not demonstrate a strong model fit (but may be considered satisfactory, see Figures S2A-S2F). Regarding the hierarchical IRT model, each item of every method (with the exception of CFT2 local) indicated a non-normal residual distribution based on the results of a Kolmogorov-Smirnov test, suggesting poor model of diffusion IRT models was assessed by Mr statistics implemented in the diffIRT package. We encountered severe computational issues in estimating this statistic. Even though this procedure was computationally infeasible due to the higher number of items for CFT2 and E-CSA-WA, it was safe to assume that the model fits for the Q-diffusion IRT model were unacceptable. This finding was also supported by QQ plots, which demonstrated significant inconsistencies between the observed and predicted RTs for each item separately. To evaluate the shifted Wald model, three diagnostic tools proposed by Anders et al. (2016) were applied and R functions by Faulkenberry (2017) were adopted. First, the QQ plot of the estimated and predicted deciles of RT was examined for each item. No curvature was detected, indicating a good model fit. However, the model slightly underestimates the middling deciles of CFT 2 global and local and ECSA global (see figures S2G-S2L). Second, the residual distribution plot of each decile was examined. The residual distributions of each decile indicated positive skewness in the data since the magnitude of the residuals increased with the magnitude of RT (see Figures S2A- S2F). Third, the residual summary statistics for each participant were obtained. The average residuals yielded increased values, however, the correlation coefficient between the standard deviation of residuals and average residuals was low, indicating a good model fit. Unfortunately, the model fit for the Bayesian shifted Wald model has not yet been implemented in R packages. We assume that its model fits will be very similar to the shifted Wald model. A summary of the results of all approaches are given in Table S2A. Table S2A: Evaluation of models fits Model Indicator CFT1 CFT2 E-CSA-WA Analytic Holistic Analytic Holistic Analytic Holistic LNIRT Kolmogorov-Smirnov Test 100% 100% 90% 100% 100% 100% (% of items that have non- normally distributed residuals) % of extreme residuals 0% 0% 0% 0% 0% 0% 74 VALIDATION OF METHODS MEASURING COGNITIVE STYLES % of misfitting items 0% 0% 0% 0% 0% 0% diffIRT M r 116237 29609 computationally infeasible df 116 116 computationally infeasible p-value < .001 < .001 computationally infeasible AIC 836 -270 26921 21499 9489 14989 BIC 1018 -90 27349 21925 9920 15420 Shifted-Wald ∆¯ 1.27 1.18 0.86 0.86 0.78 0.75 distribution σ x 213 204 615 593 476 753 ρ∆σ 0.08 0.04 -0.13 -0.16 -0.13 0.13 Figure S2A: Model fit of CFT 1 Global using posterior predictive check Figure S2B: Model fit of CFT 1 Local using posterior predictive check 75 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure S2C: Model fit of CFT 2 Global using posterior predictive check Figure S2D: Model fit of CFT 2 Local using posterior predictive check 76 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure S2E: Model fit of E-CSA-WA Holistic using posterior predictive check Figure S2F: Model fit of E-CSA-WA Analytic using posterior predictive check 77 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure S2G: Shifted-Wald model fit for CFT1 local Figure S2H: Shifted-Wald model fit for CFT1 global 78 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure S2I: Shifted-Wald model fit for CFT2 local Figure S2J: Shifted-Wald model fit for CFT2 global 79 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Figure S2K: Shifted-Wald model fit for E-CSA-WA local Figure S2L: Shifted-Wald model fit for E-CSA-WA global 80 VALIDATION OF METHODS MEASURING COGNITIVE STYLES 81 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Appendix 3: Robustness check Table S3A: Intraclass correlation coefficients for CFT1, CFT2 and E-CSA-WA. CFT 1 CFT 2 E-CSA-WA Local Global Local Global Analytic Holistic Raw RT: Median .900 .857 .871 .854 .795 .739 [.865, .926] [.807, .894] [.701, [.962, [.608, [.349, .868] .932] .919] .880] Ex-Gaussian: Tau .302 .255 .469 .332 .451 .639 [.052, [-.011, [.265, [.081, [.053, [.416, .765] .487] .451] .616] .516] .661] LNIRT: Theta .062 .057 .740 .845 .474 .475 [-.250, [-.274, [.639, [.786, [.268, [.270, .622] .298] .302] .813] .888] .622] diffIRT: Theta .168 .282 .456 .690 .340 .514 [-.117, [.030, .470] [.246, [.572, [.090, [.328, .649] .381] .608] .776] .523] Shifted-Wald: Drift .215 .492 .439 .389 .409 .463 [-.061, [.310, .626] [.225, [.158, [.084, [.250, .615] .420] .593] .557] .607] Bayesian shifted-Wald: .505 .538 .570 .616 .573 .411 Drift [.329, .671] [.373, .660] [.351, [.416, [.374, [-.039, .708] .740] .704] .634] Table S3B: Heterotrait-monotrait ratio of correlations of CFT2 and E-CSA-WA with personality Method Indicator extraversion agreeableness conscientiousness negative open emotionality mindedness CFT2: Raw RT: Me .101 .137 .060 .119 .134 Local Ex-Gaussian: .556 .390 .553 .389 .428 tau LNIRT: theta .242 .263 .269 .201 .230 diffIRT: .084 .156 .113 .092 .110 theta Shifted- .450 .416 .297 .273 .389 Wald: Drift Bayesian .237 .177 .172 .191 .294 shifted- Wald: Drift Raw RT: Me .102 .126 .060 .095 .136 82 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Ex-Gaussian: .398 .271 .261 .242 .347 tau LNIRT: theta .125 .165 .207 .113 .218 diffIRT: .080 .091 .090 .062 .075 CFT2: theta Global Shifted- .381 .342 .278 .336 .448 Wald: Drift Bayesian .191 .159 .147 .183 .256 shifted- Wald: Drift E-CSA- Raw RT: Me .144 .100 .085 .090 .122 WA: Local Ex-Gaussian: .196 .183 .172 .105 .138 tau LNIRT: theta .202 .210 .133 .158 .249 diffIRT: .166 .178 .127 .128 .193 theta Shifted- .150 .173 .151 .114 .211 Wald: Drift Bayesian .106 .129 .114 .101 .147 shifted- Wald: Drift E-CSA- Raw RT: Me .121 .086 .087 .070 .118 WA: Global Ex-Gaussian: .234 .200 .144 .189 .150 tau LNIRT: theta .272 .211 .252 .207 .293 diffIRT: .250 .204 .134 .141 .171 theta Shifted- .190 .170 .076 .120 .093 Wald: Drift Bayesian .163 .116 .076 .108 .118 shifted- Wald: Drift Table S3C: Discriminant validity of ART, CFT1 and CFT3 with personality Method Indicator extraversion agreeableness conscientiousness negative open emotionality mindedness ART: |∆M| .106 .075 .192 -.059 .052 Analytic 83 VALIDATION OF METHODS MEASURING COGNITIVE STYLES ART: |∆M| .077 .079 .064 -.038 -.021 Holistic CFT1: Raw RT: Me -.033 .154 .101 -.055 -.022 Local Ex-Gaussian: -.144 .046 .128 -.067 -.029 tau LNIRT: theta .111 -.087 .056 .012 .013 diffIRT: theta .132 -.097 .040 -.041 .019 Shifted-Wald: .131 -.012 -.066 -.010 -.016 Drift Bayesian .115 -.092 -.104 .021 -.019 shifted-Wald: Drift CFT1: Raw RT: Me -.056 .161 .106 -.048 -.050 Global Ex-Gaussian: .081 .027 .128 -.078 .019 tau LNIRT: theta .050 -.021 -.095 .027 .105 diffIRT: theta .072 .012 -.039 .009 .111 Shifted-Wald: -.008 .026 -.039 .010 .029 Drift Bayesian -.048 -.026 -.079 .009 -.002 shifted-Wald: Drift CFT3: Hierarchical -.137 -.086 -.080 .067 -.134 Analytic LBA: Drift CFT3: Hierarchical .056 -.090 .205 * -.061 .063 Holistic LBA: Drift Table S3D: Discriminant validity with intelligence Method Indicator matrix rotation ART: Analytic |∆M| -.228 -.248 ART: Holistic |∆M| -.247 -.360 * CFT1: Local Raw RT: Me -.248 -.175 Ex-Gaussian: tau -.124 -.116 LNIRT: theta .067 .101 84 VALIDATION OF METHODS MEASURING COGNITIVE STYLES diffIRT: theta .212 .127 Shifted-Wald: Drift .145 .100 Bayesian shifted-Wald: Drift .256 .161 CFT1: Global Raw RT: Me -.266 -.168 Ex-Gaussian: tau -.180 -.183 LNIRT: theta .121 .164 diffIRT: theta .128 .175 Shifted-Wald: Drift .164 .098 Bayesian shifted-Wald: Drift .204 .128 CFT2: Local Raw RT: Me .054 .115 Ex-Gaussian: tau .069 .030 LNIRT: theta .264 .362* diffIRT: theta .233 .361* Shifted-Wald: Drift -.039 .035 Bayesian shifted-Wald: Drift .029 .047 CFT2: Global Raw RT: Me .130 .157 Ex-Gaussian: tau .036 .100 LNIRT: theta .334 .278 diffIRT: theta .253 .248 Shifted-Wald: Drift -.048 .120 Bayesian shifted-Wald: Drift -.042 -.064 CFT3: Analytic Hierarchical LBA: Drift -.116 -.081 CFT3: Holistic Hierarchical LBA: Drift -.048 -.025 85 VALIDATION OF METHODS MEASURING COGNITIVE STYLES E-CSA-WA: Local Raw RT: Me .075 .086 Ex-Gaussian: tau .027 -.016 LNIRT: theta .296 .300 diffIRT: theta .181 .230 Shifted-Wald: Drift .016 .053 Bayesian shifted-Wald: Drift .057 .105 E-CSA-WA: Global Raw RT: Me -.006 -.049 Ex-Gaussian: tau .058 .020 LNIRT: theta .187 .236 diffIRT: theta .109 .099 Shifted-Wald: Drift -.033 .022 Bayesian shifted-Wald: Drift -.053 -.024 Table S3E: Multi Trait Multi Method Matrix for CFT1, CFT2, CFT3 and E-CSA-WA Indicator of RTs Same Trait-Different Same Method-Different Different Method-Different Method Trait Trait Raw RT: Me .213 .147 .254 Ex-Gaussian: tau .082 .345 .030 LNIRT: theta .038 .024 .077 diffIRT: theta .085 .030 .120 Shifted-Wald: Drift .048 .073 .121 Bayesian shifted-Wald: .113 .075 .153 Drift Table S3F: Spearman's rank correlation coefficients between related AH subtests Method Indicator CFT2 CFT3 ratio E-CSA-WA CFT1: Local Raw RT: Me .38 [.24, .48] *** -.01 [-.15, .13] .31 [.15, .39] *** Ex-Gaussian: tau .03 [-.09, .15] .03 [-.09, .14] .02 [-.10, .20] 86 VALIDATION OF METHODS MEASURING COGNITIVE STYLES LNIRT: theta -.06 [-.21, .07] .03 [-.11, .13] .05 [-.09, .17] diffIRT: theta .11 [-.05, .22] -.06 [-.18, .04] .16 [.03, .27] * Shifted-Wald: Drift -.05 [-.19, .08] -.01 [-.11, .13] .10 [-.06, .23] Bayesian shifted-Wald: Drift .09 [-.04, .24] .01 [-.10, .12] .18 [.05, .29] * CFT1: Global Raw RT: Me .33 [.22, .45] *** .00 [-.14, .12] .41 [.30, .49] *** Ex-Gaussian: tau .00 [-.14, .11] .01 [-.13, .12] .14 [.05, .26] * LNIRT: theta .16 [.04, .31] * -.03 [-.14, .07] -.11 [-.22, .01] diffIRT: theta .17 [.07, .28] *** -.02 [-.15, .20] .08 [-.03, .20] Shifted-Wald: Drift .10 [-.03, .24] -.04 [-.16, .09] .01 [-.12, .14] Bayesian shifted-Wald: Drift .08 [-.07, .23] -.03 [-.15, .10] .06 [-.06, .18] CFT2: Local Raw RT: Me - .02 [-.14, .17] .36 [.20, .47] *** Ex-Gaussian: tau - .01 [-.14, .17] .21 [.04, .33] * LNIRT: theta - .05 [-.08, .19] .22 [.07, .35] *** diffIRT: theta - .01 [-.11, .12] .30 [.16, .43] *** Shifted-Wald: Drift - -.03 [-.18, .09] .20 [.06, .33] *** Bayesian shifted-Wald: Drift - -.07 [-.22, .07] .26 [.13, .38] *** CFT2: Global Raw RT: Me - -.05 [-.18, .10] .32 [.14, .42] *** Ex-Gaussian: tau - .00 [-.15, .16] .19 [.03, .30] ** LNIRT: theta - -.06 [-.18, .04] .25 [.12, .37] *** diffIRT: theta - -.04 [-.14, .08] .17 [.05, .30] ** Shifted-Wald: Drift - .00 [-.12, .13] .16 [.04, .29] ** Bayesian shifted-Wald: Drift - .02 [-.12, .16] .23 [.12, .35] *** E-CSA-WA: Local Raw RT: Me - -.11 [-.23, .03] - 87 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Ex-Gaussian: tau - -.05 [-.17, .07] - LNIRT: theta - -.04 [-.15, .06] - diffIRT: theta - -.03 [-.20, .10] - Shifted-Wald: Drift - .06 [-.06, .19] - Bayesian shifted-Wald: Drift - .05 [-.07, .18] - E-CSA-WA: Global Raw RT: Me - -.10 [-.23, .04] - Ex-Gaussian: tau - -.03 [-.13, .08] - LNIRT: theta - -.09 [-.20, .01] - diffIRT: theta - .04 [-.08, .15] - Shifted-Wald: Drift - .06 [-.05, .17] - Bayesian shifted-Wald: Drift - .06 [-.06, .17] - Table S3G: Spearman's rank correlation coefficients between subtests within one measure Method Indicator Associations between subtests CFT1 Raw RT: Me .83 [.78, .87] *** Ex-Gaussian: tau .03 [-.08, .14] LNIRT: theta -.06 [-.17, .05] diffIRT: theta .15 [.04, .26] ** Shifted-Wald: Drift .16 [.04, .27] * Bayesian shifted-Wald: Drift .20 [.08, .33] ** CFT2 Raw RT: Me .89 [.86, .92] *** Ex-Gaussian: tau -.02 [-.12, .09] LNIRT: theta .55 [.48, .62] *** diffIRT: theta .49 [.40, .58] *** 88 VALIDATION OF METHODS MEASURING COGNITIVE STYLES Shifted-Wald: Drift .28 [.18, .38] *** Bayesian shifted-Wald: Drift .62 [.52, .68] *** CFT3 Hierarchical LBA: Drift -.38 [-.51, -.26] *** E-CSA-WA Raw RT: Me .83 [.78, .86] *** Ex-Gaussian: tau .50 [.39, .60] *** LNIRT: theta .55 [.43, .65] *** diffIRT: theta .44 [.35, .54] *** Shifted-Wald: Drift .45 [.32, .54] *** Bayesian shifted-Wald: Drift .56 [.46, .66] *** Table S3H: Differences between participants from lower and upper mid socioeconomic status at their level of AH Indicator Lower mid Upper mid t df p d M M CFT1: Local Raw RT: Me 1.139 0.996 4.13 81.95 < 0.823 .001 Ex-Gaussian: tau 0.186 0.124 2.28 72.29 .025 0.493 LNIRT: theta -0.013 -0.029 0.15 61.00 .880 0.035 diffIRT: theta 0.991 1.046 -0.61 65.40 .545 0.135 Shifted-Wald: Drift 2.844 3.289 -2.28 63.69 .026 0.519 Bayesian shifted-Wald: 4.134 4.327 -1.95 65.03 .028 .308 Drift CFT1: Global Raw RT: Me 1.012 0.874 3.28 81.80 .002 0.643 Ex-Gaussian: tau 0.171 0.177 -0.16 51.74 .871 0.039 LNIRT: theta -0.003 -0.059 0.74 46.55 .465 0.184 diffIRT: theta 0.963 0.905 0.61 48.61 .547 0.149 Shifted-Wald: Drift 2.956 3.021 -0.26 48.12 .798 0.064 Bayesian shifted-Wald: 3.602 3.618 -0.19 50.52 .851 0.046 Drift CFT2: Local Raw RT: Me 2.229 2.069 1.72 39.34 .093 0.475 Ex-Gaussian: tau 0.555 0.541 0.21 32.36 .834 0.062 89 VALIDATION OF METHODS MEASURING COGNITIVE STYLES LNIRT: theta 0.036 0.080 -0.62 44.98 .540 .163 diffIRT: theta 0.948 1.029 -0.69 46.88 .495 0.179 Shifted-Wald: Drift 1.712 1.699 0.11 40.30 .916 0.029 Bayesian shifted-Wald: 2.279 2.357 -0.72 36.27 .478 0.203 Drift CFT2: Global Raw RT: Me 2.244 2.062 1.82 38.04 .0763 0.509 Ex-Gaussian: tau 0.515 0.537 -0.43 50.30 .671 0.109 LNIRT: theta -0.005 -0.005 0.01 38.22 .992 0.003 diffIRT: theta 0.975 1.043 -0.60 32.84 .555 0.175 Shifted-Wald: Drift 1.711 1.743 -0.33 39.88 .744 0.090 Bayesian shifted-Wald: 2.339 2.472 -1.32 36.66 .194 0.375 Drift CFT3: Analytic Hierarchical LBA: Drift 0.812 0.774 0.33 68.92 .630 1 CFT3: Holistic Hierarchical LBA: Drift 2.285 2.372 -0.85 68.61 .200 1 E-CSA-WA: Raw RT: Me 1.533 1.328 2.27 75.33 .026 0.486 Local Ex-Gaussian: tau 0.678 0.584 1.447 69.19 .153 0.324 LNIRT: theta 0.001 -.006 0.82 64.70 .418 0.188 diffIRT: theta 1.032 1.015 0.25 65.71 .804 0.057 Shifted-Wald: Drift 1.994 2.131 -0.82 65.53 .418 0.186 Bayesian shifted-Wald: 2.681 2.837 -1.19 62.10 .240 0.276 Drift E-CSA-WA: Raw RT: Me 1.293 1.195 1.57 75.95 .120 0.331 Global Ex-Gaussian: tau 0.460 0.409 0.95 68.02 .344 0.215 LNIRT: theta -0.006 0.002 0.67 60.00 .503 0.158 diffIRT: theta 1.001 1.048 -0.78 56.46 .437 0.187 Shifted-Wald: Drift 1.435 1.568 -1.09 60.68 .280 0.255 Bayesian shifted-Wald: 2.045 2.272 -1.76 61.18 .084 0.410 Drift * Note: Unlike the main analyses, the two-tailed t-tests are reported here without Holm-Bonferroni correction. 90