Research methodology for ethnostatistics in organization studies: towards a historical ethnostatistics

Introduction

In the last couple of decades, demands for transparency, accountability and efficiency have resulted in the proliferation of measures crafted to evaluate the performance of both individuals and organizations. In management and organization studies, a specific focus on the complex dynamics of the construction, use and effects of quantitative measures has been proposed by ethnostatistics (Gephart, 2006, 1988). Ethnostatistics is “the study of the construction, interpretation, and display of statistics in quantitative social research” (Gephart, 1988, p. 9). Therefore, ethnostatistics is an attempt “to bridge the gap between quantitative and qualitative research” (Gephart, 1988, p. 6). Tracing its roots in Garfinkel's (1967) ethnomethodology, the ethnostatistics underlying assumptions rely on the premise that individuals have “native” or culturally specific systems and methods for comprehending and acting. Such sensemaking practices include tacit assumptions and informal practices used by actors in their everyday professional life which are somehow neglected and taken for granted by conventional social science. The object of enquiry of ethnostatistics is examining exactly those methods and practices and more specifically challenging the taken-for-granted nature of quantification. To put it simply, ethnostatistics seeks to describe what scientists or professionals actually do in the process of quantification (from data gathering, through data elaboration to final representation) and assumes a certain degree of variability in the undertaken practices and the practical reasoning employed by statistics producers which can ultimately lead to diverging results.

This underlying premise is elaborated in the three levels of ethnostatistical enquiry as described by Gephart (2009, 1988). First-level ethnostatistics (producing a statistic) studies how statistics are constructed by observing “the activities, meanings, and contexts involved in producing variables and statistics” (Gephart, 1988, p. 13). Therefore, it studies the cultural features of groups of scientists or professionals at work and in general the social enterprise of quantitative knowledge creation deployed by them. This level of analysis is concerned with producing a thick description of how a statistic is constructed focussing on “who said what to whom, how, when, where and why during the processes of counting social phenomena, assigning numbers or numerals to measure observations, and applying statistical analysis to measured observations” (Gephart, 2009, p. 90).

Second-level ethnostatistics (statistics at work) focusses on the assumptions and the appropriateness of the statistical techniques used by scientists and professionals. At this level, quantitative methods along with computer simulations are used to explore and understand the technical and practical assumptions employed in the production and use of statistics (Gephart, 1988). In particular, it addresses “how human and social features and contexts influence measurement outcomes, statistical results and findings” (Gephart, 2009, p. 92).

Third-level ethnostatistics (the rhetoric of statistics) treats quantitative documents (reports, analysis, etc.) as rhetoric. Rhetoric is defined as the “art of persuasion that involves production of an argument claiming validity for a particular audience” (Gephart, 1988, p. 47). At this level of analysis, it is assumed that the interpretation and use of statistics and measurement are an artful and strategic process by which numbers (and quantification in general) are used to create persuasion rather than an objective process aimed at reporting pure facts.

It is clear that such critical approach to quantification can grant certain practical benefits on both business and academic level, such as improving the effectiveness and validity of measurement. However, the method never became a mainstream in organizational research despite Gephart's invitation (2009). Some rare representative exceptions employing ethnostatistical approach in management and organization research have appeared, such as Landrum and Boje's (2008) critical examination of the 1997 study released by Nike to resolve wage controversies in subcontracted Asian factories; Boje et al.'s (2006) investigation of the widely publicized failure of Enron Corporation; Mills et al.'s (2006) case of Canadian business schools and universities’ usage of comparative rankings and performance measures to signal to audiences about selected features and characteristics of their institutions; Carlon et al.'s (2006) clever explanation of the link between executives' compensation plans and financial performance by uncovering how compensation strategies have become fetishes; and Kilduff and Oh's (2006) examination of the scholarly organizational discourse related to the seminal study “Medical Innovation: A Diffusion Study” by Coleman et al. (1966) (see the following dedicated section for a full literature review).

While some of those valuable contributions are great examples of the application of ethnostatistics in organizational research, they have some limitations in terms of the “by definition” use of the ethnostatistical method. They are not in fact producing real-time ethnographies (first level of ethnostatistics) in their analysis and instead rely on “quasi-historical” methods to reconstruct the contexts in which events were taking place. The reasons for such drawbacks are obvious and can be sought on a pure practical level falling under either of the following conditions: 1) in order for an event to become an object of ethnostatistical enquiry, it should represent an instance of controversy and often major controversial cases or scandals are identified as such after a substantial time lag making a real-time ethnography practically impossible to conduct; 2) even in the absence of a critical event, the access to organizational sites can be extremely difficult (potentially more difficult than usual in organizational ethnographies) due to the intrinsic nature of the ethnostatistical enquiry to focus on organizational performance measures and indicators which could represent relatively sensitive topic for any organization.

Although we can easily accept that such practical constraints represent major hurdles in the “by definition” application of the ethnostatistical method, we argue that the use of historical data as substitutions for conducting real-time ethnographies must be done with caution as it poses additional methodological concerns. We accept that in the case of practical impossibility to conduct real-time ethnographies, there is no other option than to resort to historical (secondary) data. Whilst we draw widely on the literature arguing that organizational scholars and historians can approach historical data in different ways, we find that resorting to historical methods (in the ways in which historians do) could be a way of overcoming some limitations existing in ethnostatistics. We argue that the use of organizational histories (which historians define as secondary sources) must be done with caution as they tend to be used for specific organizational purposes in the present being part of a wider organizational rhetoric. Further, archival materials (primary sources) can highlight elements of metrics and measurement construction that are not evident in secondary sources. However, a correct use of such sources requires the methodological awareness of their limitation and partiality. All historical sources (both primary and secondary) are, in fact, secondary data for organizational researchers, that is, they are not constructed by the researcher to answer their specific research questions (Rowlinson et al., 2014, pp. 255-258). Archival materials are evidences that need to be interpreted in order to provide meaningful clues to reply to the researcher's queries. While acknowledging that other disciplines (e.g. anthropology) have already paid specific attention to these issues, we wish to magnify this important argument through the lenses of organization and management scholarship, thus contributing to a greater understanding of the problems related to hybridization of methods in general.

To address this matter, this study aims at discussing and proposing a pluralistic method for conducting ethnostatistical research in live organizational settings by presenting practical techniques that could overcome the obstacles that ethnostatistics admittedly met with. Furthermore, as such it attempts to contribute at a methodological level to the ethnostatistical field by discussing the intersection between history and ethnography while suggesting ways for their complementary use in organization and management research. To accomplish these purposes, we first discuss in more depth the three levels of ethnostatistics and its underlying assumptions. Then we review how this method has been applied by management and organization scholars. Followed by this, we discuss the methodological limitations of the ethnostatistical method. Finally, we propose a pluralistic method (model) for conducting ethnostatistical research in organizational settings based on the integration of 1) research practices employed by one of the authors conducting ethnostatistical research in a large multinational organization and 2) best practices from ethnographic and historical research. The paper concludes by outlining the potential benefits from the use of the suggested pluralistic method in the organization and management research domains.

The ethnostatistical method and its underlying assumptions

As previously described, the goal of ethnostatistics is to “describe, analyze, explain and understand how statistics are actually accomplished and used in the research process” (Gephart, 1988, p. 10). This can be achieved by examining three distinct levels in the process of quantification, that is, the construction, interpretation and final presentation of statistics. The emergence of ethnostatistics can be traced back in the late 1980s, in Gephart's seminal work (1988), with the recognition that while statistics as a discipline offers formal rules and guidelines for producing measures and statistics, the actual production of statistics is not exempt from informal practices, tacit assumptions and conventions which are used by professionals but lie outside the formal statistical and measurement rules. Furthermore, such practices are somehow inseparable and needed in the everyday practice of constructing and elaborating numerals. To give more precise examples of what is meant by informal practices, conventions and tacit assumptions inherent in the production of quantitative knowledge, in what follows we will describe how those practices can be found on each level of constructing quantitative data and how this may impact its final outcome.

In first-order ethnostatistical analysis, which explores the “how” of creating quantitative knowledge (Gephart, 2009), informal practices and conventions can be discovered in the data gathering process. For example, Gubrium and Buckholdt (1979), in a classic field work on the production of hard data in social service institutions, found that in the process of recording patients' behaviour, not every event that should be countable gets counted but instead countable events become recorded after being “interpreted” by the staff. To illustrate that, the authors describe cases in which a patient's act of non-normal behaviour was chosen not to be counted by the staff because the patient had the motivation to behave “normally” but did not due to some extraordinary circumstances and therefore such abnormal behaviour was not accounted for in the records. It can be seen from the example that the practical reasoning and the in-depth contextual knowledge of the data gatherers (in this case the nurses) impose a certain degree of imprecision in the collected data.

In second-order ethnostatistics, certain technical and practical assumptions (and conventions) can be identified in the process of elaborating data (i.e. producing statistics). Such assumptions can include the appropriateness of using statistical procedures with data measured using particular types of scales (nominal, ordinal, interval and ratio). For example, at a technical level, it is often assumed that Likert-type scaling, which offers subjects ordered choices (e.g. strongly agree to strongly disagree on a 1–5- or 7-point scales in a questionnaire), is appropriate. Analysts often use parametric strategy (a means for transforming ordinal data into interval-level measures) to treat Likert-type responses as interval data. On a practical level, it is assumed that subjects can consistently and reliably assign numerals to categories, that the phenomena being measured have real quantitative underlying dimensions that correspond directly to measurement scales, that all subjects assign observations to categories and values in an equivalent manner and can report true values (Cicourel, 1964; Pawson, 1982). It is, however, common knowledge that social actors lack complete information, behave inconsistently and have difficulties assigning numbers to experiences (Gephart, 1988). Hence, if the assignment of numbers is arbitrary, different actors will assign different meanings for the same values and variables. One can still assume that there exist true interval values for variables, but actors have difficulties reporting them or measurement instruments are not well equipped to capture them. The common practice for social scientists in this situation is to produce ordinal measures to capture this phenomenon. The implication stemming from this is that, for example, two different subjects who share same meanings can report different values (i.e. 4 and 5). In similar veins, subjects can forget previous selections and report 4 one time and 5 the next time holding the same meaning. Furthermore, some subjects have wider intervals while others narrower, meaning that, for example, good for one could be 90 on a 100-point scale, while to another good could be 2 on a 3-point scale. It is possible to question whether researchers are justified in the common assumption and practice of assigning interval values to ordinal data so that the data can be treated as interval for the purposes of further statistical analyses (such as regression analysis). In particular, if those technical and practical assumptions are justified, results from computer simulations where ordinal data were converted to interval data should be equivalent to results from true interval measures. To illustrate this case, Gephart (1983) designs a simulation experiment to test whether imprecise measurement will produce statistics similar to true values. For these purposes, he compares the results from running regressions among variables with true values and imprecisely measured values (obtained through four transformations). The main results from the experiment show that transformations of ordinal to interval scales representing data in equal intervals are appropriate and capture fairly and precisely the phenomenon being measured. When representing data in unequal intervals, however, observed measures diverge substantially from true values. Furthermore, further statistical analysis run on both true values and transformed values gives divergent results. Based on this experiment, the author illustrates that imprecise measurement (e.g. transforming ordinal into interval data) will not produce statistics similar to true values. Thus, second-order ethnostatistical analysis shows that common practices (i.e. conventions) employed by analysts in performing statistical analysis may distort final results.

In third-order ethnostatistical analysis, which uses textual analysis methods to examine the persuasive properties of statistics in documents, another set of informal practices are found to be used in order to enforce the validity of the argued claims. A nice illustration of this phenomenon is Gephart's (1986) examination of the rhetoric used in 16 quantitative methodology papers exploring the adequacy of the parametric strategy. After having made a comparative thematic analysis of the papers, Gephart (1986) found that when discussing correlation coefficients (a statistical measure of the strength of the relationship between the relative movements of two variables) in language terms, similar parts of speech (in specific adjectives) were applied to different values, as well as similar values were linked to different descriptive terms and those language tools were used to persuade the audience about the researchers' claims. In other words, the interpretation of correlation coefficients appeared to be a rhetorical process in which meanings were created by adjectives while absolute values of numerals or tests of statistical significance had minor role, confirming the claims made by McCloskey (1985). Such conventions and arbitrary use of language in quantification texts can surely challenge the taken-for-granted objectivity of hard data production.

The state of the art of ethnostatistics in organization and management research

While there is an ample number of studies illustrating the use of each level of ethnostatistics in isolation from one another (which could be found in sociological, anthropological and organization and management literature), ethnostatistical studies in organization and management research (our field of enquiry) embracing holistically the different levels of analysis are relatively rare (some significant examples include Landrum and Boje, 2008; Boje et al., 2006; Carlon et al., 2006; Kilduff and Oh, 2006; Mills et al., 2006). Specifically, we observe that management researchers have adopted ethnostatistical lenses even before being potentially inspired by Gephart's formal invitation (2009) using it as a research method and/or as a theoretical perspective. For example, Mills et al. (2006, 2004) combined ethnostatistics with Weick's sensemaking framework while exploring Canadian business schools and universities to deconstruct the production, meaning and rhetoric used by them when they draw on accreditation and rankings to demonstrate how and why certain discourses influence the popularity of particular measurements at certain points in time. Davidson et al. (2004) used reasoning grounded agency theory to disentangle earnings management practices employed by CEOs in dual (serving as both a CEO and a chair of the board of directors of corporations) and non-dual positions to find that earnings management occurs more frequently following duality-creating successions than otherwise because CEO chairs have greater control of the impressions created by their firms' financial reports and are operating under greater expectations of positive results. In similar veins, Carlon et al. (2006) used ethnostatistics to disentangle executive compensation strategies explaining why scholars have found inconsistent empirical links between various compensation strategies and firm performance. Other interesting contributions are continuing to appear in organization and management research in the form of conference proceedings (such as Klopp and Saylors, 2019, where the authors used ethnostatistics as a method for establishing the importance of a replication study in entrepreneurship) or in interdisciplinary journals (such as in Winiecki's, 2008, ethnostatistical analysis of performance measurement). It is, however, evident that ethnostatistics is still lacking enough attention in management and organization research.

As one of the main objectives of our research is methodological, in what follows we will focus on two exemplar studies using holistically ethnostatistics to extrapolate how management and organization scholars have applied the ethnostatistical method.

In one of the exemplar studies employing first- and second-order ethnostatistical analysis, Kilduff and Oh (2006) assess the classical text on diffusion of innovation “Medical Innovation: A Diffusion Study” by Coleman et al. (1966) and its three subsequent re-analyses. Their purpose is to demonstrate how reconstructing the context in which the original study was conducted and seeking for tacit assumptions employed in the level of statistical elaboration could shed light on the existing scholarly debate stemming from the controversial results reported in the subsequent studies. The scholarly debate revolving around the classical study on the diffusion of the antibiotic drug among medical practitioners and its subsequent re-analyses was concerned with understanding whether physicians resolve the uncertainty of adopting the new drug through conversations with colleagues (i.e. the cohesion explanation) or through their perception of the action as proper for an occupant of their position in the social structure of colleagues (i.e. the structural equivalence explanation). The main controversy in this scholarly debate was that while the results from the original study supported the cohesion explanation, its three subsequent re-analyses using the same data (i.e. Srang and Tuma, 1993; Marsden and Podolny, 1990; Burt, 1987) supported either the same explanation or the alternative argument. In order to shed light on the debate, Kilduff and Oh (2006) applied first- and second-order ethnostatistical analysis by 1) reconstructing the historical context in which the data for the original study were gathered and 2) examining the statistical analysis applied in the subsequent re-analyses by focussing on how each study modified or preserved statistical assumptions and techniques contained in the previous study. Drawing on the reconstruction of the context and the economic climate in which the original study was conducted, the authors showed that early prescribers of the new drug were more integrated into the medical community if compared to later prescribers. This finding therefore supported the cohesion explanation. The first re-analysis of this study, however, focussed on a subset of the original data, and the conclusion of the original study was discarded offering a new one based on the idea of competition between similarly situated physicians (structurally equivalent). The further re-analyses showed a continuous struggle between the competing arguments reporting controversial results stemming from the arbitrary choice of focussing on different parts of the data that can support different theoretical claims. It is clear that the ethnostatistical analysis provided by Kilduff and Oh (2006) could shed more light on the debate by locating data in the context from which it was generally abstracted. One can, however, ask what was the explicit method by which the contextual reconstruction was guided.

Another representative study of the use of ethnostatistics in management and organization research is presented by Boje et al. (2006), who investigated how Enron Corporation, which underwent one of the largest bankruptcies in the US history, presented statistical and accounting data in a way to create a positive misleading image of the company, applying all three levels of ethnostatistics. What Boje et al. (2006) did in order to disentangle the case was 1) to reconstruct the context in which the company operated (first-order ethnostatistics), 2) to evaluate how the company used predictive statistical models to ground the pricing of their long-term contracts (second-order ethnostatistical analysis) and 3) to uncover how the rhetoric of numbers was accomplished theatrically in staged settings in order to persuade relevant audience about the company's success (third-order ethnostatistical analysis). They found that the context in which the company operated was a new highly deregulated market. This context easily allowed using impression management tactics, such as immediate recording of projected revenues from long-term contracts at arbitrary price assumptions, therefore making it possible to inflate revenue figures as needed or keeping numerous debt-related “off-the-balance-sheet deals” to demonstrate financial strength. At the second ethnostatistical level, the authors found that Enron traders could wrongly mark the forward pricing curves, producing fictitiously enlarged revenues, which granted them enlarged bonuses. Furthermore, Boje et al.'s (2006) analysis uncovered that the risk management model used by the company included overly optimistic assumptions, thus presenting misleading information about risk to investors. Finally, at the third level of ethnostatistics, the authors discovered how in each year between 1998 and 2001, Enron created temporary stage set and simulated a real trading floor, thus persuading targeted audience (i.e. Wall Street analysts and other relevant stakeholders) that the business was “booming”. In this way, the rhetorical power of numbers was enacted theatrically to create a misleading image of the company. Notwithstanding the cleverness and insightfulness of Boje et al.'s (2006) study, one can ask again for the explicit methodology used for creating the historical contextual reconstruction employed in their study.

In fact, both the representative studies applied ethnostatistical analysis, starting by referencing Gephart's (1988) seminal work methodology. However, it was somehow not explicit how they methodologically delivered the historical reconstruction in their studies. While for real-time ethnographies, organization and management scholars seem to be well equipped with methodological tools and usually pay significant attention to methods description, in those studies the “retrospective” ethnography produced somehow lacked a description of employed methods. While we acknowledge that such shortcomings are not compromising the rigour and the validity of the obtained results, we use this observation to position our argument, namely that in the lack of the practical possibility to conduct real-time ethnographies, specific attention to historical sources should be paid as their use poses additional methodological concerns which are explicated in more detail in the subsequent sections.

Limitations in the application of ethnostatistics in organization and management studies: a historical approach as a possible solution

As seen from the literature review presented earlier, organization and management scholarship applied ethnostatistical analyses in different settings delivering insightful explanations about ongoing controversies and uncovering hidden contextual facts to supplement their analysis. In all cases, however, the first-order ethnostatistical analysis was done in retrospect. While we do not question the validity and the power of the delivered histories in those studies, we draw on an argument explained in detail in the literature (Rowlinson et al., 2014) that the treatment of time (i.e. the past) for organization theorists and historians differs, therefore potentially raising methodological concerns. Specifically, Rowlinson et al. (2014) suggested that 1) while historians are preoccupied with narrative construction, organization scholars tend to subordinate narratives to analyses; 2) while historical research uses verifiable documentary sources, organization scholars construct data following replicable procedures; and 3) whilst historians construct periodization from sources and historical contexts, organization theorists “tend to treat time as constant or else import periodization as given from historiography” (Rowlinson et al., 2014, p. 251). Bringing those arguments to the application of ethnostatistics in organization and management research, we would like to pay specific attention to the fact that historical approaches can substantially complement the development of the contextual reconstruction, but only if conducted with methodological awareness. Furthermore, historical approaches focussing on source criticism can supplement data triangulation and allow for a critical assessment of the changing meaning of measurement practices and results in different contexts.

These considerations imply for the researcher an additional effort to assess the cultural and material conditions of quantification practices by checking all the eclectic but verifiable documentation that is available. References to data and literature should be supplemented by archival sources spanning from public records to private correspondence, and what actors say about what they do needs to be assessed on other evidence and interpreted following the different contexts and audiences. At the same time, such an inquiry should be extended retrospectively to further sources concerning the antecedents and, where possible, the effects of their practices. This way, it becomes possible to identify the complex dynamics that explain the variability in time and space of both the contextual meaning and the contingent circumstances of the production and use of statistics.

The intersection of history and ethnography: implications for ethnostatistics

An historical approach can address some of the gaps coming from the limitations of ethnostatistical methods, as they are explicitly stated by Gephart (2009). History has a descriptive and explanatory approach in common with the ethnographic method, which puts prescriptive and transformative aims out of its reach. Yet the two disciplines diverge when political implications and intentional manipulations are concerned. The ethno-statistician accepts that the latter “do occur” and that the former are relevant, but assumes that they “are not fundamental to quantitative social and management research” (Gephart, 2009, p. 100). On the other hand, historical studies on the development of quantitative enquiry provide useful insights on the relationship between intentional and unintentional selection and representation biases, showing that political questions were crucial to the development of new methods (Desrosières, 2002; Yeo, 2003) and that very sophisticated manipulations can be embodied in technicalities (Stapleford, 2009).

These threads of research in the history of statistics show also that history can shed light on the political and historical context in ways that are precluded to ethnographic analysis, as far as it refers to the past only as it remains in the present perception of the actors (Clark and Rowlinson, 2004, p. 345). In particular, a retrospective historical approach can highlight processes of black-boxing, reusing, reinterpreting data, measurements and metrics that only take place in the medium and long term. The social life of data is not limited to their creation and immediate rhetoric use, as they remain as a reference for future quantification, for comparison and to build up trends and scenarios that frame the perception of a phenomenon and, finally, decision-making. In this perspective, putting into relationship documents, materials and events located in different times allows to follow the change of meaning that measurement can take, and the process of alterations involving different actors in different moments, highlighting the role of “boundary objects” that measurements and the resulting quantitative data can exert between different places and times (Merton, 1973).

Consequently, historical approaches, with their focus on source criticism and contextual reconstruction, can usefully join ethnography, computer simulations and rhetorical analysis, complementing these methods especially where the interpretations and re-contextualization in the medium and long term by subsequent users of statistics are concerned. An historical ethnostatistics can make it possible to understand the long-period dynamics of quantification and to identify specific historical trajectories, particularly important where the transmission and validation of knowledge are involved (Barth, 2002; Bourdieu, 2004). Examples of similar inquiries include explorations into the origins and evolution of economic indicators as the consumer price index in the United States (Stapleford, 2009), research on the development of statistical instruments as time series analysis (Klein, 1997) or the history of specific metrics as the QI test (Carson, 2006). All these long-period studies exploited a variety of sources, from interviews with relevant witnesses to the analyses of the private papers and published production of scientists and technicians. They also adopted different methods, including discourse and textual analysis, an ethnographic approach to the reconstruction of quantification practices inside of statistical bureaus and a quantitative reassessment of their results, together with a retrospective historical interpretation of the connections among the resulting plurality of data.

Anthropology in its turn is not new to the hybridization of its methods with historical ones, neither to reflecting on the role of assumptions about temporality in the interpretation of cultures. Bourdieu (1984) was crucial in highlighting the critical role of history. As any established order produces “the naturalization of its own arbitrariness” (Bourdieu, 1994), the historical inquiry into its construction through time can provide the ethnographer with critical insights to deconstruct assumptions that are taken for granted. However, as Greenhouse (1996) highlights, the relationship of ethnography with temporality is complex, as its analysis of sensemaking and legitimation processes de-constructs the apparent linearity of historical time. Time was also identified as an instrument through which the ethnographers take distance from their subjects, denying their status as coeval to the observer (Fabian, 1983). Historical approaches can then imply the imposition of an evolutionary or otherwise distancing time frame on the relationship with the other.

The necessity of admitting a pluralism of possible historical interpretations is explicitly evoked by Comaroff and Comaroff (1992) in a series of articles. They argue for the necessity for anthropologists to apply their ethnographic sensibility when working with archives, deciphering the multiple, overlapping interpretations that both the authors of sources and the historical actors propose. Their theoretical proposal for a historical ethnography, however, has been criticized precisely for not considering the peculiarity of historical techniques (Vansina, 1993). Consequently, there are evident difficulties and problems in complementing or substituting ethnography with a historical approach. By building a historical narrative based on secondary data, the ethnostatistical researcher takes on the unavoidable risk of imposing their own interpretation on the sources (Norman, 1991). Also, by using secondary data as historical primary or secondary sources, researchers apparently resort to what Van Maanen (1988, p. 76) defines as “pre-packaged” materials, so contradicting a main element of the ethnographic method.

Both the earlier cited objections can be dealt with by the means of what Kipping and Usdiken (2014) have defined as “historical cognizance”, that is, the ability to incorporate the complexity of historical processes and the inherent partiality of historical sources in the process of theorization itself. First, the researcher can cope with the problem of imposing an interpretation when using historical materials (however, without solving it) only by means of an explicit avoidance of objectivism, that is, by making clear to the reader how the evidence was found and interpreted and neatly distinguishing between what this evidence is saying and what is the result of their best possible interpretation. Second, it is true that archival sources may be “pre-packaged”. However, the historical researcher has many instruments of source criticism to “unpack” them, from cross-comparison with other sources, to the study of the context and conditions presiding to their production, up to textual analysis. For the ethnographic purposes of ethnostatistics, in fact, the ability to read historical sources “against the grain”, that is, using them in similar way as an ethnographer would use texts to interpret a culture, is crucial. Among the different available historical approaches, it is clearly the “ethnographic history”, as identified in Rowlinson et al. (2014, pp. 266-267), which appears the most fit to complement the classical ethnostatistical approach.

It is worth highlighting, however, that, in doing this, the ethnographer needs to be aware of the epistemological peculiarities of historical enquiry (Vansina, 1993). Historical sources are idiosyncratic by definition and cannot be treated under the assumption that what matters is the replicability of data construction and analysis, simply because they are not data. They are not constructed, but found, and provide an unavoidably partial view on whatever is the subject of the research. Hence, they cannot be simply analysed, but they need to be retrospectively interpreted, leaving room for the possibility of multiple interpretations.

To be clearer, the bias in sources and in their historical interpretations can be recognized and accounted for, but it cannot be avoided entirely. It is not something that the researcher might strive to stamp out, or something in spite of which interpretations are constructed; instead, as Comaroff and Comaroff (1992) have suggested, the recognition of the multiple perspectives embodied in sources and data is the only possible way to build proper thick descriptions [1]. Such a perspective may be identified as critical realism, that is, an approach that adopts a realistic ontology and a relativist epistemology at the same time (Gorski, 2013; Steinmetz, 2014). This implies the assumption that social reality has an autonomous existence, but our knowledge of it is unavoidably partial, being “always historically, socially and culturally situated” (Archer et al., 2016).

Source criticism which relies on different instruments in historical research is a powerful tool to rigorously undertake this task and can supplement an ethnographic approach not only by highlighting the partiality of sources but also by assessing their (partial) reliability. From such a methodological approach, the researcher gains the ability to interpret retrospectively in context the complex and contingent causality that affects change dynamics (Andrews and Burke, 2007).

Towards a historical ethnostatistics

In what follows we outline a pluralistic method for conducting ethnostatistical research in organizational settings based on the integration of research practices employed by one of the authors conducting ethnostatistical research in a large multinational company and best practices from ethnographic and historical research. The practical experience of conducting a real-time ethnostatistical enquiry was gained in the period of seven months from September 2016 to March 2017 when one of the authors had the unique opportunity to explore the work of the Sustainability Department located in the headquarter of a large multinational company which has been one of the industry leaders in sustainability in the sector in which it operates according to independent rating agencies assessment for five consecutive years (i.e. 2009–2016). Over this period, she was able to make regular site visits ranging from one to three days in length lasting on average of 7–8 h per day which included participating in team meetings, teleconferences and conducting formal interviews and informal talks (15 formal semi-structured and unstructured interviews). The data collection process was complemented using additional secondary sources such as the company’s publicly available data in the form of official documents available in the company's website, sustainability reports, press releases and news articles, rating agencies' questionnaire and reports, competitors' websites and sustainability reports.

In order to complement the suggested research methodology, the authors extracted best practices and techniques of conducting ethnographic research available in literature (i.e. Risi and Wickert, 2017; Adderio, 2014; Slager et al., 2012; Stigliani and Ravasi, 2012; Turner and Rindova, 2012; Kaplan, 2008; Gioia and Chittipeddi, 1991) as well as best practices for conducting historical research. The integration of those research techniques is illustrated in Figure 1. As the figure shows for each level of ethnostatistical enquiry (including one more level, i.e. the history of statistics), different research techniques have been suggested, further additional materials to be treated as secondary data have been identified and finally, some research strategies are presented to complement the suggested analysis.

At the first-level of ethnostatistics (i.e. conducting an ethnography of measurement production), this framework suggests data collection methods commonly used for ethnographic research (e.g. Van Maanen, 1979) spreading from participant observation, formal in-depth semi-structured interviews and informal conversations. To supplement the data collection process, visual documentation of used and produced artefacts during the participant observation has been suggested in an exemplar ethnographic research (i.e. Stigliani and Ravasi, 2012). Such technique can prove to be specifically crucial especially in the cases in which sensitive data (such as key performance indicators) are being elaborated or discussed. The actual documentation of such artefacts can be done by direct photographs (if permitted by the hosting organization) or by the researcher's own graphic or descriptive means – by attempting to recreate graphically and descriptively the artefacts at stake. Within this level of analysis, a complementary set of secondary data, ranging from officially disclosed information through internal documents and other publicly available meta-data sources (see full list in Figure 1), has been suggested to be used for further complementing the creation of a more holistic picture about the general “climate” in which the organization operates. Within this level, we encourage that the researcher creates an initial repertoire of organizational rhetoric (i.e. seeking and documenting specifics in the language or attempting to develop a repertoire of the professional slang used by the observed organizational actors). This initial repertoire will be used later to compare with the official organizational rhetoric in the search for potential divergence that can inform the researcher in which cases the language tools are used to emphasize or de-emphasize a specific event.

Our framework suggests that a useful technique to develop the second level of ethnostatistical analysis (statistics at work) is by conducting it as an action research (Lewin, 1947). In this method of research, it is presumed that the benefit from combining “action” with “research” is to overcome important social and organizational issues together with those practitioners who are experiencing the issues (McKay and Marshall, 2001). Thus, as opposed to case studies or simple participant observation methods, in action research, members (both the researchers and organizational actors) actively participate in the cyclical process of planning, taking action, evaluating that action, leading to further planning, rather than merely being objects of the study (Olesen and Myers, 1999). Therefore, the actors work together so that any issues may be resolved or an organizational system could be improved. While such a research can have operational value, on a purely practical level its value for ethnostatistics can be to grant the researcher potentially with a wider access to organizational documents, operating procedures and even closer, colleague-like relationships with the organizational actors, thus allowing them to have more holistic insights on the actual process of creating statistics. Again, at this level of research any of the secondary sources listed in Figure 1 can supplement emerging insights. At analytical level in this stage, it is suggested that the researchers use the obtained sources as if they were historians (i.e. focussing on source criticism to uncover what the developed documents at stake are an answer to and which specific bias they project on the results). Further it is suggested as useful to gain more knowledge about the organizational actors' educational and professional background in order to give the researchers a potential anticipation and explanation of the “conventions” and informal practices they might be using in the process of data elaboration.

In the third level of ethnostatistical enquiry, the rhetoric of statistics, a specific focus on the organizational textual and visual presence, is suggested as a means for evaluating how, where and why quantitative facts can be potentially used to create a sounder validity of organizational claims. At this level of analysis, we suggest evaluating the official company's disclosure against the developed (in the first stage of the analysis) repertoire of organizational rhetoric. Further it is suggested that the official organizational disclosure should be analysed critically as a literary document searching for differences in the writing styles used in different documents, points of time or among sections within the same document. For example, too emotional or positive style can be a sign of the use of language as a strategic tool rather than to disclose pure facts. In this stage, again, possibilities for cross-checking with additional data are suggested as a tool to complement the analysis and to give the researcher an indication of whether and why certain events might be discarded from the official organizational rhetoric.

Finally, we suggest adding a fourth level of analysis, that is, an ethnographic history of statistics, focussing on the longer-term evolution and change of the meaning assigned to the constructed metrics and to the collected data. As it is common knowledge, metrics and methods are often transferred from and/or to other organizations and/or institutions (i.e. standardized indicators, assessment methods, key performance indicators). In this process, a complex dynamic of domestication and adaptation is involved. We suggest that such mechanisms can be studied using a retrospective approach resorting to the secondary data concerning the past phase and connecting it to the present situation that is then subject of ethnographic observation. Connecting the two phases requires establishing an interpretive connection between materials having different origin and characteristics. Such an interpretation is impossible to achieve focussing solely on the ethnographic study of quantification practices in a live setting, as the metrics that are transferred are “black-boxed”, that is, they are taken for granted in their working, focussing only on the results, this way concealing the details of their origin and making their history unknown to their users (Latour, 1999, p. 304). This in fact excludes the possibility to appreciate the details of their construction and transfer as they are not explicitly “embedded in the daily practices and symbolic life of the group studied” (Van Maanen, 1988, p. 72). The same is true whenever the users of data take for granted the procedures by which these data were produced elsewhere or in a different time and go on elaborating and interpreting them for practical purposes. Furthermore, the “social life of data” is also a field of enquiry which requires almost by definition a historical approach in order to be extended further than a limited scope and to assess what was lost and what was acquired in the transfer through different contexts (Blackman, 2015).

Therefore, at this level of analysis, we suggest that exploration of the social life of data and the path of transferred metrics is needed to uncover the micro-dynamics of quantification processes (including building or adopting measurement tools and data generation) which grant the researcher the ability to trace and show the long-term trajectories of data which is lacking if one relies solely on the observation of data generation and use in the present. At this level of analysis, to open their black boxes, historical approaches for source criticism can be used, ranging from cross-comparison with other sources, the study of the context and conditions in which data and metrics were produced, to textual analysis. As illustrated in Figure 1, additional secondary sources can supplement this level of analysis.

Conclusion

As far as the professional world is becoming more and more performance-driven, reliance on quantitative data has become a milestone to support the business decision-making process and to create objective knowledge. In such a context, it is extremely important for both academics and business professionals to adopt critical lenses in the process of constructing and using measures. In this paper, we discuss how a promising method from organization and management research, ethnostatistics, can be used to guide such critical analysis. In particular, we suggest how historical approaches can successfully join ethnostatistical enquiries in an attempt to overcome some limitations in existing conventional methods. Such an approach can bring certain benefits to both the academic and business world by increasing the methodological rigour in producing scientific knowledge at a theoretical level and by improving the measurement practices employed by business professionals at a practical level.

We observe that organization and management scholars delivered some excellent contributions by applying ethnostatistical analysis to solve highly debated and controversial cases. However, the practical impossibility of conducting real-time ethnographies has made historical reconstruction used as a substitution. We use this observation to position our main argument, namely that historical approaches which focus on the retrospective interpretation of eclectic sources can be useful in solving existing problems in conventional ethnographies. To address these issues, we outline a pluralistic method for conducting ethnostatistical research encompassing practical research techniques and offer a methodological discussion on how historical approaches which focus on source criticism and contextual reconstruction could overcome the stated limitations of ethnostatistics.

Finally, although we state explicitly that this research has been informed by the practical experience of one of the researchers complemented with best practices evident in the available literature, we wish to acknowledge that the suggested framework is a conceptual one, and further empirical research is needed to ensure its applicability and further enhancement. Therefore, a promising avenue for future research is to apply the suggested method in real case studies (although not necessarily focussed on pure quantification problems, but also on the rhetoric of quantification). We believe that such research agenda and future improvements of the methodology can narrow the conceptual distance between practice and theory and within academia between historical and ethnographic organizational and management research. While we acknowledge that such a method can be time-consuming, we see it as a promising contribution to methods. In specific, we believe that this paper contributes to the ethnostatistical field by discussing the intersection between history and ethnography while suggesting ways for their complementary use in organizational and management research.