https://orcid.org/0000-0002-9603-6686
Figures
Abstract
This paper offers a comparative evaluation of the scientific impact of a citizen science program in ecology, ‘‘Vigie-Nature”, managed by the French National Museum of Natural History. Vigie-Nature consists of a national network of amateur observatories dedicated to a participative study of biodiversity in France that has been running for the last twenty years. We collected 123 articles published by Vigie-Nature in international peer-reviewed journals between 2007 and 2019, and computed the yearly amount of citations of these articles between 0–12 years post-publication. We then compared this body of citations with the number of yearly citations relative to the ensemble of the articles published in ecology and indexed in the ‘‘Web of Science” data-base. Using a longitudinal data analysis, we could observe that the yearly number of citations of the Vigie-Nature articles is significantly higher than that of the other publications in the same domain. Furthermore, this excess of citations tends to steadily grow over time: Vigie-Nature publications are about 1.5 times more cited 3 years after publication, and 3 times more cited 11 years post-publication. These results suggest that large-scale biodiversity citizen science projects are susceptible to reach a high epistemic impact, when managed in specific ways which need to be clarified through further investigations.
Citation: Bedessem B, Julliard R, Montuschi E (2021) Measuring epistemic success of a biodiversity citizen science program: A citation study. PLoS ONE 16(10): e0258350. https://doi.org/10.1371/journal.pone.0258350
Editor: José Gutiérrez-Pérez, University of Granada: Universidad de Granada, SPAIN
Received: March 30, 2021; Accepted: September 25, 2021; Published: October 11, 2021
Copyright: © 2021 Bedessem et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have no competing interests to declare. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Introduction
The participation of non-professional scientists in the production of scientific knowledge is hardly a new phenomenon. The history of the natural sciences, from XIXth century botany to contemporary ecology and conservation sciences, shows several instances of amators engagement in scientific practices ([1, 2]). However, calls for a larger implication of lay citizens in the process of knowledge and expertise production are on the rise, while inclusiveness in scientific research is increasingly valued by institutional discourse [3]. Within scientific practices themselves, this trend is particularly notable through the growth of so-called ‘‘citizen science”, which includes a large variety of forms of participation of non professional scientists (citizens, some members of NGOs) in the production of scientific knowledge [4].
The value of this type of public engagement in science for citizen empowerment [5], public learning and understanding of science ([6, 7]), and the development of new tools of governance [5, 8, 9] is well recognized. However, the growth of citizen science practices goes hand in hand with epistemic concerns about whether and how ‘‘the public can make actual contributions to science” [10]. For instance, doubts have been raised about the quality of the data collected in crowdsourcing programs [11, 12], or about the neutrality of the participants which sometimes pursue a political agenda [13]. In this context, a fundamental question for most scientists concerns the ability of citizen science projects to deliver results that can significantly advance scientific knowledge [14–18]. In response to these concerns, many contributions have been recently proposed, which aim to demonstrate the epistemic value of citizen science. This is notably the case in ecology and environmental science, where participative practices have proved their utility for biodiversity evaluation [19], the knowledge and management of natural resources [20, 21], or the assessment of ecosystemic services [21]. Given these purported successes, a more general question emerges about the conditions which maximize the scientific ‘‘impact and reuse” of citizen science data and results [22]. In this context, the systematic evaluation of the scientific outputs of citizen science projects is a central challenge in terms of how both to identify good participative practices and to increase the trust of the whole scientific community in the ability of those projects to produce reliable and useful knowledge. However, if the epistemic evaluation of research is a complex and controversial issue in general [23], it seems to be even trickier in the case of citizen science. Among the reasons put forward by [24]: the results produced by citizen science projects are not always presented and disseminated in the form of peer-reviewed scientific publications, which despite limitations, are normally adopted as ‘‘raw material” for scientometric studies aiming to evaluate scientific productivity [15]. A further reason that is advocated for is the technical difficulty of gathering large numbers of existing scientific publications produced by citizen science projects, since these are rarely labeled as such in literature database such as Web Of Science or Google Scholar [25].
Recently, [15] collected 143 publications in peer-reviewed journals resulting from 23 distinct citizen science projects in the field of astronomy. He conducted a citation study of these articles over a 15-year period, and found that their citation rate achieves a peak within three years from publication, by which time they have been significantly more cited than the average astronomy papers. This result is interpreted in terms of scientific ‘‘impact” both by the author and by a data-brief published in Nature Index [26]: citizen science publications in astronomy have a strong short-term relative impact, and this impact decreases on the mid and long-term. Of course, the very notion of scientific impact is an abstract construct which can endorse different meanings, and be quantified through different types of measurements, for instance by using social network analysis [27]. However, the citation rate (the number of citations of a given article in subsequently published papers) is an old tool [28] which is still largely used as a proxy for the impact of scientific production, even if it is well recognized that it constitutes a partial and imperfect variable for research evaluation [29], including ecological science [30].
In this paper, we aim to contribute to the debate about the evaluation of the epistemic value of public engagement in scientific research by conducting a citation study of citizen science-based publications in the field of ecology and conservation science. Following [15], we consider the temporal evolution of the citation rate as a proxy for the scientific impact of a given publication. However, while [15] considered an aggregate of heterogeneous citizen science projects, our study draws on one large-scale, long-term citizen science program: the Vigie-Nature network of participative observatories of biodiversity, managed since 1989 by the Muséum National d’Histoire Naturelle (MNHN) in Paris, France.
Materials and methods
The ‘‘Vigie-Nature” program
Vigie-Nature started in 1989, with the aim of improving our knowledge of biodiversity by engaging a large diversity of people (amateur naturalists, managers of green areas, pupils, farmers and other citizens) in the collection of field data in the whole French territory (for more information, one can refer to http://www.vigienature.fr/fr/presentation-2831). Vigie-Nature mobilizes non-professional citizens to quantify the abundance of species, in order to characterize the evolution of ecosystems in terms of species assemblage. It gathers different participative observatories, focusing notably on pollinator insects (Spipoll project), birds (BirdLab), or urban vegetation (Sauvage de ma rue).
The program is managed by the MNHN, with the collaboration of naturalist associations. Data quality is controlled by scientific experts, which verify species’ identifications. The researchers use these collected data to advance research in ecology and conservation science by publishing results in scientific journals such as Biodiversity and Conservation, Journal of Ornithology, Biological Conservation, Landscape Ecology, Global Change Biology, and Frontiers in Ecology and the Environment. Between 2007 and 2019, 171 articles were published within the frame of this program. The corresponding references are available in a dedicated data base (https://www.zotero.org/groups/85788/vigienature/items/collectionKey/4M3I4ZUR). Within this body of publications, 123 articles present scientific results on the basis of collected data, while the remaining articles propose meta-reflections about citizen science. The Vigie-Nature program is thus well adapted for the purpose of scientometric studies since it allows to pursue statistical analysis on the basis of a large sample of publications. Whereas studies such as Odenwald’s (2018) rely on aggregated data from distinct citizen science programs, the bulk of Vigie-Nature’s publications is large enough to allow for a study of the scientific impact of one specific project.
Data collection
To conduct our study, we considered the 123 scientific articles in the field of ecology and environmental sciences from the Vigie-Nature bibliography published between 2007 and 2019, with a total of 5813 citations in peer-reviewed journals (hereafter, VN group of articles). Fig 1 gives the distribution of this sample with respect to publication year.
NB: At the time of our analysis, the whole data for 2019 were not yet available, which explains the relative low number of articles for this year.
To collect the number of citations on each year post-publication for each of these publications we used the Web Of Science (WoS) database—a reliable and user friendly platform, widely considered the ‘‘gold standard” for citation studies [31]. We first computed the number of self-citations (we considered an article A is self-cited by B when at least one author of A is also an author of B) in order to allow for a more comprehensive quantification of the scientific impact of Vigie-Nature’s result. We found a self-citation rate of about 11% in average, which appears to be close to the average value of self-citations in contemporary science (about 12% following [32]), but lower than the self-citation rate in the ecological sciences (about 16% following [33]). S2 Appendix compares the citation profile obtained with and without self-citation. In the following, we do not exclude self-citation from the citation rates.
We then calculated the number of citations for each article from their year of publication (shifted to year 0) up to 2019. This produces data on a range from 0 to 12 years post-publication, with 123 articles for the first point (time = 0 year) and 6 articles (corresponding to those published in 2007) for the last one (time = 12 years). This method of calculation might introduce a bias because a paper published, for instance, on 2019-01-01 is treated as identical to a paper published on 2020-12-31. This bias may be expected to be influential for the first point of the yearly citation profile (time = 0 years). However, we suggest that it rapidly attenuates after this point. Besides, we suggest that this effect operates similarly for the Vigie-Nature samples and for the comparison groups of papers. S1 Appendix presents simulation results which strengthen the relevance of this approximation.
This set of data was compared with the citation history of three bibliographic ensembles. The first one consists of the articles published between 2007 and 2019 and referenced in WoS in the category ‘‘Biodiversity Conservation” (sample size: 60847 articles, hereafter BC group of articles). This sample was constituted by using the filters ‘‘Biodiversity Conservation” (as a WoS category), ‘‘2007–2019”, and ‘‘Article” (as document type).
For every publication year, the annual citation rate of these articles were then extracted by creating a ‘‘citation report” in WoS. This citation report contains, for a given year, the total number of articles under the BC category, and the yearly number of citations from 2007 to 2019. These data are given in S3 Appendix. We then computed an average citation rate per year post-publication following the same methods as the one used for Vigie-Nature (VN) articles.
The second set of data includes all the articles published between 2007 and 2019 within the ensemble of scientific journals where the Vigie-Nature articles are published (hereafter, VN journals groups). To build Fig 3, the citation history of every VN article was compared to the average citation history of the articles published in the corresponding journal. More precisely, for each article A from the Vigie-Nature bibliography, published in the journal J during the year Y, and which got, t years after publication, a number of citations NA(t), we computed R = NA(t)/Nj(t), with Nj(t) being the average number of citations t years after publication of the articles in the journal J during the same year Y. This ratio R represents, for each VN article, its relative amount of citations compared with the whole publications in the same journal. Journals’ citation histories are easily extracted from WoS by creating a ‘‘citation report” (after selecting the publication year, the document type, and the journal title). Let us note that some journals are not referenced in WoS, and were not taken into account in the statistical analysis (namely: Diversity; Current Zoology; Nature Climate Change; Peer-Review Journal; Computational management science; Methods in ecology and evolution; Urban ecosystems; Regional Environment change; British birds; Ecology and evolution; Conservation Letters; Insect Conservation and Diversity). 17 Vigie-Nature articles are then excluded. Besides, we also excluded from the analysis articles published in generalist journals, namely: Science, Nature, Proceeding of the Royal Society: Biology, Biology Letters, Plos One. 13 VN articles were then removed from the analysis. Finally, this second data set includes 78988 articles. Third, we also used the articles from the ‘‘Environment and Ecology” category from Clarivate Analytics. Between 2009 and 2019, this category gathers 841 673 articles. We directly extracted some statistical figures from the data base (citation thresholds for the 10%, 20% and 50% most cited papers. These data are available here: https://esi.clarivate.com/BaselineAction.action).
Statistical analysis.
All the calculations, and in particular statistical tests were carried out by using Matlab 2019b. The statistics were developed through a longitudinal analysis using the mvregress function in Matlab (specifically conceived to lead this kind of longitudinal analysis aiming to compare temporal series by modeling them under the form of polynomials). With this function, we operated multivariate regressions to fit, with different models, the time-series of yearly citations for the different groups of papers: VN (Vigie-Nature articles), BC (Biology and conservation articles), and VN journals articles. In its more complex form, the regression model takes into account the time T (0 to 12 years), the origin O of the articles (BC, VN or VN journals), and the interaction between time and origin TxO. For a two-degree polynomial, the global equation model is the following:
with yij the value (number of citations) for article i at time tij (in years), Gi the origin of the paper (equal to 1 for VN and 0 for BC or VN journals), εij the error in fitting yij, and (βo.…β5) the coefficients to be found. Apart from this global model, two reduced models can be designed: the first one ignores the O variable (MR1), and the other one ignores only the interaction terms β4 and β5 (MR2). The mvregress function then finds the β terms and the likelihood objective function for these models (the global one and the reduced ones). We then led two kinds of analysis. First, for the general comparison of citation history, we computed the MR1 model and we compared it to the global model which includes O as a variable. We then compared these two models in their ability to fit the data, by using a likelihood ratio test (using the chi2cdf function in Matlab). The null hypothesis is that the two models are equivalent, which would mean that the reduced model without O is sufficient, and thus that the origin of the paper does not determine the citation history.
Second, for the analysis of the evolution of the relative citation rate of Vigie-Nature articles, we compared MR2 and the global model (which includes interaction terms). We applied the same likelihood ratio test to determine if the interaction of time and origine had a significant effect on the citation history profile. In other terms, this analysis allows to determine if the differences measured in the yearly citation rate change in time. The results we obtained are presented in the next section.
Results
Comparison of citation history
We first computed the citation history for the 123 papers extracted from the Vigie-Nature (VN) bibliographic database. We calculated the average number of annual citations as a function of the time post-publication from year 0 to year 12. We present the corresponding graph on Fig 2. Error bars are classically calculated as σ/(N^0.5), with σ the standard deviation and N the sample size. In Table 1, we give the number of values used to calculate each point (depending on the number of available articles, see Fig 1).
We plotted the average citation history for Vigie-Nature’s articles (blue, total sample size: 123 articles), for the ‘Biodiversity and Conservation’ articles from WoS (red, sample size: 60847 articles), and for the articles from the Vigie-Nature journals group (black, sample size: 78988 articles).
On the same figure we drew the average annual amount of citations for the articles in the ‘‘Biodiversity Conservation” (BC) category of WoS. We also drew the citation history for the articles from the VN journal group. The average citation history profile of these two groups of standard (non citizen science based) research articles shows an increase in the amount of annual citations during the first 4 years after publications to reach a plateau of about 3 citations/year from year 4 up to year 12 post-publication. The average citation history of Vigie-Nature articles seems to follow a similar profile, but with an ‘‘excess” of citations from 0 to 12 years post-publication. To test the significance of this effect, we led a longitudinal analysis of the citation history (see the ‘‘Method” section for details). Using a multivariate regression (to fit the data with a polynomial function using a maximum likelihood estimation), we first tested if the VN and BC profiles are significantly distinct. To do so, we compared two models in their ability to fit the data (see the Methods section for details). This test gives a p-value of 0.005. We thus accept the hypothesis that taking into account the origin of the paper improves the fit: in other words, we conclude that the difference between the Vigie-Nature citation history and the Biodiversity and conservation one is significant with a risk inferior to 5%. We performed the same test to compare VN and VN journals articles. We found a p-value<10−5, which shows that the Vigie-Nature citation history is also significantly distinct from the VN journals one.
In addition, it seems that this difference in the time-series of yearly citations grows with time. In the next sub-section we explore the hypothesis that the relative long-term impact of Vigie-Nature articles (compared to standard ones) is higher than their relative short-term impact.
Evolution of the relative amount of citations of Vigie-Nature articles
In order to visualize this time-dependant ‘‘excess” of citations, we computed the average relative amount (R) of Vigie-Nature annual citations, as explained in the Methods section: for each VN article, the number of citation is divided by the average annual citations of the papers published in the corresponding journal, as a function of time (Fig 3). The results indicate that the average relative amount of citations for VN articles seems to be superior to 1 from 0 to 12 years, which is coherent with our previous analysis. Furthermore, it seems that the ‘‘excess” of citations of VN articles grows with time, at least between 0 and 11 years. Indeed, the ratio after 11 years is significantly higher that the ratio between 0 and 4 years (3.3 against about 1.4). This might indicate that the differences between Vigie-Nature articles and ‘‘mainstream” ones in terms of impact become more relevant in the long-term. To test the significance of this effect, we extend the longitudinal analysis presented in the previous section (see the ‘‘Method” section for full details). As for the comparison of citation history, we compared two models in their ability to fit the data. The first one (model 1) does not consider the interaction terms between time T and origin O of the papers, and the second one introduces interaction terms. We then led a likelihood-ratio test in order to test the null hypothesis that the two models are equivalent. For the comparison between the BC and VN groups, the test gives a p-values of 0.026 (<0.05). We thus accept the hypothesis that taking into account the interactions between T and O improves the fit. This means that the VN and BC citation history profiles present different tendencies (that is to say, curves in Fig 2 are not similar, in the sense that they are not ‘‘parallel”). The same test performed for the VN and VN journals groups gives a p-value<10−6, which confirms the previous results. By comparing this result to the evolution of the average relative VN citation rates (Fig 3), it seems reasonable to argue that these difference in general tendencies are due to a time-dependant growing ‘‘excess” of citation of Vigie-Nature articles. In other words, the relative impact of VN articles (compared to ‘‘mainstream” ones published in the same journals) might significantly grows with time (at least from 0 to 11 years).
We plotted the relative amount of annual citation of Vigie-Nature articles with respect to the ensemble of articles published in the same journals during the same year.
Statistical analyses of the dispersion of the citation rates
At this point, an issue can be raised about the statistical distribution of the overall ‘‘excess” of citations of Vigie-Nature articles. Does it only concern a few highly-cited articles, or is it a more equally distributed feature regarding Vigie-Nature publications? To get a more precise picture of the citation profile, we computed the value of the median of the set of Vigie-Nature articles citations at each time t post-publication (that is to say, the number X of citations such as half of the papers are more cited than X, t years after publication). It appears that from 0 to 12 years, the median value is slightly higher than the average citation rate of the articles in the ‘‘Biodiversity and Conservation” category from WoS (see Fig 4). This means that about half of the Vigie-Nature articles are annually more cited than the mainstream publications in the same area. We also computed the deciles partition of the total number of citations of Vigie-Nature articles. Table 2 presents the citation thresholds of the 10%, 20% and 50% most cited papers both for Vigie-Nature and the whole ‘‘Environment and Ecology” category (from Clarivate Analytics) between 2009 and 2019. The 9th decile (the 10% threshold) is about 2.3 times higher in the Vigie-Nature set of articles, whereas the 5th decile (the 50% threshold) is about 3.5 times higher. Together, these results suggest that the ‘‘excess” of citations of Vigie-Nature productions is quite equally distributed among the articles. At the very least, the average citation history of Vigie-Nature papers is clearly not led by a small amount of highly cited ones.
The median (in blue) is compared to the average citation rate of the articles in biodiversity and conservation from WoS (in red, sample size: 60847 papers).
Interpretation and discussion
We have computed the citation history of 123 papers published in peer-review journals (between 2007 and 2019) resulting from Vigie-Nature, a long-term and large-scale citizen science program in ecology and environment sciences. We found that Vigie-Nature articles are on average significantly more cited from 0 to 12 years after publication than the total of articles in the same domain. This average value refers not only to a few highly cited papers. In addition, the difference in yearly citations rate strongly increases with time. We evaluated that between 0 and 4 years post-publication, Vigie-Nature publications are cited each year about 1.4 times more than the mainstream ones (BC and VN journals groups). After 11 years, they are on average cited 3.3 times more per year. Quite interestingly, this long-term impact strongly contrasts with what was obtained by [15] when studying the citation history of publications issued from citizen science projects in astronomy. The author shows that the aggregated citation rate of 143 published papers resulting from 23 citizen science projects is very high 3 years after publication (about 4 times the average of a sample of classical papers). This short-term peak is followed by a sharp decline: after 8 years, citizen science articles from his sample are significantly less cited than mainstream ones. However, our results are fare for being directly comparable to [15]’s ones, since the projects’ structures and scientific domain are very different.
At this stage, a central issue is that of the generality of the conclusions we can draw from our results for citizen science practices. Vigie-Nature is indeed a very specific program, characterized by its large scale (the whole French territory) and its longevity (more than thirty years). In addition, the produced data are massively diffused through peer-reviewed scientific articles, which is rarely the case for biodiversity citizen science projects: [34] have shown that only 12% of the 388 projects they surveyed reach scientific literature. In addition, they suggest that the public availability of the data, the control of scientific rigor, and the projects’ spatial scale and longevity positively influence publication. Similarly, [35] have shown that the accessibility of the data, and the presence of coordinators aiming to maintain data quality enhance the scientific impact of biodiversity citizen science projects. Furthermore, projects dedicated to long-term monitoring also seem to have a greater impact. Together with these studies, our results might allow to formulate some hypothesis about the conditions which promote the epistemic success of biodiversity citizen science projects. First, projects exhibiting a relatively large temporal and spatial extent appear to have a (relatively) high scientific impact. Here, it would be relevant to compare the impact of Vigie-Nature with that of non-participative large-scale, long-term networks of observatories of biodiversity, such as the countryside survey ([36], https://countrysidesurvey.org.uk/) in UK or the worldwide CTFS-forestgeo ([37], https://forestgeo.si.edu/) project. Second, the management structure of the projects might also be a central determinant of their epistemic success. Vigie-Nature is characterized by a very centralized structure, with a few number of well-identified researchers managing a vast network of observatories and a vast amount of collected data. These managers are assisted locally by naturalist associations which recruit participants and animate the programs by organizing meetings and by sending newsletters. This structure might enhance the scientific impact of the projects because data are collected, controlled and used by a stable, dedicated team of researchers. These ones got a good expertise in producing peer-reviewed papers on the basis of participative data collection. Apart from this specific structure of the citizen science programs, the constitution of the research team might also play a role in itself, notably through its professional quality, its notoriety, its insertion within the research networks, etc. To have a more complete view of the factors influencing epistemic success of biodiversity citizen science project, it would then be necessary to address a range of comparisons with other similar participative projects in ecology—such as, for instance, the eBird program ([38], https://ebird.org/home) from Cornell Laboratory of Ornithology.
We also should keep in mind that scientific impact cannot be the only determinant of what should count as ‘‘good practices” of participation. The ethical issue of the visibility of the participants should also be considered seriously. As noted by [39], various citizen science programs in environmental sciences still make contributing citizens invisible in scientific productions, in spite of advances in securing data quality. More generally, this lack of recognition of volunteer contributors is a well-recognized ethical problem [40, 41]. Yet, Vigie-Nature productions do not explicitly include recognition of citizen contributions. In addition, this lack of recognition also questions the influence of citizens’ invisibility on the diffusion of the corresponding scientific results [39]. As many scientists manifest a lack of trust as respects to citizen science data and results [42], the fact that participants are made invisible in Vigie-Nature productions might contribute to their epistemic impact. This would be a crucial point to consider in the future of this citizen science program. Some inspiring propositions have been made in literature regarding the online sharing of biodiversity data. For instance, [43] insist on the importance of recognizing the work of data publishers when citing datasets in scientific papers. This suggestion points to an explicit reference to Vigie-Nature participatory datasets within the Vigie-Nature scientific publications. Moreover, Vigie-Nature data could be shared online through international biodiversity data networks, such as the Global Biodiversity Information Facility [44]. This would give a visibility to this specific type of data, and could increase the epistemic trust placed by researchers in citizen science.
Supporting information
S1 Fig. Comparison of the original sample citation profile (red curve) and the computed one (blue curve).
The blue curve was obtained by applying the approximation method we use in our study. The sample size is n = 100 papers.
https://doi.org/10.1371/journal.pone.0258350.s001
(TIF)
S2 Fig. Evaluation of the robustness of our citation rate calculation.
Average relative error generated by our method of calculation of the yearly citation rate as a function of the sample size.
https://doi.org/10.1371/journal.pone.0258350.s002
(TIF)
S3 Fig. Average yearly citation rate for the Vigie-Nature articles.
The blue curve includes self-citations, and the red one excludes self-citations.
https://doi.org/10.1371/journal.pone.0258350.s003
(TIF)
S1 Appendix. Robustness of the calendar year method.
https://doi.org/10.1371/journal.pone.0258350.s004
(DOCX)
References
- 1. Alberti SJMM. Amateurs and Professionals in One County: Biology and Natural History in Late Victorian Yorkshire. J Hist Biol. 1 mars 2001;34(1):115–47.
- 2. Couvet D, Prevot A-C. Citizen-science programs: Towards transformative biodiversity governance. Environ Dev. 1 janv 2015;13:39–45.
- 3. Strasser BJ, Baudry J, Mahr D, Sanchez G, Tancoigne E. “Citizen Science”? Rethinking Science and Public Participation. Sci Technol Stud. 15 mai 2019;52–76.
- 4. Eitzel M, Cappadonna J, Santos-Lang C, Duerr R, West SE, Virapongse A, et al. Citizen science terminology matters: Exploring key terms. Citiz Sci Theory Pract. 2017;1–20.
- 5. Richter A, Dörler D, Hecker S, Heigl F, Pettibone L, Serrano Sanz F, et al. Capacity building in citizen science. In JSTOR; 2018.
- 6. Bonney R, Phillips TB, Ballard HL, Enck JW. Can citizen science enhance public understanding of science? Public Underst Sci. 2016;25(1):2–16. pmid:26445860
- 7.
Edwards R, Kirn S, Hillman T, Kloetzer L, Mathieson K, McDonnell D, et al. Learning and developing science capital through citizen science. In UCL Press; 2018.
- 8.
Nascimento S, Rubio Iglesias JM, Owen R, Schade S, Shanley L. Citizen science for policy formulation and implementation. In Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen Science: Innovation in Open Science, Society and Policy. London: UCL Press; 2018. pp. 219–240.
- 9.
Smallman M. Citizen Science and Responsible Research and Innovation. In Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen Science: Innovation in Open Science, Society and Policy. London: UCL Press; 2018. pp. 241–253.
- 10. Golumbic YN, Orr D, Baram-Tsabari A, Fishbain B. Between Vision and Reality: A Study of Scientists’ Views on Citizen Science. Citiz Sci Theory Pract. 2017;2(1):6.
- 11. Aceves-Bueno E, Adeleye AS, Feraud M, Huang Y, Tao M, Yang Y, et al. The accuracy of citizen science data: a quantitative review. Bull Ecol Soc Am. 2017;98(4):278–290.
- 12. Van Brussel S, Huyse H. Citizen science on speed? Realising the triple objective of scientific rigour, policy influence and deep citizen engagement in a large-scale citizen science project on ambient air quality in Antwerp. J Environ Plan Manag. 2019;0(0):1–18.
- 13. Gura T. Citizen science: amateur experts. Nature. 2013;496(7444):259–261. pmid:23586092
- 14. Bonney R, Shirk JL, Phillips TB, Wiggins A, Ballard HL, Miller-Rushing AJ, et al. Next steps for citizen science. Science. 2014;343(6178):1436–1437. pmid:24675940
- 15. Odenwald S. A Citation Study of Citizen Science Projects in Space Science and Astronomy. Citiz Sci Theory Pract. 2018;3(2).
- 16. Odenwald SF. A citation study of earth science projects in citizen science. Plos One. 2020;15(7):e0235265. pmid:32673329
- 17.
Shirk JL, Bonney R. Scientific impacts and innovations of citizen science. In: Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen Science: Innovation in Open Science, Society and Policy. London: UCL Press; 2018. pp. 41–51
- 18. Brouwer S, Hessels LK. Increasing research impact with citizen science: The influence of recruitment strategies on sample diversity. Public Underst Sci. 2019;28(5):606–621. pmid:30995163
- 19. Roger E, Turak E, Tegart P. Adopting Citizen Science as a Tool to Enhance Monitoring for an Environment Agency. Citiz Sci Theory Pract. 2019;4(1).
- 20. Yamamoto YT. Values, objectivity and credibility of scientists in a contentious natural resource debate. Public Underst Sci Bristol Engl. 2012;21(1):101–25.
- 21. Schröter M, Kraemer R, Mantel M, Kabisch N, Hecker S, Richter A, et al. Citizen science for assessing ecosystem services: Status, challenges and opportunities. Ecosyst Serv. 2017;28:80–94.
- 22.
Williams J, Chapman C, Leibovici D, Loïs G, Matheus A, Oggioni A, et al. Maximising the impact and reuse of citizen science data. In: Hecker S., Haklay M., Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen Science: Innovation in Open Science, Society and Policy: Maximising the Impact and Reuse of Citizen Science Data. London: UCL Press; 2018. pp. 321–336.
- 23. Viola M. Evaluation of Research (ers) and its Threat to Epistemic Pluralisms. Eur J Anal Philos. 2017;13(2):55–78.
- 24. Wiggins A, Bonney R, LeBuhn G, Parrish JK, Weltzin JF. A science products inventory for citizen-science planning and evaluation. BioScience. 2018;68(6):436–444.
- 25. Kullenberg C, Kasperowski D. What is citizen science?–A scientometric meta-analysis. PloS One. 2016;11(1):e0147152. pmid:26766577
- 26.
Mallapaty S. Data brief: Citizen science papers have more impact. Nature Index [Internet]. 2018; 23. https://www.natureindex.com/news-blog/data-brief-citizen-science-papers-have-more-impact
- 27. Bollen J, Van de Sompel H, Hagberg A, Chute R. A principal component analysis of 39 scientific impact measures. PloS One. 2009;4(6). pmid:19562078
- 28.
Price D. Statistical studies of networks of scientific papers. In: Statistical Association Methods for Mechanized Documentation: Symposium Proceedings. US Government Printing Office; 1965. p. 187.
- 29. Larivière V, Kiermer V, MacCallum CJ, McNutt M, Patterson M, Pulverer B, et al. A simple proposal for the publication of journal citation distributions. BioRxiv. 2016;062109.
- 30. Leimu R, Koricheva J. What determines the citation frequency of ecological papers? Trends Ecol Evol. 2005;20(1):28–32. pmid:16701337
- 31. Peoples BK, Midway SR, Sackett D, Lynch A, Cooney PB. Twitter predicts citation rates of ecological research. PloS One. 2016;11(11):e0166570. pmid:27835703
- 32. Van Noorden R, Singh CD. Hundreds of extreme self-citing scientists revealed in new database. Nature. 2019;572(7771):578. pmid:31455906
- 33. Krauss J. Journal self-citation rates in ecological sciences. Scientometrics. 2007;73(1):79–89.
- 34. Theobald EJ, Ettinger AK, Burgess HK, DeBey LB, Schmidt NR, Froehlich HE, et al. Global change and local solutions: Tapping the unrealized potential of citizen science for biodiversity research. Biol Conserv. 2015;181:236–44.
- 35. Tulloch AIT, Possingham HP, Joseph LN, Szabo J, Martin TG. Realising the full potential of citizen science monitoring programs. Biol Conserv. 2013;165:128–38.
- 36. Tipping E, Davies JAC, Henrys PA, Kirk GJ, Lilly A, Dragosits U, et al. Long-term increases in soil carbon due to ecosystem fertilization by atmospheric nitrogen deposition demonstrated by regional-scale modelling and observations. Sci Rep. 2017;7(1):1–11.
- 37. Marchand P, Comita LS, Wright SJ, Condit R, Hubbell SP, Beckman NG. Seed-to-seedling transitions exhibit distance-dependent mortality but no strong spacing effects in a Neotropical forest. Ecology. 2020;101(2):e02926. pmid:31729025
- 38. Jose V S, Nameer PO. The expanding distribution of the Indian Peafowl (Pavo cristatus) as an indicator of changing climate in Kerala, southern India: A modelling study using MaxEnt. Ecological Indicators. 2020; 110: 105930
- 39. Cooper CB, Shirk J, Zuckerberg B. The invisible prevalence of citizen science in global research: migratory birds and climate change. PloS One. 2014;9(9):e106508. pmid:25184755
- 40. Guerrini CJ, Majumder MA, Lewellyn MJ, McGuire AL. Citizen science, public policy. Science. 2018;361(6398):134–136. pmid:30002244
- 41. Wehn U, Collins K, Anema K, Basco-Carrera L, Lerebours A. Stakeholder engagement in water governance as social learning: lessons from practice. Water Int. 2018;43(1):34–59.
- 42. Riesch H, Potter C. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Underst Sci. 2014;23(1):107–120. pmid:23982281
- 43. Escribano N, Galicia D, Ariño AH. The tragedy of the biodiversity data commons: a data impediment creeping nigher? Database. 2018;2018. pmid:29688384
- 44. Faith D, Collen B, Ariño A, Koleff PKP, Guinotte J, Kerr J, et al. Bridging the biodiversity data gaps: Recommendations to meet users’ data needs. Biodivers Inform. 2013;8(2).