76
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Ten Simple Rules for a Successful Cross-Disciplinary Collaboration

      editorial

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Introduction Cross-disciplinary collaborations have become an increasingly important part of science. They are seen as key if we are to find solutions to pressing, global-scale societal challenges, including green technologies, sustainable food production, and drug development. Regulators and policy-makers have realized the power of such collaborations, for example, in the 80 billion Euro "Horizon 2020" EU Framework Programme for Research and Innovation. This programme puts special emphasis on “breaking down barriers to create a genuine single market for knowledge, research and innovation” (http://ec.europa.eu/programmes/horizon2020/en/what-horizon-2020). Cross-disciplinary collaborations are key to all partners in computational biology. On the one hand, for scientists working in theoretical fields such as computer science, mathematics, or statistics, validation of predictions against experimental data is of the utmost importance. On the other hand, experimentalists, such as molecular biologists, geneticists, or clinicians, often want to reduce the number of experiments needed to achieve a certain scientific aim, to obtain insight into processes that are inaccessible using current experimental techniques, or to handle large volumes of data, which are far beyond any human analysis skills. The synergistic and skilfulcombining ofdifferent disciplines can achieve insight beyond current borders and thereby generate novel solutions to complex problems. The combination of methods and data from different fields can achieve more than the sum of the individual parts could do alone. This applies not only to computational biology but also tomany other academic disciplines. Initiating and successfully maintaining cross-disciplinary collaborations can be challenging but highly rewarding. In a previous publication in this series, ten simple rules for a successful collaboration were proposed [1]. In the present guide, we go one step further and focus on the specific challenges associated with cross-disciplinary research, from the perspective of the theoretician in particular. As research fellows of the 2020 Science project (http://www.2020science.net) and collaboration partners, we bring broad experience of developing interdisciplinary collaborations. We intend this guide to be for early career computational researchers as well as more senior scientists who are entering a cross-disciplinary setting for the first time. We describe the key benefits, as well as some possible pitfalls, arising from collaborations between scientists with very different backgrounds. Rule 1: Enjoy Entering a Completely New Field of Research Collaborating with scientists from other disciplines is an opportunity to learn about cutting-edge science directly from experts. Make the most of being the novice. No one expects you to know everything about the new field. In particular, there is no pressure to understand everything immediately, so ask the “stupid” questions. Demonstrating your interest and enthusiasm is of much higher value than pretending to know everything already. An interested audience makes information sharing much easier for all partners in a collaboration. You should prepare for a deluge of new ideas and approaches. It is a good practice to read relevant textbooks and review papers, which your collaborators should be able to recommend, in order to quickly grasp the vocabulary (see Rule 3) and key ideas of the new field. This will make it easier for you to establish a common parlance between you and your collaborators, and allow you to build from there. You should try to discuss your work with a range of scientists from complementary fields. As well as getting feedback, this can help you identify new collaborative opportunities. Remember that contacts that do not lead directly to collaborations can still prove useful later in your career. Rule 2: Go to the Wet Lab It is vitally important to understand where specific data sets come from. Just like mathematical and computational models, experiments have their own in-built assumptions, strengths, and weaknesses that you need to understand. What was the exact process of data collection? How many experiments can be performed in a given timeframe and how much do they cost? What were the constraints that led to the design of the experiments—how will you include this in your interpretation? If you plan to use the resulting data for model calibration or parameter fitting then try to obtain sufficient information to reproduce the experiment in silico. Papers in different domains have different perspectives and might not contain the data you are looking for in sufficient detail. Visiting the lab in person is often the most efficient way to get the information you need. A good understanding of the experimental setup might also suggest appropriate testcases for the computational studies. Try to talk to both the junior and senior scientists in the lab as they may give you different perspectives. There are social, as well as scientific, reasons for understanding life in the wet lab. As a computational scientist, it is easy to underestimate the commitment and resources necessary to acquire experimental data (see rule 4). Visiting a lab, and taking an interest in data collection, is a way of acknowledging your colleagues’ effort and the value of their data and expertise. Rule 3: Different Fields Have Different Terminologies: Learn the Language Science is full of subcultures using diverse and evolving jargon. Forming a successful cross-disciplinary relationship requires that you fully understand your collaboration partner. From classification schemes and methods to journals and research philosophy, it can be hard enough keeping up with developments in your own field, let alone others. For instance, neologisms can be ubiquitous in computational and biological sciences, where new terminology continually emerges from new methods, tools, and knowledge. Learn the other field’s jargon early on in the collaboration and ask basic questions about the meanings of words. For example: Ambiguity: “Model” is probably the most ambiguous word in science. Mathematical, statistical, experimental, observational, theoretical, computational, analytical, verbal, legal, mental, graphical, geometrical, structural, and workflow models all have different meanings. Almost every field will have its own interpretation of “model” and the semantics differ significantly. Synonyms: For example, removing entities above and below certain thresholds is termed “positive and negative selection” in immunology, while it is called “band-pass filter” in signal transduction. Context often matters, so try to understand nuances in the use of terms. It can be beneficial to build up a technical glossary. Evaluate your understanding by presenting it back to new colleagues and observe where your rudimentary understanding needs more work. Finally, agree on a joint nomenclature with your collaborators early in the project. Write equations and code in a consistent manner, standardise data formats, and use consistent style schemes in figures. Then talk through your outputs to discuss your collaborators’ understanding and involvement. A good relationship is based on mutually understandable communication. Rule 4: Different Fields Move at Different Speeds: Do not Become Impatient A huge variety of cultures and expectations regarding research and subsequent publication exist in different scientific disciplines. However, these differences can lead to stress when embarking upon multidisciplinary collaborations, unless they are acknowledged and effectively communicated at an early stage. It is important to accept the different pace of different fields, communicate well, and be patient. Research in experimental biology, for example, often involves long and arduous experiments, taking perhaps months or even years to complete. Animals or tissues may need to be grown, and weekends or nights spent in the lab tending to cell cultures and repeating experiments may be necessary. Some projects generate publications and co-authorships several years after a theoretician may have actually performed their in silico contribution to the work. Vice versa, computational aspects often involve more than simply pressing a button and computational resources may be limited. Do not make assumptions about how hard fellow collaborators are working based on how long they take to get back to you with results. Here, communication is of particular importance. Similarly, journals in different disciplines might have different periods of time from submission to publication. This can have knock-on effects when demonstrating your research output (see rule 5). Early communication of how long your part of the work is likely to take and why this amount of time is needed will help your collaboration to run more smoothly. Rule 5: Different Fields Have Different Reward Models: Know What You Can Expect It is important to recognise that the publication culture in the life sciences, and in experimental biology particularly, differs from that of the theoretical sciences. Such differences can include: Publication speed varies greatly. In experimental biology, publishing often takes several years, while certain theoretical papers can be published in a much shorter timescale (see also rule 4). Metrics, such as the impact factor (IF), are used by many organisations to evaluate your research [2,3]. Be aware that different fields have different impact factor scales. The journal impact factors mainly depend on the average length of reference lists in the field. For example, a journal with an impact factor of 3 in mathematics (295 journals, median IF 0.57, maximum IF 3.57) might be more prestigious than a journal with an impact factor of 30 in cell biology (184 journals; median IF 3.2; maximum IF 37.16) (based on Journal Citation Reports of Thomson Reuters, version 2012). In some fields, such as information technology, it can be the norm to publish new research in peer-reviewed conference proceedings instead of journals. The preferred ordering of authors on a manuscript may also depend strongly on the academic environment. The first author might be the scientist who contributed most or whose surname comes first alphabetically. The last author may be the principal investigator, the author who contributed least, or the author with the last surname alphabetically. The corresponding author can be seen as “in charge of the paper,” the principal investigator, or the person who volunteered for dealing with the correspondence. In some areas of biology, large consortia of authors are needed to conduct research. In some theoretical fields, people tend to publish with fewer authors. Thus, the definition of a “significant” contribution to a manuscript might differ markedly. It is important to be aware of these differences. Make sure that you discuss all these issues early with your collaborators to avoid misunderstandings and frustration. A well-planned publication strategy is fundamental in order to fulfil everyone’s expectations and accommodate the potential mismatch of timescales of theoretical/experimental work (see rule 4). Does your field and that of your partner value less frequent, higher impact publications or a series of smaller publications? One option is to start with methodological papers (both theoretical and experimental) while final publications describing the major breakthrough and how all the components are brought together could follow. Early methodological papers should already highlight the benefits of collaborating, e.g., theoretical work with experimentally sound assumptions and parameters, experimental work with solid data analytics. It is advisable to design initial publications without forgetting the greater scope of the collaboration. However, be aware that preceding papers might weaken your main publication if they anticipate parts of the results. Rule 6: What Different Fields Mean by “Data” Be prepared that scientists with experimental backgrounds might not have the same structured view on data and terminologies (see also rule 3) as you have. For scientists with a background in computer science, the lowest level of data organisation might be a spreadsheet where each column and row is well defined. For scientists with non-technical backgrounds, such a spreadsheet might represent the highest form of data organisation. Whenever possible, ask your collaboration partner for a standardized data format. Good guides on how to share data can be found in [4,5]. Always favour electronic forms of data and always keep a copy of the original file. You might also consider writing minutes about meetings and data specifications to avoid later misunderstandings. Do not blindly trust experimental data. Always perform “sanity checks” on the data you receive (graphs, frequency tables, mutually exclusive data, unnatural distributions, etc.). This will help you to see if you have interpreted the data correctly and allow you to ask questions if you have any doubts. Rule 7: Assess the Advantages and Disadvantages of Service Work Theoretical scientists can be a huge asset to multidisciplinary projects by providing experimentalists with computational tools to gather data, predictive methods, and advanced statistical modelling. Theoretical scientists often do substantial amounts of “service work” in a collaborative project, for example, by maintaining computer infrastructure and databases, keeping their in-house code base up-to-date, and statistically analysing data. Service work is often an excellent way to establish a collaboration, get the partners to trust in your ability and expertise, and learn enough about other disciplines to start making direct contributions, whilst, at the same time, co-authoring high-quality publications. Service work will show that you take the collaboration seriously and help you to establish a reputation as a reliable and analytically keen scientist who delivers fast, structured, and correct results. Nonetheless, service work is also risky, as it may take more time than you anticipated. Therefore, make sure to evaluate the amount of service work on a regular basis and be clear with your collaborators about what you expect in return before engaging in service work. To gain more insight into the “cost” of service work, keep a record of the amount of time spent on service tasks. This not only prevents your collaborators from treating your contributions lightly but also gives you a clear idea as to whether it is worthwhile to engage in such tasks and/or take on new ones. Crucial to minimizing the service “'load” is to make it easy to delegate tasks to others. When starting to develop analytical tools for others (whether software or mathematics), always take the perspective that these tools won't primarily be used by you, but by other collaborators. Hence, making your tools user-friendly, for example, by providing illustrative examples and documenting your code extensively [6], is essential. Rule 8: Create and Manage Structural Bonds Cross-disciplinary collaborations require structural bonds between the collaborators. It is only possible to break the silos of scientific disciplines and to become truly cross-disciplinary if a proper framework for scientific exchange is established. This can include regular meetings, workshops, symposia, attendance of each other’s group meetings, and co-teaching of courses. However, to keep your collaboration efficient, be careful about imposing too many obligations. While it is often necessary to leave the “comfort zone”of scientific disciplines, it is equally important not to frustrate your collaboration partners with too many details not relevant to their endeavours. Therefore, keep the number of meetings at a reasonable level and set clear agendas. Moreover, establishing these bonds often requires financial support, which can be achieved in different ways. For the initial setup phase, seed funding schemes maybe a useful resource. On the basis of these initial bonds, applications for larger grants can be submitted collaboratively. Many funding bodies offer special calls for cross-disciplinary research or favour cross-disciplinary proposals for both national and international settings. Examples include the "Horizon 2020" EU Framework (http://ec.europa.eu/programmes/horizon2020/en/what-horizon-2020), the Human Frontiers in Science Program (http://www.hfsp.org/), the US NSF/BIO–UK BBSRC Lead Agency Pilot (http://www.bbsrc.ac.uk/funding/internationalfunding/nsfbio-lead-agency-pilot.aspx), or internal projects created to achieve inter-faculty cooperation within a university. Once funding is secured, shared PhD students and postdocs can further strengthen bonds between collaboration partners. Shared supervision rewards with constant knowledge exchange, shared publications, and an interdisciplinary training for all. However, it is also important to protect junior scientists from “getting lost in cross-disciplinary collaboration.”In particular, there is a risk that they fall between two stools in their attempts to comply with the expectations and advice of two or more supervisors from completely different fields. Rule 9: Recognise When Things Are Not Working Well Unlike a marriage, collaborations are not necessarily intended to be continuous and permanent. A pragmatic approach can favour both parties. If major problems arise that cannot be solved after a couple of attempts, there are several possible next steps: Ostrich approach: Pretend that nothing has happened and hope that normality will soon be restored. We strongly advise against this tactic as it might lead to more frustration and potential damage to the relationship. Pause: Sometimes, one of the collaborators may find it harder than expected to deliver the agreed results or is overwhelmed with other work-related duties. If your collaborator has trouble with their side of the workload, it is often better to introduce a pause in the collaboration than to exert pressure on them. Deliberate pauses take pressure off your collaborations, and can save potential frustration on your side as well. Search for alternatives: Collaborations do not need to be exclusive. If you prioritise the collective work more highly than your collaborator, you might consider establishing a fresh collaboration with someone else who also values the work more. But be aware of your actions and the consequences they might have on your current collaboration. End a collaboration: If a collaboration has become unworkable or reached a natural terminus, it may be best to make a clean cut and end the collaboration once and for all. Many failures in collaborations could have been avoided by an early, proactive approach on arising problems. In cross-disciplinary settings, it could just be a problem of understanding (rule 3), impatience (rule 4), or lack of reward (rule 5). If you decide to end a collaboration, make sure to keep a working relationship with your former collaborator. This will allow you to properly handle existing structural bonds (rule 8) and allow you to potentially initiate other collaborations with the same partner. Rule 10: Be Synergistic Probably the most important quality of collaborations is the mutual gain that emerges. This usually works best when scientists with different, but complementary, skills decide to work together. One example might be the application of a novel, high-throughput computer algorithm to a vast quantity of experimental data. While the algorithm by itself might be brilliant, and publishable by benchmarking it on a publicly available dataset, it will shine even more when applied to a huge, unpublished dataset. At the same time, the huge dataset could be analysed by standard, semi-manual methods. This might also be publishable, but would take a long time and essential insights may be missed, which the novel algorithm would have delivered. Only by combining contributions from both sides does the work become more than the sum of its parts, achieving useful things for each partner and enabling insights that would not have been possible for either alone. Initiating and nurturing successfully synergistic relationships is an important and valuable skill. The success of an interdisciplinary collaboration depends on any number of factors, including which partner approached which and over what timescale each party envisages collaboration. One very important property for a long-term, effective, and mutually beneficial relationship is that both sides should feel they are winners during and, especially, after a project (see also rule 5). An outstanding book on how such win/win situations can be achieved is Fisher, Ury, and Patton’s Getting to Yes: Negotiating Agreement Without Giving In [7]: their advice includes inventing options for mutual gain, making sure to always give enough credit to partners, and caring for their interests as you would for your own. Then you will establish a truly successful and synergistic collaboration.

          Related collections

          Most cited references5

          • Record: found
          • Abstract: found
          • Article: found
          Is Open Access

          The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations

          Author summary Subjective assessments of the merit and likely impact of scientific publications are routinely made by scientists during their own research, and as part of promotion, appointment, and government committees. Using two large datasets in which scientists have made qualitative assessments of scientific merit, we show that scientists are poor at judging scientific merit and the likely impact of a paper, and that their judgment is strongly influenced by the journal in which the paper is published. We also demonstrate that the number of citations a paper accumulates is a poor measure of merit and we argue that although it is likely to be poor, the impact factor, of the journal in which a paper is published, may be the best measure of scientific merit currently available. Introduction How should we assess the merit of a scientific publication? Is the judgment of a well-informed scientist better than the impact factor (IF) of the journal the paper is published in, or the number of citations that a paper receives? These are important questions that have a bearing upon both individual careers and university departments. They are also critical to governments. Several countries, including the United Kingdom, Canada, and Australia, attempt to assess the merit of the research being produced by scientists and universities and then allocate funds according to performance. In the United Kingdom, this process was known until recently as the Research Assessment Exercise (RAE) (www.rae.ac.uk); it has now been rebranded the Research Excellence Framework (REF) (www.ref.ac.uk). The RAE was first performed in 1986 and has been repeated six times at roughly 5-yearly intervals. Although, the detailed structure of these exercises has varied, they have all relied, to a large extent, on the subjective assessment of scientific publications by a panel of experts. In a recent attempt to investigate how good scientists are at assessing the merit and impact of a scientific paper, Allen et al. [1] asked a panel of experts to rate 716 biomedical papers, which were the outcome of research funded, at least in part, by the Wellcome Trust (WT). They found that the level of agreement between experts was low, but that rater score was moderately correlated to the number of citations the paper had obtained 3 years after publication. However, they also found that the assessor score was more strongly correlated to the IF of the journal in which the paper was published than to the number of citations; it was therefore possible that the correlation between assessor scores, and between assessor scores and the number of citations was a consequence of assessors rating papers in high profile journals more highly, rather than an ability of assessors to judge the intrinsic merit or likely impact of a paper. Subsequently, Wardle [2] has assessed the reliability of post-publication subjective assessments of scientific publications using the Faculty of 1000 (F1000) database. In the F1000 database, a panel of experts is encouraged to select and recommend the most important research papers from biology and medicine to subscribers of the database. Papers in the F1000 database are rated “recommended,” “must read,” or “exceptional.” He showed, amongst ecological papers, that selected papers were cited more often than non-selected papers, and that papers rated must read or exceptional garnered more citations than those rated recommended. However, the differences were small; the average numbers of citations for non-selected, recommended, and must read/exceptional were 21.6, 30.9, and 37.5, respectively. Furthermore, he noted that F1000 faculty had failed to recommend any of the 12 most heavily cited papers from the year 2005. Nevertheless there is a good correlation between rates of article citation and subjective assessments of research merit at an institutional level for some subjects, including most sciences [3]. The RAE and similar procedures are time consuming and expensive. The last RAE, conducted in 2008, cost the British government £12 million to perform [4], and universities an additional £47 million to prepare their submissions [5]. This has led to the suggestion that it might be better to measure the merit of science using bibliometric methods, either by rating the merit of a paper by the IF of the journal in which it is published, or directly through the number of citations a paper receives [6]. Here we investigate three methods of assessing the merit of a scientific publication: subjective post-publication peer review, the number of citations a paper accrues, and the IF. We do not attempt to define merit rigorously; it is simply the qualities in a paper that lead a scientist to rate a paper highly; it is likely that this largely depends upon the perceived importance of the paper. We also largely restrict our analysis to the assessment of merit rather than impact; for example, as we show below, the number of citations, which is a measure of impact, is a very poor measure of the underlying merit of the science, because the accumulation of citations is highly stochastic. We have considered the IF, rather than other measures of journal impact, of which there are many (see [7] for list of 39 measures), because it is simple and widely used. Results Datasets To investigate methods of assessing scientific merit we used two datasets [8] in which the merit of a scientific publication had been subjectively assessed by a panel of experts: (i) 716 papers from the WT dataset mentioned in the introduction, each of which had been scored by two assessors and which had been published in 2005, and (ii) 5,811 papers, also published in 2005, from the F1000 database, 1,328 of which had been assessed by more than one assessor. For each of these papers we collated citation information ∼6 years after publication. We also obtained the IF of the journal in which the paper had been published (further details in the Materials and Methods). The datasets have strengths and weaknesses. The F1000 dataset is considerably larger than the WT dataset, but it is papers that the assessors considered good enough to be featured in F1000; the papers therefore probably represent a narrower range of merit than in the WT dataset. Furthermore, the scores of two assessors are not independent in the F1000 dataset because the second assessor might have known the score of the first assessor, and F1000 scores have the potential to affect rates of citation, whereas the WT assessments were independent and confidential. The papers in both datasets are drawn from a diverse set of journals covering a broad range of IFs (Figure 1). Perhaps not surprisingly the F1000 data tend to be drawn from journals with higher IF, because they have been chosen by the assessors for inclusion in the F1000 database (Mean IF: WT = 6.6; F1000 = 13.9). 10.1371/journal.pbio.1001675.g001 Figure 1 The distribution of the impact factor in the two datasets. Subjective Assessment of Merit If scientists are good at assessing the merit of a scientific publication, and they agree on what merit is, then there should be a good level of agreement between assessors. Indeed assessors gave the same score in 47% and 50% of cases in the WT and F1000 datasets, respectively (Tables 1 and 2). However, we would have expected them to agree 40% of the time by chance alone in both datasets, so the excess agreement above these expectations is small. The correlations between assessor scores are correspondingly modest (WT r = 0.36, p 20 compared to those with IF 20, respectively. If we remove the influence of IF upon assessor score, the correlations between assessor scores drop below 0.2 (partial correlations between assessor scores controlling for IF: WT, r = 0.15, p 30 in the F1000 dataset. Number of Citations An alternative to the subjective assessment of scientific merit is the use of bibliometric measures such as the IF of the journal in which the paper is published or the number of citations the paper receives. The number of citations a paper accumulates is likely to be subject to random fluctuation—two papers of similar merit will not accrue the same number of citations even if they are published in similar journals. We can infer the relative error variance associated with this process as follows. Let us assume that the number of citations within a journal is due to the intrinsic merit of the paper plus some error. The correlation between assessor score and the number of citations is therefore expected to be where and is the error variance associated with the accumulation of citations (see Materials and Methods for derivation). Hence we can estimate the error variance associated with the accumulation of citations relative to variance in merit by simultaneously considering the correlation between assessor scores and the correlation between assessor scores and the number of citations. If we assume that assessors and the number of citations are unaffected by the IF of the journal, then we estimate the ratio of the error variance associated with citations to be approximately 1.5 times the variance in merit (WT rc  = 1.5 [0.83–2.7]; F1000 rc  = 1.6 [0.86–2.6]) and if we assume that the correlation between assessor score and IF is entirely due to bias then we estimate, using the partial correlation between score and citations, controlling for IF, that the ratio of the error variance to the variance in merit within journals to be greater than 5-fold (WT rc  = 5.6 [1.2–42]; F1000 rc  = 9.8 [4.0–31]). These estimates underestimate the error variance because they do not take into account the variance associated with which journal a paper gets published in; the stochasticity associated with this process will generate additional variance in the number of citations a paper accumulates if the journal affects the number of citations a paper receives, as analyses of duplicate papers suggest [9]–[11]. Impact Factor The IF might potentially be a better measure of merit than either a post-publication assessment or the number of citations, since several individuals are typically involved in a decision to publish, so the error variance associated with their combined assessment should be lower than that associated with the number of citations; although such benefits can be partially undermined by having a single individual determine whether a manuscript should be reviewed or by rejecting manuscripts if one review is unsupportive. Unfortunately, it seems likely that the IF will also be subject to considerable error. If we combine n independent assessments we expect the ratio of the error variance to the variance in merit in their combined qualitative assessment to be reduced by a factor n. Hence, if we assume that pre-publication assessments are of similar quality to post-publication assessments, and that three individuals have equal influence over the decision to publish a paper, their combined assessment is still likely to be dominated by error not merit; e.g., if we average the estimates of rs from the correlation between scores and between scores controlling for IF we have  = 3.7 and 3.9, for the WT and F1000 datasets, respectively, which means that the ratio of the error variance associated with the combined assessor score will be ∼1.2× the variance in merit; i.e., the error variance is still larger than the variance in merit. Discussion Our results have some important implications for the assessment of science. We have shown that scientists are poor at estimating the merit of a scientific publication; their assessments are error prone and biased by the journal in which the paper is published. In addition, subjective assessments are expensive and time-consuming. Scientists are also poor at predicting the future impact of a paper, as measured by the number of citations a paper accumulates. This appears to be due to two factors; scientists are not good at assessing merit and the accumulation of citations is a highly stochastic process, such that two papers of similar merit can accumulate very different numbers of citations just by chance. The IF and the number of citations are also likely to be poor measures of merit, though they may be better measures of impact. The number of citations is a poor measure of merit for two reasons. First, the accumulation of citations is a highly stochastic process, so the number of citations is only poorly correlated to merit. It has previously been suggested that the error variance associated with the accumulation of citations is small based on the strong correlation between the number of citations in successive years [12], but such an analysis does not take into account the influence that citations have on subsequent levels of citation—the citations in successive years are not independent. Second, as others have shown, the number of citations is strongly affected by the journal in which the paper is published [9]–[11]. There are also additional problems associated with using the number of citations as a measure of merit since it is influenced by factors such as the geographic origin of the authors [13],[14], whether they are English speaking [14],[15], and the gender of the authors [16],[17] (though see [15]). The problems of using the number of citations as a measure of merit are also likely to affect other article level metrics such as downloads and social network activity. The IF is likely to be poor because it is based on subjective assessment, although it does have the benefit of being a pre-publication assessment, and hence not influenced by the journal in which the paper has been published. In fact, given that the scientific community has already made an assessment of a paper's merit in deciding where it should be published, it seems odd to suggest that we could do better with post-publication assessment. Post-publication assessment cannot hope to be better than pre-publication assessment unless more individuals are involved in making the assessment, and even then it seems difficult to avoid the bias in favour of papers published in high-ranking journals that seems to pervade our assessments. However, the correlation between merit and IF is likely to be far from perfect. In fact the available evidence suggests there is little correlation between merit and IF, at least amongst low IF journals. The IF depends upon two factors, the merit of the papers being published by the journal and the effect that the journal has on the number of citations for a given level of merit. In the most extensive analysis of its kind, Lariviere and Gingras [11] analysed 4,532 cases in which the same paper had been published in two different journals; on average the two journals differed by 2.4-fold in their IFs and the papers differed 1.9-fold in the number of citations they had accumulated, suggesting that the higher IF journals in their analysis had gained their higher IF largely through positive feedback, not by publishing better papers. However, the mean IF of the journals in this study was less than one, and it seems unlikely that the IF is entirely a function of positive feedback amongst higher IF journals. Nevertheless the tendency for journals to affect the number of citations a paper receives means that IFs are NOT a quantitative measure of merit; a paper published in a journal with an IF of 30 is not on average six times better than one published in a journal with an IF of 5. The IF has a number of additional benefits over subjective post-publication review and the number of citations as measures of merit. First, it is transparent. Second, it removes the difficult task of determining which papers should be selected for submission to an assessment exercise such as the RAE or REF; is it better to submit a paper in a high IF journal, a paper that has been highly cited, even if it appears in a low IF journal, or a paper that the submitter believes is their best work? Third, it is relatively cheap to implement. And fourth it is an instantaneous measure of merit. The use of IF as a measure merit is unpopular with many scientists, a dissatisfaction that has recently found its voice in the San Francisco Declaration of Research Assessment (DORA) (http://am.ascb.org/dora/). The declaration urges institutions, funding bodies, and governments to avoid using journal level metrics, such as the IF, to assess the merit of scientific papers. Instead it promotes the use of subjective review and article level metrics. However, as we have shown, both subjective post-publication review and the number of citations, an example of an article level metric, are highly error prone measures of merit. Furthermore, the declaration fails to appreciate that journal level metrics are a form of pre-publication subjective review. It has been argued that the IF is a poor measure of merit because the variation in the number of citations, accumulated by papers published in the same journal, is large [9],[18]; the IF is therefore unrepresentative of the number of citations that individual papers accumulate. However, as we have shown the accumulation of citations is highly stochastic, so we would expect a large variance in the number of citations even if the IF were a perfect measure of merit. There are however many problems with using the IF besides the error associated with the assessment. The IF is influenced by the type of papers that are published and with the way in which the IF is calculated [18],[19]. Furthermore it clearly needs to be standardized across fields. A possible solution to these problems may be to get leading scientists to rank the journals in their field, and to use these ranks as a measure of merit, rather than the IF. Finally, possibly the biggest problem with the IF is simply our reaction to it; we have a tendency to overrate papers published in high IF journals. So if are to use the IF, we need to reduce this tendency; one approach might be to rank all papers by their IF and assign scores by rank. The REF will be performed in the United Kingdom next year in 2014. The assessment of publications forms the largest component of this exercise. This will be done by subjective post-publication review, with citation information being provided to some panels. However, as we have shown, both subjective review and the number of citations are very error prone measures of merit, so it seems likely that these assessments will also be extremely error prone, particularly given the volume of assessments that need to be made. For example, sub-panel 14 in the 2008 version of the RAE assessed ∼9,000 research outputs, each of which was assessed by two members of a 19 person panel; therefore each panel member assessed an average of just under 1,000 papers within a few months. We have also shown that assessors tend to overrate science in high IF journals, and although the REF [20], like the RAE before it [21], contains a stipulation that the journal of publication should not be taken into account in making an assessment, it is unclear whether this is possible. In our research we have not been able to address another potential problem for a process such as the REF. It seems very likely that assessors will differ in their mean score—some assessors will tend to give higher scores than other assessors. This could potentially affect the overall score for a department, particularly if the department is small and its outputs scored by relatively few assessors. The REF actually represents an unrivalled opportunity to investigate the assessment of scientific research and to assess the quality of the data produced by such an exercise. We would therefore encourage the REF to have all components of every submission assessed by two independent assessors and then investigate how strongly these are correlated and whether some assessors score more generously than others. Only then can we determine how reliable the data are. In summary, we have shown that none of the measures of scientific merit that we have investigated are reliable. In particular subjective peer review is error prone, biased, and expensive; we must therefore question whether using peer review in exercises such as the RAE and the REF is worth the huge amount of resources spent on them. Ultimately the only way to obtain (a largely) unbiased estimate of merit is to have pre-publication assessment, by several independent assessors, of manuscripts devoid of author's names and addresses. Nevertheless this will be a noisy estimate of merit unless we are prepared to engage many reviewers for each paper. Materials and Methods We compiled subjective assessments from two sources. The largest of these datasets was from the F1000 database (www.F1000.com). In the F1000 database a panel of experts selects and recommends papers from biology and medicine to subscribers of the database. Papers in the F1000 database are rated “recommended” (numerical score 6), “must read” (8), or “exceptional” (10). We chose to take all papers that been published in a single year, 2005; this was judged to be sufficiently recent to reflect current trends and biases in publishing, but sufficiently long ago to allow substantial numbers of citations to have accumulated. We restricted our analysis to those papers that had been assessed within 12 months of publication to minimize the influence that subsequent discussion and citation might have on the assessment. This gave us a dataset of 5,811 papers, with 1,328 papers having been assessed by two or more assessors within 12 months. We chose to consider the 5-year IFs, since it was over a similar time-scale to the period over which we collected citations. However, in our dataset the 2-year and 5-year IFs are very highly correlated (r = 0.99). Citations were obtained from Google Scholar in 2011. We also analysed the WT data collected by Allen et al. [1]. This is a dataset of 716 biomedical papers, which were published in 2005, and assessed within 6 months by two assessors. Papers were given scores of 4, landmark; 3, major addition to knowledge; 2, useful step forward; and 1, for the record. The scores were sorted such that the higher score was usually allocated to the first assessor; this will affect the correlations by reducing the variance within the first (and second) assessor scores. As a consequence the scores were randomly re-allocated to the first and second assessor. Citations were collated from Google Scholar in 2011. As with the F1000 data we used 5 year IFs from 2010. Data have been deposited with Dryad [8]. Because most journals are poorly represented in each dataset we estimated the within and between journal variance in the number of citations as follows. We rounded the IF to the nearest integer then grouped journals according to the integer value. We then performed ANOVA on those groups for which we had ten or more publications. Estimates of the error variance in assessment relative to variance in merit can be estimated as follows. Let us assume that the score (s) given by an assessor is linearly dependent upon the merit (m) and some error (es ): s = m+es . Let the variance in merit be and that for the error be , so the variance in the score is . If two assessors score the same paper the covariance between their scores will simply be and the hence the correlation between scores is (1) where . If we similarly assume that the number of citations a paper accumulates depends linearly on the merit and some error (with variance ) then the covariance between an assessor's score and the number of citations is and the correlation is (2) where . It is therefore straightforward to estimate rs and rc , and to obtain confidence intervals by bootstrapping the data. Supporting Information Table S1 The correlations, partial correlations, and standardized regression coefficients between assessor score (AS) and IF and the number of citations (CIT). ***p<0.001. (DOCX) Click here for additional data file. Table S2 Spearman correlation coefficients between assessor scores and assessor scores and the number of citations and the IF. ***p<0.001. (DOCX) Click here for additional data file. Table S3 The correlations, partial correlations, and standardized regression coefficients between assessor score (AS) and the log of IF and the log of the number of citations (CIT). ***p<0.001. (DOCX) Click here for additional data file.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            10 Simple Rules for the Care and Feeding of Scientific Data

            This article offers a short guide to the steps scientists can take to ensure that their data and associated analyses continue to be of value and to be recognized. In just the past few years, hundreds of scholarly papers and reports have been written on questions of data sharing, data provenance, research reproducibility, licensing, attribution, privacy, and more, but our goal here is not to review that literature. Instead, we present a short guide intended for researchers who want to know why it is important to "care for and feed" data, with some practical advice on how to do that.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Ten Simple Rules for a Successful Collaboration

              Scientific research has always been a collaborative undertaking, and this is particularly true today. For example, between 1981 and 2001, the average number of coauthors on a paper for the Proceedings of the National Academy of Sciences U S A rose from 3.9 to 8.4 [1]. Why the increase? Biology has always been considered the study of living systems; many of us now think of it as the study of complex systems. Understanding this complexity requires experts in many different domains. In short, these days success in being a biologist depends more on one's ability to collaborate than ever before. The Medical Research Centers in the United Kingdom figured this out long ago, and the new Janelia Farm research campus of the Howard Hughes Medical Institute in the United States has got the idea, as it strongly promotes intra- and inter-institutional collaborations [2]. Given that collaboration is crucial, how do you go about picking the right collaborators, and how can you best make the collaboration work? Here are ten simple rules based on our experience that we hope will help. Additional suggestions can be found in the references [3,4]. Above all, keep in mind that these rules are for both you and your collaborators. Always remember to treat your collaborators as you would want to be treated yourself—empathy is key. Rule 1: Do Not Be Lured into Just Any Collaboration Learn to say no, even if it is to an attractive grant that would involve significant amounts of money and/or if it is a collaboration with someone more established and well-known. It is easier to say no at the beginning—the longer an ill-fated collaboration drags on, the harder it is to sever, and the worse it will be in the end. Enter a collaboration because of a shared passion for the science, not just because you think getting that grant or working with this person would look good on your curriculum vitae. Attending meetings is a perfect opportunity to interact with people who have shared interests [5]. Take time to consider all aspects of the potential collaboration. Ask yourself, will this collaboration really make a difference in my research? Does this grant constitute a valid motivation to seek out that collaboration? Do I have the expertise required to tackle the proposed tasks? What priority will this teamwork have for me? Will I be able to deliver on time? If the answer is no for even one of these questions, the collaboration could be ill-fated. Enter a collaboration because of a shared passion for the science . . . Rule 2: Decide at the Beginning Who Will Work on What Tasks Carefully establishing the purpose of the collaboration and delegating responsibilities is priceless. Often the collaboration will be defined by a grant. In that case, revisit the specific aims regularly and be sure the respective responsibilities are being met. Otherwise, consider writing a memo of understanding, or, if that is too formal, at least an e-mail about who is responsible for what. Given the delegation of tasks, discuss expectations for authorship early in the work. Having said that, leave room for evolution over the course of the collaboration. New ideas will arise. Have a mutual understanding up-front such that these ideas can be embraced as an extension of the original collaboration. Discuss adjustments to the timelines and the order of authors on the final published paper, accordingly. In any case, be comfortable with the anticipated credit you will get from the work. The history of science is littered with stories of unacknowledged contributions. Rule 3: Stick to Your Tasks Scientific research is such that every answered question begs a number of new questions to be answered. Do not digress into these new questions without first discussing them with your collaborators. Do not change your initial plans without discussing the change with your collaborators. Thinking they will be pleased with your new approach or innovation is often misplaced and can lead to conflict. Rule 4: Be Open and Honest Share data, protocols, materials, etc., and make papers accessible prior to publication. Remain available. A trusting relationship is important for the collaborative understanding of the problem being tackled and for the subsequent joint thinking throughout the evolution of the collaboration. Rule 5: Feel Respect, Get Respect If you do not have respect for the scientific work of your collaborators, you should definitely not be collaborating. Respect here especially means playing by Rules 2–4. If you do not respect your collaborators, it will show. Likewise, if they don't respect you. Look for the signs. The signs will depend on the personality of your collaborators and range from being aggressive to being passive–aggressive. For example, getting your tasks done in a timely manner should be your priority. There is nothing more frustrating for your collaborators than to have to throttle their progress while they are waiting for you to send them your data. Showing respect would be to inform your collaborator when you cannot make a previously agreed-upon deadline, so that other arrangements can be made. Rule 6: Communicate, Communicate, and Communicate Consistent communication with your collaborators is the best way to make sure the partnership is going in the planned direction. Nothing new here, it is the same as for friendship and marriage. Communication is always better face-to-face if possible, for example by traveling to meet your collaborators, or by scheduling discussion related to your collaborations during conferences that the people involved will attend. Synchronous communication by telephone or video teleconferencing is preferred over asynchronous collaboration by e-mail (data could be exchanged by e-mail prior to a call so that everyone can refer to the data while talking). Rule 7: Protect Yourself from a Collaboration That Turns Sour The excitement of a new collaboration can often quickly dissipate as the first hurdles to any new project appear. The direct consequence can be a progressive lack of interest and focus to get the job done. To avoid the subsequent frustrations and resentment that could even impact your work in general, give three chances to your collaborators to get back on track. After all, your collaborators could just be having a difficult time for reasons outside of their control and unanticipated at the time the collaboration started. After three chances, if it feels like the collaboration cannot be saved, move on. At that point try to minimize the role of your collaborators in your work: think carefully about the most basic help you need from them and get it while you can (e.g., when having a phone call or a meeting in person). You may still need to deal with the co-authorship, but hopefully for one paper only! Rule 8: Always Acknowledge and Cite Your Collaborators This applies as soon as you mention preliminary results. Be clear on who undertook what aspect of the work being reported. Additionally, citing your collaborators can reveal your dynamism and your skills at developing prosperous professional relationships. This skill will be valued by your peers throughout your career. Rule 9: Seek Advice from Experienced Scientists Even though you may not encounter severe difficulties that would result in the failure of the partnership, each collaboration will come with a particular set of challenges. To overcome these obstacles, interact with colleagues not involved in the work, such as your former advisors or professors in your department who have probably been through all kinds of collaborations. They will offer insightful advice that will help you move beyond the current crisis. Remember, however, that a crisis can occasionally lead to a breakthrough. Do not, therefore, give up on the collaboration too easily. Rule 10: If Your Collaboration Satisfies You, Keep It Going Ever wondered why a pair of authors has published so many papers together? Well, it is like any good recipe: when you find one that works, you cook it again and again. Successful teamwork will tend to keep flourishing—the first paper will stimulate deeper and/or broader studies that will in turn lead to more papers. As you get to know your collaborators, you begin to understand work habits, strengths but also weaknesses, as well as respective areas of knowledge. Accepting these things and working together can make the work advance rapidly, but do not hurry: it takes time and effort from both sides to get to this point. Collaborations often come unexpectedly, just like this one. One of us (PEB) as Editor-in-Chief was approached not just with the idea for these Ten Rules, but with a draft set of rules that needed only minor reworking. As you can see, we have obeyed Rule 8. 
                Bookmark

                Author and article information

                Journal
                PLoS Comput Biol
                PLoS Comput. Biol
                plos
                ploscomp
                PLoS Computational Biology
                Public Library of Science (San Francisco, CA USA )
                1553-734X
                1553-7358
                30 April 2015
                April 2015
                : 11
                : 4
                : e1004214
                Affiliations
                [1 ]Department of Statistics, University of Oxford, Oxford, United Kingdom
                [2 ]Centre for Computational Science, Department of Chemistry, University College London, London, United Kingdom
                [3 ]Centre for Mathematics, Physics and Engineering in the Life Sciences and Experimental Biology, University College London, London, United Kingdom
                [4 ]Department of Computer Science, University of Oxford, Oxford, United Kingdom
                [5 ]Wolfson Centre for Mathematical Biology, Mathematical Institute, University of Oxford, Oxford, United Kingdom
                [6 ]Department of Mathematics, Imperial College London, London, United Kingdom
                [7 ]Department of Genetics, Evolution and Environment, University College London, London, United Kingdom
                [8 ]Centre for Information & Innovation Law, University of Copenhagen, Copenhagen, Denmark
                [9 ]Guest Scholar, Harvard Law School, Cambridge, Massachusetts, United States of America
                [10 ]Visiting Research Fellow, University of Oxford, Oxford, United Kingdom
                [11 ]Department of Dermatology, Medical University of Vienna, Vienna, Austria
                [12 ]Laboratory of Gene Regulation and Signal Transduction, Departments of Pharmacology and Pathology, University of California, California, San Diego, United States of America
                [13 ]Department of Mathematical Sciences, University of Bath, Bath, United Kingdom
                Author notes

                The authors have declared that no competing interests exist.

                Article
                PCOMPBIOL-D-14-01972
                10.1371/journal.pcbi.1004214
                4415777
                25928184
                44a15305-24f9-4669-9d29-f4f9db0ba801
                Copyright @ 2015

                This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

                History
                Page count
                Figures: 0, Tables: 0, Pages: 7
                Funding
                The 2020 Science programme is funded through the EPSRC Cross-Disciplinary Interface Programme (grant number EP/I017909/1). The authors received no specific funding for this article.
                Categories
                Editorial

                Quantitative & Systems biology
                Quantitative & Systems biology

                Comments

                Comment on this article