The outputs from scientific research are many and varied, including: research articles reporting new knowledge, data, reagents, and software; intellectual property; and highly trained young scientists. Funding agencies, institutions that employ scientists, and scientists themselves all have a desire, and need, to assess the quality and impact of scientific outputs. It is imperative that scientific output be measured accurately, evaluated wisely, and used thoughtfully.
The Journal Impact Factor is frequently used as the primary parameter with which to measure the scientific output of individuals and institutions. This metric, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment.
These limitations include: A) citation distributions within journals are highly skewed; B) the properties of the Journal Impact Factor are field-specific; it is a composite of multiple, highly diverse article types, including primary research papers and reviews; C) Impact Factors can be manipulated by editorial policy; and D) data used to calculate the Impact Factors are neither transparent nor openly available to the public.
Improving the evaluation of research quality
Below we make a number of recommendations for improving the way in which the actual quality of research output is evaluated. Outputs other than research articles will grow in importance in assessing research effectiveness in the future, but the peer-reviewed research paper will remain a central research output that informs research assessment.
Our recommendations therefore focus primarily on practices relating to research articles published in peer-reviewed journals, but can and should be extended by recognising additional products, such as datasets, as important research outputs. The recommendations are aimed at funding agencies, academic institutions, journals, organisations that compile metrics, and individual researchers.
A number of themes run through these recommendations:
- the need to eliminate use of journal-based metrics, such as impact factors, in funding, appointment and promotion considerations;
- the need to assess research on its own merits rather than on the basis of the journal in which the research is published;
- the need to capitalise on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).
We recognise that many funding agencies, institutions, publishers, and researchers are already encouraging improved practices in research assessment. Such steps are beginning to increase the momentum toward more sophisticated and meaningful approaches to research evaluation that can now be built upon and adopted by all of the key constituencies involved.
The signatories of the San Francisco Declaration support the adoption of the following practices in research assessment:
General Recommendation
1. Do not use journal-based metrics, such as journal impact factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion or funding decisions.
For funding agencies
2. Be explicit about the criteria used in evaluating the scientific productivity of grant applicants and clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
3. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
For institutions
4. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
5. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
For publishers
6. Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics that provide a richer view of journal performance.
7. Make available a range of article-level metrics to encourage a shift toward assessment based on the scientific content of an article rather than publication metrics of the journal in which it was published.
8. Encourage responsible authorship practices and the provision of information about the specific contributions of each author.
9. Whether a journal is open-access or subscription-based, remove all reuse limitations on reference lists in research articles and make them available under the Creative Commons Public Domain Dedication.
10. Remove or reduce the constraints on the number of references in research articles, and, where appropriate, mandate the citation of primary literature in favour of reviews in order to give credit to the group(s) who first reported a finding.
For organisations that compile metrics
11. Be open and transparent by providing data and methods used to calculate all metrics.
12. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data.
13. Be clear that inappropriate manipulation of metrics will not be tolerated; be explicit about what constitutes inappropriate manipulation and what measures will be taken to combat this.
14. Account for the variation in article types (e.g., reviews versus research articles), and in different subject areas when metrics are used, aggregated, or compared
For researchers
15. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.
16. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.
17. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs.
18. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote best practice that focuses on the value and influence of specific research outputs.
For a list of signatories see http://am.ascb.org/dora/
Receive the Funding Newswire [full access requires a subscription] each Tuesday, our Policy Bulletin each Thursday, and news about bridging Europe’s east-west innovation gap twice a month in The Widening. 
A unique international forum for public research organisations and companies to connect their external engagement with strategic interests around their R&D system.
