Full Text
Home > Policy > White Paper, Notice, Announcement > White Paper > JAPANESE GOVERNMENT POLICES IN EDUCATION, SCIENCE, SPORTS AND CULTURE 1997 > Scientific Research Chapter 3 Section 3 4

Scientific Research: Opening the Door to the Future
Chapter 3 New Trends in the Promotion of Science
Section 3: Improving Evaluation Systems for Scientific Research
4. Improving Research Evaluation

(1) Creating an Overall Evaluation Policy

The Science and Technology Basic Plan, decided on by the Cabinet in July 1996, stipulates that impartial evaluation must be a facet of efforts to create a new research and development system. It also states that the evaluation process should be facilitated through the formulation of basic guidelines to be used for all government research and development, with due respect for public opinion. The Subcommittee on Evaluation Guidelines was established under the Committee on Policy Matters of the Council for Science and Technology and commenced deliberations to formulate a draft policy. In July 1997 the Council for Science and Technology submitted an opinion to the Prime Minister. This was followed by the formulation of "Views Concerning National General Guidelines on the Method of Evaluations for Government Research and Development," which was adopted by the Prime Minister and is now serving as guidelines for evaluations.

The guidelines cover the entire range of government-funded research and development carried out by organizations within government ministries or agencies or by research and development institutions, including national experiment and research institutes, national universities, and special corporations. The guidelines do not cover research in the humanities and social sciences, however, and they also stipulate that evaluations should take into account the characteristics of scientific research.

(2) Science Council Deliberations

Because evaluation is indispensable to making scientific research more lively and creative, the Science Council has long been considering approaches to evaluation across the entire spectrum of scientific research.

In a July 1992 report titled "Strategies for Comprehensive Promotion of Scientific Research with the Prospect of the Twenty-first Century," the Science Council emphasized the importance of regular self-monitoring and self-evaluation and continuous improvement efforts by research organizations as a way of maintaining an environment in which researchers can function dynamically and the organizations can sustain their vitality, The report also states that, while cases in other countries may serve as references, Japan needs to develop its own evaluation system to ensure the prioritized allocation of research funds.

Since this report was issued, the Science Council's Special Committee on Scientific Research Systems has continued its deliberations. In July 1997 it produced an interim report titled "Evaluation in Scientific Research," which also took account of progress in the Science Council deliberations described above. Subsequently the committee has been soliciting opinions from various quarters and conducting deliberations with a view to formulating a recommendation.

Overview of "Evaluation in Scientific Research (Interim Report)"


Basic Thinking on Evaluation

Evaluations should focus primarily on academic significance, but, depending on the field and research objectives, should also take into account contributions to society and the economy.
Criteria for academic significance should include the level and originality of the research, the potential for future progress, and contributions to other fields. Criteria for contributions to society and the economy should include the creation of new technology, the formation of intellectual property (patents, etc.), the development of new industrial infrastructure, the improvement of the infrastructure for daily living, policy formation, and the advancement of humanity and culture. Contributions to education and human resource development must also be given consideration.
The humanities and social sciences are closely related to culture and tradition, so evaluation in these areas must reflect a diversity of values. Because these areas are not amenable to simple, uniform criteria, full consideration must be given to the limitations this places on the effectiveness of numerical indicators and on the general applicability of evaluations.
A nationwide system is needed to support the collection, collation, analysis, and distribution of information about objective data that can be used as evaluation indicators, international research trends, evaluation findings, and the distribution of research resources, as well as other information from throughout Japan. The quality and quantity of data in the databases of the National Center for Science Information Systems, and systems for utilizing that information, must be improved.
Active steps to inform the public of evaluation results is important.


Evaluation of Research Projects


Evaluation of research projects receiving ordinary research funds

Peer review systems for specialist researchers will need to be further refined, and approaches to self-imposed evaluation developed.
Evaluation could also be made a facet of self-monitoring and self-examination undertaken by the university on institution of affiliation.


Evaluation of research projects receiving general research grants

Disclosure of reasons for nonacceptance should be expanded, and consideration should also be given to the creation of systems to inform society at large of outstanding research achievements.
Consideration must also be given to allocation screening suited to the objectives and characteristics of each research category, as well as to improving interim and postcompletion evaluations.


Evaluation of research projects promoted from the viewpoint of science and technology policy

For creative basic research (under the New Program System), further efforts are needed in such areas as disseminating evaluation results to the wider community and evaluating the project selection process itself.
In the case of the Research for the Future Program of the Japan Society for the Promotion of Science, the methods used in interim and postcompletion evaluations must be considered thoroughly and a suitable system arranged.

(Evaluation of large-scale research projects)

Further efforts are needed to enhance evaluation methods (by, for example, seeking the opinions of foreign researchers and outside experts) and to make evaluation results known to the public.
In the case of international joint research, it will be necessary to carry out evaluations effectively and efficiently in cooperation with governments or other organizations in the countries concerned.


Evaluation of the Research Functions of Universities and Affiliated Research Institutions


Self-monitoring and self-evaluation

Greater efforts must be made to set up implementation arrangements that suit individual universities' circumstances, to encourage the participation of outside researchers and experts in evaluations, to improve and enhance evaluation criteria and methods, and to establish the practice of carrying out self-monitoring and self-evaluation at regular intervals. It will also be necessary to support such efforts on the part of universities.
Steps should be taken to make the results of self-monitoring and self-evaluation available to the public in more comprehensible forms.


Evaluation of inter-university research institutes

It will be necessary to create an organization covering all inter-university institutes to carry out organized, systematic self-monitoring and self-evaluation in which users and other outside researchers participate.
Increased use should be made of the evaluation functions of boards of trustees and management councils, and positive steps should be taken to make the results known to the wider community.
It will be necessary to improve support systems for external evaluations carried out independently by inter-university research institutes and joint-use research institutes attached to national universities. Measures are needed to provide active support for the dissemination of information about research activities, accomplishments, and evaluation results in Japan and overseas.

(C)COPYRIGHT Ministry of Education, Culture, Sports, Science and Technology

Back to Top   MEXT HOME