Home > Policy > White Paper, Notice, Announcement > White Paper > JAPANESE GOVERNMENT POLICES IN EDUCATION, SCIENCE, SPORTS AND CULTURE 1997 > Scientific Research Chapter 3 Section 3 3

Because of the diversity and broad scope of scientific research, the assessment of research themes must employ systems that reflect the characteristics of research projects, including objectives, nature, methods, scale, duration, and field. Similarly, systems developed for evaluating research institutions must reflect the aims for which those institutions were established, the fields in which they operate, and other characteristics.

Examiners are the most important element in evaluation systems, and high-quality personnel must be recruited to ensure appropriate evaluation.

Participants in the Survey on Scientific Research and Evaluation were asked to state their views on the methods used to select examiners. The majority felt that the views of research groups should be taken into account when choosing decision-makers (persons who plan evaluations and make some kind of decision on the basis of evaluation results) ( Figure 3-12 ).

Survey respondents were also asked to state their views on the tenure of examiners. Over 80% felt that, to prevent concentration on specific individuals, examiners should be replaced every 2-3 years, and as many people as possible should participate. With regard to the disclosure of examiners' names, approximately one-half of researchers thought that examiners' names should not be disclosed until the completion of the evaluation process but should be disclosed thereafter. The names of examiners for grants-in-aid for scientific research are always disclosed after the completion of the evaluation process, if not before.

Figure 3-12: Researchers' Attitudes Regarding Examiners

The majority of researchers thought that the reasons for evaluations should be disclosed either generally or just to the those whose work was assessed. Disclosure of reasons for nonacceptance has already been introduced for some types of grants-in-aid for scientific research. MESSC is considering extending the use of this approach, since disclosure not only provides guidance to those who undergo evaluation, but also contributes to assessing the evaluation process itself.

The evaluation of research themes can be carried out in three stages: before the work starts, midway through a project, and after completion. Evaluations must be scheduled in ways suited to the content, nature, and other characteristics of research projects. Thus, for example, interim evaluations may be omitted in the case of short-term research, but several interim evaluations may be carried out at regular intervals during medium-and long-term projects. What is more, the various types of evaluations are closely related: Postcompletion evaluations are also further evaluations of the results of preproject and interim evaluations and provide useful information for selecting new research themes.

Examples of evaluatin reports from inter-university research institutes and joint-use research institutes attached to national universities.

Objective data that can be used in research evaluations include the number of papers published, the number of citations from papers, the number of patents obtained, the number of invitations to speak at international conferences, and awards won. According to the Survey on Scientific Research and Evaluation, papers published and citations from papers are ranked among the most important evaluation criteria, and recognition by research groups also scored high ( Figure 3-13 ). Respondents felt that objective data are important to getting evaluation results accepted by researchers and society at large and such data should be used effectively as reference information for evaluations. The survey results also reflect the view that ultimately evaluations should be based on a synthesis of quantitative and qualitative aspects, but that at present, numerical criteria are not necessarily adequate, and their limitations need to be taken into account when such information is used.

Figure 3-13: Researchers' Views on Evaluation Criteria

To support the appropriate implementation of evaluations, and also to ensure that the process does not seriously disrupt the core activities of the researchers who either implement or undergo evaluations, support systems must be developed and improved. In particular, improvements are needed in the management of related organizations, such as examination committees, and in the collection, collation, and analysis of objective data (numbers of papers, citations from papers, information about books, etc.) that can be used as evaluation criteria, of information about international research trends, and of evaluation results. Systems for developing and maintaining databases must also be set up.

The full value of evaluations cannot be realized unless the results are properly utilized. Evaluation findings must be reflected appropriately in areas like: research planning; the allocation of funds, personnel, and other research and development resources; improving the management of research institutes; and improving the research systems that embrace individual research themes.

As mentioned earlier, it is necessary not only to disclose evaluation results and reasons to those who undergo evaluation, but also to take active steps to use various forums and means to disseminate evaluation findings to the wider community so as to assure fairness and suitability and to gain the understanding and support of society. Each year MESSC publicizes general information about grants-in-aid for scientific research, including details of the projects selected, screening policies and time tables, evaluation criteria, and the membership of examination panels. The majority of universities and other institutions that undertake self-monitoring and self-evaluation procedures also publicize the results.