Summary of discussion on how best to evaluate at National Research and Development Agency Council， Ministry of Education， Culture， Sports， Science and Technology
August 22， 2017
National Research and Development Agency Council
The discussion on how best to evaluate the National Research and Development Agencies conducted at the National Research and Development Agency Council of the Ministry of Education， Culture， Sports， Science and Technology is as follows
○ In the case of extremely challenging objectives to achieve （outcomes）， the PDCA cycle can be effectively implemented that the setting of challenging objectives by Agencies， appropriate efforts toward the achievement， and the continuation of motivation trying to make use of a failure for next improvement by adopting the evaluation method that incorporates the evaluation of processes leading to outcomes．
○ For research， etc． requiring a long‐term period， there is room for devising how best to evaluate， such as the setting of evaluation axes according to the time scale．
○ Assuming outcomes to achieve， an evaluation should be made with milestones set for each year， but in case progress of research， etc． is hard to find in its process， caused by the continued influence of specific events， it could be an alternative measure to use the rate of change as an additional indicator， for instance， considering the aspect of activity to enhance research infrastructure．
○ In the event that a new viewpoint such as an emergency response was not originally planned， the outcomes should be evaluated fairly （cases already being done with plural Agencies）．
○ It will be necessary to examine the way of dealing with the relationship between the cause of the event occurring in a year other than the objective year of the evaluation and the original target year for evaluation （e．g．， a special note being written with no consideration in the concerned fiscal year）．
○ Since very good efforts for management‐related evaluation items can be seen individually， the way of setting KPI that will raise the evaluation may as well be considered．
○ As for self‐evaluation， a certain scheme should be devised so that the discussion can be deepened in the Sub‐Committee， by describing points changed from the initial plan， devised points， and points that should have been done differently， aside from the outcomes to be originally evaluated．
○ It is important to take action based on the whole picture， including national strategy， needs of industry， and resources of National Research and Development Agencies and other institutions．
○ The Council should place emphasis on efforts of organization／mission‐type research themes contributing to the country， which could not be dealt with at universities， etc．
○ It would be desirable to have a place to discuss the activation of cooperation among National Research and Development Agencies by setting specific themes such as cross appointment from the viewpoint of strengthening Japan’s international competitiveness．
○ Since researchers tend to be confined to their own research areas， the Council needs to create opportunities for them to contact the surrounding fields （including non‐research items）．
○ Information dissemination has been conducted internationally （exchanging opinions with experts in Japan and overseas） and to the general public as well． Since the continuation of information dissemination is indispensable for promoting understanding， we hope to further enhance information dissemination．
○ Towards improvement of the effectiveness of the evaluation from a viewpoint of “maximization of research achievements，” it is important to evaluate the degree or speed of attainment which is related to the outcome of researches．For the evaluation，attention should be paid to medium to long－term objectives shown by a flowchart（visualized flow from initiatives to outputs，outcomes and impacts）．
○ Each agency should provide a road map showing how to implement medium to long－term objectives／plans，and clarify how its initiatives are advancing．
○ When preparing a flowchart and a roadmap，their contents should be as specific as possible in order to support the validity of the evaluation．Achievements which were not intended in original objectives and not described in a flowchart or a roadmap should also be evaluated justly．
○ When conducting the evaluation，not only achievements but also issues should be presented so as to contribute to solving the issues．
○ Upon evaluation in each sub－committee of National Research and Development Agency Council，basic stances including that toward the judgment standard of evaluation should be clarified by each item and shared by sub－committee members who are involved in the evaluation．
○ When each sub－committee implements the evaluation，it is necessary to note whether excessive differences are generated in judgment of each sub－committee，from the viewpoint of balance of weighting of respective evaluation item（for example，the item of maximization of R&D achievements，that of appropriate and effective business operation，and that of efficiency improvement of business operation）in the entire evaluation of an agency．
○ A view capable of implementing evaluation of the management of an agency including management of R&D and cooperation with other organizations is required．It is favorable to establish a certain directionality by making draft common evaluation items of management．
○ When self－evaluation is made by a national research and development agency or a party to be evaluated makes an explanation，they should accumulate grounds which are as objective and specific as possible，such as indexes established for measuring effects of their efforts as well as results of such efforts．
○ When evaluating achievements applicable to a plurality of evaluation items，basically，each item should be evaluated by a different viewpoint so that the same achievement is not evaluated redundantly．At the same time，cautions are also required not to underestimate the achievements due to distribution and subdivision of each item of achievements．
○ Other than relative evaluations to determine whether achievements of the agency are excellent or not in light of global standard，absolute evaluations based on the standard which can be assumed considering potential level of the agency have been implemented．It is required to clarify that，in light of the relation to outcomes／outputs included in medium to long－term objectives and the characters of business operations，which type of evaluation should be implemented preferentially．Regarding this issue，some views should be investigated and improved by all the evaluators．
○ The ideal way to attain “maximization of research and development achievements” should be explored considering differences of the type of researches，and the idea obtained should be reflected on design or evaluation of medium to long－term objectives／plans．
○ It is necessary for a national research and development agency to take international viewpoints into setting or evaluation of objectives appropriately，aiming at “maximization of research and development achievements．”
○ As far as the maximization of R&D achievements is concerned，each agency should aim at further embodiment and clarification to realize maximization of R&D achievements by taking the characteristics of an agency’s own projects into consideration．Under this condition，the council should deepen the study of the evaluation and the way of establishing objectives for the National Research and Development Agency．
○ Even for R&D with high risk where it is difficult to anticipate a certain level of achievement within a short－term period，an appropriate evaluation for such a given agency should be made in accordance with its own traits．It is advised that such an evaluation should not retard challenging and long－term R&D．
○ Unexpected R&D achievements irrelevant from the original objective may be made． Although it is difficult to evaluate such achievements under the condition of the current objective－oriented evaluation methods，such achievements should be proactively highly evaluated as well．
○ Regarding R&D being conducted collaboratively by multiple agencies，even in case some issues may arise as a whole joint project while an agency itself has been making an achievement，further consideration about an appropriate evaluation method is required．
○ As far as self－evaluation is concerned，the range of self－evaluation should be expanded to issues and their responses relating to an agency’s R&D as well as a result of self－evaluation．The reasons for a rating should be understandable and explicit．In addition，further ingenuity to deepen discussions is required．
○ In order to promote seamless R&D，it is desired that researchers should be motivated to match their objectives and research orientation to a given agency．
○ Whether “a management team” headed by such an agency’s chief director functions effectively through the nurture of each other’s skill or not is important． Such a director is required to manage the whole organization through the utilization of their “management team” without relying on his／her own management capability．
○ An anonymous written interview（upward feedback）targeted at the managers at the National Research and Development Agency was conducted．This written interview was about asking them how they evaluated the chief director’s leadership and management competency．As a result of conducting a feedback against the director，a positive response was also received，whose comment was that it was meaningful．
○ The National Research and Development Agency conducts R&D by utilizing society’s needs and university seeds，therefore it is considered that research seamlessly develops from basic to applied．As a result，a wide range of partnership between the National Research and Development Agency，universities，and companies should be proactively promoted．
○ As far as the National Research and Development Agency is concerned，the proportion of fixed expenses for maintenance management is usually high． Considering this，if subsidies for operating expenses were reduced uniformly，the fact that this may affect R&D projects should be taken into account．
○ Since the accounting standard for the National Research and Development Agency is principally project－achievement－oriented，flexible management should be considered based on the traits of the R&D that the agency conducts．
○ As far as the nation’s policy realization and the improvement of science and technology’s standards are concerned，what kinds of roles the National Research and Development Agency takes on should be sufficiently considered．
科学技術・学術政策局企画評価課