Main Comments to be Reviewed in National Research and Development Agency Council of Ministry of Education，Culture，Sports，Science and Technology（MEXT）
August 24，2016
In addition to those described in the evaluation report of each national research and development agency，main comments which are common in evaluation of the agencies are shown below． Comments in FY2015 are shown in the attachment．
1．Implementation of PDCA utilizing flowchart，etc．
○ Towards improvement of the effectiveness of the evaluation from a viewpoint of “maximization of research achievements，” it is important to evaluate the degree or speed of attainment which is related to the outcome of researches．For the evaluation，attention should be paid to medium to long－term objectives shown by a flowchart（visualized flow from initiatives to outputs，outcomes and impacts）．
○ Each agency should provide a road map showing how to implement medium to long－term objectives／plans，and clarify how its initiatives are advancing．
○ When preparing a flowchart and a roadmap，their contents should be as specific as possible in order to support the validity of the evaluation．Achievements which were not intended in original objectives and not described in a flowchart or a roadmap should also be evaluated justly．
○ When conducting the evaluation，not only achievements but also issues should be presented so as to contribute to solving the issues．
2．Upgrading and improvement of evaluation method
○ Upon evaluation in each sub－committee of National Research and Development Agency Council，basic stances including that toward the judgment standard of evaluation should be clarified by each item and shared by sub－committee members who are involved in the evaluation．
○ When each sub－committee implements the evaluation，it is necessary to note whether excessive differences are generated in judgment of each sub－committee，from the viewpoint of balance of weighting of respective evaluation item（for example，the item of maximization of R&D achievements，that of appropriate and effective business operation，and that of efficiency improvement of business operation）in the entire evaluation of an agency．
○ A view capable of implementing evaluation of the management of an agency including management of R&D and cooperation with other organizations is required．It is favorable to establish a certain directionality by making draft common evaluation items of management．
○ When self－evaluation is made by a national research and development agency or a party to be evaluated makes an explanation，they should accumulate grounds which are as objective and specific as possible，such as indexes established for measuring effects of their efforts as well as results of such efforts．
○ When evaluating achievements applicable to a plurality of evaluation items，basically，each item should be evaluated by a different viewpoint so that the same achievement is not evaluated redundantly．At the same time，cautions are also required not to underestimate the achievements due to distribution and subdivision of each item of achievements．
○ Other than relative evaluations to determine whether achievements of the agency are excellent or not in light of global standard，absolute evaluations based on the standard which can be assumed considering potential level of the agency have been implemented．It is required to clarify that，in light of the relation to outcomes／outputs included in medium to long－term objectives and the characters of business operations，which type of evaluation should be implemented preferentially．Regarding this issue，some views should be investigated and improved by all the evaluators．
3．Maximization of research and development achievements
○ The ideal way to attain “maximization of research and development achievements” should be explored considering differences of the type of researches，and the idea obtained should be reflected on design or evaluation of medium to long－term objectives／plans．
○ It is necessary for a national research and development agency to take international viewpoints into setting or evaluation of objectives appropriately，aiming at “maximization of research and development achievements．”
National Research and Development Agency Council of Ministry of Education，Culture，Sports，Science and Technology（MEXT）Main comments to be reviewed
August 21，2015
In addition to the comments described in the evaluation form of each agency，the main common points to be reviewed for the National Research and Development Agency are as follows．
1．Regarding the evaluation of the National Research and Development Agency
○ As far as the maximization of R&D achievements is concerned，each agency should aim at further embodiment and clarification to realize maximization of R&D achievements by taking the characteristics of an agency’s own projects into consideration．Under this condition，the council should deepen the study of the evaluation and the way of establishing objectives for the National Research and Development Agency．
○ Even for R&D with high risk where it is difficult to anticipate a certain level of achievement within a short－term period，an appropriate evaluation for such a given agency should be made in accordance with its own traits．It is advised that such an evaluation should not retard challenging and long－term R&D．
○ Unexpected R&D achievements irrelevant from the original objective may be made． Although it is difficult to evaluate such achievements under the condition of the current objective－oriented evaluation methods，such achievements should be proactively highly evaluated as well．
○ Regarding R&D being conducted collaboratively by multiple agencies，even in case some issues may arise as a whole joint project while an agency itself has been making an achievement，further consideration about an appropriate evaluation method is required．
○ As far as self－evaluation is concerned，the range of self－evaluation should be expanded to issues and their responses relating to an agency’s R&D as well as a result of self－evaluation．The reasons for a rating should be understandable and explicit．In addition，further ingenuity to deepen discussions is required．
○ In order to promote seamless R&D，it is desired that researchers should be motivated to match their objectives and research orientation to a given agency．
○ Whether “a management team” headed by such an agency’s chief director functions effectively through the nurture of each other’s skill or not is important． Such a director is required to manage the whole organization through the utilization of their “management team” without relying on his／her own management capability．
○ An anonymous written interview（upward feedback）targeted at the managers at the National Research and Development Agency was conducted．This written interview was about asking them how they evaluated the chief director’s leadership and management competency．As a result of conducting a feedback against the director，a positive response was also received，whose comment was that it was meaningful．
2．Regarding the management of the National Research and Development Agency system toward the promotion of R&D
○ The National Research and Development Agency conducts R&D by utilizing society’s needs and university seeds，therefore it is considered that research seamlessly develops from basic to applied．As a result，a wide range of partnership between the National Research and Development Agency，universities，and companies should be proactively promoted．
○ As far as the National Research and Development Agency is concerned，the proportion of fixed expenses for maintenance management is usually high． Considering this，if subsidies for operating expenses were reduced uniformly，the fact that this may affect R&D projects should be taken into account．
○ Since the accounting standard for the National Research and Development Agency is principally project－achievement－oriented，flexible management should be considered based on the traits of the R&D that the agency conducts．
○ As far as the nation’s policy realization and the improvement of science and technology’s standards are concerned，what kinds of roles the National Research and Development Agency takes on should be sufficiently considered．
科学技術・学術政策局企画評価課