VIII. Review Guidelines

A. General Review Considerations

For a CERC application to be assigned an overall impact score, at least three component projects that extend for the duration of the project must have been judged to have sufficient scientific merit and received impact scores. NCCAM is interested in supporting only the most rigorous research; individual research projects that are relatively lower in merit may not be funded under the umbrella of the project mechanism. Therefore, each project will be assigned a separate impact score, taking into consideration only its merit as an individual research project. Each project must fit and contribute to the theme of the overall project, but these factors are judged separately and have no bearing on a project’s individual impact score. Instead, these considerations will be addressed later with respect to the merit of the overall CERC.

It is expected that individual components, in order to receive funding, should in general receive impact scores similar to those for funded R01 grants. However, a project whose score is somewhat poorer than currently funded R01 grants may contribute significantly to the overall project, whereby synergism with other components and use of core facilities significantly enhance its own value and the value of the other projects. Such considerations would be expected to have an impact on the overall impact score assigned by the reviewers to the project.

Both the applicant and the reviewers should address the contribution of requested cores to each project in both scientific and budget terms.

Questions on the review process for CERCs should be addressed to the Chief of the NCCAM Review Branch.

B. Review Criteria for Individual Research Projects

Reviewers will provide an overall impact score for each project to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research fields involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

1. Reviewers will consider each of the five core review criteria below in the determination of scientific merit and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may still be essential to advance a field.
   
   Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventive interventions that drive this field?
   
   Investigators. Are the project leader, collaborators, and other researchers well suited to the project? If they are early-stage investigators or new investigators or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their fields? If the project is collaborative, do the investigators have complementary and integrated expertise? Are their leadership approaches, governance, and organizational structures appropriate for the project?
   
   Innovation. Does the proposed project challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?
   
   Approach. Are the overall strategy, methodology, and analyses well reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? If the project involves human subjects research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed? Are data adequate in support of feasibility of recruiting sufficient participants from the proposed population to achieve the stated objectives?
   
   Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are institutional support, equipment, and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?
2. Additional Review Criteria. As applicable for the project or core proposed, reviewers will consider the following additional items in the determination of scientific and technical merit but will not give separate scores for these items:
   
   Protections for Human Subjects. For research that involves human subjects but does not involve one of the six categories of research exempt under Title 45, Code of Federal Regulations, Part 46, the committee will evaluate the justification for involvement of human subjects and the proposed protections from research risk relating to their participation according to the following five review criteria: 1) risk to subjects, 2) adequacy of protection against risks, 3) potential benefits to the subjects and others, 4) importance of the knowledge to be gained, and 5) data and safety monitoring for clinical trials.
   
   For research that involves human subjects and meets the criteria for one or more of the six categories of research that are exempt under Title 45, Code of Federal Regulations, Part 46, the committee will evaluate: 1) the justification for the exemption, 2) human subjects involvement and characteristics, and 3) sources of materials. For additional information, please refer to the Human Subjects Protection and Inclusion Guidelines.
   
   Inclusion of Women, Minorities, and Children. When the proposed project or core involves clinical research, the committee will evaluate the proposed plans for inclusion of minorities and members of both genders, as well as the inclusion of children. For additional information, please refer to the Human Subjects Protection and Inclusion Guidelines.
   
   Vertebrate Animals. The reviewers will evaluate the involvement of live vertebrate animals as part of the scientific assessment according to the following five points: 1) proposed use of the animals and species, strains, ages, sex, and numbers to be used, 2) justifications for the use of animals and for the appropriateness of the species and numbers proposed, 3) adequacy of veterinary care, 4) procedures for limiting discomfort, distress, pain, and injury to that which is unavoidable in the conduct of scientifically sound research, including the use of analgesic, anesthetic, and tranquilizing drugs and/or comfortable restraining devices, and 5) methods of euthanasia and reason for selection if not consistent with the AVMA Guidelines on Euthanasia. For additional information, please refer to the Worksheet for Review of the Vertebrate Animal Section.
   
   Biohazards. Reviewers will assess whether materials or procedures proposed are potentially hazardous to research personnel and/or the environment and, if needed, determine whether adequate protection is proposed.
   
   Resubmissions. For resubmissions, the reviewers will evaluate the projects or cores as now presented, taking into consideration the responses to comments from the previous scientific review group and changes made.
   
   Renewals. For Renewals, the committee will consider the progress made in the last funding period. Were the the goals and objectives of the prior project or core achieved? How many papers directly related to the prior project or core were published in high impact journals? How have the results from the prior funding advanced the science related to CAM?
3. Additional Review Considerations. As applicable for the project or core proposed, reviewers will consider each of the following items, but will not give scores for these items, and should not consider them in providing an impact score.
   
   Applications from Foreign Organizations. Not Applicable. Applications from foreign organizations are not permitted..
   
   Select Agent Research. Reviewers will assess the information provided in this section of the application, including 1) the Select Agent(s) to be used in the proposed research, 2) the registration status of all entities where Select Agent(s) will be used, 3) the procedures that will be used to monitor possession use and transfer of Select Agent(s), and 4) plans for appropriate biosafety, biocontainment, and security of the Select Agent(s).
   
   Resource Sharing Plans. Reviewers will comment on whether the following Resource Sharing Plans, or the rationale for not sharing the following types of resources, are reasonable: 1) Data Sharing Plan; 2) Sharing Model Organisms; and 3) Genome Wide Association Studies (GWAS).
   
   Budget and Period of Support. Reviewers will consider whether the budget and the requested period of support are fully justified and reasonable in relation to the proposed research.

C. Review of Research Cores

The review criteria for the research cores are given below (cores receive merit descriptors rather than numeric scores, and individual criterion scores are not provided):

1. Technical merit and justification of the proposed core. Does each scientific core provide critical facilities or services for two or more funded research projects?
2. Qualifications, independent research track record, and commitment of the personnel involved in the core
3. Adequacy of plans for use of the core by the research projects, including criteria for prioritization and the benefit to the projects from core resources or services
4. Appropriateness of the timetable in relation to the scope of the proposed research support
5. If human subjects, vertebrate animals, or biohazards are to be used in the core, the adequacy of the following sections must be assessed and will be considered in determining the merit descriptor of the individual core.

D. Review of Overall CERC

Reviewers will provide an overall impact/priority score to reflect their assessment of the likelihood for the Center as a whole to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the Center proposed).

In assigning the impact score for the application as a whole, although primary emphasis will be placed on scientific merit of the research projects, the assessment of scientific synergy (i.e., the extent to which the potential for scientific impact of the proposed CERC as a whole is deemed likely to be greater than the sum of its component research projects and cores) may contribute significantly to the overall score.

For a P01 to receive an impact score, it must consist of at least three scored individual projects for the duration of the project period. Each core must provide essential functions or services for at least two of these projects.

Specific factors to be evaluated in the consideration of the overall program project are as follows:

1. The potential impact on the field, scientific merit of the program as a whole, and that of individual projects;
2. Scientific synergy (i.e., the extent to which the potential for scientific impact of the proposed CERC as a whole is deemed likely to be greater than the sum of its component research projects and cores, or greater than achievable if each project were supported separately) should contribute significantly to the overall score.;
3. The evaluation of the the leadership and scientific ability of the PD(s)/PI(s), their commitment and their track record of ability to effectively drive and manage a highly productive, multi-disciplinary research team, the presence of a well-defined central research focus, innovation, approach, and environment;

Overall Impact

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the Center to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the Center proposed).

Scored Review Criteria—Overall

Reviewers will consider each of the review criteria below in the determination of scientific merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a Center that by its nature is not innovative may be essential to advance a field.

• Significance
  Does the Center address an important problem or a critical barrier to progress in the field? If the aims of the Center are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?
• Investigator(s)
  Are the PD(s)/PI(s), collaborators, and other researchers well suited to the Center? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD(s)/PI(s), do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project? Do(es) the Center PD(s)/PI(s) have (a) track record(s) of consistently producing highly significant research publications in one or more of the research areas proposed for the Center? Do(es) the Center PD(s)/PI(s) have (a) track record(s) demonstrating the ability to effectively and productively manage a large, interdisciplinary project in the proposed research area(s)? Do the proposed Project and Research Core Leaders have track records demonstrating their ability to independently complete their proposed research projects? Is there a clear, detailed plan for managing the Center’s research and administration, ensuring appropriate prioritization of research, needed course corrections and problem identification and resolution, and effective sharing of resources, that conveys a high likelihood of effective, productive management of the Center as whole?
• Innovation
  Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?
• Approach
  Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the Center? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?
  
  If the Center involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed? Is there adequate data in support of feasibility of recruiting sufficient participants from the proposed population to achieve the stated objectives?
• Environment
  Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements? Is there evidence of substantial institutional commitment on the part of each participating institution? Is there evidence of prior productive collaboration of the research team (including in the P01 application)? Are the application and investigator track records indicative of the establishment of a strong collaborative environment for the program project?

Additional Review Criteria—Overall

As applicable for the Center proposed, reviewers will evaluate the following additional items while determining scientific and technical merit, and in providing an overall impact score, but will not give separate scores for these items.

• Synergy
  In assigning the impact score for the application as a whole, although primary emphasis will be placed on scientific merit of the research projects, the assessment of scientific synergy (i.e., the extent to which the potential for scientific impact of the proposed CERC as a whole is deemed likely to be greater than the sum of its component research projects and cores) may contribute significantly to the overall score.
  
  Each application will receive a merit descriptor (highly synergistic, synergistic, not synergistic) that reflects the degree of synergy of the proposed CERC.
  
  In particular, program synergy will be evaluated on the extent to which the combined approaches to the research question proposed for the CERC are synergistic, and the integration and parallel pursuit of the projects and cores are likely to advance this research area to a greater extent than could be achieved without Center support.
• Protections for Human Subjects
  For research that involves human subjects but does not involve one of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate the justification for involvement of human subjects and the proposed protections from research risk relating to their participation according to the following five review criteria: 1) risk to subjects, 2) adequacy of protection against risks, 3) potential benefits to the subjects and others, 4) importance of the knowledge to be gained, and 5) data and safety monitoring for clinical trials.
  
  For research that involves human subjects and meets the criteria for one or more of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate: 1) the justification for the exemption, 2) human subjects involvement and characteristics, and 3) sources of materials. For additional information on review of the Human Subjects section, please refer to the Human Subjects Protection and Inclusion Guidelines.
• Inclusion of Women, Minorities, and Children
  When the proposed project involves clinical research, the committee will evaluate the proposed plans for inclusion of minorities and members of both genders, as well as the inclusion of children. For additional information on review of the Inclusion section, please refer to the Human Subjects Protection and Inclusion Guidelines.
• Vertebrate Animals
  The committee will evaluate the involvement of live vertebrate animals as part of the scientific assessment according to the following five points: 1) proposed use of the animals, and species, strains, ages, sex, and numbers to be used; 2) justifications for the use of animals and for the appropriateness of the species and numbers proposed; 3) adequacy of veterinary care; 4) procedures for limiting discomfort, distress, pain and injury to that which is unavoidable in the conduct of scientifically sound research including the use of analgesic, anesthetic, and tranquilizing drugs and/or comfortable restraining devices; and 5) methods of euthanasia and reason for selection if not consistent with the AVMA Guidelines on Euthanasia. For additional information on review of the Vertebrate Animals section, please refer to the Worksheet for Review of the Vertebrate Animal Section.
• Biohazards
  Reviewers will assess whether materials or procedures proposed are potentially hazardous to research personnel and/or the environment, and if needed, determine whether adequate protection is proposed.
• Resubmissions
  For Resubmissions, the committee will evaluate the application as now presented, taking into consideration the responses to comments from the previous scientific review group and changes made to the Center.
• Renewals
  For renewals, the committee will consider the progress and achievements specific to Center since the previous review. Were the goals and objectives of the prior grant period achieved? Have the results from the prior funding period advanced the science related to CAM? Are new research goals a logical extension of ongoing work? Is there justification for deleting components? The extent of collaboration will be assessed by publications and resource sharing, including the performance and use of the cores.
• Revisions
  For Revisions, the committee will consider the appropriateness of the proposed expansion of the scope of the Center.

Additional Review Considerations

As applicable for the Center proposed, reviewers will consider each of the following items, but will not give scores for these items, and should not consider them in providing an impact score.

• Applications from Foreign Organizations
  Not Applicable. Applications from foreign organizations are not permitted.
• Select Agent Research
  Reviewers will assess the information provided in this section of the application, including 1) the Select Agent(s) to be used in the proposed research, 2) the registration status of all entities where Select Agent(s) will be used, 3) the procedures that will be used to monitor possession use and transfer of Select Agent(s), and 4) plans for appropriate biosafety, biocontainment, and security of the Select Agent(s).
• Resource Sharing Plans
  Reviewers will comment on whether the following Resource Sharing Plans, or the rationale for not sharing the following types of resources, are reasonable: 1) Data Sharing Plan; 2) Sharing Model Organisms; and 3) Genome Wide Association Studies (GWAS).
• Budget and Period of Support
  Reviewers will consider whether the budget and the requested period of support are fully justified and reasonable in relation to the proposed research.