How to Prepare an Effective SAES-422

The 1993 Federal Government Performance and Results Act[2] (GPRA) significantly changed the expectations for Executive Branch agencies. Henceforth all agencies, including those funding scientific research activities, were required to develop performance-based plans that were to be reported against annually. The anticipation is that budget decisions in the future will be related to demonstrated performance by the managing agencies, based on their reported results.

Missing was any organized plan for gathering information generated external to the agency regarding performance and results. This was notably true for scientific research activities which are notoriously difficult to predict as outputs and outcomes of research trials yet-to-be conducted.

The GPRA requirements were imposed across all agencies ranging from the Internal Revenue Service, and the Department of Defense, to the science-based agencies such as the National Science Foundation and the National Institutes of Health. However, results for the U.S. Army or the Social Security Administration have little resemblance to the results of research. The ramp up period to full GPRA reporting is now over, and the pressure is on to provide what is now seen as competitive information on why programs should be funded.

The Agricultural Research, Extension and Education Reform Act of 1998 (a.k.a. the '98 Farm Bill) requires all federal formula funded activities at public institutions to be organized in state-based Plans of Work. The expectation from Congress, it is presumed, was to provide the funding agency [i.e., the Cooperative State Research, Education, and Extension Service (CSREES) of the USDA] with reliable information that could be assembled into performance reports for GPRA compliance.

Federal formula funding is viewed as an essential component of the Federal-State Partnership in agricultural research and extension. However, formula funding is under attack as not being accountable enough, or has been assumed to be an institutional entitlement. Some critics of formula funding have proposed that these funds be converted to competitive grants

Formula funds are used for, inter alia, maintaining facilities, paying faculty salaries, hiring technicians, feeding research animals, buying fertilizers for crop trials, and supporting the next generation of scientists. Matching non-federal funding is required in equal amounts, and they are often overmatched in ratios of five- or even ten to one. SAES directors are very interested in assuring proper accounting for the use of formula funds, and they are in fact looking for ways to increase federal formula allocations. Loss of formula funding would be devastating to agricultural research as we know it.

An additional '98 Farm Bill requirement for formula funding is "not less than 25% must be used for multistate research projects, and twice the amount of the 1997 base (or not less than 25% of the formula funds, which ever is less) must be used for integrated activities with extension". The justification for these requirements rests with the perspective that federal tax dollars should not be used for studying local or state-specific problems or issues. The use of federal funds should be invested more towards regional and national needs and opportunities, and building stronger links between research and extension activities.

To address this requirement to account for federal funding investments in agricultural research new reporting methods were developed for Multistate Research Projects. One of these changes was the design of an SAES-422 format (see Appendix 1) that asks for annual reports of research outcomes and impacts for Multistate Research projects.

Based on the review of the 2001 SAES-422 submissions there is a pressing need to improve the content and quality of the submissions. This publication was developed to provide guidance for Multistate Research technical committees. We have included an example of what we think is an exemplary SAES-422 (see Appendix 2).

Traditionally, scientific research has been pretty much left alone to pursue discoveries in search of knowledge that may or may not benefit the nation’s economy, society, health, or the environment. Consider the 1945 publication entitled "Science, The Endless Frontier" by Vannevar Bush[3]. This policy document was in force for nearly 50 years, and in essence asked for major federal investments in post-WWII science with the general pledge that such investments would yield huge payoffs. Indeed such payoffs did occur, but the documentation on the extent of the payoffs is scanty. For most of the second half of the 20^th Century the federal government invested heavily in scientific research. In return health improvements are legendary. Food supplies have been vastly increased and food costs have dropped to less than 9% of the average American's disposable income. The environment has been greatly improved. All of these results and much more, have been a direct result of public and private investments in scientific research.

Agricultural research programs funded by the federal government must be responsive with information that justifies their continuation. This means that each project or committee should do its best to report results in a way that information can be compiled into more comprehensive reports for decision makers. To facilitate this responsibility, and hopefully better support the preservation of this important source of research funds, the regional associations of State Agricultural Experiment Station directors adopted a set of guidelines that were designed to implement the intent of the changes in the 1998 Farm Bill and shift the reporting instruments into an outcomes-oriented system. Among those changes was the requirement to submit an annual SAES-422 accomplishment report.

The first round of SAES-422 reports focused mostly on activities (e.g., a meeting attended, a common set of plants grown, or an animal fed for so many days). Unfortunately, none of these measures tells anyone about the outputs of the research, the accomplishments of the project, or the benefits of the application of research knowledge.

To address the need for clear communication we will use the terminology of the Consultative Group on International Agricultural Research (CGIAR) that has been adapted with some changes to align the terms with current American English and to accommodate terminology in use at the USDA.

Intermediate Goals: Benefits directly resulting from the uptake of an innovation that includes research outputs from the Federal-State Partnership (see Appendix 3 for the current intermediate national research goals).

Themes: Topics of information organized to describe progress in addressing one or more goals (or intermediate goals). The utility of themes (versus a programmatic organization) is said to be in the versatility of themes, which can be flexed to respond to shifting priorities as administrations change in Washington, D.C. The application of thematic organization is commonly done through text searches for key words (e.g., food safety).

Outcomes: Quantitative, measurable benefits of the research outputs as experienced by those who receive them. Examples include the adoption of a technology, the creation of jobs, reduced cost to the consumer, less pesticide exposure to farmers, access to more nutritious food, a cleaner environment, or healthier communities.

Indicators: Qualitative surrogate observations or indirect measures of quantitative performance measures which permit monitoring the achievement of outcomes when direct measurement of performance is difficult, too costly, or not possible. An indicator of cultivar adoption might be seed certification records, rather than actual land area planted to that cultivar.

Outputs: Defined products (tangible or intangible) that are delivered by a research project. Examples of outputs are reports, data, information, observations, publications, and patents.

Milestones: Key intermediate targets necessary for achieving and/or delivering the outputs of a project, within an agreed timeframe. Milestones are useful for managing complex projects. For example, a milestone for a biotechnology project might be "To reduce our genetic transformation procedures to practice by December 2004."

Activities: Organized and specific functions or duties carried out by individuals or teams using scientific methods to reveal new knowledge and develop new understanding.

Objectives: A specific target (or set of targets) against which a set of research activities are planned.

Inputs: Human, financial and intellectual capital made available for an activity. Allocations of inputs may be made as a competitive grant, institutional project support, access to technicians, etc. each with a monetary value, at least measured indirectly.

If only the world were as neat as this typology. But it is not. Many research results find utility in some far off applications. Some research discoveries must be combined with other findings to make a difference. Often research success comes over many years, and may be clustered with other scientific advances. Many times a research discovery must await other "market" factors before being applicable. Frequently, the associated measures of research impact are not gathered or are not available. So who gets credit? Who does the study?

What is needed is a system of providing to CSREES information suitable for summary statements for reports to policy and budget decision makers. The following section sets out how that task can be made easy for both the researcher and the agency analyst.

A particularly troublesome point for most projects is the impossibility of having over-the-wall home runs to report every year. This point is well accepted and the agreement with the Federal-State Partnership is not every project will have impacts to report every year, but at least once in the life of a project some measurable progress will be reported. When research progress or accomplishments were reported previously, reference to that report will be acceptable. Or, when the work is not yet completed it will be acceptable to mention that a report will be forthcoming. What is to be avoided is a five-year project ending with nothing ever reported.

Research project proposals should be organized to facilitate evaluation including reporting of outputs and outcomes. If a project has as an objective to reduce the use of a pesticide, how might that reduction be measured? If another project is intending to improve child nutrition, what statistics are available as benchmarks, and how might the intended change be documented?

Selection of research objectives will need to become more closely tied to meaningful measures in the future. Some measures are relatively simple, such as calculating internal rates of return on a research investment. If a project cost $500,000 to accomplish and provided the target audience with $10 million in benefits the following year, the annual rate of return on the investment is easily calculated.

In another example, the cost of a project versus the benefits in terms of gains for the intended users can be expressed as a ratio (the cost: benefit ratio). Another measure might be the proportion of an area affected, or the percentage of individuals adopting a specific technology.

In many cases direct measurement of an outcome is not possible, or it would be dubious. In these instances indicators might be used to express the outcomes derived from research investments. In other cases testimonials from beneficiaries may be an adequate substitute (e.g., a quote from a pleased farmer). In fact, many newspaper reporters use anecdotal information to prove a point. Politicians particularly use case examples to make their points. Long eschewed by science, case studies may in fact be necessary method for defending some types of research accomplishments. This may be especially true for those project outcomes that cannot be easily measured, such as environmental quality improvements.

Plausible associations are also useful in establishing research accomplishments. For example, it is plausible to claim that reduced food costs are the result of higher yields and other factors. In particular, cheap potatoes are the result of high yielding cultivars and better cultivation technologies. Proving that claim might be difficult, but claiming it to be true seems plausible.

Sharing credit for research outcomes is problematic in many cases. This is especially true when the work of many contribute to a success story. A common example of this happens when several institutions work with the private sector to complete an activity. Who gets the credit? In all fairness, everyone should be able to claim the success.

Ex ante and ex post impact studies can be helpful in documenting research outcomes. But in many cases the expense of doing such an analysis may not be worthwhile. In these instances substitutes should be sought.

Milestones are particularly helpful in reporting outputs and outcomes. If a research project is cleverly designed with well thought out milestones, reporting progress and achievements can be greatly facilitated. Thus, time invested in project design pays off in the reporting requirements. This thought motivated the recent changes to the suggested multistate project outline, which is now outcomes oriented with a milestones structure.

CONCLUSION

Much will soon be developed as improvements to the way we report our achievements in agricultural research. And these changes are necessary if we are to sustain (indeed increase) our funding for agricultural research. Our staunch advocates want to double the funding for agricultural research, as has been done for the National Institutes of Health and the National Science Foundation in recent years. We need to give our advocates within and outside the Federal government the ammunition they need to make a case for increased funding. To do this, research scientists need to provide clear, concise and up-to-date information on research outputs and outcomes. Equally important is the need to provide clear statements about what it all means, especially for the intended beneficiaries. The SAES-422 is the recommended way to fill that need.

APPENDIX 1

National Multistate Research Guidelines

Appendix D

SAES-422

Format for

Multistate Research Activity

Accomplishments Report

Note: This report is submitted each year of an activity's duration and is due 60 calendar days following the annual meeting. The SAES-422 is forwarded electronically by AAs to their Executive Director. Annual Reports for MRF projects are then forwarded to CRIS by the Executive Director.

Project/Activity Number:

Project Title:

Period Covered:

Date of This Report:

Annual Meeting Dates:

Participants: Provide a list of those who attended, and their employing institution. As an alternative, list the URL for the meeting minutes, if that report contains the list of those who were present. If available, add the address for your list server as well.

Brief summary of minutes of annual meeting: Provide information with a focus on the decisions made. As an alternative, list here the URL for your meeting minutes.

Accomplishments and Impacts: In this section focus on intended outcomes and potential impacts. This information should be built around the activity's milestones, as they were identified in the original outline. The report should also reflect on the items that stakeholders want to know, or want to see. Also, describe your plans for the coming year in no more than one or two short paragraphs.

Publications: List the publications for current year only (with the authors, title, journal series, etc.).

Authorization: Electronic signature of the Administrative Advisor, with the date.

Agricultural Experiment Station Directors

Multi-State Research Project

Project Title: Development of New Potato Clones for Environmental and Economical Sustainability in the Northeast

Participants: Bill Brodie (NY), Barb Christ (PA), Walter De Jong (NY), Don Halseth (NY), Mel Henninger (NJ), Chad Hutchinson (FL), Matt Kleinhenz (OH), Dave Lambert (ME), Bill Lamont (PA), David MacKenzie (MD, NERA), Keith Perry (NY), Robert Plaisted (NY), Greg Porter (ME), Joe Sieczka (NY), Rikki Sterrett (VA), George Tai (NB, AAFC), Lesley Wanner (BARC, USDA), Marion White (FL), Craig Yencho (NC).

NE184 rewrite. Greg Porter presented version 2.0 of the proposal. It has been examined twice by the rewrite committee. The length of the document needs to be reduced to fit within national guidelines. Detailed discussion of the document was tabled until the next morning, when all present will have had a chance to read it. The revised proposal will be reviewed by five scientists outside the NE region. After their comments are incorporated, the proposal will go to the NERA Multistate Advisory Committee for a recommendation and then to the NERA Directors for a final vote.

State, Federal and Provincial Site Reports for 2001. A discussion arose concerning continued Canadian participation. It was noted that NE184 seed meets Canadian Federal standards but provincial rules in NB or PEI prevent us from sending seed from a farm with the ring rot pathogen present. State reports updated the meeting participants on clone performance and weather conditions during trials.

Comments from Industry Representatives. No industry representatives were present. C. Yencho suggested that we should invite industry representatives to one of our meetings every few years, and that we have a session where we listen to them (rather than have them listen to us). D. Halseth suggested that we could also invite them to observe some of our trials.

Results using cluster and AMMI analysis (G. Tai). AMMI and cluster analyses results were distributed. AMMI provides a simple, easy way to interpret and summarize multistate trial data. D. Halseth and a graduate student assembled the 1998-2001 data for analysis, which was distributed as a summary report.

Seed Orders. G. Porter distributed a ‘shopping list” of advanced clones. He noted that there were some seed quality problems this past year, and expressed hope to do better this coming year. They are now performing extra testing to ensure freedom from viruses and ring rot. Breeders' seed constitutes an increasing percentage of the original propagules multiplied by NE-184 each year.

Old Business. B0564-8 and AF1753-16 were nominated to represent NE184 in the 2002 national trial. G. Porter reminded the group that last year we had decided to place all our reports up on a web site. A committee of C. Hutchinson, D. Halseth, B. Christ, D. Lambert formed to investigate development of a NE184 database and web site, and will report back to the technical committee next year. They will evaluate if we want/need a common data format for future field-testing performance reporting.

Site committee selection report. Next years meeting is to be held in North Carolina, possibly Charlotte, Raleigh or Myrtle Beach, South Carolina. Date - January 23 and 24, 2003.

Nominations committee report. Next year's Executive Committee are -- Matt Kleinhenz, Chair; Walter De Jong, Vice-chair; and Craig Yencho, Secretary.

"Whereas Dr Alvin F. Reeves, potato breeder and long-time NE-184 participant, passed away in May of 2001, let us observe a moment of silence in his memory and gratefully recognize his many NE-184 accomplishments and the years of friendship that we shared with him."

The NE-184 project was formed in the mid-1970s to respond to the trend to ever increasing proportions of potatoes being industrially processed for consumption. One goal was to replace pesticide dependency with naturally resistant selections carrying improved industrial-processing characteristics. Another goal was to increase the quality attributes of potatoes grown in the region to enhance their suitability for processing markets and fresh consumption. A third goal was to develop replacement potato cultivars that were less costly to grow and better suited to processing and regional fresh market niches.

2001 Outputs: Advanced clone testing: Nineteen advanced breeding lines were evaluated against 10 NE-184 "standards" in replicated field trials in 2001. As expected, performance varied by location. But several selections outperformed standard lines at several locations. Measures of performance included yield, yield of US#1 tubers, chipping color after storage, cooking characteristics, and specific gravity (an indicator of dry matter content).

Germplasm trials: Several sites tested breeding materials of varying stages of development against named cultivars. Performance measures included total yield and size distributions, specific gravity and cooking characteristics. Additionally disease resistance and physiological defects were recorded.

Project Impacts: As a result of this project’s efforts many replacement potato cultivars have been adopted region-wide by growers. As a consequence of these cultivar replacements pesticide dependency has been reduced significantly in the region. Although reliable numbers on pesticide sales are not available from pesticide companies, testimonials by East coast potato growers attest to the reduced need to spray crops for diseases and insect pests on the replacement cultivars. Moreover, the substitution of golden nematode resistant potato cultivars in New York is said to have saved the state’s industry from what would have been otherwise its complete foreclosure.

One of the greatest successes of the project is the ability to predict on-farm cultivar performance, based on field test results obtained by the project from plots planted from maritime

Canada

to Florida. These tests have allowed for the selection of cultivars that are either broadly adapted, or adapted to specific environment. As a result, the NE-184 developed cultivars are known to perform well in varied soil and climatic conditions. Moreover, potato grower participation in the on-farm demonstration of these selections has let to their rapid adoption, and the consequent impacts. Further, the project is currently testing and validating new statistical models that enable scientists to predict a potato selection’s performance, allowing for more efficient breeding and variety development schemes in the future.

A study by the International Potato Center (Lima,

Peru

) on the economic returns for NE-184 investments has shown an annual rate of return from the NE-184 activities to exceed 40%, for a total net farm value that exceeds $14 million. This rate of return and the resulting pay-off competes favorably with some of the best internal rates of return obtained from agricultural research and extension investments.

PUBLICATIONS: G. Porter is preparing a summary document to be put online. D. Halseth is organizing a project to put all other project reports online.

Experiment Station Section

The following five research areas (in bold) were selected as the highest priority areas for future research activities by the SAES/ARD Directors at their Workshop in New Orleans, September 26-28, 2000. The statement below each area (in italics) represents the primary outcome-oriented goal desired from research in that area. The bullet list[5] under each area indicates areas for more specific research activities of critical national need within each high priority area.

Biobased products will be central to sustaining the economy and the environment.

[1] Executive Directors, Northeastern, Southern, North Central, and Western Regional Associations of State Agricultural Experiment Station Directors, respectively. For more information contact the first author at dm184@umail.umd.edu

[2] http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html

[3] http://www.nsf.gov/od/lpa/nsf50/vbush1945.htm

[4] The USDA’s five goals are: An agricultural system that is highly competitive in the global economy; A safe and secure food and fiber system; A healthy, well nourished population; An agricultural system which enhances natural resources and the environment; and Enhanced economic opportunity and quality of life for Americans. The Federal-State Partnership in agricultural research has adopted the 5 goals of the USDA

[5] Bullet lists are not in priority order.

Preparing an Effective

By

CONCLUSION

Agricultural Experiment Station Directors

Multi-State Research Project

Experiment Station Section