Research organisation schemes:
Portfolio Evaluation Synthesis

Published: 1 Oct 2005 Evaluation Unit, Foundation? for Research, Science and Technology, 2005.

Scope:

The report synthesises eight portfolio evaluations carried out between 2001 and 2005. The first two, covering seafood research and freshwater management research, were by the nature of pilot studies that established the feasibility of the evaluation approach. Subsequent surveys developed the approach in studies of manufacturing, natural ecosystems, food products (excluding dairy), agricultural production/agritechnology, energy, and building and construction.

The summary synthesises and compares the portfolio evaluation findings, and examines the factors affecting performance of Foundation-funded RS&T.

The report begins with the following:

• The major evaluation questions.
• Conclusions on performance.
• Key recommendations.

The remainder of the report comprises evidence from the consolidated portfolio evaluation findings together with contextual factors and comparisons with the Technology Learning and Knowledge Application Review (completed 1998, see related work below) results.

Summary

The portfolio-based evaluations of Foundation-funded research focussed on RS&T performance. Specifically, the evaluations examined research user’s uptake of research, their relationships with research organisations, their research and development capability, and outcomes and benefits they attribute to the research.

RS&T performance

The consolidated evaluation evidence presented in the report demonstrates that positive and successful outcomes are being achieved from Foundation-funded research by some research users. This was most evident in sectors where there are strong networks, long associations between researchers and research user organisations, and good internal research and development capability in research user organisations.

The portfolio evaluations were aimed at ‘research users’ who could be expected reasonably to have a significant interest in, and commitment to research and development as a vital part of their business. The evaluations found that in each portfolio there were a few research users with significant RS&T performance achievements, which were associated with high levels of awareness of Foundation-funded RS&T, and high levels of engagement with researchers and/or outcomes.

Research users

Research users generally expressed appreciation of Foundation-funded research, and a desire to know more about the research and be more involved with external researchers. Many research users felt frustrated, however, in their ability to keep up-to-date with current research and research organisations, and in their ability to influence RS&T funding priorities and strategic directions.

Segmentation

The evaluations segmented research users, within their sectors, on the basis of both research focus and commercial parameters. This included segmentation for co-funding organisations, recipients of Technology NZ funding, and organisational types (e.g. emerging/high/low growth companies, New Zealand Trade and Enterprise, corporate bodies). The evaluation findings did not show significant differences between the segments of research users, however, but tended to be similar across different segments in terms of variability in awareness, engagement and uptake of Foundation-funded research.

Conclusions

In comparing the consolidated portfolio evaluation findings on outcomes with results from the Foundation’s earlier Technological Learning and Knowledge Application Review (1998), it was apparent that there have been real improvements in RS&T performance in the last decade. The conclusions from the recent evaluations, however, suggest that there are still substantial challenges for the Foundation, for research organisations and for research users in achieving more effective uptake of research and outcomes.

Two challenges stand out. Firstly, for research organisations and the Foundation to venture beyond the existing, well established RS&T networks to draw in potential, innovative research users who are currently unaware or excluded.

Secondly, the whole RS&T sector needs to further the integration of research users as both informed and effective players in Public Good Science and Technology ?(PGS&T) strategies and processes.

It is also important that future actions complement the current structural strengths of Foundation-funded research (e.g. research platforms supported by basic and applied research) and that any actions implemented do not undermine these strengths. Increasing engagement with research users, therefore, must not be at the expense of other research capabilities.

 

Related work

The eight portfolio evaluations are listed as the prior items in this section. Also discussed in the report is an earlier report ‘Technology Learning and Knowledge Application Review’ Foundation for Research, Science, and Technology, April 1998. (Three printed volumes comprising this report and its annexes are held by the evaluation unit. An electronic version is not known to exist).