Investment processes:
Stakeholder Survey 2003

Andrew Fletcher Consulting Ltd.

Scope:

This survey was the first annual repeat of the benchmark survey (2002). The survey  and aimed to:

• inform the Foundation? of the external stakeholder assessment of its performance and areas for improvement
• gauge how well the Foundation processes the impact on provider ability to deliver outcomes, and assess changes in processes and their impact on providers over time
• continue to enhance the Foundation's accountability reporting, particularly the Statement of Service Performance to the Minister and Parliament.

The survey comprised 27 individual interviews with key stakeholders, and Internet surveys of 208 stakeholders and 58 Technology NZ stakeholders (68% response rate).

Summary

Involvement with the Foundation

Funding contracts

72% of respondents had a current contract with the Foundation compared to 80% in the 2002 survey, which was partly due to changes in respondent profile since 2002.

The proportion of respondents having a current contract was much higher for the Foundation survey (82%) than the Technology NZ survey (34%).

There had been more applications made for funding than in the 2002 survey, and the proportion that had made only one application had dropped from 60 % in 2002 to 42%. This change applied across all roles and provider types.

The proportion of respondents who said that they spent between 1 and 5 days on their largest application had fallen from 38% (2002) to 15%, indicating that respondents were spending longer preparing applications due to larger, more complex bids.

In 2002, 61% described their applications as successful compared to 36% in this survey due to the fact that in 2003 respondents made more applications for funds and for greater amounts than before.

Impact of Foundation funding

Half of all respondents described the impact of Foundation funding on their organisation as positive, but a quarter said it had been negative. Research Associations (60%), CRIs (42%) and those whose applications had been unsuccessful (45%) were more likely to say the impact had been negative. Universities (66%) and private sector businesses (58%) were more likely to describe the impact as positive.

Overall performance

The difference in ratings between 2002 and 2003 was not significant and rated at 5.2 (scale 1-9), a little above the mid-point. Interviewees regarded the Foundation as doing a hard job under sometimes difficult circumstances.

Organisations whose funding applications were successful rated the Foundation’s performance higher (5.8) than those whose applications were unsuccessful or partially successful (4.6). This was also true for organisations for whom Foundation funding had had a positive impact (5.6), than for whom the impact had been negative (3.8).

Relationships

The Foundation’s relationships with stakeholders was seen positively (5.5) although this is lower than the 5.8 response in 2002.

The Foundation’s perceived relationship with users was rated below average (4.6) which is the same level as 2002, and its’ perceived relationship with research providers is rated above mid-point (5.4) compared to 5.3 in 2002.

A smaller number of respondents reported non-funding dealings with the Foundation (39%) compared to 2002 (51%). Participants rated the Foundation below the mid-point for the effectiveness of its non-funding activities, however there was a strong desire across all groups for greater involvement in non-funding activities which was in line with the findings in 2002.

Foundation strategy

The Foundation’s strategy was no clearer in the survey than it was in 2002, nor did stakeholders perceive it to be well aligned to their organisation’s objectives. Respondents rated the clarity of the Foundation’s strategy very poorly (3.1-3.7).

PGST process

Four out of five respondents said that they deal with the Foundation’s Public Good Science and Technology ?(PGST) funding, although understanding of the process had not improved since 2002.

In particular, perceptions of the effectiveness of the use of portfolios to define funding areas and setting priorities within research funding areas had declined slightly since 2002.

Overall quality of the Foundation’s decision process is rated 4.5 compared to 4.8 in 2002.

The Foundation rated well for most aspects of communicating its funding steps, except for feedback on funding decisions which rated 4.2 compared to 4.7 in 2002. There was relatively high and widespread dissatisfaction with this aspect of the Foundation’s communications.

Technology New Zealand

40% of those taking part in the survey had dealings with Technology NZ which rated very highly for dealing with responses to applications in a timely manner and helping to achieve technical success within specific projects. All other attributes rated above the mid-point except for its integration with other government agencies which rated below average.

Maori advancement

Maori advancement is not seen as being effectively applied, the exception being PGST recognition of Maori responsiveness in its decision process.

Communication

Technology NZ companies rated the effectiveness of Foundation communications low. This applied particularly to IP policy and the Research Consortia? Initiative. Respondents to the Foundation survey rated the Foundation lower for all areas of communication compared to 2002. Respondents to the Technology NZ Survey rated the Foundation as “OK” for communication of RS&T achievements in New Zealand, but rated all other communication areas lower than in 2002.

All vehicles for communication rated as reasonably valuable with personal contacts and site visits rated particularly valuable.

Impact on stakeholders adversely affected by Foundation funding decisions

The stakeholders who had been adversely affected by Foundation funding decisions said it would impact them in various ways:

• impact on organisation – loss of productive time, inability to do research at desired level, loss of corporate memory, negative impact may be offset by funding received from other organisations
• impact on research – loss of science capability within organisation and nationally
• impact on staff – low morale, job losses, discouragement of young researchers.

Conclusions

The 2002 and 2003 stakeholder surveys were notable for the absence of change in the ratings of the Foundation’s performance (especially when allowing for sampling error). This probably reflects the shortness of time between the two surveys, and, also, that changes in perception and understanding among the Foundation’s stakeholders is likely to take time, even with the Foundation’s best efforts. Many of the issues were reinforced in other evaluations.

Significant changes in the way the Foundation operates are now being introduced, both internally (re-structuring) and externally (e.g. a wider choice of RS&T investment instruments). The Foundation will continue to assess the extent to which these and recent changes address the issues reported by stakeholders.

Related work

Stakeholder Survey, 2002, 2006, Andrew Fletcher Consulting Ltd.