Chapter 5 - Keeping track of performance and effectiveness

“Ko r-atou e rapu ana te hunga ka kite”

The ministry’s approach to performance measurement is to monitor both overall outcomes in terms of indicators of sector progress and  immediate results or impacts of specific interventions.

Cost-effectiveness and value for money

The ministry is continuing to invest in the development of information sets, capability and tools to enable it to undertake more sophisticated analysis of cost-effectiveness and value for money. The objective of work on cost-effectiveness and value for money is to achieve better outcomes for any given level of expenditure.

An analysis of cost-effectiveness in education seeks to make comparisons such as:

• changes in student outcomes over a specified period with expenditure to achieve those outcomes
• the relative impact of different interventions to achieve comparable outcomes at a specified point in time.

Cost-effectiveness analysis in education is a problematic process for a number of reasons. In 2005 Treasury noted that
“measuring the performance and productivity of schools is a difficult and complex task, which has been the subject of much research and debate over the last 50 years particularly in the USA. By way of illustration, a meta-analysis of all published estimates of the production function for education up to 1995 for the USA covers 89 individual publications containing 376 production function estimates.*

* UNESCO (2005) EFA Global Monitoring Report 2005, cited in New Zealand Treasury (2005) Value for Money and Productivity in the Secondary School Sector: a Case Study.

Some of the issues that need to be addressed include:

• clarity of the outcomes sought
• the ability to attribute causality
• understanding of lags in terms of realisation of broader benefits
• the adequacy of information, including consistency of information sets over time.

To overcome these issues the ministry is investing in developing the information sets and capability that will enable us to undertake both evaluations of the effectiveness of educational interventions and studies of cost-effectiveness. This work includes:

Defining outcomes

Progress is being made to develop and agree clear goals and outcomes for the education system which provide the basis for cost-effectiveness assessment. Progress has included outcomes established through:

• development of the Tertiary Education Strategy 2007 – 2012
• the Pasifika Education Plan 2006 – 2010
• this Statement of Intent 2007 – 2012
• continued work on the Early Childhood Strategic Plan 2002 – 2012
• Better Outcomes for Children: an Action Plan for GSE 2006 – 2011.

Aligning inputs to outputs

Work to redevelop the ministry’s output plan has progressed well. This Statement of Intent contains a redeveloped departmental output structure. The new output classes reflect the ministry’s leadership role in the sector and the delivery of services to its key stakeholders: students; teachers; providers; families and communities. The structure includes the activities that deliver each output. These activities can now be used as a common base for planning and budgeting. The planning process links each activity to the ministry’s outcomes and priorities; budgets are then developed for each activity. This creates a clear link between outcomes and appropriations, allowing financial data to be better matched with non-financial data in order to prioritise work programmes and assess their value for money.

Continuing to develop the evidence and indicator base

Considerable progress has been made by the ministry in recent years to redevelop a robust and extensive evidence and indicator base.

Access to the most up-to-date information, research and analysis on education outcomes is available through the Education Counts website. Education Counts looks to manage and simplify access to the increasing volume and complexity of information becoming available about education statistics and research. It focuses on operational and policy issues of strategic importance, anticipating and responding to information needs, providing information, analysis and expert advice. (Go to: www.educationcounts.edcentre.govt.nz) For particular areas (including ECE and Pasifika education) there are published sector targets and routine monitoring against them.

In addition, there has been considerable progress in areas of data development and its use in the ministry, the sector and for the public, including initiatives to:

• develop basic data within the system – including ENROL (which was implemented in secondary and  intermediate schools in 2006/07 and will be implemented in primary schools during 2007/08) – and in particular to collect data, at the individual school student level, that will enable sophisticated longitudinal analysis of outcomes
• pull together different datasets, including the tertiary integrated dataset,  and use these to examine the links between education and the labour market
• present performance information to different audiences – including through Education Counts website (which has a large amount of performance-related information), a new State of Education annual report on performance of the system, and the inclusion of student achievement data on the Team-Up website.

Ongoing data and research developments include:

• further development and dissemination of Best Evidence Syntheses (BES),  which seek to provide consolidated best practice information on what makes a difference to student outcomes in teaching and learning and the relative effect size of different approaches. The BES programme represents a significant and important change programme that has had a considerable level of sector engagement and is highly respected internationally as a leading-edge initiative. During 2007/08 there will be a focus on dissemination and use of results based on the new BES analysis that has been produced in 2006/07
• collection of quantitative data on impacts of particular interventions, for example, impacts of the numeracy strategy.

Evaluation Strategy

The focus of the Evaluation Strategy is on improving the quality of evaluation undertaken by the ministry and at the same time improving organisational capability to use evaluation effectively in decision-making.

The Strategy’s key actions include providing business planning guidance, establishing an evaluation register and standards, setting up a governance and management body for evaluation across the organisation, providing professional development and training opportunities and developing ministry-specific evaluation resources.

Progress on key actions
Work on an evaluation register has progressed and will provide a platform for the coordination and alignment of the ministry’s evaluation programme. It will support better understanding of the results of evaluations and the impact evaluation is having on policy, operations and programme delivery.

The development of evaluation standards for the ministry is well underway. They will provide a clear model of what constitutes ‘good evaluation’ practice within the ministry. They will form the building blocks for shaping the ministry’s future evaluation practice and use of evaluative information.

Professional workshops with managers and staff are continuing. They cover: an introduction to evaluation, its uses and relationship to other forms of evidence and information; using performance measurement as an effective evaluative tool; needs assessment; an introduction to intervention logic and its uses; what good evaluation questions look like; and evaluating complex programmes/interventions. In 2007/08 work will begin on developing a set of ‘core skills’ modules on evaluation use and practice that will link to the organisation’s broader capability development strategy.

Analytical tools

Over 2006/07 a value-for-money analytical tool has been developed and used to help capture and cross-check assumptions about how new initiatives will work and capture some comparisons in terms of breadth and depth of impact and relative cost.

Measures of progress

The ministry’s approach to performance measurement is to monitor both overall outcomes in terms of indicators of sector progress and immediate results or impacts of specific interventions.

The table below provides a summary of some of the outcome indicators being monitored. They highlight presence, engagement and achievement across the sectors, and for some indicators show monitoring against international measures of achievement.

In each of the measurement areas we break down information as much as possible so we can monitor the progress of a diverse range of learners, including Māori students and Pasifika students. Whenever possible we also break it down further to look at children and students with special education needs and disabled students. The ministry reports annually on the most up-to-date indicator information.

PIRLS – Progress in International Reading Literacy Study
TIMSS – Third International Mathematics and Science Study
ALL – Adult Literacy and Life Skills Survey
PISA – Programme for International Student Assessment
NEMP – National Education Monitoring Project
OECD – Organisation for Economic Cooperation and Development
PBRF – Performance Based Research Fund
asTTle – Assessment Tools for Teaching and Learning
NQF – National Qualifications Framework

In 2007/08 monitoring against specific impacts will be focused on the key areas as set out in the table below: