Ensuring outcomes-based policies are meaningful throughout the organisation's hierarchy

A paper for the 2nd Annual Developing, Managing &
Measuring Outcomes-Based Policy in the Public Sector Conference John
BishopSocial Commentator Wellington31st March and 1st April 2004

Introduction
Ensuring any kind of policies mean something throughout the organisation is
always a challenge for senior management teams and particularly for Chief
Executives and their communications people.
So on the face of it, ensuring that "outcomes-based policies are meaningful
throughout the organisation's hierarchy", is simply other internal
communications issue. From another perspective it's one more challenge for the
HR people, the planners and senior management to get people in the organisation
to sit up and take notice about something. And if that's all there was to say
about the matter, I'd stop now and we could have an early finish to the day.

So is there something new here?
I think there is certainly something going on, and this afternoon I will try
and unravel it for you and then seek to answer the four specific questions that
are posed in the brief.
First of all what is Managing for Outcomes?
According to the SSC website Managing for Outcomes in the Public Service is
"an outcome-based approach to departments' planning, management and reporting.
Its aim is to improve the performance of the Public Service and requires
departments to adopt a strategic and outcome-focused approach to planning,
management and reporting while focusing on delivering outputs."
So what's the big deal about it?
In preparing this presentation I went to the State Services Commission and
talked to Deputy Commissioner Tony Hartevelt.
It is clear that the Commission feels itself under pressure to improve the
use of public funds. He made a number of points to me on this matter:

  1. Departments have developed sophisticated systems for bidding for new money.
    The difficult part is assessing the effectiveness of baseline money. If you've
    got it, you tend to keep it, even in an environment where funding is scarce.
    Public sector processes for evaluating and recommending changes in the baseline
    lag well behind the processes for getting new money.
  2. Efforts to measure outcomes over a shorter period have provided too
    difficult. Results take half a decade or longer to emerge and often it's hard to
    prove cause and effect.
  3. Chief executives have to have the courage and the information to go to
    Ministers and say that they ought to do less of this and more of something else.
    Getting them to do that has proved very difficult. That is what Managing for
    Outcomes is all about.
  4. We want them to specify the "vital few things" that they really have to get
    right.
  5. Build an explanation that they can support with evidence. We call it
    "intervention logic".
  6. We talk co-operation and co-ordination but we don't practice collaboration.
    The extent to which we have collaborated is despite the system not because of
    it. Where it has happened it's because of determined professionals on the ground
    made it happen
  7. Collaboration is an agency saying we'll stop doing this and you (another
    agency) can do it instead. Success will be when the whole system is greater than
    the sum of its parts.

Mr Hartevelt explained that last year was the first time that all 35
departments had been required to do a Statement of Intent, and that the
shortcomings in the SOIs were only to be expected, predictable, and would be
remedied as departments gained more experience.
He might be right, but it is worth looking at the report of the team that
evaluated what the departments did.
The team was led by Tim Blackmore of the Treasury and was published in August
last year. It's available on the SSC website. So what did the report say. Here's
their summary.

  • Departmental capacity to undertake meaningful organisational capability
    appraisal appears limited. Departments do not have a clear picture of the
    current state or future capability requirements, or access to common capability
    appraisal metrics.
  • Measurement of outcomes - Outcome indicators or performance measures are
    virtually non-existent in the majority of 2003/04 SOIs. Most departments have
    identified this as an area for future development.
  • Intervention logic - The quality of intervention logic is generally not
    good. The SOIs have rationale statements but the question is whether these are
    well founded. In some cases, the causal links between outcomes and outputs are
    not clear.
  • Identification of risk and risk management is very limited across the SOI
    set. Demonstration of an understanding of risk, and the link between risk and
    operating environment, is non-existent.
  • Audience assessment is weak. Currently departments focus too much on meeting
    central agency expectations in their tabled SOIs, greater referencing or
    hypertexting to other lower level planning documents and processes would be
    welcome, as would more simply worded and structured SOIs.
  • Management of shared outcome development and sectoral reporting could be
    improved. At the moment departments tend to assert linkages to each others'
    outcome sets rather than describe how interdepartmental collaboration or shared
    outcomes contribution will work. OAG officials suggest that effective sectoral
    reporting is dependent on designated single agency leadership, and clarity in
    how accountability for outputs can be approached in a context of shared
    contribution to outcomes.

It's hard to see that as much more than a C+, must try harder, could do a lot
better, kind of a report card.
And the SSC has made it clear that it does want and expect more, and it's put
in place various measures to help departments to do.
Mr Hartevelt said "The SSC is now highlighting that it is not going to let up
on MfO and getting SOIs done well. We will be looking for demonstrable progress
in the next round. Fair enough.
Survey of Departments
Departments would typically start their planning and preparation for their
SOIs in February/March so I decided to run a small survey to see whether the
matter was being taken more seriously than last year.
I developed a short questionnaire and asked 10 departments to take part. The
questions were about what happened last year, and what was happening this year,
and the survey also asked about internal communications and internal
leadership.
Respondents could complete the survey anonymously and online. I got seven
replies, and these are attached to the paper.
Looking first at what happened last year; the picture that emerges is
that

  • The language of Managing for Outcomes was important - five of the seven
    Chief Executives used it a lot or sometimes
  • The approach was very important or reasonably important for all seven
    agencies
  • And this year its likely to be more important than last year for four of the
    seven agencies and about the same for the other three

I was also interested in how they had communicated to or involved staff last
year.

  • Only one of the seven had prepared a communications plan
  • Six put stuff on the intranet, and five sent staff a message of some kind
  • Only three sought staff input in any formal way but
  • All seven advised stakeholders and key external audiences

The conclusion I draw is that the external audiences are seen as more
important than the internal audience, and while there were efforts made to
inform staff the efforts to engage or involve staff were rather less.
So were things going to change this year? What would people do differently
this time round? There were a variety of answers

  • more input from third tier managers
  • Outputs and performance measures have been reviewed.
  • We have started earlier to allow plenty of time for consultation and
    thinking. A significant amount more time is being placed into ensuring
    robustness of outcomes, definitions and measurement.
  • Engaged Minister on a number of occasions Focus of senior staff day to raise
    awareness
  • and understanding. More formal and frequent briefings of staff
  • With a new Act and a new strategic plan and hence new goals our planned
    outcomes will be different

These are constructive and sensible steps to take.
So were there greater efforts being made in terms of time, resources and
priority?
Four of the seven were putting in more effort on the SOI and three were
putting more effort into the communications as well. The others were doing about
the same as last year on both counts.
One of the key features of the SOI/Managing for Outcomes process is what the
SSC calls "the intervention logic", the chain of reasoning that links outcomes
to outputs to the activities of the agency. How well are those linkages
understood by the staff, and for that matter by the leadership.
The responses from the seven agencies indicated that they thought that staff
understood the relationship

between activities and outputs
very well
3

quite well
4

but the relationship between outputs and outcomes were much less well
understood

between outputs and outcomes
very well
1

quite well
1

only somewhat
4

not very well at all
1

Here's the interesting bit. I asked the agencies the same question about how
well the leadership understood these matters.

between activities and outputs
very well
6

quite well
1

but the relationship between outputs and outcomes were much less well
understood

between outputs and outcomes
very well
3

quite well
3

only somewhat
1

not very well at all
0

Survey Conclusions
In short the two main findings are:

  • The leadership understand both sets of relationships better than the staff,
    and
  • Understanding of the relationship between activities and outputs is higher
    for both staff and leadership than it is of the relationship between outputs and
    outcomes.

I find that rather disturbing because it suggests that departments and
agencies know how to carry out their activities and can see how this advances
the department's outputs, but after that it becomes rather bleary.
As Tony Hartevelt put it to me.... "the penny has not dropped that (the SOI) is
a document that is about changing the way of managing public resources based on
evidence of effectiveness. This is only slowly emerging."
So final question, is the leadership doing anything about this. Are they
clearly explaining those relationships to staff. Three are, and three do so
sometimes. One respondent checked the box saying 'they try'.
(Full results are in Appendix One)
Answering the Questions
So let's answer the four questions or matters that were raised in the
brief.
They were

  • Implementing a strategic focus on communicating outcomes based management
    throughout the organization

The answers I think are the same answers as for another other organisation
wide programme. One, the change has to be led, endorsed and modeled by the
senior management.
Two, they have to mean it, and be serious and credible about it and make it a
big deal. If necessary back it with sanctions.
Three, you have to show that the new ways of doing things are important to
the organisation and to the staff at all levels.
Four, you have to show staff the connection between your public service
management planning model and their tasks. Just as the SSC is looking for
intervention logic between the outcomes, outputs and the activities of the
department, so too spell out the internal intervention logic between the
activities of the department and the tasks any one person or group carries out.
Answer the question: what does this mean for me as a staff member, what do I
have to do differently?
Five, measure and evaluate, and six, celebrate. The message is clear, what
gets measured gets done, and what is celebrated is what is important. You want
this stuff taken seriously: measure it and celebrate it.

  • Establishing a strategy for making outcomes accessible, purposeful and of
    value to different departments and levels of staff

I think I have answered that question already. The key to me is in the
breaking down of the activities into tasks performed by individuals and
groups.
One point that does emerge here. I said earlier that you needed to be able to
answer the questions: what does this mean for me, what do I have to do
differently?
For a lot of staff a lot of the time, the answer will be that they don't have
to do things differently. The organisation might see the activity differently
but if the tasks don't change then the shift in thinking doesn't mean much to
the people doing the work.
That's important because it makes it even harder to engage them in the
planning cycle, and the Managing for Outcomes model because their tasks remain
disconnected from it.

  • Improving outcomes based policy development by integrating staff feedback

Several agencies in my survey said that they were making greater efforts to
involve either third level managers or staff in key front line areas in the
planning round, but it was proving very difficult.
At least part of the reason is what I was talking about before: no change in
their jobs means little incentive to engage.
However I would like to make two suggestions:
One is that staff feedback need not be feedback about the plan or statement
of intent, which is a hard thing to get your brain around.
Staff feedback can and ought to be a continuous process. Why not designate
one person on the planning team to bring together all the feedback formal and
informal that has recorded during the year and feed that into the planning
round.
Not only should that help inform the planners, the communications people will
get excited, because you have created an opportunity to show staff that their
feedback has been heard, taken into account and is actually making a difference.
That's quite a powerful message, which will encourage more feedback in future.

Of course you do have to have feedback gathered during the year, and if you
haven't got any, you are probably planning in a vacuum anyway. My point is that
staff feedback does not have to be feedback about the plan itself; it could be
feedback about their activities and experiences, which is then reflected in the
content of the plan.
The second suggestion is that you don't necessarily have to get feedback
about all of the plan. Why not select say three or four "flashpoints" if you
like and seek feedback on those.
By flashpoints I mean issues on which there is going to be keenly held views,
so there is something to talk about and debate. Few people will be motivated to
feedback anything about routine tasks adequately performed, particularly where
the tasks are not either mission critical or inherently interesting in
themselves.
They are much more likely to feedback on points where there is controversy,
uncertainty, differing expectations, variations in performance and the like.

So organise some sessions where each of these points is discussed by those
who are interested in doing so, and record the suggestions, comments, and
reactions. Use a facilitator to probe and explore and get more than just top of
mind views. Make the groups think. There will be merit in some of their
suggestions, and that makes the exercise worthwhile.

  • Using outcomes to drive motivation towards achieving strategic objectives

This seems to me to reverse the usual direction of planning logic. The
conventional approach is to set outcomes, derive outputs from those and work
down to a set of objectives and activities for the agency. What this appears to
suggest is that the desire for outcomes can be used to motivate people to
achieve the objectives of the agency.
For example the desire for a healthier New Zealand can be used to motivate
health professionals to do more immunization, because immunization makes a
direct contribution to a healthier New Zealand.
I am not at all sure that I agree with the logic and certainly I do not know
of any empirical evidence that shows that the logic works in that way.
It might, and I am not saying it doesn't. I am saying I don't know.
On the face of it the evidence is mixed. Doctors, teachers and police
officers may well all share a desire to make society a better place...and good on
them for that...but that doesn't seem to make recruitment to Kaitaia, south
Auckland or the West Coast of the South Island any easier.
On the other hand a passionate belief in a healthy New Zealand clearly
underlies the activities of those who campaign for a smoke free New Zealand or
to reduce obesity, or to remove the stigma of mental illness. So perhaps
believing in ends does drive people to work harder on the intermediate goals.

Despite the lofty goals of greater efficiency to which the SSC wishes
departments to aspire and preferably reach, the reality is somewhat
different.
One agency said ...it is a challenge to engage non-managerial people in
public sector management frameworks. The key is face to face briefings at all
levels - this time commitment can be a big ask when pressing day to day
realities compete for the attention of managers.
And this really brings me back to the central problem. If Managing for
Outcomes and its most tangible symbol the Statement of Intent are to become
significant tools of logic justifying state intervention and the use of public
money, then departments really have to do better than this.
Secondly, if departments and agencies are to become well focused on
delivering their activities which advance outputs, which are a step to outcomes
then the whole organisation needs to understand that bottom to top and down
again. The evidence would seem to be that they don't - or at least not yet
anyway.
Finally, saying that it is hard to make planning frameworks meaningful is
really to say that management hasn't yet found ways to make the work
understandable and interesting enough that staff will find it valuable to take
an interest.
And if management can't motivate staff, achieve goals that contribute to good
social outcomes, and if they can't prove that their intervention helps people
live better lives in some way, or make a difference to the quality of our
society, you have got to ask why are we allowing them to be in charge of a
significant chunk of the government's and our nation's resources.
I think it is a very simple proposition at the end of the day: if you can't
prove it makes a positive difference, you can't spend it. That rule, rigorously
applied, would certainly focus a few minds and might, in turn, improve the
quality of government expenditure.
John BishopSocial Commentator1
April 200432 Disley StreetHighburyWellingtonTel 04 475
8650Mob 0274 482 247Web www.johnbishop.co.nzEmail john@johnbishop.co.nz
Data Summary: Managing for Outcomes/Statements of Intent

John Bishop - Communicator for a paper to the "Second Annual Developing,
Managing and Measuring Outcomes Based Policy in the Public Sector" Conference
Wellington 31 March & 1 April 2004

Ten large and important government agencies were asked about their
experiences with the development of Statements of Intent and Managing for
Outcomes. Seven responded, and the questions, answers and comments are captured
below
Q1 In preparing your Statement of Intent last year, did your Chief Executive
and senior management team use the term Managing for Outcomes?

Yes
A lot
4

Sometimes
1

Hardly at all
1

No

4

Not sure

0

Q2 How important was the Managing for Outcomes approach in the preparation of
your Statement of Intent last year?

Very important
4

Reasonably important
3

Not very important
0

Didn't really count for much at all
0

Didn't use it
0

Q3 Departments and agencies have probably started preparations for this
year's Statement of Intent. Compared to last year in your department, is the
Managing for Outcomes approach likely to be

more important
4

less important
0

about the same
3

won't be using that approach
0

Q4 What steps did you take last year to a) communicate the Managing for
Outcomes approach to staff; and b) incorporate their feedback into the planning
for the Statement of Intent? Did you for example (check all that apply)

Prepare and implement a communications plan
1

Place material on your intranet
3

Send a message email or newsletter to staff
5

Hold briefings for staff
2

Seek staff comment or reaction in any formal way
3

Give all staff a copy of your SOI
4

Place the Statement of Intent on your intranet
6

Advise other stakeholders and key external

audiences about the SOI
7

Involve any other non government stakeholders

in the development of the SOI
4

Hold a social function when the SOI was completed
2

Take any other communications action (please describe briefly)

A mini version of the SOI was produced for all staffWe don't use the
language "Managing for Outcomes" but that is what we do The department has
set up teams to develop outcomes, intervention logic, outputs and performance
measures. The results are used for planning and SOI document production.Held
consultation with key government departments, including SSC,TPK and Treasury
Currently developing Ministry wide strategy for improving stakeholder
communications that draws on intermediate outcomes in the SOISOI was
developed in work plans across Ministry and discussed in team meetings

Q5 Thinking about the preparation of this year's Statement of Intent, what
will your department or agency do differently from last year?
more input from third tier managersOutputs and performance measures have
been reviewed.We have started earlier to allow plenty of time for
consultation and thinking. A significant amount more time is being placed into
ensuring robustness of outcomes, definitions and measurement. Engaged
Minister on a number of occasions Focus of senior staff day to raise awareness
and understanding More formal and frequent briefings of staff With a new Act and
a new strategic plan and hence new goals our planned outcomes will be different

Q6 Is the amount of effort (time, resources and priority) involved in
preparing the SOI this year likely to be

greater than last year
4

about the same
2

less than last year
1

we're really not doing anything at all
0

Q7 Is the amount of effort (time, resources and priority) involved in
communications about the SOI this year likely to be

greater than last year
4

about the same
3

less than last year
0

we're really not doing anything at all
0

Q8 How well do the staff in general in your department/agency understand the
relationship between their activities and your outputs and between the outputs
and the government's desired outcomes

between activities and outputs
very well
3

quite well
4

only somewhat
0

not very well at all
0

between outputs and outcomes
very well
1

quite well
1

only somewhat
4

not very well at all
1

Q9 How well does the leadership of your department/agency understand these
relationships?

between activities and outputs
very well
6

quite well
1

only somewhat
0

not very well at all
0

between outputs and outcomes
very well
3

quite well
3

only somewhat
1

not very well at all
0

Q10 Do the leaders of your department/agency clearly explain those
relationships to staff?

Yes
3

No
0

Sometimes
3

They try
1

Comments
It is a challenge to engage non-managerial people in public sector management
frameworks. The key is face to face briefings at all levels - this time
commitment can be a big ask when pressing day to day realities compete for the
attention of managers