Selecting your own questions

Does one size fit all?

We have been providing 360 degree appraisal systems for 6 or 7 years now. One trend that we have noticed is a move to having recipients have some control over the questions that are asked. At one extreme we have responded to this with a full self service appraisal solution www.myown360.com . But for most of our corporate clients the requirement is to allow the recipient to tailor their questionnaire or simply select the competencies that they should be assessed upon.

We see this as a great development. It assists the process of gaining the recipient's buy-in to their own feedback, reduces the number of questions that are not relevant, and often reduces the overall question count as there is no need for a superset of questions that everyone is asked.

A free-for-all is not the answer of course. Instead, we encourage organisations to set the overall framework - perhaps including some core competencies that are asked of everyone. Optional competencies are then selected from a range designed by HR or with our involvement.

As well as selecting competencies we are also seeing requests to select the questions that are to be asked. For a given competency the company may offer up 6 alternatives with the recipient selecting the 4 they see as most appropriate.

We beleive the benefits of an online 360 degree appraisal system are starting to be realised as the introduction of practices that improve 360 are being introduced at the same time as costs are reduced through a move from a paper-based procedure.

360 degree appraisals from Bowland Solutions

Online 360 degree appraisals

Online versus paper

As a provider of online 360 degree appraisals it will come as no surprise that we see value in moving the process away from paper-based. If you are having to make the case for an investment in a 360 system then here is a list of what we see as the key benefits. Some obvious, some less so.

• Reduced administrative overload - the key reason people approach us
• Flexibility on questionnaires - it is very simple to have subsets of questions for peers compared to direct reports, say. You could do this on paper, but I wouldn't recommend it!
• Able to monitor and chase progress easily
• Higher completion rates. I'm sure someone has worked out why, but people respond to email better than they respond to paper.
• Great flexibility on reporting - and reporting is instant
• Reduced ongoing cost
• Simple routes to get consolidated reports - that analyse strengths and weaknesses over a range of demographic information
• Integrate into your performance review process - or even your course delegate management system.
• Repeating the process is trivial

Frankly, for more than 10 people I would not consider a paper-based 360 process. And even for smaller numbers, solutions like our self-service 360 www.myown360.com make more sense than paper.

Brendan

360 degree appraisals from Bowland Solutions

Releasing the report

The big moment

Another client, another day. Today the topic was - when should the recipient get their feedback report and, more importantly, should the manager get to see it first.

For budgetary reasons and straightforward resourcing issues it was not feasible for an external or non-line manager to run the debriefing session and so the question came; should my manager see my 360 degree feedback before I did?

I boldly recommended that the manager should see it first, prepare for the debrief session, and then take the recipient through their feedback during a meeting. But our client was adamant that the recipient should be in control of the process and should not find themselves meeting their manager in the corridor and getting a "you should see what's in your 360" comment.

The ideal is, I believe, that a non-line person gives a debrief and that is the first time the person sees the report. Two key reasons; it allows the report to be understood and it allows a context to be provided. A third consideration is it allows a trained person to observe whether the recipient is emotionally affected by the feedback.

This couldn't happen here. I'm frankly uncertain. I am always nervous of someone reading their report without any assistance on understanding and context. Yet, having the recipient in control of the process is also one of our recommendations.

The client chose to have the recipient remain in control and "release" the report to their manager in advance of the feedback meeting.

Brendan

Find out more about 360 degree appraisal solutions with Bowland Solutions

Who should pick the respondents in 360 degree feedback

They'll pick their mates!

I've touched on cultural influences on 360 feedback processes before. Who picks the respondents/raters is another classic debate we get into where the organisation's culture drives much of the debate and the answer.

Our baseline position is that the recipient should pick the respondents. They are best placed to understand who can give accurate feedback on themselves and engaging them in the process is crucial to overall success and ensuring a high response rate.

Many clients believe that recipients will pick their mates - pick people who will give them favourable ratings.

The purpose of the 360 will be influential. If there is any link to pay then the 'rating' on the 360 becomes something of value outside of the intrinsic development process. The recipient has an interest in getting high 'scores'. We would recommend avoiding such a link but if it does exist then some form of managerial review of the selected respondents is inevitable.

But, all of our experience is that people approach this from a development perspective - and actively seek balanced feedback - as long as the systems / processes surrounding the 360 are supportive and used for development rather than evaluation. If you see 360 as a long-term process then even those people that in the first year are suspicious and load their selections with favourable colleagues, over time they will move to selecting an appropriate mix of colleagues and direct reports.

Brendan

Find out more about 360 degree appraisal solutions with Bowland Solutions

Editing responses in 360 degree feedback

Dubious meddling or good management?

We were discussing our 360 system and annual appraisal system with a prospective client yesterday. During the discussion we advised the prospective client that it was possible within our system to 'exclude' a respondent from the feedback report. This sparked a lively debate.

We didn't touch on it in our conversation but as well as excluding one person's responses from a report we also get asked to allow an administrator to have the option of editing a particular response.

Should it be allowed?

I see the real issue here as what is driving the request. If the request is driven from an interest in ensuring that the feedback someone receives is balanced, supportive, and constructive then being able to exclude a respondent or edit feedback can be appropriate. It is not that you wish to ignore the respondent just that their feedback may be distorting the whole report and it is better handled separately. Editing a response can be a judicial solution to an ill-judged narrative that is likely to derail the feedback session.

If, however, the drive to edit or exclude comes from a desire to massage the results of feedback, ignore controversial comments, or simply reflects a culture of interfering then that is not appropriate.

Organisational culture plays a big part on decisions you have to make when implementing 360 feedback and I see this as another example of cultural influences.

Our solution for the prospective client? Well, our online 360 feedback solution allows us to switch features on or off for each client. For this client, we would be switching it off.

Brendan

Find out more about 360 degree appraisal solutions with Bowland Solutions

Getting them to do it

How to get a 95% and above completion rate

A number of our clients approach us because they have had a poor experience with 360 feedback and yet they still see it as a tool that will offer effective feedback for their team or organisation. One of the key issues they face is that they cannot get enough people to complete the feedback.

Here are some of our hints and tips that we provide people on ensuring they hit a high completion rate.

Let me get the "you would say that, wouldn't you" one out of the way. We recommend you put the 360 online. Bowland Solutions provides online 360 degree feedback so there is no surprise there. Now, lets get a list of reasons why this will help and what you need to do to increase that feedback rate.

• Make sure the questions, the branding, the reports, the communications are all tailored - don't use some off-the-shelf set of questions and standard set of reports. People spot quickly that this is not quite right for them.
  
• Keep the questions limited to just what you need to get the feedback required. As a rule of thumb, something like 30 questions is often right.
• Keep the numbers giving feedback to a sensible amount
• Watch out for the peer group - they often have the most feedback to give.
• Limit to 3 peers per recipient
• Consider asking the peers a subset of the questions
• Communicate why this is a good thing
• Give people enough time to complete - we recommend 3 weeks
• If you can - make the recipient the person in control of the process. That way they will chase down their respondents for you.
  
• Chase them by email
• Chase them by email again
• Phone them

We hit 95% completion feedback rates - not every time, but often enough to know that it is readily achievable.

Brendan

Find out more about 360 degree appraisal solutions with Bowland Solutions

Technorati Profile

How many is too many?

Deciding how many raters to have in your feedback

This is way up there in the "what do you get asked most" table. Many of our clients immediately recognise the conflict between having lots of raters - to get the most rounded feedback - and keeping rater numbers low to stop the procedure becoming too burdensome.

Let me start with our recommendation and then work back. We recommend 8-12 raters. Anything under 6 and we start to worry; anything over 15 and we worry about that as well (we worry about our clients see!)

The factors that move you toward a lower number are:

• burden (linked to how many questions there are in the appraisal - I'll write about that another time),
  
• desire to get a tight set of feedback results, and
• a desire to ensure everyone who gives feedback has something useful to say.

The factors that make you move to a larger number are:

• anonymity - another broad topic for another day, but you do need a quorum of responses if you wish to retain anonymity
• wishing to make the process as inclusive as possible
• having a range of roles you want feedback from (external peers, internal peers, customers, suppliers, etc.) - a real benefit of an online 360 degree appraisal system
  

What we would say is that big is not necessarily better. After a certain level you will find that you can't see the wood for the trees. When giving feedback you actually find the volume of information - particularly from narrative sections - overwhelming and so will the recipient.

As a sweeping generalisation, we recommend boss, self, 3-4 peers, 3-4 direct reports. We see, and recommend, a great deal of variation from that set-up but it's a good starting place.

Brendan

Find out more about 360 appraisal systems with Bowland Solutions

Ratings versus narrative in 360 degree appraisals

When most people think of 360 degree feedback, they think of rating people against a set of statements. Often these scores are then averaged (but see my previous blog on the problems of averaging in 360 feedback) and presented back as a wholesome view of the individual.

When completing my MBA, I wrote a thesis on the narrative element of 360 degree feedback. If any school friends are reading this then, yes, I do still enjoy sport, a beer, and other normal activities! In writing that document I came in with some bias that from my experience the narrative parts of a 360 often contained richer information than the rating scales. The thesis only confirmed this, with the following key learning points.

• The narrative is often used to place the ratings in context, e.g. "Although I answered 'poor', this relates to a specific incident where Jenny...."
• The narrative can be helpful in drawing out the way each role or person is scoring (see What the boss says in 360 degree feedback)
• Without the narrative it is very difficult to give feedback to the individual....you know how people feel about the person but you do not know why nor the impact of the identified behaviour.

Although the narrative element suffers again from role bias (bosses use directional language such as "must", "needs to" whereas direct reports soften any blows "sometimes", "Jenny is brilliant but could perhaps...") it is much easier to interpret when giving feedback.

We recommend that you increase the narrative elements of your 360, preferably having one narrative question per competency. That question should be written in a way that asks for additional commentary, some examples and the impact of the behaviour.

Brendan

Find out more about 360 appraisal systems with Bowland Solutions

What the boss says in 360 degree feedback

"My boss doesn't like me"

Bowland Solutions provide an online 360 degree appraisal system . We get to see a lot of completed feedback and often we give debriefing sessions based on that feedback. What do we see?

Well, one of issues that interests academics is whether the role a person plays influences their responses. For us mere mortals, there is probably no surprise that it does.

We have found that bosses consistently score lower. Because you see the same person play out multiple roles (sometimes they respond as a peer, sometimes as boss, etc) you can isolate the role influence. Clearly you can't rely on it happening on each appraisal but, overall, bosses score lower.

In my previous article on scoring in 360 degree appraisals I warned against averaging scores from a 360. Here is another reason why it doesn't work well. If the boss is scoring in a range 1-4 and everyone else tends to score in a range 2-5 then you are not comparing apples with apples if you combine their scores.

What to do? We recommend that you use a bar chart style report that shows the boss' score clearly. When feeding back, armed with your new knowledge, you can place the boss score in context and you can see any consistent underscoring over the whole appraisal.

In a future article we will also discuss how boss' write narratives compared with others - and that gives you the main clue on how to interpret the scores.

Brendan

Find out more about 360 appraisal systems with Bowland Solutions

360 degree appraisals and the Russian judge

It’s a 5.6 from the Russian judge

The benefit of averaging in 360s has been likened to the removal of rogue scores from ice dancing competitions. Take 9 scores, remove the two extremes and average the rest and noone can deny Torville and Dean the gold. But is this valid for a 360? Aren’t we interested in why two people scored at the extreme? Wouldn’t we be better reflecting back the detail of the feedback rather than averaging out the variety?

It seems that people like numbers - it gives clarity. Bosses like to feel it is getting to the nitty gritty of someone's performance. I'm not so sure.

First - you lose something. If three people say I am poor and three people say I am fantastic then being told that overall I am average simply isn't correct and, worse, is misleading.

Second - the average isn't correct. It assumes that people evenly score on the range. But, particularly us Brits, people don't like scoring at the extremes....so, when they do, it needs to carry more weight.

Third - 360 degree feedback is meant to give you a rounded opinion, a range of feedback. Leave the detail in, is what I say.

And finally, the conversations from the 360 are often painful. They start with "I'm 3.7 on Leadership and 3.6 on Customer Service, so I need to work on Customer Service". The precision of numbers is misleading.

Together with colleagues, I set up a company concentrating on 360 degree appraisal to give companies a chance to handle 360s effectively and efficiently. Our recommendation is always that the feedback report should give a simple representation of what has been said and keep numbers away. If there is a craven need for a score - then use something like a spidergraph to show an overall picture but don't create scores for each behaviour.

Brendan

Find out more about online 360 degree appraisal systems with Bowland Solutions