Benchmarking comparisons for Employee and Customer Satisfaction Surveys
Why Norms?
When people are looking at the results from their survey,
they often wonder how they compare with those found in other organisations.
People often think the results don't mean much by themselves, so they
want to put them in context by finding out if they are better or worse
than other organisations get, or much the same.
What are norms?
To meet this need, we have developed a sophisticated
norms management system, comprising a database of over a million data
points, each of which represents one informant's
response to one item
in a survey. By summarising all these prior responses to the same
question, and arranging them in ascending order of the score the informant
gave, we can tell you how your result compares with results from those
other surveys by reporting the percentage of those other responses
which yours is better than, so for each question in your survey which
has a parallel in our norms we can report that "Your result was
better than X% of the normative data."
This allows you to put your results in context.
Caution
We have some serious reservations about the use of benchmarking or normative data, however, so while we are pleased
to be able to offer a service which meets a wish expressed by many
clients, we offer it with the serious health warnings set out below.
Read David Lusty's article Debunking the benchmarking myth which has appeared in a number of journals.
Expectation
This is the most serious drawback of benchmarking, because
it can actually lead to a "good" organisation getting worse
satisfaction scores than a "bad" one. We think that's perverse, but there's
nothing anyone can do to prevent it.
The responses people give to questions about their satisfaction
with anything don't depend only on the experience they have had. Exactly
the same experience might lead to quite different satisfaction responses
depending on what the informant had expected. Someone who had not
expected to be treated particularly well might be very pleased if
their actual treatment came as a nice surprise, even if it wasn't
particularly wonderful. On the other hand, if they had expected excellent
treatment, exactly the same not particularly wonderful experience
would leave them disappointed so this time they would answer a satisfaction
question in a survey less favourably. So satisfaction is a function of
experience and expectation.
The harder you have worked at satisfaction, the higher
the expectation of your informants is likely to be and that makes it harder
for you to get high satisfaction scores. When you compare your satisfaction
scores with those obtained in another organisation, you won't know
what their informants were expecting. Another organisation might not
in the past have tried half as hard as yours, and their informants,
expecting to be very shabbily treated, might have been pleasantly
surprised even to be treated in a manner your organisation would regard as unacceptably
poor.
Should you then be concerned if your satisfaction scores
are no better than that other organisation's? Should you invest management
time and scarce resources in dealing with the "problem"?
We don't think so.
Of course you won't be comparing your results with just
one other organisation's, so the wider norms group will include some
"good" and some "bad" organisations. All the same,
if your result doesn't compare as well as you had hoped you can never
be sure if that is because people are getting a less good experience
from your organisation or because they have learned to expect more
from you.
For their own very good reasons, clients often find
a standard item wording inappropriate to their organisation, so they
adapt it to meet their needs. Norms include items using the identical
standard wording, but they also often include items expressed in equivalent
terms. So where the standard item reads I always get the equipment
and/or facilities I need to do my job, the data might also include items where
clients have preferred I have the equipment and facilities
I need to do my job; or We always get the equipment
and/or facilities we need to enable us to do our jobs;
or I have sufficient resources to enable me to do my job.
This pragmatic approach is intended to provide clients with the widest
possible normative group to compare with but it doesn't provide like for like
comparisons.
Response frame
Clients use different response
frames, so while one survey is scored on a four point scale, Very
dissatisfied, Dissatisfied, Satisfied, Very satisfied; another might
use a seven point agreement scale, Strongly disagree, Disagree, Tend
to disagree, Neither agree nor disagree, Tend to agree, Agree, Strongly
agree.
To incorporate data gathered using any response frame,
each individual response is expressed in the normative database as
a percentile score; that is converted as if its response frame had
been a scale from 0 to 100. This is a useful device for comparing data gathered
using different response frames, but it is a bit rough and ready. For a discussion of the use of percentiles
for comparing results on different scales, see Percentiles
in the QUANTIFY glossary of terms.
Sequence
The response to a question can be influenced by the other
questions which come before it in the questionnaire. Imagine asking
people to rate their overall happiness on some scale. If the question
is preceded by one asking them to rate the happiness of their marriage
/ relationship, this narrows the perception of the overall question,
so people reply to it much more in the context of their relationship
than they would if the general question was put before the marriage
/ relationship one.
Even if we compare data derived from items using identical text the different questions which have preceded
it in the various questionnaires will have coloured responses in varying ways which we can't predict or control,
so once again, we aren't strictly comparing like with like.
Conclusion
All these influences mean that while normative comparisons
might give some clue to how your survey results relate to other people's, they should
be approached with great caution and corroborative evidence should
be sought before you invest effort and resources in addressing any perceived
competitive disadvantage.
|