Why Norms?
When people are looking at the results from their survey, they often wonder how they compare with those found in other organisations. People often think the results don't mean much by themselves, so they want to put them in context by finding out if they are better or worse than other organisations get, or much the same.
What are norms?
To meet this need, we have developed a sophisticated norms management system, comprising a database of over a million data points, each of which represents one informant's response to one item in a survey. By summarising all these prior responses to the same question, and arranging them in ascending order of the score the informant gave, we can tell you how your result compares with results from those other surveys by reporting the percentage of those other responses which yours is better than, so for each question in your survey which has a parallel in our norms we can report that "Your result was better than X% of the normative data."
This allows you to put your results in context. The norms cover a collection of 62 core employee satisfaction items and the sample sizes (the number of individual responses to the question which are included in the norms) vary according to the popularity of each item in our clients' surveys over the years. In the norms chart below, we can see for example, that an average result which is a percentile score of 60 is only at the 43rd percentile of the norms i.e. better than 43% of all the responses included in the selected norms set.
Click to see a graphical representation.
We don't expect you to figure out these numbers from a chart, though. Here is how this item might appear in a report if your average result was at the 59th percentile of the range of responses to the question. This example happens to come from the bottom of a page, so you can also see the page footer. There were 322 responses to the item in the client's survey, and 1,431 replies in the normative data the client chose to use.
Click to see a tabular representation.
Where do the norms come from?
They are collected from the surveys we have handled for clients over the years. You can choose the normative data your results are compared with, though. To secure the biggest samples to compare with, you may choose to compare with all our other clients. Or you may wish to choose specific norms data to compare your results with. This will naturally reduce the sample sizes but the norms we use for your report can be selected according to
- gender of informant,
- profit / non-profit sector,
- industry,
- survey date,
- Question text identical / identical & equivalents (see wording below),
- Specific clients (though we won't identify any specific client to you).
Caution
We have some serious reservations about the use of benchmarking or normative data, however, so while we are pleased to be able to offer a service which meets a need expressed by many clients, we offer it with the health warnings set out below.
Read David Lusty's article Debunking the benchmarking myth in Human Capital Management.
Expectation
This is the most serious drawback of benchmarking, because it can actually lead to a "good" organisation getting worse satisfaction scores than a "bad" one.
The responses people give to questions about their satisfaction with anything don't depend only on the experience they have had. Exactly the same experience might lead to quite different satisfaction responses depending on what the informant had expected. Someone who had not expected to be treated particularly well might be very pleased if their actual treatment came as a nice surprise, even if it wasn't particularly wonderful. On the other hand, if they had expected excellent treatment, exactly the same not particularly wonderful experience would leave them disappointed so they would answer a satisfaction question in a survey differently. So satisfaction is a function of experience and expectation.
The harder you have worked at satisfaction, the higher the expectation of your informants is likely to be, making it harder for you to get high satisfaction scores. When you compare your satisfaction scores with those obtained in another organisation, you won't know what their informants were expecting. Another organisation might not in the past have tried half as hard as yours, and their informants, expecting to be very shabbily treated, might have been pleasantly surprised even to be treated in a manner you would regard as unacceptably poor.
Should you then be concerned if your satisfaction scores are no better than that other organisation's? Should you invest management time and scarce resources in dealing with the "problem"? We don't think so.
Of course you won't be comparing your results with just one other organisation's, so the wider norms group will include some "good" and some "bad" organisations. All the same, if your result doesn't compare as well as you had hoped you can never be sure if that is because people are getting a less good experience from your organisation or because they have learned to expect more from you.
Wording
For their own very good reasons, clients often find a standard item wording inappropriate to their organisation, so they adapt it to meet their needs. Our norms include items using the identical standard wording, but they also include items expressed in equivalent terms. So where the standard item reads I always get the equipment and/or facilities I need to do my job, we also include items where clients have preferred I have the equipment and facilities I need to do my job; or We always get the equipment and/or facilities we need to enable us to do our jobs; or I have sufficient resources to enable me to do my job. This pragmatic approach is intended to provide clients with the widest possible normative group to compare with but we acknowledge that we are not strictly comparing like with like, so we offer the option of norms based on the smaller samples including only surveys using the standard text word for word.
Response frame
Clients use different response frames, so while one survey is scored on a four point scale, Very dissatisfied, Dissatisfied, Satisfied, Very satisfied; another might use a seven point agreement scale, Strongly disagree, Disagree, Tend to disagree, Neither agree nor disagree, Tend to agree, Agree, Strongly agree.
To incorporate data gathered using any response frame, each individual response is expressed in the normative database as a percentile score; that is converted as if its response frame had been a scale from 0 to 100. For a discussion of the use of percentiles for comparing results on different scales, see Percentiles in the QUANTIFY glossary of terms.
Sequence
The response to a question can be influenced by the other questions which come before it in the questionnaire. Imagine asking people to rate their overall happiness on some scale. If the question is preceded by one asking them to rate the happiness of their marriage / relationship, this narrows the perception of the overall question, so people reply to it much more in the context of their relationship than they would if the general question was put before the marriage / relationship one.
Even if we restrict ourselves to the identical text of the standard core question, the different questions which have preceded it in the various questionnaires will have coloured responses to the standard question in varying ways which we can't predict or control, so once again, we aren't strictly comparing like with like.
Conclusion
All these influences mean that while normative comparisons can help to provide a context for your survey results, they should be approached with great caution and corroborative evidence should be sought before you invest effort and resources in addressing a perceived competitive disadvantage.
