Many organizations develop surveys by gathering smart people together in a room to create a list of questions thought to be meaningful. But surprisingly, few organizations validate their surveys in any rigorous way. When one small consulting company took the time to perform this validation, the disappointing results led to one of the greatest innovations in the company’s history.
Background: A small, rapidly growing safety-consulting company had long understood that its clients’ organizational environments influenced the effectiveness and outcomes of the consulting company’s services. In order to gather information about client context, a team of subject-matter experts developed a survey. The resulting document consisted of 29 questions the expert team believed to be critically important factors; a judgment based on their 20 years of collective consulting experience. They used the survey to gather data from their clients and followed each client for at least one year. When the opportunity arose, they attempted to use the data to predict client success, but were sorely disappointed to discover that only 1 of 29 questions predicted client outcomes, and it only did so indirectly.
The Innovation Process: Still convinced that the organizational context mattered to client success, but requiring data to prove that premise, the consulting firm turned to psychologist Dr. David Hofmann. Dr. Hofmann had been studying organizational safety climate and was able to identify 30 studies containing clues to which contextual variables were most likely to predict the success and outcomes of an organization’s safety efforts. Based on review of these studies, he proposed 10 variables for the new instrument and a preliminary model describing the relationships between the variables.
The next step was to test the validity of the instrument and of the model. This was done in collaboration with a large mining company.
Immediate Results: The results were spectacular: The literature review shortened what would have taken over 20 years by trial-and-error. The validity study took just a few months from the beginning of data collection to the final outcomes, and the data showed just what the literature review suggested: After a few refinements of the instrument it was ready for client use and could be trusted to yield valid, meaningful data.
Long-Term Results: The organizational culture assessment instrument stood up to many subsequent validity studies over the next 15+ years, including a study of 94 organizations from 18 countries. By 2012, over 1,000 organizations and nearly 200,000 people had used the instrument.
