On 14 October 2021, the Australian Prudential Regulation Authority (APRA) shared insights from its new risk culture survey. This was piloted with 10 general insurers and will be rolled out to other insurance, banking and superannuation entities over the next 12 months.

The Risk Culture survey measures responses across 10 Dimensions that allow APRA to access comparable data in a consistent way across regulated entities to assess and benchmark risk culture. One of these is performance management and incentives.

APRA sent surveys to every employee at each participating entity (11,600 potential respondents in total). Participation in the survey was voluntary and the average response rate was 62 percent.

APRA also included an attention check question in the survey to help assess data quality. The purpose of the attention check is to differentiate between people who provide thoughtful responses – based on paying attention to the content of the survey – and employees who are not paying close enough attention, thereby making the data unreliable and not representative of an entity. Where respondents failed the attention check question, their responses were excluded.

The average attention check failure rate in the pilot survey was 20 per cent, which is noticeably higher than in previous APRA risk culture surveys (i.e., conducted with individual entities in risk culture deep dive reviews).

There have been media reports where this high “failure” rate has been criticised in terms of design.

Rather surprisingly, one chart shared by APRA indicates that entities with lower survey participation had the strongest risk cultures. However, there is no evidence that this also reflects companies with high reject rates. Usually in social research these phenomena can be explained after a little effort. And the findings are invariably interesting. In this regard we trust APRA will provide these.

On the design question the high “failure” rate should not be a concern, providing other checks on the “rejects” do not reveal any systemic cohort issues. Sound psycho- and socio- metric design incorporate attention and “lie” scales to indicate and refine data validity. In high sample numbers designers can afford to be choosier in terms of the confidence limits applied, so that false negatives will not impact data quality.

See the APRA results HERE.

© Guerdon Associates 2024 Back to all articles