This paper considers both the considerable opportunities and the issues associated with using individual-level data and citizen-based analytics to inform social policy development and implementation. This growing application of big data analytics occurs within the context of new insights from biological, behavioural and social sciences that allow for greater understandings of how data can be better applied to assist in the evaluation both ex ante and ex post of social sector interventions. In New Zealand, this approach is being developed to underpin social policy development and implementation and has been termed ‘Social Investment’. In Europe, this type of analysis aligns with ongoing work of the European Commission’s Joint Research Centre in the areas of fairness and resilience.

The basic argument underpinning this general approach is that interventions at one stage of the life-course will have impacts later in the life-course, and often with a much broader range of outcomes than those initially targeted. While this general understanding is not new, the scientific, behavioural and social science that seeks to explain these relationships has advanced rapidly, allowing for a broader range of possible policy developments to be considered.

The emerging challenge for policy-makers is to better understand these complex relationships in a way that allows them to decide which interventions, delivered to which individuals, at what stage in their life-course, will do the most to boost resilience later in life – leaving individuals better able to cope with the inevitable stressors of life. Because there is an almost infinite range of possible social interventions, governments are placed in a position of having to choose between many possible options. Each has its advocates and the decisions are inevitably politically charged. Further, the evidence needed on which to decide to terminate an ineffective programme, or to enhance an effective one, is often absent.

Individual life-courses are affected by many influences including biological, family, social and environmental factors. Governments and social sector providers are increasingly able to collect and use knowledge related to a growing number of these factors. By combining such data, especially longitudinal and multi-domain data, in large databases under the appropriate guidelines and controls, it is possible to obtain group-level data that provides a greater understanding of how these factors interact, and of the potential for prevention, amelioration or remediation. This is the basis of ‘citizen-based analytics’. The compelling scientific, ethical, economic and policy arguments for this approach are presented.

From both the State’s and the individual’s perspective, optimal social outcomes require integration and delivery of effective and efficient services. Citizen-based analytics must be supported and informed by scientific understandings to avoid a large number of potential interpretative errors. Whereas citizen-based analytics generally only need anonymised but 2 of 25 individually linked data, there are some circumstances in service delivery assessment and management where identifiable client level data may be. Therefore, it is important that citizen-based analytics are supported by appropriate data governance and safeguards, accountability and oversight that take into account and justify the distinct purposes for which data may be needed.

This paper discusses these distinct dimensions, including: the need for social licence and transparency around data use; the need for governance and data management structures; the need for clarity of system architecture; the importance of high quality data; and a number of related issues including the data use for service improvement. Citizens need clarity about the purposes for the collection, curation and use of their data. The paper discusses the considerable utility and also the limitations of citizen-based analytics, the broader implications of this approach, and the potential impact on the policy process and service delivery.

Download discussion paper