Evidence for democracy – Transparency

Eyes on Evidence II, Evidence for Democracy, 31 January 2022

An assessment of the transparency of evidence usage in the Government of Canada

Policy-making is a complex and nuanced process, where policy actors must make sense of constantly evolving information in order to deliver policies that best serve the needs of the people. Putting evidence at the heart of policy-making is critical to not only illustrate the realities of a particular issue, but also to challenge what we might think is common sense when it comes to making decisions.

In recent years, the Government of Canada has repeatedly committed to using the best available science and evidence in its decision-making. If the public is to assess what progress has been made in this endeavour, it must be given the opportunity to scrutinize the relationship between evidence and policy decisions. This gives members of the public the chance to consider whether they agree with how evidence is being used to formulate and justify public policy. In this sense, transparency is crucial.

In Eyes On Evidence: A framework for evaluating evidence use in Canada (2021), we adapted a framework from the United Kingdom to evaluate the transparency of evidence underlying policy decisions, and tested the framework on seven federal policies. Now, in this study, we applied the same transparency framework to assess the transparency of evidence use in a total of 100 policies from 10 departments and agencies in the Government of Canada.

Key Findings

We applied a transparency framework to assess the transparency of evidence usage in 100 randomly selected policies across ten federal departments and agencies. Simply put: can the evidence behind policy decisions be found by the lay public? The framework consists of four categories:

- Diagnosis: What do policy-makers know about the issue?
- Proposal: What is the government’s chosen intervention, and why was it chosen?
- Implementation: How will the chosen intervention be rolled out, and why was this method chosen?
- Testing and evaluation: How and when will we know if the policy has worked?
Overall, our assessment found that policies scored low on the transparency of evidence usage, meaning that it’s very difficult for members of the public to find the evidence behind government policy. Too often, policies failed to provide a reference or citation for any evidence mentioned, and rarely acknowledged alternative policy options, or any absent, weak or contradictory evidence.
Transparency scores varied across different departments. The department that received the highest scores was Environment and Climate Change Canada, in part due to detailed regulatory impact statements. The departments which received the lowest scores were Canadian Heritage, and Health Canada. But higher assessment scores do not necessarily require documents to be longer — policies can achieve high transparency scores without being lengthy.
There were some shared shortcomings across departments. Too often, policies failed to provide a reference or citation for any evidence mentioned. Across all departments, almost all policies scored poorly (i.e., received a 0) in the testing and evaluation section (i.e., to know how and when a policy has worked), and rarely explored the merits of alternative policy options, or acknowledged any absent, weak or contradictory evidence.
The transparency framework is not a perfect measure. Our framework does not assess the quality of evidence. Additionally, if the evidence behind a policy decision cannot be found, it doesn’t mean that the evidence doesn’t exist, or that it wasn’t considered in the policy-making process.
The commitment to evidence-informed decision-making is inherently political. In applying the transparency framework, a bigger question emerges: what does it really mean to translate the political commitment to the principles of best available science and evidence in policy-making to the systems, structures and resources of the public service?
Transparency is not the end goal, but instead, is the first step on a path to fostering public trust and realizing the political commitment to evidence-informed decision-making. The next stage of this project will extend this transparency assessment to the provincial level, as policy-making is distributed across several levels of government jurisdiction. We will also continue discussions with governments across the Canadian federation to better understand what challenges may impact the transparency of evidence usage in policy-making processes.

Head to our blog post to learn about our key findings, or check out the full report here.