Analysis of GPT-4 competence in assessing complex legal language: Example of Bill C-11 of the Canadian Parliament. - Part 1

If your at all familiar with Canadian politics over the past few years you may have heard of this mysterious ‘C-11’ (a.k.a. ‘the Online Streaming Act’) being mentioned in various places.

It refers to a complex bill attempting to regulate an area that is notoriously difficult to regulate in a fair and balanced way. Namely, online media, which includes the streaming giants, news, radio, but also smaller scale productions such as podcasts, streamers, etc. It attempts to mirror the regulations that broadcast and radio content conform to in Canada while accommodating the unique characteristics of the internet.

It notably excludes purely personal communication such as LW posts, though not without some asterisks if there is any for-profit component, hence the polarizing nature of the legislation, which will be touched on in part 2.

I chose this example to represent complex legal language because it’s:

Really controversial, hence it’s written in a dense and ‘cover all angles’ way, with references to sub sections, sub paragraphs, and so on, with many topics interweaved throughout, so much so that it is genuinely difficult to parse and understand for the normal reader.
Also not a gigantic omnibus bill that would exceed the capacity of the ChatGPT variant of GPT-4 to remember, as C-11 already requires breaking up into 9 parts to input. (after editing out the french parts of the bill, which yes is co-mingled with the English text...)
Publicly available for anyone to reference (I’m basing this off the ‘Third reading’ found via the link above)
Topical, outside of the GPT-4 aspect, for at least some portion of the reader-base
An exemplar of the way Canadian Parliament bills are generally structured, which are a source of confusion in of itself, because of billingualism and various other quirks, so this served as a way to test parsing capabilities on a larger, structural, scale.
Amending an existing law, so the older versions of the law, ‘the Broadcasting Act’, can be useful as comparisons for later analysis if anyone wants to run it through GPT-4 and see the major changes.

Keep in mind that although the bill is primarily about amending an existing law, there are various other laws amended as well by the bill, which have been excluded from this analysis so that the potential reader-base will be larger than a handful of Canadian law experts.

(The countries I know of have law(s) broadly similar to the Broadcasting Act regulating TV, radio, etc… but in case you’ve never heard of this before I would recommend skimming through the wikipedia page at least)

So let’s begin.

At first I asked for a quick summary without providing any of the bill’s text, relying entirely on GPT-4′s knowledge.

However to make things interesting I provided only the URL link without an associated title. The response was:

Bill C-11, also known as the Digital Charter Implementation Act, 2020, is a piece of legislation aimed at modernizing Canada’s privacy laws and enhancing consumer protection in the digital age. The bill is a response to the rapid growth of technology and the increasing concerns about personal data privacy.
The Digital Charter Implementation Act consists of two parts:
Consumer Privacy Protection Act (CPPA): The CPPA replaces the Personal Information Protection and Electronic Documents Act (PIPEDA) and introduces new privacy regulations for private-sector organizations. The key provisions of the CPPA include:
a. Consent: Organizations must obtain individuals’ consent to collect, use, or disclose their personal information. Consent should be meaningful, clear, and easy to understand.
b. Transparency: Organizations must provide individuals with clear information about their privacy practices, including the purposes for which personal information is collected, used, or disclosed.
c. Right to access and delete: Individuals have the right to access their personal information held by organizations and request corrections or deletion of their data.
d. Data portability: Individuals have the right to request their personal information in a structured, commonly used, and machine-readable format to transmit it to another organization.
e. Algorithmic transparency: Organizations using automated decision-making systems (e.g., algorithms) must provide clear explanations of how such systems work and how they might impact individuals.
f. Privacy management programs: Organizations must establish and maintain comprehensive privacy management programs to ensure compliance with the CPPA.
g. Breach notification: Organizations must report significant data breaches to the Office of the Privacy Commissioner (OPC) and notify affected individuals.
Personal Information and Data Protection Tribunal Act: This act establishes the Personal Information and Data Protection Tribunal (PIDPT), an independent body responsible for adjudicating appeals of decisions made by the OPC under the CPPA. The PIDPT has the authority to impose administrative monetary penalties and issue orders related to privacy matters.
In addition to these provisions, the Digital Charter Implementation Act also grants the OPC greater enforcement powers, including the authority to conduct audits and investigations, issue orders, and impose fines for non-compliance.
Overall, the Digital Charter Implementation Act seeks to strengthen privacy protections for Canadian consumers and hold organizations accountable for their data practices in the digital age.

So right off the bat, we see a big problem, although it got C-11 right, the ‘Digital Charter Implementation Act’ is not C-11 at all! That’s Bill C-27 which hasn’t even been passed yet.

As a summary of C-27 it’s fine enough for someone who needing an overview, and remarkably competent considering it must have based this just from prior training while being prompted by a parsing error.

But this is clearly not what we want. So I continued, but without pointing out the mistake, to test if it could self-correct its mistake:

What are the downsides of this proposal?
While Bill C-11, the Digital Charter Implementation Act, 2020, is designed to strengthen privacy protections for Canadian consumers and hold organizations accountable for their data practices, there are potential downsides to consider:
Increased compliance costs: The new regulations and requirements under the Consumer Privacy Protection Act (CPPA) may impose additional financial and administrative burdens on businesses, particularly small and medium-sized enterprises (SMEs). The costs of implementing privacy management programs, data protection measures, and complying with the new rules could be substantial for some organizations.
Regulatory complexity: The introduction of the CPPA could result in a more complex regulatory landscape for businesses operating across different jurisdictions. Canadian organizations that are also subject to other privacy laws, such as the European Union’s General Data Protection Regulation (GDPR), might need to navigate multiple sets of privacy regulations, which could be challenging and resource-intensive.
Potential impact on innovation: Strict privacy regulations may hinder innovation in some sectors, such as artificial intelligence and data analytics, by limiting the ability of businesses to collect, use, and share data. The algorithmic transparency requirements may also discourage the development and use of certain automated decision-making technologies due to the obligation to provide explanations.
Uncertainty in enforcement: The increased powers granted to the Office of the Privacy Commissioner (OPC) could lead to more aggressive enforcement actions, which might cause uncertainty among businesses. This may result in organizations adopting more conservative approaches to data collection and processing to avoid potential penalties and legal challenges.
Overreliance on consent: The CPPA emphasizes the importance of obtaining individuals’ consent for data collection, use, and disclosure. However, this approach might not always be effective in addressing privacy concerns, as consumers may not fully understand the implications of their consent or may experience consent fatigue.
Balancing privacy and public interest: In some cases, the interests of individual privacy may conflict with broader public interests, such as public safety or public health. Striking the right balance between these competing interests can be challenging, and the Digital Charter Implementation Act might not always provide clear guidance in such situations.
Despite these potential downsides, Bill C-11 aims to provide stronger privacy protections and more control for individuals over their personal data in the digital age. The implementation of this legislation will require ongoing monitoring and evaluation to address any unintended consequences and ensure that the intended goals are achieved.

Although this is still a continuation of analyzing C-27, we’re indeed getting substantive analysis if that was the topic! And quite easily too considering I wrote the briefest of prompts.

It still mentions ‘C-11’ so it seems to be quite confused. It’s not the first time I’ve seen this happen so it’s indicative of how fragile it can be.

In this aspect GPT-4 does not improve upon GPT-3.5, nonetheless, the level of summarization performance is encouraging for our intended analysis.

Moving on to the main show, I directly copy-pasted the entire text of the summary section of C-11 and asked for it to identify any logical inconsistencies, to make sure it understood correctly. (I find that GPT-4 can’t convincingly BS beyond more than 1 layer of abstraction so a multitude of possible tricks could be used here)

Reading along with the actual text is highly recommended if your curious about the nuances that it gets right and those that it does not, which are too many to elaborate on in detail. The Bill Summary is very accessible and written in simpler language, without any French.

Upon reviewing the provided bill summary, no apparent logical inconsistencies are found. The summary describes amendments to the Broadcasting Act that aim to address various aspects of the Canadian broadcasting system, including online undertakings, diversity, accessibility, regulation, and enforcement. The amendments appear coherent and consistent with the overall objective of modernizing the Broadcasting Act.
The summary outlines the following key points:
Adding online undertakings as a distinct class of broadcasting undertakings.
Excluding user-uploaded content on social media services from the Act, unless prescribed by regulation.
Updating the broadcasting policy for Canada to serve diverse communities, including Indigenous persons and persons with disabilities.
Enhancing the vitality of English and French linguistic minority communities in Canada.
Specifying the ways the Canadian Radio-television and Telecommunications Commission (CRTC) must regulate and supervise the Canadian broadcasting system.
Amending the procedure for the issuance of policy directions by the Governor in Council to the Commission.
Replacing the Commission’s power to impose conditions on a license with a power to make orders.
Providing the Commission with the power to require expenditures to support the Canadian broadcasting system.
Authorizing the Commission to provide information to the Minister, the Chief Statistician of Canada, and the Commissioner of Competition.
Amending the procedure by which the Governor in Council may set aside a decision of the Commission or refer it back for reconsideration and hearing.
Specifying that a person shall not carry on a broadcasting undertaking without a license or exemption.
Harmonizing punishments for offences under the Broadcasting Act and clarifying the due diligence defence.
Allowing for the imposition of administrative monetary penalties for violations of certain provisions.
These points seem to be consistent with the overall goal of modernizing the Broadcasting Act and addressing the contemporary challenges and opportunities in the Canadian broadcasting landscape.

The points are broadly correct as far as I understand, and it’s a decent reduction in word length from the original, mostly from cutting out the sub-points, but not nearly as much as I expected. This points to the original summary being well written enough that not much more can be cut.

Interestingly, it does not give the points with subpoints any more length for explanation, such as the fifth point, which in the actual bill has 7 subpoints:

“(e) specify that the Canadian Radio-television and Telecommunications Commission (the “Commission”) must regulate and supervise the Canadian broadcasting system in a manner that
(i) takes into account the different characteristics of English, French and Indigenous language broadcasting and the different conditions under which broadcasting undertakings that provide English, French or Indigenous language programming operate,
(ii) takes into account, among other things, the nature and diversity of the services provided by broadcasting undertakings,
(iii) ensures that any broadcasting undertaking that cannot make maximum or predominant use of Canadian creative and other human resources in the creation, production and presentation of programming contributes to those Canadian resources in an equitable manner,
(iv) facilitates the provision to Canadians of Canadian programs in both official languages, including those created and produced by official language minority communities in Canada, as well as Canadian programs in Indigenous languages,
(v) facilitates the provision of programs that are accessible without barriers to persons with disabilities,
(vi) facilitates the provision to Canadians of programs created and produced by members of Black or other racialized communities, and
(vii) takes into account the variety of broadcasting undertakings to which the Act applies and avoids imposing obligations on any class of broadcasting undertakings if that imposition will not contribute in a material manner to the implementation of the broadcasting policy;”

If you had just read GPT-4′s summary you would might not have guessed how much importance the bill drafters attached to this part specifically.

On the other hand, how many folks would actually read it instead of skimming it?

So although it might be different from how a trained lawyer would summarize the text, it doesn’t seem altogether inferior for the purposes of the casual reader.

As it did not identify any logical inconsistencies I tried to prompt in an alternative way with:

Are there any loopholes, confusing wording, or any other type of inconsistency?

Based on the provided bill summary, it is difficult to identify specific loopholes, confusing wording, or inconsistencies without examining the full text of the bill and its specific provisions. The summary appears to be a coherent and comprehensive overview of the proposed amendments to the Broadcasting Act.
However, in any piece of legislation, there could be room for improvement or clarification. Here are a few potential areas where the summary may benefit from further examination or elaboration:
Exclusion of user-uploaded content on social media services: The summary states that the Act does not apply to programs uploaded to an online undertaking that provides a social media service by a user of the service, unless the programs are prescribed by regulation. The scope and criteria for such prescriptions could be further clarified to avoid potential ambiguity.
Definition of “online undertakings”: The summary mentions adding “online undertakings” as a distinct class of broadcasting undertakings. The definition and scope of this term could be further clarified to prevent confusion or misinterpretation.
Supporting the Canadian broadcasting system: The summary mentions that the Commission will have the power to require persons carrying on broadcasting undertakings to make expenditures to support the Canadian broadcasting system. The criteria and methods for determining such expenditures might need further clarification to ensure fairness and transparency.
Administrative monetary penalties: The summary allows for the imposition of administrative monetary penalties for violations of certain provisions. The specific provisions subject to these penalties and the process for determining the penalty amounts could be further elaborated.
It is essential to remember that this analysis is based solely on the summary provided. To fully understand the scope, language, and potential issues within the bill, a thorough examination of the complete text of the legislation is necessary.

It begins sounding like boiler-plate but there are some interesting points raised, which correspond to what’s later expanded upon in the main body of the bill without having read it, so GPT-4′s ‘judgement’ does seem to have some correlation with human ‘judgement’.

Though how much so would depend heavily on personal views or perhaps legal philosophy of how detailed laws need to be.

Moving on to the main text, my method was copy-pasting each part as input prompt, the first part begins immediately after the Bill Summary, up until Subparagraph 3(1)‍(d)‍(iii.7), and GPT-4 provided what it thinks of the possible inconsistencies in each. This will be repeated for each subsequent part. With the overall conclusions being made at the end.

The reason for this approach is because simply summarizing the bill in each part seemed superfluous when there’s already a summary of the overall bill that is accessible enough for most readers. Plus it makes the wording choices more interesting to analyze.

Upon reviewing the first part of the full text of the bill, the following potential issues could be considered for further clarification or elaboration:
Definition of “social media service”: The text provides exclusion criteria for social media services and user-uploaded content. However, the term “social media service” is not explicitly defined. Providing a clear definition of “social media service” could help ensure consistent interpretation and application of the provisions.
Criteria for exemptions: The bill outlines several exemptions related to online undertakings and transmissions of programs over the Internet. Further clarification of the criteria for these exemptions could help ensure that they are applied consistently and fairly.
Addressing potential overlap or conflicts: The bill introduces new definitions and amends existing ones. A more in-depth analysis of the relationships and potential overlaps or conflicts among these definitions might be necessary to avoid unintended consequences or confusion.
Roles and responsibilities of online undertakings: While the bill establishes online undertakings as a distinct class of broadcasting undertakings, further clarification of their roles and responsibilities within the Canadian broadcasting system would help ensure a consistent application of the legislation.
It’s important to note that any potential issues or areas of improvement identified here are based on a review of the first part of the full text of the bill. A more comprehensive analysis would require examination of the entire bill text.

The bit about ‘social media service’ not being explicitly defined is remarkably spot on. The bill does indeed continue to use this specific term, a dozen times total throughout, without actually defining it!

‘Addressing potential overlap or conflicts’ is also a solid insight as the bill continues to be quite obtuse in this regard, though perhaps GPT-4 just guessed this.

To get an idea of how wordy the main text is, an amendment to a single subparagraph in the actual bill, albeit the lengthiest in the first part, is as follows:

Subparagraph 3(1)‍(d)‍(iii) of the Act is replaced by the following:
(iii) through its programming and the employment opportunities arising out of its operations, serve the needs and interests of all Canadians — including Canadians from racialized communities and Canadians of diverse ethnocultural backgrounds, socio-economic statuses, abilities and disabilities, sexual orientations, gender identities and expressions, and ages — and reflect their circumstances and aspirations, including equal rights, the linguistic duality and multicultural and multiracial nature of Canadian society and the special place of Indigenous peoples within that society,
(iii.‍1) provide opportunities to Indigenous persons to produce programming in Indigenous languages, English or French, or in any combination of them, and to carry on broadcasting undertakings,
(iii.‍11) provide opportunities to Black and other racialized persons in Canada by taking into account their specific needs and interests, namely, by supporting the production and broadcasting of original programs by and for Black and other racialized communities,
(iii.‍2) support the production and broadcasting of original French language programs,
(iii.‍3) enhance the vitality of official language minority communities in Canada and support and assist their development by taking into account their specific needs and interests, including through supporting the production and broadcasting of original programs by and for those communities,
(iii.‍4) support community broadcasting that reflects both the diversity of the communities being served, including with respect to the languages in use within those communities and to their ethnocultural and Indigenous composition, and the high engagement and involvement in community broadcasting by members of those communities, including with respect to matters of public concern,
(iii.‍5) ensure that Canadian independent broadcasting undertakings continue to be able to play a vital role within that system,
(iii.‍6) support the production and broadcasting of programs in a diversity of languages that reflect racialized communities and the diversity of the ethnocultural composition of Canadian society, including through broadcasting undertakings that are carried on by Canadians from racialized communities and diverse ethnocultural backgrounds,
(iii.‍7) provide opportunities to Canadians from racialized communities and diverse ethnocultural backgrounds to produce and broadcast programs by and for those communities,

And there are quite a few of these later on too. Most normal readers would likely have stopped here if they bothered reading past the Bill Summary, so there’s huge value in being able to quickly condense all the legalese into a more readable format.

For comparison here is how GPT-3.5 interprets the same section:

As an AI language model, my goal is to provide an analysis of the text in a neutral and unbiased manner. I do not hold any political views or affiliations.
Bill C-11 is a lengthy document that proposes amendments to the Broadcasting Act and other related Acts. It aims to modernize the regulatory framework for broadcasting in Canada and address issues related to the digital age. However, there are several inconsistencies, loopholes, confusing wording, and logic errors in the text that could be problematic. Below are some examples:
Definition of “broadcasting” The definition of “broadcasting” in subsection 2(1) of the Broadcasting Act is ambiguous and confusing. It includes any transmission of programs by radio waves or other means of telecommunication for reception by the public but excludes any such transmission made solely for performance or display in a public place. This definition could be interpreted in various ways, and it is not clear what constitutes a “public place.” The term “broadcasting” also includes on-demand transmissions, which could create confusion regarding the scope of regulation.
Exclusion of social media services from the definition of “broadcasting undertaking” Bill C-11 excludes persons who use a social media service to upload programs for transmission over the internet from the definition of “broadcasting undertaking.” This exclusion only applies if the person is not the provider of the service, the provider’s affiliate, or the agent or mandatary of either of them. This exclusion creates a loophole that could allow social media companies to avoid regulatory oversight, even if they provide broadcasting services to Canadians.
Programming Control The definition of “programming control” in subsection 2(1) of the Broadcasting Act is not consistent with the definition used in other sections of the Act. It defines programming control as “control over the selection of programs for transmission” but excludes “control over the selection of a programming service for retransmission.” This definition is unclear and could lead to confusion regarding the scope of regulation.
Objectives of the broadcasting policy The amendments to paragraph 3(1)(d) of the Broadcasting Act include several objectives related to promoting Canadian expression, diversity, and inclusion. While these objectives are laudable, they are not clearly defined and could be interpreted in different ways. The Act could benefit from more precise language that specifies how these objectives will be achieved.
In conclusion, Bill C-11 contains several inconsistencies, loopholes, confusing wording, and logic errors that could be problematic. These issues could undermine the effectiveness of the proposed amendments and create confusion regarding the scope of regulation. It may be necessary to clarify and refine the language of the Act to address these concerns.

Immediately noticeable is the different approach of GPT-3.5, it really gets down to the weeds without bothering to mention, or consider, that these issues could be addressed in the remaining text of the bill. Although each point seems well reasoned, in context it seems somewhat ‘dumber’ than GPT-4.

Though they do both immediately, and correctly, identify ‘social media services’ as an ambiguous, not defined, term.

There’s a lot more that could be said but due to length, the post will be continued in Part 2, depending on demand, so comment below if you want to read more.