Te Kete o Karaitiana Taiuru (Blog)

The New Zealand media continue to report on AI harms from a western, academic, middle-class perspective while continuing to make invisible communities silent and ignored. This is a te ao Māori perspective of the recent MSD automated Decision Making law that was passed, delving deep into community harms for Māori.

I have spent years documenting what I call digital colonialism: the imposition of technological systems on Māori, without consent, without co-design, and without regard for the asymmetric harm those systems cause.

Under parliamentary urgency a procedure explicitly used to prevent public submissions, select committee scrutiny, and Waitangi Tribunal analysis the coalition government passed the Social Security (Modernisation) Amendment Bill on 29 May 2026, authorising the Ministry of Social Development (MSD) to deploy automated systems to make consequential decisions about welfare entitlements, the bill passed in a day.

The government’s Budget projects it will reduce benefit payments to New Zealande beneficiaries by $54.5 million over four years due to an update to the expected reduction in receipt of Temporary Additional Support. We either have a massive fraud issue, or many of our most vulnerable will be discriminated against while loosing money?

As a technologist, I’m not opposed to technology in public administration, but I am opposed to the deployment of automated systems that make life altering decisions about people without skilled human oversight, without checks against bias, and without accountability when those systems cause harm. In this article, I aim to set out what responsible deployment requires: who the human in the loop must be, what they must know, and what safeguards must be built in from the start.

MSD have followed the correct procedure and have created an Automated Decision-Making Standard ADM Standard), that many of the media have overlooked.

MSD’s ADM Standard contains some meaningful human oversight provisions, but they are largely reactive rather than proactive. The standard requires that any challenge or appeal of an automated decision must involve human decision-makers not another automated process and those reviewers must be empowered to correct decisions and look beyond the narrow specifics of the original automated determination. It also mandates that where bias or discrimination cannot be removed or sufficiently mitigated, substantial human involvement must be incorporated into the process at which point it would no longer qualify as an automated decision. Regular monitoring and three-yearly compliance reviews are required, and accountability is assigned to a named Business Owner. These may appear reasonable baseline safeguards.

However, the standard is light on proactive human involvement. There is no requirement for a human to be in the loop before a decision is made or at the point of decision itself. Human oversight only activates after the fact, through a complaints or appeals channel, and the standard explicitly defines an automated decision as one where “there is no substantial human involvement in making the decision.” For Māori and other vulnerable groups who may face compounding disadvantages, limited digital access, language barriers, distrust of institutions, the burden of initiating a challenge sits entirely with the affected person. The standard also places risk acceptance squarely with the Business Owner internally, with no requirement for independent external oversight, co-design with affected communities, or specific consideration of Māori rights under Te Tiriti. For decisions affecting welfare entitlements, that is a significant gap.

Another protection mechanism is the New Zealand the Algorithm Charter. I have also written and spoken about the lack of protections and the international peer review that found many issues with it. Recently, another critique was published from a Culturally And Linguistically Diverse CALD community (Ubuntu) perspective  highlighting again the shortcomings and need for government to consult widely across New Zealand and ensure there are community representatives, not just people on the books.

The Human Rights Commission released a public statement about their concerns has concerns about significant expansion of the use of ADM without proper public scrutiny and the extent to which automated systems can fairly reflect the complexity of people’s circumstances when making decisions, even when the situation appears straightforward. They too welcomed provisions for a human in the loop.

Denise Barlow has written stories about real life examples to the harms our most vulnerable people are discriminated on her LinkdeIn and recent blog The Room . When we read MSD’s AMD standards, then compare them to her stories more concerns are raised.

Don Tewhaiti has created an algorithm to prevent bias in algorithms called Objective Integrity Model (OIM) , yet to date there is no widespread up take of his work.

My 2026 paper Responsible AI in New Zealand Requires Māori Governance: Beyond Voluntary Compliance to Treaty Consistent Practice establishes that New Zealand’s existing AI governance instruments including: the Algorithm Charter  (2020), Privacy Act 2020, Digital Identity Services Trust Framework Act 2023 , and Public Service AI Framework (2025) formally comply with procedural requirements while continuing to shift power away from Māori communities. The Social Security (Modernisation) Amendment Bill  shows the same documented pattern.

 

The Colonial Pattern in Technology

To understand why automated welfare systems are dangerous for Māori, you need to understand the history. The Crown has repeatedly used administrative and institutional mechanisms to manage, suppress, and dispossess Māori, each time with confident assertions that the system was neutral, modern, and in Māori best interests.

The Tohunga Suppression Act 1907  criminalised traditional Māori healing and knowledge keeping, replacing them with state-sanctioned systems that Māori had no role in designing. The Native Schools Act  imposed English only education on Māori communities, severing te reo Māori from generations of children through an institutional mechanism that appeared bureaucratic and benign until its repeal in 1969. The Hunn Report of 1960  recommended the accelerated integration and urbanisation of Māori on the explicit assumption that Māori culture was a deficit to be managed out of existence, a policy that drove mass displacement of whānau from traditional communities.

The Waitangi Tribunal has documented, across decades of hearings, how these mechanisms caused cumulative, compounding harm: the Wai 262 Report on mātauranga Māori and intellectual property; Wai 2575  on Māori health; and the Oranga Tamariki urgent inquiry (Wai 2600)  on the removal of Māori children. The Royal Commission on Abuse in Care, which reported in 2023, found systemic, racially disproportionate abuse of Māori children and disabled people in state care. Māori children were taken from their families at vastly higher rates. They experienced harsher treatment, and the institutions responsible operated with minimal oversight, little accountability, and confident assertions that their practices were appropriate.

MSD’s own regulatory impact statement for recent welfare changes (2025) acknowledges that “Māori are disproportionately represented in the welfare system” and that proposals “may disproportionately negatively impact Māori and do not align with Te Tiriti o Waitangi/Treaty of Waitangi principles of active protection and equity.” Māori make up approximately 36% of all welfare recipients while comprising about 17% of the total population MSD Benefit Fact Sheets, 2025  and Welfare Expert Advisory Group, 2019 .

I see the same structure in the Social Security (Modernisation) Amendment Bill, a system designed without Māori consultation, operating on data that reflects historical discrimination, deployed at scale with cost-reduction targets, and shielded from scrutiny by urgency. The Abuse in Care Royal Commission found that state institutions with financial and administrative incentives to minimise costs consistently prioritised those interests over the welfare of the people in their care.

International Evidence

Australia’s Robodebt

Australia’s Robodebt scheme  generated automated debt demands against welfare recipients using flawed income-averaging methodology, many of them among the country’s poorest. Over $746 million AUD was wrongfully recovered from 381,000 individuals and later refunded. The Royal Commission found evidence of deaths associated with the scheme, including confirmed suicides, with the Commission stating it was “confident these were not the only tragedies of this kind.

“Ministers knew by late 2016 that it was unlawful , but they continued it, and when victims spoke out, officials unlawfully released their confidential welfare files to media to discredit them  .

Māori make up 36% of welfare recipients in New Zealand. A Robodebt-equivalent here would be racially disproportionate from day one, on data that already encodes decades of over-sanction and under-support of Māori.

The lesson from Robodebt is one the Abuse in Care Commission would recognise: once an automated system is deployed with a cost-reduction target, the institutional incentive is to protect the system, not correct it. The machine achieves the budget outcome while whistleblowers are silenced.

The Dutch Toeslagenaffaire

The Dutch childcare benefits scandal saw an automated algorithm target approximately 26,000 to 35,000 families  for alleged fraud. The Dutch Data Protection Authority confirmed the algorithm used nationality as a fraud proxy , disproportionately flagging non-Dutch migrants and minorities. Amnesty International, in its 2021 report “Xenophobic Machines”, described the system as discriminatory by design. Tens of thousands of families were financially destroyed and the entire Dutch cabinet resigned .

The mechanism was identical to what the Waitangi Tribunal has documented in Māori contexts: a system trained on historically biased data learns to reproduce that bias, applies it consistently at scale, and provides no meaningful pathway for affected people to understand or challenge what has happened to them.

A peer-reviewed 2023 analysis of predictive risk modelling in New Zealand’s care and protection system found that Māori over-representation “might be incidentally intensified by predictive risk models, leading to possible cycles of bias.

MSD’s own 2014 paper on a prior predictive tool acknowledged it “over-identified Māori children and under-identified Pākehā children.” The Dutch pattern is already present in New Zealand’s algorithmic history.

The EU AI Act

The European Union’s AI Act (Regulation (EU) 2024/1689) entered into force on 1 August 2024. AI Act (Regulation (EU) 2024/1689). It classifies AI used for welfare eligibility decisions as high-risk under Annex III, triggering mandatory requirements for human oversight, conformity assessments, transparency, decision logging, and the right to explanation. While most high-risk AI obligations were originally set to apply from August 2026, the Digital Omnibus amendments adopted in 2026 have pushed those obligations back to December 2027 for stand-alone high-risk systems. The point stands regardless of timeline: New Zealand is deploying, under urgency, a system the EU considers inherently dangerous with none of the safeguards the EU considers mandatory.

Why Human Oversight Is the Central Safeguard

Automated systems do not introduce new discrimination into welfare administration; they automate existing discrimination and deploy it faster and at greater scale than any human bureaucracy could. The Tohunga Suppression Act, the Native Schools Act, and the Hunn Report’s integration policy were administered by humans with discretion.

Human oversight is where institutional knowledge, cultural competency, tikanga, and conscience can be applied to a decision before it destroys someone’s or their whanau’s life. For many Māori, the welfare system has only been navigable through relationships with culturally competent advocates who understand the person, their whānau, and their context. Automation eliminates this and the person becomes their data record, while whakapapa, whānau circumstances, and lived reality disappear.

My Kaupapa Māori Framework for AI Agents (2026) establishes that AI governance in New Zealand “has been almost entirely derived from Western philosophical and technical traditions.” This structural incapacity means no automated welfare system can assess Māori wellbeing on terms Māori recognise whakapapa, mauri, mana, whānau obligations are not data fields. The framework provides the basis for Māori-centred human oversight requirements.

The Abuse in Care Commission found that state institutions consistently failed Māori and disabled people because oversight was inadequate, accountability was absent, and institutional incentives ran against acknowledging harm. Human oversight in automated welfare systems must be designed to avoid exactly this failure mode.

Automation bias is the tendency of human reviewers to defer to machine recommendations is well documented. The Abuse in Care Commission found that staff who raised concerns in state institutions were routinely silenced or marginalised. The same dynamic will apply here unless independence is structurally protected.

Required Skills

These are the minimum competencies for any person with decision authority over automated welfare determinations.

Skill	What It Requires |
Algorithmic literacy	Understand how models generate recommendations, what training data means, what error rates signify, and where models typically fail. You do not need to be a data scientist. You need to know which questions to ask.
Statistical reasoning	Read disaggregated outcome data, recognise disparate impact patterns, understand base rates and error types. Know the difference between correlation and causation. Te Tiriti competency
Māori cultural competency	Working knowledge of tikanga Māori, te ao Māori frameworks, and the structural barriers Māori face in health and welfare systems, including the legacy of the Tohunga Suppression Act, Native Schools Act, and Hunn Report integration policy. Understand the Crown’s active Te Tiriti obligations in technology deployment: tino rangatiratanga, active protection, partnership. Non-negotiable for anyone making decisions affecting Māori.
Trauma-informed practice	Recognise and respond to cumulative harm from state systems. The Abuse in Care Commission documented the devastating impact of institutionalised indifference. Do not replicate it.
Administrative law	Understand natural justice obligations, the right to reasons, and the legal standards governing reviewable decisions about fundamental entitlements. |
Bias identification	Detect proxy discrimination: where neutral-appearing variables such as postcode, benefit history, contact frequency, function as ethnic or disability proxies in model outputs. |
Decision documentation	Produce records detailed enough for internal review, external audit, Ombudsman scrutiny, or Waitangi Tribunal examination. Explain the reasoning, not just the outcome.
Māori data sovereignty literacy	Understand the [Waitangi Tribunal WAI 2522 (2023 https://waitangitribunal.govt.nz/inquiries/kaupapa-inquiries/wai-2522-data-sovereignty-inquiry/ definition of Māori data as taonga and the Crown’s resulting governance obligations. Able to apply. My six Te Tiriti-based AI principles https://www.taiuru.co.nz/ai-principles and identify when an automated recommendation violates them.

 

Conclusion

This is not the first time the state has deployed an institutional mechanism that harms Māori while claiming neutrality and efficiency. The Tohunga Suppression Act, the Native Schools Act, the Hunn Report, the child welfare systems documented by the Abuse in Care Commission all operated on the same logic: that the state knows best, that Māori input is unnecessary, and that the harm caused will be invisible or acceptable.

An automated welfare system makes decisions in milliseconds and applies its learned patterns to thousands of people simultaneously. The institutional incentive to correct errors as Robodebt demonstrated, is overwhelmed by the sheer volume and velocity of automated determinations.

The human in the loop is not a bureaucratic safeguard but the point at which tikanga, cultural competency, and conscience can interrupt a machine, but only if that human has the skills to recognise what the machine has gotten wrong, the authority to act on that recognition, and the institutional protection to do so without career consequences.

None of those conditions are currently met. This legislation creates the machine without building the human infrastructure that makes the machine safe. For Māori who are over-represented in the welfare system, whose data reflects generations of colonial policy, and whose tikanga cannot be encoded in decision trees, the stakes are not abstract. They are the same stakes that have always attended the Crown’s confident assertions that its systems are neutral, modern, and in Māori best interests. The Waitangi Tribunal’s WAI 2522 finding that Māori data is taonga, subject to Māori governance. That obligation should not be ignored.