In 2016, Arkansas stopped asking nurses how many hours of home care a person needed and started asking an algorithm. Tammy Dobbs, who has cerebral palsy and relies on an aide to get out of bed, eat, and use the bathroom, watched her weekly allocation drop from 56 hours of care to 32. She had not changed. Her needs had not changed. A formula had simply produced a smaller number, and nobody in the room could tell her why.
That story, reported by The Verge and documented by the Center for Democracy and Technology, is not a cautionary tale from the distant past. It is the clearest picture we have of what happens when a system that hands out care gets replaced by a model nobody can read. And a version of it is arriving in disability services across North America right now.
What the algorithms are already doing
Automated tools are no longer a future worry in disability services. They are running today, and they show up in the three places that matter most to anyone who depends on public support.
No spam. No inspiration porn.
Our best writing for adults with disabilities, weekly and free.
Get the newsletterThe first is eligibility: whether you qualify for a program or a level of funding, increasingly scored by weighing your file against patterns in old data. The second is service hours: how much personal care, respite, or home support you get, set by an assessment tool instead of a clinician’s judgment. The third is risk classification, which quietly sorts people into tiers that shape how much scrutiny and service they receive.
When Arkansas moved to a new assessment tool called ARIA in 2019, built by the consulting firm Optum, the Benefits Tech Advocacy Hub records that the redesigned system left over a quarter of the people in the state’s program determined wholly ineligible for care, with many of those still eligible capped by a budget limit and a time-task tool. That is not a rounding error. That is thousands of people losing care because a model decided they no longer fit.
The part that should worry you
The problem is not that these tools make mistakes. Every system makes mistakes. The problem is that they are often built so the mistakes cannot be seen, questioned, or traced.
When a nurse cut your hours, you could ask the nurse why. When the algorithm did it in Arkansas, the answer was some version of “that is what the system calculated.” The weighting was proprietary. The training data was not disclosed. And when the litigation finally pried the system open, it turned out a coding error had been quietly cutting hours for people with cerebral palsy and diabetes. The person who built the tool blamed the officials who ran it. The officials blamed the people who coded it. Tammy Dobbs lost her care while they argued.
This is what is at stake for self-determination, which is the whole reason disability services exist. Self-determination means you hold authority over the decisions that shape your life. A score you cannot see, cannot question, and cannot meaningfully challenge takes that authority and moves it somewhere you cannot reach.
The Canadian picture
Canada has not had an Arkansas-scale failure that anyone has documented the same way. But the conditions are here. Roughly 8 million Canadians aged 15 and older, about 27 per cent of the population, have one or more disabilities, according to the 2022 Canadian Survey on Disability. Provincial disability programs assess eligibility and funding using exactly the kind of criteria that get automated, and governments everywhere are under pressure to process more files with fewer people.
The law has not caught up. PIPEDA, the federal private-sector privacy law, has no express provisions on automated decision-making. The Privacy Commissioner of Canada has recommended that people subject to an automated decision get a right to a meaningful explanation and a right to contest it. Parliament had a chance to write that into law through Bill C-27 and its Consumer Privacy Protection Act, which would have required organizations to explain automated decisions that significantly affect someone. The bill died on the Order Paper in January 2025. The gap it was meant to close is still open.
The practical effect: if an automated tool touches your benefits today, your protection depends heavily on which province you live in and which program you are dealing with. That is a thin guarantee to rest a person’s care on.
What you can ask
This section is general information, not legal advice. You do not need to understand machine learning to follow what is happening with your file. The process generally allows a person to ask questions and put them in writing. When a decision about eligibility, hours, or classification comes back, a person can ask the agency or caseworker directly:
- Was any part of this decision made or scored by an automated tool? Ask plainly. The answer should not be a secret.
- What factors went into the score, and how was my information weighted? You are entitled to understand the basis of a decision that affects you.
- Did a person review this before it was finalized, or did the system decide on its own?
- What is the appeal or reconsideration process, and what is the deadline? Almost every benefit decision can be appealed. Find the channel and use it before the clock runs out.
- Can I see the assessment in writing, including the inputs you used? A written request creates a record.
Keeping copies helps. Dates, names, and what a person was told all build a record. If hours or eligibility change without a clear reason, that pattern is exactly what the successful challenges in Arkansas were built on. The people there did not win by reading the code. They won by documenting the harm and bringing it to legal aid and elected officials.
Where to take a problem
If you believe an automated decision has affected your benefits unfairly, the route is the official one. Use the program’s internal appeal or reconsideration process first, within the deadline. Community legal clinics and provincial legal aid offices handle benefit appeals and often take them at no cost. Your provincial privacy commissioner, or the federal Privacy Commissioner where a federal program is involved, can look at how your personal information was used in an automated decision. These channels exist so that a person, not a model, has the final say.
The takeaway
Automation is not the enemy here. A transparent, regularly audited tool could in principle spread support more consistently than a stretched human system ever could. The trouble is the version that keeps showing up: opaque, unexplained, and built so the people it decides for cannot see how it works or push back when it is wrong. Until the law guarantees a real right to explanation and review, the protection is the one you build yourself. It is worth asking whether a machine touched your file, getting the answer in writing, and using the appeal process for anything that does not add up. The number on the page is not the final word; you are.
Sources
- Arkansas Medicaid Home and Community Based Services Hours Cuts, Benefits Tech Advocacy Hub
- What Happens When Computer Programs Automatically Cut Benefits That People With Disabilities Rely on to Survive, Center for Democracy and Technology
- Arkansas’s Opaque Algorithm to Allocate Health Care Excessively Cut Down Hours for Beneficiaries, AI Incident Database
- The Algorithms Too Few People Are Talking About, Human Rights Watch
- Policy Proposals for PIPEDA Reform to Address Artificial Intelligence, Office of the Privacy Commissioner of Canada
- Privacy and Artificial Intelligence in Canada, HillNotes (Library of Parliament)
- Canadian Survey on Disability, 2017 to 2022, Statistics Canada
