Research
Ontological Blindness: The Hidden Assumptions in Healthcare AI
The complex reality
Consider a widely deployed healthcare risk-prediction algorithm. To prevent bias, designers intentionally made the system “race-blind” by excluding race as an input. Instead, the system used past healthcare costs as a proxy for medical need, assuming that patients who spent more money were sicker.
The failure
Because marginalised patients historically spend less on healthcare due to systemic inequalities, the algorithm missed a crucial part of the picture. Its representation of the system was “ontologically blind.” It mistakenly assumed Black patients were healthier than they actually were, requiring them to be significantly sicker than White patients to reach the same risk score and receive the same care.
Traditional compliance tools missed this entirely. The software ran perfectly without technical bugs. The fracture was a catastrophic semantic mismatch between the engineering definition of “risk” and the real-world definition of “care.”
An Epistemic Engineering view
The architecture encoded a consequential assumption: healthcare cost could stand in for medical need. Removing race from the input did not remove inequity from the system’s reasoning.
Artificial Apperception makes consequential assumptions explicit within a living, machine-native system model. Within that model, claims like Healthcare Cost = Medical Need can be traced through an Assumption Graph, stakeholder perspectives can be compared in a Synthesis Matrix, and anti-requirements can be defined for outcomes the system must avoid. Linking those artefacts to evidence and decisions gives teams a reviewable basis for challenging the proxy and validating the design before deployment.