AI: When models update, systems break

[Chris Johnston]

This is a somewhat technical article, but I encourage you to give it a read anyway:
When Claude changed, everything changed: Managing AI blast radius in production

The important thing for me is the level of risk this represents in healthcare. If systems are built to be dependent on external AI models, then every update becomes a major hazard since it's impossible for even experienced staff to anticipate how changes in the model might impact the system, and even experienced staff become complacent. Complacency leads to lower quality standards, under-spec'd solutions and in healthcare that inevitably leads to patient harms which could and should have been avoided.

I'd like to think that they'd be running updates in a sandbox environment, to test and revalidate model behaviour before letting it loose on a live system. But even when that's a documented requirement, it's not always followed rigorously. And even testing might not reveal all the hidden quirks, unless done thoroughly.

For this reason, I firmly believe that as patient partners we should be pushing for more patient involvement at every stage of every AI implementation and beyond into regular ongoing monitoring of AI systems. We need to be persistent in asking the awkward questions and insisting on rigorous checks, testing and validation at every point - not just when models update but also against model drift. We should also question inconsistencies that might indicate deeper issues, and insist on explainable rather than black box AI models to facilitate safety auditing.

But to do that, we need to be pushing to be involved more widely, more deeply and for the long haul.

[Debra Turnbull]

@Chris-Johnston
I found myself smiling as I was reading this - 'cause it's so true !!! Ignoring the technical jargon, the cause and effect is still relateble. When basic testing procedures get ignored - all hell breaks loose. Love the term "blast radius".

"...successful upgrades had trained us to believe those gaps were safe."

Complacency is a problem in deployment. Non-AI systems may be a little more stable when they go haywire, but at least they are contained. We just had a building full of scientist, analysts and managers running around: "the server has crashed!!" Power down, power back up, locate and block the last change made.

AI is a differenct beast. First - how do you detect if something has gone wrong? Second - how do you contain when it does go wrong? Third - how do you disable it without affecting your other systems? Some of these things do not have a simple "off" switch.

Ignoring my analytical comments - thanks for the giggle this morning !... and on a Monday, no less !!!