The important thing for me is the level of risk this represents in healthcare. If systems are built to be dependent on external AI models, then every update becomes a major hazard since it's impossible for even experienced staff to anticipate how changes in the model might impact the system, and even experienced staff become complacent. Complacency leads to lower quality standards, under-spec'd solutions and in healthcare that inevitably leads to patient harms which could and should have been avoided.

I'd like to think that they'd be running updates in a sandbox environment, to test and revalidate model behaviour before letting it loose on a live system. But even when that's a documented requirement, it's not always followed rigorously. And even testing might not reveal all the hidden quirks, unless done thoroughly.

For this reason, I firmly believe that as patient partners we should be pushing for more patient involvement at every stage of every AI implementation and beyond into regular ongoing monitoring of AI systems. We need to be persistent in asking the awkward questions and insisting on rigorous checks, testing and validation at every point - not just when models update but also against model drift. We should also question inconsistencies that might indicate deeper issues, and insist on explainable rather than black box AI models to facilitate safety auditing.

But to do that, we need to be pushing to be involved more widely, more deeply and for the long haul.