Nature: 'A fairer way forward for AI (artificial intelligence) in health care', 2019, Nordling

Andy

Andy

Retired committee member
When data scientists in Chicago, Illinois, set out to test whether a machine-learning algorithm could predict how long people would stay in hospital, they thought that they were doing everyone a favour. Keeping people in hospital is expensive, and if managers knew which patients were most likely to be eligible for discharge, they could move them to the top of doctors’ priority lists to avoid unnecessary delays. It would be a win–win situation: the hospital would save money and people could leave as soon as possible.

Starting their work at the end of 2017, the scientists trained their algorithm on patient data from the University of Chicago academic hospital system. Taking data from the previous three years, they crunched the numbers to see what combination of factors best predicted length of stay. At first they only looked at clinical data. But when they expanded their analysis to other patient information, they discovered that one of the best predictors for length of stay was the person’s postal code. This was puzzling. What did the duration of a person’s stay in hospital have to do with where they lived?

As the researchers dug deeper, they became increasingly concerned. The postal codes that correlated to longer hospital stays were in poor and predominantly African American neighbourhoods. People from these areas stayed in hospitals longer than did those from more affluent, predominantly white areas. The reason for this disparity evaded the team. Perhaps people from the poorer areas were admitted with more severe conditions. Or perhaps they were less likely to be prescribed the drugs they needed.

The finding threw up an ethical conundrum. If optimizing hospital resources was the sole aim of their programme, people’s postal codes would clearly be a powerful predictor for length of hospital stay. But using them would, in practice, divert hospital resources away from poor, black people towards wealthy white people, exacerbating existing biases in the system.
Click to expand...
https://www.nature.com/articles/d41586-019-02872-2
 
Today, in people finding about how poverty is bad for your health, for numerous reasons that often compound with one another.

Although for the US the problem is massively aggravated by the broken system of private insurance meaning millions of people will completely avoid health care until they are in a dire state.

It's pretty interesting that it took an AI to flag this for humans to see the obvious they should have already known about. AI will bring out a lot of interesting blind spots like that in the next few years.
 
The thrust of the article seems to be that if the current system is so full of biases based on race and socio economic status, it can not be relied on to train the new AI systems. Using current practice to train AI systems will result in an AI system that entrenches the current biases.
 
BruceInOz said:
The thrust of the article seems to be that if the current system is so full of biases based on race and socio economic status, it can not be relied on to train the new AI systems. Using current practice to train AI systems will result in an AI system that entrenches the current biases.
Click to expand...
Wasn't there a chatbot that had been developed on public online interactions between people and it was found to be racist?
 
Andy said:
Wasn't there a chatbot that had been developed on public online interactions between people and it was found to be racist?
Click to expand...
I guess you mean this:
https://en.m.wikipedia.org/wiki/Tay_(bot)
Wikipedia said:
Tay was an artificial intelligencechatter bot that was originally released by Microsoft Corporation via Twitter on March 23, 2016; it caused subsequent controversy when the bot began to post inflammatory and offensive tweets through its Twitter account, causing Microsoft to shut down the service only 16 hours after its launch.[1] According to Microsoft, this was caused by trolls who "attacked" the service as the bot made replies based on its interactions with people on Twitter
Click to expand...
 
BruceInOz said:
The thrust of the article seems to be that if the current system is so full of biases based on race and socio economic status, it can not be relied on to train the new AI systems. Using current practice to train AI systems will result in an AI system that entrenches the current biases.
Click to expand...
Depends. There has been a trend in the past few years showing that machine learning systems do best when given the fewest assumptions, ideally none at all. So at least there is enormous pressure in the way research is showing much more useful results out of systems that specifically go out of their way to give the most freedom and the least constraints.

In some cases there have been experiments with systems learning with almost no supervision at all and coming up with far better solutions, many of which were shockingly creative.

This would be a significant problem if the trend were the other around. There is bias in all data so I don't think medical data would be significantly different. The main problem would be in missing data, as for example we ourselves are largely missing or incomplete, but biased AI systems do so poorly that I think those with fewer, or none, assumptions will precisely highlight the discrepancy that we have been screaming about for years.

Specifically, the kind of biased research that supports the psychosocial model of ME will no doubt be flagged as below garbage-tier and of no value whatsoever besides being perfect examples of how not to do research.
 
Last edited:
Andy said:
Wasn't there a chatbot that had been developed on public online interactions between people and it was found to be racist?
Click to expand...
Unmoderated data will always be lousy, doesn't matter if AIs are involved or now. People actually went out of their way to influence the chatbot.

The quality of medical data will never be perfect but should generally be relatively free of Nazi sentiments, various controversial opinions and hate speech. Personal files no doubt have some less than stellar comments but those are unlikely to be found anywhere but on a sheet of physical paper and never submitted to centralized systems.
 
You must log in or register to reply here.
Back
Top Bottom