Hacker News

points by andix 2 days ago | 0 comments

Can AI also fully legalize racial profiling?

Some criminal behaviors are somewhat correlated with skin color, age or gender. It would be extremely hard to train an AI to ignore those correlations. But would it even matter? Nobody knows why the AI flagged suspicious behavior, it usually can't tell even if you are able to ask follow up questions. So nobody could prove racial profiling.

[–] throwaway-medic 2 days ago

I work on software that Dr and Nurses use. The company I work for trains its own models to provide automated insight on patient history, etc. I've been pretty concerned about these kinds of problems you describe. I sat down with one of the data scientists and talked about biases a while ago. She walked me through the (high level) process they go through to try and detect common biases in the model before publishing.

For example, it is a common belief within the medical community that Black people have higher pain tolerance, and thus don't need as much or as strong of pain killers. People in pain sitting in beds in the ER will be delayed or refused pain killers because of this. This belief has been proven false, but it shows up in papers, notes, etc. and nurses and doctors still will treat their patients as if it's true.

This is just one example of many many things they test for. When training the model the data-science team actually tries to see if they can produce output that exhibit those identified biases/problems and block it from ever been used in production if they do.

I've had people internally tell me how frustrated they are that I or other people aren't letting them get their new-fangled AI-powered feature in front of customers fast enough... Without a lot of discipline and guardrails people would be using ChatGPT or Claude APIs instead.

My point is: I highly doubt that kind of care is being taken in many industries/companies. It's very concerning to me.

[–] GuB-42 2 days ago

> For example, it is a common belief within the medical community that Black people have higher pain tolerance, and thus don't need as much or as strong of pain killers. People in pain sitting in beds in the ER will be delayed or refused pain killers because of this. This belief has been proven false, but it shows up in papers, notes, etc. and nurses and doctors still will treat their patients as if it's true.

Nurses and doctors don't refuse giving painkillers because they enjoy seeing their patients suffer, painkillers come with side effects. So they are trying to find a balance between preventing suffering and long term health. We can look at the wrong idea that black people have a higher pain tolerance as causing black patients unnecessary suffering, but you can also see it as protecting the long term health of black patients more. That's the same thing, just seen from a different angle.

But now that we know of this bias, we can of course try to fix it, but how. Now that we also have data: do black people have less painkiller-related problems? If they do, maybe we should give the black treatment to others, if they don't maybe they could get some more.

[–] steveBK123 2 days ago

You don't even need there to be any kernel of reality statistically.

If a model is trained on data that shows some bias due to prior human decisions, it would continue to follow those biases while being a lot harder to prove/sue. There were examples of this with AI assisted resume screening & criminal sentencing for example.

[–] preg_match 2 days ago

Yes, AI like organizations are very good for laundering accountability. Even before AI you could just hire Organization X or Consultancy Y and they can be as racist as they want, and boom you're just laundered all your accountability. Now go have fun, do whatever, who cares.

It's the same reason why Flock is so popular. It's just cameras, who cares right? Well Flock is popular for the organization, it's an accountability sink. The government can't just stand up a bunch of cameras, people will complain. Ahh, but Flock can. The entire economic value of Flock is purely the accountability laundering, the technology is just a footnote.

Really, many organizations in the US boil down to this. Producing value is hard, producing fake value is easy.

[–] derwiki 2 days ago

In regulated industries like money lending, ML outputs have to be explainable (and explained for an ECOA case). I’m not an ML expert by any stretch, but IIUC this is why we end up using decision tree models.

[–] smallmancontrov 2 days ago

On the contrary, you can feed the model counterfactuals and see what it does. Edit the person to be white then feed the image back into the model and see if its opinion changes.

[–] theonemind 2 days ago

Maybe. I'm not so sure. If you change skin color, it might identify race by other aspects. If you stage the same situation with an entirely different person or otherwise digitally edit it, it might detect in some way that it's not a real situation or an altered image and respond based on that. They're black boxes.

I remember someone talking about AI detecting lung cancer from MRIs, and they discovered that it was assigning higher probabilities based on the age of the machine that took it (or maybe fMRI, and maybe it wasn't lung cancer, I'm forgetting the details). So those communities were poorer and had worse health and a higher incidence, or they'd have had the newer machine. You never really know how it's deciding anything (though in that case they did figure it out, it might not be impossible, but in the end, you probably end up figuring it out at a statistical level--I suspect the counter-factuals wouldn't be reliable), so you're introducing variables attempting to make a counter-factual, which makes me think that in the end, it'll come down to statistical analysis from real data/people/situations. Which is after you've let it loose doing the thing, unless you run it in parallel for a while before relying on it at all and statistically analyze that before relying on it. I'm not sure you can isolate a variable sufficiently except by statistical means on real data, because it looks at everything, and it might find a correlate to your correlate, and a correlate to that correlate, and reach the same conclusion based on your counter-factual not being able to sufficiently hide what it was looking at

I don't know, random thoughts

[–] the_doctah 2 days ago

>Some criminal behaviors are somewhat correlated with skin color, age or gender.

Well, yes, because so is crime.

[–] andix 2 days ago

potato, potahto

[–] ToucanLoucan 2 days ago

Except crime occurs in pretty constant amounts over large populations. What really sets minority neighborhoods apart is far more aggressive policing, which finds more crime to be prosecuted for. Which is then used as both evidence backing racist claims by politicians, and reason to increase patrols further, which then find more crime, and so it continues.

It's also worth noting that numerous crimes, the most egregious examples probably being cocaine/crack and marijuana, are explicitly racialized in their enforcement. Cocaine in rich white suburbs is a health issue. Crack in the ghetto is a crime wave. It's literally the same substance, the only difference is how it looks and who is taking it. (and if it was possibly supplied by the CIA...)

Meanwhile white offenders of pot abuse get off with a few months here and there, and black people caught with it frequently, you know, fucking die.

[–] andix 2 days ago

This is not true in practical terms. Crime is highly dependent on the culture and the living conditions of a group of people. For example many immigrants are not allowed to buy guns and therefore commit more violent acts with knifes. Which creates a clear correlation between ethnicity and their preferred weapons. Their choice of weapon is not based on their ethnicity, but it still correlates.

[–] Windchaser 2 days ago

Yeah, I remember reading that a study found that white and black youth used marijuana at roughly equal rates, but black youth were about 4x as likely to be arrested for it. Due to differences in the # of stops, then differences in searches, then differences in actual arrests, with white youth getting more lenient treatment at each step.

And then this just reinforces the idea: "well, look at the crime statistics, black kids are more crime-prone, right"

[–] ToucanLoucan 2 days ago

And to be clear, I was IN this. We got busted for pot YEARS ago, but again, we're white, middle class. We confirmed what we took, we confirmed it was all gone, and I never saw so much as a fine. Cops asked to look around my house, I said sure thing help yourselves, they looked through maybe 1/3 of my home, and then left. And I've never heard a thing about it since then.

The fact that this is the same kind of life event that can literally, without any hyperbole, cost a black person their life, fucks with me sometimes.

[–] the_doctah 2 days ago

Now do firearm related crimes.

Anecdotes fall apart when we don't cherry-pick.

[–] Windchaser 2 days ago

Ancedotes are personal stories, like "oh, my dad was down at the post office the other day, and he saw..."

This is not a personal story, not an anecdote. I'm reporting on a study that looked at incarceration rates for drug use vs the drug use itself.

Why is that important? Because the study itself is solid, and it shows that there is some sort of bias between police/policing and african-american communities.

Sure, maybe that bias doesn't show up in other areas like firearm-related crimes. Or: maybe it does, and we haven't yet figured out how. Or, maybe it plays an indirect role, further eroding the relationship between the AA communities and the police, gradually building up a mutual dynamic that subjects black people to an unhealthy chronic level of stress and despair and harassment while also "justifying" the prejudicial activities of the cops.

There are whole books written about this. But at a minimum, I can't imagine just brushing over a 4x higher arrest rate for the same level of crime. That's pretty damn big.

[–] the_doctah 2 days ago

You're literally just describing a chicken-and-egg problem. Curious what your brain thinks is the reason African-American communities are policed more intensely. Surely every police organization in the United States is just racist.

[–] scheme271 2 days ago

So there is data on some policing in NYC. Because the police need to record stop and frisk encounters, it turns out that black populations get stopped and searched at 2-3 times the rate that other racial groups get stopped and frisked (whites got 10% of the encounters despite being something like 33% of the pop and black+latinos got 90% of the encounters). And it's not like the frisks turned up anything, with the vast majority (90%) of the frisks finding nothing. I believe there's similar data for police in other cities.

At the very least, this suggests that police in NYC and probably elsewhere were disproportionately singling out Black (and Latinos) for intrusive searches without evidence and on a pretextual basis.

[–] ToucanLoucan 2 days ago

I mean their officers keep getting caught posting racist shit, participating in racist chats, police unions regularly donate to republican politicians, and they murder black people so often that the phrase “driving/walking while black” is common slang for when the aforementioned African Americans get gunned down for every day misdemeanors.

So, every police institution? I’m sure there are exceptions, but there’s enough of a preponderance to safely assume it and the data bears that out.

[–] yieldcrv 2 days ago

it's all related

most of the violent crimes are from people with prior crimes, but the prior crimes themselves weren't necessarily violent, but just enough to marginalize that person from ability to participate in society with steady jobs

such as the possession of now-legalized substances, and the possession stop itself was based on flimsy pretexts that could have been fought, but it made more sense to take a plea-deal for that person or they were not aware of how to not engage with police and damned themselves without a lawyer present

[–] kbrannigan 2 days ago

What is a criminal? What is crime? In the USA, it was a crime for an enslaved Black American to learn to read and write? Attempting to escape was also a crime.

What about the crimes happening in company boardrooms rather than on the street corners?

People often call crime, individual offenses and think of certain racial groups, while ignoring real crimes committed by corporations led by other racial groups.

The crazy thing is that the greatest harms are caused by people who are not target of surveillance technology .

[–] ToucanLoucan 2 days ago

It will never not blow my mind that wage theft, the act of employers by one way or another fudging compensation to employees, eclipses every other form of larceny combined, and authorities do basically nothing about it.

If you work at a store and you take $100 from the register, you can be detained and jailed. Maybe prosecuted depending where you are.

If you work at a store and short five workers $20 each on their checks, and pocket it, they all have to individually pursue it as a civil matter, which itself costs money and time they probably don't have. It's fucking wild behavior.

I have to disagree though, it's not that crazy. Crime has always been a social construct much as that triggers a certain kind of person to hear, it just is. We decide the bounds of "good" behavior as a society and then determine punishments for those who transgress but at the same time, the law enforcement and justice systems are themselves systems made of people, and therefore inherit their biases. There have always been crooked cops, crooked judges, crooked prosecutors, etc. Like this isn't revolutionary shit I'm talking about, this is a standard plot point in movies going back to before movies had sound.

[–] the_doctah 2 days ago

Literally every single thing you said needs some kind of supporting evidence because it's all nonsense.

[–] ToucanLoucan 2 days ago

Google "study racialized policing" and you will find thousands of articles, going back decades, from all over America. This is not new information.

Oh, unless you mean the CIA thing. Then google "Contra CIA bloods crips" and you'll find everything on that.

[–] nonethewiser 2 days ago

Have you ever seen those AI videos where countries are competing in a farcical olympic event? Say, longest cold shower or something. And the Indian will just quit and fall to the ground and yell "I NEED CURRY!!!!!!!!"