cross-posted from: https://lemmy.sdf.org/post/61241446
The criticism and warnings are not new, but now they come from within China.
[…]
A recent government-funded study out of Sichuan University’s Law School published in the Journal of Sichuan Normal University has triggered a fresh wave of scrutiny regarding China’s social credit system. Authored by doctoral candidate Lin Jianxing, the paper cautions that pouring advanced artificial intelligence and predictive algorithms into social rating frameworks risks turning administrative tools into instruments for mass surveillance.
The core issue isn’t new, but the source is. Critics from Western nations have flagged privacy concerns for years. Now, internal legal scholars are pointing out the exact mechanical pitfalls from within the system’s own academic backyard.
[…]
When the social credit initiative first gained traction years ago, its stated goals sounded straightforward. Officials designed it to build market trust, crack down on corporate fraud, and help banks assess default risks for loans. Think of it as a corporate and financial reliability index, meant to keep bad actors out of public contracts and penalize contract breakers.
Over time, things shifted. Local governments began testing personal credit metrics that tracked minor infractions on public transit, school attendance, and everyday civic habits.
Lin’s research highlights a profound transition. By utilizing big data mining and predictive algorithms, administrative bodies can translate everyday online and offline choices into quantifiable scores. This process aims to forecast a person’s future actions before they happen. That predictive capability is where convenience crosses the line into surveillance.
[…]
When you hand regulatory decisions over to predictive algorithms, several distinct problems emerge.
- Opaque Logic: When an algorithm lowers a score, finding out why can feel impossible. Citizens face automated restrictions without knowing which specific data point triggered the penalty.
- Amplified Discrimination: Predictive models rely on historical data. If past enforcement favored certain demographics or penalized specific behaviors unfairly, the algorithm bakes those biases into future decisions.
- Massive Surveillance Footprint: Real-time tracking requires constant data collection. Facial recognition feeds, smartphone logs, and financial transactions must flow continuously into a centralized scoring apparatus.
[…]
Internal critiques published in state-funded journals show that policy debates inside the country are far from monolithic. Legal scholars and researchers are actively pushing back against the overreach of automated systems, warning that hyper-efficient administrative control carries a steep societal cost.
If algorithms become the ultimate arbiter of who gets a loan, who boards a train, or who qualifies for public services, trust stops being a social contract between people. It becomes a compliance score issued by a machine.
[…]
The article begins with the following two sentences:
Algorithms don’t just track behavior anymore. They predict it.
That’s an ironic LLM-ism if I ever saw one.
That doesn’t make the article wrong, of course. It just makes the person who “wrote” it a piece of worthless incompetent shit.
You’re right to push back – that excerpt does resemble LLM produced text.
You should have committed with an actual emdash: —
How do you recognize that from these sentences?
People who use AI a lot (like me) recognize it instantly. Recent studies find if you take 5 people who use AI a lot and have then grade human or AI output the majority vote is 99% accurate.
LLMs right now massively overuse “it doesn’t just X, it Y”. Just like how they overuse “you are right to push back”.
So I guess the excerpt raises suspicions even though it could just be the author writing that.
I see. Yeah, when I see the whole article, it’s mostly clear if it’s ai or not. But the above excerpt could just be an ai enthusiast praising it.
Let’s not forget about the other classics, like “checking how X (statement) ‘landed’” or whether “Y ‘hit’ (the way it was expected to)”. I swear I’ve never read or heard either of those turn of phrase in any other context, so I have no idea where the LLMs picked them up. Somebody must have fed them executive briefs or something.
Then there the tiresome constant use of similes. Something can’t just be. It has to be like something. “The similes were scattered throughout the LLM generated text, like pebbles on a beach.”
Inserting linebreaks to make every sentence it’s own paragraph.
Avoiding the use of commas and the word “and” like they were the plague. Even if that means making unnecessary new sentences.
Don’t forget the loudest tell: Quietly.
Oh yes. How could I have forgotten that one?
It’s not so much an LLM thing as it is a data/curation thing. Idk the specifics, but I do remember back when I used it, chatgpt occasionally presented me with multiple responses, and asked me to pick the one that I liked best. It’s like how Facebook/tiktok/whatever curate your feed by examining how much time you spend eg looking at one post vs another. There’s nothing inherent about LLMs that cause them to prioritize emdashes, thats all the result of extra massaging the operator has applied to the model.
This is also why I feel it’s important to not focus on perceived “LLMisms” as a way to detect slop. It’s extremely trivial to hide that, and the ones that don’t are low hanging fruit.
LLMs love advertising / speechwriting techniques. Wikipedia has a good list here: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
If you use AI regularly these things are very very obvious.
Now, internal legal scholars are pointing out the exact mechanical pitfalls from within the system’s own academic backyard.
Yeah no fucking way a human wrote that lmao
