Il paradosso dell’automazione

Automation ‘But what’s happening?’ Flight 447 and the Jennifer unit: When human messiness protects us from computerised disaster – Messy: How to Be Creative and Resilient in a Tidy-Minded World – Tim Harford –

***

Punti chiave: L’automazione diffusa erode le nostre competenze – Il pilota automatico in aeronautica – I guai di un aereo troppo sicuro –  paradosso dell’automazione – multe in automatico, azzeramento capacità d’indagine e deresponsabilizzazione –  I giapponesi finiti nell’oceano con il gps – Il robot che più preoccupa non è quello che ci sostituisce nel lavoro ma nel giudizio – Meteorologi: prima prevedere, poi consultare il computer – I problemoni dell’auto che si guida da sé – Soluzioni al paradosso dell’automazione – Puo’ aiutare un computer che sbaglia più frequentemente – Una strada più incasinata ci rende guitadori più concentrati e prudenti –

***

Yellow highlight | Location: 2,654
Pierre-Cédric Bonin, thirty-two, was young and inexperienced. David Robert, thirty-seven, had more experience but he had recently become an Air France manager and no longer flew full-time. Captain Marc Dubois, fifty-eight, had experience aplenty but he’d been touring Rio with an off-duty flight attendant. It was later reported that he had only had an hour’s sleep.

Note:UNA CREW CON I SUOI DIFETTI

Yellow highlight | Location: 2,665
paradoxically, there is a risk to building a plane that protects pilots so assiduously from even the tiniest error. It means that when something challenging does occur, the pilots will have very little experience to draw on as they try to meet that challenge.

Note:GLI INCONVENIENTI DI UN AEREO TROPPO SICURO

Yellow highlight | Location: 2,717
the real source of the problem was the system that had done so much to keep A330s safe across fifteen years and millions of miles of flying: the fly-by-wire. Or more precisely, the problem was not the fly-by-wire system, but the fact that the pilots had grown to rely on that system…Aggravating this mode confusion was Bonin’s lack of experience in flying a plane without computer assistance. While he had spent many hours in the cockpit of the A330, most of those hours had been spent monitoring and adjusting the plane’s computers rather than directly flying the aircraft….

Note:FLY BY WIRE

Yellow highlight | Location: 2,734
This problem has a name: the paradox of automation…The better the automatic systems, the more out-of-practice human operators will be,…

Note:PARADOSSO DELL’AUTOMAZIONE

Yellow highlight | Location: 2,738
The psychologist James Reason, author of Human Error, wrote: ‘Manual control is a highly skilled activity, and skills need to be practised continuously in order to maintain them.

Note:SKILL E ABITUDINE

Yellow highlight | Location: 2,743
automatic systems accommodate incompetence

Note:INCOMPETENZE SMASCHERATE

Yellow highlight | Location: 2,745
even if operators are expert, automatic systems erode their skills

Note:EROSIONE

Yellow highlight | Location: 2,746
systems tend to fail either in unusual situations or in ways that produce unusual situations, requiring a particularly skilful human response.

Note:SITUAZIONI ECCEZIONALE

Yellow highlight | Location: 2,749
A customer service webpage may be able to handle routine complaints and requests, so that customer service staff are spared repetitive work and may do a better job for customers

Note:TECNOLOGIA SENZA PARADOSSI

Yellow highlight | Location: 2,787
Earl Wiener, a cult figure in aviation safety who died in 2013, coined what’s known as ‘Wiener’s Laws’ of aviation and human error. One of them was, ‘Digital devices tune out small errors while creating opportunities for large errors.’

Note:LA LEGGE DI WIENER

Yellow highlight | Location: 2,808
At 14 seconds after 8.08 p.m. on 20 December 2013, his car had been blocking a bus stop in Bradford, Yorkshire, and had been photographed by a camera mounted in a traffic enforcement van driving past. A computer had identified the licence plate… There was just one problem: Mr Hankins hadn’t been illegally parked at all. He had been stuck in traffic….Yellow highlight | Location: 2,813
There was just one problem: Mr Hankins hadn’t been illegally parked at all. He had been stuck in traffic….

Note:LA MULTA AUTOMATICA AL SIG HANKINS

Yellow highlight | Location: 2,814
In principle, such technology should not fall victim to the paradox of automation. It should free up humans to do more interesting and varied work – checking the anomalous cases

Note:AZZERATA LA CAPACITÀ D’INDAGINE

Yellow highlight | Location: 2,820
Google unveiled a neural network that could identify house numbers in photographs…what if Google improves its accuracy by a factor of a million?…

Note:GOOGLE CI GUARDA IN CASA

Yellow highlight | Location: 2,832
when someone says the computer made a mistake, we will assume they are wrong or lying.

Note:ERRORI DEL COMPUTER

Yellow highlight | Location: 2,832
What happens when private security guards throw you out of your local shopping centre because a computer has mistaken your face for that of a known shoplifter?

Note:LA FACCIA SBAGLIATA

Yellow highlight | Location: 2,841
Rahinah – visiting her home country of Malaysia – was told at the airport that her US student visa had been revoked without notice. Despite being the mother of an American citizen, she would never be able to return to the United States…Rahinah had been put on a no-fly list by mistake – possibly the result of confusion between Jemaah Islamiyah, a terrorist group, which killed 202 people with a car bomb in Bali in 2002, and Jemaah Islah Malaysia, a professional association of Malaysians who have studied overseas. Rahinah was a member of the second group, not the first. Once that error had entered the database it acquired the iron authority of the computer….

Note:RAHINAH

Yellow highlight | Location: 2,855
Yet automatic systems want to be tidy. Once an algorithm or a database has placed you in a particular category, the black-and-white definitions of the data discourage argument and uncertainty.

Note:LA MALEDIZIONE DEPL BIANCO E NERO

Yellow highlight | Location: 2,858
We are now on more lists than ever before: lists of criminals; lists of free-spending shoppers;

Note:LISTE

Yellow highlight | Location: 2,865
we have not yet acknowledged how imperfectly a tidy database maps on to a messy world.

Note:DATI CHIARI MONDO CAOTICO

Yellow highlight | Location: 2,871
the database and the algorithm, like the autopilot, should be there to support human decision-making.

Note:SOLO LA COPPIA FUNZIONA

Yellow highlight | Location: 2,873
In March 2012, three Japanese students visiting Australia decided to drive to North Stradbroke, guided by their GPS system. For some reason the GPS was not aware that their route was blocked by nine miles of the Pacific Ocean…in thrall to their technology, they drove their car on to the beach and across the mud flats towards the ocean….

Note:GIAPPONESI IN AUSTRALIA

Yellow highlight | Location: 2,880
Automated systems tend to lull us into passivity.

Note:PASSIVI

Yellow highlight | Location: 2,885
This tendency to passively accept the default option turns out to apply to automated decisions too; psychologists call it automation bias.

Note:AUTOMATATION BIAS

Yellow highlight | Location: 2,887
Driving your car into the sea is an extreme example of automation bias, but most GPS users will recognise the tendency in a milder form….Not knowing why the GPS failed me, I have no way of predicting when it might do so again….

Note:GPS

Yellow highlight | Location: 2,901
When the algorithms are making the decisions, people often stop working to get better.

Note:MIGLIORARE

Yellow highlight | Location: 2,904
engineers make the problem worse by deliberately designing systems to supplant human expertise by default; if we wish instead to use them to support human expertise, we need to wrestle with the system.

Note:AFFIANCAMENTO E RIMPIAZZO

Yellow highlight | Location: 2,909
We worry that the robots are taking our jobs, but just as common a problem is that the robots are taking our judgement.

ote:ROBOT E LAVORO

Yellow highlight | Location: 2,917
You could even argue that the financial crisis of 2007–8, which plunged the world into recession, was analogous to absent-mindedly driving a car into the Pacific.

Note:CRISI FINANZIARIA

Yellow highlight | Location: 2,921
if that grizzled market participant had been able to talk to the computers, the computers would have been able to demonstrate the catastrophic impact of such a crash on the value of CDOs. Unfortunately, there was no meeting of minds

Note:AUTOMATISMI LASCIATI A SE STESSI

Yellow highlight | Location: 2,925
veteran meteorologists would make weather forecasts first by looking at the data and forming an expert judgement; only then would they look at the computerised forecast

Note:LA STRATEGIA DEI METEOREOLOGI

Yellow highlight | Location: 2,931
Chris Urmson, who runs Google’s self-driving car programme, hopes that the cars will soon be so widely available that his sons will never need to have a driving licence.

Note:SELF DRIVING CAR

Yellow highlight | Location: 2,933
unlike a plane’s autopilot, a self-driving car will never need to cede control

Note:UN PERENNE PILOTA AUTOMATICO

Yellow highlight | Location: 2,937
we can look forward to a more gradual process of letting the car drive itself in easier conditions…‘There will always be some edge cases where things do go beyond anybody’s control.’…

Note:RAJKUMAR E IL GRADUALISMO

Yellow highlight | Location: 2,946
if we expect the human to leap in and take over, how will the human be able to react appropriately?

Note:L’ASPETTO DIFFICOLTOSO

Yellow highlight | Location: 2,948
Human beings are not used to driving automated vehicles, so we really don’t know how drivers are going to react when the driving is taken over by the car,’ says Anuj K. Pradhan

Note:L‘INCOGNITA

Yellow highlight | Location: 2,952
And when the computer gives control back to the driver, it may well do so in the most extreme and challenging situations.

Note:SITUAZIONI AL LIMITE

Yellow highlight | Location: 2,956
No matter how many years of experience a driver has, her skills will slowly erode if she lets the computer take over.

Note:L’ESPERIENZA CONTA POCO

Yellow highlight | Location: 2,969
‘I wasn’t parked illegally, I was stuck in traffic’; or ‘That’s not a terrorist group, it’s an alumni association.’ Does more efficient service in the majority of cases justify trapping a small number of individuals in Kafkaesque battles against bureaucracy?

Note:BUROCRAZIA KAFKIANA

Yellow highlight | Location: 2,973
Until the late 1970s, one could reliably expect at least twenty-five fatal commercial plane crashes a year. In 2009, Air France 447 was one of just eight crashes, a safety record. The cost–benefit analysis seems clear: freakish accidents like Flight 447 are a price worth paying,

Note:IL TREND

Yellow highlight | Location: 2,977
One priority could be to make semi-automated systems give feedback in a way that humans feel more viscerally.

Note:SOLUZIONE FEEDBACK

Yellow highlight | Location: 2,983
Some senior pilots urge their juniors to turn off the autopilots from time to time to maintain their skills.

ote:SOLUZIONE ALLENAMENTO

Yellow highlight | Location: 2,986
An alternative solution is to reverse the role of computer and human. Rather than letting the computer fly the plane with the human poised to take over when the computer cannot cope

Note:SOLUZIONE INVERTIRE I RUOLI

Yellow highlight | Location: 3,003
Unsurprisingly, the scientists showed that reaction times and other measures of performance dramatically deteriorated as the hours ticked by.

Note:TEMPI DI REAZIONE

Yellow highlight | Location: 3,007
behaviour suggests that when humans are asked to babysit computers, the computers themselves should be programmed to serve up occasional brief diversions. Even better might be an automated system that demanded more input, more often, from the human

Note:UN COMPUER CHE SBAGLIA DI FREQUENTE PUÒ AIUTARE

Yellow highlight | Location: 3,015
Control measures such as traffic lights and speed bumps frustrated drivers, who would often speed dangerously between one measure and another.

Note:LIMITARE LA VELOCITÀ

Yellow highlight | Location: 3,016
He suggested that the road through Oudehaske be made to look more like what it already was: a road through a village.

Note:SOLUZIONE MONDERMAN

Yellow highlight | Location: 3,025
Where once drivers had, figuratively speaking, sped through the village on autopilot – not really attending to what they were doing – now they were faced with a messy situation and had to engage their brains…As Tom Vanderbilt describes Monderman’s strategy, ‘Rather than clarity and segregation, he had created confusion and ambiguity.’…

Note:UNA SITUAZIONE INCASINATA CI FA CONCENTRARE