ATC expert here. Sorry for the long comment 😅
I’m not defending FAA at all, they are doing shit and overworked their employees for too long. But if you remove engineers and don’t invest in your ATC infrastructure during decades don’t be pikashu face shocked when it fails.
But it is also very easy to bring down those systems, it happened in my country several times and I’ve seen it in neighboring European countries as well.
You lose NTP for whatever reason? You lose radars because you cannot be certain a plot (the dot on the radar) is correct anymore.
You fuck up one part of the FDP (flight data processing) software update you fuck up everything. Because no flight plans = No correlation with the radar plots (a dot with a label become a simple dot lost among thousands of others), and you also lose your capacity prediction (I think that’s what they call SMART).
You lose the MET(eo) system you reduce capacity to at least 50%.
You lose VHF? Guess what… And VHF are old analog to digital back to analog, with modulation/demodulation of frequency to be able to transport voice between the controller mic to the antenna probably far away.
GND to GND voice is VOIP (probably not at FAA, lol) but GND to AIR is still old stuff.
Intercenter coms (sending an aircraft from one airspace to the next one, or OLDI in Europe) is also a delicate system and if you can’t transfer a flight to your neighbor or colleague you have to rely on good old phones, so reduced capacity.
Etc etc etc… So much stuff can take down a sky…
Civil aviation networks are built around a star shaped network at a national level. Those national stars are interlinked between them at the international level to form a meshed network, but if you zoom back to a national level you have a single point of entry to the AFS network, usually called the COM Center. You can find some (old) charts of those network in this ICAO document: https://www.icao.int/sites/default/files/EURNAT/Documents/EUR and Nat Docs/EUR Documents/EUR Documents/ATS Messaging Routing Directory/AMRD-Part-IV-COM-Charts-v18_0-v1.pdf
To have the same for the national network you have to work at the ATC here, but everything ICAO is public.
And guess what? You lose the COM Center, you lose everything. Of course they have network and system redundancies but shit still happens. France for example had its COM Center taken down by Italy a few years back.
Italy used a fucked up system they had no choice to use (from a Berlusconi company), this system sent erroneous data to the French, France felt for hours, closed sky etc. Redundancy, backups or not, as long as Italy was sending errors to them the core of the star network kept crashing every time they tried to get back up.
I have plenty more of silly examples like that around Europe and it’s one of the most funded and well managed ATC.
So FAA with no investment since the end of the Cold War? Yeah no shit Sherlock…
If you’re curious about how the civil aviation ground to ground communications works I think Wikipedia has some articles. Keywords would be AFS (aeronautical fixed services), AFTN (aeronautical fixed telecommunications network), AMHS (Aeronautical message handling system, formerly CIDIN which stands for I don’t Remember…)
PS: But closing the sky or reducing its capacity (so your flight is cancelled) is also a “good thing”, it’s done for safety reasons. We rather have you (you the pax, the self loading cargo) pissed off on the ground at the airport than dead in a crash because we weren’t able to guarantee a reliable and safe ATC anymore 🤷🏻♂️
Dave Gunson released a record/tape called What Goes Up … Might Come Down!” (1982). I think you would like it, if you can find a copy.
It’s a very funny recording of an after dinner speech which must still be available on the internet, by a British Air Traffic Controller who worked through the 70s and 80s. A great speaker. One of the things he covered was when the French went on strike. He would normally ring up his equivalent on the phone and pass him a group of dots. His French equivalent would say “Thank you very much” and they were his responsibility until he did the same and passed them to Spain.
Then one day, the phone was answered and the French said, “No thank you, we are on strike” and put the phone down. Gunson had to route his blips up and around French airspace for several days. They were only on strike for one day, and when they were rung to see if they would take some more blips they said “Certainly, it’s been very quiet”. They’d only had a one day strike, but hadn’t told anyone they were back so all international air traffic was still routing around their airspace thinking it was closed.
https://www.gordonpoole.com/talent/david-gunson/
lol that looks like my jam thanks.
Yeah we of course have French strikes on a regular base. I’d say once a year but I don’t remember the last one, it might be several years ago.
Usually all EUR/NAT ACCs (area control center) receive a message (via the above mentioned AFTN) which say something like “unmanned, reason: Industrial action” and routings have to be adapted in consequence.
Once a colleague had to call them for something tech related (so not an aircraft in flight, not an emergency) and the dude at Bordeaux was on site, answering, saying “ah yes I see it’s broken, but I won’t do anything until tomorrow”. He was just there staring at the screens and probably drinking coffee. So they are not unmanned (that would be a huge safety risk), they are just idling at work.
Italy is also one of our usual funny interactions, they are very relaxed, sometimes too much (but me as well so I won’t blame them).
The ones that seems perfect are of course the Brits and the Germans, but they sometimes fail just as much as the rest of us, they just looks more professionals while doing it.
This is notably weak reporting from The Verge, which is usually appropriately skeptical of unsubstantiated explanations.
Here’s the entire article:
At a press conference today, Transportation Secretary Sean Duffy said that Friday’s nationwide ground stop ordered by the FAA was related to work replacing outdated copper wire with a fiber optic line at a “remote FAA facility.” The secretary said that this was “routine maintenance,” rather than the result of a major modernization effort. He told reporters that it was “such an old wire that it was being discontinued, meaning that line was going to get shut down,” necessitating the upgrade.
What wasn’t clear, according to Secretary Duffy, was the root cause of the problem. He said that it could have been faulty hardware, a misconfigured router, or “incompetence” on the part of contractor L3Harris. Regardless, what followed was the newly installed router “flooding” other routers in the Air Traffic Control system. Duffy said that there were “over 1,000 connection attempts every 4 to 5 seconds,” causing disruption to radar feeds and communication with planes.
Secretary Duffy also took the opportunity to call on additional funding from Congress for modernizing the Air Traffic Control system, and to tout the work the FAA has done so far. “We are building, we are building safely, but we are building fast,” he said, but said that Congress had only given him a “partial payment” to complete the necessary work.
There’s no reason to trust Sean Duffy’s explanation here, and skepticism is the only rational response given the Trump administration frequent lies about anything and everything that goes wrong. The headline should be “says Trump administration” or “says Sean Duffy,” and judiciously avoid endorsement of that explanation in the article. Instead, not only does the headline adopt the explanation as uncontroverted truth, there isn’t any attempt in the body to fact check or leave open the possibility of alternate explanations, despite confounding/contributing factors (like the new AI system rollout) that are being discussed elsewhere.
This is the fifth or sixth explanation I’ve heard. Are they purposefully trying to sew confusion?
Begging for money to embezzle to his friends.
Extorting. In one breath he says he needs $50 billion, and in another he says it could happen again at any time. If it were simply an operational mistake by some low voltage technician, he wouldn’t need the money and could keep it from happening again.
You don’t take a copper circuit offline until you verify the new fiber optic one is functional. This is just lies because they don’t want anyone looking behind the curtain and the failed AI roll-out.
Just FYI, it’s “sow”, as in planting crops.
https://en.wikipedia.org/wiki/Firehose_of_falsehood
Oh, look, Russia again. Huh.
I’m neurospicy, and also used to programming errors, but to me that detects as incompetence. It tells me they don’t know what they’re talking about.
When Trump kept saying the Hormuz Strait was open to freight when it wasn’t, I quickly concluded he doesn’t know the conditions that determine open. Or he’s misinformed. Or he’s lying. Now, it’s general knowledge Trump is not a source.
Duffy is definitely not a source either. He doesn’t know how transportation infrastructure works. He’s a reality-television star, which is why he’s been doing a travel themed reality TV show rather than actually doing his job.
Hey, it gives us something we can do for the resistance, which is teach others to not rely on official lines from the regime, but what we can gather from news based on leaks and investigation. It’s what we had to do in the George W. Bush era.
[Duffy is] a reality-television star
I wasn’t aware of that, but I can’t help notice a curious coincidence. Are there more incompetent US officials with a background in television rather than whatever their actual profession is?
Secretary of education is the wife and cofounder of the fake wrestling WWE. A whole career in professionally faking it, and she’s in charge of education in this country.
Yes. While a lot of Americans’ primary source of media has been the TV in the past couple decades, it’s even worse with Trump. He, personally, spends a lot of time watching Fox News, so that’s his primary source for what’s going on in the country. And that’s why you have people like Jeanine Pirro (had a show on Fox during his first term) as DC prosecutor, and Dr. Oz in charge of Medicaid and Medicare.
sew
And a bone apple tea to you!
“Random explanation generator” is a mandatory module in every company communication software.
Sounds like bullshit to save face on the AI debacle.
Are you trying to suggest STP wasn’t enabled on one device and could propagate to the entire system? A system that’s been fine for years? Highly unlikely. Not everyone is as stupid as you are Mr Duffy.
Probably to steal more public money than save face.
Oh my god, they’re switching over to the Russian playbook on Government news.
Appear to be incompetent and the only viable information. Basically dilute the news with as many random ass explanations as you please, and people will either stop talking about it, or worse, take the Russian attitude of “its not my concern.”
He pretty much had me until “such an old wire that it was being discontinued, meaning that line was going to get shut down”. Does he know what wire is? And what is, for the FAA, a remote facility? What “line” was being shut down? It’s all very lazy and hand wavy.
At a press conference today, Transportation Secretary Sean Duffy said that Friday’s nationwide ground stop ordered by the FAA was related to work replacing outdated copper wire with a fiber optic line at a “remote FAA facility.” The secretary said that this was “routine maintenance,” rather than the result of a major modernization effort. He told reporters that it was “such an old wire that it was being discontinued, meaning that line was going to get shut down,” necessitating the upgrade.
What wasn’t clear, according to Secretary Duffy, was the root cause of the problem. He said that it could have been faulty hardware, a misconfigured router, or “incompetence” on the part of contractor L3Harris. Regardless, what followed was the newly installed router “flooding” other routers in the Air Traffic Control system. Duffy said that there were “over 1,000 connection attempts every 4 to 5 seconds,” causing disruption to radar feeds and communication with planes.
I feel that might be actually worse than the supposed AI integration mistake, that a single mistake in one place can bring almost everything down. I don’t know which is really true, but wow. What a mess.
It’s almost as if Elon Musk cutting those 400 engineers from the FAA might not have been a great idea.
It would be worse if it were true, but they’re lying to cover their ass about the disastrous attempt to integrate slop into a previously functional system.
ATC systems are designed with redundancies, you cannot knock them offline by snipping one cable.
Yeah, that sounds like they told someone who knew nothing about the tech to make something up to cover, not realizing the lie has to make sense.
Aircraft have at least two of every important system for exactly that reason, someone forgot to tell the towers.
It’s literally listed in the project 2025 playbook though so not ATCs fault they’re being used as political pawns
The towers have backup systems too. Like a hospital, if the computers go down, they switch to physical charts and telephones.
Honestly, calling a national ground stop made this into a much bigger problem than it needed to be.
Outlook servers with a noose around the neck: First time?
If this article is accurate even the air traffic controllers aren’t buying Duffy’s FAA BS
https://www.commondreams.org/news/duffy-ground-stop-explanation
Regardless, what followed was the newly installed router “flooding” other routers in the Air Traffic Control system. Duffy said that there were “over 1,000 connection attempts every 4 to 5 seconds,”
I’m not a networking person. Is that a lot?
I was a network person before i changed disciplines.
A direct answer is no, that is not a lot of connections, in fact, it is trivial. What we don’t know is any level of meaningful detail. If there were thousands of endpoints each making thousands of connections all at once, that changes the scale of the problem. Furthermore, most of the time, we assume that one host is trying to connect to another host. However, if those connection attempts were network broadcasts, that’s a much bigger problem. A network broadcast is sent from one host to ALL hosts on a given network, and then they ALL respond. If all hosts on a network are sending broadcast frames endlessly, that will flood a network. It’s the same concept as a denial of service attack; one host can initiate traffic that will exhaust resources, but takes very little resource to do it. Thusly it can cripple a network with very little need for power or bandwidth.
My guess is that they fucked up the config on the router and was passing traffic that it should not have or initiated traffic that it couldn’t handle the responses from. It started using up all resources on itself and probably other network equipment and the whole thing shits the bed. Techs go around trying to reboot network gear, not sure where the problem is, the gear is overwhelmed and won’t respond. They have to dick around with out-of-band management (does not use the clogged network for management, there is a separate network for management), try to reboot or do some diagnostics, but as soon as the network device comes back up, it’s saturated again making diagnosis slow and painful. People panic, bosses start freaking out, more and more people are called in adding to confusion and now the first people to respond are not working, they’re answering endless questions about what’s going on, even though they don’t really know yet. Vendors are called, contractors are called, calls spiral and fracture, bosses demand updates while people are guessing at what to do next because they aren’t allowed to say “I don’t know” when planes are in the air. Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched before everything shit the bed. They go look at the config, see the fuckup, back it out to the last known good config and the shitstorm dies.
this is 100% speculation treat is as such.
I’m above average familiar with networking, but likely not as much as you. Currently working on going into that field tho.
I’m also semi-familiar with how ATC works from touring their local facilities as part of a course on aerodynamics.
I’m fairly certain they’re lying to cover their ass.
Everything in the ATC system has redundancy. They’re saying they clipped one copper wire and the whole system went down, but there would be another established path for that connection, severing one copper wore wouldn’t stop any communications, they would transmit through a fail safe.
Also, the last time they had to ground all flights was 09/11/2001, and let’s just say it was not because of issues with the extremely reliable technology used in ATC.
They installed their AI slop bullshit a few days ago.
They think we’re all stupid, so they didn’t bother asking anyone who understands the systems before formulating their low-quality lie (in a similar fashion to how they came up with their slop ATC system - without speaking to anyone who knows how the shit works).
You are correct. One physical disconnect of a copper cable isn’t going to kill off an entire system. As you noted, they have redundant network paths, equipment, etc. Those systems were designed not to go down hard, and until this week, never did.
Copper maxes out at 10gbps in a production environment and if they were still using copper Ethernet, they were trunking multiple cables together for higher throughput. The trunking also had to be redundant, so it would take a lot of cutting or unplugging to take it all down as an oopsy.
Again, all speculation, but I’m still betting on a human-grade fuckup. When in doubt, blame DNS.
With network-loop type shit, it’s less likely they clipped the copper wire, and more likely they brought up the fiber before taking down the copper link and things sent over the copper were coming back on the fiber, looping and exploding everything.
It is possible but the old routing protocols that used to do this (spanning tree) have protections in place to prevent it.
At one point, routing protocols were a little more promiscuous and you could create a loop by plugging a single Ethernet cable into two ports on the same switch. Cue Benny hill music as everyone went running around trying to figure out what happened.
Now there are mechanisms like BPDU guard and portfast configs that should prevent this. In most cases, they would be enabled by default. However, I’m sure there are some folks out there who have no idea how to handle an old port config done at the command line. They could have been tinkering to try and make something work, turned off some of the protective features and now they’re back to the 90s with routing loops and broadcast storms. The song is Yakety Sax fwiw, always have it at the ready.
If you ever wonder why the networking people are grouchy, this is why. Networks were and remain complex, highly interdependent, challenging to build and maintain. It takes only a few ignorant people to send something tumbling.
this is 100% speculation treat is as such.
Will do.
Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched
I’d hope that those records exist. I think there’s value in treating this as a dry run response to a cyber attack, and so far I’m not inspired (mostly with the immediate call for money).
For anything made in the last 30 years, no. He could be completely off the mark (see also Sen. Ted Stevens’ “series of tubes” speech) or the equipment might actually be that old and unable to handle it.
All bad options. :-(
