Did We Create A World Too Complex For Ourselves?

[alphacat]

It’s complicated
Human ingenuity has created a world that the mind cannot master. Have we finally reached our limits?

Despite the vastness of the sky, airplanes occasionally crash into each other. To avoid these catastrophes, the Traffic Alert and Collision Avoidance System (TCAS) was developed. TCAS alerts pilots to potential hazards, and tells them how to respond by using a series of complicated rules. In fact, this set of rules — developed over decades — is so complex, perhaps only a handful of individuals alive even understand it anymore. When a TCAS is developed, humans are pushed to the sidelines and, instead, simulation is used. If the system responds as expected after a number of test cases, it receives the engineer’s seal of approval and goes into use.

While the problem of avoiding collisions is itself a complex question, the system we’ve built to handle this problem has essentially become too complicated for us to understand, and even experts sometimes react with surprise to its behaviour. This escalating complexity points to a larger phenomenon in modern life. When the systems designed to save our lives are hard to grasp, we have reached a technological threshold that bears examining.

For centuries, humans have been creating ever-more complicated systems, from the machines we live with to the informational systems and laws that keep our global civilisation stitched together. Technology continues its fantastic pace of accelerating complexity — offering efficiencies and benefits that previous generations could not have imagined — but with this increasing sophistication and interconnectedness come complicated and messy effects that we can’t always anticipate. It’s one thing to recognise that technology continues to grow more complex, making the task of the experts who build and maintain our systems more complicated still, but it’s quite another to recognise that many of these systems are actually no longer completely understandable. We now live in a world filled with incomprehensible glitches and bugs. When we find a bug in a video game, it’s intriguing, but when we are surprised by the very infrastructure of our society, that should give us pause.

One of the earliest signs of technology complicating human life was the advent of the railroads, which necessitated the development of standardised time zones in the United States, to co-ordinate the dozens of new trains that were criss-crossing the continent. And things have gotten orders of magnitude more complex since then in the realm of transportation. Automobiles have gone from mechanical contraptions of limited complexity to computational engines on wheels. Indeed, it’s estimated that the US has more than 300,000 intersections with traffic signals in its road system. And it’s not just the systems and networks these machines inhabit. During the past 200 years, the number of individual parts in our complicated machines — from airplanes to calculators — has increased exponentially.

The encroachment of technological complication through increased computerisation has affected every aspect of our lives, from kitchen appliances to workout equipment. We are now living with the unintended consequences: a world we have created for ourselves that is too complicated for our humble human brains to handle. The nightmare scenario is not Skynet — a self-aware network declaring war on humanity — but messy systems so convoluted that nearly any glitch you can think of can happen. And they actually happen far more often than we would like.

We already see hints of the endpoint toward which we seem to be hurtling: a world where nearly self-contained technological ecosystems operate outside of human knowledge and understanding. As a scientific paper in Nature in September 2013 put it, there is a complete ‘machine ecology beyond human response time’ in the financial world, where stocks are traded in an eyeblink, and mini-crashes and spikes can occur on the order of a second or less. When we try to push our financial trades to the limits of the speed of light, it is time to recognise that machines are interacting with each other in rich ways, essentially as algorithms trading among themselves, with humans on the sidelines.

It used to be taken for granted that there would be knowledge that no human could possibly attain. In his book The Guide for the Perplexed, the medieval scholar Moses Maimonides opined that ‘man’s intellect indubitably has a limit at which it stops’ and even enumerated several concepts he thought we would never grasp, including ‘the number of the stars of heaven’ and ‘whether that number is even or odd’. But then the Scientific Revolution happened, and with it, a triumphalism of understanding. Hundreds of years later, we now know the exact number of objects in the night sky visible to the naked eye — it’s 9,110 (an even number).

But ever since the Enlightenment, we have moved steadily toward the ‘Entanglement’, a term coined by the American computer scientist Danny Hillis. The Entanglement is the trend towards more interconnected and less comprehensible technological surroundings. Hillis argues that our machines, while subject to rational rules, are now too complicated to understand. Whether it’s the entirety of the internet or other large pieces of our infrastructure, understanding the whole — keeping it in your head — is no longer even close to possible.

One example of this trend is our software’s increasing complexity, as measured by the number of lines of code it takes to write it. According to some estimates, the source code for the Windows operating system increased by an order of magnitude over the course of a decade, making it impossible for a single person to understand all the different parts at once. And remember Y2K? It’s true that the so-called Millennium Bug passed without serious complications, but the startling fact was that we couldn’t be sure what would happen on 1 January 2000 because the systems involved were too complex.

Even our legal systems have grown irreconcilably messy. The US Code, itself a kind of technology, is more than 22 million words long and contains more than 80,000 links within it, between one section and another. This vast legal network is profoundly complicated, the functionality of which no person could understand in its entirety. Michael Mandel and Diana Carew, of the Progressive Policy Institute in WashingtonDC, have referred to this growth of legal systems as ‘regulatory accumulation’, wherein we keep adding more and more rules and regulations. Each law individually might make sense, but taken together they can be debilitating, and even interact in surprising and unexpected ways. We even see the interplay between legal complexity and computational complexity in the problematic rollout of a website for Obamacare. The glitches in this technological system can affect each of us.

And this trend is accelerating. For instance, we now have 3D printers, vast machinery to help construct tunnels and bridges, and even software that helps with the design of new products and infrastructure, such as sophisticated computer-aided design (CAD) programs. One computational realm, evolutionary programming, even allows software to ‘evolve’ solutions to problems, while being agnostic on what shape that final solution could take. Need an equation to fit some data? Evolutionary programming can do that — even if you can’t understand the answer it comes up with.

A number of years ago, a team of research scientists tried to improve the design of a certain kind of computer circuit. They created a simple task that the circuit needed to solve and then tried to evolve a potential solution. After many generations, the team eventually found a successful circuit design. But here’s the interesting part: there were parts of it that were disconnected from the main part of the circuit, but were essential for its function. Essentially, the evolutionary program took advantage of weird physical and electromagnetic phenomena that no engineer would ever think of using in order to make the circuit complete its task. In the words of the researchers: ‘Evolution was able to exploit this physical behaviour, even though it would be difficult to analyse.’

This evolutionary technique yielded a novel technological system, one that we have difficulty understanding, because we would never have come up with something like this on our own. In chess, a realm where computers are more powerful than humans and have the ability to win in ways that the human mind can’t always understand, these types of solutions are known as ‘computer moves’ — the moves that no human would ever do, the ones that are ugly but still get results. As the American economist Tyler Cowen noted in his book Average Is Over (2013), these types of moves often seem wrong, but they are very effective. Computers have exposed the fact that chess, at least when played at the highest levels, is too complicated, with too many moving parts for a person — even a grandmaster — to understand.

So how do we respond to all of this technological impenetrability? One response is to simply give up, much like the comic strip character Calvin (friend to a philosophical tiger) who declared that everything from light bulbs to vacuum cleaners works via ‘magic’. Rather than confront the complicated truth of how wind works, Calvin resorts to calling it ‘trees sneezing’. This intellectual surrender in the face of increasing complexity seems too extreme and even a bit cowardly, but what should we replace it with if we can’t understand our creations any more?

Perhaps we can replace it with the same kind of attitude we have towards weather. While we can’t actually control the weather or understand it in all of its nonlinear details, we can predict it reasonably well, adapt to it, and even prepare for it. And when the elements deliver us something unexpected, we muddle through as best as we can. So, just as we have weather models, we can begin to make models of our technological systems, even somewhat simplified ones. Playing with a simulation of the system we’re interested in — testing its limits and fiddling with its parameters, rather than understanding it completely — can be a powerful path to insight, and is a skill that needs cultivation.

For example, the computer game SimCity, a model of sorts, gives its users insights into how a city works. Before SimCity, few outside the realm of urban planning and civil engineering had a clear mental model of how cities worked, and none were able to twiddle the knobs of urban life to produce counterfactual outcomes. We probably still can’t do that at the level of complexity of an actual city, but those who play these types of games do have a better understanding of the general effects of their actions. We need to get better at ‘playing’ simulations of the technological world more generally. This could conceivably be geared towards the direction our educational system needs to move, teaching students how to play with something, examining its bounds and how it works, at least ‘sort of’.

We also need interpreters of what’s going on in these systems, a bit like TV meteorologists. Near the end of Average Is Over, Cowen speculates about these future interpreters. He says they ‘will hone their skills of seeking out, absorbing, and evaluating this information… They will be translators of the truths coming out of our networks of machines… At least for a while, they will be the only people left who will have a clear notion of what is going on.’

And when things get too complicated and we end up being surprised by the workings of the structures humanity has created? At that point, we will have to take a cue from those who turn up their collars to the unexpected wintry mix and sigh as they proceed outdoors: we will have to become a bit more humble. Those like Maimonides, who lived before the Enlightenment, recognised that there were bounds to what we could know, and it might be time to return to that way of thinking. Of course, we shouldn’t throw our hands up and say that just because we can’t understand something, there is nothing else to learn. But at the same time, it might be time to get reacquainted with our limits.

[kay]

This is kinda true, but the systems they talk about are ones which have grown up organically, inefficiently and haphazardly. All the systems the article used as examples also suffer from the fact that all of them were assembled by multiple independent teams over a number of years, which means that there was no cohesive structure/plan as to how to set things up logically. If someone were to start from scratch today, they would most likely be able to come up with a less complex (or at least more streamlined) version of something that was made 20 years ago.

Having said that, complex systems now represents a field of study with growing importance.

Of course, it doesn't take lines and lines of code or years and years of revisions in law to make things complex. Simple rules can also lead to complexity/chaos/unpredictability under certain circumstances. Take the first simulation of life programmes. Very simple rules to define how blocks moved, propogated, mated and died. But with very complex results.

[syrup]

yes

[sigbowls]

im even more scared of planes now

[test_recordings]

It's kind of funny, humans invented systems to control nature, then the universe at large. This grew out of a fear of unpredicability, much as the tricky Greek god of nature, Pan, was transformed in to the Christian devil. In traditional tribal societies, groups have a shaman, or other leader, tasked with understanding their surroundings and utilising them for their benefit. They are chosen and trained to be able to read what's happening without explicit knowledge of what is happening, even conservation is handled by them sometimes.

Now, humans have created another unpredictable universe and require their own 'techno shamans' to be able to guide them through the maze!

I agree with Kay though. You can see actual examples of sensible planning in places like Japan. The place got heavily bombed in WW2 and has had numerous natural disasters that have rendered large areas as blank slates. Add in a pretty centralised planning system ('collective capitalism', but the government also works hand-in-hand with private entities for mutual benefit a lot of the time) and you have a very organised, efficient system. There is a heavy emphasis on cooperation, though.

Take, for example, the Tokyo subway system for travel around the city's neighbourhoods and zones (the overground Japan Railway system is another entity in itself, but that is essentially a government-licensed monopoly in certain areas to guarantee transport in, out and around major areas). I used to live on the Saitama Railway line just next to the Tokyo metropolitan area - there was one driver for that line from the end furthest from Tokyo to the start of the next, fully connected, line in the Tokyo area, the Namboku line. There was no train change, but a driver from a different company (I think the Toyko Metro network) was waiting on the platform and swapped with the driver from the Saitama line (I assume the Saitama driver swopped back on to a train going up the Saitama line). The same train then covered the whole Namboku line with the Tokyo Metro driver until it gets to another station where the train company changes - same thing, Tokyo Metro driver jumps out and yet another company's (Tokyu) driver gets on to drive it down the Tokyu Meguro line (in fact, the Tokyo Meguro line actually interoperates with another company, Toei, on their Toei Mita line at the Meguro station).

So that's the same train line operated by three companies with three different drivers indirectly interacting with a fourth company at one point... and it all works perfectly That's because everything is p-lanned to fuck and everyone knows their part. All the companies actually share software to coordinate train times in case something fucks up somewhere because there's so many interconnections, you can't have people piling up at stations because you can have hundreds of people on a train and there's trains coming in every 2 minutes at rush hour (even then, people are employed to push people on to trains for 200% seating capacity). There's also 37,000,000 million people in the surrounding area. I fucking hated going to work in that, couldn't board an express past 7 in the fucking morning...

[sigbowls]

the whole world is fucked thats all i got to say

[nousd]

Human culture has always had specialization so that nobody understood everything
so that we've long been reliant on the cooperation of indivualized knowledges.
The point of this article seems to be:

A number of years ago, a team of research scientists tried to improve the design of a certain kind of computer circuit. They created a simple task that the circuit needed to solve and then tried to evolve a potential solution. After many generations, the team eventually found a successful circuit design. But here’s the interesting part: there were parts of it that were disconnected from the main part of the circuit, but were essential for its function. Essentially, the evolutionary program took advantage of weird physical and electromagnetic phenomena that no engineer would ever think of using in order to make the circuit complete its task. In the words of the researchers: ‘Evolution was able to exploit this physical behaviour, even though it would be difficult to analyse.

Today's incarnation of a chainsaw would have been equally as arcane to cavemen without axes. The accumulation of design evolving ideas has been sped up to make this process seem incomprehensible. imo essentially what's changing is the rate of change. Because not many people understand second order calculus, we're increasingly assigning that function to machines.
Even the recommendation to have more people working to predict system outcomes is a call to speed up feedback rather than previously leisurely observation and adjustment.
And why? Perhaps we've crossed an evolutionary (or environmental) threshold whereby mutation and migration are becoming more urgent...after all, we've only got a few billion years to successfully leave this solar system and survive.

[kay]

Perhaps it's more a realisation that if you leave systems to grow as they will, the chances of something fucking up will grow as well. The larger and more complex a system, the worse the effects of a fuck up.

We used to think that systems with simple rules would remain easily predictable. In most cases this is probably true. But the situations where it has proven untrue have forced us to learn the hard way that we need to pay attention to how things work and how we put things together in order to avoid unwanted interactions.

Trying to make sense of big data has made many inroads towards better understanding of complex systems. I think that the monkey wrench in the whole system is actually humans. Poor documentation, poor cooperation, poor communication, poor thinking, poor system design, poor architecture and poor rigour are probably greater contributions to system failures than the complexities of systems themselves.

[wysockisauce]

Like sd5 said division of labor is something to consider. It allowed us to get this far (technologically) in the first place. As soon as it is instituted it takes more than one person to understand something/make it work.

I agree that humans are the weakest link here. We built (through division of labor) complex interrelated systems that have to work together perfectly to get good results. Unless the creators and those maintaining the systems work together perfectly as well we're going to have problems.

Progress on finding the solution to the human problem is actually pretty good. Just look at the internet.

Obligatory

[test_recordings]

Humans really need to know more about themselves to progress. Even scientists are prone to horribly obvious cognitive biases, but often don't think they can be.

"Check yourself" should be the phrase of the century imho

Also, evolution made a pretty good job of humans etc so why not just do the same with software that has a generation span trillions of times faster than humans and see what happens? We don't even completely understand nature and probably never will but that's not stopping anyone making use of it...

[Jizz]

We havent realised the size of the monster we have created. THE END IS NIGH

[nowaysj]

@sd5 I think you may be glossing an important point of this article: that progress may become impossible under the bulk of these unknowable systems. That to push these systems to new levels won't be effective without a complete understanding of the systems, and due to their complexity, a complete understanding is impossible, therefore we may hit a complexity barrier where no further significant work can be done.

---

Interesting idea above regarding man's relation to nature. That in our quest to mitigate the chaos/unpredictability/complexity of nature, we create somewhat equivalent chaos/unpredictability/complexity. Essentially that we have removed the existential threat of unpredictable weather (if you're white), and replaced it with an unpredictable market with existential implications?


----

Fragility - as our systems are getting more complex and interconnected, are they becoming more robust or more fragile/vulnerable? I know two very important US systems are extremely fragile - the NYSE and the US power grid. Both are extremely central to every aspect of life in the US, and both are apparently totally fragile.


All this definitely feels like we're trapped in a fractal, and we're adolescently shifting scale, looking for an escape, but we keep finding the same shit at every level.

[nousd]

NOWAYS: I think you may be glossing an important point of this article: that progress may become impossible under the bulk of these unknowable systems.

hmmm...yeah...consciously controlled progress for sure

All this definitely feels like we're trapped in a fractal

ah fractals, very topical
with fractals we can zoom in and examine detail,
manipulate it and zoom back out
(I'm currently incorporating all my art & writing into one opus this way).

being trapped is illusory imo
just like tripping turns to psychosis when you try to rationalize aspects of experience that aren't intellectually consistent
solution: leave to detail to those interested in resolution
and keep driving the over-view, the path

progress suggests a judgement of what is happening,
an ability to manipulate it
what we need is clarification and prognosis.

[nowaysj]

being trapped is illusory imo

progress suggests a judgement of what is happening,
an ability to manipulate it
what we need is clarification and prognosis.

It may suggest judgement, but it is intended in my usage as our progenitors intended it, simply to walk forward. To walk where we have not been. In regards to moving through scale in a fractal: progress is impossible, as it is all reiteration. We are not walking where we have not been, we're constantly arriving at where we just left. We are leaving the ravages of weather and arriving at the tempest of a market capable of collapse in milliseconds.

Call that not a trap if you like, but that feels like one to me. It is a room with two opposing doors, they are the same door, exit one, and enter the other. We are not going anywhere.

[kay]

With regards complexity barriers, I think that any system that grows organically will hit this barrier at some point, simply because new functional bits are tacked on as the system grows.

At that point, the system collapses into unpredictability and is no longer able to grow meaningfully. In natural systems, I guess that leads to a dead end or status quo of sorts. In human-engineered systems, on the other hand, we have the option of re-engineering the system and building a whole new unified architecture from scratch (which then complexifies again).

The problem is that people/organisations are really bad at taking the initiative to build new versions of things until it is absolutely imperative, by which time it's too late and shit has already hit the fan.

[test_recordings]

I remember reading on wired.com or somewhere that, because of all the separation of responsibility in production, technology is often outdated even on it's release. Old versions of software, hardware, etc often 5 years or more. That means there's not only a huge number of ways to exploit it, but also that it can't be patched because there's just no patches for it anymore and it would require a complete overhaul. Now Apple's in-house, closed system design actually looks better than I thought it was it as they know everything about their own products.

[Pedro Sánchez]

I fucking hate JavaScript, bane of my life.

[kay]

I remember reading on wired.com or somewhere that, because of all the separation of responsibility in production, technology is often outdated even on it's release. Old versions of software, hardware, etc often 5 years or more. That means there's not only a huge number of ways to exploit it, but also that it can't be patched because there's just no patches for it anymore and it would require a complete overhaul. Now Apple's in-house, closed system design actually looks better than I thought it was it as they know everything about their own products.

That's probably true. It's even more true for hardware and physical products. It usually takes a year or two to go from finalised design to production and manufacturing. So by the time anything is released, companies are already working on the next iteration (or even the one after).

[sigbowls]

i like alphacat threads. more

[alphacat]

Thank you kindly.