The Ghost of Cognition Past, or Thinking Like An Algorithm

[Image: Wiring the ENIAC; via Wired]

One of many things I love about writing—that is, engaging in writing as an activity—is how it facilitates a discovery of connections between otherwise unrelated things. Writing reveals and even relies upon analogies, metaphors, and unexpected similarities: there is resonance between a story in the news and a medieval European folktale, say, or between a photo taken in a war-wrecked city and an 18th-century landscape painting. These sorts of relations might remain dormant or unnoticed until writing brings them to the foreground: previously unconnected topics and themes begin to interact, developing meanings not present in those original subjects on their own.

Wildfires burning in the Arctic might bring to mind infernal images from Paradise Lost or even intimations of an unwritten J.G. Ballard novel, pushing a simple tale of natural disaster to new symbolic heights, something mythic and larger than the story at hand. Learning that U.S. Naval researchers on the Gulf Coast have used the marine slime of a “300-million-year old creature” to develop 21st-century body armor might conjure images from classical mythology or even from H.P. Lovecraft: Neptunian biotech wed with Cthulhoid military terror.

In other words, writing means that one thing can be crosswired or brought into contrast with another for the specific purpose of fueling further imaginative connections, new themes to be pulled apart and lengthened, teased out to form plots, characters, and scenes.

In addition, a writer of fiction might stage an otherwise straightforward storyline in an unexpected setting, in order to reveal something new about both. It’s a hard-boiled detective thriller—set on an international space station. It’s a heist film—set at the bottom of the sea. It’s a procedural missing-person mystery—set on a remote military base in Afghanistan.

Thinking like a writer would mean asking why things have happened in this way and not another—in this place and not another—and to see what happens when you begin to switch things around. It’s about strategic recombination.

I mention all this after reading a new essay by artist and critic James Bridle about algorithmic content generation as seen in children’s videos on YouTube. The piece is worth reading for yourself, but I wanted to highlight a few things here.

[Image: Wiring the ENIAC; via Wired]

In brief, the essay suggests that an increasingly odd, even nonsensical subcategory of children’s video is emerging on YouTube. The content of these videos, Bridle writes, comes from what he calls “keyword/hashtag association.” That is, popular keyword searches have become a stimulus for producing new videos whose content is reverse-engineered from those searches.

To use an entirely fictional example of what this means, let’s imagine that, following a popular Saturday Night Live sketch, millions of people begin Googling “Pokémon Go Ewan McGregor.” In the emerging YouTube media ecology that Bridle documents, someone with an entrepreneurial spirit would immediately make a Pokémon Go video featuring Ewan McGregor both to satisfy this peculiar cultural urge and to profit from the anticipated traffic.

Content-generation through keyword mixing is “a whole dark art unto itself,” Bridle suggests. As a particular keyword or hashtag begins to trend, “content producers pile onto it, creating thousands and thousands more of these videos in every possible iteration.” Imagine Ewan McGregor playing Pokémon Go, forever.

What’s unusual here, however, and what Bridle specifically highlights in his essay, is that this creative process is becoming automated: machine-learning algorithms are taking note of trending keyword searches or popular hashtag combinations, then recommending the production of content to match those otherwise arbitrary sets. For Bridle, the results verge on the incomprehensible—less Big Data, say, than Big Dada.

This is by no means new. Recall the origin of House of Cards on Netflix. Netflix learned from its massive trove of consumer data that its customers liked, among other things, David Fincher films, political thrillers, and the actor Kevin Spacey. As David Carr explained for the New York Times back in 2013, this suggested the outline of a possible series: “With those three circles of interest, Netflix was able to find a Venn diagram intersection that suggested that buying the series would be a very good bet on original programming.”

In other words, House of Cards was produced because it matched a data set, an example of “keyword/hashtag association” becoming video.

The question here would be: what if, instead of a human producer, a machine-learning algorithm had been tasked with analyzing Netflix consumer data and generating an idea for a new TV show? What if that recommendation algorithm didn’t quite understand which combinations would be good or worth watching? It’s not hard to imagine an unwatchably surreal, even uncanny television show resulting from this, something that seems to make more sense as a data-collection exercise than as a coherent plot—yet Bridle suggests that this is exactly what’s happening in the world of children’s videos online.

[Image: From Metropolis].

In some of these videos, Bridle explains, keyword-based programming might mean something as basic as altering a few words in a script, then having actors playfully act out those new scenarios. Actors might incorporate new toys, new types of candy, or even a particular child’s name: “Matt” on a “donkey” at “the zoo” becomes “Matt” on a “horse” at “the zoo” becomes “Carla” on a “horse” at “home.” Each variant keyword combination then results in its own short video, and each of these videos can be monetized. Future such recombinations are infinite.

In an age of easily produced digital animations, Bridle adds, these sorts of keyword micro-variants can be produced both extremely quickly and very nearly automatically. Some YouTube producers have even eliminated “human actors” altogether, he writes, “to create infinite reconfigurable versions of the same videos over and over again. What is occurring here is clearly automated. Stock animations, audio tracks, and lists of keywords being assembled in their thousands to produce an endless stream of videos.”

Bridle notes with worry that it is nearly impossible here “to parse out the gap between human and machine.”

Going further, he suggests that the automated production of new videos based on popular search terms has resulted in scenes so troubling that children should not be exposed to them—but, interestingly, Bridle’s reaction here seems to be based on those videos’ content. That is, the videos feature animated characters appearing without heads, or kids being buried alive in sandboxes, or even the painful sounds of babies crying.

What I think is unsettling here is slightly different, on the other hand. The content, in my opinion, is simply strange: a kind of low-rent surrealism for kids, David Lynch-lite for toddlers. For thousands of years, western folktales have featured cannibals, incest, haunted houses, even John Carpenter-like biological transformations, from woman to tree, or from man to pig and back again. Children burn to death on chariots in the sky or sons fall from atmospheric heights into the sea. These myths seem more nightmarish—on the level of content—than some of Bridle’s chosen YouTube videos.

Instead, I would argue, what’s disturbing here is what the content suggests about how things should be connected. The real risk would seem to be that children exposed to recommendation algorithms at an early age might begin to emulate them cognitively, learning how to think, reason, and associate based on inhuman leaps of machine logic.

Bridle’s inability “to parse out the gap between human and machine” might soon apply not just to these sorts of YouTube videos but to the children who grew up watching them.

[Image: Replicants in Blade Runner].

One of my favorite scenes in Umberto Eco’s novel Foucault’s Pendulum is when a character named Jacopo Belbo describes different types of people. Everyone in the world, Belbo suggests, is one of only four types: there are “cretins, fools, morons, and lunatics.”

In the context of the present discussion, it is interesting to note that these categories are defined by modes of reasoning. For example, “Fools don’t claim that cats bark,” Belbo explains, “but they talk about cats when everyone else is talking about dogs.” They get their references wrong.

It is Eco’s “lunatic,” however, who offers a particularly interesting character type for us to consider: the lunatic, we read, is “a moron who doesn’t know the ropes. The moron proves his [own] thesis; he has a logic, however twisted it may be. The lunatic, on the other hand, doesn’t concern himself at all with logic; he works by short circuits. For him, everything proves everything else. The lunatic is all idée fixe, and whatever he comes across confirms his lunacy. You can tell him by the liberties he takes with common sense, by his flashes of inspiration…”

It might soon be time to suggest a fifth category, something beyond the lunatic, where thinking like an algorithm becomes its own strange form of reasoning, an alien logic gradually accepted as human over two or three generations to come.

Assuming I have read Bridle’s essay correctly—and it is entirely possible I have not—he seems disturbed by the content of these videos. I think the more troubling aspect, however, is in how they suggest kids should think. They replace narrative reason with algorithmic recommendation, connecting events and objects in weird, illogical bursts lacking any semblance of internal coherence, where the sudden appearance of something completely irrelevant can nonetheless be explained because of its keyword-search frequency. Having a conversation with someone who thinks like this—who “thinks” like this—would be utterly alien, if not logically impossible.

So, to return to this post’s beginning, one of the thrills of thinking like a writer, so to speak, is precisely in how it encourages one to bring together things that might not otherwise belong on the same page, and to work toward understanding why these apparently unrelated subjects might secretly be connected.

But what is thinking like an algorithm?

It will be interesting to see if algorithmically assembled material can still offer the sort of interpretive challenge posed by narrative writing, or if the only appropriate response to the kinds of content Bridle describes will be passive resignation, indifference, knowing that a data set somewhere produced a series of keywords and that the story before you goes no deeper than that. So you simply watch the next video. And the next. And the next.

“A medieval cathedral was a sort of permanent and unchangeable TV program”

notredame1[Image: Notre Dame, Paris, courtesy of the Library of Congress].

I’ve always loved Umberto Eco’s observation, from a text he delivered for the opening of the Bibliotheca Alexandrina back in 2003, that “a medieval cathedral was a sort of permanent and unchangeable TV program that was supposed to tell people everything indispensable for their everyday life, as well as for their eternal salvation.”

The carved statuary, the stone ornament, the careful placement of scenes: it was all part of an edited visual narrative that you could return to again and again, like a 3-dimensional comic book or a collection of film stills in the center of your city, a body of symbolic storylines and characters given architectural form.

At the time of these cathedrals’ construction, Eco explained, “manuscripts were reserved to a restricted elite of literate persons, and the only thing to teach the masses about the stories of the Bible, the life of Christ and of the Saints, the moral principles, even the deeds of national history or the most elementary notions of geography and natural sciences (the nature of unknown peoples and the virtues of herbs or stones), was provided by the images of a cathedral.” Then, the sentence I quote above: “A medieval cathedral was a sort of permanent and unchangeable TV program that was supposed to tell people everything indispensable for their everyday life, as well as for their eternal salvation.”

notredame2[Image: Notre Dame, Paris, courtesy of the Library of Congress].

I’ve long been a fan of Eco’s writing, even as a kid growing up in a variety of houses where we seemed to always have a copy of The Name of the Rose stored somewhere in the family-room bookshelves. Well before I could even conceivably read such a thing in full, yet captivated by its original cover art, I’d flip through the book to find descriptions of imposing monastery walls or hidden courtyards, of mirrored libraries concealed inside stone towers. I even memorized, for no particular reason, the monastic hours that Eco enumerates at the book’s beginning.

It’s also a novel, I’d eventually see, full of superb lines: “As I lay on my pallet,” Eco’s monastic narrator at one point writes, “I concluded that my father should not have sent me out in the world, which was more complicated than I had thought. I was learning too many things.” Or: “How beautiful the world would be if there were a procedure for moving through labyrinths.”

notredame3[Image: Notre Dame, Paris, courtesy of the Library of Congress].

But Foucault’s Pendulum—way too quickly dismissed today as a kind of hipster Da Vinci Code—is a novel I’ve read so many times I am embarrassed to admit the number. It’s a book I’ve obsessively traveled with, having read it now in Greece, Berlin, Warsaw, County Donegal, even Beijing. A mere ten days ago, I picked it up again here in New York City, for a variety of reasons, to give it one more spin.

So the news that Umberto Eco died yesterday was both sad and, for me, oddly timed; it’s also news I feel compelled to mention here, for both personal and architectural reasons.

In fact, I was thinking explicitly of Eco when I wrote a piece recently for Cabinet Magazine about rare-book thefts at a French monastery near the border with Germany.

Let’s start with the obvious: the fractal library in The Name of the Rose, a fictional architectural construct that belongs up there with other mythical buildings, from Kafka’s Castle to Daedalus’s Labyrinth or the Tower of Babel. The library, Eco explains, is a fortified architectural complex doubly protected by a weird system of mirrors and winds:

“The library must, of course, have a ventilation system,” William [the book’s non-narrating protagonist] said. “Otherwise the atmosphere would be stifling, especially in the summer. Moreover, those slits provide the right amount of humidity, so the parchments will not dry out. But the cleverness of the founders did not stop there. Placing the slits at certain angles, they made sure that on windy nights the gusts penetrating from those openings would encounter other gusts, and swirl inside the sequence of rooms, producing the sounds we have heard. Which, along with the mirrors and the herbs, increase the fear of the foolhardy who come in here, as we have, without knowing the place well. And we ourselves for a moment thought ghosts were breathing on our faces.”

What we would now call the building’s HVAC system was deliberately engineered to induce the aeolian illusion of other humans. It was a kind of super-sensory burglar alarm for spooking uninvited guests—spatial hauntings in surroundsound.

libraryrose[Image: The fractal stairs of the breeze-haunted library in The Name of the Rose; courtesy Twentieth-Century Fox/Columbia Pictures].

Or take the building that isn’t really a building in Foucault’s Pendulum.

One of that book’s minor characters mentions a house in Paris that is simultaneously more and less than it appears. Parisians “walk by” this house every day, Eco writes, but “they don’t know the truth. That the house is a fake. It’s a facade, an enclosure with no room, no interior. It is really a chimney, a ventilation flue that serves to release the vapors of the regional Métro. And once you know this you feel you are standing at the mouth of the underworld…”

Or consider Eco’s honeycomb of artificial caves beneath the French town of Provins, also in Foucault’s Pendulum and something I have also written about before.

There, an over-excited former colonel explains that “something” has been in Provins “since prehistoric times: tunnels. A network of tunnels—real catacombs—extends beneath the hill.”

Some tunnels lead from building to building. You can enter a granary or a warehouse and come out in a church. Some tunnels are constructed with columns and vaulted ceilings. Even today, every house in the upper city still has a cellar with ogival vaults—there must be more than a hundred of them. And every cellar has an entrance to a tunnel.

In 1894, the colonel continues, two Chevaliers came to the village and asked to be taken down into the tunnels beneath a granary:

Accompanied by the caretaker, they went down into one of the subterranean rooms, on the second level belowground. When the caretaker, trying to show that there were other levels even farther down, stamped on the earth, they heard echoes and reverberations. [The Chevaliers] promptly fetched lanterns and ropes and went into the unknown tunnels like boys down a mine, pulling themselves forward on their elbows, crawling through mysterious passages. [They soon] came to a great hall with a fine fireplace and a dry well in the center. They tied a stone to a rope, lowered it, and found that the well was eleven meters deep. They went back a week later with stronger ropes, and two companions lowered [one of the Chevaliers] into the well, where he discovered a big room with stone walls, ten meters square and five meters high. The others then followed him down.

Eco excelled at these sorts of allegorical details: rooms that served to mask the presence of other rooms, a town built atop a subterranean twin of itself, a library that conceals a parallel, clandestine collection of books, another library somehow tucked inside its very walls, even an island lost on the precise border between today and yesterday.

[Image: Mont-Sainte-Odile; photo via Wikipedia, related to a marginal note, above].

Among many other reasons, Foucault’s Pendulum remains an amazing novel for revealing the seemingly endless extent of one’s own gullibility—that is, the often overwhelming need to believe in or to pursue something, to connect together things you think are signs or clues in fits of irrationality and inspiration, to give your life, your cause, your project, your movement its larger emotional meaning or narrative gravity; only to realize, in retrospect, that these were all just neutral facts of the world you temporarily and needlessly seized upon. They were there when you needed them—or it all made sense at the time.

In fact, the novel contains its own fantastic distillation of this argument in an early scene, set in a Milanese bar. The world, we read, consists of only four types of people: “cretins, fools, morons, and lunatics.” “And that covers everybody?” the book’s narrator asks. “Oh, yes, including us.” I’d risk copying the entire book if I continue on like this in any detail, but I particularly love Eco’s description of “lunatics.” It is an excellent cautionary tale.

A lunatic, he writes, is “a moron who doesn’t know the ropes. The moron proves his [own] thesis; he has a logic, however twisted it may be. The lunatic, on the other hand, doesn’t concern himself at all with logic; he works by short circuits. For him, everything proves everything else. The lunatic is all idée fixe, and whatever he comes across confirms his lunacy. You can tell him by the liberties he takes with common sense, by his flashes of inspiration…”

In any case, as my own tendency to over-re-read Foucault’s Pendulum undoubtedly shows, Eco’s books are perfect for people who are too willing to believe that truth can be found in reading—even if the stories they return to again and again are published not with words at all, but on the façade of a cathedral, in a theological sci-fi of intertwined saints, symbols, and landscapes.

Even if found in the narrative ornament of “a sort of permanent and unchangeable TV program,” as Eco once wrote, these stories we tell ourselves promise a truth it is always wiser to question.

(If you are an American fan of Umberto Eco, there’s a good chance you read his work through the translations of William Weaver, who also passed away recently. Meanwhile, the quotation about cathedrals as TV programs was originally published on Al-Ahram, but is no longer on their site; Nettime has an archived version).

Mineral TV and the Archipelago of Abandoned Shopping Malls

“A mediaeval cathedral was a sort of permanent and unchangeable TV programme that was supposed to tell people everything indispensable for their everyday life, as well as for their eternal salvation.” So says Umberto Eco, speaking at the Bibliotheca Alexandrina, Egypt, 2003.

[Image: Cathedral at Bourges, by Arnaud Frich].

This makes me wonder if everyone on Earth could take everything they know and carve it into a cliffside somewhere – or a mountain – sculpting all that rock into a cathedral; and, then, if they could take that hulking monolith of information and minerals and break it off, launch it into orbit, send it drifting through space… It’d be a kind of moving table of contents for the human species. A knowledge-object.
Would that have a better chance than NASA’s so-called Golden Record, that got sent out with Voyager, of explaining the Earth and human history to distant civilizations?

[Image: NASA’s Golden Record, “intended to communicate a story of our world to extraterrestrials.” The record is really “a 12-inch gold-plated copper disk containing sounds and images selected to portray the diversity of life and culture on Earth,” including the sound of “surf, wind and thunder, birds, whales, and other animals,” and a signed letter from then-president Jimmy Carter. A Menudo video was reportedly removed at the last minute].

Or, instead of demolishing old buildings, perhaps we should detach them from the Earth’s surface and send them into space as lessons for alien species. Like that Michael Crichton novel. You could learn about the Earth by studying its architecture – because the planet flings buildings everywhere. Constantly.
Archipelagoes of abandoned shopping malls pulled slowly toward distant planets. There goes the Mall of America…
A new film directed by Jerry Bruckheimer.