Olfaction, sense of smell, is one of the most primitive senses that we have, in evolutionary terms at least. Smells enhance the functioning of memory and evoke emotions, even when we are not consciously aware of them. Our consciousness of smells is very private and subjective, and hard to describe. Since our sense of smell is generally unreliable and vague, we don’t put much trust in it. Many people are completely unable to smell, and we don’t treat them like they were blind or deaf; in the modern world there is very little handicap in missing the ability to smell. [Unless your job depends on it; Cooks and chefs can lose their jobs if they lose their sense of smell, for example due to a head trauma.]
We do know that there are many species of animal with much more sensitive noses. Dogs are able to perform amazing feats of chemosensing, such as distinguishing identical human twins by their smells. Through our inter-species cooperation, we have had access to canine super-senses for millennia, but only recently have we been able to compare canine senses to our scientific instruments.
What if there was a way to receive the kind of information from the environment that a dog gets through smell, through technological means? For our more accessible senses, seeing and hearing, we have technologically advanced replacements, cameras and microphones, which are already quite small and ubiquitous. Despite many attempts to make technological sensors, trained dogs are still today the best alternative for versatile chemosensing in many practical situations.
Just as we have optical instruments that surpass our eyes, we have developed instruments for detecting chemical compounds that are more accurate than even the olfactory sense of a trained tracking dog. Gas chromatography-mass spectrometry (GC-MS) is the “gold standard” used in forensic research. But such instruments are bulky, nowhere near the portability of digital cameras. For the kind of instant superolfaction that a dog is able to perform, the chemosensor would have to be about the same size as the miniature camera in a smartphone, and work continuously without consuming anything else besides a small amount of electrical power.
Some technological development is of course needed before this becomes reality; The miniaturization of photographic cameras from bulky glass plate devices into digital miniature marvels did not happen overnight either. But the development of cameras and microphones may have also been driven by human familiarity with the senses that we rely on most. Because sense of smell is so vague and impractical in humans, we cannot easily imagine how it would benefit to develop it artificially. [There seems to be at least one ongoing research project for OFET-based e-noses, called PlasticARMPit, which seems technically promising, but the silly name and the idea of using it to sell deodorant shows a serious lack of imagination.]
What could you do with an aerial chemosensor attached to your mobile computing device, as sensitive as the noses of dogs or the antennnae of bees, as programmable as an app?
- Hold it above a served meal, and you will instantly see if it contains any compounds you are allergic to, or just prefer to avoid in your diet.
- With food products and many consumables, you can check the ingredients are what they are claimed to be, just by waving your device above it. This can help avoid product forgeries, and empower consumers to make more informed choices in general.
- Your device automatically alerts you if it detects any harmful substances in your surroundings. If the sensor is generic enough, tuning your device to look for any specific stuff is just a matter of downloading a software signature from an online store.
- An instrument that is mechanical or simulated can also learn smells that would be cruel or impractical to teach to dogs, like deadly nerve toxins.
- An olfactory device can be taught smells by machine learning, and since the results of the training are just a software signature, it can be transferred and downloaded to any similar device. Just as mobile cameras can enable people to become citizen reporters or semi-pro photographers, having an olfactory device with you at all times can enable people to become collectors of airborne chemical signatures.
Advanced artificial superolfaction of this kind is not easy to break into smaller steps. Our natural olfaction is less structured than vision and hearing, which does not help: Whereas digital photographs and audio recordings have the concept of resolution, numbers of samples per second or pixels per area, no such natural structure exists for smells.
There must be some dimensions or components to an odor, like primary colors for vision, or the five dimensions of taste [or six, if capsaicin sensitivity is included], but there are no established names for elementary odor components. In fact, by conservative estimates, there are hundreds of dimensions of smell. Through controlled double-blind testing, the minimum number of different combinations of odorant molecules that the average human can distinguish is estimated to be about one trillion [“Humans Can Discriminate More than 1 Trillion Olfactory Stimuli”, Bushdid et al., Science vol 343, March 2014. The “trillion” in the title is short-scale, the estimate they present is 1.72·10^12. To put this number in perspective: if you were to sample a distinct smell every one second of your life, you would die before experiencing 1% of all possible smells.]
“Primary odors” must be even rarer in nature than primary colors, so it makes no evolutionary sense to name them as abstract entities, separated from the objects that produce them. We name things when we come across them, so the names of smells are also the names of the things that smell like them. Smells are harder to imitate than other sensory properties, so we easily identify them with the “essence” of a thing [insert Shakespeare quote here].
The subjectiveness of the aesthetics of scents and flavors is the reason there can be “secret” recipes for products with fragrance or taste. While the ordinary consumer can appreciate the flavor of prepared seasonings and sauces, it is an impossible task for the average taster to break it down in his mind into a recipe to recreate it. [Of course there are chefs who say they can do exactly that, taste a dish and tell exactly the ingredients and recipe used. This claim is still subjective, and like any acquired skill, subject to continuous training and calibration.] It is because the “qualia” of olfaction run in the trillions, that scent can augment identity and memory. But due to the vast number of possible smells this augmentation is in one direction only. We can recognize hundreds and thousands of scents when we come across them again, but it is hard to “imagine” smells in the mind, or recall or describe them from just memory. We can read descriptions of what dodo meat tasted like, but unless we bring the species back, there is no way to recreate the experience. This rich subjective tapestry of scents and flavors goes well with human omnivorousness and curiosity: despite having less olfactory sensitivity than most other animals, we seem to be the the only ones who regularly cook and season our food, and find pleasure in combining several elements to create new sensory experiences.
Since we don’t understand the coding of odors, efficient digital representation, like we have for digital video and audio, is not a very meaningful goal for artificial olfaction. And if we want a detector to be as general as possible, and more sensitive than our native sense, we should try to retain as much detailed information as possible in the representation. This is why it makes little sense to implement artificial olfaction in the manner of a phonograph, or a camera: as a series of recording samples with no further analysis. The trace chemical composition of air must be analysed on the spot, with access to a large database, using statistical models capable of parallel recognition, based on uncompressed detailed molecular measurements and high sensory data bandwidth locally. In other words, artificial superolfaction requires a connected smart device to be useful.
Because the design space of possible molecules is so vast, the expedience of evolution means that our sense of smell has most likely been optimized to detect the kinds of organic molecules that normally occur in biochemistry. Many of the chemicals that we naturally consider malodorous serve as warning signs, something to avoid for our own health. Of course we also attach more private emotional associations to smells, but more universal preferences seem to exist as well, encoded somewhere deep in our biochemical program code.
An artificial chemosensor does not need to be optimized towards organic molecules. Pretty much anything that produces airborne particulates and molecules could be analysed in detail with artificial olfaction. Modern environments have increasing amounts of artificial chemicals that don’t occur naturally in nature. They could have long-term health effects, but our biological senses do not smell them as harmful to us, if they smell them at all.
Just as with any technological advancement, universally available superolfaction will not be an unqualified blessing. Ubiquitous devices capable of revealing the unseen chemistry that surrounds us every moment of our lives would increase transparency and safety in many ways. But this kind of capability has dangers as well, if used by unscrupulous individuals. Just as ubiquitous portable cameras and microphones have diminished the normal expectations of privacy, ubiquitous chemosensing could be used to diminish privacy even more. Facial recognition as currently implemented requires large databases which can (in principle) regulate their access. But the capability for everyone to sample their surroundings and record chemical signatures of the people they meet, provides an astonishing capability for “stalking” them later. Private or illegal databases of chemical signatures might pop up, which could be used to create virtual “bloodhound mobs” to target individuals for whatever reason.
Our biological selves operate at the level of biochemistry. In theory it would be possible to technosniff an individual and instantly detect:
- what they have eaten recently (past few days even)
- where they have been, and with whom
- some medical conditions they have, or medications they are on
This of course requires that the device is programmed to detect such chemical signatures. And of course, even deeper analysis would be possible by taking solid samples, small pieces of dead skin and hair, and analysing them later by aerating them in front of the device.
[The title of this post obviously refers to a book by Nick Bostrom. It was not my original intention to review books on this blog, but I will make an exception here.
Now, for something completely different … ]
Comparative Review of Wiener and Bostrom
Cybernetics: or Control and Communication in the Animal and the Machine by Norbert Wiener (1894-1964) was something of a surprise bestseller in 1948, much like Superintelligence: Paths, Dangers, Strategies by Nick Bostrom (1973-) was in 2014. Both books were also published in multiple, revised editions by their respective authors; Wiener also published a more approachable book, The Human Use of Human Beings: Cybernetics and Society, in 1950 as a less technical book on the topic, more intended to the lay public. In 1964, the year of his death, Wiener returned to the topic with God & Golem, Inc.: A Comment on Certain Points Where Cybernetics Impinges on Religion. [I will treat the three books by Wiener as essentially three parts of the same work in the following. From Wiener, I have used the second edition of Cybernetics from 1961, the revised Human Use from 1954, and God & Golem from 1964. For Bostrom’s Superintelligence, I have used the second edition from 2016.]
While both books can be characterized as cross-disciplinary efforts, situated somewhere between philosophy of engineering and futurology, they seem to cover much of the same ground; in particular the relationship between humans and the powerful machines they have created, or will create in the future. With continuing advancement of technology this topic remains at least as important as it was 70 years ago.
Exploring these two authors in parallel in 2018 has been a very enlightening experience in many ways. The closeness of the topic highlights both the similarities and the differences between the approaches that Wiener and Bostrom have made. The two authors are largely ignorant of each others work; Wiener was of course not influenced by Bostrom, but neither does Bostrom seem directly influenced by Wiener’s writings. [Wiener does get mentioned in a footnote on AI pioneers, Superintelligence pg 326]
Both authors have varied educational backgrounds. Wiener studied mathematics, zoology, and philosophy before his doctorate in mathematical logic at Harvard in 1912-1914. Bostrom explored physics, mathematics, logic, and computational neuroscience before his doctorate in philosophy in 2000. This interdisciplinary curiosity remained as a strong creative force for Wiener, who attended seminars, conferences in wide variety of fields and initiated discussions with experts in many fields throughout his career. He was consistently interested in physiology, and collaborated closely with Walter Cannon (who coined “homeostasis”) and Arturo Rosenblueth when forming his ideas. Bostrom has been active in many forward-thinking interdisciplinary groups, like the Transhumanists, and has been influenced by the ideas of Eric Drexler and Anders Sandberg [as well as Eliezer Yudkowsky, who is referenced often in Superintelligence].
The backgrounds have dissimilarities relevant to their work as well. Wiener’s insight into society and communication was formed after living through two world wars, and seeing the political landscape change. It could be that being part of a minority (agnostic jew) affected his view on society and politics as well. In contrast, Bostrom’s home country, Sweden, has not been directly involved in a war in two centuries, and until recently has not had much ethnic or cultural diversity. [The above should not be construed as an apology for either author’s views, but as an attempt at placing them to some context.] [Even though Sweden did not fight in the wars, they did design and manufacture weapons for them. In Swedish language, the word commonly used for powered missiles is “robot”, introduced by Chief Engineer Tore Edlén in the 1940s.]
Even though they handle similar topics, the styles of the two authors could hardly be more different. Wiener’s prose is pithy but rambling; it is piecewise smooth and continuous everywhere, but each chapter progresses like a maximum entropy random walk. An amazing array of topics gets randomly covered in just a few pages, but the delivery never seems hurried or impatient. There are few footnotes in Wiener’s text, but there are many small allusions to outside literature. Example:
The chance of the quantum theoretician is not the ethical freedom of the Augustinian, and Tyche is as relentless a mistress as Ananke. [Cybernetics, ch. 1]
In spite of the concise style, Wiener’s rhetoric gets quite passionate at times, for example towards the end of God&Golem, where he compares “gadget worshipers” to the sin of sorcery, but his target is not clear and the implications are veiled. In a warning tone, he cites Goethe’s The Sorcerer’s Apprentice, The Fisherman and the Jinni from Arabian Nights, and The Monkey’s Paw by W.W.Jacobs, as timeless examples of assumed mastery revealed as tragic illusions of control.
Bostrom’s prose is less deep, and as he laments in his introduction, peppered with weasel wordings (“perhaps”, “likely”, “could” etc.) that act to dilute his message. Classics education is today marginal, and Bostrom does not bother to reference ancient myths; his audience would not get the references anyway. But he does start his book with an unfinished fable written in a timeless style about sparrows and owls. The cover of the book also depicts an owl, but the half-finished fable just raises more questions than it answers. Does it refer to the owl of Athena, the virgin goddess of strategy and wisdom? Or is the owl just the biological owl, stealth predator of a specialized niche? How did Skronkfinkle the sparrow lose his eye? Would a cuckoo’s egg be more to the point than an owl’s egg?
Where Wiener’s voice is piecewise smooth, Bostrom’s arguments often have rough and jittery edges, which he manages with the forementioned weasel wordings, as well as by oscillating between many alternative scenarios in parallel. His anxiety seems quite genuine, he really wants to convince the reader of his fears for humanity’s future; but like the time-anxious White Rabbit from Alice’s Adventures, he has a tendency to lead the reader down rabbit holes. Although they seem far-fetched, his scenarios are not the usual thought experiments that philosophers like to employ to illustrate an idea or a contradiction. The extreme hypotheticals that he presents one after the other are meant as actual possibilities of future development of mankind and its instruments. But the book is not really futurology either, more like a pamphlet with an almost exhaustive list of talking points. [I will not use my unpaid time to tackle most of them in this review.]
For convincing his reader, Bostrom’s rhetoric comes a bit too close to a sales pitch at times; for example the false dichotomy of extreme alternative futures [Drs Pangloss and Strangelove?]. The “limited time” anxiety is also a known method to force people to make quick and bad decisions. The sheer exhaustion of the detailed scenarios and original invented terminology [partly overlapping other disciplines, like “oracle”, “singleton”] also serve more to confuse than to enlighten the reader. Whereas Wiener could devote pages to deriving mathematical formulas as well as gently educating his readers on how machines and animals actually work, Bostrom is not using his pages for basic education. He assumes that the “adults” among his readers are capable of searching the Internet for more details, if they need them.
The word “cybernetics” itself seems to have gone out of fashion, but derived terms like “cyber-crime” and “cyborg” make appearances in Bostrom’s book [even “cyborgization”, pg 55]. The word “cyborg” was coined in 1960, while Wiener was still alive, but I don’t know what he thought of the portmanteau. The original “cyborg” was intended as a means of overcoming biological shortcomings in the context of manned space exploration, and was based on extended homeostasis. It immediately became popular in science fiction, and today it seems to mean any merging of artificial and biological prostheses, in space or elsewhere, with or without homeostasis. [A further degeneration are the fictional Borg of Star Trek: The Next Generation, where only the letter B remains from the word “cybernetics”.]
Since the original meaning of cybernetics has become so muddled, I will take a brief detour to try and filter out the original meaning from the accumulation of background noise.
Control Problem vs. Control Theory
Systems with realtime regulation, either designed or natural, have been studied for quite a while. In the Alcibiades dialogue, as recorded by Plato, Sokrates refers to the art of the pilot [“kubernetikhe”] when discussing how to be a good ruler of men [Alcibiades, 125d]. This later became the commonly used metaphor for government in general, especially in the latin translation, “gubernatoria”. Clerk Maxwell started to formalize the theory of self-regulating devices in the 19th century, in an article on “governors” of steam engines. Partly to avoid confusion with civic government, Norbert Wiener brought the word back to its greek roots with “cybernetics”. [Although to add to the confusion, similar back-formation was made in 1834 by André-Marie Ampère to describe a science of civil government.]
Basic mechanisms of cybernetics, feedback and forward amplification, were already intuitively known in antiquity, exemplified by the way the pilot controls the rudders from the back of the ship, using quite small movements to turn a large ship. But their detailed formalization could not be made until a theory of realtime signaling or communication was available. Pure information theory is not enough for cybernetics; realtime latency and bandwidth must also be characterized.
The necessity of feedback is due to the inherently incomplete knowledge of the controlling agent. Instead of a copy of the actual world it acts in, the control agent has just a map, a simplified model of the world that it can internally refer to. However accurate its internal model may be, no agent has a reliable map of the future. An agent with telos, acting in a world with incomplete information, needs to know whether it has succeeded or failed, or how close or how far it has come to reaching its goal.
Here is how Wiener puts it:
[…] control of a machine on the basis of its actual performance rather than its expected performance is known as feedback [Human Use, pg 24]
[…]The simplest feedbacks deal with gross successes or failures of performance, such as whether we have actually succeeded in grasping an object that we have tried to pick up, or whether the advance guard of an army is at the appointed place at the appointed time. [pg 59]
It is important for the agent to get accurate feedback as soon as possible, so that it can correct its internal model and avoid wasting resources at a faulty target. The fundamental delay of the round-trip time from taking an action and receiving the results of that action is called latency. The concept of latency is very well known to players of online multiplayer games, where having too much network latency towards the game server puts even the best players at a disadvantage. Single-player games and virtual reality helmets also have maximum latency requirements for the feedback loop between moving a controller and seeing the results, to make the virtual world seem real to the player, and not cause nausea. Although it is desirable to minimize latency, it cannot be completely removed in physical systems, ultimately due to the maximum transfer speed of information, the speed of light.
Oscillation due to feedback latency is a realtime phenomenon which can be seen in many natural systems. In addition to intention tremor described by Wiener, cycles of boom and bust in economy, cycling of political parties into opposition and out of it, can be seen as large-scale examples of delayed feedback oscillation. In a wide sense, even natural selection in biology can be seen as a form of feedback from the environment, operating at population scale with generational latency. [Plato also describes the “kuklos” of history, a larger, slower oscillation of socio-historical forces where individual human lives are just hapless passengers.]
The living animal takes feedback through the senses. Innate senses even seem to have built-in valuation heuristics, indicating short-term success and failure through pleasure and pain. Pain is perhaps more fundamental, and required even in the safety-conscious modern world for long-term survival. Humans have complex emotions, able also to imagine and empathize with pleasures and pains not currently present. For an agent wishing to act in the real world, the contingency of receiving either pleasant or painful feedback is unavoidable. Medical and technological means can short-circuit the feedback loop, replacing real pleasures and pains with virtual ones. This usually results in addiction.
Even simple animals with innate senses move towards pleasure and away from pain, seeking local optimums of their environment with little need for conscious memory. These pleasures and pains are rarely experienced with absolute idempotency, however; when the need has already been satisfied, less pleasure is received from e.g. eating more food. This natural lessening of urgency can also lead to orbits or oscillations in long-term patterns of behaviour, with interspersed periods of attraction and avoidance. In complex social situations the multivariate fears and desires, avoidances and attractions, of individuals are signaled in various ways, leading to complex guessing games of what thing will become desirable or undesirable in the future. This kind of marketplace signaling of buying and selling, delaying and preparing, works ultimately on probing and then getting feedback; there is no a priori way of deciding market prices for non-essentials [which do often display oscillations at many interval scales].
A modern version of the ship pilot metaphor is driving a car. Forward amplification of control signals is called power steering and power brakes; feedback takes place through a multitude of instrument displays, but perhaps more importantly through the driver’s vision of the road, as well as feeling the accelerations via the seat of his pants. When a driver presses a pedal, the complete feedback loop progresses through the wheels, contacts the surface of the road, and makes the whole vehicle accelerate or decelerate. When the contact is temporarily lost between the wheel and the road, for example when driving over a slick, the driver feels the loss of traction as directly as if his own feet were slipping.
With self-driving cars having the potential to prevent millions of accidents as well as transport goods more economically, they seem on the verge of becoming a reality. Enhanced computer vision systems have already become standard for assisting the driver with lane drift or emergency braking. In these scenarios the driver is still sending all the forward signals, but the computer may censor them for safety, and preferably indicate to the driver that he is about to do something stupid. Eventually it may be that the computer vision and safety logic gets advanced enough that the forward signals will also be triggered by a computer instead of the human. There could still be an overlap period when a human driver is needed as a backup, in case the machine makes a mistake. But switching from automatic driving back to human driver is not as easy as a driving instructor taking over the controls from a student [which is itself not that simple, or foolproof]. The driving instructor continuously looks for signals from the human student, to see when the driving student is about to do something dangerous. These signals are non-verbal, biological signals that humans are evolved to interpret as gestalts, such as direction of the eyes or body language. What kind of signaling should a self-driving car be designed to use, to show to the human backup driver that it is about to do something stupid and the human should take over? [Marketing will of course claim that their self-driving car never behaves stupidly, but please sign all these waivers saying that you are prepared to take over the wheel in case of emergency.]
The theory of games, founded by von Neumann, has the idea of “sums”, valuations which the players use to formulate their goals of attraction and avoidance during play. In both the cybernetic model and the game theoretic model there are agents trying to achieve goals with incomplete and delayed information; one difference is that in game theory the incompleteness of information is caused by agents purposely not revealing their strategies to the other players, and the delays are imposed by the rules of play. In the cybernetic model the incompleteness of information is a result of entropic noise, an unescapable fact of the world, and delay is ultimately due to the maximum speed of information transfer, another unescapable fact of the world.
It may be down to the individual temperament of individual thinkers, how much uncertainty they are prepared to live with, that informs their intuition on epistemic humility. Here is Wiener’s characterization of the two sources of incomplete knowledge:
The scientist is always working to discover the order and organization of the universe, and is thus playing a game against the arch enemy, disorganization. Is this devil Manichaean or Augustinian? Is it a contrary force opposed to order or is it the very absence of order itself? The difference between these two sorts of demons will make itself apparent in the tactics to be used against them. [Human Use, pg 34]
Most, if not all, of the descriptions of the superintelligence in Bostrom’s book cast the AI decidedly into the role of the Manichean devil. At least in this book, Bostrom does not seem much interested in how intelligence (natural or otherwise) arises, only in what kind of opponent it will turn out to be when it inevitably does.
As an example, Bostrom presents “domesticity methods” for containing what he calls “oracles”:
For example, we might stipulate that it should base its answer on a preloaded corpus of information, such as a stored snapshot of the internet, and that it should use no more than a fixed number of computational steps. [Superintelligence, pg 178]
Using a preloaded snapshot as the only input of a process removes entirely what Wiener calls feedback, the ability of an agency to know if it has succeeded or failed in its goal, or how close it was to achieving it. Instead of a using a pilot mechanism, Bostrom expects AI to work by dead reckoning.
The best way that we currently know how to build an “oracle” like this is machine learning: basically a process of curated input with a feedback valuation function telling how close the agent was to succeeding. Rather than removing the valuation feedback completely when learning is considered “finished”, a less severe way of isolating the agent would be to add artificial latency to its inputs. [Latency management is actually used in high-frequency trading, to ensure a level playing field between automized agents.]
Information As Message – Contingency And Interpretation
In one of the more speculative passages in Human Use, [ch. 5, “Organization as the message”] Wiener poses that it would be theoretically possible to transmit a living organism, even a human being, via telegraph. This is handled both as a thought experiment to clarify metaphysical intuition about human identity (discussed later in more detail by Derek Parfit), as well as an exploration of the developing field of information.
A pattern is a message, and may be transmitted as a message. How else do we employ our radio than to transmit patterns of sound, and our television set than to transmit patterns of light? It is amusing as well as instructive to consider what would happen if we were to transmit the whole pattern of the human body, of the human brain with its memories and cross connections, so that a hypothetical receiving instrument could re-embody these messages in appropriate matter, capable of continuing the processes already in the body and the mind, and of maintaining the integrity needed for this continuation by a process of homeostasis. [Human Use, pg 96]
[…]This takes us very deeply into the question of human individuality. The problem of the nature of human individuality and of the barrier which separates one personality from another is as old as history. [pg 98]
[…]Let us then admit that the idea that one might conceivably travel by telegraph, in addition to traveling by train or airplane, is not intrinsically absurd, far as it may be from realization. The difficulties are, of course, enormous. It is possible to evaluate something like the amount of significant information conveyed by all the genes in a germ cell, and thereby to determine the amount of hereditary information, as compared with learned information, that a human being possesses. In order for this message to be significant at all, it must convey at least as much information as an entire set of the Encyclopedia Britannica. In fact if we compare the number of asymmetric carbon atoms in all the molecules of a germ cell with the number of dots and dashes needed to code the Encyclopedia Britannica, we find that they constitute an even more enormous message; and this is still more impressive when we realize what the conditions for telegraphic transmission of such a message must be. Any scanning of the human organism must be a probe going through all its parts, and will, accordingly, tend to destroy the tissue on its way. To hold an organism stable while part of it is being slowly destroyed, with the intention of re-creating it out of other material elsewhere, involves a lowering of its degree of activity, which in most cases would destroy life in the tissue. [pg 103]
The notion of transporting people as signals is of course fascinating, and a fictional version of it became famous in the TV show Star Trek, ten years after Human Use was first published.
But just as the next generation of Star Trek took the ideas of the original show even further, so has the idea of people as information been taken further in Bostrom’s time. Bostrom spends a lot of effort on the simulation argument, questioning the distinction between a real physical process and a computational simulation of it. Again, Bostrom takes the wildest of thought experiments at face value, even ballparking the number of human lives that can be simulated using the “spare” bits of matter and energy scattered around in space. Since this is obviously a very high number, compared to the number of humans that have ever lived, the probability is high that you (the reader) are in fact a computer simulation instead of a real physical person.
This kind of a thought twister is of course a daily exercise for the professional philosopher, who can entertain six impossible thoughts before breakfast, but in the less mentally flexible population it can take a more sinister guise: a mindgame pitch for a cult, replacing outdated ontologies of phairies and aliens with “science” and “computers”. From the looks of it, there are plenty of people on the Internet quite serious about an upcoming digital rapture (of course only for people who have demonstrated proper subservient good-will towards our future robotic overlords).
In the area of theory-making, intellectual humility is not always proportionate with the significance of new theories. Nor is it always the people who first formalize a theory who foresee their most important future significance. We must be very careful in our fascination of life as information, as a pattern. Could this become an excuse for lazy thinking, in a similar way that the transporter of Star Trek became sometimes an excuse for lazy writing?
Today, information is everywhere; digital, ubiquitous, mundane even. But just because it is everywhere and easily available, does not mean that its nature is universally understood. We treat information like a commodity or a fuel, flowing like substance between containers, measuring it in bandwidth and storage capacity, compressing it with algorithms to save storage space. But in the communication theory of information, every last bit of information represents a contingency, a choice between two equally likely possibilities. Such a thing has no independent existence, it is always relative to a specific context.
To cover this aspect of communication engineering, we had to develop a statistical theory of the amount of information, in which the unit amount of information was that transmitted as a single decision between equally probable alternatives. This idea occurred at about the same time to several writers, among them the statistician R. A. Fisher, Dr. Shannon of the Bell Telephone Laboratories, and the author. Fisher’s motive in studying this subject is to be found in classical statistical theory; that of Shannon in the problem of coding information; and that of the author in the problem of noise and message in electrical filters. Let it be remarked parenthetically that some of my speculations in this direction attach themselves to the earlier work of Kolmogoroff in Russia, although a considerable part of my work was done before my attention was called to the work of the Russian school. [Cybernetics, Introduction, pg 10]
The contextual, contingent and relative nature of information-as-message is what enables techniques like compression algorithms and cryptography: without a correct interpreter, or the correct key, information is just meaningless random noise, an incoherent stream of bits. This processing is purely statistical. As Shannon puts it: “[…] semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages.” [C. E. Shannon, 1948, emphasis in original]
Part of the allure of information-as-existence comes from developments in modern physics, as well as trends in philosophy of mind. Rather than giving up on metaphysics as unsolvable, many people seem attracted to minimalist ontologies, often some version of “everything is just information”, or “everything is mathematics, nothing else exists” [one proponent of the latter is Max Tegmark, a compatriot of Bostrom’s]. [Unfortunately, the concept “every/all” always holds the seeds of a paradox.]
Another explanation for the category mistake is the natural desire for an ultimate terra firma, a fundamental something that everything else can (at least in principle) be mapped to; the original language of the cosmos at its most profound level. If only we could access such a philosopher’s touchstone, why, we could (in principle) map the semantics of all that we can say or see in terms of it. Unfortunately that is just not the nature of mathematical information theory. The only touchstone is noise, its statistical properties in time and space, and not some universal a priori Rosetta stele.
Information(-as-message) is meaningful only for agents with incomplete knowledge of the world. The cybernetic agent does not know everything, so it opens its receptors for incoming data, pokes things to see how they react, oscillates to find out what moves and what doesn’t. The perception of time as flowing is also a common source of incomplete knowledge; although statistically at the large scale the world evolves with mathematical certainty, the small scale agent perspective includes significant contingencies, things that may or may not happen, that have consequences for the agent. The semantics that arise from all this are those of survival, of countless trials and errors recorded in genetic memory.
This seems to be a crucial difference in the thinking represented by Wiener and Bostrom, and is repeated in many of the scenarios they discuss. One example is “breaking out of the simulation”, where an intelligent agent, either natural or artificial, is able to deduce that its senses are not dealing with a fundamental reality, but rather a virtual reality simulated by some process. An earlier variant of this thought experiment was famously given by Rene Descartes: a brain in a vat with a demon intercepting all nerve inputs and outputs. Unable to accept limits to human knowledge and intellectual capacity, Descartes posed metaphysical dualism as a way for humans [and only humans, not other animals] to bypass the limited/incomplete nature of existence as a physical agent in a physical world.
Bostrom’s loose definition of intelligence does not rule out a possible ability to access reality at a more fundamental level: “[…] superintelligence (or even just moderately enhanced human intelligence) would outperform the current cast of thinkers in answering fundamental questions in science and philosophy.” [Superintelligence, pg 315] This ability has nothing to do with more accurate sensory inputs, it is just a result of “more” thinking, even in complete absence of feedback.
Decidability And Algorithmic Agnosis
All agents are limited in knowledge in a fundamental way. The currently accepted limits of a priori logical certainty are expressed by the Church-Turing Thesis. This is a theoretical boundary on what facts can be proven from a set of given inputs to a theoretical model. Both the model and the set of inputs could be considered “a priori”, in the sense that their empirical validity is not in question; the only thing that the Church-Turing thesis is interested in is the breadth of knowledge that can be logically inferred from a given set of assumptions.
Since Euclid, mathematical proofs have been reduced into rigorous, elementary steps, so simple that even the most passionate skeptic can approve each of them. Combining such steps using strict rules of logic makes the complete proof acceptable, a secondary fact derived from previously accepted facts. The complexity of mathematics is that it is far from obvious what secondary facts are both true and provable.
Computers were first built for mathematical purposes. A traditional computer performs each elementary operation mechanically, and follows the rules of logic much more strictly than any human mathematician could. The reason we trust the results of a mechanical computation is the same as for accepting a mathematical proof: each elementary step is simple, and the rules for combining the steps is followed rigorously. Computer have pushed human error out of the details, into the bigger picture: inputs and models. Even so, we are still stuck in the centuries-long habit of algorithmic decomposition as a model of ascertainment: if we trust each atomic step of a process, and the connection that each atomic part has with its neighbors, we can trust the results of the computation equally. [This is why so much emphasis was put on “computer error” in the early days of computers. Today computation is so cheap that we can just run our calculations redundantly, in the rare case that random hardware faults make a difference.]
Kurt Gödel was the first to show that any sufficiently non-trivial theoretical model is logically incomplete: that is to say, the complete scope of a priori knowledge cannot be known a priori. This untidyness in the systems that provide us with the most exact and universal knowledge available, mathematics and logic, seems to expose an epistemic loophole of some kind, a blind spot within the foundations of metaphysics.
In the same breath as we marvel at this fundamental incompleteness of logic, we note that for practical use of mathematics and logic, it matters very little. Before the deciding of decidability, we must already have chosen our theoretical model and other a priori assumptions. It makes a kind of sense that a limited set of choices can only lead to a limited amount of possibilities. But as finite agents, what choice do we have?
For a cybernetic agent, risks and probabilities are much more useful than limiting activities to just provable a priori certainties. A cybernetic system is a pilot mechanism, working by default outside of terra firma, relying on environmental feedback, continuously updating its maps with fresh data as it navigates the flows of winds and seas. It does not matter so much what can be done with a fixed model and input; rather how fast the continuous stream of input data can be processed and integrated into the live model, while staying within the parameters of a somewhat negotiable set of goals. Life in the pilot seat is a world of continuous choosing; there is usually some freedom of choice, but rarely freedom from not taking any risks.
Finding some meaning within this world of choice is not possible without true contingency. Many thinkers have gone on record stating that the freedom of will that people think they have is an illusion; In the true picture of physics, they say, the world is completely deterministic, and can in principle be predicted down to the least detail, including the choices any single person makes at any single instant. Of course we don’t have such a coherent theory of everything yet, and we cannot access the initial conditions of the Universe, but these things must exist nonetheless, even though we currently don’t know them. Right?
But if the world is nothing more than the mathematics that describe its regularities and irregularities, it must be either trivial, incomplete, or infinite in its axioms; in accordance with Gödel, Church, and Turing. If physics is not trivial, then its determinism is something that a finite mathematician, biological or mechanical, can never be able to predict with complete accuracy and certainty.
Philosophy of Engineering: Limits
Many people are passingly familiar with the Philosophy of Science. Maybe not by that name exactly, but at least the important questions that fall in its area of responsibility will sound familiar: What are the limits of scientific knowledge? How can we be certain that a theory is correct? Widely accepted answers to these questions are readily available.
When it comes to Philosophy of Engineering, however, embarrassing gaps of familiarity appear. What are the limits of man’s technological abilities? Is technological progress inevitable? What are the most popular ethical frameworks used in engineering? No authoritative answers seem to be available, least from engineers themselves.
But what exactly is the relationship between engineering, technology, and science? Science can give definite answers about the absolute limits of human possibilities, and the limits of engineering must fall inside them. But there are subtleties to these questions, related to actual human collaborative capabilities, as well as economic and political contingencies of societies in general.
For Bostrom, technical and scientific development seems a matter of discovery: the more effort is made, the earlier a result can be found. “Think of a “discovery” as an act that moves the arrival of information from a later point in time to an earlier time. The discovery’s value does not equal the value of information discovered but rather the value of having the information available earlier than it otherwise would have been.” [Superintelligence, pg 315, from the context “value” means instrumental, strategic value]
Wiener had a fairly unique perspective in that he was a classically trained philosopher who went on to have a long career at MIT. He never called himself a “philosopher of engineering” [the word “engineering” probably did not have the right connotations for him], but he did dictate a draft for a book that is relevant to the human activity that we now choose to call engineering. It was posthumously published as Invention: the Care and Feeding of Ideas in 1989.
The largest computer that Wiener envisions in his texts is the size of a skyscraper, with as many transistors as the human brain has neurons [God&Golem, ch 6]. This is of course nothing compared to the computers imagined today. Although the minimum size of a working transistor has been shrunk, yet we find it necessary to envision even larger computers. In fact, what Bostrom calls the “cosmic endowment” of mankind is all the accessible matter and energy in the Universe essentially turned into a giant distributed computer. What kind of calculations it will be used for is the most important decision that mankind will have to make, according to Bostrom.
The general idea would be to send out small but super-smart robotic agents out into the galaxy, where they would self-replicate using any raw material they come across, multiply and spread out in all directions based on some prewritten instructions. Pretty soon (in post-human terms) the sky would be filled with Dyson Sphere Computroniums, all just to simulate endless trillions and trillions of happy human subjects (remember, Bostrom cannot distinguish simulations from real persons).
But if life on Earth is nothing exceptional, and billions of years have passed since Big Bang, why have no other space civilization done the same thing? Why are our telescopes not detecting Kardashev scale objects in the endless sky? While physicists are good at estimating the physical makeup of stars and galaxies [well, apart from Dark Matter], it is another thing to look at that matter as modeling clay, to measure the available plasticity it has to be molded into useful shapes. While there is no such general theory of physical plasticity yet in existence, there are some things that can be said about the prospects of space engineering.
The average density of visible space is of a very thin gas. Although there are sizable chunks, islands of solid matter even, in the larger scale these chunks behave like particles of a gas. What we see as “fixed” stars in the night sky are nothing of the sort, our lives are just too short to detect their constant motion.
Arranging the particles floating in this thin gas of space is not dissimilar to Maxwell’s demon sorting the gas particles in a chamber to hot and cold compartments. Arranging all or even most of the matter floating in space is impossible for much the same reason as Maxwell’s demon is, even if the demon is allowed to use self-replication to accomplish its task.
To clarify, I do believe that self-replicating machines are a possible way to “seed” nearby systems by diffusion, if the machines are small and capable of adapting to local conditions and power sources. (In fact, this is a possible origin of biological life on Earth, with the self-replication and adaptation still ongoing in the form of biological evolution.) But I do not think that this method can be used to produce Kardashev scale artifacts, at least stable ones. At this point in the evolution of the Universe, the stable large-scale shapes are the ones that we see out there. [There can still be some interesting “structures” that we do not normally see. For example, astrospheric and galactic current sheets are a fairly new finding.]
The part of our “cosmic endowment” that is plastic enough for us to easily mold with any future technology is not going to be very large portion. And like anything we build or grow, it will not last forever. The statistical nature of thermodynamics means that the large-scale shape of the future of the Universe remains unmovable. Parts of intelligent life may survive for very long times in one form or another, not in any kind of total dominance of all matter and energy, but in the nooks and crannies of the Universe, as statistically insignificant islands of order in a vast sea of chaos.
Yet we may succeed in framing our values so that this temporary accident of
living existence, and this much more temporary accident of human existence, may be taken as all-important positive values, notwithstanding their fugitive character.
In a very real sense we are shipwrecked passengers on a doomed planet. Yet even in a shipwreck, human decencies and human values do not necessarily vanish, and we must make the most of them. We shall go down, but let it be in a manner to which we may look forward as worthy of our dignity. [Human Use, pg 40]
Philosophy of Engineering: Ethics
In the widest sense, engineering or technology is the ability to purposely change some part of the physical world in some way. It consists of techniques, skills, arts, tools, and crafts, which make use of the available plasticity in the physical world to effect the changes. It can be as simple as collecting seeds from plants and planting them in a different place, or as complex as collecting pieces of rock from an asteroid and bringing them back.
It is perhaps nice to imagine that the changes made by technology are expressions of human spirit or power, the will of a genius imposing his order onto a chaotic world. In reality, the forms of plasticity that exist in the world usually come with costs, and take a lot of trials and errors to master. Technology is a world of engineering trade-offs, and if it becomes “indistinguishable from magic” for some observers, it is because someone has purposely hidden all the wiring. [Incidentally, some of the mechanical inventions of Leonardo were inspired by his secret dissections of human and animal carcasses. More recently, neural network classifiers were also inspired by studies of anatomy; the “exposed wiring” of Nature’s engineering.]
Is the creativity of a poet the same kind of creativity that an engineer has? Are the thousands of programmers writing the software for your self-driving car artists? From current parlance, software companies are mainly looking for “coders”, not “rock stars” or other divas. Writing software is probably the most flexible form of expression yet invented, and is available to pretty much everyone willing to learn. Many have used their creative impulses to write open source software, outside of any established software company. Within technology companies, creativity can also flourish with “skunkworks” budgets, which are unfortunately becoming a rarity. [Many of the most widely used software tools today, from Unix to C to Linux, originated either as skunkworks or as open source projects.]
In a way, the layering of “software” on top of hardware represents the ultimate idea of plasticity as pure, noise-free signals, communicated as intangible information. But it also reminds us that there are two sides to plasticity, that something must remain unchanged (hardware) even when all that can be changed (software), changes.
Even putting aside the unavoidable trade-offs, mankind is not today in a position to plan its use of technology very well. The collective society seems mostly drawn by economics and marketing, when it comes to applying and developing technology. If technology is about using the inherent plasticity around us, and discovering methods to change the physical world in more ways, should these changes not be planned ahead, instead of reacting to one technology related crisis after another?
How does humanity as a whole make plans and choices? A possibility of meaningful one-to-one feedback sessions between all of the billions of people on Earth is not feasible [even less between 10^58 simulated humans]. Useful information has a way of diffusing through even large societies and cultures, via marketplaces of goods and ideas. The existence of these global markets is also an enabler for the progressive development of technologies, through market competition. [It should be noted that market competition as an engine of progress is not infallibly qualitative, as it deals almost exclusively in quantitative terms.]
Many systems of rules-based rule, both ancient and modern, have been developed to govern individual nations and other institutions with some measure of fairness [and of course with powers to enforce the rules themselves]. Some have searched for a universal theory of ethics by which civic government could be run deterministically, like a programmed machine. Although people and animals are capable of feeling pain, a simple calculus of total suffering at one particular moment in time is not a very satisfactory measure of success or failure, but it has been suggested by many as a possible way arrive at universal ethics.
As mentioned previously, experiencing pain is itself a part of the short-time cybernetic guidance system that evolution has built inside us. We are quite capable of ascending above momentary pains, to suffer in the short term for a worthy long-term goal. The stick-and-carrot of short-term pain and pleasure cannot be reverse engineered to reveal what our long-time goals should be, much less any collective goals that transcend the lives of individuals. Besides, our technological means can already short-circuit these atavistic signals in the body, and close our minds inside a circle of addiction. But I don’t think anyone would consider enclosing all human minds inside virtual feedback loops as a satisfactory way of eliminating human suffering.
A goal that I probably share with both Wiener and Bostrom is to keep humanity around in some form or another, for a while longer at least. This must mean having some lookout for the adverse effects, long-term or short, of our collective technologies. When changing the physical world around us, we should try to keep our options open for future changes as well; not paint ourselves to a corner, or destroy the ladders we just climbed up, or burn every bridge we pass along our triumphant march forward. Future generations will not accept a pre-programmed existence any more than we would; true contingency is necessary for meaningful living.
Technology is not a free lunch, there are always some hidden offsets that we need to be aware of when making decisions. Marketing can try to hide the downsides, or pass the secondary costs to the public. For this reason, transparency and regulation will be more important than ever, as technology keeps progressing.