Synthetic Biology, Artificial Intelligence, and Quantum Computing

 

Abstract

We envisage a world where genetic engineering, artificial intelligence (AI), and quantum computing (QC) will coalesce to bring about a forced speciation of the Homo sapiens. A forced speciation will drastically reduce the emergence time for a new species to a few years compared to Nature’s hundreds of millennia. In this chapter, we explain the basic concepts that would allow a forced speciation of the Homo sapiens to occur and its consequences on life on Earth thereafter. Accelerating speciation mediated by Homo sapiens via domestication, gene splicing, and gene drive mechanisms is now scientifically well understood. Synthetic biology can advance speciation far more rapidly using a combination of clustered regularly interspaced short palindromic repeats (CRISPR) technology, advanced computing technologies, and knowledge creation using AI. The day is perhaps not far off when Homo sapiens itself will initiate its own speciation once it advances synthetic biology to a level where it can safely modify the brain to temper emotion and enhance rational thinking as a means of competing against AI-embedded machines guided by quantum algorithms.

1. Introduction The great journey of building rational scientific knowledge includes observing, making conjectures, and severely verifying them for flaws, limitations, and errors. When conjectures falter, scientists revisit, revise, abandon, start afresh, search for alternatives, etc. They seek unity in diversity or generalize to include diversity, with the knowledge that “truth” is not knowable. In this journey, they seek to be rational, parsimonious1 in making conjectures, and methodical, open, transparent, and consistent when sharing them. Conjectures are deemed scientifically valid only if there is potential scope of finding an error [1]. “Though [a mistake] stresses our fallibility it does not resign itself to scepticism, for it also stresses the fact that knowledge can grow, and that science can progress—just because we can learn from our mistakes” [1]. The process is criticism controlled. In the last few decades, technology has provided some remarkable tools to accelerate, not merely speed up, this process, and these tools have tremendous potential of becoming even more versatile. In the context of synthetic biology, the tools include the triad: clustered regularly interspaced short palindromic repeats (CRISPR) gene editing technology in genetic engineering, artificial intelligence (AI), and quantum computing (QC). There is also a torrential gathering of data since the Human Genome Project [2] published a draft sequence and initial analysis of the human genome in February 2001 [3]. The new sources include data flowing from the Human Cell Atlas project, which plans to identify and locate every type of cell we possess [4], and various brain projects initiated in the US, Europe, Japan, and Korea, and privately funded Allen Institute for Brain Science. China and Taiwan are also getting in the fray [5]. To make sense of the growing mountains of data in terms of finding “the molecular logic of the living state” in a timely manner rather than drowning in it will require data curation and analysis tools and resources that presently only CRISPR, AI, and QC can provide. This appears fortuitous since we anticipate a catastrophic speciation of the Homo sapiens to occur soon because of a rapidly changing environment that will likely lead to its decimation unless synthetic biology comes to the rescue. This chapter is therefore written for the millennials on whose shoulders will fall the responsibility of navigating through a socioeconomic epochal change that is already under way—the emerging postindustrial era—and a possibly unanticipated speciation of the Homo sapiens. The aim is to show that the time is ripe for synthetic biology, AI, and QC to join hands and form a purposeful, integrated discipline to further explore the secrets of life, create new life, and find harmonious ways by which the Homo sapiens can speciate in a controlled manner. 2. Time for human speciation is near Biology is a game of creation, survival by adaptation, and annihilation; it is a game that is “red in tooth and claw”. Survival of the fittest (also called natural selection) means survival of those best able to adapt to the environment they are in. This is not about individual survival but of cohesive groups belonging to a species capable of exchanging genes or interbreeding. Natural selection is an ultraslow process in which sudden, dramatic changes in the environment generally mean sudden decimation of species living in it. Homo sapiens already find themselves in this unenviable but self-created situation that includes climate change (that also brings deadly heat, spreads diseases, overwhelms hospitals2), epidemics, automation initiated unemployment, large-scale immigration due to ism-related (e.g., political, religion related dogmatism) strife, concentration of information and wealth in the hands of fewer and fewer people, depletion of natural resources faster than its replenishment by Nature, the rising irrelevance of rote education, the escalating cost and deterioration of health care, a rising global population that embeds a disproportionately rising population from the less developed countries (see Figure 1), the rapidly rising population of the aged whose needs must be paid for by a shrinking, less fecund, younger working population (itself worried about an insecure and financially bleak future), etc. Each by itself is a major stress creator; collectively, they are approaching a crescendo portending an environmental catastrophe that leads to speciation or extinction, and destruction of the biosphere’s existing order. Figure 1. (Left) World population growth. (Right) World population growth, 1950–2050. Source: World Population Prospects: The 2010 Revision, United Nations, 2011, http://www.un.org/en/development/desa/population/publications/pdf/trends/WPP2010/WPP2010_Volume-I_Comprehensive-Tables.pdf Note the rapidly increasing population size in the less developed countries. With the benefit of hindsight, we can discern the heralding signs of speciation that went unnoticed. In the rapidly growing global population (presently at 7.7 billion plus), the collective population of the more developed countries (characterized by high living standards and education, and low birth rate) since the last several decades has stabilized to about 1.3 billion (including immigrants), while that of the less developed countries (with opposite characteristics) is steadily rising. Concurrently, globally wealth has concentrated into fewer and fewer hands. In January 2018, Oxfam reported that “82% of all wealth created in the last year went to the top 1%, and nothing went to the bottom 50%”, that the wealthiest 42 people now had as much wealth as the poorest half, and two-thirds of billionaires wealth come from inheritance, monopoly, and cronyism [7, 8]. The environment for the poorest (hence unfittest) is already brutal. When natural speciation starts, its largest and earliest victims will come from the less developed countries before it hits the developed ones. In this respect, Africa appears to be highly vulnerable; it “has become the source of some of the greatest threats to the global economic order. Rather than capitalizing on opportunities, international engagement is increasingly focused on mitigating risks” [9]. When speciation begins, these risk mitigation efforts will be in vain because it is the global socioeconomic structure itself that will be disintegrating. The Homo sapiens’ incommensurate brain power will then make it vulnerable to extinction. The historical legacy of the Homo sapiens will not be its fossil record, but its amazing science, technology, engineering, and mathematics (STEM) record for successor species, if any, to peruse. Speciation is about adapting to the environment. Homo sapiens is the only known species to have developed substantial capacity to change the environment to its needs. Thus, it reduced the pressure for speciation since the agricultural era by adopting a socioeconomic structure built around division of labor and a tolerable taxation dogma of “from each according to his ability, to each according to his need” to temper Nature that is “red in tooth and claw”. That dogma is increasingly unsustainable because of an escalating need to subsidize the less well off. The affluent 1.3 billion can no longer subsidize the life span of the rest of the unemployable world. But there is the tantalizing possibility that since synthetic biology is ultrafast in editing DNA (deoxyribonucleic acid) and with advancing AI and QC, it will be even faster and better as compared to natural mutation and it may enable the Homo sapiens to initiate its own speciation in a programmed manner and survive extinction. What we cannot predict and may even fail to control once initiated are the unintended consequences that will certainly follow. If Ray Kurzweil’s prediction about the future capabilities of AI machines (“By 2029, computers will have human-level intelligence” [10]), turn out to be reasonably true, and genetic engineering continues at its present rate of development aided by advances in QC and in understanding RNA (ribonucleic acid)-mediated cellular activity using AI, artificially induced speciation of Homo sapiens by the end of this century may become possible before natural selection steps in anger. Kurzweil also forecasts that the future will provide opportunities of unparalleled human-machine synthesis: 2029 is the consistent date I have predicted for when an AI will pass a valid Turing test and therefore achieve human levels of intelligence. I have set the date 2045 for the ‘Singularity’ which is when we will multiply our effective intelligence a billion-fold by merging with the intelligence we have created. [11] Kurzweil’s forecasts are based on his “law of accelerating returns” that enunciates that fundamental measures of information technology follow predictable and exponential trajectories seemingly unaffected by dramatic socioeconomic events such as war or peace, and prosperity or recession, paralleling Moore’s law in computer technology—the number of transistors on integrated circuit chips doubles approximately almost every 2 years. Indeed, it turns out that once a technology becomes de facto information technology, it comes under the grip of the law of accelerating returns because computer simulation of any technology is all about mathematics and computation. The exponential change is the inevitable effect of our ability to conceptualize in larger and larger conceptual blocks by aggregating and augmenting smaller conceptual blocks discovered earlier. This simple mechanism enables the human mind to deal with and find solutions to more complex problems by using the same number of but more versatile concepts rather than an unmanageably larger number of simpler concepts. The method is no different than what mathematicians do. We were first exposed to this method when we studied Euclidean geometry in school. Mathematicians start with simple, primitive concepts they call axioms and build more and more complex theorems as they go along. It works if the axiomatic system is consistent because once a theorem is proven, its validity can be taken for granted even by those who know nothing about mathematics, for example, by machines. This is how machines acquire “intelligence.” During the industrial era just behind us, most people reached their peak capacity to educate and skill themselves in activities (including earning a living) that required mechanizable “intelligent” rote education. That AI machines, in principle, can far surpass humans in such activities had become evident when Alan Turing showed how arithmetical calculations can be mechanized [12] and Gödel had earlier shown that any axiomatic system can be arithmetized [13]. This meant that any form of rational knowledge could be axiomatized and rote education programmed into computers. While creating new knowledge would still require human creativity, once that knowledge had matured and was formalized into an axiomatic system, it would be mechanizable and expandable. It would then be a matter of time that humans would increasingly face competition from machines and eventually be overwhelmed by them. Kurzweil’s prediction that this would happen during 2029–2045 is bolstered by recent advances in AI. Ongoing advances in deep learning by machines indicate that through self-learning they can become highly creative and creators of original technology (the patent system will go for a toss) and scientific discoveries without human intervention may well become the norm [14]. Of some 150 predictions since the 1990s, Kurzweil claims an 86% accuracy rate [11, 15, 16]. Since synthetic biology has now come under the grip of mathematics, its exponential development is certain. Synthetic biology is now a part of information technology. It is only in the last few years that the enormous significance of the exponential growth property of the law of accelerating returns has sunk in the minds of people. As one can see from Figure 2 (left), till one reaches the vicinity of the knee of the curve, the curve looks deceptively linear with a mild slope. This allows human minds to extrapolate into the future from gathered knowledge and experience. At the knee, the curve bends upward so rapidly that the human mind cannot respond fast enough to absorb, assess, contemplate, and react rationally. Knee-jerk reaction is about the best humans are capable of in such a situation. Homo sapiens now find themselves in an environment which they neither understand nor have the intellectual ability to rationally cope with. This is germane for triggering a speciation event. Figure 2. Exponential growth. Source: (top left) Author. Nature of exponential growth. (Top right) Steve Jurvetson. An updated version of Moore’s Law (based on Kurzweil’s graph). Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Moore%27s_Law_over_120_Years.png Once AI breaches a certain threshold, one should expect a runaway technological growth resulting in a phase transition in human civilization, including perhaps the speciation of the Homo sapiens. A likely component of the phase transition may well be that AI enters self-improvement cycles (feedback loops) that eventually cause it to evolve into a powerful level of superintelligence that would qualitatively surpass intelligence levels of all Homo sapiens. The accelerating progress of STEM in concert has also brought about commensurate changes in our lifestyle, expectations from life, and erosion of superstition and belief in religion. In the last few decades, the exponential nature of these changes has become noticeable and taxing enough even for the socioeconomic upper strata Homo sapiens to cope with. When, to mitigate their anxiety about AI, people claim that AI cannot do this or that which humans can, they often forget to ask if those tasks are worth doing. Exponential growth in AI has advanced the possibility that artificially induced speciation of Homo sapiens may occur by the end of this century. Recent findings show that Homo sapiens evolved about 300,000 years ago [17, 18].3 In recent times, their socioeconomic environment too has changed dramatically. Billions face the prospect of AI machines depriving them of sustainable livelihood and a dignified existence in society. Under such dramatic conditions of environmental change, Nature will force speciation toward life forms with an evolved brain far superior to that of the Homo sapiens. The very process may start too late and move too slowly and lead to the extinction of the Homo sapiens. Artificially induced speciation may therefore be the only means that may allow the Homo sapiens to transition to a new species in a controlled manner. On the flip side, one or more renegade group of Homo sapiens may strategize to surreptitiously create a colony of new species with the aim of dominating the Earth and decimating the Homo sapiens as an unnecessary burden on Earth. 3. DNA is an information molecule The language of information now pervades molecular biology—genes are linear sequences of bases (like letters of an alphabet) that carry information (like words) to produce proteins (like sentences). For the process of going from DNA sequences to proteins, we use words like “transcription” and “translation,” and of passing genetic “information” from one generation to another. It is rather uncanny that molecular biology can be understood by ignoring chemistry and treating the DNA as a computer program (with enough input data included) in stored memory residing in a computer (the cellular machinery). It is this aspect that bioinformatics exploits. It is analogous to viewing Euclidean geometry not in terms of drawings but in terms of algebra. In a sense, in the DNA sequences in our cells, written using an alphabet of only four letters, lies hidden the story of who we are and where we come from. For all we know, it might even tell us where we might be going. Albert Lehninger wrote: … living organisms are composed of lifeless molecules … that conform to all the laws of chemistry but interact with each other in accordance with another set of principles—the molecular logic of the living state. [19] It is this “molecular logic of the living state” that is yet to be completely understood, and therein may lie our ability to understand emotion, cognition, and intelligence. So, in a deep sense, the DNA is the master molecule of life. A marvelous thing about cells is that they are so designed that for many purposes one can totally ignore their chemistry and think just about their logic. The fact that one can get away with this is one of the most elegant aspects of molecular biology. The algorithmic side of molecular biology is bioinformatics, the study of information flows in living matter. Bioinformatics is about the development and application of algorithms and methods to turn biological data into knowledge of biological systems. Of fundamental interest is the organization and control of genes in the DNA sequence, the identification of transcriptional units in the DNA, the prediction of protein structure from sequence, and the analysis of molecular function. If there is mathematical logic in living things, then one naturally seeks to determine the formal mathematical system that governs life, that is, how information in the DNA is stored and used by the rest of the cell’s machinery to do the myriad of things that it does. We already know that a DNA molecule—a genotype—is converted into a physical organism—a phenotype—by a very complex process, involving the manufacture of proteins, the replication of the DNA, the replication of cells, the gradual differentiation of cell types, and so on. This epigenetic process is guided by a set of enormously complex cycles of chemical reactions and feedback loops. By the time the full organism appears, there is no discernible similarity between the physical characteristics of the organism and its genotype. Yet molecular biologists attribute the physical structure of the organism to the information encoded in its DNA, and to that alone. This is because there is overwhelming experimental evidence that only DNA transmits hereditary properties. The genotype and the phenotype are isomorphic. However, this isomorphism is so complex that so far it has not been possible to divide the phenotype and genotype into parts, which can be mapped onto each other directly, unlike as, say, in the case of a music record and a record player where portions of a record’s track can be easily mapped to specific musical notes [20]. One hopes that AI and QC together will enable us to find this complex mapping. It is all about information processing. By information we mean the precise determination of sequence, either of bases in the nucleic acid or of amino acid residues in the protein. We gain knowledge of biological systems when we can interpret information in some “meaningful” way without it being easily refuted. That is, we make conjectures and put them through rigorous tests of refutations. Molecular biologists are becoming increasingly sure that “life is a partnership between genes and mathematics” [21]. Indeed, we increasingly tend to believe as Max Tegmark does about the Universe itself: Our reality isn’t just described by mathematics – it is mathematics … Not just aspects of it, but all of it, including you. [In other words,] our external physical reality is a mathematical structure. [22] One can well imagine the enormous strides synthetic biology will make when researchers get a deeper understanding of the Book of Life, with AI software becoming their research assistant, and quantum computers executing all the computing required by the AI software. 4. The technology triad The time has come for synthetic biology, AI, and QC to join hands and form a purposeful, integrated discipline to further explore the secrets of life, create new life, and find harmonious ways by which Homo sapiens can speciate. The main responsibility will fall on the shoulders of the millennials. The technology triad (CRISPR, AI, and QC) share some important properties, the ability to create, share, process, and communicate information in digital form. This means they can be supported and integrated with the full power of mathematics and physics. As Richard Feynman notes: Mathematics is a language plus reasoning; it is like a language plus logic. Mathematics is a tool for reasoning. … [I]t is impossible to explain honestly the beauties of the laws of nature in a way that people can feel, without their having some deep understanding of mathematics. [23] Mathematics is the lingua franca of the physicists because a formal mathematical statement to be of any value is either true or false; it cannot be true to some and false to others. This is the reason why knowledge based on any axiomatic system, that is, a consistent system in which every valid statement or query has a “yes” or “no” answer, can be arithmetized (i.e., translated into arithmetical statements), encoded in a binary string, and processed in a digital computer. Mathematics is the language that binds men and machine together in a rational dialog. In short, axiomatic systems permit men and machines to mutually communicate without ambiguity or confusion. This is the foundation on which artificial intelligence (AI) rests. It is why Pierre Simon Marquis de Laplace did not even acknowledge God as the creator of the Universe in his mathematical magnum opus on celestial mechanics [24]. He famously told Napoleon Bonaparte, “I had no need of that hypothesis” [25]. Creating and advancing rational knowledge, inter alia, requires an ability to communicate thoughts concisely, precisely, and accurately apart from refining knowledge by trial and error, that is, by making our conjectures fitter and fitter for survival. Benjamin Lee Whorf (1897–1941) said, “Language shapes the way we think, and determines what we can think about.” And Ludwig Wittgenstein (1889–1951) said, “The limits of my language mean the limits of my world.” Mathematics provides fewer limitations than any other language known to Homo sapiens. The power of mathematics lies in its ability to extract unity from diversity by abstraction, that is, by eliminating unnecessary context; it helps in discovering group properties (abstract or otherwise) common to all members of the group, for example, the DNA of a species. Both AI and QC are inseparable from mathematics; they are powerful means of processing and interpreting information (e.g., in the DNA) as well as aiding in inventing novel DNA for specific purposes. Both support and are supported by 3D printing that began by making plastic widgets, but now make guns, houses, prosthetic limbs, vehicle parts, etc. from inanimate matter. The day is not far off when it will advance to printing living, breathing, bio-organs, such as hearts [26] and kidneys, using nanotechnology, computer-aided engineering, and inanimate biodegradable or biocompatible materials and chemicals to build stem cells. Replicating and growing cells, say, in petri dishes is well established, and such cells are already in use as bioink in bioprinters. 3D printing offers the possibility of printing an entire organ, along with a system of arteries, capillaries, and veins that can support it [27, 28]. A major issue in developing this technology is to make it immune-system friendly, since the body may reject organs or cells thus produced, something that can occur even when tissue from one area of the body is put into another. 4.1 CRISPR technology connects synthetic biology with information technology CRISPR technology has enabled a simple and affordable method of manipulating and editing DNA that has radically changed the ambitions of synthetic biologists. The technology promises to revolutionize how Homo sapiens may deal with the world’s biggest problems, for example, finding cures for cancer, blindness, and Alzheimer’s disease, improving food and eliminating food shortages, fulfilling organ transplant needs, and producing fuel and manufacturing chemicals. Biotechnologists are racing to develop the most efficient, precise, versatile, affordable, and commercially viable genome-editing tools possible. This will be a long and exciting race that may eventually lead to the Homo sapiens creating a super species that far exceeds them in the evolutionary path in a controlled manner. CRISPR is a series of short repeating DNA sequences with “spacers” separating them. The CRISPR technology harnesses an ancient bacteria-based defense system. Bacteria use these genetic sequences to “remember” the viruses that have attacked them by the simple mechanism of incorporating the virus’ DNA into their own bacterial genome. The viral DNA thus resides as spacers in the CRISPR sequence as identification tags the bacteria can use to mount an attack if the virus attacks again. Accompanying the CRISPR are locally stationed genes called Cas (Crispr-associated) genes. Once activated, these genes produce enzymes that act as “molecular scissors” that can cut into DNA with specificity. The significance is that in subsequent virus attacks, the bacteria can recall the virus signature and send RNA and Cas to locate and destroy the virus. Among the Cas enzymes derived from bacteria, Cas9 is the best-known molecular scissors enzyme for cutting animal and human DNA. Although the CRISPR sequence was first discovered in 1987, its function was discovered only in 2012. The ability to cut DNA allows one to either knock out, say, an unwanted disease-causing gene, or splice a “fixed” version of a gene into the DNA. This is analogous to the “Find & Replace” function in text editing software. Indeed, CRISPR technology has advanced so rapidly beyond the Find & Replace function that by December 2017, the Salk Institute had designed a version of the CRISPR-Cas9 system that could switch on or off a targeted gene without even editing the gene. The basic ingredients of gene editing are (1) a piece of RNA, called the guide RNA, that locates the targeted gene, (2) the scissors (the CRISPR-associated protein 9), and (3) the desired DNA segment for insertion after the break. Once the guide RNA locates the targeted gene, Cas9 makes a double-stranded break in the DNA carrying the targeted gene and replaces it with the desired DNA segment. A quick tutorial on CRISPR is available at [29]. CRISPR-based therapies are still nascent. As expected, single-gene disorders are among the best understood because of their simple inheritance patterns (recessive or dominant) and relatively simple genetic etiology (cause). Such disorders include cystic fibrosis, hemochromatosis, Tay-Sachs, and sickle cell anemia. For example, cure for sickle cell disease (an inherited form of anemia in which distorted red blood cells—rigid, sticky, and shaped like sickles—are present in such numbers as to prevent adequate oxygen supply throughout the body) has gained prominence because it is related to an abnormal hemoglobin molecule, which comes from a well-understood genetic mutation. Hence efforts are concentrated on creating therapeutic strategies for fixing the mutated gene. Online Mendelian Inheritance in Man® (OMIM®) provides an Online Catalog of Human Genes and Genetic Disorders, a comprehensive database provides information about the etiology, clinical symptoms, and a bibliography of thousands of genetic conditions.4 4.2 Artificial intelligence (AI) The true test of intelligence is not how much we know how to do, but how we behave when we don’t know what to do. [30] This behavior is a product of the brain-mind system an individual is born with and the environment it finds itself in. From conception to death, behavior and intelligence evolve through intimate interaction between the individual and the environment where the individual essentially tries to coexist with the environment by exploring networking strategies, inter alia, based on its information gathering and processing abilities (see Section 5.2). In the past few decades, in an ongoing process, the Homo sapiens using technology they have intelligently developed have already acquired massive amounts of information and placed it in easily accessible public repositories along with some sophisticated automated information processing services. This has happened unexpectedly, suddenly, and on a massive scale at an exponential rate in multiple disciplines (including molecular biology) due to breakthroughs in communication and computing technologies engineered by an exceptionally intelligent group of Homo sapiens. This development is well on its way to dwarfing the intellectual abilities of almost all Homo sapiens. In comparison, individual human brain capacity to understand, assimilate, create, and deal with knowledge appears pathetic and along with it, its ability to find gainful employment in the future. Machines are rapidly learning to create and deal with knowledge. On the positive side, There is a paradox in the growth of scientific knowledge. As information accumulates in ever more intimidating quantities, disconnected facts and impenetrable mysteries give way to rational explanations, and simplicity emerges from chaos. [31] It is this scientific knowledge ferreted out by a few geniuses among the Homo sapiens, which has allowed the species to extend their life span and improve their lifestyle by not just adapting to an environment but also by aiding the environment to adapt to humans. Along the way, Claude Shannon provided a mathematical theory that highlighted an important aspect of how data can be condensed and communicated efficiently in binary bitstreams. This was an important step in handling data by finding structure in data to reduce redundancy in data representation [32]. Big data revolution, development and deployment of wearable medical devices, and mobile health applications have provided new powerful tools to the biomedical community for applying AI and machine learning algorithms to vast amount of data. Its impact in predictive analytics, precision medicine, virtual diagnosis, patient monitoring, and drug discovery and delivery is already being felt. More powerful advances are anticipated in the near future. Even at this early stage, AI excels even human experts in certain well but narrowly defined tasks. AI is at a stage where basic building blocks are being built. Soon we will learn to network these blocks and build increasingly powerful systems and subsystems that will solve increasingly complex problems and even create new knowledge. We already have a glimpse of it in Alphabet’s AlphaGo Zero’s ability to learn complex decision-making from scratch [33, 34]. “Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. In doing so, it quickly surpassed human level of play and defeated the previously published champion-defeating version of AlphaGo by 100 games to 0” [34]. It acquired this ability within 40 days of self-training in an essentially iterative manner. The key here is the iterative strategy it used. Indeed, Homo sapiens too acquire knowledge iteratively but slowly over years and generations, collaboratively across space and time with other Homo sapiens, by making conjectures and refutations. It is rather uncanny that the essence of the process and its unusual power is mathematically captured by the Mandelbrot set in fractal geometry (see Section 5.3). Notwithstanding AlphaGo’s success, many real-life problems are still far too difficult not just for current AI systems but also for the vast-vast majority of Homo sapiens. The competition is really between two classes of geniuses: Homo sapiens who create ab initio knowledge and Homo sapiens who develop AI. Eventually, the latter is expected to win even if they must create an artificial brain using synthetic biology and place it in a humanoid! The task is enormously complex but not out-of-reach, in principle. What is needed is the ability to automate the task of observing and collecting data about the world and about us, create categories, data structures, and algorithms that would enable the collected data to be condensed into a computer program that can calculate the observations. This necessarily means that the size of the computer program (say, as represented by a binary string) must be as compact as possible (an index of the AI system’s intelligence) compared to the collected data (also represented by a binary string). Till this is accomplished, the collected data would remain incomprehensible, that is, algorithmically random, theory-less, unstructured, and irreducible [35]. This is what Homo sapiens in the genius class devote themselves to. As Oren Etzioni notes, machine learning is still 99% human work: The equation for AI success is to take a set of categories (for example, cats and dogs) and an enormous amount of data (that is labeled as to whether it is a cat or a dog), and then feed those two inputs through an algorithm. That produces the models that do the work for us. All three of those elements—categories, data, algorithm—are created through manual labor. [36] The solution to eliminating manual labor may well be the creation of an artificial brain using synthetic biology. For the present, AI serves mainly by “augmenting human intelligence”. But then automation too had begun by augmenting brawn (muscle) power to eventually become the superbrawn power during the industrial revolution. It only required the Homo sapiens to intelligently harness and control steam by first connecting water, heat, and work and then creating the thermodynamics, the science that would allow machines to make human brawn power look insignificant. Today’s augmented intelligence appears destined to become superintelligence. We have learnt to harness and control reasoning by first connecting logic, axiomatic systems and theorem proving. We are now advancing rapidly into understanding information theory so that quantum computers can become information engines to do intelligent work. It is interesting that the concept of entropy appears fundamental both in thermodynamics and information theory. Both are offsprings of rational thought in physics, and both are intimately related.5 4.3 Quantum computing will power synthetic biology and AI Quantum mechanics deals with the world inhabited by photons, electrons, protons, atoms, molecules, etc. and how they interact among themselves to create larger matter entities. It is an incredibly mysterious world understood only in the language of advanced mathematics. This is the part of physics that tells us how atoms congregate into molecules by adjusting the electrons they carry into configurations that we call chemical bonds, how strong or weak those bonds will be or whether they will bond at all, what a congregation’s physical and chemical properties will be. It has led to many technical innovations and many more are expected, for example, in synthetic biology. The success of quantum mechanics in using mathematical abstractions is such that to a lay person it appears mystical, which even religious mystics cannot understand! Its remarkable success comes even though we still do not know what is meant by measurement in the quantum world and how the measurement process captures the information it outputs and why it releases information in a randomized way. Yet its success is undeniably visible: Quantum mechanics is an immensely successful theory. Not only have all its predictions been experimentally confirmed to an unprecedented level of accuracy, allowing for a detailed understanding of the atomic and subatomic aspects of matter; the theory also lies at the heart of many of the technological advances shaping modern society – not least the transistor and therefore all of the electronic equipment that surrounds us. [38, 39] Understanding quantum mechanics is out of reach except for a few thousand people in the world at any given time! This should immediately alert us to the fact that human intelligence needed to cope with AI-QC combination in the future will be very high and successor species of the Homo sapiens must evolve in the direction of better and smarter brains rather than any other physical trait. Computation, comprehension, and cognition are all a part of the brain’s activity, and we may assume that a sharper brain will come with a sharper mind. And we may further assume that comprehension and cognition are driven by computation in addition to using intuition, serendipity, flashes of inspiration, and inputs from the environment, etc. The keys are computation, problem-solving algorithms, and rational decision-making processes. These can be simulated by a classical computer, which itself has an abstract mathematical description we call the universal Turing machine (UTM) [12]. Computing technology has now advanced to a stage where quantum computers can do everything that a UTM can do, and some more. A quantum computer’s phenomenal computing power comes from the extraordinary laws of quantum mechanics that include such esoteric concepts as superposition of quantum states, entanglement (“spooky action at a distance”), and tunneling through insulating walls, which, though highly counterintuitive, play extremely useful roles in understanding Nature at subatomic levels. However, it is not clear if these concepts can be ignored in biology and living processes in the way they are ignored in the design of cars and airplanes. May be not because there are areas in biology where quantum effects have been found, for example, in protein-pigment (or ligand) complex systems [40]. Thus, while the role of quantum mechanics is clear in quantum computing and hence in advancing both AI and synthetic biology research, it is not yet known if in the design of DNA, knowledge of quantum mechanics is required or that natural selection favors quantum-optimized processes. Essentially, we do not know if any cellular DNA maintains or can maintain sustained entangled quantum states between different parts of the DNA (even if it involves only atoms in a nucleotide). But we cannot rule out the possibility that sporadic random entanglements do occur that result in biological mutations or that researchers will not be able to achieve it in the laboratory and find novel uses for it in synthetic biology [41]. For example, in principle, it is possible to design molecular quantum computers, insert them in cells that can observe cellular activity, and activate select chemical pathways in the cell in a programmed manner. There is increasing speculation that some brain activity, for example, cognition, may be quantum mechanical [42]. 5. Integrating the triad: mechanization of speciation A combination of emerging technologies such as CRISPR, AI, and QC; new delivery models for products and services that form the core around which Homo sapiens organize themselves through collaborative division of labor; and talent migration, driven not by rote education but by innate creativity and global opportunities for employment open to them is disrupting and changing the character of the global talent pool that society needs today. Globalization has created opportunities for the talented to reach the skies, but in a resource-constrained world, it also means that many others must be or feel deprived. Sections 5.2 and 5.3 provide some glimpses of the dynamics of this situation captured in mathematics. Because mathematics is abstract, the depicted dynamics apply to entities and situations whether they are animate or inanimate. A resource-constrained world provides ample scope for adversarial dynamics in which some are predators and others are preys. Globalization has accentuated the problem at all levels of social structure, and since speciation is triggered by a changing environment, it affects the DNA. This has created survivability demands on the Homo sapiens. As this pressure mounts beyond endurance, Homo sapiens will face speciation by natural selection with uncertain outcomes. However, in the case of Homo sapiens, this process too may face a disruptive change because the highly intelligent among them may boldly initiate speciation using upcoming advances in synthetic biology, perhaps after perfecting their techniques by creating humanoids (a hybrid creation of life with embedded intelligent machinery). This will be a watershed event where a species takes on the task of speciation on itself. This remarkable possibility arises because Homo sapiens created and mastered mathematics, rational thought, computing machinery, and eventually deep data analytics so that life could be designed by them in the laboratory to create superior species. Synthetic biology, using methods and rational knowledge of molecular biology, physical sciences, and engineering, aims to design and construct novel biological parts, artificial biological pathways, devices, organisms, and systems for useful purposes. This will also permit us, at all levels of the hierarchy of biological structures (molecules, cells, tissues, and organisms), to redesign existing natural biological systems and may even help us recreate certain extinct species (if we can also recreate the environment, they had adapted to). It is not surprising that an extinct species has never revived itself since speciation and environment go together. Successes of synthetic biology will change the face of human civilization and almost certainly bring in new elements into play when Homo sapiens eventually speciate by playing an active role in it. Since the discovery of the double-helix structure of cellular DNA by James Watson and Francis Crick in 1953 [43] and its significance that the “precise sequence of the bases is the code which carries the genetical information …” (emphasis added) [44], the jargon and theory of information has invaded molecular biology (see Section 3). This enriched biotechnology and computational biology with nomenclature, definitions, concepts, and meanings. This also facilitates integration of synthetic biology with AI and QC. DNA is an information-carrying polymer. It is an organized chemical information database that inter alia carries the complete set of instructions for making all the proteins a cell will ever need. Just 20 years after Watson and Crick, in 1973 Cohen and Boyer published their pioneering work in recombinant DNA [45] and gave birth to genetic engineering and the biotechnology industry based on their patents [46] under liberal licensing terms. The next landmark was the creation of a bacterial cell controlled by a chemically synthesized genome by Craig Venter and his group in 2010 [47]. In 2014, Floyd Romesberg and colleagues [48] reported the creation of a semisynthetic organism with an expanded genetic alphabet by creating artificial nucleotides not found in Nature. Since its discovery in 2012 [49, 50, 51], CRISPR gene editing technology pioneered by Jennifer Doudna and Emmanuelle Charpentier, and Feng Zhang has come to occupy center stage in molecular biology as a new way of making precise, targeted changes to the genome of a cell or an organism. It has set the stage for major advances in synthetic biology (see Section 4.1). Another major advance was reported by Venter and his research group in March 2016 following their successful creation in 2010 of a bacterial cell controlled by a chemically synthesized genome noted above. In fact, they succeeded in creating a bacterium that contains the minimal genetic ingredients needed for free living. The genome of this bacterium consists of only 473 genes, including 149 whose precise biological function is unknown. It is a minimalist version of the genome of Mycoplasma mycoides [52, 53]. Synthesis capabilities have developed at a pace where DNA synthesis is now automated. All one needs to do is to provide the desired DNA sequence to a vendor. Researchers in synthetic biology are now inching toward anticipating and preempting evolutionary events that if left to themselves would perhaps take a few million years to occur, and of even resurrecting extinct species. The time is ripe to integrate synthetic biology with AI and QC with a common language to enable seamless communication among them, connect with, and discover conceptual similarities for consistent integration of subsystems and validation of the whole system. That common language is mathematics; it comes with the added benefit that it can be used to also communicate between humans and machines. It is fortuitous that the DNA serves as the “Book of Life” that appears to have structure and grammar amenable to translation into mathematics. Once translated, biologists will discover some amazing patterns that have a direct bearing on life at the molecular level. We introduce a few of these below in brief. 5.1 The molecular logic of the living state All macromolecules are constructed from a few simple compounds comprising a few atoms. It appears paradoxical that the DNA that serves as the epitome of life is itself lifeless. The molecule conforms to all the physical and chemical laws that describe the behavior of inanimate matter. All living organisms extract, transform, and use energy by interacting with the environment. Unlike inanimate matter, a living cell has the unique capacity, using the genetic information contained completely within itself, to grow and maintain itself and do mechanical, chemical, osmotic, and other types of work. But its most unique attribute is its programmed capacity to self-replicate and self-assemble. The great mystery that engulfs molecular biology is: “How does life emerge from an interacting collection of inanimate molecules that constitute living organisms to maintain and perpetuate life?” Once this is understood, chemical engineers will create a new life industry and commoditize it! Imagine buying customized pets as starters. As noted in Section 3, the mystery of life is almost certainly encoded in mathematics. The chemical basis of life is one indication because chemistry now has a strong mathematical foundation via quantum chemistry. Even more striking is the fact that all living organisms—bacterium, fish, plant, bird, animal—share common basic chemical features, for example, the same basic structural unit (the cell), the same kind of macromolecules (DNA, RNA (ribonucleic acids), and proteins) built from the same kind of monomeric subunits (nucleotides and amino acids), the same pathways for synthesis of cellular components, the same genetic code, and evolutionary ancestors. The monomeric subunits can be covalently linked in a virtually limitless variety of sequences just as the 26 letters of the English alphabet or the two binary numbers (0, 1) in binary arithmetic can be arranged into a limitless number of strings that stand for words, sentences, books, computer programs, etc. Organic compounds of molecular weight less than about 500, such as amino acids, nucleotides, and monosaccharides, serve as monomeric subunits of proteins, nucleic acids, and polysaccharides, respectively. A protein molecule may have a thousand or more amino acids linked in a chain, and DNA typically has millions of nucleotides arranged in sequence. Only a small number of chemical elements from the periodic table of chemistry appear in biomolecules. The carbon atom dominates and, by virtue of its special covalent bonding properties, permits the formation of a wide variety of molecules by bonding with itself, and atoms of hydrogen, oxygen, nitrogen, etc. Nature has placed further constraints. DNA is constructed from only four different kinds of subunits, the deoxyribonucleotides; the RNA is composed from just four types of ribonucleotides; and proteins are put together using 20 different kinds of amino acids. The 8 kinds of nucleotides (4 for DNA and 4 for RNA) from which all nucleic acids are built and the 20 amino acids from which all proteins are built are identical in all living organisms. So, at this level, living organisms are remarkably alike in their chemical makeup. This by itself provides a tantalizing hope that the DNA may indeed be completely decipherable as to its grammar and information content. The above observations strongly suggest the likelihood of an underlying, as yet undiscovered set of “axioms” of life that enforce emergent, organizing principles around which diverse life forms evolve and adapt to the environment at various levels, without transgressing any physical or chemical law. The organizing principles appear to include (1) Nature is red in tooth and claw (species are connected to each other in a predator-prey, food-chain relationship in a sparse resource matrix), (2) rules of genetic inheritance, (3) rules of environmental adaptation, and (4) rules of speciation. At each level, the rules are likely to appear stochastic given that there are innumerable interacting factors ranging from nature to nurture. 5.2 Law of network phase transition In 1960, Erdős and Rényi [54, 55] proved a remarkable result in graph theory, which implies that when a large number of entities (e.g., men, machines, ideas, or arbitrary combinations of them represented by dots) begin to connect (link) randomly, a critical condition arises, following which a phase transition occurs in the way the entities form or reform into clusters of connected entities. The critical condition is reached when in a set of n dots, n/2 random links are made. The phase transition abruptly creates a giant connected component, while the next largest component is quite small. Such giant components then grow or shrink rather slowly with the number of dots as they continue to link or delink. Such behavior is observed in protein interaction networks, telephone call graphs, scientific collaboration graphs, and many others [56]. This immediately suggests an involuntary mechanism by which a society at various levels of evolution, by connections alone, spontaneously reorganizes itself as nodes (people, machines, resources, etc.) link or delink in apparent randomness. It is highly pronounced in an Internet of Things (IoT) connected world where the millennials spontaneously polarize on issue-based networks that concern them on social media. Synthetic biologists must never forget that between the molecular and environmental levels, there are multiple intermediate levels through which regulated command and control communications pass. At all levels, level-related phase transitions and predatory fights for resources can occur and spread to other levels. In fact, the intimately coupled relationship between Homo sapiens and the environment is often overlooked. We rarely note what Richard Ogle has that [I]n making sense of the world, acting intelligently, and solving problems creatively, we do not rely solely on our mind’s internal resources. Instead, we constantly have recourse to a vast array of culturally and socially embodied idea-spaces that populate the extended mind. These spaces … are rich with embedded intelligence that we have progressively offloaded into our physical, social, and cultural environment for the sake of simplifying the burden on our own minds of rendering the world intelligible. Sometimes the space of ideas thinks for us. [57] The deep significance of this intimate bonding between the Homo sapiens and the environment is that while they are adapting to the environment, they are also helping the environment to adapt to them. When entities connect, they also acquire emergent properties by virtue of the relationships they are bound by. Certain static group properties emerge based on the network’s topology, while dynamic properties emerge depending on the rate at which entities make, break, or modify connections. The fluctuating dynamics witnessed in the social media, for example, is common among the millennials. Rapidly increasing connectivity among men and machines has imposed upon the global socio-politico-economic structure, a series of issue-dependent phase transitions. More will occur in areas where massive connectivity is in the offing. Immediately before a transition, existing man-made laws begin to crack, and in the transition, they break down. Posttransition, new laws must be framed and enforced to establish order. Since such a phase transition is a statistical phenomenon, the only viable way of managing it is to manage groups by abbreviating individual rights. The emergence of strongman style of leadership and its contagious spreading across the world is thus to be expected because job-seeking millennials will expect them to destroy the past and create a new future over the rubble. It appears inevitable that many humans will perish during the transition for lack of jobs or their inability to adapt to new circumstances. Robots and humanoids will gain domination over main job clusters, while society undergoes radical structural changes. Ironically, robots neither need jobs, nor job satisfaction, nor a livelihood. There will be ruthlessness in the reorganization. 5.3 The logistic map and the Mandelbrot set Consider the iteration xn + 1 = r xn (1 – xn), called the logistic map, and a number-pair (r, x0) where r > 0 and 0 < x0 < 1, and plot the points (r, xn → ∞). Note our interest is only in the long-term trajectory of x0 and not in its transitory phase. Note xn + (1 – xn) = 1. The plot (Figure 3) has numerous 2-pronged pitchforks and hence is called the bifurcation diagram. Depending on r, xn may be settled as for 0 < r ≤ 3, and beyond r = 3 migrating from one prong to another of available pitchforks for a given r in the bifurcation diagram. At r = 4 and beyond, migration is chaotic. In between r = 3.5 and 4, there is an intuitively unexpected white band where migration options are few. Such and other unexpected (not discussed here) display of rich complexity tethered to r independent of x0 (i.e., the starting state) caught researchers by great surprise. Figure 3. The logistic map. There are countless situations for which the logistic map captures the essence of a situation. For example, in genetics it describes the change in gene frequency in time, or in epidemiology the fraction of the population infected at time t, or in economics it depicts the relationship between commodity quantity and price, or in theories of learning the number of bits of information one can remember after an interval, or in the propagation of rumors the number of people who have heard the rumor after time t, etc. The logistic map allows us to assess the volatility of an adversarial environment by assessing r, that is, the ferocity with which the predators and preys are battling for resources. Now consider the following complex iteration. Given the complex variable z = x + iy, where i=−1‾‾‾√ i = − 1 and the complex constant c = a + ib, pick a value for c, and iterate with the seed z0 = 0. If the iterations diverge, then c is not in the Mandelbrot set (it is in the escape set), otherwise (even when it is trapped in some repeating loop or is wandering chaotically), it is in the Mandelbrot set (black points in Figure 4) M. (Setting z0 equal to any point in the set that is not a periodic point gives the same result.) This is perhaps the most famous mathematical object yet known. It is a fractal object, an object that is irregular or fragmented at all scales. It is a major discovery of the late 20th century. It cannot be replicated in Euclidean geometry. Figure 4. Mandelbrot set. In 1981–1982, Adrien Douady and John H. Hubbard [58] proved that the Mandelbrot set is connected. Quite astoundingly, the Mandelbrot set, when magnified enough, is seen to contain rough copies of itself, tiny bug-like objects (molecules) floating off from the main body, but no matter how great the magnification, none of these molecules exactly match any other (see Figure 5 and follow the white-bordered square from left to right). The boundary of M is where a Mandelbrot set computer program spends most of its time deciding if a point belongs there or not. The simplicity of the iterative formula and the complexity of the Mandelbrot set leave one wondering how such a simple formula can produce a shape of great organic beauty and infinite subtle variation. Figure 5. Infinite variations of the Mandelbrot set are embedded in the set itself. Source: Ishaan Gulrajani, A zoom sequence of the Mandelbrot set showing quasi-self-similarity, 01 October 2011, https://commons.wikimedia.org/wiki/File:Blue_Mandelbrot_Zoom.jpg (Placed in public domain). Since the logistic map and the Mandelbrot set map quadratic functions, and both represent behavior under iteration, it is not surprising that a one-to-one correspondence exists between the constants r and c and that the bifurcations created by r correspond to features that come with changes in c along the real axis where the Mandelbrot set compresses the information in the bifurcation diagram, that is, the map shows the points where the map converges to periodic oscillations and its periodicity, while the Mandelbrot set marks all the points, which end up oscillating, but the periodicity information is encoded in the bulbs of the set (see Figure 6). Figure 6. (Left) Connection between the logistic map and the Mandelbrot set. (Public domain) Source: Georg-Johann Lay, 07 April 2008, at https://commons.wikimedia.org/wiki/File:Verhulst-Mandelbrot-Bifurcation.jpg. (Right) Frank Klemm, Mandelbrot set with periodicity of limiting sequences. 12 August 2017. https://commons.wikimedia.org/wiki/File:Mandelbrot_Set_%E2%80%93_Periodicities_coloured.png licensed under the Creative Commons Attribution-Share Alike 3.0 Unported. It appears that the Mandelbrot set, inter alia, mimics the working of the mind. Its infinitely many variations embedded within itself seem to say that once the mind latches on to an idea and begins to deeply explore it, it does so by investigating its many variations, often in a random fashion (i.e., choosing c randomly), but does not abandon the core idea (the iterated function, equivalent of a law of Nature). On the other hand, if a mind randomly discovers a few of the dispersed similar looking sets, it begins a search for the mother set, M, itself. Is it then surprising that researchers often tackle new problems through random exploration based on a hunch (the iterated function), and if they are persistent enough, a solution finally emerges if the hunch is right? We see a game of conjectures and refutations at play here. On the other hand, the logistic map appears to work on a species scale where random interactions among minds lead to forming of societies (say, along the lines of the Erdős & Rényi theorem) functioning under constrained resources and an adversarial predator-prey law where the bifurcation points stand for points of speciation (measured in geological time scales). The pace at which a system is driven through cyclic (iterative, also called self-referential) processes, that is, cycles of construction and destruction constrained by recyclable finite resources, has a profound effect on how the system evolves. A remarkably simple model as the logistic map shows an amazing variety of nonintuitive dynamics that a nonlinear system can display. It too provides a basic involuntary mechanism by which a society spontaneously reorganizes itself. In his seminal paper on the logistic map, Robert May, a theoretical ecologist and former President of the Royal Society (2000–2005) was so struck by the deep relationship between complexity and stability in natural communities that he exhorted: Not only in research, but also in the everyday world of politics and economics, we would all be better off if more people realised that simple nonlinear systems do not necessarily possess simple dynamical properties. [59] What lessons can we draw from such simple mathematical models? For one, the logistic map indicates that the Earth’s supply chain (the environment) has been grossly disrupted. In this predator-prey game where some Homo sapiens turn into predators and the rest into preys, a massive capture of supplies by predators results in a massive population of preys, and the preys must mutate or speciate to survive or die. The logistic map decides how the selfish genes play the game while the Homo sapiens mainly decide the value of r. The Mandelbrot set tells us that while the laws of Nature need not change for the environment to change, it does contain enough complexities in the form of fractal structures whereby the environment may change enough to force speciation to take place in niches. In the present innovation-driven environment, speciation will push to enhance the brain-mind system of the Homo sapiens. In the process, synthetic biology may discover life as we do not know it. The survival of the fittest is a statistical law and hence it rests on an ensemble being available. The world’s current population certainly fulfills that. In the present global environment, saturated by connectivity between humans, machines, and ideas, the largest component emerging in any socioeconomic context is populated by the deprived who cannot fend for themselves. Inter alia, this is highly visible at multiple scales of population size (global, national, provincial, urban, etc.) and context (employment, access to health care, education, skill development, etc.). A wide spectrum of power, opportunities, and assets are grabbed by a minority by simply ignoring the plight of the desperate. This alone enforces a massive decimation of the Homo sapiens’ gene pool. Among the predators, many with inherited wealth (and hence generally lacking survival skills but not the means) too will become preys. In this planetary-scale debacle, a unique minority endowed with an exceptional brain-mind system, perhaps aided by AI and QC, will strive to improve their gene pool by artificial speciation6 using synthetic biology and insulate themselves in an artificially created environment to improve their cognitive abilities, life span, and fecundity. A look at the logistic map shows that as the new species advance even more rapidly, increasingly wild fluctuations in their fortunes will take place within their insulated, resource-constrained environment unless they reduce r by allowing the environment to replenish itself. In the absence of irreversible ecological damage, it is possible that, in the early stages, replenishment may happen by itself since Nature would have decimated a large component of the population from the less developed countries, thus presenting the survivors with a sudden increase in per capita resources. We may infer by analogy from the Mandelbrot set that once a new species survives long enough to avoid extinction (because it begins with a small population, which needs time to grow into adulthood), even if it is in some remote fringes of the set, it will likely someday reach the main (central) part of the set since the set is connected. Once this happens, the new species will likely continue for a very long time until it is decimated by the Sun entering its dying phase by turning into a giant red star. That will be a few billion years hence. 5.4 Creating novel DNA algorithmically The way we acquire knowledge is iterative and nonlinear—we conjecture and put our conjectures on trial, that is, put them to severe critical tests (refutations). As the trial progresses, we edit, discard, refine, and add to our conjectures in a pseudorandom manner controlled by criticism, driven by instinct, hunches, inspiration, etc. Conjectures and refutations in scientific research are deemed self- and community-driven adversarial processes. We connect the dots. At every step of linking the dots, we consult the axioms (conjectures) and the rules for deriving conclusions (theorems) to ensure that we are within the axiomatic system we have put on trial. This means that the process leads us to understand the Universe solely based on our chosen beliefs (axiomatic system). As we learn from our mistakes our knowledge grows, even though we may never know—that is, know for certain. Since our knowledge can grow, there can be no reason here for despair of reason. And since we can never know for certain, there can be no authority here for any claim to authority, for conceit over our knowledge, or for smugness. [1, Preface] As far as we can tell, creating an axiomatic system is a nonmathematical and a highly intelligent act. Developing a sequence of theorems with a specific nontrivial goal in mind (developing algorithms) is also a highly intelligent act. However, executing an algorithm, once developed, can be mechanized and does not require intelligence, in fact, none at all. If the most useful aspect of intelligence is algorithmic, then it must be mechanizable and converted into computation. We believe the DNA is a book of knowledge about the birth and death of life. In principle, it is in machine-readable form. AI and quantum computing are the most powerful tools we presently have to decipher it. When AI drives our lives, it is the algorithm that really drives us. Some recent bold experiments using CRISPR gene editing have provided glimpses of DNA editing as a new source of creating a variety of biomatter and life forms. For example, experiments are in progress for producing meat (beef, pork, poultry, and sea food) without killing animals by growing meat in the laboratory from cultured stem cells by multiplying them dramatically and allowing them to differentiate into primitive fibers that then bulk up to form muscle tissue. This would substantially reduce environmental costs of meat production and eliminate much of the cruel and unethical treatment of animals [62]. Another example is producing offsprings from same-sex mice parents, again using stem cells and CRISPR gene editing technology [63]. In another development, till recently it was believed that mitochondrion DNA (mtDNA) in nearly all mammals (including humans) is inherited exclusively from the mother. However, recently, Luo et al. [64] have uncovered multiple instances of biparental inheritance of mtDNA “spanning three unrelated multiple generation families, a result confirmed by independent sequencing across multiple unrelated laboratories with different methodologies. Surprisingly, this pattern of inheritance appears to be determined in an autosomal dominant like manner.” Given that the mitochondrion is an energy-producing organelle in the cell, this discovery will have profound implications in synthetic biology and in the design of new drugs. Once humans master the art of designing DNA for self-replicating, multicellular organisms (we already know how to design cells not found in Nature and edit DNA), they will create living species of their own design. We also anticipate that when AI machines master the art of learning from mistakes (i.e., the art of making conjectures and refuting them in a spiraling process toward better knowledge, a possibility that mathematically exists), they would have taught themselves how to handily beat humans in intelligent activities and thereby break the human monopoly on intelligence. The seeds of this were sown when the AI program called AlphaGo decisively defeated the world’s greatest Go players in 2016 [65, 66]. AlphaGo has achieved what many scientific researchers had dreamed of achieving. It means that a machine can teach itself in a tiny fraction of the time it takes humans to explore ab initio any axiomatic system. The last bastion of human supremacy over all other creatures on Earth in the form of intelligence has been cracked by AI machines. This is the world the millennials have stepped into. We have no idea how AI machines may organize themselves into networks and network with humans and vice versa. Will the future be written and created by humanoids with humans finding themselves relegated to footnotes and appendices once biotechnology and AI integrate? (See, e.g., [14].) So, what comes after Homo sapiens? Given the accelerating march of AI and computing, everything points to the dominating power of algorithms created and executed by quantum computers. It is a matter of understanding how to create novel DNA sequences and creating an environment for it to thrive. It is about writing lengthy books of life using natural and artificial nucleotides. With AI-embedded quantum computers capable of surpassing human intelligence, and the smartest among them developing Godlike abilities, the raw material they will be hunting for is massive amounts of data and mining that data for usable information for the welfare of one or more new species to whom the Homo sapiens will be ancestors. 6. Conclusions The stage appears set for some remarkable advances in synthetic biology including artificial speciation as an alternative to the natural evolution of species. Homo sapiens are now poised to change the evolutionary destiny of life forms (including their own) they choose to target and even design-to-order new life forms. The ramifications are far and wide (see, e.g., [67]). Creating species that can thrive on other planets, colonizing the Moon with single-celled life, etc. are no longer science fiction fantasies. We, the Homo sapiens,7 have been around for about 300,000 years [17, 18]. Records of our civilization date back approximately 6000 years. Since Homo sapiens are still evolving, speciation may yet produce superior creatures with new attributes that can give them superior knowledge of the Universe and its origin. After all, it is speciation that made the Homo sapiens overwhelmingly superior in intellect from the great apes and our cousins, the chimpanzees with whom we share 96% of our DNA sequence. “Darwin wasn’t just provocative in saying that we descend from the apes—he didn’t go far enough. We are apes in every way, from our long arms and tailless bodies to our habits and temperament.”8 Yet, at an intellectual level, within a span of few centuries, at the knee of the exponential curve that breathed energetic intellectual life into our neural and socioeconomic networks, we have attained such remarkable feats as formalizing and mechanizing axiomatic systems, discovering deep secrets of the Universe, partially mechanizing brain-mind activities, developing technologies that augment, supplement, and amplify our comparatively puny brain and brawn capacities. Within the past century or so, we have fathomed the power and limitations of rational thought and binary arithmetic to express it in, mechanized arithmetical calculations to unimaginable heights, and used this mechanization to develop robotics, 3D precision manufacturing, biotechnology, AI, QC, cloud computing, etc. These developments are now rapidly networking, the scale of which is such that we now see the combined effects of phase transition of graph theory in the Internet of Things (IoT) (creation and destruction of interlinked man-machine-idea components), of the logistic map in the rapidly changing socioeconomic scenarios that have increasingly made predicting the future at all levels of aggregating individuals a game of dice. The relationship between the logistic map and the Mandelbrot set implies that the future of Homo sapiens will indeed be so complex that a new species capable of handling that level of complexity must either evolve or be artificially created. The raw physical limitations of the Homo sapiens’ brain-mind system is distressingly visible in its waning ability to earn a living. Barring exceptional Homo sapiens, our search for meaning in life is now propelled by search engines roaming the Internet and not by our brains. The World Wide Web (WWW) has changed the way we think, what we think about, and how we communicate our thoughts. The millennials’ cognitive abilities are very different from those they were born with and weaned on before the Internet invaded their lives. They are shaped not just by what they read but by how they read. Not only has their lifestyle changed but also has their thought style. All the work of the mind—deep thinking, exhaustive reading, deep analysis, introspection, etc.—is now delegated to AI machines. Humans have thus relinquished their right to control their individual lives and direct their souls (maybe deep inside they already know there is no soul!). If machines can outdo humans so easily without a soul, then perhaps the soul is holding humans back from reaching their potential. Perhaps it is time, AI machines became our role models and our mentors [14]. Modern computers have made increasingly powerful and compute intensive mathematical algorithms accessible to even those not trained in science and mathematics for solving complex problems. Rapid advances in artificial intelligence (AI) and quantum computing show an inevitable trend that a vast array of human activities that till now required intelligent Homo sapiens to perform and earn a livelihood will soon be performed by AI-enabled computers, including the design of cellular life forms. When this happens, can human-designed speciation of life forms, its DNA coded for superintelligence, and other designed characteristics be prevented by the Homo sapiens’ instinct for survival? One day, nanotechnology will enable biocompatible, implantable, programmable quantum computers to be embedded into our organs or even introduce specialized new miniature organs, and we will be on our way to creating humanoids. We do not know how this will affect the speciation of the Homo sapiens. But before insight-driven complex experimentation aided by deep computing can happen; AI, new quantum algorithms, and embeddable quantum computers will have to evolve. Some early successes, for example, creation of artificial nucleotides, designed cells, attempts at resurrecting extinct species, etc. in molecular biology, indicate that once we master the biochemistry of very-very large molecules, for example, the DNA, RNA, proteins, by understanding their structure and their chemical-structural dynamics through quantum mechanical models, interactions between living and nonliving matter will undergo a sea change. We therefore anticipate a forced speciation of the Homo sapiens. It will drastically reduce the emergence time for a new species to a few years compared to Nature’s hundreds of millennia. Accelerated speciation by Homo sapiens via domestication, gene splicing, and gene drive mechanisms is now scientifically well understood. Synthetic biology can advance speciation far more rapidly using a combination of CRISPR technology, advanced computing technologies, and knowledge creation using AI. There is no reason why Homo sapiens themselves will not initiate their own speciation once synthetic biology advances to a level where it can safely modify the brain to temper emotion and enhance rational thinking as a means of competing against AI-embedded machines guided by quantum algorithms. Rapidly advancing research in the life sciences, while promising tools to meet global challenges in health, agriculture, the environment, and economic development, some of which are already on the horizon, also raises the specter of new social, ethical, legal, and security challenges. These include the development of ethical principles for human genome editing, establishment of regulatory systems for the safe conduct of field trials of gene drive-modified organisms, and many others. Additional concerns arise since the knowledge, tools, and techniques resulting from such research could easily lead to the development of bioweapons, facilitate bioterrorism, and the extinction of the Homo sapiens themselves. All these concerns are global not merely national [69]. The subject of this chapter goes beyond such concerns because here the concern is the possibility of self-initiated speciation of the Homo sapiens. The ramification of such a self-referential (iterative) process akin to that of the logistic map and the Mandelbrot set involving, in addition, phase transitions seen in graph theory is unknown. The perspective presented in this chapter is vastly different from that of Erwin Schrödinger (among the pioneers of quantum mechanics) expressed in 1944 [70]. Much water has flown under the bridge since then. A decade later, in 1953, when the structure of the DNA and its role in replicating life was discovered by Watson and Crick [43, 44], molecular biology was born. That led to genetic engineering [45] and synthetic biology [47]. As we write, CRISPR-Cas9 has been used to alter the embryonic genes of twin girls born in December 2018 in China [60, 61], which has elicited deep concern in the scientific community and an immediate response from the WHO: “Gene editing may have unintended consequences, this is uncharted water and it has to be taken seriously … WHO is putting together experts. We will work with member states to do everything we can to make sure of all issues—be it ethical, social, safety—before any manipulation is done” [71]. On the heels of this report comes the news that the world’s first baby born via womb transplant from a dead donor has been successfully achieved in Brazil [72]. With CRISPR, AI, and QC, the Homo sapiens are now on the threshold of creating new life forms and initiating even their own speciation. Notes Following William Occam (1287–1347) who recommended a principle of parsimony (les parsimoniae), famously known as Occam’s razor, "plurality is not to be posited without necessity". Its consequences on human health was recently highlighted in [6]. Prior to these papers, Homo sapiens were said to have been around for about 200,000 years. "OMIM contain information on all known mendelian disorders and over 15,000 genes. OMIM focuses on the relationship between phenotype and genotype. It is updated daily, and the entries contain copious links to other genetics resources." http://omim.org/about This we know from the explanation of the Maxwell’s demon paradox in thermodynamics. See, e.g., [37]. A controversial experiment to this effect seems to have been successfully conducted by He Jiankui who recently presented his work at the Second International Summit on Human Genome Editing in Hong Kong, November 27–29, 2018, http://www.nationalacademies.org/gene-editing/2nd_summit/index.htm [60, 61]. The term Homo sapiens was coined by Carl Linnaeus in 1758. A quote from Frans de Waal, a primate scientist at Emory University in Atlanta, Georgia, as it appeared in [68].

Do you ever wonder how this Artificial Intelligence thing can help in autobiopsy of brain and lung for infections, viral and tumors

 

Artificial Intelligence, 'Virtual Biopsies' May Be The Future Of Understanding Brain Tumors

Artificial intelligence i s being used increasingly in medicine to make services more efficient and improve patient care. Now, local researchers plan to use the technology to perform "virtual biopsies" of brain tumors

Brain Biopsy of a Suspected Cerebellar Lymphoma

AI System Approaching Neuroradiologist-level Differential Diagnosis Accuracy at Brain MRI 5,235 views•Jul 7, 2020

12 COVID Autopsy Cases Reveal the TRUTH How COVID 19 Patients Dying - Doctor Explains

The Pathologic Manifestations of COVID 19

Creative Sketch Generation

Sketch-related AI so far has primarily focused on mimicking the human ability to perceive rich visual information from simple line drawings [1, 2] and to generate minimal depictions that capture the salient aspects of our visual world [3, 4]. Most existing datasets contain sketches drawn by humans to realistically mimic common objects [5, 6, 7, 8]. Surprisingly, in spite its popularity, the role of sketching or doodling as a creative exercise has been largely unexplored in AI research.

Figure 1: Cherry-picked example sketches from our proposed datasets: Creative Birds (left) and Creative Creatures (right). See random examples in Figure 2.

In this work we focus on creative sketches. AI systems that can generate and interpret creative sketches which can inspire, enhance or augment the human creative process or final artifact. Concrete scenarios include automatically generating an initial sketch that a user can build on, proposing the next set of strokes or completions based on partial sketches drawn by a user, presenting the user with possible interpretations of the sketch that may inspire further ideas, etc.

AI for creative sketches is challenging. They are diverse and complex. They are unusual depictions of visual concepts while simultaneously being recognizable. They have subjective interpretations like aesthetics and style, and are semantically rich – often conveying a story or emotions.

Figure 2: Random sketches from existing datasets (left) and our creative sketches datasets (right).

To facilitate progress in AI-assisted creative sketching, we collect two datasets – Creative Birds and Creative Creatures (Figure 1) – containing 10k creative sketches of birds and generic creatures respectively, along with part annotations (Figure 2 right columns). To engage subjects in a creative exercise during data collection, we take inspiration from a process doodling artists (e.g., Michal Levy) often follow. We setup a sketching interface where subjects are asked to draw an eye arbitrarily

∗The work was done when the first author interned at Facebook AI Research.

4th Workshop on Machine Learning for Creativity and Design at NeurIPS 2020, Vancouver, Canada.

                                   QuickDraw Bird TU-Berlin Bird Sketchy Bird Creative Birds Creative Birds Creative Creatures (Jonge-jan et al., 2016) (Eitz et al.,2012) (Sangkloy et al., 2016) (w/ part labels) (w/ part labels)

 

around a random initial stroke generated by the interface. Subjects are then asked to imagine a bird or generic creature that incorporates the eye and initial stroke, and draw it one part at a time. Figure 2 shows example sketches from our datasets. Notice the larger diversity and creativity of birds in our dataset than those from existing datasets with more canonical and mundane birds. More sketch examples can be found in Figures 6 and 7.

We focus on creative sketch generation. Generating novel artifacts is key to creativity. To this end we propose DoodlerGAN – a part-based Generative Adversarial Network (GAN) that generates novel part appearances and composes them in previously unseen configurations. During inference, the model automatically determines the appropriate order of parts to generate. This makes the model well suited for human-in-the-loop interactive interfaces where it can make suggestions based on user drawn partial sketches. A schematic of our approach is shown in Figure 3.

(a) Training Phase (b) Inference Phase

Figure 3: DoodlerGAN. (a) During training, given an input partial sketch represented as stacked part channels, a part selector is trained to predict the next part to be generated, and a part generator is trained for each part to generate that part. (b) During inference, starting from a random initial stroke, the part selector and part generators work iteratively to complete the sketch.

Figure 4: DoodlerGAN generates sketches of hybrid creatures

We do quantitative evaluation and human studies to show that our approach generates more creative and higher quality sketches than existing vector-based generation model SketchRNN [3] and raster- based generation model StyleGAN [9]. The results can be found in Table 1 and Figure 9. In fact, subjects prefer sketches generated by DoodlerGAN over human sketches from the Creative Birds dataset! We find that our model often generates hybrids of creatures (e.g., a cow and a parrot, see Figure 4) – a strong indication of creativity!

   True part Predicted part

Cross Entropy Loss

GAN Loss

      Input partial sketch

Classifier

 Part generator:

                       Input partial Encoder StyleGAN2 Discriminator sketch generator

Part selector:

   Stroke sampler

Random initial stroke

Intermediate partial sketch 1

Intermediate partial sketch 2

Intermediate partial sketch N

Part selector

Part selector

Part selector

Part selector

Eye selected

𝑧~𝑁(0,1)

Head selected

𝑧~𝑁(0,1)

Beak selected

𝑧~𝑁(0,1)

Part generator (eye)

Part generator (head)

Part generator (beak)

Final sketch

                         Completion signal

        Squirrel Sea turtle

Cow Parrot

Kangaroo Frog Parrot Penguin

Panda Raccoon Snake Owl

Parrot Penguin

Penguin Mouse

Horse Cat

Bear Sea turtle

Shark Bird

Monkey Parrot

      Finally, we convert the generated raster images of sketches to vector representation results with open source software mkbitmap and Potrace. We also intentionally introduce ar- tifacts by controlling the stoke styles and shapes. We found such artifacts to be more visually appealing and preferred by human subjects 96% of the time. Examples are shown in Figure 5. Our datasets, code, a web demo, and human evaluation protocol will be made publicly available.

Figure 5: Vectorized sketches at higher resolution.

     2

References

[1] Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-Net that Beats Humans. In BMVC, 2015.

[2] Lei Li, Changqing Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, and Chiew-Lan Tai. Sketch- R2CNN: An Attentive Network for Vector Sketch Recognition. arXiv:1811.08170, 2018.

[3] David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. In ICLR, 2018.

[4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image Translation

with Conditional Adversarial Networks. In CVPR, 2017.

[5] Mathias Eitz, James Hays, and Marc Alexa. How Do Humans Sketch Objects? In SIGGRAPH,

2012.

[6] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The Sketchy Database:

Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (TOG), 2016.

[7] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. Quick,

Draw! The Data, 2016. Licensed under (CC-BY 4.0).

[8] Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning Robust Global

Representations by Penalizing Local Predictive Power. In NeurIPS, 2019.

[9] TeroKarras,SamuliLaine,andTimoAila.AStyle-BasedGeneratorArchitectureforGenerative

Adversarial Networks. In CVPR, 2019.

[10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS, 2017.

[11] Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. AI-Sketcher: A Deep Generative Model for Producing High-quality Sketches. In AAAI, 2019.

3

A Example sketches from datasets and generations

See Figures 6 and 7 for example sketches from the Creative Birds and Creative Creatures datasets respectively. See Figure 8 for randomly sampled sketches generated by DoolderGAN.

Figure 6: Random example sketches from our Creative Birds dataset.

 4

 Figure 7: Random example sketches from our Creative Creatures dataset.

5

 (a) Creative Birds.

 (b) Creative Creatures.

Figure 8: Uncurated creative sketches generated by DoodlerGAN.

6

B Evaluation

Table 1: Quantitative evaluation of DoodlerGAN against baselines on Fréchet inception distances (FID)) [10], generation diversity [11], characteristic score (CS) and semantic diversity score (SDS). CS that checks how often a generated sketch is classified to be a bird (for Creative Birds) or creature (for Creative Creatures) by the trained Inception model. SDS measures how diverse the sketches are in terms of the different creature categories they represent.

 Methods

Training Data

SketchRNN Uncond. SketchRNN Cond. StyleGAN2 Uncond. StyleGAN2 Cond. Percentage-based

DoodlerGAN

Creative Birds Creative Creatures

FID(↓) DS(↑) CS(↑) FID(↓) DS(↑) CS(↑) SDS(↑)

   - 19.40 0.45

79.57 17.19 0.20 82.17 17.29 0.18 71.05 17.49 0.23

130.93 14.45 0.12 103.79 15.11 0.20

39.95 16.33 0.69

- 18.06 0.60 1.91 60.56 15.85 0.43 1.22

54.12 16.11 0.48 1.34 103.24 14.41 0.18 0.72 56.81 13.96 0.37 1.17 57.13 13.86 0.41 1.17

43.94 14.57 0.55 1.45

    SketchRNN Unconditional StyleGAN2 Unconditional Percentage-based Creative Datasets SketchRNN Conditional StyleGAN2 Conditional QuickDraw Birds

     80 60 40 20

                                Creative

Looks Like like bird

Human drawn

Initial stroke integrated

Creative

Looks like creature

Like

Human drawn

Initial stroke integrated

Figure 9: We ran human studies using Amazon Mechanical Turk (AMT) on Creative Birds (left) and Creative Creatures (right) for five questions: which one of two sketches (1) is more creative? (2) looks more like a bird / creature? (3) they like better? (4) is more likely to be drawn by a human? For the conditional baselines, we also ask (5) in which sketch is the initial stroke (displayed in a different color) better integrated with the rest of the sketch? We evaluated 200 random sketches from each approach. Each pair was annotated by 5 unique subjects. Higher values → DoodlerGAN is preferred over the approach more often.

7

% times DoodlerGAN wins

Audio-reactive Latent Interpolations with StyleGAN

 Abstract

StyleGAN allows for fine-grained control of image generation through its hierarchy of latent and noise inserts. Using musical information like onset envelopes and chromagrams, latent vectors and noise maps can be generated to create interpo- lation videos that react to music. This paper introduces techniques that create convincing audio-reactive videos by combining multiple interpolations that react to individual musical features. The amount of musical information that can be visually recognized is maximized by mapping different musical features to different parts of the latent and noise hierarchy.

1 Introduction

There have been many music videos that use machine learning techniques [3, 11, 12, 17, 22, 23]. However, most either stylize pre-edited clips, suffer from size limitations, or do not visually react to the music. Lee et al. [14] propose a framework that learns to synthesize videos of people dancing from music. However, the architecture relies on domain-specific features and so is limited to dance videos only. Previous audio-reactive videos that use StyleGAN [7, 13, 15, 19, 20] are either focused on syncing the lips with the text, are not able to capture the wide range of elements in the music, or do not make full use of the expressiveness of StyleGAN’s architecture.

In this paper, multiple audio-reactive techniques are introduced that better capture the most important aspects of music. We also show how these techniques can be combined while preserving their individual clarity. This increases the amount of musical information that can be encoded visually by leveraging the full expressive capacity of StyleGAN.

The StyleGAN(2) [8, 9] generator uses a hierarchy of convolutional blocks each responsible for different scales of visual information (this is discussed further in Appendix A). Each block is controlled by a latent vector which encodes the semantic structure of that scale and a noise matrix which encodes the stochastic layout of the scale. This multi-scale control of image generation offered by StyleGAN is important for generating convincing audio-reactive interpolations.

Creating a video that reacts well to the music is a process that is unique to each audio source. The artist’s task is to find which features in the music can best be mapped to StyleGAN’s inputs. Spreading the modulation over different scales prevents the interpolation from becoming too chaotic. Good audio-reactive videos are characterized by how well separate visual movements can be recognized as corresponding to elements of the music. Therefore, having changes in the video which are clearly distinct from each other allows more "bandwidth" to represent the music.

2 Extracting Musical Feature Envelopes

Audio-reactive interpolations should correspond to the music both on a short-term scale and long-term scale. Harmonic and rhythmic features can be used to generate recognizable visual changes on short timescales. Mixing together multiple interpolations designed to match each section of the song ensures the global visual narrative fits the musical one, as well.

 4th Workshop on Machine Learning for Creativity and Design at NeurIPS 2020, Vancouver, Canada.

Short-Term Features The two most important musical features for local reactivity are the chroma- gram and onset envelope. A chromagram divides the frequency spectrum into categories by pitch (e.g. the 12 notes of the western musical scale). This gives separate envelopes which are high when the given frequencies are being played and low when they are not. The onset envelope peaks when there is a sudden change in the audio spectrum (e.g. when a drum is hit). This captures the rhythmic character of the music.

Often it is desirable to calculate these features for individual instruments, but sometimes it is not possible to find a band of frequencies where the instrument is isolated. One way to alleviate this is by applying harmonic-percussive source separation algorithms [5] and then filtering the separated signals. More recently, neural network based approaches also allow more targeted source separation [4, 6, 21].

Long-Term Features On top of these short-timescale features, the longer-term structure of the music can be incorporated. The RMS represents the average energy across the entire spectrum. Louder sections with more instruments will have a higher RMS value while softer sections will be low. Alternatively, algorithms like Laplacian segmentation [16] can be used to analyze the hierarchical pattern structure of music to automatically detect different sections.

3 Generating Audio-reactive Interpolations

An expressive audio-reactive interpolation is created by combining multiple sequences of latent vectors or noise maps that react to individual aspects of the music. Video examples showing results can be found at https://jcbrouwer.github.io/audio-reactive-stylegan-supplement.

Latents Two types of interpolations can be used as building blocks: looping patterns synchronized to a set amount of bars of music and chromagram-weighted sums of a set of latent vectors (discussed further in Appendices B and C, respectively). Using looping patterns tends to work well due to the repetitive nature of music. The chromagram-weighted sum creates a sequence where each note has a unique visual style which blend together based on the notes being played. Using either loops or a chromagram sequence as a base, other interpolations can be added by multiplying the new interpolation by the onset of an instrument and the base interpolation by its inverse. This causes a visual shift towards the instrument’s latents whenever it plays. Often a single, static latent is sufficient to mix in by onset, especially for transient envelopes like drums.

Noise For noise maps, the exact values have less effect on the resulting image. Rather the standard deviation, spatial location, and smoothness over time are visually apparent. Multiplying a noise sequence with drum onsets will increase its standard deviation whenever the drums hit causing a short, chaotic spasm.

Mixing The key to ensuring that different reactive components are recognizable is to map them to different parts of the noise and latent hierarchies. Both hierarchies can be split into about 3 levels: large structure, medium structure/color, and fine detail. On top of this, noise maps have a spatial component and so different parts of the image can be used by different reactive components (e.g. center vs. edges). Sequences designed for each section of a song (e.g. different noise intensities, different latents, faster loops, etc.) can be blended based on long-term features, like RMS, to ensure the video follows the narrative of the music.

Network Bending Another way of influencing image generation is by applying transformations to intermediate outputs before passing them to the next layer. This is called network bending [2]. Simple spatial transformations like zooming or rotating applied to lower layers propagate to the output image with a fluid, morphing quality due to the upper layers filling details into the transformed features. Classification networks can be trained to recognize which internal outputs are related to semantic features and transform those individually (e.g. scaling the eyebrows or translating the ears). A related technique called model rewriting [1] works similarly but applies changes to the weights of the generator instead. These techniques both open a wide range of specific targets for audio feature modulation which are clearly distinguishable from latent or noise effects–increasing the musical information that can be conveyed in a given period of time.

2

Broader Impact

The techniques described in this paper open new creative pathways towards expressive audio-reactive content. Hopefully this will enable artists in the future to design better accompaniments to their music and convey the messages behind their work more easily.

Generative networks like StyleGAN provide expressive control over high-level structural and textural aspects of images. Where previously artists would have to program a rich generative/procedural system manually, now GANs can be trained to learn representations of arbitrary image data. While the computational requirements to train such GANs are still high, further innovations will continue to lower the barrier to entry. This will allow people without extensive knowledge of visual design programs or music information retrieval to create audio-reactive videos easily.

This might automate away the jobs of today’s audio-reactive designers, but it would do so by opening the door for others to create such content more easily. The techniques described in this paper still need to be selected and fine-tuned for different songs or genres for the best results. This means that, for the time being at least, a human in the loop can still provide value over automatic systems.

One interesting consideration that arises from the use of generative networks is the question of how derivative the audio-reactive videos are. A network could, for example, be trained on copyrighted works, causing its outputs to strongly resemble the originals. On the one hand, this could be used for creative remixing of a music video to accompany a remix of the original song, on the other, it might enable profiting off of the visual style of an artist who does not provide consent or receive compensation.

As always there are secondary effects that have the potential to have a negative impact. However, hopefully people will instead focus their effort on using these techniques to create inspiring new works of audio-reactive art.

Acknowledgments and Disclosure of Funding

Thank you to Lydia Chen, Wiep van der Toorn, Thore Roepman, and Xander Steenbrugge for providing valuable feedback and support.

This work was partially supported by monetary contributions from Subverse bass music collective.

References

[1] David Bau et al. Rewriting a Deep Generative Model. July 30, 2020. arXiv: 2007.15646 [cs]. URL: http://arxiv.org/abs/2007.15646.

[2] Terence Broad, Frederic Fol Leymarie, and Mick Grierson. Network Bending: Manipulating The Inner Representations of Deep Generative Models. May 25, 2020. arXiv: 2005.12420 [cs]. URL: http://arxiv.org/abs/2005.12420.

[3] Mike Burakoff and Hallie Cooper-Novack, director. MGMT - When You Die (Official Video). In collab. with Lucia Pier and Jamie Dutcher. Dec. 12, 2017. URL: https://www.youtube. com/watch?v=tmozGmGoJuw.

[4] Alexandre Défossez et al. Music Source Separation in the Waveform Domain. arXiv: 1911. 13254. URL: http://arxiv.org/abs/1911.13254.

[5] Jonathan Driedger. “Extending Harmonic-Percussive Sepearation of Audio Signals”. In: ISMIR. 2014.

[6] Romain Hennequin et al. “Spleeter: A Fast and State-of-the-Art Music Source Separation Tool with Pre-Trained Models”. In: (2019), p. 2.

[7] Fly Jonathan. Scatman John - Scatman (Wav2Lip Interpolation Video). URL: https:// twitter.com/jonathanfly/status/1300976651380109313.

[8] Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Gener- ative Adversarial Networks. Mar. 29, 2019. arXiv: 1812.04948 [cs, stat]. URL: http: //arxiv.org/abs/1812.04948.

[9] Tero Karras et al. Analyzing and Improving the Image Quality of StyleGAN. Mar. 23, 2020. arXiv: 1912.04958 [cs, eess, stat]. URL: http://arxiv.org/abs/1912.04958.

3

[10] Tero Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation. Feb. 26, 2018. arXiv: 1710 . 10196 [cs, stat]. URL: https : / / github . com / tkarras / progressive _ growing _ of _ gans / blob / 35d6c23c578bdf2be185d026c6b3d366c1518120/util_scripts.py#L60.

[11] Mario Klingemann, director. Automatically Generated BigGAN Music Video. Dec. 12, 2018. URL: https://www.youtube.com/watch?v=U7Jz17ZZyOg.

[12] Mario Klingemann, director. Experimental Music Video Made with Neural Face Generator. Feb. 3, 2017. URL: https://www.youtube.com/watch?v=-elV0lT5-H4.

[13] Mario Klingemann, director. Hair by CoronaTM - StyleGAN2 Generated Music-Reactive Clip. In collab. with Pitx. Apr. 4, 2020. URL: https://www.youtube.com/watch?v= yWcYcA2HLNc.

[14] Hsin-Ying Lee et al. Dancing to Music. Nov. 5, 2019. arXiv: 1911.02001 [cs]. URL: http://arxiv.org/abs/1911.02001.

[15] Robert Luxemburg. Culture_shock.Py. URL: https : / / gist . github . com / rolux / 48f1da6cf2bc6ca5833dbacbf852b348.

[16] Brian McFee and Daniel P. W. Ellis. “Analyzing Song Structure with Spectral Clustering”. In: ISMIR. 2014.

[17] Jana Sam and Alex Mordvintsev, director. "Neverending Story" Music Video. In collab. with Cerebral Party and Alice Virt. Jan. 1, 2019. URL: https://www.youtube.com/watch?v= IVBmLRPiQjQ.

[18] Scipy.Interpolate.Splrep — SciPy v1.5.2 Reference Guide. URL: https://docs.scipy. org/doc/scipy/reference/generated/scipy.interpolate.splrep.html#scipy. interpolate.splrep.

[19] Xander Steenbrugge, director. When the Painting Comes to Life...: An Experiment in Visualizing Sound Using AI. In collab. with Kupla x j’san. Oct. 29, 2019. URL: https://vimeo.com/ 369531132.

[20] Xander Steenbrugge. WZRD. URL: https://wzrd.ai/.

[21] Fabian-Robert Stöter et al. “Open-Unmix - A Reference Implementation for Music Source

Separation”. In: JOSS 4.41 (Sept. 8, 2019), p. 1667. ISSN: 2475-9066. DOI: 10.21105/joss.

01667. URL: https://joss.theoj.org/papers/10.21105/joss.01667.

[22] Keijiro Takahashi. NGX: Pix2Pix VJ Generator & Mixer. Oct. 4, 2020. URL: https://

github.com/keijiro/Ngx.

[23] Pinar & Viola and Sofia Crespo, director. PANSPERMIA // Music Video for Aski - Se. In

collab. with Dark Fractures and Gene Kogan. Sept. 4, 2019. URL: https://www.youtube.

com/watch?v=6Ag_OQ7Ev5Q.

[24] Tom White and Soumith Chintala. Issue #14 · Soumith/Dcgan.Torch. URL: https://github.

com/soumith/dcgan.torch/issues/14.

4

A StyleGAN Architecture

The StyleGAN(2) [8, 9] generator uses blocks of progressively larger style-modulated convolutions with a noise insert at each scale. With an output resolution of 1024x1024 there are 9 distinct scales at which latent vectors and noise matrices are inserted to the network. The lower blocks, which have smaller spatial size, model large-scale structural information while the higher, larger blocks model the details. The latent vector controls high-level structural or textural information (e.g. face shape or hair color in a network trained on FFHQ) while the noise controls the stochastic variation in this information (e.g. the exact pattern of hair or skin pores).

Figure 1: Schematic representation of the structure of the StyleGAN generator. Dark blue boxes represent targets that audio features can control.

Figure 1 shows the structure of the StyleGAN generator. While there are differences between the architectures of StyleGAN1 and StyleGAN2, they do not affect the control points that can be targeted and so the details are left out of the schematic. The green boxes show the values that propagate through the network step by step. The dark blue boxes show the targets that musical features can be mapped to.

While there are 9 sets of 4 of these features in a network of resolution 1024x1024, generally the effects of consecutive blocks will not be distinguishable from each other when being modulated at the same time–especially in the highest blocks. This can vary per trained network, though, so groups of blocks that are visually separable should be evaluated on a case-by-case basis.

B Looping Latent Sequences

There are multiple ways to generate latent sequences that loop in a visually pleasing way: using spline interpolation, Gaussian filters, or geodesics on the unit n-sphere.

The spline approach is the most straightforward and works well when done in StyleGAN’s mapped latent space, W. For a collection of latents, calculate the 1-dimensional spline through each of their coordinates returning to the first latent at the end[18].

   Latent Vector Noise Matrix

StyleGAN Block

  1024x1024 (Final Image)

5

The second trick comes from the official implementation of Progressive Growing of GANs [10]. Here a 1-dimensional Gaussian filter is applied along the time dimension of a set of latents. This weighted moving average creates a smooth interpolation. This is useful to make any latent or modulation signal smoother. Note that with higher standard deviations of the Gaussian filter (which give smoother results), the latents are drawn closer to the average latent. Normalizing by the standard deviation of the latents after filtering retains output variety, however, this can cause the final path through latent space to miss the exact latent vectors that were originally filtered.

The unit sphere geodesics approach is less applicable in StyleGAN’s W-space as this is a semantically decoupled latent space. However, in the z-space, interpolating along these geodesics tends to give higher quality images than linear or spline interpolation. A discussion of this can be found in a GitHub issue in the DCGAN repository [24].

However the loops are generated, setting their length so that they repeat in time to patterns in the music helps create the feeling that the video matches the audio. While this trick relies on the human tendency to seek connections even when there are none, mixing a few other audio-reactive sequences into a base sequence that is simply looping creates convincing audiovisual interpolations.

C Chromagram-weighted Latent Sequences

A chromagram-weighted latent sequence can be created by taking the dot product of a normalized chromagram and a tensor of latent vectors. When the chromagram is normalized to sum to one at each time step, the result is an average of the latent vectors weighted by the spectral energy in the frequencies of each note.

Time

Figure 2: A 6-bin chromagram’s magnitudes are plotted in the center. The images corresponding to the latent vectors of each bin are along the right side. The chromagram-weighted sequence is along the top–the sum of each chromagram bin times its latent vector. The dotted lines show the point in time that each frame occurs.

Figure 2 shows the magnitude of each bin of an example 6-bin chromagram over time. On the right are the images which each latent vector produces and along the top are is the chromagram-weighted sequence. When the magnitude of the bin is high, its corresponding latent vector is blended into the image more strongly. Due to StyleGAN’s learned, semantically decoupled latent space, averaging latent vectors produces images that have traits of each of the latent vectors that are mixed in.

                        6

Note Presence

Humans can easily hear when different chords or notes are played and so using latent sequences that change according to these tones is a way to ensure that the video’s movements can clearly be recognized as stemming from the music.

D Post-processing Envelopes

Often signals such as onset or chromagrams calculated from audio will be noisy and unstable. This can cause unpleasant visual jitter or lead to visual changes which are less/more emphatic than their audio counterparts. While using better source separation or filtering the audio can help, there are a couple of ways to post-process these envelopes that can make a big difference as well.

The best audio-reactive results come from carefully tweaking combinations of these post-processing effects on each musical feature’s envelope.

Filtering A moving average or Gaussian filter (as discussed in Appendix B) helps eliminate jitter from envelopes. This reduces the dynamic range of the envelope so should usually be followed with a normalization step to compensate. For transient envelopes, the second half of the filter kernel can be reduced (or set to zero) to make the filter more causal (although the kernel must be normalized to sum to one again). This smooths the signal without looking forward in time. This helps prevent visual transients from starting to ramp up before the corresponding audio transient has occurred.

Compression and Expansion Another important set of effects are expansion, compression, and clipping. These effects work differently on the signal based on its value. For an example of expansion, a threshold can be set at 0.5, above which the signal is multiplied by 2 and below which the signal is divided by 2 (again, usually this is followed by normalizing the signal back to between 0 and 1). This creates a more extreme envelope which is only high when the musical feature is strongly present.

Flipping the multiplication and division to the opposite sides of the threshold has the opposite effect: compressing the envelope’s dynamic range. This is useful when the visual reaction is not strong enough compared to the audio.

Another version of compression is to simply multiply the entire signal and then clip any values above a threshold to that maximum. This creates flattened peaks which can be especially useful for long term envelopes (e.g. for mixing in latent sequences for different sections).