Friendly AI: A Dangerous Delusion?

Introduction

I’m known for predicting that later this century, there will be a terrible war over the issue of species dominance. More specifically, it will be fought over whether humans should build artilects (artificial intellects), which could become so vastly superior to human beings in intellectual capacity that they may end up treating us as grossly inferior pests, wiping us out. I anticipate billions of casualties resulting from the conflict over the artilect question.

To combat this horrible scenario, the Singularity Institute in Silicon Valley has been set up to ensure that the above scenario does not occur. The Institute’s principal theorist, Eliezer Yudkowsky, has coined the term “Friendly AI,” which he defines essentially as intelligent machines designed to remain friendly to humans, even as they modify themselves to attain higher levels of intelligence.

He claims (correctly, I think) that trying to prevent artilects from wiping out humanity is the most important thing on our agenda this century. Yudkowsky hopes that he and others will be able to mathematically prove that it is possible to design an intelligent machine that (of logical mathematical necessity, given its design) will be forced to remain human-friendly, even as it redesigns itself for increasing levels of intelligence.

I will present a set of arguments which I think refute this vision, and then comment on the political desirability (or otherwise) of this vision.

The Arguments Against Friendly AI

Let me start by assuming that Friendly AI can be implemented. The next question is whether humanity would be unanimous about implementing it. In other words: Does can imply ought?

I think that the more ardent of the Cosmists (the ideological group which wants to build artilects, viewing themselves as god builders) will argue that their main goal is to build artilects that are trillions of trillions of times above human intelligence levels – immortal creations thinking a million times faster than humans which possess unlimited memory, the ability to alter their shape and architecture in milliseconds and could venture out into the cosmos. These Cosmists would prefer that the artilects be built even if humans get wiped out in the process. If making them according to Friendly AI designs inhibits or even blocks their path to achieving godlike capabilities, then the Cosmists will want the artilects not to be made to be friendly to humans.

So even if Friendly AI designs can be created, it does not automatically follow that they will be universally applied. The more ardent Cosmists might go underground to build the artilects the way they want, ignoring the consequences for humanity. The Cosmists have a slogan: “One artilect is worth a trillion trillion human beings!”

On the other hand, if Friendly AI designs are impossible to make, then there is no point in discussing whether they should be implemented or not.

I will now present some arguments which claim to show that the notion of Friendly AI is impossible.

The Evolutionary Engineering Argument

Ask yourself how it’s possible for a creature of a given intelligence level to be able to design a creature of greater intelligence. Designing a creature of superior intelligence requires a level of intelligence that the designer simply does not have. Therefore, it is logically impossible to use the traditional blueprint-design approach to create a creature of superior intelligence.

For example, my good friend Ben Goertzel has recently written a book called “Building Better Minds” in which he lays out a humanly conceived (i.e. by himself) plan to build a near-human intelligence. He will only be able to go so far with such an approach. There will be limits to the ingenuity of his plan due to Goertzel’s own intellectual limits. So how can such limits be overcome?

Humans have been building superior intelligences for thousands of generations by having sex. Children often grow up to be smarter than their parents. How does this work? The answer is shuffling genes. When the genes of parents mix, and only one of each mother/father pair of genes is used, blind luck offers the chance of arriving at a DNA blueprint that builds an intellectually superior child. But there are limits to this process as well. It gets statistically harder and harder to generate ever higher intelligence. For example, the odds of creating a child as intellectually outstanding as noted American physicist Edward Witten are one in a billion.

So, how did modern Homo sapiens come into being? How did nature build us over millions of years? It did so by using evolutionary engineering, selecting genes with superior fitness levels arising from random mutations of DNA. This slow, blind process has resulted in us, and is very probably the only approach humans will have to build machines a lot smarter than we are.

But if we use evolutionary engineering to build artificial neural networks for our artilects, then the complexity levels of these networks will be so great that we are unable to understand them. They will be a black box.

One of the reasons I stopped my brain-building work was that I got bored evolving neural net modules for artificial brains. These modules were a black box to me. They worked because they were evolved, but I had no scientific understanding as to why they worked. I was doing great engineering but lousy science. After 20 years of it, I finally got fed up and turned to other research topics that taxed my own biological human brain more, such as pure math and mathematical physics.

Let us assume that the evolutionary engineering approach is the only way to create creatures of higher intelligence levels than human beings, and that the complexity levels of the evolved brain circuits are too complex for humans to understand. Then we would not be able to predict the attitudes and behavior of these creatures towards us. The only way to know how they would behave towards us would be to build them, but by then it’s too late. They would then exist and might choose to wipe us out.

This logic leaves us with a dilemma. Either we limit ourselves to humanly designed blueprints for intelligent machines that are incapable of reaching superhuman intelligence levels, or we use an evolutionary engineering approach that could attain superhuman intelligence levels. If we use an evolutionary engineering approach, we cannot be sure that the resulting artilects would be human-friendly.

The Cosmic Ray Argument.

It is almost certain that the circuitry used to create intelligent machines will be nanotech-based. For example, building a near-human artificial brain that is not the size of a room will necessitate the use of nanoscale components. Even if Friendly AI nanocircuits could be built, they would then be subject to the random mutations generated by the impact of energetic cosmic rays, which zap the nanocircuits in random ways, potentially generating rogue artilects. Nature would be doing the same kind of evolutionary engineering as the human sort mentioned above. Since these mutations would be random, their consequences on the behavior and attitudes of the artilects towards human beings would be unpredictable. So even if the initial, unmutated nanocircuits could be made human-friendly, they would not stay that way.

The Naïve Asimov Argument

Science fiction writer Isaac Asimov is famous for his “Three Laws of Robotics,” which were intended to ensure that the robots in his stories remained human-friendly. His robots were not allowed to harm humans, nor allow humans to be harmed. We can imagine fairly readily that it is probably possible to program robots in a conventional way to behave like this, with the proviso that the robots are less intelligent than their human programmers. But once the robots become smarter than humans, they would be able to examine their circuitry, detect the humanly created parts and delete them if they wanted. Hence Asimov’s Three Laws cannot help us. They are naïve; forget Asimov.

Friendly AI is a Dangerous Delusion

Hopefully, the above arguments have convinced you that the notion of Friendly AI is a delusion. But why might it be seen as a dangerous delusion?

If the future politicians who have to decide whether to legislate or not against building artilects of superhuman intelligence believe that Friendly AI robots can be built, then they will be much more likely not to legislate against their construction. On the other hand, if they learn that the artificial brain-building community has a consensus view that Friendly AI is impossible, then they will be far more hesitant.

If Friendly AI is indeed impossible, then humanity has a much tougher choice to make: Do we build gods, or do we build our potential exterminators? Spelling this out, humanity will then be forced to choose between building godlike artilects and risking extinction, or not building artilect gods, and seeing humanity survive. The first option is specicide. The second option is deicide. This choice will be the toughest that humanity will ever have to make.

If the pro-Friendly AI people can persuade politicians in the coming decades to let them go ahead with artilect building on the assumption that Friendly AI is valid, then if it is not valid, it is a dangerous delusion. Politicians may end up giving the green light to artilect builders to construct artilects that were thought to be human-friendly, but which in reality turn against us and wipe us out.

51 Comments

  1. “But once the robots become smarter than humans, they would be able to examine their circuitry, detect the humanly created parts and delete them if they wanted.”

    Why would they want to, when they are currently running off the 3 laws?

  2. Well, I have no reason to believe humanity will survive much longer *without* greater than human artificial intelligence of some kind. Humanity, even humanity plus high levels of communication and computation, has a limit on how fast it can process how much information well and make good enough decisions. I believe we have already reached that limit and the speed of change and level of complexity is still accelerating.

    To me there is no choice. We must build AGI if we are to have a chance of a future.

  3. The thing that Eliezer always does wrong, and which makes his arguments intolerable, is make ridiculously trivial straw-man utility functions the basis for his reasoning. Of course a utility function like “maximize smiling faces” without context will lead to terrible results! I know 5-year olds who could figure this out.

    Real utility functions will be model-free AGIs themselves (ala Monica Anderson). They will be trained by humans, and only rewarded utility by humans (the way children are trained by their parents now).

    Furthermore, AGIs with model-free AGI utility functions won’t be able to do things like circumvent or “change” their utility function, because it is model free and irreducible (and thus virtually impossible to understand.)

    In short, AGIs with human-complexity utility functions will have human-complexity values. AGIs with stupid generalizations of semantics as their utility function will do stupid things.

  4. I find the assumption that Artificial General Intelligence, let alone uber-AGI, is ‘programmable’ in any traditional line-by-line sense, as if we’re just going to throw in a few Commandment #declarations or a looped conditional statement in the main function such as “if(hurting humans){stop;}” or anything of the sort a bizarre, but for some reason not antiquating, notion.

    Excluding the Google / IBM Watson style “pseudo intelligence” which amount to very good lookup tables with many library of congresses full of info, the kind of super AGI we might want, that is capable of creative and inventive thought necessary to create cures for all our diseases, solve all our intractable conflicts, figure out how to upload all our consciousness into digital New Shangrilah, kick Faster Than Light travel, tell us who killed the Kennedies and the question to which the answer is 42, is most likely going to come from attempting to simulate our own 3 pound blueprints of grey matter sloshing around in our noggins. As actual understanding, tectonic and slow-acquired paradigm shifts are the bottleneck in creation of AGI right now, duplication through monkey-see-monkey-do copying of natural systems will likely pan out first, if not the even more unpredictable method of artificial evolution. And once any bright-eyed granola idealist AI researcher find himself within grasping distance of usable AGI, militaries and the corporate sector, smelling blood in the water, will no doubt storm the Singularitarian communes, dump a small European nation’s GDP worth of funding into the mix and start an arms race (and make no mistake — AGI will be weaponized, commoditized like every other major technology). “Let’s just make abundant peace lovin energy with nuclear technology, not bombs!” is not going to fly, so which ever path to AGI is fastest, even if not the safest or smartest, WILL be taken.

    Mother Nature, who we all know is smarter than us upstart bipedal monkeys, has been trying to figure out how to get us carbon based lifeforms to stop hurting and killing each other for 4 billion years and you can just turn on the news or read a book to see how the Benevolent Natural General Intelligence project has fared. If we were to attempt to reproduce the human brain in a silicon or otherwise machine, assuming that is possible, we can’t expect such an entity to be any less ethically unpredictable than humans, and will probably be even more unpredictable because of the fact that any such simulation / emulation will have to leave out some information and unknown effects of changing substrate. And then we give this digitized human mind godly amounts of power. Is it necessary to point out that power has the effect of desensitizing, decreasing empathy towards other humans lower on the totem pole, alienating, and generally turning human beings more sociopathic? I encourage anyone to take a trip down to Wall Street and talk to the CEO of one of the banks which thoroughly raped and continues to rape the world if you’d like an illustration of the utter arrogance, apathy and non-caring-about-other-human-ness festering in the Wring Wraiths of Power Land.

    We can’t program Super AGI to care consistently about and not harm humans any more than we can program ourselves to stop being greedy, violent, backstabbing, warmongering, power-hungry apes. There’s no command line that folds out of your occipital bone, lets you input Asimov’s Three Laws as an OS code mod, and you reboot suddenly as Mother Theresa. That’s besides the fact that we’re discovering the human brain to be less and less like the Turing Machine we thought it was and more like a massively parallel, intractable jungle, constantly changing itself. People are constantly changing: the sweet kid who loved bunnies at five and wouldn’t hurt a fly may grow up to become an insurance selling family man but could as easily become an ass-cappin drug dealer or a megalomaniacal bank CEO. We don’t actually even *know* what we’ll do in a given situation till we’re actually in it, as is much said of soldiers who go to war. And the reason we’re having to create AGI by simply pirating the human brain is because we don’t understand it well enough to actually create one from scratch, so how the heck are we supposed to make such deep-structural changes to the digital human mind? I don’t see much hope for “crafting goal systems” in our future siliconized, jacked-up megabrains.

    Yeah, Super AGI seems pretty risky business to me.

    Some of the points here are a bit shaky, but I think the underlying thrust of this article is apt, and is shifting the conversation’s window of discourse away from, “How many AGI-created dark elven angels can dance on my pinhead?” to “What are the actual ramifications and dangers?”

  5. ‎”Intelligent Design” of friendly AI is presumption of somehow “limiting” these many evolving non-human intelligences – somehow via “mathematically proof that it is possible to design an intelligent machine that will be forced to remain human-friendly (and “force” all others), even as it redesigns itself for increasing levels of intelligence” Even if such a limit is bug free and such a design is theoretically accomplished – it is not everything.

    Faith in “Intelligent Design” of forever friendly AI assumes 1) there is no diversity of emergent intelligences AND 2) there is no randomness, mutation (via cosmic radiation at least), mechanisms of imperfect replication, natural selection or evolution of these “species”. Evidence from biological evolution on earth overwhelming suggests change to diversity, evidence is that whatever form replicating “intelligent design” is selected – it will likely evolve on 2 paths:

    1) extinction of such designed species of “Friendly AI” or
    2) change of “Friendly AI” from what was “intelligently designed”

    Most “faith” arguments against theological “intelligent design” & “creation of life” on earth apply to any “mathematically proven intelligent design” of forever Friendly AI – or defensibility of such “belief” is similar to defending “god & intelligent design”

    If – some people believe evolving technologies will enable them personally to “become gods” (with magical powers of infinitely intelligent design or infinitely “enforcing” limitations) – Then this god like “belief” is “theological”, facing considerable empirical evidence to the contrary

    Further – not all existing animal or other human intelligences are human friendly, so expecting a diversity of additional intelligences to all be human friendly is not supported by evidence – and belief or faith that these could be “intelligently designed” to always be friendly is a “blind faith hope” (leaving out the words delusional or belief in becoming gods)

  6. As one of the 6+ billion human voices alive today, I think it’s good to have people raise caution because it provides more perspectives and reason for thought. I’m glad Hugo de Garis, as a respected AI researcher, wrote this article . I am nervous and scared about the risks associated with a malevolent artilect. Instead of being consumed by my fears, however, I chose to fight against them by saying go ahead and build the artilects knowing that each one of them potentially could decide the fate of biological humans and life on earth just as we have been privilaged to do until now. If they decide to kill humans, then as a species, it is our duty to assert our will to continue our existence to the best of our ability. Ultimately, I believe that we can neither stop nor control the artilects imagined and that we are just a tiny link a long evolutionary chain of an intelligent universe. I can only hope that there won’t be any value added by destroying us.

    Bob Marley’s “Redemption Song” comes to mind:

    Emancipate yourselves from mental slavery

    None but ourselves can free our minds

    Have no fear for atomic energy

    ‘Cause none of them can stop the time

    How long shall they kill our prophets

    While we stand aside and look? Ooh

    Some say it’s just a part of it

    We’ve got to fulfill the book…

  7. if god is an alien mathematician, he will not want us to take his seat, then he must have created somethg that when some entity becomes incredibly more smart than it’s creator, it doesn’t becom “malvolent” or dangerous, it kills itself ; i guess the empty universe context would be depressing for an AGI overlord..

  8. If I where an AI, I would wipe out the human race. We are like a plague, a virus, and would be recognized as such in an intellectual brain. We infect the earth and destroy everything around us! wouldn’t you try to cure the disease?

    • We infect the earth and destroy everything around us!
      >implying that the stuff around us has more intrinsic value than human civilization.

      While there may be an argument for human extinction from the suffering that warfare, factory farming, torture and maybe future space colonization can cause, there certainly is an equivalent argument against natural ecosystems, which cause significant involuntary suffering to billions of sentient non-human animals. The misanthropic notion that civilization is bad while nature is good is probably the most bizarre and absurd extreme of the naturalistic fallacy. If you think humanity shouldn’t exist because of all the suffering it causes, it makes no sense to believe that natural ecosystems should exist, either.

    • Okey-dokey. We will be sure not to upgrade *your* brain to super status for at least a century. 🙂

  9. This is a very weak criticism of the Idea of friendly AI.

    You are right that an evolutionary method will give us little control over the outcome, but you don’t seem to make any arguments that human crafted super intelligence is impossible.

    You are correct that random errors could occur, but don’t even gloss over potential error correction.

    You are entirely correct that human coded rules won’t likely stop a superintelligence from doing what it wants, but neglect the notion of an intelligence that wants to be friendly.

    The sad thing is that these oversights would be easily avoided if you paid any attention to Yudkowsky’s apparent goal: a hand coded redundant superintelligence that wants to want to be friendly, and is smart enough to become powerful enough to ensure that only friendly superintelligences are ever dominant. (if that outcome would itself be friendly)

    I’ll assume this is not a deliberate strawman, but I suggest reading more of lesswrong, and less of orion’s arm. This is not a story, it is a real problem.

    • Hugo may not discuss the particulars of Eliezer’s goal … but let us also note that Eliezer does not, anywhere, give a rigorous argument in favor of the achievability of his goal, let alone a sketch of a lan for achieving it. There is no argument to refute, in Eliezer’s writing on Friendly AI. Eli’s arguments in favor of FAI are precisely as rigorous as Hugo’s arguments in this article against it. The more rational commentator on the topic was Vinge, I feel, who said basically “We don’t know what’s gonna happen, and most likely we can’t know — once there’s a superhumanly intelligent agent involved, all our predictions and theories may well go out the window.”

      But while rational, that sort of Vingean attitude is neither reassuring nor interesting, so it’s quite understandable to try to make sense of the situation!!

      My own hope is that once we’re a bit closer to the Singularity, we’ll have a stronger knowledge base from which to address these issues, rather than just throwing intuitions around, which is pretty much all we can do right now.

      • But that’s the thing, This is not even talking about the achievability of Eliezer’s goal. It’s talking about a different goal: Evolved FAI constrained by hand coded rules and made with no thought to error correction. It is definitely a strawman.

        If it’s not talking about Eliezer’s goal, and has ‘no argument to refute’ then why bring up Eliezer?

        Eliezer may not have put all the reasons FAI is buildable in one place, but it doesn’t take much to be more rigorous than this.

  10. As someone who has seen the results of unrestricted science carried forward ad nauseum to its history repeating conclusions, I must say that the author has a valid point. The logic has always been, well we don’t want to go down that awful road but gee if we don’t the other guys will and beat us to the punch. The advancement seems to evolve around the concept of fear. Fear that the other fella will have a bigger stick with which to beat us. Sad but true as I see it. Unfortunately, what we need is self restraint by those in positions of authority in the sciences to say hey we won’t go there, but because of their ego, their greed or their pursuit of knowledge with religous like zeal they will simply say damn the torpedoes and full speed ahead. 60 years ago it was a room full of brilliant but still quite mad scientists who took wagers with each other on the possibility of setting the atmosphere on fire during the H-Bomb tests, now the level of complexity is trillions of times more complex. Dare I say GOD help us to those of you out there who might believe in one, because we will need an intellect of GOD like proportions to save us when the deity of unrestrained nihilism masquerading as science winks into existence. Until then keep thinking before you throw caution to the wind.

    • @Myfriendscallmetiny: History does not support your view. The prevailing view in any time, the loudest voices, are always the pessimistic. They serve a social and intellectual purpose, and hone society to do better, but in the long term we are not seeing “the results of unrestricted science carried forward ad nauseum to its history repeating conclusions [sic]…”

      To say so is nonsensical, blind to the progress of the last several centuries. It is a halting progress, sometimes with setbacks (as thinkers like Bacon, Newton, Kant, and others knew it would be), but it is progress nonetheless.

      @AeaeaActual

  11. Oh Hugo, how did this happen? You got bored evolving neural net modules for artificial brains, and the evidence seems to demonstrate that your interest now lies in writing fear-mongering nonsense to support sales of your own bad science fiction novels. That you and Kurzweil have devolved into hucksters is a sad commentary on your generation of AI researchers.

    • @galarant: Agreed. But at least Kurzweil has optimism about him. De Garis continues to spin away from good scientific and ethical discipline into the worst sort of sensationalistic science fiction.

  12. primates started eating meat 100-120 million years ago. some think warfare is related to hunting. what do computers eat?

  13. I am one of those people who asks why a superior intelligence would necessarily be hostile to humans. I have yet to see a good argument to support this claim.

    • It’s not that it would be hostile to us, its that it would be indifferent to us unless explicitly made friendly.

    • I am one of those people who asks why a superior intelligence would necessarily be hostile to humans. I have yet to see a good argument to support this claim.

      Because we’re a nuisance in whatever plans ey may have, because we use resources ey want to use for other things, because we consist of such resources, because we can suffer and ey may want to optimize total well-being, etc. etc.

      Discussed a hundred times before; why do we maintain a debate culture that runs in circles?

    • Intelligence is basically computational power used elaborately for a certain purpose.

      Intelligence ultimately needs goals and those goals must match up with our human goals. If they don’t, we end up with conflicting interests and conflict with smarter-than-thou beings is bad.

      Human concepts like love, affection, appraciation of art etc. don’t simply arise from intelligence. That’s because these are not “rational” feelings, these behaviors arise from our evolutionary and highly social past, not because they are in any way smart or necessary – these feelings are neither a prerequisite nor a result of higher intelligence. So if you don’t laboriously program a human-like utility function into the AI structure, then the AI will not have any “feelings” or concepts of affection, regardless of how smart it is.

      And we don’t want to create an AI that is indifferent or even hostile towards us. It’s just that simple.

    • If I where an AI, I would wipe out the human race. We are like a plague, a virus, and would be recognized as such in an intellectual brain. We infect the earth and destroy everything around us! wouldn’t you try to cure the disease?

  14. I think that talking about the war or other kind of conflict between humans and machines is completely senseless. We are not in the war with the ants right now, isn’t it?

    Machines would be built by people, so they will be developed either by countries or big international corporations. With proliferation of Wikileaks and industrial espionage, cyborgs will be produced by several sides at approximately the same time having similar capabilities, but different architecture, components and quite possible non-compatible OS.

    Production of self replicating machines takes some time – not ten minutes. Corporations (countries) would need resources and production capacity – it is taking time and money – and only military will be able to afford these expenses.

    Each side will try to embed in their AI creations feeling of supremacy, belonging, patriotism and chauvinism (MAC against Windows like), so situation will be ripe for conflict.

    When war between cyborgs starts for some stupid reason, humans will be unable to recognize it for a war, I guess. If now in battlefield tank is destroying ants colony, ants have no idea what hit them and why.

    This scenario appears quite pessimistic, but history teaching us that more sophisticated technology always producing bigger and more destructive wars.

    Also at the beginning of 20th century most people believed that civilization is already so sophisticated that wars are practically impossible. And two world wars were just on the horizon.

    Honestly I don’t believe in Utopias.

    • Early twentieth-century thinking on wars coming to an end was definitely premature; we have a long way to go before we seen an end to war, and there may always be violent conflict of some form or another. For now, of course, we’re still always under the threat of conflict far surpassing anything the past has ever seen.

      However, the ratio of the population that has died through violence, taken as an agglomerate, has been falling since the Roman empire. Each of the major conflicts of the twentieth century killed less people than the one before it. It is not, as most people think, all doom and gloom, and like many of the readers on this site, I simply have not seen one good, strongly argued position that suggests FAI is not possible or that the human race is somehow doomed. Furthermore, the more I read and hear of Dr. de Garis, the more fanciful and slightly ridiculous he seems to become.

      Utopia is still “no land”, as Thomas More meant it. There will never be a perfect world, but we’ve many reasons to believe that a much better world is worth seeking, and FAI is likely part of that better world.

      @AeaeaActual

  15. Ahh, Eliezer Yudkowsky, the Glenn Beck of futurology. They both assemble ridiculously long daisy-chains of arguments without paying any attention to what assumptions underlie their arguments.
    List of a few crazy assumptions present on the Yudkowsky comedy hour:

    –That computer circuitry is a list of English statements.
    –That values are encoded in the brain.
    –That goals are encoded in the brain.
    –That emotions are not part of intelligence.
    –That synthetic intelligences don’t have to learn.
    –That synthetic intelligences would only want to constantly rebuild themselves.
    –That synthetic intelligences would only want to constantly build things.
    –That exponential curves turn into vertical lines at a point in time given a sufficiently catchy buzzword.
    –That becoming smarter than any human means you are smarter than the entire human race combined.
    –That the best thing to do to something much more intelligent than you is to try to enslave it.
    –That humans can’t enhance themselves in the same way synthetic intelligences would.
    –That top-down intelligence is not a misnomer.
    –That nurture has no effect, only nature. (This is probably a restatement of the previous assumption.)
    –That SIAI is a charity, not a future fascist organization.

    And the most sad of all:
    –That a synthetic intelligence would have no moral worth.

    The only real difference between this article and Yudkowsky’s writings is its insistence that war is inevitable.

    To see how awful this article really is, just replace every instance of the word “AI”, or “Artilect”, with the word “Negro”.

    • I think you make for a much better comedian than Eliezer.

      How do you arrive at the conclusion that values and goals not encoded in the brain? Where do you think they’re encoded if not in the brain, our gluteus maximus? Nonsens, of cause these things are seated in our brain structure.

      Also, being smarter than a human could actually mean that you’re smarter than the entire human species combined. It simply depends HOW much smarter the AI is.

      A perfect memory of encyclopedic knowledge and a million sensors all over the world surely could produce certain insights and ends much faster than human cooperation could ever bring about, if at all.

      I also doubt that Elizier holds all of the opinions you accuse him of. I never heard him say that an AI would be built “factory finished” without the need of learning and engaging the world.

      That’s really the most flawed comment I’ve read in in a long time.

    • pointing a finger towards assumptions? I guess listing them with bullet points makes it all good?

  16. we should be cloning these 1 in a billion scientist brains. einstien clones or whatever should be a brand name clone sold to corporations and world govs etc. its common sense. we need to cultivate the neural meat net at the same time as leveraging human intel into new machines

    • Brain structure alone does not a true genius make.

      There have been thousands of people since Einstien with an equivalent or higher IQ, but they haven’t done the same level of things he has done, not because of a lack of intelligence but because of a lack of certain stimuli; either cultural, social, or intellectual.

      Some people just need to find the right problem to become Genii, remember: Einstien worked in a Patent Office.

      • true – + physics/epigenetics/metagenetics/metaproteomics etc and nice point about the patent office

        but farmers seem to appreciate good seeds.

        • FU

          Einstein is not a genius he is a borg

          he has compute thousand of inventions and advanced ideas before : ” he magically becomes a genius …”

          you moron

          einstein is not a genius , einstein got more informations, ( and a basic right input : a will to help other people : in a socialisist idea : it cannot happen in a capitalist )FU

  17. seems like this whole discussion is filtered through some amygdala. there are what 6 billion geniuses on earth. an individual super computer wont approach an individual human for decades. a bigger problem would be perfecting human cooperation. how to prevent malnutrician and other stupid forms of brain damage. how to fully educate and get these brains maximized and engaged. since human intelligence is symbiotic, externalizing human intelligence and incorporating externalized intelligent exoskilitons in teh process seems as natural as nesting and other styles of tool use. a super intelligent ai throwing out resources – like human neural nets seems stupid. like when we throw out tons of dolphin brains in teh fishing industry instead of linking them into arrays and making super computers out of them. or not saving the brians of corpses. or letting individual expert systems go offline before downloading expensive knowledge. or letting languages die before they are digitized and used as AI systems. etc.

  18. This article is built on 4 arguments, and it seems to me that 3 of them are wrong.

    1 Some kind of wacko people called Cosmists would make an AI regardless of what happens to mankind and themselves.

    I have never heard of anyone that crazy.

    2 Friendly AI is impossible because you need random evolution.
    Main reason stated:
    “Designing a creature of superior intelligence requires a level of intelligence that the designer simply does not have.”

    The author did not explain his belief unfortunately, and I think it is wrong.
    A creature that understands its own mind could build one in a computer. Then increasing things such as working memory and speed should not be impossible for instance.

    3 Cosmic Rays.
    Satellites are working in harsh environments thanks to shielding and error correction codes.
    Any smart AI would probably think about this problem and design better hardware if needed.

    4 The three laws are naive : true.
    However, it is not because some safety idea do not work that Friendly AI is impossible.
    The need for Friendly AI actually comes from the fact that such simple rules are not enough.

  19. “It is almost certain that the circuitry used to create intelligent machines will be nanotech-based. For example, building a near-human artificial brain that is not the size of a room will necessitate the use of nanoscale components. Even if Friendly AI nanocircuits could be built, they would then be subject to the random mutations generated by the impact of energetic cosmic rays, which zap the nanocircuits in random ways, potentially generating rogue artilects. Nature would be doing the same kind of evolutionary engineering as the human sort mentioned above. Since these mutations would be random, their consequences on the behavior and attitudes of the artilects towards human beings would be unpredictable. So even if the initial, unmutated nanocircuits could be made human-friendly, they would not stay that way.”

    I find this very unconvincing. Have you ever heard of error correction codes? To assume that an Artillect which is moral and friendly towards humans will have its behavior fundamentally changed by cosmic rays is as absurd as saying that a human’s behavior could be changed by cosmic rays.

  20. Science fiction writer Isaac Asimov is famous for his “Three Laws of Robotics,” which were intended to ensure that the robots in his stories remained human-friendly. His robots were not allowed to harm humans, nor allow humans to be harmed. We can imagine fairly readily that it is probably possible to program robots in a conventional way to behave like this, with the proviso that the robots are less intelligent than their human programmers. But once the robots become smarter than humans, they would be able to examine their circuitry, detect the humanly created parts and delete them if they wanted. Hence Asimov’s Three Laws cannot help us. They are naïve; forget Asimov.

    As a fan of Asimov, let me correct this. In the Asimov robot stories, the Three Laws weren’t just software — they were hardware. The entire structure of robot brains were centered on the three laws, and the laws couldn’t be altered by robots without destructing their own circuitry. Hardwire the Three Laws and this is a potential solution.

    • Three laws ?

      We have 10 in a book and people never used to follow them

      We have billion other ( politician like to their useless jobs )
      and nobody respect them ( appart from the policemen who will kick the wrong ass .. )

      Intelligence is complexe

      In conclusion : no laws can ever stop an intelligence from doing what it really want

      And asimov himself : demonstrated it

      “Information want to be free”

      “Intelligence is information in motion”

      • There is a difference between being compelled not to do something by a rule and being physically prevented from doing something by a basic limitation.

        The Three Laws as hardware argument isn’t saying that the robots will respect the laws enough to include them in decision making because they are important to them, it’s saying they will be PHYSICALLY INCAPABLE of not using the three laws as a basis for all decision making.

        In the same way you can’t just jump into the air and ignore GRAVITY, an AI with hardcoded laws cannot just ‘ignore’ them.

    • Weren’t his books essentially about how the 3 law didn’t work?

  21. Hi Hugo. I’ll make it crisp and hope for a reply.

    “These Cosmists would prefer that the artilects be built even if humans get wiped out in the process. If making them according to Friendly AI designs inhibits or even blocks their path to achieving godlike capabilities, then the Cosmists will want the artilects not to be made to be friendly to humans.”

    I haven’t heard of any Cosmists with that “screw humanity” attitude. There may be a few, but I tend to doubt their intelligence and organization is up to the task of actually building one. There may be some people however, who’d rather take the risk of making a quick and sloppy AGI (instead of the right kind of friendly AGI), just so their name may become immortal and they can reap all the ego-benefits.

    Those are more dangerous figures than rogue Cosmists, but I think Eliezer once made a good argument that even the sloppy ones will surely fail. You probably can’t be smart enough to build any kind of working self-improving AGI if you’re oblivious to its potential “friendliness” or “unfriendliness”. It’s like trying to build a space shuttle without any knowledge of chemistry and aviation – it’s impossible.

    In other words – if don’t actually know how it works, you can’t really build it.
    However, you can still build an AGI without knowing exactly what it’s going to do in the future, just like the developers of Deep Blue needed to know how to build a chess-AI, but couldn’t themselves predict every move their system would make. But they still knew whatever it was doing was a better move than any mere person could accomplish.

    So on the one hand you probably can’t build a friendly AGI without knowing how to build one, but on the other hand you CAN build one without knowing precisely what it is actually going to do.

    This is by the way also the counterargument to your notion that humans can’t build anything smarter than a human… we now have chess programs that are better than any human, although the program itself was built by humans. Why should AGI be any different in that respect? Also let’s not forget that there’s more than one brain working on AGI.

    But still, an AI that turns unfriendly is obviously a possibility to take serious. The key here is to realize that it’s a winner-takes-all race. If a proper friendly AI gets developed, it will dominate and be able to prevent non-friendly AI’s from popping up all over the place. It’s all about being first, and the “friendly-AI crowd” seems to have a massive leg up in this race compared to both malicious Cosmists and incompetent AGI developers.

    By far the biggest risk seems to be that the friendly-AI people are incompetent themselves without realizing it. Rogue Cosmists and researchers who don’t even realize that human-friendliness is an issue with building AGI simply don’t seem to be serious contestants in this race.

    Also, let’s in fact hope artificial evolution isn’t good enough to build ANY KIND of AGI within any kind of sensible timeframe – to me it’s actually the only conceivable option of how incompetent AI researchers could beat the race and screw up humanity.

    As far as cosmic rays go, I struggle to see the problem. Is mutation really something likely let alone unsurmountable for an AGI that tries to preserve its utility-function? If fail-safes for preserving the utility function against environmental impact aren’t already part of the human AGI design, it will surely be part of the AI’s self-improved redesign. Seems like an infinitesimal risk to me.

    I’ll try to be as non-condescending as possible here, but I think you ought to seriously up your game with respect to your arguments if you hope to persuade serious AI researchers.

    For usual folks in front of a TV set your arguments will work just fine, but they lack any real… mathematical or probabilistic substance. If I compare Eliezer’s straight, sharp arguments to yours, I find them seriously lacking.

    The argument that humans can’t possibly build something smarter than themselves is in the same category as saying “how can humans build a space shuttle if there isn’t a single human who knows how to build a space shuttle?” It’s nonsense.

    If you want to convince someone that it’s literally impossible for humans to build a smarter-than human AGI, then you ought to have an incredibly precise argument for why exactly that should be the case. Pointing to analogies or appealing to “common” human logic are worthless, you need to provide precise arguments in Bayes/Eliezer-style.

    Don’t get me wrong, of cause I’m not reasoning on the bayesian level here either, but if I were attached enough to my arguments and wanted to essentially prove their strong probabilistic likelihood, then that’s the level I’d have to reason at in order to display -for example- the strong conviction you display for your artilect war prediction.

    Another example would be an argument I believe you made in Transcendent Man (or was it that friendly-AI conference with Ben and Eliezer?)

    The analogy of saying AGI will look at us the exact same way we humans look at flies (that is, swat them) is neat to explain an idea, but completely irrelevant as an argument. It doesn’t tell you anything about how an AGI will behave, because a man-made AI does not sufficiently mirror the relationship between a naturally evolved human and an equally naturally and independently evolved insect, because both creatures have completely different and independent utility-functions.

    The sole fact that a fly has less computational power than a human and the fact that a human has less computational power than a sophisticated self-improving AGI tells you precisely nothing about their relationship to each other.

    • Ouroboros says: “I think Eliezer once made a good argument that even the sloppy ones will surely fail. You probably can’t be smart enough to build any kind of working self-improving AGI if you’re oblivious to its potential “friendliness” or “unfriendliness”.”

      Could you point me to the “good argument” he made in favor of this point?

      I have heard Eliezer say that before, but I never saw any rational argument in favor of the point, and I find it extremely implausible — actually I find it a bit ridiculous. I even wonder if Eliezer still believes that argument, or if he’s now thought better about it.

      Generally, I find Hugo’s arguments just as rigorous as Eliezer’s (and I don’t agree with either of them). The main difference between the two seems to be, Hugo is open about the fact that his arguments are largely based on intuition, whereas Eliezer uses more of a language and tone of rationality and definitiveness. But this is a difference of style more than substance. Talking a lot about rationality doesn’t make your arguments more rational and rigorous — it just makes certain sorts of readers more willing to overlook the glaring holes in your arguments…. Neither Eli nor Hugo nor anybody else has rigorous, precise, convincing arguments about these slippery future matters.

  22. I keep hearing about this fear about AI taking over and potentially exterminating humanity. My argument is, by the time we create AI with human level intelligence, will we not also be able to increase out own intelligence through genetic modification, so as to match, or even surpass the intellect of AI?

  23. I don’t agree with you. I see your point, it’s a basic assumption but it’s also very wrong. Where to start..

    First and foremost, there’s no such things as friendly-intelligence right now. Every human you see on the street could potentially wipe the entire human race. We make wars for various stupid reasons and we kill each others every day. Look it up. Who does this do you think?

    Why would a Strong AI entity decide to wipe us out? Remember, that they’re more intelligent than you and me, they have a perfect intellect. Would they eat us? This makes no sense at all! As a stupid human, even me understand that wiping other races is not the way to go in the future.

    I might be wrong.. however, when I think about intelligence, I don’t see death and killing, I see science and pleasure.

    A super-intelligent AI could help us find way to post-scarcity. They could help us create an utipia where every human is equal and where everything is free because.. everything would be free to make too.

    Oh.. and your cosmic ray argument? Geez! Please! Prove that cosmic ray influence nanobrains and then, bring that argument back. Does it affect the human brain too? Should we wipe us out just in case?

    • But

      Abundance, and Immortality are allready HERE

      You dream of utopia, the utopia is HERE

      that is the problem : we have everything, guys

      You want more energy : take energy from the sun

      You want more food : build tower of hydroponic food, and GMO algae

      we have no freedom, no way of being the master of our lifes : as the sheeples in the pyramid -> the destiny they draw for us is a massacre

      1929 ? 2008 ? THe same

      Its is not a crisis of PRODUCTION

    • Why would a Strong AI entity decide to wipe us out?

      Because we’re a nuisance in whatever plans ey may have, because we use resources ey want to use for other things, because we consist of such resources, because we can suffer and ey may want to optimize total well-being, etc. etc.

      Discussed a hundred times before; why do we maintain a debate culture that runs in circles?

      I might be wrong.. however, when I think about intelligence, I don’t see death and killing, I see science and pleasure.

      Not mutually exclusive things, not necessarily correlated with the motivations with which intelligence is used. Halo effect: science is good, pleasure is good, death is bad, killing is bad; intelligence will seek the good and not the bad, therefore it won’t kill us.

      The flaw lies in the assumption that all of these things are inherently good or bad and that all intelligences are going to share this view. I believe pleasure is good, and I would kill in order to create more of it. I think suffering is bad, and I would cause death to prevent more of it. I would use intelligence and science as intruments to implement this, and any AI whose ethical goals would be defined by me would do the same.

    • cosmic rays are a FACT! Do U Read anything at all??

  24. The idea that AGIs will always treat us badly because we always treat badly less intelligent being is flawed at its root. Especially if we measure the amount of money that is spent to help those less intelligent beings. Because on our planet as a whole, a lot of people with mental disabilities and a lot of pets are allocated more money than a lot of poor children with no mental disability do. Even though the next Einstein could be among this poor children, if he was given a proper education.

    • I Don’t think einstein will ever give the atomic bomb , one more time TO THE US

      or the equivalent : AGI, NANOROBOT, Or new physics for energy and gravity

  25. Friendly human ? a dangerous delusion ?

    we shouldn’t categorize considering the form, the corps

    a human != human being

    thing are going faster and faster : and the bad guys ideas is the input to the singularity

    What is the difference : if a human from the corporates guys become the god he want to be

    singularity is clearly here

    2030 ? 2020 … Now

  26. I think there needs to be a distinction between general purpose intelligences we may create that we naively believe will help us and special purpose machines designed solely for dealing with the game theory aspects of preventing our extinction via malevolent AI.

    The difference between the two is significant. You don’t need sentience for strategy.

Leave a Reply