Singularity: The Post-Human Era

Singularity:
Since the rise of Homo sapiens, human beings have been the smartest minds around. Sometime in the next few decades, we can expect technological advancements to break the upper bound on intelligence that has held for tens of thousands of years. The Singularity presents the human species with some difficult issues, to which almost no one is paying attention because they're too busy watching television.

Calvin: I imagine bugs and girls have a dim perception that Nature played a cruel trick on them, but they lack the intelligence to really comprehend its magnitude.
~ Bill Watterson

Like the sponge cells and the slime mold amoeba, you and I are parts of a vast population whose pooled efforts move some larger creature on its path through life. Like the sponge cells, we cannot live in total separation from the human clump. We are components of a superorganism.
~ Howard Bloom

God is what mind becomes when it has passed beyond the scale of our comprehension.
~ Freeman Dyson

You cannot "serve" God. You don't serve entities. You serve purposes. Asking "What is the meaning of life?" and getting back "God" is like asking "What is two plus two?" and getting back "Spackling paste." It's not even a religious issue. It's a category error, pure and simple. When I ask what two plus two equals, I expect a number. When I ask what the meaning of life is, I expect a goal. That doesn't mean that God can't exist and be a goal in some sense I don't understand at all, because the Universe is a weird place; but it does mean that equating God with a goal will lead you to make a lot of silly mistakes by trying to "serve God" the way you'd serve another human being. If you're religious and you want to be really hubristic, you can say: "Serve God? Of course not, but I serve the same purpose God does."
~ Eliezer S. Yudkowsky

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.
~ I. J. Good

Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended. Any intelligent machine of the sort . . . would not be humankind's "tool", any more than humans are the tools of rabbits or robins or chimpanzees.
~ Vernor Vinge

• Question: Can an AI, starting from a blank-slate goal system, reason to any nonzero goals?
• Answer: Yes.
• Logic: Either life has meaning or it doesn't. >> Gotta be one or the other. >> But we don't know which. >> If we don't know, we should figure it both ways. >> If life is meaningless, nothing makes a difference. Even bemoaning the pointlessness is pointless. >> Since nihilism has absolutely nothing to say, only the "meaning hypothesis" is relevant. >> It doesn't matter whether the probability of the "meaning hypothesis" is 1% or 99%. As long as it's not 0%, the relative value of choices and goals is the same as if the probability were 100%, absolute certainty. >> When it comes to making choices, you can assume that life has meaning and work from there. >> We know a goal exists; let's translate that knowledge into an actual Goal object and try to achieve it. >> Some methods are pretty useful no matter what you're trying to do. For example, "think about how to do it" or "pay someone else to do it" or "try to create a superintelligence which can figure out what G1 (any goal) is and do it". >> All done: We have something specific to do.
• Conclusion: In other words, it isn't necessary to have some nonzero goal when the system starts up. It isn't even necessary to assume that one exists. Just the possibility that a nonzero goal exists, combined with whatever heuristics the system has learned about the world, will be enough to generate actions. We don't have direct access to the real meaning of life. But whatever it is, it's a good guess that the Minds on the other side of Singularity have a better chance of achieving it, so the Singularity is the interim meaning of life. You don't have to know what the meaning of life is in order to work towards it.
~ Eliezer S. Yudkowsky

Trying to speak about the ultimate reality is like sending a kiss through a messenger.
~ Anonymous

Calvin: You know, I don't understand why humans evolved as such thoughtless, shortsighted creatures.
Hobbes: Well, it can't stay that way forever.
Calvin: You think we'll get smarter?
Hobbes: That's one of the two possibilities.
~ Bill Watterson