[alancalverd]

An existential risk is any risk that has the potential

Aha!

Merriam Webster

existential
[ˌeɡzəˈsten(t)SH(ə)l]
ADJECTIVE
relating to existence.
philosophy
concerned with existence, especially human existence as viewed in the theories of existentialism.
logic
(of a proposition) affirming or implying the existence of a thing.

Silly me, using the logical meaning of the word, which is exactly opposite to the use you quoted!

[alancalverd]

Quote
„Religions exist because people would rather have a wrong answer than no answer at all.“ —  Chuck Palahniuk, book Doomed

Source: https://quotepark.com/quotes/1489111-chuck-palahniuk-religions-exist-because-people-would-rather-have-a/
So, perhaps the best way to remove it is to find the right answer.

But there's the problem. Every religion claims to be the right answer, and they are all different.

The honest, scientific answer is usually "we don't know, but this equation summarises what we have seen and usually predicts what happens next" and nobody is despised for edging closer to the truth.

[hamdani yusuf (OP)]

An existential risk is any risk that has the potential

Aha!

Merriam Webster

existential
[ˌeɡzəˈsten(t)SH(ə)l]
ADJECTIVE
relating to existence.
philosophy
concerned with existence, especially human existence as viewed in the theories of existentialism.
logic
(of a proposition) affirming or implying the existence of a thing.

Silly me, using the logical meaning of the word, which is exactly opposite to the use you quoted!

The meaning is reversed by the word risk. Health risk means there is significant severity and probability of losing one's health.

[hamdani yusuf (OP)]

Quote
„Religions exist because people would rather have a wrong answer than no answer at all.“ —  Chuck Palahniuk, book Doomed

Source: https://quotepark.com/quotes/1489111-chuck-palahniuk-religions-exist-because-people-would-rather-have-a/
So, perhaps the best way to remove it is to find the right answer.

But there's the problem. Every religion claims to be the right answer, and they are all different.

The honest, scientific answer is usually "we don't know, but this equation summarises what we have seen and usually predicts what happens next" and nobody is despised for edging closer to the truth.

If we just say that we don't know and then stop, they will stick to their previous answers, like Palahniuk said. It would be better if we can give our latest best answer, and then update it as new information becomes available, and our answer becomes less and less wrong.

[alancalverd]

Never as clearly demonstrated as when Jonathan Van Tam appeared on TV and talked honestly about COVID, including all the unknowns and unknowables. Massively favorable  public response after a year of being told lies  presented as absolute truth by politicians.

[hamdani yusuf (OP)]

Never as clearly demonstrated as when Jonathan Van Tam appeared on TV and talked honestly about COVID, including all the unknowns and unknowables. Massively favorable  public response after a year of being told lies  presented as absolute truth by politicians.

Or this journalist.

[hamdani yusuf (OP)]

On the subject of AI,  why would anyone build a new machine that dislikes people, can deploy lethal force in its own defence, and can commandeer all the resources it needs to keep functioning? We already have religion and politics.

[hamdani yusuf (OP)]

However, we cannot simply dismiss ideas that are non-rational as a whole. The great David Hume famously realised this in his Enquiry Concerning the Principles of Morals. This quotation is worth showing in full (if only to have an excuse to relish in the man’s writing).

It appears evident that the ultimate ends of human actions can never, in any case, be accounted for by reason, but recommend themselves entirely to the sentiments and affections of mankind, without any dependence on the intellectual faculties. Ask a man why he uses exercise; he will answer, because he desires to keep his health. If you then enquire, why he desires health, he will readily reply, because sickness is painful. If you push your enquiries farther, and desire a reason why he hates pain, it is impossible he can ever give any. This is an ultimate end, and is never referred to any other object.

Perhaps to your second question, why he desires health, he may also reply, that it is necessary for the exercise of his calling. If you ask, why he is anxious on that head, he will answer, because he desires to get money. If you demand Why? It is the instrument of pleasure, says he. And beyond this it is an absurdity to ask for a reason. It is impossible there can be a progress in infinitum; and that one thing can always be a reason why another is desired. Something must be desirable on its own account, and because of its immediate accord or agreement with human sentiment and affection. (from An Enquiry into the Principles of Morals, Appendix 1, V.)

It's unfortunate that Hume stopped at pleasure as the final answer to why question. He could have continued that pain and pleasure  helped our ancestors to survive and thrive, by telling them in advance if their latest actions would likely get them killed, or continue to survive and thrive. He could still chase the why question one more time. The answer would be, only surviving conscious beings can think, and have some control over their own future. In the end, only conscious entities can ask all of those why questions in the first place.

[hamdani yusuf (OP)]

On the subject of AI,  why would anyone build a new machine that dislikes people, can deploy lethal force in its own defence, and can commandeer all the resources it needs to keep functioning? We already have religion and politics.

An introduction to AI Safety, remastered from a talk at "AI and Politics" in London

[hamdani yusuf (OP)]

It's unfortunate that Hume stopped at pleasure as the final answer to why question. He could have continued that pain and pleasure  helped our ancestors to survive and thrive, by telling them in advance if their latest actions would likely get them killed, or continue to survive and thrive. He could still chase the why question one more time. The answer would be, only surviving conscious beings can think, and have some control over their own future. In the end, only conscious entities can ask all of those why questions in the first place.

Pain and pleasure can be felt almost instantaneously. They don't require complex neural network to predict future event accurately. Nonetheless, moving away from painful situations can have significant reduction in risk of death.

More advanced version of reward functions are found in the form of happiness and sorrow. They can be viewed as expectation of pleasure and pain in the future, respectively. Kids can be happy when you give them money. They don't feel the pleasure yet, but they expect to get pleasure in the near future, made available by the money you give them.
This kind of reward functions require complex neural networks to predict future events accurately up to some point.

Some people can only simulate the events to near future, so they act to maximize pleasure and minimize pain in near future. They feel happiness or sorrow when they are expecting to get pleasure and pain in near future. They may become addicted to drugs, alcohol, or junk foods. They avoid exercise and healthy foods. Some may be involved in crime, such as stealing.

Some other people can simulate the events to further future, so they act to maximize pleasure and minimize pain in further future. They feel happiness or sorrow when they are expecting to get pleasure and pain in further future. They avoid drugs, alcohol, or junk foods. They enjoy exercise and healthy foods, and feel happy for doing those activities.

Some other people can simulate the events to even further future, beyond their own lifetime. So they act to maximize pleasure and minimize pain of their next generations. They feel happiness or sorrow when they are expecting to give pleasure and pain to their successors in further future. They try to build a better environment for their successors to live and thrive.

[hamdani yusuf (OP)]

Plato realised that immoral actions are usually caused by lack of thoughtfulness. But we know that not all mistakes are classified as immoral. They are usually caused by shortsightedness. They sacrifice long term goals just to try to achieve shorter term goals.

[hamdani yusuf (OP)]

There's another cause that makes us think that an action is immoral. It's when the action is based on deeply flawed world view.
This means that the action is based on false assumptions which are strongly held, or it's among the highest priorities in someone's beliefs. So if one day a new information contradicting it becomes available, he/she tends to ignore that and keeps the old belief instead.

An example may make it clear. Some people think that their terminal goal is to live forever in heaven after they die. They strongly believe that heaven really exist, and they believe that some actions will make them live again there forever after they die. Some of those actions are seen to be immoral by those who don't share the same beliefs or assumptions.

[hamdani yusuf (OP)]

Plato realised that immoral actions are usually caused by lack of thoughtfulness. But we know that not all mistakes are classified as immoral. They are usually caused by shortsightedness. They sacrifice long term goals just to try to achieve shorter term goals.

Mistakes caused by false beliefs in shallow/peripheral level can be dismissed from immoral actions. For example, a doctor misdiagnoses a patient due to a new/rare/unknown disease, and gives a wrong prescription which harms the patient. But intentional negligence or ignorance can be classified as immoral, although the boundary might be fuzzy.

[alancalverd]

All the more reason to adopt a simple test of morality with no invocation of belief. Would you behead yourself or the person you love most? If yes to the first test, then why haven't you done it?

[hamdani yusuf (OP)]

All the more reason to adopt a simple test of morality with no invocation of belief. Would you behead yourself or the person you love most? If yes to the first test, then why haven't you done it?

I've already addressed your tests.

"Kill infidels" thus becomes a moral imperative. But it fails my tests: you wouldn't want to kill yourself (if you think the order is legitimate, you aren't an infidel) and your nearest and dearest (by definition of near and dear).   

Your moral tests, are two steps algorithm to determine if an action is considered moral or immoral. It depends on following assumptions:

• Everyone is pursuing self preservation
• Everyone loves someone
• Everyone is rational

It turns out that those assumptions are not always true.

Sometimes someone stops pursuing self preservation. But commonly used moral standards don't grant them right to harm others for that. So you add the second test, based on second assumption. In your opinion, moral actions must pass both tests. This additional test reduces the case where an action can pass your tests while failing commonly used moral standards. 

But the second assumption is not always true either. Although extremely rare, it's still possible that someone loves noone. If it happens that they don't love themselves either, they can do immoral things without failing your tests.

Perhaps the most cases of immoral actions are done when third assumption turns out to be false. In this case, they can do immoral actions without failing your tests.

Let me know if you have something new to say.

[alancalverd]

If he has no loved ones he can't answer "yes" to Q2.

The problem with any argument around "false beliefs" is the impossibility of falsifying some beliefs, such as an eternity in paradise for the killer of infidels. 

Intentional ignorance is a strange concept, and I'm uncomfortable with the idea that the morality of an action has anything to do with the knowledge or intention of the person doing it. That leads dangerously close to the evangelical concept of damnation by education. The principle there is that our merciful god doesn't condemn or punish the ignorant, but as soon as I have preached the gospel to you, you are damned if you disobey the rules I have just set out.

On the other hand the legal principle is that ignorance of the law does not excuse you from obeying it.  And yet we have the concept of "reading the riot act" that requires those in authority to proclaim that authority before taking punitive action.

[hamdani yusuf (OP)]

If he has no loved ones he can't answer "yes" to Q2.

Does it make everything he does immoral?

[hamdani yusuf (OP)]

The problem with any argument around "false beliefs" is the impossibility of falsifying some beliefs, such as an eternity in paradise for the killer of infidels. 

Although perfect knowledge is impossible, we can still use practical knowledge based on probability and statistics.
The key point here is determining how much likely they will repeat their actions which produce bad consequences to the society.

[hamdani yusuf (OP)]

Intentional ignorance is a strange concept, and I'm uncomfortable with the idea that the morality of an action has anything to do with the knowledge or intention of the person doing it. That leads dangerously close to the evangelical concept of damnation by education. The principle there is that our merciful god doesn't condemn or punish the ignorant, but as soon as I have preached the gospel to you, you are damned if you disobey the rules I have just set out.

1. A kid treats his new friend with a peanut cookies  not  knowing that she is allergic to peanut.
2. A kid treats his new friend with a peanut cookies after knowing that she is allergic to peanut.

What matters in determining morality of an action is its intended consequences.

[hamdani yusuf (OP)]

On the other hand the legal principle is that ignorance of the law does not excuse you from obeying it.  And yet we have the concept of "reading the riot act" that requires those in authority to proclaim that authority before taking punitive action.

There are differences between legal and moral principles, although there are some similarities too.
In some countries, making a joke on their supreme leaders are illegal.