Here is some examples to demonstrate that moral judgment is closely related to knowledge and uncertainty.
You are in a tall and large building, and find a massive time bomb which makes it impossible to move before disarming it first. You can see red and blue wires on the detonator, and a counting down clock showing that there is only 2 minutes left before it explodes. You are an expert in explosives, sou you know for certain the following premises:
- if you cut the red wire, the bomb will be disarmed.
- If you cut the blue wire, the bomb will explode immediately, destroying the entire building and killing thousands inside.
- If you do nothing about the bomb, the timer will eventually trigger the bomb.
Which is the most moral decision you can take, which is the least moral, and why?
The decision will depend on other information. Is the bomb on the top floor or the ground floor? Are there any other people in the building? If this is an office block and it's empty at night, no one sane would snip either wire when they can just run out of there in the two minutes that are available.
However, let's assume the higher floors of the building are full of people who can't possibly get out in two minutes (or even be warned within two minutes), that the bomb is on the ground floor, and the the building will collapse as soon as it blows. There is nothing immoral about not risking your own death in order to have a 50:50 chance of saving lots of other people, so you are entitled to run out of there and let it blow. If AGI is making the decision though, it could lock you in with the bomb so that you don't have a choice - that would be its moral decision. You would then cut one of the wires, randomly selected.
There are some other factors though. If the person who has to run or cut a wire is more valuable to humanity than the sum total worth of all the other people in the building, AGI will not lock him/her in the room, but will order him/her to get out of there and let the building blow. If the building is full of Nazis who are attending a conference, that could well happen.
The following users thanked this post: hamdani yusuf