Naked Science Forum

General Discussion & Feedback => Just Chat! => Topic started by: alancalverd on 03/07/2021 18:22:24

Title: What do you mean by "random"?
Post by: alancalverd on 03/07/2021 18:22:24
Common problem: I need to traceably randomise 100 patients for a clinical trial. So as they are recruited, I assign each one a random number from 00 to 99, then, say, give the trial drug to those with odd numbers and the placebo to the evens.How do I generate those random numbers?

A. Take two 10-sided dice, one numbered 0 to 9 and the other, 00 to 90. Roll the dice and there's a random two-digit integer for each patient, with a 1 in 100 chance of it being any value from 00 to 99.

B. put 100 counters in a hat, give it a good shake, draw one at a time and discard it: same probability of it being any value from 00 to 99.

What is the probability that I will assign a given number to two patients?  In case A, almost 1. In case B, zero.

In case A I cannot guess what the next number will be, so that's really random. In case B, my guess will become more accurate as the numbers are assigned, and I know exactly what the 100th number will be, so it's not quite as random.

It all depends on what you mean by "random".
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 03/07/2021 18:41:06
Or you could just toss a coin for each patient.
If you are concerned that your coin may be balanced, toss it twice.
If you get HH or TT toss twice more (repeatedly if needs be).
If you get HT then assign them to the placebo group, if you get TH assign them to the treatment group.

There's no  guarantee that the number in each group will be the same, but it will probably be close.
In reality, I suspect everyone uses excel.
Title: Re: What do you mean by "random"?
Post by: hamdani yusuf on 03/07/2021 22:40:02
Or you could just toss a coin for each patient.
If you are concerned that your coin may be balanced, toss it twice.
If you get HH or TT toss twice more (repeatedly if needs be).
If you get HT then assign them to the placebo group, if you get TH assign them to the treatment group.
If the first coin is biased towards head, or the second coin is biased towards tail, then the patient is more likely to get placebo, and vice versa.
Title: Re: What do you mean by "random"?
Post by: alancalverd on 03/07/2021 23:47:32
The specific application isn't the point. It is to demonstrate that of two ways of generating a set of random numbers, one is almost certain to produce duplicates but with no possibility of predicting anything about the remainder of the set, whilst the other cannot produce duplicates and increasingly gives you information about the remainder to the point that you can predict the last number with absolute certainty.

3-way randomisation is sometimes used e.g. to investigate superiority or tolerability between treatments X, Y and placebo. Coin tossing becomes more difficult, but it's easy with a random number table because one third will be divisible by 3, one third by 2 but not 3, and one third not divisible by 2 or 3. But the problem remains as to what you mean by "random".

Research protocols usually refer to an unnamed "computer program that generates random numbers", and as far as I am concerned a set of 2-digit integers generated by Excel RAND is indeed adequately random, but you still need to filter out duplicates.
Title: Re: What do you mean by "random"?
Post by: Petrochemicals on 04/07/2021 00:03:07
That the numbers 1, 2, 3, 4, 5, 6, can win the lottery, every week since its inception just as easily as the string of numbers that has followed for the last 25 years.
Title: Re: What do you mean by "random"?
Post by: Eternal Student on 04/07/2021 20:44:28
Hi.

    Go with the counters in a hat concept.  It's easier.
    It's random at the time when you first started (before pulling any counters).  It doesn't really matter if you know more as the process continues.   Nothing is random after observations have been made and information is available, then it's just a fact not a random variable with an associated probability distribution.  For example the result of a coin toss I made yesterday is not random, it's outcome is an established fact.   The outcome of yesterday's coin toss is never going to be random ever again.
   Just make sure you go through with the selection - don't interfere with it near the end (for example, it would be quite human to give everyone a chance to be in the active group rather than the placebo group and you could do this by tampering with the process to keep at least one active group counter in the bag at all times).
    Randomise the outcomes twice, so that you can't know what will happen.  You can have the hat and the counters but all you're doing is assigning people to group A or group B with no way of knowing which will be the placebo or the active treatment group.  Someone else will decide which group is the placebo by some other method (e.g. tossing a coin) and they should not be informed of who was put into which group.   Keep all records of the random outcomes locked away until it becomes necessary to trace a patient.

     Discussing the true nature of proability and randomness could take a long time and there are chapters on this in most good books about probability.  I'm sure you know this already.
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 04/07/2021 22:43:17
Or you could just toss a coin for each patient.
If you are concerned that your coin may be balanced, toss it twice.
If you get HH or TT toss twice more (repeatedly if needs be).
If you get HT then assign them to the placebo group, if you get TH assign them to the treatment group.
If the first coin is biased towards head, or the second coin is biased towards tail, then the patient is more likely to get placebo, and vice versa.
Did you notice that I said "coin" not "coins"?

It's not me you are arguing with
"John von Neumann invented a simple algorithm to fix simple bias and reduce correlation. It considers two bits at a time (non-overlapping), taking one of three actions: when two successive bits are equal, they are discarded; a sequence of 1,0 becomes a 1; and a sequence of 0,1 becomes a zero. It thus represents a falling edge with a 1, and a rising edge with a 0. This eliminates simple bias, and is easy to implement as a computer program or in digital logic. This technique works no matter how the bits have been generated."

From
https://en.wikipedia.org/wiki/Hardware_random_number_generator#Software_whitening
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 04/07/2021 22:51:53
as far as I am concerned a set of 2-digit integers generated by Excel RAND is indeed adequately random, but you still need to filter out duplicates.
No you don't.
You put the numbers 0 to 99 in a column, Put random numbers into the next column and then sort both columns by the random number
Then you use the top haul of the first column as your "treatment" group and the bottom half as your "placebo" group.

If you want to split them into 3 you can do the same thing (as long as your numbers of patients is divisible by 3).


but it's easy with a random number table because one third will be divisible by 3, one third by 2 but not 3, and one third not divisible by 2 or 3.
More generally, you can split them into n sub groups by finding the remainder when you divide by n
So, for example, if you divide by 4 the remainders are 0, 1, 2 or 3.
Title: Re: What do you mean by "random"?
Post by: hamdani yusuf on 05/07/2021 06:09:43
Or you could just toss a coin for each patient.
If you are concerned that your coin may be balanced, toss it twice.
If you get HH or TT toss twice more (repeatedly if needs be).
If you get HT then assign them to the placebo group, if you get TH assign them to the treatment group.
If the first coin is biased towards head, or the second coin is biased towards tail, then the patient is more likely to get placebo, and vice versa.
Did you notice that I said "coin" not "coins"?

It's not me you are arguing with
"John von Neumann invented a simple algorithm to fix simple bias and reduce correlation. It considers two bits at a time (non-overlapping), taking one of three actions: when two successive bits are equal, they are discarded; a sequence of 1,0 becomes a 1; and a sequence of 0,1 becomes a zero. It thus represents a falling edge with a 1, and a rising edge with a 0. This eliminates simple bias, and is easy to implement as a computer program or in digital logic. This technique works no matter how the bits have been generated."

From
https://en.wikipedia.org/wiki/Hardware_random_number_generator#Software_whitening

I'm sorry for missing the first sentences.
What would happen if the coin has a strong tendency to flip from the last toss? I think it can be done using a simple mechanism.
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 05/07/2021 08:33:10
What would happen if the coin has a strong tendency to flip from the last toss?
Then it would be cheating to use it.
(and it probably wouldn't be a coin).
Title: Re: What do you mean by "random"?
Post by: alancalverd on 05/07/2021 22:16:41
Put random numbers into the next column and then sort both columns by the random number
As Mrs Beeton said, "first catch your rabbit..."

Slightly off the subject (as always) when cooking with my chef son, he once told me to "Just....." and I asked "what exactly do you mean by "just" to which he replied "study and practice for 20 years, then just...." So, to construct 100 random numbers, start with a random number table, then just.....
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 05/07/2021 22:25:33
Put random numbers into the next column and then sort both columns by the random number
As Mrs Beeton said, "first catch your rabbit..."

Slightly off the subject (as always) when cooking with my chef son, he once told me to "Just....." and I asked "what exactly do you mean by "just" to which he replied "study and practice for 20 years, then just...." So, to construct 100 random numbers, start with a random number table, then just.....
no, just put random numbers in tte h2nd column.
Like you said
and as far as I am concerned a set of 2-digit integers generated by Excel RAND is indeed adequately random,
Title: Re: What do you mean by "random"?
Post by: Eternal Student on 05/07/2021 23:27:19
Hi.

There's no interesting discussions going on today, so I'm joining this one.
An old view is to claim that nothing in nature is random.   Probability doesn't exist in nature, it's just an abstract mathematical concept.   There are mathematical objects that are random variables but nature does not have any such variables within it.   

    However, Quantum Mechanics presents some evidence that probability is something inherently found in nature.  So the answer to your problem is simple.   Go down to the Physics lab and get their experimental set up of an electron being fired through some slits and impacting on a screen at the end.   Draw a score board on the screen (like a darts board) and have a few games of quantum darts to generate some random numbers.
Title: Re: What do you mean by "random"?
Post by: hamdani yusuf on 08/07/2021 05:10:40
However, Quantum Mechanics presents some evidence that probability is something inherently found in nature.  So the answer to your problem is simple.   Go down to the Physics lab and get their experimental set up of an electron being fired through some slits and impacting on a screen at the end.   Draw a score board on the screen (like a darts board) and have a few games of quantum darts to generate some random numbers.
Newest experiment shows that quantum jump can be sensed before actually occuring. It can even be canceled. It's not instantaneous.
Title: Re: What do you mean by "random"?
Post by: Europa on 21/07/2021 15:01:50
Common problem: I need to traceably randomise 100 patients for a clinical trial. So as they are recruited, I assign each one a random number from 00 to 99, then, say, give the trial drug to those with odd numbers and the placebo to the evens.How do I generate those random numbers?

A. Take two 10-sided dice, one numbered 0 to 9 and the other, 00 to 90. Roll the dice and there's a random two-digit integer for each patient, with a 1 in 100 chance of it being any value from 00 to 99.

B. put 100 counters in a hat, give it a good shake, draw one at a time and discard it: same probability of it being any value from 00 to 99.

What is the probability that I will assign a given number to two patients?  In case A, almost 1. In case B, zero.

In case A I cannot guess what the next number will be, so that's really random. In case B, my guess will become more accurate as the numbers are assigned, and I know exactly what the 100th number will be, so it's not quite as random.

It all depends on what you mean by "random".
While it would be possible to select 100 individuals randomly there is no way to achieve an accurate representation of the population of a country or of the Earth with just 100 individuals so random can not actually be achieved   
Title: Re: What do you mean by "random"?
Post by: hamdani yusuf on 21/07/2021 23:02:04
While it would be possible to select 100 individuals randomly there is no way to achieve an accurate representation of the population of a country or of the Earth with just 100 individuals so random can not actually be achieved 
It's more like rounding error, representing population with less bit of information, rather than impossibility to get randomness.
Title: Re: What do you mean by "random"?
Post by: Europa on 22/07/2021 02:10:03
While it would be possible to select 100 individuals randomly there is no way to achieve an accurate representation of the population of a country or of the Earth with just 100 individuals so random can not actually be achieved 
It's more like rounding error, representing population with less bit of information, rather than impossibility to get randomness.

Again there are too many human variables to represent the population with just 100 people.  There are 195 countries in the World so this study would leave out entire groups
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 22/07/2021 08:52:49
While it would be possible to select 100 individuals randomly there is no way to achieve an accurate representation of the population of a country or of the Earth with just 100 individuals so random can not actually be achieved 
It's more like rounding error, representing population with less bit of information, rather than impossibility to get randomness.

Again there are too many human variables to represent the population with just 100 people.  There are 195 countries in the World so this study would leave out entire groups
Will whoever is keeping a tally of the things Europa doesn't understand please add this one to the list, thanks.
Title: Re: What do you mean by "random"?
Post by: evan_au on 22/07/2021 11:08:57
A few weeks ago I was doing a Monte-Carlo simulation (yes, that is a thing!), and needed to generate a lot of random sequences; I was using EXCEL to generate them.
See: https://en.wikipedia.org/wiki/Monte_Carlo_method

There were a number of methods on the web that would generate random numbers "with replacement", but I wanted to do it "without replacement".
- If you are running a clinical trial, you can't give two different patients the same bottle of medicine
- And you don't want any bottles left over at the end
- And you don't want to allocate more people to one arm of the trial than the other
- What you want is a random permutation:
       - Randomly assign 100 people to 100 bottles of medicine; once you have assigned a bottle to a person, that bottle is removed from consideration.
       - It also works in the opposite direction: once you have assigned a person to a bottle, that person is removed from consideration.

I found a method on the web that is pretty much what bored chemist suggested, generating a random permutation:
- Generate a row of random numbers (however many you need): EXCEL has a RND() function.
- Lookup the list of random numbers and pick the nth largest, using the "LARGE()" function
- Find the index of that entry using the "MATCH()" function
- It seems rather roundabout, but it works well.
- I wanted to generate a hundred random permutations: Just copy the equations 100 times
- I wanted to run the simulation many times with different inputs: Every time you click "Calculate", it generates a new simulation and graphs the results...
This is the method I used: https://superuser.com/questions/972507/can-i-produce-a-row-in-excel-which-is-random-permutation-of-another-row

But I think Alan only wanted to generate 1 random permutation.

Apparently, EXCEL in Microsoft365 has a new RANDARRAY function that can be used to randomly permute a list by coupling it with the SORTBY function:
https://www.ablebits.com/office-addins-blog/2020/07/15/excel-randarray-function-generate-random-numbers/
Title: Re: What do you mean by "random"?
Post by: alancalverd on 23/07/2021 22:45:06
While it would be possible to select 100 individuals randomly there is no way to achieve an accurate representation of the population of a country or of the Earth with just 100 individuals so random can not actually be achieved   

Try reading the first line of the question.

I have 100 patients, that is, individuals who have been diagnosed with  the same illness. They are by definition not representative of the population, 99.9% of whom are not sick.

I want to see if toadleg accelerates their recovery. So I need to give half of them a dose of toadleg, and the other half a bottle that tastes the same but doesn't contain my magic gloop, and nobody must know or be able to calculate who got which until the end of the trial. 
Title: Re: What do you mean by "random"?
Post by: Eternal Student on 23/07/2021 22:58:39
Hi @alancalverd

I thought several ideas had already been presented.   Are you looking for some improvement, refinement or continued discussion?
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 24/07/2021 00:58:49
A few weeks ago I was doing a Monte-Carlo simulation (yes, that is a thing!), and needed to generate a lot of random sequences; I was using EXCEL to generate them.
See: https://en.wikipedia.org/wiki/Monte_Carlo_method

There were a number of methods on the web that would generate random numbers "with replacement", but I wanted to do it "without replacement".
- If you are running a clinical trial, you can't give two different patients the same bottle of medicine
- And you don't want any bottles left over at the end
- And you don't want to allocate more people to one arm of the trial than the other
- What you want is a random permutation:
       - Randomly assign 100 people to 100 bottles of medicine; once you have assigned a bottle to a person, that bottle is removed from consideration.
       - It also works in the opposite direction: once you have assigned a person to a bottle, that person is removed from consideration.

I found a method on the web that is pretty much what bored chemist suggested, generating a random permutation:
- Generate a row of random numbers (however many you need): EXCEL has a RND() function.
- Lookup the list of random numbers and pick the nth largest, using the "LARGE()" function
- Find the index of that entry using the "MATCH()" function
- It seems rather roundabout, but it works well.
- I wanted to generate a hundred random permutations: Just copy the equations 100 times
- I wanted to run the simulation many times with different inputs: Every time you click "Calculate", it generates a new simulation and graphs the results...
This is the method I used: https://superuser.com/questions/972507/can-i-produce-a-row-in-excel-which-is-random-permutation-of-another-row

But I think Alan only wanted to generate 1 random permutation.

Apparently, EXCEL in Microsoft365 has a new RANDARRAY function that can be used to randomly permute a list by coupling it with the SORTBY function:
https://www.ablebits.com/office-addins-blog/2020/07/15/excel-randarray-function-generate-random-numbers/

Am I missing something?
Number the bottle 1 to 100 in any way you see fit. If it helps, include the "placebos"
Put the patients in a list- for example, by DoB- it won't matter
Put a random number next to each patient (that's why it doesn't matter)

Sort the patients by the random number.
That puts the patients into a random order.
Give bottle1 to the fist patient, bottle 2 to the second and so on.

You will give exactly 100 patients exactly 100 bottles in a way where each patient gets one bottle

Title: Re: What do you mean by "random"?
Post by: Bored chemist on 24/07/2021 01:00:36
I thought several ideas had already been presented.   Are you looking for some improvement, refinement or continued discussion?
I'm looking for a refinement that doesn't include trollish conspiracy theorists.
Title: Re: What do you mean by "random"?
Post by: hamdani yusuf on 24/07/2021 07:30:23
The purpose of randomization is to remove bias other than the difference of treatment, which can be caused by some correlation between how the patients are treated and other parameters that actually affect the results, such as age, gender, habits, genetics, and other preexisting medical conditions. So, in most cases, procedure proposed by BC should be enough.
Title: Re: What do you mean by "random"?
Post by: alancalverd on 24/07/2021 10:27:33
As usual, the discussion has moved way off the original point, which was to note that there are different interpretations of "random" within a closed set, some of which may permit (or even guarantee) duplicates.

Suddenly, DNA does not seem all that improbable!
Title: Re: What do you mean by "random"?
Post by: Bored chemist on 24/07/2021 11:24:37
which was to note that there are different interpretations of "random" within a closed set, some of which may permit (or even guarantee) duplicates.
Noted.
(But a bit dull)
Title: Re: What do you mean by "random"?
Post by: Eternal Student on 24/07/2021 14:06:45
Hi.
As usual, the discussion has moved way off the original point, which was to note that there are different interpretations of "random" within a closed set, some of which may permit (or even guarantee) duplicates.
    OK.  Well then let's briefly discuss the notion of "random" again.

Quote
A random variable X is a measurable function X : Ω → E from a set of possible outcomes Ω to a measurable space E. The technical axiomatic definition requires Ω to be a sample space of a probability triple ( Ω , F , P ).

The probability that X takes on a value in a measurable set S ⊆ E is written as

    P ⁡ ( X ∈ S ) = P ⁡ ( { ω ∈ Ω ∣ X ( ω ) ∈ S } )
[Based on Wikipedia entry for "Random variable"

    So "random" really doesn't mean "equally likely" or imply that there is no bias, or that there would be no pattern discernible in the outcomes. 
    You are very fortunate in that all of your sets are finite, so there won't be any probelms constructing the measure spaces we need.  You could choose to assign every patient to the placebo group.  That is "random" just with an unusual probability distribution.
    Scientists tend to use the term "random" in a slightly different way but that's not their fault.  They don't have a lot of time to study probability or measure theory.
     Until recently I would have been quite certain that probability is not something that exists in Nature but instead it is only an abstract mathematical object.  It's just that we don't have sufficient information and therefore some things appear to be random. 
     Quantum Mechanics may change some of this.  It seems that for some systems we cannot have all the information:  It's not that we haven't tried hard enough to gather the information but just that some quantities (like position and momentum) cannot be known simultaneously no matter how hard you try.

Best wishes to everyone.
Title: Re: What do you mean by "random"?
Post by: alancalverd on 24/07/2021 14:45:42
Quantum mechanics is actually a bit "harder" than not knowing. I think there has been a longstanding mistranslation of Heisenberg's Indeterminacy Principle as "uncertainty", and it would help if mathematicians could tell the public that there is a difference! I am quite certain that the ratio 0/0 is correct, but it is certainly indeterminate. Likewise the position of an electron in a hydrogen atom is indeterminate: it could be anywhere, but we can be quite certain about the probability of it being in certain places.