Probability/Properties of Distributions

Introduction Recall that pdf (or cdf) describes the random behaviour of a random variable completely . However, we may sometimes find the pdf (or cdf) to be too complicated, and only want to know some partial information about the random variable. In view of this, we study some properties of distributions in this chapter, which provide partial descriptions of the random behaviour of the random variable. Some examples of such partial descriptions include location (e.g. pdf is 'located' at left or right?), dispersion (e.g. 'sharp' of 'flat' pdf?), skewness (e.g. pdf is symmetric, skewed to left, or skewed to right?), and tail property (e.g. pdf have 'light' or 'heavy' tails?). We can qualitatively describe them, but such descriptions are quite subjective and inaccurate. To give a more objective and accurate measure to such descriptions, we evaluate them quantitatively using some quantitative measures derived from the pdf (or cdf) of the random variable. We will discuss some of such quantitative measures in this chapter. Among these, the expectation is the most important one, since many of other properties base upon the concept of expectation. Expectation We have different alternative names for expectation, e.g. expected value and mean. (Expectation) The expectation of a random variable is (i) (if is discrete) in which is pmf of ; (ii) (if is continuous) in which is pdf of ; (iii) (if is mixed) If in which is pmf of and is pdf of . The expectation of is what we would expect of its value if we are to take an observation of . It is a weighted average of all possible values attainable by (i.e. ), with heavier weighting to points with higher value of pmf or pdf. Expectation tells us the 'centre' of distribution of , and 'average' position of when generated in the long run. Actually the '' is not needed, since the pmf or pdf will equal zero for input outside its support. Let be the number facing up after throwing a fair six-faced dice once. Then, the expectation of is If the dice is loaded, and thus the probability that number '6' faces up becomes 0.5, and there are equal probability for the other five numbers facing up, the expectation of becomes (Expectation of uniform distribution) Let , the uniform distribution with parameters and . Then, the pdf of is and the expectation of is In a process, we first toss an unfair coin one time, with probability for the head to come up. If head comes up in the first toss, we toss another unfair coin one time, with probability for the head to come up. If tails comes up in the first toss instead, We throw an arrow to the ground one time. Let be the number of head coming up in all tosses, be the angle from the north direction to the direction pointed by the arrow, measured clockwise and in radian, and be the number we get from the process finally. Suppose that . {Choose correct expression(s) of . |type="[]"} - || What is ? - || What is ? + || - || - || in general. } {Choose correct expression(s) of . |type="[]"} + || . || Remark: recall from the chapter about random variable that the interval of the support can be closed, open or half-open, without affecting the result. - || The angle we get is not affected by . - || The angle we get is not affected by . - || The angle we get is not affected by . - || The angle we get is not affected by . {Choose correct expression(s) of . |type="[]"} + || - - - {If the two coins are fair, choose correct statement(s). |type="[]"} - increases. - decreases. + Change in depends on values of and . || Since + remains unchanged. - increases if . || If , when the coins are still unfair, || . || When the coins become fair, || . || So, actually decreases. In the following, we introduce a useful result that gives the relationship between expectation and probability, we can use expectation to ease the computation of probability using this result. (Fundamental bridge between probability and expectation) For each event , Let . Since (which is a discrete random variable), When there are multiple random variables involved, we may derive the joint pmf or pdf first to compute the expectation, but it can be quite difficult and complicated to do so. Practically, we use the following theorem more often. (Law of the unconscious statistician (LOTUS)) Let be random variables. Define for a function . Then, (i) (if is discrete) in which is joint pmf of ; (ii) (if is continuous) in which is joint pdf of . If is mixed, we can apply the definition of expectation and use the above two results for the expectations of the discrete and continuous random variables. This theorem is known as the law of the unconscious statistician, because we often tend to use this identity without realizing that it is a result of a theorem, instead of a definition. This theorem also holds when there is only one random variables involved (the joint pmf and pdf become normal pmf and pdf) , e.g. The proof is quite complicated, and hence we skip it. In the following, we will introduce several properties of expectation that can help us to simplify computations of the expectation. (Properties of expectation) For each constant and random variable , (Linearity) ; (Nonnegativity) if , ; (Monotonicity) if , ; (Triangle inequality) ; (Multiplicativity under independence) if are independent, . Linearity: for continuous random variables , Similarly, for discrete random variables , Nonnegativity: For continuous random variable , Similarly, for discrete random variable , Monotonicity: For random variables that are either both discrete or both continuous, Triangle inequality: Multiplicativity under independence: For continuous random variables , Similarly, for discrete random variables , (Nonmultiplicativity) in general. We cannot apply linearity property similarly when the function inside the expectation is nonlinear. E.g., in general. From linearity, we can see that the expectation of a constant is simply the constant itself. This is intuitive, since the value we expect for a constant is simply the constant. The converse of multiplicativity under independence is true in general, but not always true. For some special dependent random variables, the converse does not hold. Mean of some distributions of a discrete random variable (Mean of Bernoulli and binomial r.v.'s) Let and . Then, , and . . Since , in which are i.i.d. and follow Each of the Bernoulli r.v.'s acts as an indicator for the success of the corresponding trial. Since, there are independent Bernoulli trials, there are such indicators., . (Mean of Poisson r.v.'s) Let . Then, (Mean of geometric and negative binomial r.v.'s) Let and . Then, , and . Since it follows that . Since in which are i.i.d., and follow Each geometric r.v. shows the number of failure for the corresponding success., (Mean of hypergeometric r.v.'s) Let . Then, . Since in which (each of the Bernoulli r.v.'s indicates whether the corresponding draw of ball is of type 1, with probability without knowing the results of other draws since this probability is unconditional, because the corresponding mean is also unconditional, so that their sum is also unconditional mean (as in the proposition), since each draw is equally likely to be any of the balls) are dependent, but we can still use the linearity of expectation, since it does not require independence. , it follows that Mean of some distributions of a continuous random variable We will introduce the formulas for mean of some distributions of a continuous random variable, which are relatively simpler. (Mean of uniform r.v.'s) Let (). Then, . (Mean of gamma, exponential, and chi-squared r.v.'s) Let , , and . Then, , , and . It suffices to prove the formula for mean of gamma r.v.'s, since exponential and chi-squared r.v.'s are essentially special cases of gamma r.v.'s, and thus we can simply substitute some values into the formula for mean of gamma r.v.'s to obtain the formulas for them. Since , by substituting . Since , by substituting and . (Mean of beta r.v.'s) Let . Then, . We use similar approach from the previous proof. (Undefined mean of Cauchy r.v.'s) Let . Then, is undefined. (Mean of normal r.v.'s) Let . Then, . Let . It follows that . Examples (St. Petersburg Paradox) Consider a game in which the player toss a fair coin times, until a head comes up. Since , the expected value of is That is, the player requires two tosses to get a head coming up on average. The game rewards the player to play the game, but the player must pay back after getting a head coming up. Some may think that the expected net gain of the player is so the player has an advantage in this game. However, this is wrong since the correct expected net gain is instead i.e., on average the player has infinite loss! {Choose correct statement(s). |type="[]"} - for each random variable . || This is true for each nonnegative random variable, and may not be nonnegative. || Actually, if is negative, then its expected value is nonpositive, by considering and linearity. + for each random variable . || Since , this holds by nonnegativity. + for each random variable . || This follows from the nonnegativity of absolute value function. - if random variables and are pairwise independent. || The pairwise independence is not sufficient for this to hold. We need mutual independence of and . {Given that , Choose correct expression(s) for . |type="[]"} + || We can obtain this by linearity. - || This expression is undefined if . However, if , may not be undefined. + || We can obtain this by linearity. - || This is wrong since for each random variable in general. } Let us illustrate the usefulness of fundamental bridge between probability and expectation by giving a proof to inclusion-exclusion using this bridge. (Proof of inclusion-exclusion formula) Recall that the inclusion-exclusion formula is For each event and , The proof is as follows: Probability generating functions An application of expectation is probability generating functions. As suggested by its name, it can generate probabilities in some sense. (Probability generating function) Let be a discrete r.v. with support . The probability generating functions of is There is also moment generating function, which can generate moments (see next section for the definition) in some sense. We will discuss in the transformation of random variables chapter. By taking derivatives of the probability generating function, we can generate probabilities: This can be seen directly by evaluating the derivatives. Variance (and standard deviation) Indeed, variance is a special case of central moment, and is related to moment in some sense. (th moment) The th moment of a random variable is . (th central moment) The th central moment of a random variable is . (Variance) The variance of a random variable , denoted by , is its 2nd central moment, i.e. . Since is the squared deviation of the value of from its mean, in view of the definition of variance, we can see that variance measure the dispersion (or spread) of distribution, since it is what we would expect of the squared deviation if we are to take an observation of the random variable. Another term which is closed related is standard deviation. (Standard deviation) The standard deviation of random variable , usually denoted as , is . the interpretation of standard deviation is similar to that of variance standard deviation is also sometimes abbreviated as 's.d.' standard deviation of random variable has the same unit as , which is one of its advantage, and one of the reasons to use standard deviation instead of variance to measure dispersion. since standard deviation is usually denoted as , it follows that we can denote variance as , although it is not as common as the notation. (Properties of variance) (alternative expression for variance) (invariance under change in location parameter) for each constant (homogeneity of degree two) for each constant (nonnegativity) (zero variance implies non-randomness) (additivity under independence) alternative expression for variance: Let for clearer expression. and the result follows. invariance under change in location parameter: nonnegativity: it follows from . zero variance implies non-randomness: Let for clearer expression. Consider the event , in which is a positive integer. Since we have . Thus, additivity under independence: For each random variable and that are independent with means respectively, Thus, inductively, if are independent. Variance of some distributions of a discrete random variable (Variance of Bernoulli and binomial r.v.'s) Let and . Then, and . since is nonnegative. It follows that . Similar to the proof for the mean of Bernoulli and binomial r.v.'s, in which are i.i.d. and follow . Because of the independence (from i.i.d. property), (Variance of Poisson r.v.'s) Let . Then, . Hence, (Variance of geometric and negative binomial r.v.'s) Let and . Then, , and . Since it follows that . Hence, . Similarly, in which are i.i.d., and follow Each geometric r.v. shows the number of failure for the corresponding success.. Because of the independence, Variance of some distributions of a continuous random variable (Variance of uniform r.v.'s) Let . () Then, . (Variance of gamma, exponential and chi-squared r.v.'s) Let , , and . Then, , , and . Similarly, it suffices to prove the formula for variance of gamma r.v.'s. It follows that Since , by substituting . Since , by substituting and . (Variance of beta r.v.'s) Let . Then, . It follows that (Undefined variance of Cauchy r.v.'s) Let . Then, is undefined. It follows from the proposition about undefined mean of Cauchy r.v.'s and the formula (arbitrary term minus undefined term is undefined). (Variance of normal r.v.'s) Let . Then, . Let . It follows that . Hence, . {Choose correct statement(s). |type="[]"} - for each constant . || If is random variable and , , but is not non-random constant. + for each random variable , and for each constant . - || Counter example: if , || , but is random variable. - if || This is wrong by nonnegativity of variance. + Standard deviation of random variable , || Since is random variable, . Otherwise, is a non-random constant, which contradicts to the assumption that is random variable. || Since , || and equality holds if and only if , which is impossible. Coefficient of variation (Coefficient of variation) The coefficient of variation is the ratio of the standard deviation to the mean, i.e. . It is also known as relative standard deviation, since it measures the dispersion relatively (relative to the mean). Thus, it tells the dispersion more accurately than standard deviation without mean. Also, coefficient of variation has no unit. So, it is useful to compare dispersion between different data sets. It shows the extent of dispersion in relation to the mean. However, if the mean is zero, then the coefficient of variation is undefined. So, this is a limitation for it. If and , then for each , the coefficient of variation of is while the coefficient of variation of is 1/5, which equals that of if , equals negative of that of if (they are the same in magnitude, i.e. absolute value). This is expected, since the scaling of random variable itself should not affect the extent of its dispersion. {Assume is increased to 20. Calculate such that the coefficient of variation of remains unchanged. |type="()"} - 1 - 2 || We want the coefficient of variation remain unchanged, not standard deviation. + 4 || Let be the new standard deviation. || - 5 - 8 {Calculate such that the standard deviation of remains unchanged. |type="()"} - 1 + 2 || is the standard deviation of . || So, to make it unchanged, we just set its value to be the same as before. - 4 - 5 - 8 In general, when the mean is negative, then the coefficient of variation will be nonpositive, since standard deviation is always nonnegative. Quantile Then, we will discuss quantile. In particular, median and interqaurtile range are quite related to quantiles. (Quantile) Quantile of order (th quantile) of random variable is Definition of quantile is not unique. There are several alternative definitions, namely If is strictly increasing, all alternative definitions become equivalent and equal the inverse of cdf at , and thus we can calculate the th quantile by solving the equation . Practical applications focus only on . The following are some terminologies related to quantiles. (Percentile) The th percentile is th quantile. 70th percentile is 0.7th quantile. (Median) The median is 0.5th quantile. (Quartile) The th quartile is th quantile in which . 2nd quartile is 0.5th quantile, which is also median. (Interquartile range) The interquartile range is 3rd quartile minus 1st quartile. Median and interquartile range measure centrality and dispersion respectively. Recall that mean and variance measure the same things respectively. One advantage of median and interquartile range is robust, since they are always defined, while mean and variance can be infinite, and they fail to measure centrality and dispersion in those occasions. However, median and interquartile range also have some disadvantages, e.g. they may be more difficult to be computed, and may not be very accurate. (Quantile of uniform distribution) The th quantile of uniform distribution with parameters and is since and we can see that if . Then, median of uniform distribution is which is the same as its mean, and the interquartile range of uniform distribution is which is different from its variance, namely . {Choose correct statement(s). |type="[]"} - 20th quantile is 0.2th percentile || Instead, 0.2th quantile is 20th percentile. - 4th quartile is 1st quantile || 4th quartile is undefined. - 2nd quantile is undefined. || It is not undefined, it is just not on the focus of practical applications. || Indeed, th quantile is simply one if . - 0th quantile = 0th percentile = 0th quartile. || 0th quartile is undefined. + Interquartile range must be nonnegative. || This is true, since cdf must be nonnegative, so 3rd quartile must have not less than 1st quartile. - Median must be nonnegative. || It may not be true, e.g. for uniform distribution with parameters , the median is . Mode Mode is another measure of centrality. (Mode) The mode of a pmf (pdf) is the value of at which the pmf (pdf) takes its maximum value (has its local maximum). The mode is the value that is most likely to be sampled (for pmf). Mode is less frequently used than mean. The modes of the pmf of the numbers coming up from throwing a fair six-faced dice are 1,2,3,4,5 and 6, since the probability for each of these numbers coming up is 1/6, so the pmf takes its maximum value (1/6) at each of these numbers. {Suppose the dice is loaded such that the probability for the number six coming up is 1/2, while other numbers are still equally likely to come up. Which of the following is (are) the mode(s) of the pmf now? |type="[]"} - 1 - 2 - 3 - 4 - 5 + 6 || The probability for six coming up is 1/2, which is higher than that for each of the other numbers coming up (1/10). From this example, we can see that the mode is not necessarily unique. Covariance and correlation coefficients In this section, we will discuss two important properties of joint distributions, namely covariance and correlation coefficients. As we will see, covariance is related to variance in some sense, and correlation coefficient is closed related to correlation. (Covariance) For each random variable , the covariance of is (Correlation coefficient) For each random variable such that , the correlation coefficient is Both covariance and correlation coefficient measure linear relationship between and . As we will see, , are more highly correlated as increases, and has a linear relationship with if . (Properties of covariance) (i) (symmetry) for each random variable , (ii) for each random variable , (iii) (alternative formula of covariance) (iv) for each constant , and for each random variables , (v) for each random variable , (i) (ii) (iii) (iv) (v) Then, we will discuss about correlation coefficients. The following is the definition of correlation between correlation between two random variables. (Correlation between two random variables) Random variables are uncorrelated if , and are correlated if , and if and . This explains why we use covariance instead of correlation coefficient. It is because covariance is always defined, but correlation coefficient may be undefined. Covariance and correlation coefficient are similar, but they have differences. In particular, depends on variances of and , not just their relationship. Thus, this number is affected by the variances, and does not measure their relationship accurately. On the other hand, adjusts for variances of and , and therefore measures their relationships more accurately. The following is one of the most important properties of correlation coefficient. (Universal measure by correlation coefficient) Correlation coefficient lies between -1 and 1 (inclusively). For each random variable , Aim: prove that . To get rid of the square root to make the proof neater, we square both side of the inequality, and get . Recall that . So, one way to prove the rightmost inequality is expressing its left side as , as follows: Thus, the result follows. For each random variable , the higher the , the higher the correlation between because of this, we can compare the correlation of different pairs of random variables if , increases linearly with if , decreases linearly with Then, we will define several terminologies related to correlation coefficient. (Positively correlated, negatively correlated, and uncorrelated) Random variables are positively (negatively) correlated if , i.e. tends to as . They are uncorrelated if . Then, we will state an important result that is related to independence and correlation. Intuitively, you may think that 'independent' is the same as 'uncorrelated'. However, this is wrong. Indeed, 'independent' is stronger than 'uncorrelated'. (Relationship between independence and correlation) If two random variables are independent, they are uncorrelated. For each independent random variable with mean respectively, However, converse is not true, as we will see in the following example. Let such that they are independent. Set . Since , , and , their joint pmf is The covariance and so are uncorrelated. On the other hand, and so are not independent. This illustrates that 'uncorrelated' does not imply 'independent'. {Choose correct statement(s). |type="[]"} + Two random variables are uncorrelated if at least one of them is non-random constant. || If at least one of is non-random constant, e.g. is non-random constant, || || Also, if both of them are non-random constants, say and , || || The result follows. - For each random variable, it is uncorrelated with itself. || This is wrong, since which is nonzero in general. || That is, the random variable is correlated with itself in general. + For each random variable, it increases linearly with itself. || For each random variable , since . - Random variables are more highly correlated than random variables if . || This is true if we replace with . || Without the absolute value sign, this is wrong since, e.g., and , but are more highly correlated than .