Information theory and the symmetry of updating beliefs Rationality is all about handling this thing called information. Fortunately, we live in an era after the rigorous formulation of Information Theory by C.E. Shannon in 1948, a basic understanding of which can actually help you think about your beliefs, in a way similar but complementary to probability theory. Indeed, it has flourished as an area of research exactly because it helps people in many areas of science to describe the world. We should take advantage of this! The information theory of events, which Im about to explain, is about as difficult as high school probability. It is certainly easier than the information theory of multiple random variables (which right now is explained on Wikipedia), even though the equations look very similar. If you already know it, this can be a linkable source of explanations to save you writing time :) So! To get started, what better way to motivate information theory than to answer a question about Bayesianism? The beautiful symmetry of Bayesian updating The factor by which observing A increases the probability of B is the same as the factor by which observing B increases the probability of A. This factor is P(A and B)/(P(A)·P(B)), which Ill denote by pev(A,B) for reasons to come. It can vary from 0 to +infinity, and allows us to write Bayes Theorem succinctly in both directions: P(A|B)=P(A)·pev(A,B), and P(B|A)=P(B)·pev(A,B) What does this symmetry mean, and how should it affect the way we think? A great way to think of pev(A,B) is as a multiplicative measure of mutual evidence, which Ill call mutual probabilistic evidence to be specific. If pev=1 if theyre independent, if pev>1 they make each other more likely, and if pevlesswrong/lw/1y9/information_theory_and_the_symmetry_of_updating/#symmetry
Posted on: Fri, 05 Sep 2014 06:19:29 +0000