Probability for humans

Overview
NOTE: Paul listed this as three different units, but it seems like it is many facets of the same thing, so I put it on a single page. ==Qualitative Bayes == ==Quantitative Bayes/Probability for Humans ==
 * Summary:
 * Q: How do we treat arguments and observations as “rational evidence” in practice?
 * Ideas:
 * Conservation of expected evidence
 * Regression to the mean
 * Base rate neglect
 * Implicit independence assumptions
 * Filtered evidence
 * Exercises:
 * Base rate neglect exercise (split audience, consistency violation)
 * Interpreting informal arguments
 * Poke holes in bad Bayesian “arguments”
 * Logistics:
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Time: ~ 1 hour
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Teacher: ?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Pairs with: Basic Bayes
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Summary:
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Q: How do we usefully learn from unreliable evidence?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Ideas:
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Calibration
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Precommit to predictions
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Bayes as a consistency criterion
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Agreement
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Exercises:
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Calibration game
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Agreement game
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Ballparking probabilities of (real) interest
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Using limited data to answer tough trivia questions
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">I’m about to tell you...
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Logistics:
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Time: ~ 3 hours
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Teacher: ?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Pairs with: Basic Bayes, Agreement

<span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Meta-Bayes

 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Summary:


 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Q: How do we incorporate our beliefs and intuitions into formal reasoning?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Ideas:


 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Ask: what process generated this belief?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Reference class forecasting
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Leaving a line of retreat
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Use explicit reasoning where you can
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Exercises:


 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Ask about sources of our beliefs, apply techniques for patching
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Logistics:


 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Time: 1 hour
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Teacher: Paul?
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Outline: available
 * <span style="font-size: 15px; font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Pairs with: Basic Bayes

Outline for Meta-Bayes ("Why Do I Believe What I Believe")
<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">---
 * 1) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">We want to figure out when we're wrong.
 * 2) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Think about the processes producing beliefs, rather than beliefs themselves (beliefs always feel right).
 * 3) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: Election parable.
 * 4) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: look for the causes of beliefs.
 * 5) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Write things down. Force yourself to spend a few minutes thinking about it.
 * 6) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Imagine a conversation with someone who's come to different beliefs; why do you disagree? (Better yet, have one!)
 * 7) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: I think a PhD is likely to be useful
 * 8) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: leave a line of retreat
 * 9) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Spend time thinking about what you'd do if you were wrong (maybe you'd do the same thing, so this question doesn’t matter, though be wary of one argument against an army)
 * 10) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">If it's important, <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">really <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> think about it e.g. go interview at startups and hang out with startup folks, and if you don't like it look for another alternative (like you'd actually do, if you were wrong)
 * 11) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Some sort of example of polarization? Gun control, perhaps.
 * 12) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: locking my doors
 * 13) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: Once you know the evidence your brain is using, reason about it explicitly
 * 14) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Don't privilege your experiences over the experiences of others (unless it's a question about you)
 * 15) **<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Go do research that screens off your experience
 * 16) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">If that's not applicable, think about how much the evidence you have really matters
 * 17) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: importance of basic science
 * 18) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: talk to people with different views, especially if you are in a biased environment.
 * 19) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: I think it's fine for me to eat meat (parallel example: I think higher taxes on the rich inhibit economic growth)
 * 20) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: make motivated cognition harder by breaking things down into subproblems
 * 21) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">List costs and benefits / evidence for and against
 * 22) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Get curious about each subproblem (evaluating the strength of each piece of evidence, the status of each related question, or the seriousness of the cost or benefit), even if you are motivated to be sure that your overall view won't change
 * 23) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Example: beliefs about diet
 * 24) <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Moral: Use the outside view
 * 25) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Of people with beliefs like yours, how often are they right?
 * 26) *<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">In similar situations, how often have <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">you <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> been right? (especially for disagreements)

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">People are wrong a lot. When two people disagree about a question of fact, at least one of them is probably wrong. When you are making an important decision, odds are you are wrong about something relevant. So the question is: how do we notice when we are wrong?

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">One approach is just to look at a belief and say: is this belief wrong? Turns out: this doesn't work. Beliefs always feel right. That's what it means to believe something.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Better is to pick up one of your beliefs and ask: why do I believe this? Why do others believe differently? Would I believe differently, if the world were different?

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">There are some common techniques to doing this, and some tricks for becoming less wrong. I'm going to present some of the hacks I've found helpful.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">I'd like to start with a parable. [Parable about election. ~ 5 min.]

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">So maybe this is silly. You wouldn't trust Bob's campaign manager about the election, right? Well the first thing is, when you ask your brain why you believe something, it's not going to be so nice and lay out things like this. It's just going to give you the vague feeling that Bob is the better candidate. So my first recommendation is: <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">pay attention <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> to the causes of your beliefs. If it's an important thing, literally <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">get out a sheet of paper <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> and force yourself to spend 5 minutes documenting where your belief comes from. This seems to work well for many people.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Similarly, if you are in an argument with someone, consider exchanging <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">causes of beliefs <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> rather than evidence, sharing these stories, and see why you've come to believe different things. This one is less well-documented, it only works well between people who have much trust and respect for each other, and it seems to be practiced very rarely by humans. I think of it more as an ideal to live up to.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Hopefully I've made my point: even if it's completely <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">obvious <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> that a belief comes from somewhere silly, you need to take the one extra step of <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">noticing <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> where your belief comes from, before you can treat it with appropriate skepticism.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">In general, once we've noticed where a belief comes from, we should ask: does the cognitive process that produced this belief tend to produce true beliefs? How could I change the process, so that it would produce true beliefs? Asking this question about processes rather than particular beliefs is easier.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">---

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Here's the first real example I want to present. I'm deciding whether to go to graduate school. Turns out, it's a complicated issue; lots of benefits of going to graduate school, but there are lots of benefits to alternative paths as well. I feel like I should go to graduate school. Why? Well, there are lots of really smart people in graduate school, and meeting and interacting with smart people is a great thing to do, and graduate school is a good launching pad for nearly anything else I might want to do, and so on.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">OK, but that's not what's really going on. What's really going on is that there are a bunch of arguments on both sides, and the balance isn't clear, and when I'm asking my brain it's providing some good arguments for going to graduate school, but what I should be asking is <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">why <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> is my brain providing those arguments. Is it because they are clearly the most compelling arguments? No--what's going on is that going to graduate school looks like it is going to be a great time, and not going to graduate school seems like it is going to make my life uncertain and stressful. So as long as there are <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">any <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> good reasons to go to graduate school, my brain is going to offer them up whether or not they are the <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">most important <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> considerations.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Cool, so I've implemented the first step of noticing where the belief came from, but now what? At the end of the day I need to choose whether to go to graduate school. I know three important hacks here:

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">1. Leave a line of retreat. The best thing by far to do, is to remove your brain's incentive to distort the truth. To do that, I (and others) have found it really helpful to visualize concretely what I would do if I <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">shouldn't <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> go to graduate school, to visualize the worlds in which those actions really were better than going to graduate school; to get myself excited about the possible world where I'm wrong, and to remove the roadblocks to recognizing it. If the first steps you can take are cheap, like interviewing places or doing research, go out and do them, and meet the people you might work with, and see how awesome the alternatives are. Once being wrong doesn't seem so terrible, at least then your brain isn't <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">guaranteed <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> to give you the wrong answer.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">2. Use reference class forecasting. Put your current belief in an appropriate reference class ask: how often are beliefs in that reference class correct? Amongst people in my field who've felt like they should go to graduate school, how many of them do I think were making an error? When I disagree with my friends, how often am I actually right? Reference class forecasting shouldn't have the final word for important decisions, but it should highlight cases where you are <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">probably <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> making an error, suggesting you should put more time into figuring out the truth. We are much better at seeing errors in others, and easily motivated not to see them in ourselves; trying to think about population statistics can help us correct this problem in ourselves.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">3. Get advice, try and reason more exhaustively with help from other people (listing all costs and benefits and actually weighing them, asking others to help you find places you are missing something or being unreasonable), build models where you can get impartial estimates of the input parameters. Some reasoning is more reliable than other reasoning, particularly when you can get help from other people. [this should get fleshed out, maybe all of these should have examples].

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">---

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Another example: I hate locking doors. When I see people locking their car doors I sort of scoff at them. People don't really steal stuff from cars, and people who lock their doors and try and hide nice items in their car and so on are just being really overly cautious. Why do I believe this? Well, I know nothing about the real rate of thefts from cars. I've never looked up statistics. What I really know is that I've never had something stolen from a car, and I've never heard of it happening to anyone I know. My brain is pretty happy generalizing from that to "it doesn't happen." (There is also motivated cognition here, where I like feeling like a contrarian and seeming carefree, and I like thinking of most people as being too cautious and silly, and so on) At any rate, this <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">is <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> legitimate evidence, and my brain is right to infer something about the rate of such thefts. But the thing about your brain is: it doesn't really know <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">how much <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> to generalize. Maybe I've parked my car in sort of shady locations 100 times, and I get a multiplier of 10x because I would have heard about it if a friend had gotten something stolen some time I was talking with them. So that's a thousand data points--how unlikely does that make my car to get stolen? [Math!]

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Yeah, so if locking my car costs me tens of cents in inconvenience, and getting my laptop stolen costs around $1000, then this really isn't a low enough probability to justify not locking my car.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">The lesson here is, when you understand <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">what <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> your brain is generalizing from, you can often evaluate the significance of that evidence better yourself. Your brain tends to weight it's own experiences too highly, because they are particularly salient. It turns out that here there is an even better solution, which is just for me to go look up how often stuff gets stolen out of cars. About 1 minute of googling reveals that roughly 1% of residents of Texas have something stolen out of their car each year, which implies that the odds of theft aren't much lower than is suggested by an (accurate) analysis of my own observations, and that I really should be locking my doors, not leaving valuable items in view, etc.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">---

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Another example: I think basic science is more valuable than engineering. Again, lots of complex arguments on both sides. So: why do I believe what I believe? After all, lots of people disagree with me. I talk with my friend, who thinks that engineering does more on net for tech progress, and we try and pinpoint the source of our disagreement. We see basically the same important considerations, but we disagree about which direction they point. Upon inspection, my beliefs come to seem from what I perceive as the consensus of honest and smart observers. Most smart people I've talked to seem to think that basic science is more important than applications, which have led my brain to generalize that smart people think basic science is more important, which is in turn pretty compelling evidence.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">So why is it that most smart people I've talked to have liked basic science more? One natural explanation is that most smart people like basic science more. But in this case there is a much cleaner explanation: I've spent most of my time in academia, which has specifically filtered for the people who are willing to spend their lives doing basic science. So regardless of what most smart people think, I'd expect to be exposed to smart people who prefer basic science. My friend faces the reversed situation.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">What to do? Like in the last case, once we know what evidence our brain is generalizing from (conversations with people I respect), I can try and reason about how representative that evidence really is. If I figure out what respectable people believe in general, this <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">screens off <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> my particular experiences. [screening off should get talked about in the last example]

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Of course, I trust people I've interacted with more than those who I hear are respectable, partly because I am biased to trust my own evaluations of people more than the evaluations of others, and in part because there actually aren't comparably good indicators of clear thinking for people I can't interact with directly. Moreover, my brain isn't just counting the opinions of smart people I've talked with, it's aggregating impressions of their arguments' qualities, of their clarity of thought on this particular issue, and so on. So a much better solution, our fifth technique for being less wrong, is to recognize this biasing factor in our experiences, and then explicitly correct it--in this case, by searching for smart people from outside of academia and discussing these issues with them.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">(Should have 1-2 more examples)

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">--Exercise:

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Prompts for dredging up a belief: A disagreement you had with someone. A decision you were unsure about. A question where you disagree with the popular position, or where there is no unified popular position.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Pick a belief. Try and describe where the belief came from (write). Does the process that produced that belief generally produce true beliefs? Is there any aspect of the process that pushes beliefs in a direction <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;">not <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;"> correlated with truth? Consider the 5 techniques we have seen, and try and apply at least one or describe a plan for applying at least one. If you can't, think about it a bit harder (especially if others disagree with you!). If you still can't, try again with a different belief.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">1. Leave a line of retreat. <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">2. Use reference class forecasting. <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">3. Get advice / use simple explicit models. <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">4. Identify what evidence your brain is generalizing from, and generalize explicitly. <span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">5. Expose yourself to more representative evidence.

<span style="font-size:15px;font-family:Arial;color:#000000;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Repeat.