主持人:布利茨斯坦教授 Host: Professor Blitzstein
主要从事统计学实践的研究和讲学 is a professor of the practice of Statistics
他因统计110班讲授的《概率论导论》课程 He is known for teaching the popular class:
而备受关注 Stat110, Introduction to Probability
在每年秋季都会吸引超过300名学生前来旁听进修 which holds over 300 students each fall
有超过20万的人次通过iTunes You He also has over 200,000 subscribers to the class
在网上预订了他的课程 on iTunes You
他的研究方向集中在 His research interests focus on
复杂网络的统计推理 statistical inference for complex networks
布利茨斯坦教授还爱好国际象棋 Professor Blitzstein also enjoys chess
并且兼任哈佛国际象棋俱乐部顾问一职 and advises the Harvard Chess Club
下面我们就有请布利茨斯坦教授 Please join us in welcoming Professor Blitzstein
教授:感谢大家的参与 Professor: thank you all for coming
谢谢你的介绍 Thanks for the introduction
让我们现在就开始今天的课程吧 Uh, let's get started right away
我演讲的主题是 The title of my talk is
"统计之魂" "The Soul of Statistics"
在我为大家讲述什么是统计之魂前 Uh, let's start with a motivating example
让我们首先来看一个引例 before I tell you what I think the soul of statistics is
亚伯拉罕·沃尔德是一位著名的统计学家 So Abraham Wald was a famous statistician
在二战期间 and um, during World War II
许多的英国轰炸机 too many British bombers were being shot down
都被德国纳粹击落 by the Nazis
于是他就与英国方面商讨 So, he was consulting with them
应该在何处投放更多装甲部队的问题 about trying to figure out where to put more armor
要知道 若他们想投入更多装甲部队 You know, they want to put more armor
考虑到装甲的巨大重量和不菲花销 but the armor's heavy and expensive
也就意味着他们不可能四处随意投放 They can't just put armor everywhere;
那么应该在哪儿投放这些装甲呢？ so where do you put the armor?
好的 我在这儿画了一些飞机 Ok, so I drew some airplanes here
尽管它们看起来并不像轰炸机 these don't really look like the bombers but,
那也没关系 that's OK
因此 Um, so
这里有一些飞机 Um, here's some airplanes
我只画了一些弹孔 and I just drew some bullet holes:
这儿就有一个 uh, there's one
很抱歉 我并没有事先 um, sorry I didn't have a warning that
对这些带有暴力倾向的绘图进行告诫 there would be a graphic violence here so
希望不会有人心理产生纠结 hopefully no one is too squeamish
这是另外一架 There's another one
还有一架 another one
于是这里就…… Uh so these are just...
我刚才在这些飞机上画了一些弹孔 I just made up some bullet holes on these planes
这些就是他们必须观察的飞机 And these are the planes that they get to observe
他们…… They're...
好的 这只是一个例子罢了 Ok, uh just as an example:
设想他们正在向沃尔德询问 let's say so they brought in Wald to ask him
在哪里可以投放更多装甲的建议 where should they be putting more armor
那么 很明显的事情就是 Well, the obvious thing to do is:
我们看到了他们在所有飞机 look at where all the planes
承受严重损失的地区 are sustaining heavy damage
投放了更多的装甲 and armor those regions more
然而 沃尔德建议采取完全相反的策略 However, Wald said to do the exact opposite
寻找那些受损很小或没有受损 Look for the parts of the planes
并且能够顺利返航的 where there is very little or no damage on the planes
部分飞机 that came back
原因就在于 以这个例子来讲 The reason is, so for this example;
我保护了所有这些飞机的突出部分 I protected the nose on all of these planes
那么 这里究竟发生了什么？ And so, what's really going on here?
我们真正感兴趣的是 Well, what we're really interested in
那些我们没有看到的飞机 is the planes that we didn't see
都是未能返航的飞机 Those are the planes that didn't come back
而大家能够亲眼见到的飞机 the ones that you actually get to see are the planes
都是幸存下来并且成功返航的飞机 that actually survive well enough to return
因此这里就涉及到了一些统计思维 So, there's some statistical thinking in there
如何才能了解 How do you learn about the planes
那些看不到的飞机呢？ you don't get to see?
要知道 And you know,
对于那些错失的数据 that requires some statistical
这时就有必要采用一些 assumptions and statistical thinking about
统计假设和统计思维了 what do you do about the missing data
好的 如果它看起来是这样…… Alright, so maybe what if it looks like this...
这架飞机 我们没能看到 This plane, we didn't get to see it
因为它可能就是阿基里斯之踵 because that's likely the Achilles heel
或者说是飞机致命的突出部位 or the Achilles nose of the plane
也许这整支空军编队都有点类似于 Maybe there's like a whole army of these like
红鼻子的鲁道夫 like Rudolph the red-nosed aircraft
我们根本就无法看到他们 that we don't get to see any of them
因为他们都阵亡了 Because they all died
我们只看到了其它的一些部分 and we only saw the other ones
因此 这里就是投放装甲的地方 so, that's where you put the armor
基本上就是这样了 So that was the idea
回想起来这是个相当简单的想法 It's a pretty simple idea in retrospect
但如果没有统计思维就不会得出这一结论 but, easy to miss if you're not thinking statistically
好的 这是我就所说的统计学中的选择偏倚 Ok, so that's an example of what's called
而举出的一个例子 Selection Bias in Statistics
我们所知道的就是我们所看到的 What we know about is what we get to see
在那些安全返航的飞机上 which is where the bullet holes are distributed
我们看到了密布的弹孔 for the planes that actually return
但这并不是我们感兴趣的地方 But that's not what we're interested in,
我们想了解 we want to know about the planes
那些没有返航的飞机 that didn't make it back
那么 Ok,
我们怎样才能将这两者结合起来观察呢？ so how do we link those two things?
这就涉及到一个统计学的问题了 That's a statistical problem
于是我们不禁要问 什么才是统计之魂 Now we're ready to say what's the soul of statistics
我可不想跟大家卖关子 I won't keep you in suspense
条件作用就是统计之魂 Conditioning is the soul of statistics
也就是说你们之中 that means conditional probability
了解事实真相人群的条件概率 for those of you who know what that is
如果你还不清楚条件概率的概念 If you don't know what conditional probability (is)
它的意思就是我们得到信息 all it means is that we're given information
我们总会得到信息 we always have information
不论我们正在做什么事情 whatever we're doing
都是以我们所掌握的信息作为条件 it's all conditional on the information we have
条件概率告诉大家 And conditional probability tells us
如何更新我们的观点 how to update our beliefs
有了信息我们就能够观察 based on information that we're able to observe
每种可能性都是由条件而引发的 every probability is conditional
这就是问题的实质 and so that's the point of view here
不论在何时你得到了一组数据 Whenever you have a data set,
你都想弄清它是如何抽取样本的 you want to think about how it was sampled
这只是一幅图片 It's just a picture
我并不打算就此对网络做出一番探讨 I'm not really gonna talk about of the network
虽然这就是我在此工作的主要任务…… which is something I work on where a lot...
目前网络的研究 There's a huge amount of interest
变得备受关注 in networks these days
但许多研究却忽略了 but a lot of that work ignores
网络是如何形成的 where did the network come from,
以及它是如何进行采样的 how is it sampled
如果忽略了采样环节的工作 and that can lead to very misleading answers
如果不注意做事情的前提条件 if you ignore the sampling
那么往往就会得出 if you're not careful about
非常误导的结论 what you're conditioning on
要知道 And you know,
这也恰恰就是"哈佛大学 立大志" this is obviously
系列讲座的价值和意义所在 Harvard Thinks Big
接下来我将尝试进行大范围的思考 So I'm gonna try to think really big
这已不仅仅只是统计之魂了 This is just not the soul of statistics,
而可以是任何事情 this is everything
因为我们大家不会…… Because we wouldn't be...
如果我说了假话 if what I said was not true,
我们也就不会在此进行对话 we wouldn't be here having this conversation
这取决于一组 It's all conditional on a huge
我们并不会去做的 conditional probability calculation
海量条件概率计算 that we can't actually do
或者此时此地将我们大家 or things that brought us to this room
聚集一堂进行对话的动机 to have this conversation right now
凡事都是讲究条件的 everything is conditional
这里还有另外一个例子 So here's another example of
关于自1835年以来的一项关于寿命的研究 this little longevity study from 1835
银行家们热衷于 Uh Lombard was interested in
搜集大量数据 collecting huge amounts of data
这是一组让人印象非常深刻的数据 A very impressive data collection effort
其中采集的便是从事各类不同职业人群的 for how long
寿命 people in different professions live
那么 我将要给大家一些…… Ok, so I'll just give you a few...
我也选取了一些"选择偏倚"的例子 I chose a few examples that's the selection bias too,
鉴于时间有限 the ones that I felt like choosing
这些都是我精心挑选出的例子 because it's a short talk
那么 这里是职业 So here's the profession,
这里则是数据组中的平均寿命 here's the average longevity in his data set
数据结果显示排在最长寿命前两位的 So it turned out that the first or second longest
职业人群竟然是巧克力工人 living people in this data set were chocolate makers
由于我爱吃巧克力 So I was very happy to see that
对此我也就喜闻乐见了 cause I really like chocolate
他们可以活到73.6岁 They lived to be 73.6;
从1835年至今 which uh, life spans have improved
他们的寿命一直在延长 from 1835 to now
因此这在当时来讲已经是非常理想的了 So that's actually quite good back then
这并不是名单中排在第二位的职业 And uh, this is not second on the list,
但却是我所感兴趣的 but one that I was interested in:
教授可以活到66.6岁 was professors lived to be 66.6,
这就有点可悲了 so it's a little bit sad,
要知道 如果我当初选择了其它的职业 you know I could have
我的寿命没准还会更长 if I'd chosen a different career path
我本可以多活七年 I could have lived seven years longer
并且还能吃到我想吃的巧克力 and had all the chocolate that I want
这里就涉及到一个样本量大小的问题 Um, there is a sample size issue here,
这张图表只搜集了 which is he only had nine chocolate makers
九位巧克力工人的数据 in his sample
这就是非常小的一个样本量了 That's a very small sample
在这项研究中教授的样本量 There are a lot more professors
要远大于巧克力工人的样本量 than chocolate makers in this study
但总之 我…… But anyway, I...
我们继续 Let's go on:
钟表匠可以活到55.3岁 clocksmiths lived to be 55.3,
这只能说一般般了 so that's less attractive
锁匠则可以活到47.2岁 locksmiths 47.2;
我试图弄清这种 I was trying to figure out
存在于钟表匠和锁匠之间 why is there this discrepancy
寿命不一致的原因 between clocksmiths and locksmiths,
怎样来解释这种现象呢？ What could explain this?
我还没有找到答案 I don't have an answer yet
还有一项数据 One more
也许是我们大家都会很感兴趣的 that is of interest to all of us;
学生的寿命 很不幸 the students um unfortunately,
我不得不遗憾地告诉在座的各位 I have some bad news for all the students here
在他的研究中 他发现 Uh, in his study, he found
学生的平均寿命是20.2岁 the average longevity of students was 20.2 years!
因此 我可不想 So, I didn't want to be like,
一开始就让大家扫兴 get this event off to a depressing start,
作为本次活动的第一位发言者 I'm the first speaker
我肯定不希望大家由于过于沮丧的情绪 and I don't want everyone to be too depressed
而耽误了收听接下来的其它演讲 to pay attention to the other talks that are coming up,
但20.2岁 的确还是很令人担忧的 but 20.2 years, you know it's worrisome
如果没看过的话 我向你们推荐 Uh, I recommend that if you haven't seen
史蒂夫·乔布斯在2005年 Steve Jobs' commencement address
斯坦福大学毕业典礼上发表的演讲 from Stanford 2005,
当时我也在场 那时我还在斯坦福…… I was there because I was at Stanford...
伟大的演讲 谢谢 It's a great talk, thanks
最大限度地让自己过好每一天 And um about living each day to the max and all that
好的 Ok,
但这种逻辑就会存在一个明显的缺陷 对吧？ but there's an obvious problem with that logic, right?
发生这种情况的条件是什么呢？ Which is what are you conditioning on,
要知道 一个学生 you know, a student,
20岁正是一个学生风华正茂的年龄 you know 20 years old is a normal age for a student
好的 这样就很容易看出问题来了 Ok, so it's kinda obvious to see what goes wrong
这是一个展现极端情况的绝佳案例 it's a good example of looking at an extreme case
在一个人职业生涯的早期 Because the earlier professions,
还并没有明显的偏好 it's not obvious they're all kinds of biases going on
但当你看到一个学生简单 But when you see the simple
而极端的案例 and extreme case of a student,
你就能明白整件事情就显得非常可疑了 you can see that the whole thing is very suspicious
这里还有另外一件事情…… There's another issue of...
我还想指出的一点就是关于 I want to mention one more thing about that
关于真相的核实 即 Uh, about censoring, which is that:
只有当人们死去时 we only know how long people lived
我们才知道他们活了多久 after they died
那么 通常与一个死人对话 Ok, so usually it's a lot harder to talk to dead people
要比跟一个活人对话困难得多 than to talk to live people
登奇教授将会就如何与死人对话 and Professor Dench will be telling us about
向大家进行详细的讲解 talking to dead people in detail
但这是一种相对来讲更容易 But this is one case where it's much easier
处理死人的案例 to deal with the dead people
因为我们知道他们活了多久 because we know how long they lived
那么 对更大规模生存分析数据的统计 Ok, so there's a larger statistical survival analysis
就是一个尝试处理我们未知事实的领域 is a field to try to deal with the fact that we don't know
活着的人们还将继续活多久 how long living people are going to continue to live
我们不得而知…… We don't...
如果我们对这个问题置若罔闻 If we're not careful about that,
那么我们的偏好也不会好到哪里去 we're gonna get some bad bias
好的 再举一个例子 Alright, one more example,
这是一则统计学中的经典案例 which is a very famous one in statistics,
即趋均数回归 并且这种现象无处不在 is regression towards the mean, and it's everywhere
如果不熟悉的话 Uh, so just to take an example
请看这个例子 if you're not familiar with this
以考试分数来说 Uh, take test scores:
你有一组学生参加SAT考试的数据 so you have a group of students take the SAT
假设让表现非常好的学生 And then suppose that
再考一遍 the ones who did really well take it again
让表现非常差的学生也再考一遍 and the ones that did it very poorly take it again
因此我们正在做出选择 So we're doing a selection ok
那么 趋均数回归就表明 Well, regression towards the mean says that
我们所期待的那些…… we expect the one...
那些在SAT考试中表现非常好的学生 the students who did really well on the SAT,
当他们在参加重试时 when they retake it,
他们的分数会降低 their scores are gonna tend to
并向平均值靠拢 go down towards the mean
而那些在初试中表现非常差的学生的分数 The ones who did really poorly
则会提高 are gonna go up
这样的例子比比皆是 There's examples everywhere
在早期的一些例子中就有这么一个 One of the first examples:
在此我并不想对这张数据表 I'm not gonna go through
进行深入的解读 this data table here
就当做是一个历史猎奇来看待吧 But that's just for historical curiosity,
弗朗西斯·高尔顿先生 Sir Francis Galton,
作为达尔文的表兄 who was a cousin of Darwin
曾经是第一位 可能就是第一位 was one of the first, possibly the first person
清楚阐释趋均数回归的科学家 to clearly explain regression towards the mean
他是通过研究人体身高才得出这一发现的 And he discovered it by studying heights
父母和孩子的身高 Parents and children's heights
经研究 他发现 举例来讲 And he found that, for example
如果单论爸爸和儿子 if you look at fathers and sons,
其实并没有必要担心性别差异 so we don't have to worry with differences uh,
在身高上的不同 gender differences in heights.
让我们以爸爸和儿子为例 Let's just take fathers and sons
因此 他通过对高大爸爸的观察 Ok, so he looked at very tall fathers,
从而得出他们儿子的身高 how tall are their sons
相应地 这些儿子同样也会长得很高 Well, the sons are still pretty tall,
但是他们最终都会趋向于回归到平均值 but they are regressing towards the mean
那么 个头很矮的爸爸 Ok, and very short fathers,
他们的儿子也会长得比他们更高 their sons tend to be taller,
但有时也并不总是趋向于平均值 not, not all the way to the mean
这就是趋均数回归 This is regression toward the mean
并不是完全回归 Not to the mean
但却是沿着这个趋势发展 but it's moving in that direction
不论是在统计学上还是在生活上 That phenomenon is everywhere
这种现象都无处不在 in statistics and in life
因此 如果你能睁大双眼细心去观察 So, if you have your eyes open to look for it,
那就再好不过了 that's a good thing
我只想快速引述 And so um, I just wanna quickly quote uh,
丹尼尔·卡内曼的一段话 Daniel Kahneman,
这也是一段对于趋均数回归的 cause it's a beautiful quote
经典概括 about regression towards the mean
他说 He said:
"我在自己的职业生涯中 "I had the most satisfying Eureka experience
学到了最令人满意的一课 of my career
当我试图去说服飞行教官 while attempting to teach flight instructors that
在提升飞行员飞行技术的指导上 praise is more effective than punishment
采用表扬而非惩罚的方式才会更有效时……" for promoting skill learning..."
当他的言论一出 And when he gave that talk,
立即就遭到了飞行教官的反驳 a flight instructor objected:
"在多数情况下我会表扬那些 "on many occasions I've praised flight cadets
清楚执行特技飞行策略的飞行学员 for clean execution of some aerobatic maneuver,
总体来讲 他们只会越做越差 and in general when they try it again, they do worse
另一方面 On the other hand
我也会经常冲着那些表现差劲的 I have often screamed at cadets
飞行学员怒吼 for bad execution
而他们通常又会在下次做得越来越好 and then they do better the next time.
因此请不要告诉我鼓励有效 So don't tell me that reinforcement works
而惩罚无用……" and punishment does not..."
卡内曼说那是一段美好的时光 And Kahneman said that's a joyous moment,
因为他意识到 because what he realizes is
我们会奖励那些做得好的人 we tend to reward others when they do well
而惩罚那些做得不好的人 and punish them when they do badly
由于趋均数回归的存在 And because there is regression to the mean,
人们评论的态度也会趋于平均 you should say toward the mean
据统计 人们因回报他人而遭到惩罚 It's part of the human condition
并因惩罚他人而得到回报 这种现象 that we are statistically punished for rewarding others
已经成为了人类生存状况的一部分 and rewarded for punishing them
然而人们对于趋均数回归 And uh, so there are a lot of misconceptions about uh,
仍旧存在着诸多误解 regression toward the mean
由于时间关系我就不赘述了 that I don't have time to get into detail of
但是我想提供给大家一张小图 But I want to give a quick little picture
来描绘一种……假想实验 illustrating what... A hypothetical thought experiment:
"没有趋均数回归的世界 "What would the world be like
会变成什么样子" without regression toward the mean"
许多人似乎存在这样一种误解 Because a lot of people seem to have the misconception
即趋均数回归就表示 that regression toward the mean would mean that
每个人的价值 everyone is gonna be
最终都会趋于一致 eventually converging to the same value
而事实上 如果没有趋均数回归…… And actually, without regression toward the mean,
我们需要利用趋均数回归 well, we need regression toward the mean
来保持人口稳定 in order to have stability in the population
因此我虚构了一个简单的例子 so I just made up a simple example
假设我们有三组人群 Suppose we had three people,
他们的身高各不相同 three different heights
并且假设每个人只有一个孩子 and suppose each person has a child
一组人群有点矮 Who's equally likely to be a little bit shorter,
一组人群一般高 the same height
一组人群有点高 or a little bit taller
那么 我给他们每个人三个孩子 Ok, so I just gave them three kids each
有点矮 一般高 有点高 A little shorter, same height, little bit taller
因此这就是下一代人 so that's the next generation
接下来我挑出这代人 Then I just took that generation,
进行重组 just rearranged
情况还是一样的 These are the same people
同样的事情又发生了 Now, same thing happens again
这是下一代人 That's the next generation
接下来我再添加一代人 and then, I'll just take one more generation
这就是没有趋均数回归 This is what the population would look like
人口发展的样子 without regression toward the mean
大家所看到的 What you see
只是第三代和第四代的人口 is after only three or four generations
要知道 你将会看到巨人 you know, you're gonna see giant,
10英尺高的人 you know ten-foot people walking around
以及4英寸高的人 And you have little four-inch people or people like that
这就是缺乏趋均数回归后 That's what the world would be like
世界将会变成的样子 without regression toward the mean
那么 最后 Ok, lastly,
我想回到我个人 I want to tie this back to this kind of personal
对于教学和研究的观点 perspective on teaching and research
我把它称为 which is something I call:
条件黄金法则 the conditional golden rule
这是一个几年前我在提示指南上 This was a course evaluation
找到的课程评价 that I got on the cue guide a few years ago
有人就写道 Um, so someone wrote:
"这是一门充满 "It was a class
布利茨斯坦教授风格的课程 that was designed by Blitz with the credo
而我想要教授一门 'I want to teach a class that
我自己愿意学习的课程" I would want to take myself"
看到这些我非常高兴 So I was really happy when I saw that
因为这是一条非常真诚的评价 because, well, it was a nice comment,
但它也同时让我回忆起一些事情 but it also reminded me of something
或是我忘记的事情 that I had forgotten
或是我说过的事情 that I had said
我有点喜欢……我喜欢这种感觉 and I kind of liked... I liked that
总而言之 Um, however
为了得到一个圆满的结局 Uh, in the interest of full disclosure
同年还出现了另一则评论说道 that same year, there was also a comment that said
"负面评论只会导致痛苦 "Writes horrible problem sets that only cause pain,
而学不到……" not learning..."
很不幸我又钻牛角尖了 So, unfortunately I try to be logical
从这两条评论中所得出的结论即为 the logical conclusion from these two things
我是个受虐狂 is that I'm a masochist
那么 就让我…… Um, so let me just...
就让我告诉大家什么是条件黄金法则 So let me just tell you what the conditional golden rule is
然后我就可以闭嘴了 and then I'll stop
这就是我精辟的教学理念 This is my philosophy of teaching in a nutshell
"己所不欲 勿施于人" "Do unto others as you would have (them) do unto you"
除非是有条件的 对吧？ Except it's conditional, right?
凡事都要讲究条件 Everything is conditional,
这是一门我有条件接受的课程 it's the class that I would take conditionally
即它符合了我的兴趣 I'm having the same interest
背景和学识 and background and knowledge
因此我只是努力在按照 So, I just try to do
条件黄金法则来做事情 the conditional golden rule
我正在努力做好这次演讲 I'm trying to follow it right now in giving this talk
如果我失败了 And if I failed,
请怪罪于条件黄金法则吧 blame the Conditional Golden Rule
或者怪罪于条件概率太难这一事实吧 Or, blame the fact that conditional probability is hard
诚如大家所见 统计110班的课程 as you would see in Stat110
难度很大 但却是发人深思的 It's hard, but it's worth thinking about
好的 我想感谢主办方 Ok, so I just want to thank the organizers
和我的同事 and thank my colleagues
尤其是卡尔·莫里斯 Especially Carl Morris,
孟晓丽和戴夫·哈林顿 Xiao-Li Meng and Dave Harrington
以及统计110班的全体学生和助教们 And all my Stat110 students and TFs
当然还有今天在座的各位 And all of you for coming
谢谢大家 Thank you