WEBVTT 00:00:12.390 --> 00:00:16.597 I work with a bunch of mathematicians, philosophers and computer scientists, 00:00:16.597 --> 00:00:21.806 and we sit around and think about the future of machine intelligence, 00:00:21.806 --> 00:00:23.850 among other things. 00:00:23.850 --> 00:00:28.575 Some people think that some of these things are sort of science fiction-y, 00:00:28.575 --> 00:00:31.676 far out there, crazy. 00:00:31.676 --> 00:00:33.146 But I like to say, 00:00:33.146 --> 00:00:36.750 okay, let's look at the modern human condition. 00:00:36.750 --> 00:00:38.442 (Laughter) 00:00:38.442 --> 00:00:40.844 This is the normal way for things to be. 00:00:40.844 --> 00:00:43.129 But if we think about it, 00:00:43.129 --> 00:00:46.422 we are actually recently arrived guests on this planet, 00:00:46.422 --> 00:00:48.504 the human species. 00:00:48.504 --> 00:00:53.250 Think about if Earth was created one year ago, 00:00:53.250 --> 00:00:56.798 the human species, then, would be 10 minutes old. 00:00:56.798 --> 00:00:59.966 The industrial era started two seconds ago. 00:01:01.096 --> 00:01:06.321 Another way to look at this is to think of world GDP over the last 10,000 years, 00:01:06.321 --> 00:01:09.350 I've actually taken the trouble to plot this for you in a graph. 00:01:09.350 --> 00:01:11.124 It looks like this. 00:01:11.124 --> 00:01:12.487 (Laughter) 00:01:12.487 --> 00:01:14.638 It's a curious shape for a normal condition. 00:01:14.638 --> 00:01:16.336 I sure wouldn't want to sit on it. 00:01:16.336 --> 00:01:18.887 (Laughter) 00:01:18.887 --> 00:01:23.661 Let's ask ourselves, what is the cause of this current anomaly? 00:01:23.661 --> 00:01:26.213 Some people would say it's technology. 00:01:26.213 --> 00:01:30.881 Now it's true, technology has accumulated through human history, 00:01:30.881 --> 00:01:35.533 and right now, technology advances extremely rapidly -- 00:01:35.533 --> 00:01:37.098 that is the proximate cause, 00:01:37.098 --> 00:01:39.663 that's why we are currently so very productive. 00:01:40.293 --> 00:01:43.954 But I like to think back further to the ultimate cause. 00:01:44.934 --> 00:01:48.700 Look at these two highly distinguished gentlemen: 00:01:48.700 --> 00:01:50.300 We have Kanzi -- 00:01:50.300 --> 00:01:54.943 he's mastered 200 lexical tokens, an incredible feat. 00:01:54.943 --> 00:01:58.637 And Ed Witten unleashed the second superstring revolution. 00:01:58.637 --> 00:02:00.961 If we look under the hood, this is what we find: 00:02:00.961 --> 00:02:02.531 basically the same thing. 00:02:02.531 --> 00:02:04.344 One is a little larger, 00:02:04.344 --> 00:02:07.102 it maybe also has a few tricks in the exact way it's wired. 00:02:07.102 --> 00:02:10.914 These invisible differences cannot be too complicated, however, 00:02:10.914 --> 00:02:15.199 because there have only been 250,000 generations 00:02:15.199 --> 00:02:16.931 since our last common ancestor. 00:02:16.931 --> 00:02:20.780 We know that complicated mechanisms take a long time to evolve. 00:02:21.820 --> 00:02:24.319 So a bunch of relatively minor changes 00:02:24.319 --> 00:02:27.386 take us from Kanzi to Witten, 00:02:27.386 --> 00:02:31.929 from broken-off tree branches to intercontinental ballistic missiles. 00:02:32.659 --> 00:02:36.594 So this then seems pretty obvious that everything we've achieved, 00:02:36.594 --> 00:02:37.972 and everything we care about, 00:02:37.972 --> 00:02:43.200 depends crucially on some relatively minor changes that made the human mind. 00:02:44.470 --> 00:02:48.132 And the corollary, of course, is that any further changes 00:02:48.132 --> 00:02:51.609 that could significantly change the substrate of thinking 00:02:51.609 --> 00:02:54.811 could have potentially enormous consequences. 00:02:56.141 --> 00:02:59.046 Some of my colleagues think we're on the verge 00:02:59.046 --> 00:03:02.954 of something that could cause a profound change in that substrate, 00:03:02.954 --> 00:03:06.167 and that is machine superintelligence. 00:03:06.167 --> 00:03:10.906 Artificial intelligence used to be about putting commands in a box. 00:03:10.906 --> 00:03:12.571 You would have human programmers 00:03:12.571 --> 00:03:15.706 that would painstakingly handcraft knowledge items. 00:03:15.706 --> 00:03:17.792 You build up these expert systems, 00:03:17.792 --> 00:03:20.116 and they were kind of useful for some purposes, 00:03:20.116 --> 00:03:22.797 but they were very brittle, you couldn't scale them. 00:03:22.797 --> 00:03:26.230 Basically, you got out only what you put in. 00:03:26.230 --> 00:03:27.227 But since then, 00:03:27.227 --> 00:03:30.694 a paradigm shift has taken place in the field of artificial intelligence. 00:03:30.694 --> 00:03:33.464 Today, the action is really around machine learning. 00:03:34.214 --> 00:03:39.601 So rather than handcrafting knowledge representations and features, 00:03:40.331 --> 00:03:45.885 we create algorithms that learn, often from raw perceptual data. 00:03:45.885 --> 00:03:50.883 Basically the same thing that the human infant does. 00:03:50.883 --> 00:03:55.090 The result is A.I. that is not limited to one domain -- 00:03:55.090 --> 00:03:59.721 the same system can learn to translate between any pairs of languages, 00:03:59.721 --> 00:04:05.158 or learn to play any computer game on the Atari console. 00:04:05.158 --> 00:04:06.937 Now of course, 00:04:06.937 --> 00:04:10.936 A.I. is still nowhere near having the same powerful, cross-domain 00:04:10.936 --> 00:04:14.155 ability to learn and plan as a human being has. 00:04:14.155 --> 00:04:16.281 The cortex still has some algorithmic tricks 00:04:16.281 --> 00:04:18.636 that we don't yet know how to match in machines. 00:04:19.706 --> 00:04:21.605 So the question is, 00:04:21.605 --> 00:04:25.105 how far are we from being able to match those tricks? 00:04:26.065 --> 00:04:27.148 A couple of years ago, 00:04:27.148 --> 00:04:30.036 we did a survey of some of the world's leading A.I. experts, 00:04:30.036 --> 00:04:33.260 to see what they think, and one of the questions we asked was, 00:04:33.260 --> 00:04:36.613 "By which year do you think there is a 50 percent probability 00:04:36.613 --> 00:04:40.095 that we will have achieved human-level machine intelligence?" 00:04:40.605 --> 00:04:44.788 We defined human-level here as the ability to perform 00:04:44.788 --> 00:04:47.659 almost any job at least as well as an adult human, 00:04:47.659 --> 00:04:51.664 so real human-level, not just within some limited domain. 00:04:51.664 --> 00:04:55.314 And the median answer was 2040 or 2050, 00:04:55.314 --> 00:04:58.120 depending on precisely which group of experts we asked. 00:04:58.120 --> 00:05:02.159 Now, it could happen much, much later, or sooner, 00:05:02.159 --> 00:05:04.099 the truth is nobody really knows. 00:05:05.079 --> 00:05:09.491 What we do know is that the ultimate limit to information processing 00:05:09.491 --> 00:05:14.362 in a machine substrate lies far outside the limits in biological tissue. 00:05:15.061 --> 00:05:17.439 This comes down to physics. 00:05:17.439 --> 00:05:22.157 A biological neuron fires, maybe, at 200 hertz, 200 times a second. 00:05:22.157 --> 00:05:25.751 But even a present-day transistor operates at the Gigahertz. 00:05:25.751 --> 00:05:31.048 Neurons propagate slowly in axons, 100 meters per second, tops. 00:05:31.048 --> 00:05:34.159 But in computers, signals can travel at the speed of light. 00:05:34.899 --> 00:05:36.768 There are also size limitations, 00:05:36.768 --> 00:05:39.795 like a human brain has to fit inside a cranium, 00:05:39.795 --> 00:05:44.556 but a computer can be the size of a warehouse or larger. 00:05:44.556 --> 00:05:50.155 So the potential for superintelligence lies dormant in matter, 00:05:50.155 --> 00:05:55.867 much like the power of the atom lay dormant throughout human history, 00:05:55.867 --> 00:06:00.272 patiently waiting there until 1945. 00:06:00.272 --> 00:06:01.520 In this century, 00:06:01.520 --> 00:06:05.638 scientists may learn to awaken the power of artificial intelligence. 00:06:05.638 --> 00:06:09.646 And I think we might then see an intelligence explosion. 00:06:10.226 --> 00:06:14.183 Now most people, when they think about what is smart and what is dumb, 00:06:14.183 --> 00:06:17.206 I think have in mind a picture roughly like this. 00:06:17.206 --> 00:06:19.804 So at one end we have the village idiot, 00:06:19.804 --> 00:06:22.287 and then far over at the other side 00:06:22.287 --> 00:06:27.043 we have Ed Witten, or Albert Einstein, or whoever your favorite guru is. 00:06:27.043 --> 00:06:30.877 But I think that from the point of view of artificial intelligence, 00:06:30.877 --> 00:06:34.558 the true picture is actually probably more like this: 00:06:35.078 --> 00:06:38.456 AI starts out at this point here, at zero intelligence, 00:06:38.456 --> 00:06:41.467 and then, after many, many years of really hard work, 00:06:41.467 --> 00:06:45.311 maybe eventually we get to mouse-level artificial intelligence, 00:06:45.311 --> 00:06:47.741 something that can navigate cluttered environments 00:06:47.741 --> 00:06:49.728 as well as a mouse can. 00:06:49.728 --> 00:06:54.041 And then, after many, many more years of really hard work, lots of investment, 00:06:54.041 --> 00:06:58.680 maybe eventually we get to chimpanzee-level artificial intelligence. 00:06:58.680 --> 00:07:01.890 And then, after even more years of really, really hard work, 00:07:01.890 --> 00:07:04.803 we get to village idiot artificial intelligence. 00:07:04.803 --> 00:07:08.075 And a few moments later, we are beyond Ed Witten. 00:07:08.075 --> 00:07:11.045 The train doesn't stop at Humanville Station. 00:07:11.045 --> 00:07:14.067 It's likely, rather, to swoosh right by. 00:07:14.067 --> 00:07:16.051 Now this has profound implications, 00:07:16.051 --> 00:07:19.913 particularly when it comes to questions of power. 00:07:19.913 --> 00:07:21.812 For example, chimpanzees are strong -- 00:07:21.812 --> 00:07:27.034 pound for pound, a chimpanzee is about twice as strong as a fit human male. 00:07:27.034 --> 00:07:31.648 And yet, the fate of Kanzi and his pals depends a lot more 00:07:31.648 --> 00:07:35.788 on what we humans do than on what the chimpanzees do themselves. 00:07:37.048 --> 00:07:39.362 Once there is superintelligence, 00:07:39.362 --> 00:07:43.201 the fate of humanity may depend on what the superintelligence does. 00:07:44.271 --> 00:07:45.328 Think about it: 00:07:45.328 --> 00:07:50.372 Machine intelligence is the last invention that humanity will ever need to make. 00:07:50.372 --> 00:07:53.345 Machines will then be better at inventing than we are, 00:07:53.345 --> 00:07:55.885 and they'll be doing so on digital timescales. 00:07:55.885 --> 00:08:00.786 What this means is basically a telescoping of the future. 00:08:00.786 --> 00:08:04.344 Think of all the crazy technologies that you could have imagined 00:08:04.344 --> 00:08:07.142 maybe humans could have developed in the fullness of time: 00:08:07.142 --> 00:08:10.400 cures for aging, space colonization, 00:08:10.400 --> 00:08:14.131 self-replicating nanobots or uploading of minds into computers, 00:08:14.131 --> 00:08:16.290 all kinds of science fiction-y stuff 00:08:16.290 --> 00:08:19.027 that's nevertheless consistent with the laws of physics. 00:08:19.027 --> 00:08:23.239 All of this superintelligence could develop, and possibly quite rapidly. 00:08:24.269 --> 00:08:27.827 Now, a superintelligence with such technological maturity 00:08:27.827 --> 00:08:30.006 would be extremely powerful, 00:08:30.006 --> 00:08:34.552 and at least in some scenarios, it would be able to get what it wants. 00:08:34.552 --> 00:08:40.213 We would then have a future that would be shaped by the preferences of this A.I. 00:08:41.675 --> 00:08:45.424 Now a good question is, what are those preferences? 00:08:46.064 --> 00:08:47.833 Here it gets trickier. 00:08:47.833 --> 00:08:49.268 To make any headway with this, 00:08:49.268 --> 00:08:52.544 we must first of all avoid anthropomorphizing. 00:08:53.754 --> 00:08:57.055 And this is ironic because every newspaper article 00:08:57.055 --> 00:09:00.910 about the future of A.I. has a picture of this: 00:09:02.100 --> 00:09:06.234 So I think what we need to do is to conceive of the issue more abstractly, 00:09:06.234 --> 00:09:09.024 not in terms of vivid Hollywood scenarios. 00:09:09.024 --> 00:09:12.641 We need to think of intelligence as an optimization process, 00:09:12.641 --> 00:09:18.290 a process that steers the future into a particular set of configurations. 00:09:18.290 --> 00:09:21.801 A superintelligence is a really strong optimization process. 00:09:21.801 --> 00:09:25.918 It's extremely good at using available means to achieve a state 00:09:25.918 --> 00:09:27.827 in which its goal is realized. 00:09:28.267 --> 00:09:30.939 This means that there is no necessary conenction between 00:09:30.939 --> 00:09:33.673 being highly intelligent in this sense, 00:09:33.673 --> 00:09:38.335 and having an objective that we humans would find worthwhile or meaningful. 00:09:39.141 --> 00:09:42.935 Suppose we give an A.I. the goal to make humans smile. 00:09:42.935 --> 00:09:45.917 When the A.I. is weak, it performs useful or amusing actions 00:09:45.917 --> 00:09:48.434 that cause its user to smile. 00:09:48.434 --> 00:09:50.851 When the A.I. becomes superintelligent, 00:09:50.851 --> 00:09:54.374 it realizes that there is a more effective way to achieve this goal: 00:09:54.374 --> 00:09:56.296 take control of the world 00:09:56.296 --> 00:09:59.458 and stick electrodes into the facial muscles of humans 00:09:59.458 --> 00:10:02.399 to cause constant, beaming grins. 00:10:02.399 --> 00:10:03.434 Another example, 00:10:03.434 --> 00:10:06.817 suppose we give A.I. the goal to solve a difficult mathematical problem. 00:10:06.817 --> 00:10:08.754 When the A.I. becomes superintelligent, 00:10:08.754 --> 00:10:12.925 it realizes that the most effective way to get the solution to this problem 00:10:12.925 --> 00:10:15.855 is by transforming the planet into a giant computer, 00:10:15.855 --> 00:10:18.101 so as to increase its thinking capacity. 00:10:18.101 --> 00:10:20.865 And notice that this gives the A.I.s an instrumental reason 00:10:20.865 --> 00:10:23.381 to do things to us that we might not approve of. 00:10:23.381 --> 00:10:25.316 Human beings in this model are threats, 00:10:25.316 --> 00:10:28.237 we could prevent the mathematical problem from being solved. 00:10:29.027 --> 00:10:32.521 Of course, perceivably things won't go wrong in these particular ways; 00:10:32.521 --> 00:10:34.274 these are cartoon examples. 00:10:34.274 --> 00:10:36.213 But the general point here is important: 00:10:36.213 --> 00:10:39.086 if you create a really powerful optimization process 00:10:39.086 --> 00:10:41.320 to maximize for objective x, 00:10:41.320 --> 00:10:43.596 you better make sure that your definition of x 00:10:43.596 --> 00:10:46.065 incorporates everything you care about. 00:10:46.655 --> 00:10:51.039 This is a lesson that's also taught in many a myth. 00:10:51.039 --> 00:10:56.337 King Midas wishes that everything he touches be turned into gold. 00:10:56.337 --> 00:10:59.198 He touches his daughter, she turns into gold. 00:10:59.198 --> 00:11:01.751 He touches his food, it turns into gold. 00:11:01.751 --> 00:11:04.340 This could become practically relevant, 00:11:04.340 --> 00:11:06.410 not just as a metaphor for greed, 00:11:06.410 --> 00:11:08.305 but as an illustration of what happens 00:11:08.305 --> 00:11:11.142 if you create a powerful optimization process 00:11:11.142 --> 00:11:15.931 and give it misconceived or poorly specified goals. 00:11:15.931 --> 00:11:21.120 Now you might say, if a computer starts sticking electrodes into people's faces, 00:11:21.120 --> 00:11:23.385 we'd just shut it off. 00:11:24.375 --> 00:11:29.715 A, this is not necessarily so easy to do if we've grown dependent on the system -- 00:11:29.715 --> 00:11:32.447 like, where is the off switch to the Internet? 00:11:32.447 --> 00:11:37.567 B, why haven't the chimpanzees flicked the off switch to humanity, 00:11:37.567 --> 00:11:39.118 or the Neanderthals? 00:11:39.118 --> 00:11:41.784 They certainly had reasons. 00:11:41.784 --> 00:11:44.579 We have an off switch, for example, right here. 00:11:44.579 --> 00:11:46.133 (Choking) 00:11:46.133 --> 00:11:49.058 The reason is that we are an intelligent adversary; 00:11:49.058 --> 00:11:51.786 we can anticipate threats and plan around them. 00:11:51.786 --> 00:11:54.290 But so could a superintelligent agent, 00:11:54.290 --> 00:11:57.544 and it would be much better at that than we are. 00:11:57.544 --> 00:12:04.731 The point is, we should not be confident that we have this under control here. 00:12:04.731 --> 00:12:08.178 And we could try to make our job a little bit easier by, say, 00:12:08.178 --> 00:12:09.768 putting the A.I. in a box, 00:12:09.768 --> 00:12:11.564 like a secure software environment, 00:12:11.564 --> 00:12:14.586 a virtual reality simulation from which it cannot escape. 00:12:14.586 --> 00:12:18.732 But how confident can we be that the A.I. couldn't find a bug. 00:12:18.732 --> 00:12:21.901 Given that merely human hackers find bugs all the time, 00:12:21.901 --> 00:12:24.937 I'd say, probably not very confident. 00:12:26.057 --> 00:12:30.605 So we disconnect the ethernet cable to create an air gap, 00:12:30.605 --> 00:12:33.273 but again, like merely human hackers 00:12:33.273 --> 00:12:36.654 routinely transgress air gaps using social engineering. 00:12:36.654 --> 00:12:37.913 Right now, as I speak, 00:12:37.913 --> 00:12:40.302 I'm sure there is some employee out there somewhere 00:12:40.302 --> 00:12:43.648 who has been talked into handing out her account details 00:12:43.648 --> 00:12:46.394 by somebody claiming to be from the I.T. department. 00:12:46.394 --> 00:12:48.521 More creative scenarios are also possible, 00:12:48.521 --> 00:12:49.836 like if you're the A.I., 00:12:49.836 --> 00:12:53.368 you can imagine wiggling electrodes around in your internal circuitry 00:12:53.368 --> 00:12:56.830 to create radio waves that you can use to communicate. 00:12:56.830 --> 00:12:59.254 Or maybe you could pretend to malfunction, 00:12:59.254 --> 00:13:02.751 and then when the programmers open you up to see what went wrong with you, 00:13:02.751 --> 00:13:04.687 they look at the source code -- Bam! -- 00:13:04.687 --> 00:13:07.134 the manipulation can take place. 00:13:07.134 --> 00:13:10.564 Or it could output the blueprint to a really nifty technology, 00:13:10.564 --> 00:13:11.962 and when we implement it, 00:13:11.962 --> 00:13:16.359 it has some surreptitious side effect that the A.I. had planned. 00:13:16.359 --> 00:13:19.822 The point here is that we should not be confident in our ability 00:13:19.822 --> 00:13:23.630 to keep a superintelligent genie locked up in its bottle forever. 00:13:23.630 --> 00:13:25.884 Sooner or later, it will out. 00:13:26.854 --> 00:13:29.957 I believe that the answer here is to figure out 00:13:29.957 --> 00:13:34.981 how to create superintelligent A.I. such that even if -- when -- it escapes, 00:13:34.981 --> 00:13:38.258 it is still safe because it is fundamentally on our side 00:13:38.258 --> 00:13:40.157 because it shares our values. 00:13:40.157 --> 00:13:43.367 I see no way around this difficult problem. 00:13:44.377 --> 00:13:48.211 Now, I'm actually fairly optimistic that this problem can be solved. 00:13:48.211 --> 00:13:52.114 We wouldn't have to write down a long list of everything we care about, 00:13:52.114 --> 00:13:55.757 or worse yet, spell it out in some computer language 00:13:55.757 --> 00:13:57.211 like C++ or Python, 00:13:57.211 --> 00:13:59.978 that would be a task beyond hopeless. 00:13:59.978 --> 00:14:04.275 Instead, we would create an A.I. that uses its intelligence 00:14:04.275 --> 00:14:07.046 to learn what we value, 00:14:07.046 --> 00:14:12.326 and its motivation system is constructed in such a way that it is motivated 00:14:12.326 --> 00:14:17.558 to pursue our values or to perform actions that it predicts we would approve of. 00:14:17.558 --> 00:14:20.972 We would thus leverage its intelligence as much as possible 00:14:20.972 --> 00:14:23.717 to solve the problem of value-loading. 00:14:24.547 --> 00:14:26.059 This can happen, 00:14:26.059 --> 00:14:29.655 and the outcome could be very good for humanity. 00:14:29.655 --> 00:14:33.612 But it doesn't happen automatically. 00:14:33.612 --> 00:14:36.610 The initial conditions for the intelligence explosion 00:14:36.610 --> 00:14:39.473 might need to be set up in just the right way 00:14:39.473 --> 00:14:43.003 if we are to have a controlled detonation. 00:14:43.003 --> 00:14:45.621 The values that the A.I. has need to match ours, 00:14:45.621 --> 00:14:47.381 not just in the familiar context, 00:14:47.381 --> 00:14:49.819 like where we can easily check how the A.I. behaves, 00:14:49.819 --> 00:14:53.053 but also in all novel contexts that the A.I. might encounter 00:14:53.053 --> 00:14:54.610 in the indefinite future. 00:14:54.610 --> 00:14:59.347 And there are also some esoteric issues that would need to be solved, sorted out: 00:14:59.347 --> 00:15:01.436 the exact details of its decision theory, 00:15:01.436 --> 00:15:04.300 how to deal with logical uncertainty and so forth. 00:15:05.150 --> 00:15:08.252 So the technical problems that need to be solved to make this work 00:15:08.252 --> 00:15:09.365 look quite difficult -- 00:15:09.365 --> 00:15:12.745 not as difficult as making a superintelligent A.I., 00:15:12.745 --> 00:15:15.613 but fairly difficult. 00:15:15.613 --> 00:15:17.308 Here is the worry: 00:15:17.308 --> 00:15:21.992 Making superintelligent A.I. is a really hard challenge. 00:15:21.992 --> 00:15:24.540 Making superintelligent A.I. that is safe 00:15:24.540 --> 00:15:26.956 involves some additional challenge on top of that. 00:15:28.036 --> 00:15:31.523 The risk is that if somebody figures out how to crack the first challenge 00:15:31.523 --> 00:15:34.524 without also having cracked the additional challenge 00:15:34.524 --> 00:15:36.425 of ensuring perfect safety. 00:15:37.195 --> 00:15:40.526 So I think that we should work out a solution 00:15:40.526 --> 00:15:43.348 to the control problem in advance, 00:15:43.348 --> 00:15:46.008 so that we have it available by the time it is needed. 00:15:46.588 --> 00:15:50.095 Now it might be that we cannot solve the entire control problem in advance 00:15:50.095 --> 00:15:53.119 because maybe some elements can only be put in place 00:15:53.119 --> 00:15:57.116 once you know the details of the architecture where it will be implemented. 00:15:57.116 --> 00:16:00.496 But the more of the control problem that we solve in advance, 00:16:00.496 --> 00:16:04.586 the better the odds that the transition to the machine intelligence era 00:16:04.586 --> 00:16:06.126 will go well. 00:16:06.126 --> 00:16:10.770 This to me looks like a thing that is well worth doing 00:16:10.770 --> 00:16:14.102 and I can imagine that if things turn out okay, 00:16:14.102 --> 00:16:18.760 that people a million years from now look back at this century 00:16:18.760 --> 00:16:22.762 and it might well be that they say that the one thing we did that really mattered 00:16:22.762 --> 00:16:24.329 was to get this thing right. 00:16:24.329 --> 00:16:26.018 Thank you. 00:16:26.018 --> 00:16:28.831 (Applause)