WEBVTT
1
00:00:00.240 --> 00:00:03.340 line:15%
Hi, I'm Hilary Mason. I'm a computer scientist.
2
00:00:03.340 --> 00:00:05.660
And today, I've been asked to explain machine learning
3
00:00:05.660 --> 00:00:08.630
in five levels of increasing complexity.
4
00:00:08.630 --> 00:00:11.690
Machine learning gives us the ability to learn things
5
00:00:11.690 --> 00:00:14.550
about the world from large amounts of data
6
00:00:14.550 --> 00:00:18.420
that we as human beings can't possibly study or appreciate.
7
00:00:18.420 --> 00:00:21.070
So machine learning is when we teach computers
8
00:00:21.070 --> 00:00:24.320
to learn patterns from looking at examples in data,
9
00:00:24.320 --> 00:00:26.740
such that they can recognize those patterns
10
00:00:26.740 --> 00:00:30.003
and apply them to new things that they haven't seen before.
11
00:00:30.003 --> 00:00:32.670 line:15%
[playful music]
12
00:00:33.800 --> 00:00:34.633
Hi.
13
00:00:34.633 --> 00:00:35.466
Hi.
14
00:00:35.466 --> 00:00:37.123
I'm Hilary, what's your name?
15
00:00:37.123 --> 00:00:37.956
I'm Brynn.
16
00:00:37.956 --> 00:00:40.420 line:15%
Do you know what machine learning means?
17
00:00:40.420 --> 00:00:41.936
Have you heard that before?
18
00:00:41.936 --> 00:00:42.769
No.
19
00:00:42.769 --> 00:00:45.790
So machine learning is a way that we teach computers
20
00:00:45.790 --> 00:00:49.670
to learn things about the world by looking at patterns
21
00:00:49.670 --> 00:00:51.770
and looking at examples of things.
22
00:00:51.770 --> 00:00:54.400
So can I show you an example
23
00:00:54.400 --> 00:00:56.820
of how a machine might learn something?
24
00:00:56.820 --> 00:00:57.653
Sure.
25
00:00:57.653 --> 00:00:59.150
So is this a dog or a cat?
26
00:00:59.150 --> 00:01:00.580
It's a dog.
27
00:01:00.580 --> 00:01:02.520
And this one?
28
00:01:02.520 --> 00:01:03.353
A cat.
29
00:01:03.353 --> 00:01:07.103
And what makes a dog, a dog and a cat, a cat?
30
00:01:07.103 --> 00:01:12.103
Well, dogs are very playful, I think, more than cats.
31
00:01:12.110 --> 00:01:15.028
Cats lick themselves more than dogs, I think.
32
00:01:15.028 --> 00:01:15.861
That's true.
33
00:01:15.861 --> 00:01:18.660
Do you think, if we look at these pictures,
34
00:01:18.660 --> 00:01:20.457
do you think maybe we could say,
35
00:01:20.457 --> 00:01:22.610
"Well, they both have pointy ears,
36
00:01:22.610 --> 00:01:25.220
but the dogs have a different kind of body
37
00:01:25.220 --> 00:01:27.930
and the cats like to stand up a little different."?
38
00:01:27.930 --> 00:01:29.460
Do you think that makes sense?
39
00:01:29.460 --> 00:01:30.293
Yeah.
Yeah.
40
00:01:30.293 --> 00:01:31.310
What about this one?
41
00:01:31.310 --> 00:01:32.143
A dog.
42
00:01:33.774 --> 00:01:34.607
A cat.
43
00:01:35.671 --> 00:01:37.640
I think, a cat?
44
00:01:37.640 --> 00:01:40.330
Because it's more skinny.
45
00:01:40.330 --> 00:01:44.130
And also, its legs are like really tall
46
00:01:44.130 --> 00:01:46.220
and its ears are a little pointy.
47
00:01:46.220 --> 00:01:49.850
This one's a jackal. And it's actually a kind of dog.
48
00:01:49.850 --> 00:01:51.470
But you made a good guess.
49
00:01:51.470 --> 00:01:54.150
That's what machines do too. They make guesses.
50
00:01:54.150 --> 00:01:56.103
Is this a cat or a dog?
51
00:01:56.103 --> 00:01:56.936
None.
52
00:01:56.936 --> 00:01:57.870
None. What is it?
53
00:01:57.870 --> 00:01:58.980
It's humans.
54
00:01:58.980 --> 00:02:01.540
And how did you know that it's not a cat or a dog?
55
00:02:01.540 --> 00:02:03.563
Because cats and dogs...
56
00:02:03.563 --> 00:02:05.790
Because they walk on their paws
57
00:02:05.790 --> 00:02:08.970
and their ears are like right here, not right here,
58
00:02:08.970 --> 00:02:10.453
and they don't wear watches.
59
00:02:10.453 --> 00:02:13.270
And so, you did something pretty amazing there.
60
00:02:13.270 --> 00:02:16.410
Because we asked the question, "Is it a cat or a dog?"
61
00:02:16.410 --> 00:02:19.780
And you said, "I disagree with your question. It's a human."
62
00:02:19.780 --> 00:02:22.720
So machine learning is when we teach machines
63
00:02:22.720 --> 00:02:25.080
to make guesses about what things are
64
00:02:25.080 --> 00:02:27.780
based on looking at a lot of different examples.
65
00:02:27.780 --> 00:02:30.060
And I build products that use machine learning
66
00:02:30.060 --> 00:02:32.160
to learn about the world and make guesses
67
00:02:32.160 --> 00:02:33.680
about things in the world.
68
00:02:33.680 --> 00:02:36.580
When we try to teach machines to recognize things
69
00:02:36.580 --> 00:02:40.260
like cats and dogs, it takes a lot of examples.
70
00:02:40.260 --> 00:02:43.430
We have to show them tens of thousands
71
00:02:43.430 --> 00:02:45.630
or even millions of examples
72
00:02:45.630 --> 00:02:49.950
before they can get even close to as good at it as you are.
73
00:02:49.950 --> 00:02:51.440
Do you have tests in school?
74
00:02:51.440 --> 00:02:52.890
Yeah, I have.
75
00:02:52.890 --> 00:02:56.820
After every unit, we have a review and then we have a test.
76
00:02:56.820 --> 00:02:58.364
Are those like the practice problems
77
00:02:58.364 --> 00:02:59.890
you do before the test?
78
00:02:59.890 --> 00:03:02.670
Well, just like everything that's gonna be on the test
79
00:03:02.670 --> 00:03:03.830
is on the review.
80
00:03:03.830 --> 00:03:05.320
Which means that in the test,
81
00:03:05.320 --> 00:03:06.690
you're not seeing any problems
82
00:03:06.690 --> 00:03:08.440
that you don't know how to solve.
83
00:03:08.440 --> 00:03:10.753
As long as you did all your practice, right?
84
00:03:10.753 --> 00:03:11.586
Yeah.
85
00:03:11.586 --> 00:03:12.980
So machines work the same way.
86
00:03:12.980 --> 00:03:15.820
If you show them a lot of examples and give them practice,
87
00:03:15.820 --> 00:03:17.930
they'll learn how to guess.
88
00:03:17.930 --> 00:03:19.890
And then when you give them the test,
89
00:03:19.890 --> 00:03:21.860
they should be able to do that.
90
00:03:21.860 --> 00:03:24.280
So we looked at eight pictures
91
00:03:24.280 --> 00:03:26.540
and you were able to answer really quickly.
92
00:03:26.540 --> 00:03:30.580
But what would you do if I gave you 10 million examples?
93
00:03:30.580 --> 00:03:32.623
Would you be able to do that so quickly?
94
00:03:32.623 --> 00:03:33.456
No.
95
00:03:33.456 --> 00:03:36.540
So one of the differences between people and machines
96
00:03:36.540 --> 00:03:40.120
is that people might be a little better at this,
97
00:03:40.120 --> 00:03:42.630
but can't look at 10 million different things.
98
00:03:42.630 --> 00:03:45.090
So now that we've been talking about machine learning,
99
00:03:45.090 --> 00:03:47.940
is this something you want to learn how to do?
100
00:03:47.940 --> 00:03:48.860
Kind of.
101
00:03:48.860 --> 00:03:52.970
Because I kind of want to become a spy.
102
00:03:52.970 --> 00:03:56.990
And we used to do coding,
103
00:03:56.990 --> 00:03:58.950
so I may be kind of good at it.
104
00:03:58.950 --> 00:04:01.540
And machine learning is a great way to use
105
00:04:01.540 --> 00:04:04.330
all those math skills, all those coding skills,
106
00:04:04.330 --> 00:04:07.625
and would be a super cool tool for a spy.
107
00:04:07.625 --> 00:04:10.860 line:15%
[quirky music]
108
00:04:10.860 --> 00:04:11.693
Hello.
109
00:04:12.710 --> 00:04:14.330
Hi. Are you a student, Lucy?
110
00:04:14.330 --> 00:04:16.710 line:15%
Yes, I just finished ninth grade.
111
00:04:16.710 --> 00:04:18.080
Congratulations.
112
00:04:18.080 --> 00:04:19.810
Thank you. It's very exciting.
113
00:04:19.810 --> 00:04:22.090
Have you ever heard of machine learning before?
114
00:04:22.090 --> 00:04:25.870
I'm going to assume that it means humans being able
115
00:04:25.870 --> 00:04:30.870
to teach machines or robots how to learn themselves?
116
00:04:30.920 --> 00:04:31.753
That's right.
117
00:04:31.753 --> 00:04:36.200
When we teach machines to learn from data,
118
00:04:36.200 --> 00:04:40.040
to build a model from that data or a representation of that,
119
00:04:40.040 --> 00:04:41.710
and then to make a prediction.
120
00:04:41.710 --> 00:04:44.280
One of the places we often find machine learning
121
00:04:44.280 --> 00:04:48.120
in the real world is in things like recommendation systems.
122
00:04:48.120 --> 00:04:50.190
So do you have an artist that you really like?
123
00:04:50.190 --> 00:04:51.850
Yeah, Melanie Martinez.
124
00:04:51.850 --> 00:04:54.160
So I'm gonna look up Melanie Martinez.
125
00:04:54.160 --> 00:04:57.610
And it says here, "If you like Melanie Martinez,
126
00:04:57.610 --> 00:05:00.560
one of the other songs you might like is by Au/Ra."
127
00:05:00.560 --> 00:05:01.640
Do you know who that is?
128
00:05:01.640 --> 00:05:02.650
I do not.
129
00:05:02.650 --> 00:05:05.140
So let's listen to a hint of this song.
130
00:05:05.140 --> 00:05:05.973
Okay.
131
00:05:11.224 --> 00:05:14.557
[alternative pop music]
132
00:05:16.030 --> 00:05:16.920
All right.
133
00:05:16.920 --> 00:05:20.050
So why do you think Spotify might've recommended that song?
134
00:05:20.050 --> 00:05:22.470
Well, I know that in Melanie Martinez's music,
135
00:05:22.470 --> 00:05:24.560
she used a lot of the filtered voice
136
00:05:24.560 --> 00:05:27.000
to make it sound very deep and low
137
00:05:27.000 --> 00:05:28.530
and that song had that.
138
00:05:28.530 --> 00:05:30.480
And that's actually a really interesting thing
139
00:05:30.480 --> 00:05:33.330
to think about because that creepy vibe
140
00:05:33.330 --> 00:05:36.910
is something that you can perceive and I can perceive,
141
00:05:36.910 --> 00:05:39.680
but it's actually really hard to describe to a machine.
142
00:05:39.680 --> 00:05:41.630
What do you think might go into that?
143
00:05:41.630 --> 00:05:42.620
Pitch of the music.
144
00:05:42.620 --> 00:05:45.673
If it's really low or if it's super high,
145
00:05:45.673 --> 00:05:46.903
it could know that.
146
00:05:46.903 --> 00:05:49.360
What can the machine understand?
147
00:05:49.360 --> 00:05:50.300
It's a great question.
148
00:05:50.300 --> 00:05:51.870
The machine can understand
149
00:05:51.870 --> 00:05:54.040
whatever we tell it to understand.
150
00:05:54.040 --> 00:05:56.850
So there might be a person thinking about things,
151
00:05:56.850 --> 00:05:59.930
like the pitch or the pacing or the tone,
152
00:05:59.930 --> 00:06:02.160
or sometimes machines can figure out
153
00:06:02.160 --> 00:06:06.010
things about music or images or videos
154
00:06:06.010 --> 00:06:09.150
that we don't tell it to discover,
155
00:06:09.150 --> 00:06:10.610
but that it can learn
156
00:06:10.610 --> 00:06:13.280
from looking at a lot of different examples.
157
00:06:13.280 --> 00:06:15.970
Why do you think companies might use machine learning?
158
00:06:15.970 --> 00:06:19.490
Well, I think things like Facebook or Instagram,
159
00:06:19.490 --> 00:06:22.120
they probably use it to target ads.
160
00:06:22.120 --> 00:06:25.160
Sometimes, the ads you see are really uncanny.
161
00:06:25.160 --> 00:06:29.230
And I think that's because they're based on so much data.
162
00:06:29.230 --> 00:06:32.830
They know where you live. They know where your device is.
163
00:06:32.830 --> 00:06:36.130
It's also important to realize that people in aggregate
164
00:06:36.130 --> 00:06:37.990
are actually pretty predictable.
165
00:06:37.990 --> 00:06:39.290
Like when we talk to each other,
166
00:06:39.290 --> 00:06:41.610
we like to talk about the novel things,
167
00:06:41.610 --> 00:06:43.330
like here, we're having this conversation.
168
00:06:43.330 --> 00:06:44.740
We don't do this every day.
169
00:06:44.740 --> 00:06:46.890
But we probably still eat breakfast.
170
00:06:46.890 --> 00:06:48.540
We're gonna eat lunch. We're gonna eat dinner.
171
00:06:48.540 --> 00:06:50.420
You probably are going to the same home
172
00:06:50.420 --> 00:06:52.110
you go to most of the time.
173
00:06:52.110 --> 00:06:54.560
And so, they're able to take that data
174
00:06:54.560 --> 00:06:57.930
that we already give them and make predictions based on that
175
00:06:57.930 --> 00:07:00.540
as to what ads they should show us.
176
00:07:00.540 --> 00:07:04.920
So, you're saying I give them enough data as it is
177
00:07:04.920 --> 00:07:07.760
about what I might be talking about or thinking about
178
00:07:07.760 --> 00:07:10.474
that they can read my mind,
179
00:07:10.474 --> 00:07:11.307
[Hilary laughs]
180
00:07:11.307 --> 00:07:13.580
but just use the data that I've already given them.
181
00:07:13.580 --> 00:07:16.013
And it almost seems like
182
00:07:16.013 --> 00:07:17.340
they're watching us.
That's right.
183
00:07:17.340 --> 00:07:20.290
To do machine learning, we use something called algorithms.
184
00:07:20.290 --> 00:07:22.260
Have you heard of algorithms before?
185
00:07:22.260 --> 00:07:25.600
A set of steps or a process
186
00:07:25.600 --> 00:07:27.453
carried out to complete something?
187
00:07:27.453 --> 00:07:28.286
That's right.
188
00:07:28.286 --> 00:07:30.070
So do you think that we've been able
189
00:07:30.070 --> 00:07:31.300
to teach machines enough
190
00:07:31.300 --> 00:07:34.660
so that they can do things that even we can't do?
191
00:07:34.660 --> 00:07:36.407
And on the opposite side of that,
192
00:07:36.407 --> 00:07:38.367
do you think there are things that we can do
193
00:07:38.367 --> 00:07:41.360
that a machine might never be able to do?
194
00:07:41.360 --> 00:07:44.350
So there are things that machines are really great at
195
00:07:44.350 --> 00:07:46.410
that humans are actually not great at.
196
00:07:46.410 --> 00:07:50.580
And imagine watching every video posted to TikTok every day.
197
00:07:50.580 --> 00:07:53.570
So we just don't have enough time to do that
198
00:07:53.570 --> 00:07:57.080
at the rate at which we can actually watch those videos.
199
00:07:57.080 --> 00:07:59.430
But a machine can analyze all of them
200
00:07:59.430 --> 00:08:01.630
and then make recommendations to us.
201
00:08:01.630 --> 00:08:05.480
And then thinking about things that machines are bad at
202
00:08:05.480 --> 00:08:08.200
and people are good at, people are really great
203
00:08:08.200 --> 00:08:11.940
with only one or two examples of learning something new
204
00:08:11.940 --> 00:08:14.550
and incorporating that into our model of the world
205
00:08:14.550 --> 00:08:16.130
to make good decisions.
206
00:08:16.130 --> 00:08:19.490
Whereas machines often need tens of thousands of examples,
207
00:08:19.490 --> 00:08:23.280
and that's not even getting into things like good judgment
208
00:08:23.280 --> 00:08:25.010
because we care about people,
209
00:08:25.010 --> 00:08:29.210
because we can imagine a future that we want to live in
210
00:08:29.210 --> 00:08:31.080
that doesn't exist today.
211
00:08:31.080 --> 00:08:33.460
And that's something that is still uniquely human.
212
00:08:33.460 --> 00:08:35.000
Machines are great at predicting
213
00:08:35.000 --> 00:08:36.980
based on what they've seen in the past,
214
00:08:36.980 --> 00:08:38.080
but they're not creative.
215
00:08:38.080 --> 00:08:39.500
They're not going to invent.
216
00:08:39.500 --> 00:08:40.790
They're not gonna, you know,
217
00:08:40.790 --> 00:08:42.530
really change where we're gonna go.
218
00:08:42.530 --> 00:08:43.842
That's up to us.
219
00:08:43.842 --> 00:08:46.425 line:15%
[serene music]
220
00:08:49.880 --> 00:08:51.050 line:15%
I'm Sunny.
221
00:08:51.050 --> 00:08:52.600 line:15%
And what are you majoring in?
222
00:08:52.600 --> 00:08:54.510 line:15%
I study Math and Computer Science.
223
00:08:54.510 --> 00:08:55.980
So in your studies,
224
00:08:55.980 --> 00:08:58.830
have you learned about machine learning?
225
00:08:58.830 --> 00:08:59.778
Yeah, I have.
226
00:08:59.778 --> 00:09:04.778
So to me, machine learning is essentially
227
00:09:04.850 --> 00:09:06.190
exactly what it sounds like.
228
00:09:06.190 --> 00:09:11.190
It's trying to teach a machine specifics about something
229
00:09:11.600 --> 00:09:14.150
by inputting a lot of data points
230
00:09:14.150 --> 00:09:18.930
and slowly, the machine will build up knowledge
231
00:09:18.930 --> 00:09:20.090
about it over time.
232
00:09:20.090 --> 00:09:22.200
For example, my Gmail program,
233
00:09:22.200 --> 00:09:23.890
I assume that there would be a lot of, like,
234
00:09:23.890 --> 00:09:26.750
machine learning models happening at once, right?
235
00:09:26.750 --> 00:09:27.600
Absolutely.
236
00:09:27.600 --> 00:09:30.060
And that's a great example because you have models
237
00:09:30.060 --> 00:09:32.560
that are operating to do things like figure out
238
00:09:32.560 --> 00:09:34.370
if a new email is spam or not.
239
00:09:34.370 --> 00:09:35.870
So what would you think
240
00:09:35.870 --> 00:09:38.110
about if you were looking at an email
241
00:09:38.110 --> 00:09:41.680
and trying to decide if it went in one category or another?
242
00:09:41.680 --> 00:09:43.810
I'd probably look at certain keywords.
243
00:09:43.810 --> 00:09:46.550
Maybe if the recipient and the sender
244
00:09:46.550 --> 00:09:48.490
had exchanged emails before
245
00:09:48.490 --> 00:09:52.320
and generally, those fell into in the past.
246
00:09:52.320 --> 00:09:54.370
So these are things we would call features.
247
00:09:54.370 --> 00:09:57.700
And we go through a process where we do feature engineering,
248
00:09:57.700 --> 00:09:59.957
where somebody looks at the example and says,
249
00:09:59.957 --> 00:10:02.950
"Okay, these are the things that I think might allow us
250
00:10:02.950 --> 00:10:05.220
to statistically tell the difference
251
00:10:05.220 --> 00:10:07.830
from something in one category versus another.
252
00:10:07.830 --> 00:10:10.380
So for example, perhaps you don't speak Russian,
253
00:10:10.380 --> 00:10:12.520
you start getting a lot of email in Russian.
254
00:10:12.520 --> 00:10:14.760
Obviously, like the features that you just described
255
00:10:14.760 --> 00:10:18.210
are features which a person would have had to think about.
256
00:10:18.210 --> 00:10:19.043
Are there features
257
00:10:19.043 --> 00:10:21.830
which, like, the machine itself could learn?
258
00:10:21.830 --> 00:10:23.570
This is a great question
259
00:10:23.570 --> 00:10:26.210
because it really gets to the difference
260
00:10:26.210 --> 00:10:28.840
between some of our different tools
261
00:10:28.840 --> 00:10:30.420
in our machine learning tool belt
262
00:10:30.420 --> 00:10:32.520
in addressing problems like this.
263
00:10:32.520 --> 00:10:37.170
So if we were to use a supervised learning classic
264
00:10:37.170 --> 00:10:38.800
classification approach,
265
00:10:38.800 --> 00:10:41.650
a person would need to think about those features
266
00:10:41.650 --> 00:10:43.930
and creatively come up with them
267
00:10:43.930 --> 00:10:46.030
in approach we call the kitchen sink approach,
268
00:10:46.030 --> 00:10:48.450
which is just try everything you can possibly think of
269
00:10:48.450 --> 00:10:49.560
and see what works.
270
00:10:49.560 --> 00:10:53.040
Unsupervised learning, where we don't have labeled data
271
00:10:53.040 --> 00:10:57.290
and we're trying to infer some structure out of the data
272
00:10:57.290 --> 00:10:59.840
is you're projecting that data into a space
273
00:10:59.840 --> 00:11:02.120
and looking for things like clusters.
274
00:11:02.120 --> 00:11:03.800
And there's a bunch of really fun math
275
00:11:03.800 --> 00:11:07.060
about how you do that, how you think about distance
276
00:11:07.060 --> 00:11:11.290
and by distance, I mean that if we have two data points
277
00:11:11.290 --> 00:11:14.670
in space, how do we decide if they're similar or not?
278
00:11:14.670 --> 00:11:19.320
And how do the algorithms themselves usually differ
279
00:11:19.320 --> 00:11:22.050
between unsupervised and supervised learning.
280
00:11:22.050 --> 00:11:24.500
Supervised learning, we have our labels
281
00:11:24.500 --> 00:11:28.010
and we're trying to figure out what statistically indicates
282
00:11:28.010 --> 00:11:31.790
if something matches one label or another label.
283
00:11:31.790 --> 00:11:32.960
Unsupervised learning,
284
00:11:32.960 --> 00:11:35.330
we don't necessarily have those labels.
285
00:11:35.330 --> 00:11:37.250
That's the thing we're trying to discover.
286
00:11:37.250 --> 00:11:39.730
So reinforcement learning is another technique
287
00:11:39.730 --> 00:11:41.170
that we use sometimes.
288
00:11:41.170 --> 00:11:43.290
You can think about it like a turn in a game
289
00:11:43.290 --> 00:11:47.300
and you can play, you know, millions and millions of trials
290
00:11:47.300 --> 00:11:49.520
so that you're able to develop a system
291
00:11:49.520 --> 00:11:52.310
that by experimenting with reinforcement learning
292
00:11:52.310 --> 00:11:54.440
can eventually learn to play these games
293
00:11:54.440 --> 00:11:55.970
pretty successfully.
294
00:11:55.970 --> 00:11:59.810
Deep learning, which is essentially using neural networks
295
00:11:59.810 --> 00:12:03.630
and very large amounts of data to eventually iterate
296
00:12:03.630 --> 00:12:06.400
on a network structure that can make predictions.
297
00:12:06.400 --> 00:12:09.603
With reinforcement learning versus deep learning,
298
00:12:10.440 --> 00:12:13.730
it seems to me that reinforcement learning,
299
00:12:13.730 --> 00:12:15.600
is it sort of like the kitchen sink approach
300
00:12:15.600 --> 00:12:17.100
that you were talking about earlier,
301
00:12:17.100 --> 00:12:19.690
where you're just kind of trying everything?
302
00:12:19.690 --> 00:12:23.070
It is, but it also thrives in environments
303
00:12:23.070 --> 00:12:25.870
where you have a decision point,
304
00:12:25.870 --> 00:12:28.220
a pallet of actions to choose from.
305
00:12:28.220 --> 00:12:30.670
And it actually comes historically
306
00:12:30.670 --> 00:12:33.680
from trying to train a robot to navigate a room.
307
00:12:33.680 --> 00:12:36.720
If it bonks into this chair, it can't go forward anymore.
308
00:12:36.720 --> 00:12:38.410
And if it falls into that pit,
309
00:12:38.410 --> 00:12:40.640
you know, it's not going to succeed.
310
00:12:40.640 --> 00:12:44.460
But if it keeps exploring, it'll eventually get to the goal.
311
00:12:44.460 --> 00:12:45.480
Oh, like roombas?
312
00:12:45.480 --> 00:12:46.398
Yes.
313
00:12:46.398 --> 00:12:47.324
[both laugh]
314
00:12:47.324 --> 00:12:49.750
Oh, I didn't realize it was that deep, almost.
315
00:12:49.750 --> 00:12:51.830
Is there a situation which you'd want to use
316
00:12:51.830 --> 00:12:52.970
a deep learning algorithm
317
00:12:52.970 --> 00:12:55.500
over a reinforcement learning algorithm?
318
00:12:55.500 --> 00:12:57.800
So typically, you would choose deep learning
319
00:12:57.800 --> 00:13:00.990
if you have sufficient high quality data,
320
00:13:00.990 --> 00:13:03.740
hopefully labeled in a useful way.
321
00:13:03.740 --> 00:13:08.740
If you really are happy not to necessarily understand
322
00:13:09.210 --> 00:13:11.940
or be able to interpret what your system is doing
323
00:13:11.940 --> 00:13:13.380
or you're willing to invest
324
00:13:13.380 --> 00:13:16.920
in another set of work afterwards to understand
325
00:13:16.920 --> 00:13:19.450
what the system is doing once you've already trained it.
326
00:13:19.450 --> 00:13:22.320
And this also comes down to the fact that some things
327
00:13:22.320 --> 00:13:25.650
are actually really easy to solve with linear regression
328
00:13:25.650 --> 00:13:27.890
or simple statistical approaches.
329
00:13:27.890 --> 00:13:29.396
And some things are impossible.
330
00:13:29.396 --> 00:13:31.760
What would be the outcome if you were to choose
331
00:13:31.760 --> 00:13:33.760
the, quote-unquote, "wrong" approach?
332
00:13:33.760 --> 00:13:36.730
You build a system that could actually be useless.
333
00:13:36.730 --> 00:13:40.470
So years ago, I had a client that was a big telecom company
334
00:13:40.470 --> 00:13:42.470
and they had a data scientist
335
00:13:42.470 --> 00:13:45.590
who built a deep learning system to predict customer churn.
336
00:13:45.590 --> 00:13:48.600
It actually was very accurate, but it wasn't useful
337
00:13:48.600 --> 00:13:52.210
because nobody knew why the prediction was what it was.
338
00:13:52.210 --> 00:13:53.707
So they could say, you know,
339
00:13:53.707 --> 00:13:55.820
"Sunny, you're likely to quit next month."
340
00:13:55.820 --> 00:13:58.420
But they had no idea what to do about it.
341
00:13:58.420 --> 00:14:01.040
And so, I think there are a bunch of failure modes.
342
00:14:01.040 --> 00:14:03.310
Would that be an example of, like, linear regression
343
00:14:03.310 --> 00:14:05.780
where the regression is accurate, but,
344
00:14:05.780 --> 00:14:08.040
you know, for marketing purposes, it's like,
345
00:14:08.040 --> 00:14:09.780
if you don't know why I'm quitting the service,
346
00:14:09.780 --> 00:14:11.660
then how can we fix this?
347
00:14:11.660 --> 00:14:12.493
Yeah.
348
00:14:12.493 --> 00:14:15.680
This is actually a good example of a very real world
349
00:14:15.680 --> 00:14:19.840
kind of machine learning problem where the solution to this
350
00:14:19.840 --> 00:14:21.920
was to build an interpretable system
351
00:14:21.920 --> 00:14:24.960
on top of the accurate predictions not to throw it away,
352
00:14:24.960 --> 00:14:27.410
but to do a bunch more work to figure out the why.
353
00:14:27.410 --> 00:14:30.770
How can we improve machine learning algorithms?
354
00:14:30.770 --> 00:14:33.760
It's actually fairly new
355
00:14:33.760 --> 00:14:36.300
that we're able to solve all of these problems
356
00:14:36.300 --> 00:14:39.410
and start to build these products and apply it in businesses
357
00:14:39.410 --> 00:14:41.870
and apply it, you know, everywhere.
358
00:14:41.870 --> 00:14:44.970
And so, we're still developing good practices
359
00:14:44.970 --> 00:14:48.790
and what it means to be a professional in machine learning.
360
00:14:48.790 --> 00:14:51.782
We're really developing a notion of what good looks like.
361
00:14:51.782 --> 00:14:55.110 line:15%
[quirky music]
362
00:14:55.110 --> 00:14:59.490 line:15%
I'm in my first year of a PhD in Computer Science
363
00:14:59.490 --> 00:15:02.170
and I'm studying natural language processing
364
00:15:02.170 --> 00:15:03.100
and machine learning.
365
00:15:03.100 --> 00:15:04.700
So would you mind telling me a bit about
366
00:15:04.700 --> 00:15:07.090
what you've been working on or interested in lately?
367
00:15:07.090 --> 00:15:10.000
I've been looking at understanding persuasion
368
00:15:10.000 --> 00:15:15.000
in online text and the ways that we might be able to
369
00:15:15.030 --> 00:15:18.500
automatically detect the intent behind that persuasion
370
00:15:18.500 --> 00:15:20.040
or who it's targeted at
371
00:15:20.040 --> 00:15:22.670
and what makes effective persuasive techniques.
372
00:15:22.670 --> 00:15:25.060
So what are some of the techniques you're applying
373
00:15:25.060 --> 00:15:26.850
to look at that debate data?
374
00:15:26.850 --> 00:15:28.740
Something I'm interested in exploring
375
00:15:28.740 --> 00:15:32.920
is how well it works to use deep learning
376
00:15:32.920 --> 00:15:36.150
and sort of automatically extracted features from this text
377
00:15:36.150 --> 00:15:38.800
versus using some of the more traditional techniques
378
00:15:38.800 --> 00:15:41.300
that we have, things like lexicons
379
00:15:41.300 --> 00:15:43.580
or some sort of template matching techniques
380
00:15:43.580 --> 00:15:46.470
for extracting features from texts.
381
00:15:46.470 --> 00:15:48.650
That's a question I'm just interested in, in general.
382
00:15:48.650 --> 00:15:50.140
When do we really need deep learning
383
00:15:50.140 --> 00:15:52.100
versus when can we use something
384
00:15:52.100 --> 00:15:53.620
that's a little bit more interpretable,
385
00:15:53.620 --> 00:15:55.150
something that's been around for a while?
386
00:15:55.150 --> 00:15:58.000
Do you think there are going to be general principles
387
00:15:58.000 --> 00:15:59.500
that guide those decisions?
388
00:15:59.500 --> 00:16:01.440
Because right now, it's generally
389
00:16:01.440 --> 00:16:04.280
up to the machine learning engineer to decide
390
00:16:04.280 --> 00:16:05.980
what tools they want to apply.
391
00:16:05.980 --> 00:16:08.270
I definitely think there is,
392
00:16:08.270 --> 00:16:11.570
but I also, sort of, see it varying a lot
393
00:16:11.570 --> 00:16:12.780
based on the use case,
394
00:16:12.780 --> 00:16:15.520
something that, kind of, works out of the box
395
00:16:15.520 --> 00:16:17.770
and maybe works a little bit more automatically
396
00:16:17.770 --> 00:16:18.603
might be better.
397
00:16:18.603 --> 00:16:21.020
And in other cases, you do, sort of, kind of,
398
00:16:21.020 --> 00:16:22.860
you want a lot of fine grain control.
399
00:16:22.860 --> 00:16:25.000
So is that where some of that frustration
400
00:16:25.000 --> 00:16:27.230
around the lack of controllability
401
00:16:27.230 --> 00:16:28.940
and interpretability comes from?
402
00:16:28.940 --> 00:16:30.100
Yeah, if you're building a model
403
00:16:30.100 --> 00:16:31.970
that just predicts the next thing
404
00:16:31.970 --> 00:16:34.790
based off of everything it's seen from texts online,
405
00:16:34.790 --> 00:16:37.350
then yeah, you're really gonna be replicating
406
00:16:37.350 --> 00:16:39.550
whatever that distribution online is.
407
00:16:39.550 --> 00:16:42.760
If you train a model off of language off the internet,
408
00:16:42.760 --> 00:16:45.130
it sometimes says uncomfortable things
409
00:16:45.130 --> 00:16:48.960
or inappropriate things and sometimes really biased things.
410
00:16:48.960 --> 00:16:50.790
Have you ever run into this yourself?
411
00:16:50.790 --> 00:16:52.940
And then how do you think about that problem
412
00:16:52.940 --> 00:16:56.560
of potentially even measuring the bias
413
00:16:56.560 --> 00:16:58.490
in a model that we've trained?
414
00:16:58.490 --> 00:17:01.160
Yeah, it's a really tricky question.
415
00:17:01.160 --> 00:17:03.720
As you said, these models are trained to, sort of, predict
416
00:17:03.720 --> 00:17:05.340
the next sequence of words,
417
00:17:05.340 --> 00:17:06.910
given a certain sequence of words.
418
00:17:06.910 --> 00:17:09.160
So we could start with just, sort of, prompts
419
00:17:09.160 --> 00:17:11.860
like "the woman was" versus "the man was",
420
00:17:11.860 --> 00:17:13.990
and, kind of, pull out common words
421
00:17:13.990 --> 00:17:15.390
that are, sort of, more used
422
00:17:15.390 --> 00:17:17.710
with one phrase versus the other.
423
00:17:17.710 --> 00:17:20.880
So that's, sort of, a qualitative way of looking at it.
424
00:17:20.880 --> 00:17:24.150
It's not ever kind of a guarantee of how the model
425
00:17:24.150 --> 00:17:26.270
is gonna behave in one particular instance.
426
00:17:26.270 --> 00:17:28.570
And I think that's what's really tricky
427
00:17:28.570 --> 00:17:30.610
and that's why I, sort of, think it's really good
428
00:17:30.610 --> 00:17:33.720
for creators of systems to just be honest
429
00:17:33.720 --> 00:17:36.270
about, "This is, sort of, what we have seen."
430
00:17:36.270 --> 00:17:38.677
And so then, someone can make their own judgment about,
431
00:17:38.677 --> 00:17:40.310
"Is this gonna be too high risk
432
00:17:40.310 --> 00:17:42.740
for, sort of, my particular use case?"
433
00:17:42.740 --> 00:17:44.340
I imagine in the last few years,
434
00:17:44.340 --> 00:17:47.070
we've seen a lot of changes and improvements
435
00:17:47.070 --> 00:17:49.960 line:15%
in the capabilities of NLP systems.
436
00:17:49.960 --> 00:17:51.570 line:15%
So is there anything in that
437
00:17:51.570 --> 00:17:54.510
that you're particularly excited about exploring further?
438
00:17:54.510 --> 00:17:59.510
I'm really interested in, sort of, the creative potential
439
00:18:00.280 --> 00:18:02.980
that we've started to see from NLP systems
440
00:18:02.980 --> 00:18:04.490
with things like GPT-3
441
00:18:04.490 --> 00:18:06.210
and other really powerful language models.
442
00:18:06.210 --> 00:18:10.990
It's really easy to write long grammatical passages
443
00:18:10.990 --> 00:18:13.580
thinking about the way that we can then harness, like,
444
00:18:13.580 --> 00:18:16.320
the human ability to actually give meaning to those words
445
00:18:16.320 --> 00:18:17.950
and, sort of, provide structure
446
00:18:17.950 --> 00:18:20.640
and how we can combine those things with the, kind of like,
447
00:18:20.640 --> 00:18:22.930
generative capabilities of these models now
448
00:18:22.930 --> 00:18:23.860
is really interesting.
449
00:18:23.860 --> 00:18:24.900
Yeah, I agree.
450
00:18:24.900 --> 00:18:27.720 line:15%
[quirky music]
451
00:18:27.720 --> 00:18:29.437
So, hi Claudia. It's so great to see you.
452
00:18:29.437 --> 00:18:30.930
It has been far too long.
453
00:18:30.930 --> 00:18:33.500
You know, we first met 10, 11 years ago
454
00:18:33.500 --> 00:18:36.810
and machine learning has changed a lot since then.
455
00:18:36.810 --> 00:18:39.850 line:15%
Tooling that we now have, the capacity,
456
00:18:39.850 --> 00:18:43.520 line:15%
and also, an elevation of the problem sets
457
00:18:43.520 --> 00:18:47.150 line:15%
that we're dealing with and how to frame the problem.
458
00:18:47.150 --> 00:18:50.730
And I'm almost struggling to figure out
459
00:18:50.730 --> 00:18:54.670
whether it's a blessing or curse that it has become
460
00:18:54.670 --> 00:18:59.670
as accessible and as democratized and as easy to execute
461
00:18:59.830 --> 00:19:03.000
and you just build another new company from scratch.
462
00:19:03.000 --> 00:19:05.450
And so, what's been, kind of, your reflection on that?
463
00:19:05.450 --> 00:19:07.890
Well, you're absolutely right that the attention
464
00:19:07.890 --> 00:19:10.970
machine learning gets has grown dramatically.
465
00:19:10.970 --> 00:19:12.730
20 years ago, going to gatherings
466
00:19:12.730 --> 00:19:14.420
and telling people what I was working on
467
00:19:14.420 --> 00:19:16.040
and seeing the blank face or the like,
468
00:19:16.040 --> 00:19:18.030
"Where's the turn?" and walk away.
469
00:19:18.030 --> 00:19:19.570
Like, "Oh, no."
470
00:19:19.570 --> 00:19:21.130
The accessibility of the tooling,
471
00:19:21.130 --> 00:19:24.670
like, we can now do in, like, five lines of code
472
00:19:24.670 --> 00:19:27.510
something that would have taken 500 lines
473
00:19:27.510 --> 00:19:30.180
of very mathematical, messy, gnarly code
474
00:19:30.180 --> 00:19:32.150
even, you know, five years ago.
475
00:19:32.150 --> 00:19:33.750
And that's not an exaggeration.
476
00:19:33.750 --> 00:19:36.430
And there are tools that mean that pretty much anyone
477
00:19:36.430 --> 00:19:38.690
can pick this up and start playing with it
478
00:19:38.690 --> 00:19:40.170
and start to build with it.
479
00:19:40.170 --> 00:19:42.530
And that is also really exciting.
480
00:19:42.530 --> 00:19:44.850
In contrast, what I'm struggling with,
481
00:19:44.850 --> 00:19:46.400
the friend of mine who asked me
482
00:19:46.400 --> 00:19:49.080
to look at some health care data for him.
483
00:19:49.080 --> 00:19:52.020
And despite the capabilities that we're having
484
00:19:52.020 --> 00:19:55.180
in all of the, kind of, bigger societal problems
485
00:19:55.180 --> 00:19:58.480
alongside with data collection engineering,
486
00:19:58.480 --> 00:19:59.960
all the gnarly stuff,
487
00:19:59.960 --> 00:20:02.960
that is actually not the machine learning itself,
488
00:20:02.960 --> 00:20:07.020
it's the rest of it where certain data isn't available.
489
00:20:07.020 --> 00:20:10.140
And to me, it's staggering how difficult it is
490
00:20:10.140 --> 00:20:12.692
to get it off the ground and actually use.
491
00:20:12.692 --> 00:20:15.480
And part of the challenge of it
492
00:20:15.480 --> 00:20:18.060
is not the mathematics of building models,
493
00:20:18.060 --> 00:20:20.210
but the challenge is making sure that the data
494
00:20:20.210 --> 00:20:23.660
is sufficiently representative, potentially high quality.
495
00:20:23.660 --> 00:20:25.880
And how transparent do I need to build it
496
00:20:25.880 --> 00:20:28.420
for it to be adopted at some point?
497
00:20:28.420 --> 00:20:32.900
What types of biases in the data collection,
498
00:20:32.900 --> 00:20:34.930
and then also in the usage?
499
00:20:34.930 --> 00:20:37.520
We now call it the bias, but we're still struggling
500
00:20:37.520 --> 00:20:41.880
with the society not really living up to its expectations
501
00:20:41.880 --> 00:20:44.490
and then machine learning, bringing it to the forefront.
502
00:20:44.490 --> 00:20:45.323
Right.
503
00:20:45.323 --> 00:20:46.460
And so, to say that another way,
504
00:20:46.460 --> 00:20:48.920
when you're collecting data from the real world
505
00:20:48.920 --> 00:20:50.670
and then building machine learning systems
506
00:20:50.670 --> 00:20:52.770
that automate decisions based on that data,
507
00:20:52.770 --> 00:20:55.110
all of the biases and problems
508
00:20:55.110 --> 00:20:58.440
that are already in the real world then can be magnified
509
00:20:58.440 --> 00:21:00.290
through that machine learning system.
510
00:21:00.290 --> 00:21:02.770
And so, it can make many of these problems much worse.
511
00:21:02.770 --> 00:21:05.220
Feeling increasingly challenged
512
00:21:05.220 --> 00:21:09.150
that my skillset of being very good at programming
513
00:21:09.150 --> 00:21:10.950
has become somewhat secondary.
514
00:21:10.950 --> 00:21:11.936
And it's feeling...
515
00:21:11.936 --> 00:21:12.793
[both laugh]
516
00:21:12.793 --> 00:21:15.790
It's really the bigger picture understanding
517
00:21:15.790 --> 00:21:17.570
of "Who would be using that?
518
00:21:17.570 --> 00:21:19.650
How transparent do I need to build it
519
00:21:19.650 --> 00:21:21.970
for it to be adopted at some point?
520
00:21:21.970 --> 00:21:26.420
What types of biases in the data collection
521
00:21:26.420 --> 00:21:28.050
and then also in the usage?"
522
00:21:28.050 --> 00:21:32.090
I think, in certain areas, we have societal expectations
523
00:21:32.090 --> 00:21:35.040
as to what is fair and what isn't.
524
00:21:35.040 --> 00:21:39.560
And so, it's not just the provenance of that data,
525
00:21:39.560 --> 00:21:41.307
but it's, sort of, deeply understanding,
526
00:21:41.307 --> 00:21:43.160
"Why does it look the way it looks?
527
00:21:43.160 --> 00:21:44.440
Why was it collected this way?
528
00:21:44.440 --> 00:21:46.110
What are the limitations of it?"
529
00:21:46.110 --> 00:21:47.200
We need to think about that
530
00:21:47.200 --> 00:21:51.460
in entire process, how we document that process.
531
00:21:51.460 --> 00:21:53.010
This is an issue in companies
532
00:21:53.010 --> 00:21:55.150
where somebody might create something
533
00:21:55.150 --> 00:21:57.470
that even their peers can't recreate.
534
00:21:57.470 --> 00:22:00.800
What have you seen in terms of which industries,
535
00:22:00.800 --> 00:22:03.610
where they stand, like who is adopting now?
536
00:22:03.610 --> 00:22:06.550
Who is ready to utilize it?
537
00:22:06.550 --> 00:22:09.158
Where would you maybe wish that didn't even try?
538
00:22:09.158 --> 00:22:10.330
[Hilary laughs]
539
00:22:10.330 --> 00:22:11.550
These are great questions.
540
00:22:11.550 --> 00:22:14.960
So things like actuarial science, operations research,
541
00:22:14.960 --> 00:22:17.240
where they actually are not using machine learning
542
00:22:17.240 --> 00:22:18.560
as much as you might think.
543
00:22:18.560 --> 00:22:21.460
And then you have other sorts of companies
544
00:22:21.460 --> 00:22:24.930
or on the FinTech side, or even the ad tech side of things
545
00:22:24.930 --> 00:22:27.810
where they perhaps are using machine learning
546
00:22:27.810 --> 00:22:30.030
to the point of even absurdity.
547
00:22:30.030 --> 00:22:33.930
So I spent about eight years working in ad tech.
548
00:22:33.930 --> 00:22:37.280
And the motivation was really
549
00:22:37.280 --> 00:22:41.330
because it was such an amazingly exciting playground
550
00:22:41.330 --> 00:22:43.720
to push that technology
551
00:22:43.720 --> 00:22:47.053
that used to largely live in academia, really,
552
00:22:47.920 --> 00:22:50.820
out in the world and see, kind of, what it can achieve.
553
00:22:50.820 --> 00:22:53.870
It has created such a hunger for data
554
00:22:53.870 --> 00:22:56.890
that now everything is being collected.
555
00:22:56.890 --> 00:22:59.660
I'm curious, when are we going to
556
00:23:01.080 --> 00:23:03.820
make a foray into things like agriculture
557
00:23:03.820 --> 00:23:08.420
about smart production of the things we eat?
558
00:23:08.420 --> 00:23:10.440
You see and hear these interesting stories,
559
00:23:10.440 --> 00:23:13.030
but I feel like we're not ready yet
560
00:23:13.030 --> 00:23:17.680
to put that into a economically viable situation.
561
00:23:17.680 --> 00:23:20.900
So when we think about the next five to 10 years,
562
00:23:20.900 --> 00:23:24.030
the things that are really still holding us back
563
00:23:24.030 --> 00:23:29.030
are these uneven applications of resources to problems
564
00:23:30.040 --> 00:23:31.850
because the problems that get attention
565
00:23:31.850 --> 00:23:33.280
are the high value ones
566
00:23:33.280 --> 00:23:35.430
in terms of how much money you can make
567
00:23:35.430 --> 00:23:37.040
or the things that are fashionable enough
568
00:23:37.040 --> 00:23:38.760
that you can publish a paper on it.
569
00:23:38.760 --> 00:23:40.890
So what do you think is holding us back?
570
00:23:40.890 --> 00:23:44.370
I fully agree on the steps you pointed out
571
00:23:44.370 --> 00:23:45.910
and the processes.
572
00:23:45.910 --> 00:23:48.130
I think there is a chicken and egg problem,
573
00:23:48.130 --> 00:23:49.430
like your former example,
574
00:23:49.430 --> 00:23:54.430
that these areas that need to wait for data,
575
00:23:54.740 --> 00:23:56.530
the value of the data collection
576
00:23:56.530 --> 00:23:59.470
is then also slightly less apparent.
577
00:23:59.470 --> 00:24:01.040
And so, it gets delayed further
578
00:24:01.040 --> 00:24:02.890
and you'll see that happening.
579
00:24:02.890 --> 00:24:05.500
But what my experience has been,
580
00:24:05.500 --> 00:24:09.110
there's unfortunately, I feel a drifting apart
581
00:24:09.110 --> 00:24:10.820
between academia
582
00:24:10.820 --> 00:24:14.120
and the uses of AI,
583
00:24:14.120 --> 00:24:19.120
but I am somewhat frustrated with a generation of students
584
00:24:19.290 --> 00:24:22.860
who have standard data sets that they never think about
585
00:24:22.860 --> 00:24:25.190
what the model needs to be used for,
586
00:24:25.190 --> 00:24:26.740
that they never have to think about
587
00:24:26.740 --> 00:24:28.360
how the data was collected.
588
00:24:28.360 --> 00:24:31.180
So with all of these challenges ahead of us,
589
00:24:31.180 --> 00:24:34.330
how optimistic are you about this world
590
00:24:34.330 --> 00:24:38.110
that I deeply believe we can create
591
00:24:38.110 --> 00:24:40.500
and the steps towards it?
592
00:24:40.500 --> 00:24:43.440
I am incredibly optimistic and not...
593
00:24:43.440 --> 00:24:46.980
Perhaps it's a personality flaw, but I can't help but look
594
00:24:46.980 --> 00:24:51.980
at the potential of the technology to reduce harm,
595
00:24:52.250 --> 00:24:55.460
to give us information that help us make better decisions.
596
00:24:55.460 --> 00:24:57.770
And to think that we would choose
597
00:24:57.770 --> 00:25:01.100
to address the big problems ahead of us.
598
00:25:01.100 --> 00:25:03.350
I don't think we have a hope of addressing them
599
00:25:03.350 --> 00:25:05.010
without figuring out the role
600
00:25:05.010 --> 00:25:06.600
that machine learning will play.
601
00:25:06.600 --> 00:25:09.320
And to think that we would then choose not to do that
602
00:25:09.320 --> 00:25:10.700
is just unthinkable.
603
00:25:10.700 --> 00:25:14.950
Despite that the rightfully raised concerns
604
00:25:14.950 --> 00:25:16.210
about the challenges ahead,
605
00:25:16.210 --> 00:25:20.300
but I think they also make us as a society better.
606
00:25:20.300 --> 00:25:23.260
They challenge us to be a lot clearer
607
00:25:23.260 --> 00:25:26.940
of what fairness means to all of us.
608
00:25:26.940 --> 00:25:30.440
So with all of the setbacks,
609
00:25:30.440 --> 00:25:33.840
I think we have exciting years to come.
610
00:25:33.840 --> 00:25:37.810
And I am looking forward to a world where a lot more of that
611
00:25:37.810 --> 00:25:40.288
is used for the right purposes.
612
00:25:40.288 --> 00:25:43.455
[gentle upbeat music]
613
00:25:45.450 --> 00:25:48.410
I hope you learned something about machine learning.
614
00:25:48.410 --> 00:25:51.570
There has never been a better time to study machine learning
615
00:25:51.570 --> 00:25:54.160
because you're now able to build products
616
00:25:54.160 --> 00:25:57.210
that have tremendous potential and impact
617
00:25:57.210 --> 00:26:01.323
across any industry or area that you might be excited about.