Drinking from the AI Firehose at the DC AWS Summit

1
00:00:00,410 --> 00:00:02,750
Ned: Other than your mic stand, everything else okay [laugh]

2
00:00:03,240 --> 00:00:03,660
?
Chris: Yeah.

3
00:00:03,960 --> 00:00:06,090
As usual, with the things that I observe, this is something

4
00:00:06,090 --> 00:00:08,550
that I figured out, like, years ago but never put a name on.

5
00:00:08,920 --> 00:00:09,700
Ned: Mmm, okay.

6
00:00:09,910 --> 00:00:11,659
Chris: So, let’s start with the obvious.

7
00:00:11,910 --> 00:00:13,379
Coke is clearly better than Pepsi.

8
00:00:13,900 --> 00:00:14,899
It’s not a conversation.

9
00:00:14,900 --> 00:00:16,070
Put your finger back down.

10
00:00:16,070 --> 00:00:16,580
Ned: [laugh] . Okay?

11
00:00:16,690 --> 00:00:16,990
Wha—

12
00:00:17,710 --> 00:00:17,879
Chris: No, no, no.

13
00:00:17,889 --> 00:00:18,930
Don’t you okay me.

14
00:00:18,970 --> 00:00:19,520
Ned: [sputters in the background]

15
00:00:19,520 --> 00:00:19,600
.
Chris: Hey.

16
00:00:19,680 --> 00:00:21,870
Ned: Diet Pepsi is better than Diet Coke.

17
00:00:22,140 --> 00:00:23,840
Chris: First of all, they’re both bad.

18
00:00:24,140 --> 00:00:24,510
Ned: Fair.

19
00:00:24,870 --> 00:00:29,050
Chris: The only decent diet soda on Earth is Diet Dr.

20
00:00:29,050 --> 00:00:29,420
Pepper.

21
00:00:29,660 --> 00:00:30,300
There I said it.

22
00:00:30,750 --> 00:00:33,610
Ned: Ohh, I agree that it’s good.

23
00:00:34,129 --> 00:00:35,889
Moving on, what was your actual point?

24
00:00:36,219 --> 00:00:37,160
Chris: I don’t remember.

25
00:00:38,450 --> 00:00:38,480
Ned: [laugh]

26
00:00:38,480 --> 00:00:39,810
.
Chris: The other thing about Diet Dr.

27
00:00:39,810 --> 00:00:41,360
Pepper is it’s got two-and-a-half times the

28
00:00:41,360 --> 00:00:43,460
caffeine, so this episode is going to go fast.

29
00:00:49,440 --> 00:00:49,470
Ned: [laugh]

30
00:00:53,269 --> 00:00:55,475
. Hello, alleged human, and welcome to the Chaos Lever podcast.

31
00:00:56,020 --> 00:00:58,529
My name is Ned and I’m definitely not a robot.

32
00:00:58,560 --> 00:01:01,290
I am a real human person who has feelings, dreams,

33
00:01:01,590 --> 00:01:06,160
and an ordinate desire to consume caffeine and sugar.

34
00:01:06,590 --> 00:01:07,700
Like, all the sugar.

35
00:01:07,950 --> 00:01:10,330
Give me all the sugars that you have.

36
00:01:10,330 --> 00:01:12,660
I will suck them out of your blood, thank you very much.

37
00:01:13,010 --> 00:01:16,590
With me is Chris, who is also full of delicious sugar.

38
00:01:16,590 --> 00:01:17,500
Hi, Chris.

39
00:01:17,880 --> 00:01:20,570
Chris: One has to assume that vampires have an opinion.

40
00:01:21,130 --> 00:01:23,590
Ned: An opinion on blood types?

41
00:01:23,770 --> 00:01:24,860
Blood sugar levels?

42
00:01:25,110 --> 00:01:25,550
Chris: Yeah.

43
00:01:25,570 --> 00:01:27,260
So, think about beef, right?

44
00:01:27,260 --> 00:01:31,150
We know there’s a difference between corn-fed and grain-fed, right?

45
00:01:31,940 --> 00:01:34,430
Ned: [laugh] . I believe you mean grass-fed, but moving on.

46
00:01:34,800 --> 00:01:35,940
Chris: Is grass not a grain?

47
00:01:36,230 --> 00:01:39,110
Ned: Oh, I thought you said brain-fed, and I

48
00:01:39,110 --> 00:01:42,069
was like, not aware of any cows eating brains.

49
00:01:42,099 --> 00:01:44,260
Isn’t that how we got the mad cow disease?

50
00:01:44,580 --> 00:01:45,230
Chris: Yes.

51
00:01:45,740 --> 00:01:46,359
Ned: It is.

52
00:01:46,869 --> 00:01:50,140
Okay, so yes, corn-fed versus grain-fed versus

53
00:01:50,140 --> 00:01:52,899
grass-fed, there is a definitive difference.

54
00:01:53,230 --> 00:01:53,530
Chris: Right.

55
00:01:53,600 --> 00:01:57,810
So, if you were eating either human flesh or drinking human

56
00:01:57,810 --> 00:02:01,949
blood—or both I’m not here to shame anyone’s kink—you’re going to

57
00:02:01,960 --> 00:02:05,489
be interested in that person’s, let’s call them the donor’s diet.

58
00:02:06,020 --> 00:02:06,729
Ned: Absolutely.

59
00:02:06,900 --> 00:02:10,930
Chris: That’s why—I’m almost positive this is correct—on the website for the Red

60
00:02:10,930 --> 00:02:14,220
Cross, they say you should do five or six shots of whiskey before you donate,

61
00:02:14,520 --> 00:02:17,137
so that, you know, when somebody gets your blood, it’s a little pick me up.

62
00:02:17,259 --> 00:02:19,510
Ned: [laugh] . Settle down.

63
00:02:20,440 --> 00:02:22,060
I was wondering where you’re going with that.

64
00:02:22,299 --> 00:02:23,900
And it turned out it wasn’t anywhere good.

65
00:02:24,289 --> 00:02:26,700
But hey, speaking of consuming entirely

66
00:02:26,700 --> 00:02:29,400
too much stuff, guess what I did this week?

67
00:02:29,940 --> 00:02:31,130
Chris: Went somewhere that wasn’t good?

68
00:02:31,900 --> 00:02:34,050
Ned: Oh, I went somewhere that was fine, but I consumed

69
00:02:34,059 --> 00:02:36,640
entirely too much of a thing, and that thing was AI.

70
00:02:39,000 --> 00:02:39,010
Wooo.

71
00:02:39,049 --> 00:02:39,349
Chris: Yay.

72
00:02:39,980 --> 00:02:40,090
Ned: Ugh.

73
00:02:40,820 --> 00:02:44,520
So, this past week, I attended the AWS Summit in our

74
00:02:44,520 --> 00:02:48,100
nation’s capitol, and home to a really good taco place.

75
00:02:48,490 --> 00:02:50,490
Seriously, it’s called Rebel Taco.

76
00:02:50,970 --> 00:02:53,389
It’s on the corner of 6th and K.

77
00:02:53,980 --> 00:02:56,930
It’s a less than five-minute walk from the Walter E.

78
00:02:56,940 --> 00:02:58,670
Washington Convention Center.

79
00:02:58,920 --> 00:03:03,460
So, if you happen to be at that convention center: Rebel Taco.

80
00:03:03,790 --> 00:03:06,040
Walk the, like, four blocks, it’s worth it.

81
00:03:06,730 --> 00:03:08,190
I got three different tacos.

82
00:03:08,200 --> 00:03:11,530
They were all delicious, especially the lamb birria.

83
00:03:11,530 --> 00:03:12,929
That was really good.

84
00:03:13,329 --> 00:03:18,390
And the margarita I got to go with my tacos was absolutely not fucking around.

85
00:03:18,740 --> 00:03:23,810
Ohh, somebody had a very heavy hand at Rebel Taco, and I didn’t mind one bit.

86
00:03:24,440 --> 00:03:27,180
Chris: Unrelated, I saw something interesting slash terrifying.

87
00:03:27,270 --> 00:03:27,690
Ned: Okay.

88
00:03:27,920 --> 00:03:31,099
Chris: The new summer drink is a pickled juice-based margarita.

89
00:03:31,660 --> 00:03:32,380
Ned: Thanks, I hate it.

90
00:03:33,370 --> 00:03:34,509
Chris: Anyway, enjoy your weekend.

91
00:03:35,170 --> 00:03:38,359
Ned: So, when I was 15—of course I have a story for

92
00:03:38,360 --> 00:03:41,490
this, right?—when I was 15, I worked at a pizzeria.

93
00:03:41,870 --> 00:03:44,230
And I also had acne because I was 15.

94
00:03:44,610 --> 00:03:44,880
Chris: Right.

95
00:03:44,910 --> 00:03:45,600
It’s like the rule.

96
00:03:46,030 --> 00:03:48,880
Ned: And working in a pizzeria was definitely not going to this.

97
00:03:48,880 --> 00:03:50,799
The amount of grease in the air—Anyway.

98
00:03:51,059 --> 00:03:58,600
So, the owner and proprietor of said pizzeria told me that a cure for

99
00:03:58,610 --> 00:04:03,720
acne—or at least a treatment was drinking pickle juice—and being as

100
00:04:03,720 --> 00:04:07,920
we were a pizzeria that made other things, we had giant pickle jars,

101
00:04:07,929 --> 00:04:10,890
so she just poured me some pickle juice, and she was like down it.

102
00:04:10,959 --> 00:04:12,090
This will help with your acne.

103
00:04:12,320 --> 00:04:12,870
It did not—

104
00:04:13,090 --> 00:04:13,835
Chris: No, no it wouldn’t.

105
00:04:13,835 --> 00:04:15,450
Ned: —but she got to watch me drink pickle

106
00:04:15,450 --> 00:04:18,230
juice, which I think is, like, 99% of it.

107
00:04:18,470 --> 00:04:18,860
Chris: Mm-hm.

108
00:04:18,990 --> 00:04:19,340
Ned: Yeah.

109
00:04:19,670 --> 00:04:22,370
Chris: That was a game of watch the 15-year-old do something dumb.

110
00:04:22,730 --> 00:04:23,059
Ned: Mm-hm.

111
00:04:23,559 --> 00:04:24,760
That seemed to happen a lot.

112
00:04:24,969 --> 00:04:24,999
Chris: [laugh]

113
00:04:25,299 --> 00:04:26,380
.
Ned: Sometimes unprompted.

114
00:04:26,719 --> 00:04:27,759
Often unprompted.

115
00:04:28,070 --> 00:04:29,370
Almost always unprompted.

116
00:04:29,549 --> 00:04:31,240
Chris: One day, you’ll stop playing that game.

117
00:04:31,460 --> 00:04:38,060
Ned: [laugh] . When I’m dead, Chris [laugh] . Anyway, so I will not

118
00:04:38,060 --> 00:04:42,159
be drinking this cocktail, but I will be talking about AI for a while.

119
00:04:42,889 --> 00:04:46,440
We’re going to be drinking from the AI firehose, which ironically, is an

120
00:04:46,449 --> 00:04:50,650
AWS service that didn’t come up in all of the presentations that I attended.

121
00:04:51,050 --> 00:04:55,000
For those not familiar with tacos, that’s when you take a corn tortilla,

122
00:04:55,380 --> 00:04:59,670
and you place some toppings inside the tortilla, and then you fold it up—

123
00:04:59,670 --> 00:05:00,023
Chris: [coughs significantly]

124
00:05:00,200 --> 00:05:03,509
Ned: And then—wait, okay, no, wait, we were talking

125
00:05:03,509 --> 00:05:06,259
about the AWS Summit, and not delicious food.

126
00:05:06,360 --> 00:05:07,740
Sorry, I got sidetracked.

127
00:05:07,750 --> 00:05:10,180
It is lunchtime, to be fair.

128
00:05:10,700 --> 00:05:14,710
So, let me talk about the amazing French toast that I had at the Delegate.

129
00:05:15,070 --> 00:05:18,329
It was a Challah-crusted French toast.

130
00:05:19,469 --> 00:05:19,489
Chris: [sigh]

131
00:05:19,489 --> 00:05:22,390
.
Ned: The crusting was cornflakes and—stop it, Chris.

132
00:05:22,390 --> 00:05:23,229
All right, fine.

133
00:05:23,240 --> 00:05:24,440
I’m not procrastinating.

134
00:05:24,440 --> 00:05:25,000
You are.

135
00:05:25,640 --> 00:05:29,040
For those not familiar with the AWS Summits, these

136
00:05:29,050 --> 00:05:33,000
are many events put on by AWS all around the globe.

137
00:05:33,340 --> 00:05:35,719
They’re typically one or two days, and it’s kind

138
00:05:35,719 --> 00:05:39,130
of—you can think of it as a speed-run of re:Invent.

139
00:05:39,469 --> 00:05:41,179
And you don’t have to go to Las Vegas.

140
00:05:41,420 --> 00:05:42,989
So, that’s a gigantic win.

141
00:05:43,560 --> 00:05:47,890
I’ve attended the NYC Summit three or four times, and really enjoyed it.

142
00:05:48,520 --> 00:05:51,860
In fact, it’s happening again in a few weeks, and I plan to go.

143
00:05:52,270 --> 00:05:55,710
But I also wanted to try out the DC Summit since I figured it

144
00:05:55,710 --> 00:05:59,379
would have a different focus and attract a slightly different

145
00:05:59,379 --> 00:06:03,659
crowd, by which I mean public sector, government, and military.

146
00:06:03,990 --> 00:06:07,260
It was interesting to see the different types of vendors this

147
00:06:07,260 --> 00:06:11,300
attracted at the Expo, and how the sessions focused on public

148
00:06:11,300 --> 00:06:15,149
sector, and intelligence applications instead of building the

149
00:06:15,150 --> 00:06:19,330
next Uber for Tacos, a company that I would absolutely invest in.

150
00:06:19,740 --> 00:06:19,980
Chris: Yeah.

151
00:06:20,320 --> 00:06:22,790
Let’s talk about that instead of AI.

152
00:06:23,610 --> 00:06:24,200
Ned: [laugh] . We probably could.

153
00:06:24,600 --> 00:06:25,030
But no.

154
00:06:25,280 --> 00:06:28,420
While I was browsing through the sessions, trying to plan

155
00:06:28,420 --> 00:06:31,690
out my schedule for the two days, I couldn’t help but notice

156
00:06:31,690 --> 00:06:34,539
how many of the sessions had AI somewhere in the title.

157
00:06:34,549 --> 00:06:38,200
I actually talked to an AWS person the following day, and they

158
00:06:38,210 --> 00:06:44,959
said that 55% of the titles of the sessions had AI in them.

159
00:06:45,440 --> 00:06:47,450
Even more of them were about AI, but that was

160
00:06:47,450 --> 00:06:49,500
just the ones that had it directly in the title.

161
00:06:50,059 --> 00:06:54,629
Chris: Considering there are 380-something services,

162
00:06:55,160 --> 00:06:58,940
there’s no possible way that 55% of the services need AI.

163
00:06:59,290 --> 00:07:00,190
Ned: [laugh] . No.

164
00:07:00,440 --> 00:07:01,120
Chris: Explain.

165
00:07:01,520 --> 00:07:05,690
Ned: [sigh] . Well, at first, I was just trying to avoid those sessions.

166
00:07:05,950 --> 00:07:11,360
So, I was trying to pick stuff like running EKS clusters, or

167
00:07:12,199 --> 00:07:15,810
AWS networking at the edge, or scaling storage best practices.

168
00:07:15,820 --> 00:07:20,990
You know, things that I think of as, quote, “Real topics.”

169
00:07:21,360 --> 00:07:23,030
Chris: Which, ironically, would include the

170
00:07:23,030 --> 00:07:25,520
responsible and practical use of Firehose.

171
00:07:25,820 --> 00:07:26,400
Ned: Indeed.

172
00:07:26,740 --> 00:07:27,810
Chris: See, I know things.

173
00:07:28,240 --> 00:07:28,889
Ned: I know you do.

174
00:07:29,549 --> 00:07:32,600
Once I finished trying to assemble this schedule, I

175
00:07:32,600 --> 00:07:36,229
looked at it and I realized it was kind of sad and boring.

176
00:07:36,380 --> 00:07:39,020
Like, say, your dad’s old record collection.

177
00:07:39,290 --> 00:07:40,870
Chris: There’s a lot of Boz Scaggs.

178
00:07:40,870 --> 00:07:44,780
Ned: It’s all stuff that’s been around for a decade or two, feels comfortable

179
00:07:44,780 --> 00:07:48,669
and safe, and it’s really just coasting along, playing the greatest hits.

180
00:07:49,060 --> 00:07:51,530
It’s the Eagles of sessions.

181
00:07:52,050 --> 00:07:54,480
So, I decided that I needed to shake things up a bit.

182
00:07:55,000 --> 00:07:57,119
Kids are talking about this new AI stuff.

183
00:07:57,510 --> 00:07:59,750
I don’t want to be the cranky old bastard in the back

184
00:07:59,750 --> 00:08:02,909
ranting about how no one makes good technology anymore.

185
00:08:03,410 --> 00:08:08,260
I need to keep my mind open and attend these weird AI sessions.

186
00:08:09,080 --> 00:08:13,770
Chris, are you familiar with Cognitive Behavioral Therapy: CBT?

187
00:08:14,110 --> 00:08:16,000
Chris: Yeah, that’s what CBT Nuggets is about, right?

188
00:08:16,460 --> 00:08:16,930
Ned: Is it?

189
00:08:17,300 --> 00:08:17,700
No.

190
00:08:17,710 --> 00:08:19,879
Is that really what CBT stands for?

191
00:08:20,389 --> 00:08:20,729
Chris: No.

192
00:08:21,009 --> 00:08:21,919
Ned: I don’t know, man.

193
00:08:21,950 --> 00:08:23,550
I never watched any of that crap.

194
00:08:23,550 --> 00:08:24,470
Chris: [laugh]

195
00:08:25,390 --> 00:08:28,110
.
Ned: [laugh] . It wasn’t crap, but I also didn’t watch it.

196
00:08:28,450 --> 00:08:29,409
Chris: I’m sending them an email.

197
00:08:29,410 --> 00:08:30,120
You’re in trouble.

198
00:08:30,670 --> 00:08:33,140
Ned: I didn’t know they were still going concern, to be honest.

199
00:08:33,679 --> 00:08:37,380
A core premise of CBT is to face your fears

200
00:08:37,530 --> 00:08:39,969
instead of avoiding them, so that’s what I did.

201
00:08:40,369 --> 00:08:44,720
Every single session I attended was AI-related.

202
00:08:44,850 --> 00:08:46,990
I think I went to about ten sessions.

203
00:08:47,350 --> 00:08:49,670
Every single one, AI-related.

204
00:08:50,110 --> 00:08:56,200
No running to the sweet and simple embrace of EC2 instances, or VPC design.

205
00:08:56,450 --> 00:08:59,850
It was going to be mainlining AI for two days.

206
00:08:59,859 --> 00:09:02,790
I have to say, I am glad I did it.

207
00:09:02,800 --> 00:09:07,979
And that’s what I want to share today: what I learned over the two days that

208
00:09:07,980 --> 00:09:12,590
I was there, how I’m feeling about AI now, and what the future looks like.

209
00:09:13,280 --> 00:09:13,990
So, let’s get into it.

210
00:09:14,370 --> 00:09:15,280
Chris: Okay.

211
00:09:16,059 --> 00:09:18,760
Ned: We’ll start with the keynote, which makes sense, right?

212
00:09:19,100 --> 00:09:21,290
Supposed to be the beginning or, you know, the

213
00:09:21,290 --> 00:09:25,780
main thrust and theme of a general conference.

214
00:09:26,300 --> 00:09:27,970
You can watch the keynote if you’d like.

215
00:09:27,980 --> 00:09:32,690
It was presented by Federal CTO at AWS, Dave Levy.

216
00:09:33,110 --> 00:09:35,689
No relation to Eugene Levy, unfortunately.

217
00:09:36,610 --> 00:09:38,160
Two quick things about the keynote.

218
00:09:38,390 --> 00:09:42,350
One, the keynote actually happened on day one at 11 a.m.

219
00:09:42,490 --> 00:09:45,020
after sessions had been going on for two hours.

220
00:09:45,349 --> 00:09:47,839
So, I had already attended a couple of sessions,

221
00:09:47,870 --> 00:09:49,780
but I’m going to start with the keynote anyway.

222
00:09:50,160 --> 00:09:52,519
And number two, there was blessedly only

223
00:09:52,520 --> 00:09:55,410
one keynote, as God and country intended.

224
00:09:55,670 --> 00:09:56,229
Chris: Wow.

225
00:09:56,560 --> 00:10:01,550
Ned: Yeah, thank you, sweet baby Jesus, and your tiny cherub hands, one keynote.

226
00:10:02,179 --> 00:10:07,270
From the very outset, Dave made the point that GenAI is changing the world.

227
00:10:07,580 --> 00:10:11,739
He made a reference to the tale of Rip Van Winkle, which is a story

228
00:10:11,779 --> 00:10:16,050
I barely remember, but if I’m recalling correctly, and he explained

229
00:10:16,050 --> 00:10:20,920
it properly, the core conceit is that Rip falls asleep for 20 years

230
00:10:21,309 --> 00:10:25,399
before the American Revolutionary War starts, and when he wakes

231
00:10:25,400 --> 00:10:29,699
up, the revolution is over and the world has changed drastically.

232
00:10:30,160 --> 00:10:30,920
I think that’s it.

233
00:10:31,250 --> 00:10:31,959
Don’t look it up.

234
00:10:32,550 --> 00:10:35,410
The obvious parallel here is that our world is going

235
00:10:35,410 --> 00:10:38,890
through a similar sea change, and the public sector

236
00:10:38,920 --> 00:10:42,570
can’t afford to sleep on it like Rip Van Winkle did.

237
00:10:43,170 --> 00:10:46,700
In fact, he kept using the word revolution

238
00:10:47,230 --> 00:10:50,310
several times to describe what AI is doing.

239
00:10:50,790 --> 00:10:55,150
And the word revolution does feel, like, a little bit loaded, especially

240
00:10:55,150 --> 00:11:00,010
given the political climate, and the seven or eight protesters who

241
00:11:00,020 --> 00:11:04,520
stood up in the middle of his talk, and yelled stuff about Gaza, Israel,

242
00:11:04,520 --> 00:11:07,770
and how evil Amazon is, and questioned how he could sleep at night.

243
00:11:08,230 --> 00:11:11,420
So, I don’t know, maybe find a word that’s a little less evocative.

244
00:11:11,810 --> 00:11:12,280
Maybe?

245
00:11:12,640 --> 00:11:12,860
I don’t know.

246
00:11:13,430 --> 00:11:15,189
Anyhow, it has nothing to do with AI.

247
00:11:15,559 --> 00:11:16,790
Just word choices.

248
00:11:17,309 --> 00:11:19,829
His main point is that GenAI is going to change

249
00:11:19,830 --> 00:11:24,170
the world, that GenAI requires the cloud to work.

250
00:11:24,400 --> 00:11:27,830
And he quoted a study commissioned by AWS that claims

251
00:11:27,840 --> 00:11:32,030
that cloud is 4.1 times more efficient than on-prem AI.

252
00:11:33,030 --> 00:11:38,949
And that AWS has the tools you need to take your AI experiments to production.

253
00:11:39,509 --> 00:11:42,179
Not sure how much weight I put behind a study commissioned by

254
00:11:42,219 --> 00:11:46,960
AWS that finds out, shockingly, cloud is better than on-prem.

255
00:11:47,560 --> 00:11:50,530
But, I mean, what else do you expect Dave to say?

256
00:11:51,150 --> 00:11:52,480
Chris: I have thoughts, but we can—

257
00:11:52,929 --> 00:11:54,210
Ned: Feel free to share.

258
00:11:54,630 --> 00:11:58,110
Chris: Anytime you see a study like this, what do they mean by on-prem?

259
00:11:58,420 --> 00:11:58,760
Ned: Yeah.

260
00:11:58,809 --> 00:11:59,560
Chris: Who set it up?

261
00:11:59,760 --> 00:12:02,640
How much money and time and expertise did they invest in it?

262
00:12:02,910 --> 00:12:04,200
What models are they using?

263
00:12:04,710 --> 00:12:05,060
Et cetera… et cetera… et cetera.

264
00:12:05,060 --> 00:12:05,120
Ned: What

265
00:12:08,110 --> 00:12:09,480
does it mean to be more efficient?

266
00:12:09,880 --> 00:12:11,459
How do you measure that efficiency?

267
00:12:11,559 --> 00:12:12,920
Like, so many questions.

268
00:12:13,420 --> 00:12:14,859
I didn’t want to get bogged down in it.

269
00:12:15,089 --> 00:12:17,380
I will try to find the study if I can.

270
00:12:17,980 --> 00:12:21,880
We will get into AWS’s AI stack in a little bit,

271
00:12:21,880 --> 00:12:24,010
but I do want to mention a couple other things.

272
00:12:24,820 --> 00:12:27,900
First, Dave said something about how AWS

273
00:12:27,969 --> 00:12:31,500
isn’t here to build the best AI applications.

274
00:12:31,889 --> 00:12:33,590
That’s what they expect you to do.

275
00:12:34,109 --> 00:12:39,359
They’re just providing the raw materials, which inadvertently is a dig at

276
00:12:39,360 --> 00:12:45,189
their own Amazon Q product, which ostensibly is a really good AI application.

277
00:12:45,719 --> 00:12:45,899
Whoops.

278
00:12:46,469 --> 00:12:49,390
Chris: Which also should be what AWS does.

279
00:12:49,679 --> 00:12:50,230
Ned: You would think.

280
00:12:50,610 --> 00:12:52,299
Chris: Just provide the raw materials.

281
00:12:52,730 --> 00:12:53,349
Ned: And that’s good.

282
00:12:53,350 --> 00:12:54,760
I actually liked that part of it.

283
00:12:54,770 --> 00:12:56,910
Like, yeah, you give me the raw materials.

284
00:12:56,920 --> 00:12:59,910
That’s what AWS has specialized in for years is giving

285
00:12:59,910 --> 00:13:05,130
developers, here’s a big old Lego set or an erector

286
00:13:05,139 --> 00:13:07,480
set, or whatever you want to call it; build your thing.

287
00:13:08,000 --> 00:13:11,650
You also made an offhand comment about data sovereignty issues,

288
00:13:11,969 --> 00:13:15,500
which I think was a subtle dig at Microsoft’s latest stumble

289
00:13:15,510 --> 00:13:20,740
with handling GDPR and the Scotland police force, I think?

290
00:13:21,550 --> 00:13:25,459
Dave had two guests with him on stage, one at a time.

291
00:13:25,639 --> 00:13:29,539
The first one was the director of AI for the CIA,

292
00:13:29,900 --> 00:13:32,900
and the second was the head of AI for the army.

293
00:13:33,570 --> 00:13:37,829
And both chats turned into weird recruitment pitches for

294
00:13:37,830 --> 00:13:41,680
the CIA and the army, along with a plea from the army

295
00:13:41,680 --> 00:13:45,269
guy for input on their AI layered defense framework.

296
00:13:45,929 --> 00:13:49,490
What was strangely absent from both conversations was any

297
00:13:49,490 --> 00:13:54,380
mention of how the CIA or the army are using AWS for anything.

298
00:13:54,720 --> 00:13:55,439
Chris: That’s classified.

299
00:13:56,179 --> 00:13:56,800
Ned: I guess?

300
00:13:57,240 --> 00:14:02,209
But also, like, you would think if you have guests up on stage at an AWS

301
00:14:02,690 --> 00:14:07,160
event, they’re going to at least mention how they’re using AWS in some way.

302
00:14:07,750 --> 00:14:09,160
Chris: I mean, counterpoint.

303
00:14:09,240 --> 00:14:13,790
This was hosted in DC, so it could very well have been, somebody

304
00:14:13,790 --> 00:14:17,599
called AWS and explained to them that these are going to be the guests.

305
00:14:18,130 --> 00:14:18,510
Ned: Mmm.

306
00:14:19,000 --> 00:14:19,449
Could be.

307
00:14:19,460 --> 00:14:23,279
I mean, the CIA probably has the leverage to do that.

308
00:14:24,020 --> 00:14:27,700
So, since they didn’t say what AWS is doing

309
00:14:27,700 --> 00:14:31,630
with AI, let’s talk about AWS’s AI portfolio.

310
00:14:32,440 --> 00:14:36,670
It’s kind of an open secret—not even a secret—it’s just known that

311
00:14:36,679 --> 00:14:40,650
Microsoft has a first-mover advantage when it comes to AI products.

312
00:14:40,910 --> 00:14:43,360
They partnered closely with OpenAI.

313
00:14:43,630 --> 00:14:47,030
They rolled out Copilot long before AWS had

314
00:14:47,030 --> 00:14:50,889
CodeWhisperer and GitLab had their product called Duo.

315
00:14:51,559 --> 00:14:55,469
Amazon and Google both looked to be caught flat-footed when

316
00:14:55,469 --> 00:14:59,589
it came to AI, and they are both furiously trying to catch up.

317
00:15:00,320 --> 00:15:04,890
That being said, a first-mover can have a big advantage,

318
00:15:05,080 --> 00:15:08,419
but it also means that you’re likely to fuck up a lot.

319
00:15:09,059 --> 00:15:12,280
You’re the one blazing the trail, you’re not following some

320
00:15:12,280 --> 00:15:15,800
well-trodden path, so the chances of making a logistical

321
00:15:15,809 --> 00:15:19,850
or tactical error are high, even if your strategy is sound.

322
00:15:20,830 --> 00:15:24,720
Sometimes being a fast-follower can actually work in your favor because

323
00:15:24,720 --> 00:15:29,610
you can see what the first-mover stumbled over and adjust accordingly.

324
00:15:29,660 --> 00:15:33,270
And I kind of get the feeling when watching AWS talk about

325
00:15:33,270 --> 00:15:37,790
their various AI offerings, that’s kind of where they’re at.

326
00:15:38,210 --> 00:15:40,459
They’re keeping an eye on Microsoft, and going well, as

327
00:15:40,460 --> 00:15:44,960
long as we can stay relatively close behind, we’ll be fine.

328
00:15:45,750 --> 00:15:51,790
So, they don’t have a monster model like OpenAI’s GPT4o.

329
00:15:52,650 --> 00:15:57,330
I guess it’s called four-oh—why did they abandon the semantic versioning?

330
00:15:57,340 --> 00:15:58,689
I don’t understand.

331
00:15:59,180 --> 00:16:02,310
Anyway—they don’t have that running in their data centers.

332
00:16:02,490 --> 00:16:05,080
Instead, they have their own foundational model which is

333
00:16:05,080 --> 00:16:09,090
called Titan, a model that, by most accounts, is pretty mid.

334
00:16:09,450 --> 00:16:09,760
Meh.

335
00:16:10,209 --> 00:16:12,250
Chris: More like a Teen Titan, am I right?

336
00:16:12,709 --> 00:16:14,070
Ned: Ahh, I like it.

337
00:16:14,270 --> 00:16:15,930
All right, which Team Titan would it be?

338
00:16:16,340 --> 00:16:17,060
Chris: The boring one.

339
00:16:17,290 --> 00:16:17,730
Ned: Robin.

340
00:16:17,970 --> 00:16:18,350
It’d be Robin.

341
00:16:18,350 --> 00:16:18,390
Chris: [laugh]

342
00:16:20,240 --> 00:16:21,949
.
Ned: [laugh] . But we repeat ourselves.

343
00:16:22,500 --> 00:16:24,689
What is a poor cloud hyperscaler to do?

344
00:16:25,230 --> 00:16:26,889
Build Amazon Bedrock?

345
00:16:27,099 --> 00:16:28,329
Chris: The Flintstones thing?

346
00:16:28,709 --> 00:16:30,930
Ned: Yeah, yeah, they invited Fred over and

347
00:16:30,930 --> 00:16:33,850
just, like, hash it out over some giant ribs.

348
00:16:34,240 --> 00:16:35,860
Chris: And strangely, he did a cigarette ad.

349
00:16:36,490 --> 00:16:37,830
Everyone was deeply confused.

350
00:16:40,099 --> 00:16:41,840
Ned: [laugh] . Oh, hell of a callback.

351
00:16:41,900 --> 00:16:42,390
Well, done.

352
00:16:43,170 --> 00:16:47,890
So, when Bedrock was announced, most people were like, “Huh?” But essentially

353
00:16:47,890 --> 00:16:52,390
what it is—and actually find it pretty interesting now that I understand it—it’s

354
00:16:52,400 --> 00:16:57,159
essentially an abstraction layer on top of other existing foundational models.

355
00:16:57,420 --> 00:16:59,440
So, it is not an LLM.

356
00:16:59,469 --> 00:17:01,460
It’s not a model in and of itself.

357
00:17:01,840 --> 00:17:03,979
It doesn’t tie you down to a single model.

358
00:17:04,260 --> 00:17:07,870
Instead, you can choose to use one or more of their

359
00:17:08,000 --> 00:17:11,050
various supported models, or bring your own model.

360
00:17:11,589 --> 00:17:18,750
Bedrock currently supports Titan, Anthropic, Meta, Mistral, and a bunch more.

361
00:17:19,030 --> 00:17:21,800
You can look up the current list; I’m sure it’s constantly changing.

362
00:17:22,410 --> 00:17:27,160
And the first session that I attended was actually a builder session

363
00:17:27,160 --> 00:17:31,720
where the presenters walked through using Bedrock in a Jupyter Notebook

364
00:17:31,720 --> 00:17:36,340
to interact with four different models, basically, simultaneously.

365
00:17:36,740 --> 00:17:40,609
I very much jumped into the deep end of things, inadvertently.

366
00:17:41,600 --> 00:17:43,580
I [laugh] walked in, they had, like, code up on the

367
00:17:43,580 --> 00:17:45,960
screen, and I was like, what did I just get myself into?

368
00:17:46,420 --> 00:17:46,939
All right.

369
00:17:47,590 --> 00:17:51,660
The idea was to test and evaluate the responses from each model,

370
00:17:51,830 --> 00:17:56,020
and figure out which one would be best suited for a set of tasks.

371
00:17:56,309 --> 00:17:58,280
And you can go back and check that on a regular

372
00:17:58,280 --> 00:18:00,800
basis because the models are constantly changing.

373
00:18:01,570 --> 00:18:07,169
Inside of the Bedrock SDK for Python, there’s a method called Converse,

374
00:18:07,780 --> 00:18:12,070
and this method and the service behind it abstracts the required

375
00:18:12,080 --> 00:18:16,990
inputs and parameters for each model into a consistent set of inputs.

376
00:18:17,670 --> 00:18:21,600
So, every model has its own specific parameters, and the way that

377
00:18:21,600 --> 00:18:25,800
it expects you to structure requests, and the way that it structures

378
00:18:25,800 --> 00:18:29,529
its responses, especially when you’re interacting programmatically.

379
00:18:30,360 --> 00:18:34,560
The key idea here was you can use this Converse method in a method

380
00:18:34,560 --> 00:18:39,110
call to loop through as many models as you’d like at once, and it

381
00:18:39,120 --> 00:18:42,639
has a consistent interface in front of those models, so you don’t

382
00:18:42,650 --> 00:18:47,429
have to adjust your parameters and your inputs, and you know what

383
00:18:47,429 --> 00:18:50,919
form the response is going to take because it normalizes it for you.

384
00:18:51,270 --> 00:18:51,410
Chris: Hmm.

385
00:18:51,940 --> 00:18:54,160
Ned: That seems kind of useful.

386
00:18:55,510 --> 00:19:00,030
If you need to, you can put in model specific-parameters in a separate field,

387
00:19:00,050 --> 00:19:04,129
so in your configuration of the method, you can say, “For this model, add

388
00:19:04,139 --> 00:19:08,209
these two or three customized parameters.” So, you can still add that level

389
00:19:08,210 --> 00:19:11,910
of customization in there, but if you don’t want to, you don’t have to.

390
00:19:12,500 --> 00:19:17,300
And since the technology and the models are evolving so rapidly, having

391
00:19:17,310 --> 00:19:22,730
a low switching cost between models is going to be a huge benefit, versus

392
00:19:22,730 --> 00:19:27,180
just hitching your wagon to OpenAI and hoping for the best from Sam Altman.

393
00:19:27,929 --> 00:19:30,179
I don’t feel super positive about that [laugh]

394
00:19:31,080 --> 00:19:33,170
.
Bedrock also has a couple other neat things.

395
00:19:33,250 --> 00:19:35,640
One is something they’re calling Knowledge Base, which

396
00:19:35,640 --> 00:19:39,899
is essentially RAG, aka, Retrieval Augmented Generation.

397
00:19:40,320 --> 00:19:43,400
Like the name implies, you can load up additional

398
00:19:43,410 --> 00:19:47,889
documents in S3 and feed them to a generic model to help

399
00:19:47,889 --> 00:19:51,610
it better understand your unique business and requirements.

400
00:19:52,200 --> 00:19:55,660
Each model has a different way of doing RAG, so it’s

401
00:19:55,670 --> 00:19:58,360
nice to have Bedrock just handle that abstraction.

402
00:19:58,620 --> 00:20:00,830
Hey, I want to use these six different models.

403
00:20:01,429 --> 00:20:03,980
Send it in the document way they prefer from

404
00:20:03,980 --> 00:20:06,209
this S3 bucket that I’ve loaded things up with.

405
00:20:06,879 --> 00:20:10,590
The other interesting concept—and I’m still trying to wrap my brain

406
00:20:10,590 --> 00:20:15,360
exactly around how it works—but it was this concept of agents and tools.

407
00:20:15,429 --> 00:20:17,239
And I’m not talking about agents in the

408
00:20:17,240 --> 00:20:19,930
sense of chatbots, or digital assistants.

409
00:20:20,320 --> 00:20:25,010
Instead, it’s a way that Bedrock lets you define tools

410
00:20:25,330 --> 00:20:28,950
that models can pass work off to in their application flow.

411
00:20:29,580 --> 00:20:33,440
So, the tools are really just Lambda functions that interact with some

412
00:20:33,440 --> 00:20:38,160
other API or data source, and the agent is the part of Bedrock that

413
00:20:38,170 --> 00:20:42,970
does the tool invocation and passes information back to the model.

414
00:20:43,580 --> 00:20:47,300
So, you can ask your application to do something like parse a bunch of

415
00:20:47,330 --> 00:20:51,959
emails for sales orders and then add them to your ordering system, and

416
00:20:51,960 --> 00:20:56,230
the model can use natural language parsing to understand the mail content,

417
00:20:56,280 --> 00:21:00,450
and then invoke the tool to actually interact with the ordering system.

418
00:21:01,100 --> 00:21:03,850
This is the sort of thing that’s going to put a lot of interns out of work.

419
00:21:04,490 --> 00:21:06,570
Chris: Yeah, it’s basically fancified RPA.

420
00:21:07,170 --> 00:21:08,680
Ned: Yeah, that’s a great way to put it.

421
00:21:08,860 --> 00:21:14,270
It is very much like—yeah, whatever RPA stands for, Robotic Process Automation?

422
00:21:14,429 --> 00:21:15,239
Chris: There you go.

423
00:21:15,249 --> 00:21:15,454
Ned: That’s the one.

424
00:21:15,599 --> 00:21:16,429
I got there.

425
00:21:17,010 --> 00:21:21,280
But yeah, if you have existing APIs and other systems that you want

426
00:21:21,280 --> 00:21:25,550
your language models or your foundational models to interact with,

427
00:21:25,870 --> 00:21:29,879
you don’t have to write that interaction into your application.

428
00:21:30,170 --> 00:21:34,230
You just define this tool spec—I think they called it a tool spec;

429
00:21:34,230 --> 00:21:38,050
it something along those lines—and then implement that tool spec in

430
00:21:38,330 --> 00:21:42,750
Lambda, and now your foundational model can interact with whatever

431
00:21:42,780 --> 00:21:46,419
API you want to hand it off to, if it’s internal or external.

432
00:21:46,500 --> 00:21:50,879
I was like, “Oh, that sounds pretty awesome, but also possibly

433
00:21:50,880 --> 00:21:55,690
incredibly dangerous because now we’ve allowed AI to just do whatever

434
00:21:55,690 --> 00:21:59,400
the hell it wants.” [laugh] . I’m sure that won’t come back to bite us.

435
00:22:00,160 --> 00:22:01,800
So, that’s Bedrock in a nutshell.

436
00:22:02,000 --> 00:22:05,100
There’s more to it, I’m sure, but those are the pieces that jumped out to me.

437
00:22:05,889 --> 00:22:11,149
AWS also went to great pains to say they do not store your requests,

438
00:22:11,389 --> 00:22:16,340
the model responses, or the augmented data anywhere inside of Bedrock.

439
00:22:16,720 --> 00:22:20,830
All of it lives inside your account, and the models that you bring with you.

440
00:22:21,320 --> 00:22:21,900
That’s nice.

441
00:22:22,440 --> 00:22:26,570
I think they went to special pains to explain that because of the audience.

442
00:22:27,059 --> 00:22:27,359
Chris: Right.

443
00:22:28,389 --> 00:22:31,270
Ned: So, from there, I want to jump ahead to a presentation that was about

444
00:22:31,380 --> 00:22:37,430
using GenAI in DevOps, which was a topic that I found very interesting.

445
00:22:37,860 --> 00:22:44,020
The two presenters, Tyler Lynch and Pranusha Manchala—I’m hope I’m

446
00:22:44,020 --> 00:22:48,410
getting that vaguely right; apologies to Pranusha—they lead the

447
00:22:48,420 --> 00:22:52,310
chalk talk and polled the audience about issues and challenges they

448
00:22:52,310 --> 00:22:58,340
encounter at all stages of the software design and deployment pipeline.

449
00:22:59,050 --> 00:23:01,480
If you’ve never attended a chalk talk before, they basically have a

450
00:23:01,620 --> 00:23:05,779
whiteboard, and, like, a loose presentation, but it’s very audience-driven.

451
00:23:05,930 --> 00:23:09,680
So, they’re asking for input: what is your experience?

452
00:23:09,700 --> 00:23:11,159
What are you seeing out there?

453
00:23:11,630 --> 00:23:14,569
As we rolled through the issues, they tried to give

454
00:23:14,570 --> 00:23:18,470
examples of how AWS’s various developer services could

455
00:23:18,480 --> 00:23:21,840
help alleviate some of the challenges people were feeling.

456
00:23:22,029 --> 00:23:25,399
So, I mean, you’re at an AWS conference; the solution is going to be AWS, right?

457
00:23:25,690 --> 00:23:26,350
Chris: One would think.

458
00:23:26,670 --> 00:23:26,940
Ned: Yeah.

459
00:23:27,580 --> 00:23:29,030
They highlighted three products.

460
00:23:29,060 --> 00:23:31,350
One was Amazon Q Developer.

461
00:23:31,560 --> 00:23:33,570
Q Developer used to be called CodeWhisperer.

462
00:23:34,330 --> 00:23:36,130
I don’t know if that was better [laugh] or worse.

463
00:23:36,660 --> 00:23:37,629
Chris: I think you do know.

464
00:23:37,880 --> 00:23:38,430
Ned: I do know.

465
00:23:38,559 --> 00:23:39,250
They’re both terrible.

466
00:23:40,039 --> 00:23:43,000
It’s very similar to what GitHub Copilot does.

467
00:23:43,210 --> 00:23:48,040
It plugs into your IDE, and it provides suggestions and a chat interface.

468
00:23:48,450 --> 00:23:51,190
Do you want to know what a chunk of code does

469
00:23:51,280 --> 00:23:53,309
when you’re trying to get to know a codebase?

470
00:23:53,889 --> 00:23:57,530
Highlight it, ask Q to explain it to you, and it will look at

471
00:23:57,530 --> 00:24:00,260
it and say, “This is what this function does,” in plain English.

472
00:24:00,709 --> 00:24:02,930
Do you need to refactor some code to be more

473
00:24:02,930 --> 00:24:05,280
efficient or scanned for vulnerabilities?

474
00:24:05,730 --> 00:24:08,010
You can do both of those things, making

475
00:24:08,010 --> 00:24:10,229
suggestions for how you could rewrite the code.

476
00:24:10,860 --> 00:24:13,980
Q can also write unit and integration tests,

477
00:24:14,530 --> 00:24:17,209
something that basically no one ever wants to do.

478
00:24:17,529 --> 00:24:21,590
As someone who’s working on developing a Go tool right now, I can tell you

479
00:24:21,590 --> 00:24:26,350
that I spend more time writing unit tests than I do writing actual code.

480
00:24:26,990 --> 00:24:27,630
It sucks.

481
00:24:27,950 --> 00:24:28,439
Chris: Not great.

482
00:24:28,940 --> 00:24:29,379
It’s not great.

483
00:24:29,800 --> 00:24:30,030
Ned: Yeah.

484
00:24:30,580 --> 00:24:34,710
Tyler also mentioned that Q can help with code migrations to

485
00:24:34,710 --> 00:24:37,660
newer language versions, and this was really interesting to me.

486
00:24:37,970 --> 00:24:44,420
Amazon used this functionality to migrate from Java 8 to Java 17.

487
00:24:44,730 --> 00:24:48,009
Now, as someone who is not a Java developer, I wasn’t

488
00:24:48,020 --> 00:24:52,360
really sure, okay, but I guess that is a significant

489
00:24:52,369 --> 00:24:56,200
jump, and those are two largely implemented versions.

490
00:24:56,710 --> 00:24:58,200
I guess there’s not a ton in between.

491
00:24:58,660 --> 00:24:59,960
I don’t really follow the numbering.

492
00:25:00,470 --> 00:25:03,890
But basically, they said they had a thousand applications

493
00:25:04,040 --> 00:25:09,030
that use Java 8 internally at Amazon, and so they took Q and

494
00:25:09,030 --> 00:25:12,680
turned it loose on those thousand applications to upgrade

495
00:25:12,680 --> 00:25:16,950
them all to Java 17, and they were able to do it in two days.

496
00:25:17,670 --> 00:25:18,890
Eyebrow-raising indeed.

497
00:25:20,360 --> 00:25:20,389
Chris: [laugh]

498
00:25:20,599 --> 00:25:21,660
.
Ned: That’s bananas.

499
00:25:22,240 --> 00:25:25,389
If you think about the amount of work that would be for developers

500
00:25:25,389 --> 00:25:30,690
to update the code and run the tests for it, they saved a

501
00:25:30,690 --> 00:25:34,320
tremendous amount of time, and also washed away a tremendous

502
00:25:34,320 --> 00:25:38,790
amount of tech debt they had being on this older version of Java.

503
00:25:39,280 --> 00:25:42,050
They also have a similar path to get from Java

504
00:25:42,050 --> 00:25:46,909
4 to 8, and then 17 for much older applications.

505
00:25:47,039 --> 00:25:52,149
I feel like Java 4 was current when I was developing Java back in, like, 1998.

506
00:25:52,870 --> 00:25:55,630
They also are working on similar upgrade functionality

507
00:25:55,630 --> 00:25:59,440
for .NET to .NET Core, and Python 2 to 3.

508
00:26:00,090 --> 00:26:02,740
This is the kind of thing that no one ever wants to do.

509
00:26:03,950 --> 00:26:06,800
No one ever sits down, they’re, like, “Today’s the day when we’re

510
00:26:06,800 --> 00:26:10,690
going to pay off our tech debt and upgrade from Python 2 to 3.”

511
00:26:10,940 --> 00:26:11,100
Chris: Yeah.

512
00:26:11,240 --> 00:26:12,940
For most companies, that day is never.

513
00:26:13,300 --> 00:26:14,830
Ned: Right because you look at it, you’re like, there

514
00:26:14,830 --> 00:26:17,380
is no immediate business value in me doing this.

515
00:26:17,710 --> 00:26:22,260
The application still works, I am not doing anything until I absolutely have to.

516
00:26:23,000 --> 00:26:26,960
But doing that upgrade actually does come with some benefits.

517
00:26:27,420 --> 00:26:29,970
Another session I attended was put on by GitLab, and

518
00:26:30,320 --> 00:26:32,810
they were talking about their code assistant, which is

519
00:26:32,810 --> 00:26:38,040
called Duo, and Duo can do refactoring from C to Rust.

520
00:26:38,620 --> 00:26:43,040
And they talked to a public sector organization whose name they

521
00:26:43,040 --> 00:26:47,000
couldn’t say, but they talked to one, and they had this C-based

522
00:26:47,010 --> 00:26:51,460
application that was a bit of a bottleneck for what they were doing.

523
00:26:52,040 --> 00:26:54,629
They had Duo port it over to Rust.

524
00:26:54,690 --> 00:26:56,080
And this was just an experiment.

525
00:26:56,299 --> 00:26:59,449
It was supposed to be like a three-month-long migration process.

526
00:27:00,129 --> 00:27:03,639
They had Duo try to do the migration of C to Rust,

527
00:27:03,830 --> 00:27:07,200
and move it to running on Lambda instead of EC2.

528
00:27:07,630 --> 00:27:11,480
Duo and the senior developer managed to do it in two days.

529
00:27:12,110 --> 00:27:12,830
Chris: It’s this the rule?

530
00:27:12,830 --> 00:27:14,120
Everything happens in two days?

531
00:27:14,219 --> 00:27:20,030
Ned: Apparently [laugh] . The resulting application was 90%

532
00:27:20,030 --> 00:27:24,399
smaller, ran a hundred times faster, and could scale by 50x.

533
00:27:25,510 --> 00:27:26,800
In part because of Lambda.

534
00:27:27,380 --> 00:27:28,699
Chris: Well, and just because of Rust.

535
00:27:28,710 --> 00:27:32,110
And something interesting about this is it’s not just simple refactoring,

536
00:27:32,120 --> 00:27:36,130
but you’re looking at optimizing how the code does what it does.

537
00:27:37,210 --> 00:27:40,560
You know, like, the difference between a junior programmer writing this

538
00:27:40,570 --> 00:27:44,630
program for the very first time and somebody that’s got 35 years of experience

539
00:27:44,980 --> 00:27:48,030
picking up that code and saying, “This is what it should have looked like.”

540
00:27:48,429 --> 00:27:49,089
Ned: Exactly.

541
00:27:49,139 --> 00:27:52,779
And I think that was the point that the presenter made was, senior

542
00:27:52,779 --> 00:27:56,550
developers will actually get some of the most benefit out of this because

543
00:27:56,980 --> 00:28:00,490
they can look at the code that is produced, and they understand the

544
00:28:00,490 --> 00:28:06,360
context of the original code, so they have a better chance of making

545
00:28:06,360 --> 00:28:10,190
sure the code actually works after that conversion, and works properly.

546
00:28:10,629 --> 00:28:12,960
It’s just that this GitLab Duo is going to save

547
00:28:12,960 --> 00:28:16,320
them a ton of time in doing the actual code writing.

548
00:28:17,260 --> 00:28:18,980
So, that was pretty impressive to me.

549
00:28:19,170 --> 00:28:23,820
He also talked about how so many companies will do, like, a lift-and-shift,

550
00:28:24,110 --> 00:28:28,789
and then never actually refactor the code, and so you have all these

551
00:28:28,820 --> 00:28:34,530
inefficient applications eating up cycles on EC2 in AWS or wherever you

552
00:28:34,530 --> 00:28:39,250
decide to host your code, if you can go back and use one of these code

553
00:28:39,450 --> 00:28:44,639
assistants to refactor things pretty quickly—you know, instead of a six-month

554
00:28:44,639 --> 00:28:48,490
lead time, you’re now talking about a two-week lead time, maybe—that’s

555
00:28:48,490 --> 00:28:52,320
actually worth doing, and could be game-changing for some organizations.

556
00:28:52,830 --> 00:28:53,110
Chris: Yeah.

557
00:28:53,120 --> 00:28:57,020
I mean, a lot of times, people simply hide bad code behind larger instances.

558
00:28:57,800 --> 00:28:58,639
Ned: [laugh] . Sometimes?

559
00:28:58,920 --> 00:29:00,610
Sometimes it’s doing a lot of lifting there, buddy [laugh]

560
00:29:00,760 --> 00:29:00,790
.
Chris: [laugh]

561
00:29:03,290 --> 00:29:06,159
.
Ned: Yeah, the fact that we can just throw more hardware at it, and have

562
00:29:06,160 --> 00:29:10,030
done that for years, this is actually an opportunity to slim down your

563
00:29:10,030 --> 00:29:14,290
code, maybe make it safer—because if you’re moving from C to Rust, it’s now

564
00:29:14,300 --> 00:29:18,360
memory safe at least—and maybe squash some security bugs in the process.

565
00:29:18,360 --> 00:29:20,599
Like, that’s a win for a lot of teams.

566
00:29:21,650 --> 00:29:23,889
Moving on from the code assistant, the other tools

567
00:29:23,900 --> 00:29:27,180
that were mentioned are DevOps Guru and Code Guru.

568
00:29:27,790 --> 00:29:28,490
Terrible names.

569
00:29:29,010 --> 00:29:32,409
The former does scanning of CloudTrail logs looking for anomalous

570
00:29:32,410 --> 00:29:35,889
events, and the latter does code scanning, testing, and assessment.

571
00:29:36,389 --> 00:29:39,050
From what I understand, Code Guru charges by

572
00:29:39,050 --> 00:29:42,660
the line scanned and is prohibitively expensive.

573
00:29:42,880 --> 00:29:43,310
Chris: Okay.

574
00:29:43,380 --> 00:29:47,300
Ned: DevOps guru, similarly expensive for looking at CloudTrail logs.

575
00:29:48,040 --> 00:29:49,820
But they mentioned them, so I guess I should, too.

576
00:29:50,690 --> 00:29:54,859
The last part of the AWS GenAI sandwich is hardware.

577
00:29:55,730 --> 00:29:59,049
AWS has some hardware to throw at the problem, including their

578
00:29:59,090 --> 00:30:04,530
own in-house developed chips called Trainium, and Inferentia.

579
00:30:04,910 --> 00:30:05,540
Chris: Trainium?

580
00:30:06,900 --> 00:30:09,599
Ned: Trainium, which are for model training, and inference, respectively.

581
00:30:10,389 --> 00:30:11,970
But I think you already guessed that.

582
00:30:12,650 --> 00:30:14,140
Are these chips any good?

583
00:30:14,759 --> 00:30:15,510
No idea.

584
00:30:16,049 --> 00:30:17,159
Are the names any good?

585
00:30:17,809 --> 00:30:18,099
No.

586
00:30:18,879 --> 00:30:23,030
They also have Nvidia A100s and soon will have H100s, so maybe

587
00:30:23,030 --> 00:30:25,650
that’s a better bet if you’re doing this model training thing.

588
00:30:26,050 --> 00:30:26,649
Chris: Just a thought.

589
00:30:27,090 --> 00:30:27,409
Ned: Yeah.

590
00:30:27,920 --> 00:30:30,850
Another session I attended was all about the training process

591
00:30:30,850 --> 00:30:35,749
for a foundational model called Falcon, and to train the model,

592
00:30:35,859 --> 00:30:40,900
they use SageMaker, and ECR, and S3, the biggest version of

593
00:30:40,900 --> 00:30:43,860
the model that they train, they did three versions of it.

594
00:30:43,970 --> 00:30:49,430
The biggest one was called 180B, which I guess is for billion parameters.

595
00:30:50,090 --> 00:30:57,390
This thing used 3.5 trillion training tokens, 46,000 petaflop days—I’m

596
00:30:57,390 --> 00:31:03,280
not even sure what that means, but that’s what they quoted—and 4096 GPUs.

597
00:31:03,969 --> 00:31:07,560
The training process took six months, and used P4d

598
00:31:08,820 --> 00:31:12,450
EC2 instances that each have eight A100s in them.

599
00:31:12,930 --> 00:31:16,130
This was not cheap… by any stretch of the imagination.

600
00:31:16,510 --> 00:31:20,070
They actually, through this process, hit the performance limit

601
00:31:20,110 --> 00:31:24,830
of S3, and had to write new custom libraries that go beyond the

602
00:31:25,090 --> 00:31:31,010
3500 PUTs and 5500 GETs per second that S3 normally supports.

603
00:31:31,550 --> 00:31:34,370
Chris: I was going to say, S3 is not supposed to have limits.

604
00:31:34,630 --> 00:31:36,009
Ned: It does [laugh]

605
00:31:36,639 --> 00:31:36,669
—
Chris: [laugh]

606
00:31:36,890 --> 00:31:38,969
.
Ned: But you have to work real hard to get there.

607
00:31:39,469 --> 00:31:40,050
They did it.

608
00:31:40,360 --> 00:31:41,210
Congratulations.

609
00:31:41,889 --> 00:31:46,509
It turns out, if you use—there’s a version of S3 that is a single

610
00:31:46,719 --> 00:31:51,919
zone—not single region, but single zone—S3—I forget exactly what it’s

611
00:31:51,920 --> 00:31:56,320
called—if you pick that version of S3, the limits go up because it no

612
00:31:56,320 --> 00:32:00,220
longer has to do any replication or spreading across availability zones.

613
00:32:00,599 --> 00:32:00,819
Chris: Right.

614
00:32:00,870 --> 00:32:02,639
I think that version, when you look at it in

615
00:32:02,880 --> 00:32:05,000
the drop-down, is called don’t use this for DR.

616
00:32:05,289 --> 00:32:07,640
Don’t—for God’s sake—use this for DR.

617
00:32:07,890 --> 00:32:10,520
Ned: Yeah, it’s intended for exactly what they were doing,

618
00:32:10,520 --> 00:32:13,390
which is these large data models that they’re loading.

619
00:32:13,600 --> 00:32:15,359
Chris: People are going to use it for DR, aren’t they?

620
00:32:15,559 --> 00:32:16,899
Ned: They absolutely are.

621
00:32:16,899 --> 00:32:21,330
So, that session was super cool because they had AWS and folks

622
00:32:21,330 --> 00:32:25,260
from Falcon’s company—it’s called TII—talking about the model

623
00:32:25,260 --> 00:32:28,850
design, the training methodology, the data sources and efficiency.

624
00:32:29,270 --> 00:32:32,360
I feel like that could be its own episode, so I’m just going to stop there.

625
00:32:32,940 --> 00:32:33,260
Chris: Cool.

626
00:32:33,710 --> 00:32:34,260
Ned: Geez.

627
00:32:34,500 --> 00:32:34,784
I will brief—

628
00:32:34,800 --> 00:32:35,919
Chris: You said you were going to stop.

629
00:32:36,460 --> 00:32:37,839
Ned: I was going to stop talking about that.

630
00:32:37,900 --> 00:32:41,919
I’m briefly going to touch on AI security, just to mention that I

631
00:32:41,920 --> 00:32:46,530
attended a session that talked about how to implement—or security

632
00:32:46,830 --> 00:32:50,579
things to worry about when it comes to interacting with LLMs, and

633
00:32:50,580 --> 00:32:56,200
they mentioned that OWASP has a top ten for LLMs, which I had no idea.

634
00:32:56,849 --> 00:33:02,720
That report was brought together by 500 experts and 125 contributors,

635
00:33:02,800 --> 00:33:06,899
and came up with a list of possible vulnerabilities for LLMs.

636
00:33:07,500 --> 00:33:08,769
I read through part of the report.

637
00:33:08,820 --> 00:33:09,850
It is fascinating.

638
00:33:10,059 --> 00:33:12,890
Once again, this paper could be its own show, and I would

639
00:33:12,890 --> 00:33:15,760
like to have someone from OWASP or one of the contributors

640
00:33:15,980 --> 00:33:19,379
come on and explain it to me like the third-grader that I am.

641
00:33:20,020 --> 00:33:20,930
Chris: I support this plan.

642
00:33:20,940 --> 00:33:22,429
It means I have to do less work.

643
00:33:22,710 --> 00:33:23,210
Ned: Well, there you go.

644
00:33:23,540 --> 00:33:27,370
Me too [laugh] . So, just to bring it all

645
00:33:27,370 --> 00:33:31,130
down to brass tacks, am I an AI convert now?

646
00:33:31,219 --> 00:33:35,360
Was drinking from the firehose enough to bring me over to the AI side?

647
00:33:35,809 --> 00:33:36,269
No.

648
00:33:37,150 --> 00:33:40,760
My goal at the conference was to enter each session with an open mind.

649
00:33:41,120 --> 00:33:45,460
I did my best not to scoff at ridiculous claims, I tried to take

650
00:33:45,460 --> 00:33:49,350
people and the services at face value, to listen, to not judge.

651
00:33:49,710 --> 00:33:53,360
And for the most part, I was able to turn off the supercritical and

652
00:33:53,360 --> 00:33:57,700
cynical parts of my brain, and just kind of absorb it as it was presented.

653
00:33:58,320 --> 00:34:01,379
Now that I’ve had a little time to digest, here’s what I think:

654
00:34:01,840 --> 00:34:05,310
for starters, GenAI is still very much in its early days.

655
00:34:05,750 --> 00:34:09,950
There were several comparisons to the dotcom bubble, and I think that’s fair.

656
00:34:10,580 --> 00:34:14,440
There’s an embarrassing amount of money being poured into anything AI

657
00:34:14,860 --> 00:34:18,759
right now in the same way that any company that had a vague association

658
00:34:18,770 --> 00:34:22,120
with the World Wide Web was showered with funding in the late-’90s.

659
00:34:22,870 --> 00:34:24,680
That bubble must and will pop.

660
00:34:25,030 --> 00:34:28,020
There will be some genuine use cases that are worth pursuing,

661
00:34:28,369 --> 00:34:30,970
some that are a little ahead of their time, but they’ll come back

662
00:34:30,970 --> 00:34:36,020
around, and some that are just absolute crocks or complete scams.

663
00:34:36,610 --> 00:34:37,620
A lot of them are that.

664
00:34:38,429 --> 00:34:41,449
The most compelling use case I saw—and you may have already

665
00:34:41,449 --> 00:34:44,320
guessed this—was around developer and ops productivity.

666
00:34:44,880 --> 00:34:49,030
There’s a metric ton of tech debt at every large company, little

667
00:34:49,030 --> 00:34:52,360
desire to actually fix it—well, I won’t say ‘a little desire.’ There’s

668
00:34:52,360 --> 00:34:57,260
desire to fix it, but not enough time or money to fix it—this may

669
00:34:57,260 --> 00:35:01,700
actually be the best use case for GenAI AI as it exists right now.

670
00:35:02,360 --> 00:35:04,360
GenAI is good at understanding code.

671
00:35:04,379 --> 00:35:06,880
It’s good at migrating it, it’s good at testing it, and it’s

672
00:35:06,880 --> 00:35:09,860
good at ingesting a lot of data and making it available.

673
00:35:10,360 --> 00:35:13,709
So, it’s going to help Dev and Ops tremendously.

674
00:35:14,050 --> 00:35:19,360
Although I have a little concern around junior talent, and them

675
00:35:19,420 --> 00:35:22,369
being developed if we’re going to end up replacing them with AI.

676
00:35:23,250 --> 00:35:25,010
So, that is a potential problem.

677
00:35:25,790 --> 00:35:27,730
Everything else was AI whitewashing.

678
00:35:28,280 --> 00:35:31,150
Walking the expo floor, basically, every

679
00:35:31,150 --> 00:35:33,880
single booth had AI plastered on it somewhere.

680
00:35:34,130 --> 00:35:35,669
I am not exaggerating.

681
00:35:36,330 --> 00:35:39,459
Every company, from storage to monitoring to professional

682
00:35:39,460 --> 00:35:45,950
services, had AI something, and 90% of them were talking about

683
00:35:45,950 --> 00:35:50,220
some kind of shitty chatbot or some basic-ass machine learning.

684
00:35:50,990 --> 00:35:52,650
In a few years, it will be something else

685
00:35:52,690 --> 00:35:54,950
in the same way that cloud was at one point.

686
00:35:55,190 --> 00:35:56,609
But most of it is just bullshit.

687
00:35:57,270 --> 00:35:58,670
Am I glad I took the plunge?

688
00:35:59,029 --> 00:35:59,329
Yeah.

689
00:35:59,840 --> 00:36:01,780
It was informative to see what people are really

690
00:36:01,780 --> 00:36:04,950
doing versus the marketing and the general messaging.

691
00:36:05,560 --> 00:36:08,319
I can see there’s a lot of potential for disruption coming down the

692
00:36:08,330 --> 00:36:12,930
pike, but it’s not nearly as dire or revolutionary, as some suggest.

693
00:36:13,500 --> 00:36:15,710
Or I don’t know, Chris, maybe I’m wrong and AGI

694
00:36:16,010 --> 00:36:18,169
will be born next week and put me to the sword.

695
00:36:18,620 --> 00:36:19,060
Whatever.

696
00:36:19,060 --> 00:36:20,210
I’m going to be on vacation.

697
00:36:20,650 --> 00:36:22,080
My digital assistant can handle it.

698
00:36:22,740 --> 00:36:23,250
That’s it.

699
00:36:23,320 --> 00:36:24,880
Hey, thanks for listening or something.

700
00:36:24,880 --> 00:36:26,900
I guess you found it worthwhile enough if you made it all

701
00:36:26,900 --> 00:36:29,230
the way to the end, so congratulations to you, friend.

702
00:36:29,480 --> 00:36:30,689
You accomplished something today.

703
00:36:30,870 --> 00:36:33,980
Now, you can sit on the couch, query Developer Q—Q

704
00:36:33,980 --> 00:36:38,520
Developer?—for the best way to rewrite your code, and just relax.

705
00:36:38,570 --> 00:36:39,250
You’ve earned it.

706
00:36:39,420 --> 00:36:42,250
You can find more about this show by visiting our LinkedIn page,

707
00:36:42,250 --> 00:36:45,850
just search ‘Chaos Lever,’ or go to the website, chaoslever.com

708
00:36:45,850 --> 00:36:48,899
where you’ll find show notes, blog posts, and general tomfoolery.

709
00:36:49,340 --> 00:36:53,380
We’ll be off next week because I actually am genuinely going on vacation.

710
00:36:53,730 --> 00:36:57,420
So, you’ll have to wait two weeks to see what fresh hell is upon us.

711
00:36:57,710 --> 00:36:58,549
Ta-ta for now.

712
00:37:06,730 --> 00:37:08,029
Chris: When are you not on vacation?

713
00:37:08,429 --> 00:37:09,549
Ned: I’m sorry, I was on vacation.

714
00:37:09,559 --> 00:37:09,899
What?

715
00:37:10,259 --> 00:37:10,399
[laugh]