Reliability Management With SLOs

In this session, we covered:

• Resource Investment: Both building and buying an SLO solution come with significant costs. Are you ready to invest the time and money into the right path for your organization?

• Implementation Timeline & Compatibility: Speed matters, but rushing can backfire. Can your team afford delays, and will the solution seamlessly integrate with your existing infrastructure?

• SLO Creation & Management: The true value of SLOs lies in their usability. Will your team adopt and maintain them effectively, or will complexity derail your efforts?

• Risk of Failure: Getting it wrong can do more than just waste time and resources—it could undermine team morale and create resistance to future SLO initiatives.

Webinar Transcript

00:00:13.400 --> 00:00:14.300
Daniel Ruby: Oh.

2
00:00:14.480 --> 00:00:17.188
Daniel Ruby: hello! Everybody is joining us.

3
00:00:18.200 --> 00:00:21.010
Daniel Ruby: welcome to this webinar.

4
00:00:21.810 --> 00:00:24.960
Daniel Ruby: Gonna be having this round table discussion on

5
00:00:25.420 --> 00:00:29.120
Daniel Ruby: implementing slos, the build versus buy conundrum.

6
00:00:29.690 --> 00:00:32.210
Daniel Ruby: and we're gonna give about

7
00:00:32.590 --> 00:00:37.039
Daniel Ruby: 3 min for for attendees to get in. I can see y'all

8
00:00:37.450 --> 00:00:40.540
Daniel Ruby: getting on. Glad to see

9
00:00:41.070 --> 00:00:43.280
Daniel Ruby: a lot of people joining.

10
00:00:43.784 --> 00:00:50.730
Daniel Ruby: We will officially kick off the discussion at let's call it 1 0, 4 Eastern time.

11
00:00:51.340 --> 00:00:52.750
Daniel Ruby: just so we can have a

12
00:00:53.410 --> 00:00:55.420
Daniel Ruby: give people a chance to get on

13
00:00:57.150 --> 00:00:59.679
Daniel Ruby: and give me a chance to drink some water

14
00:01:01.110 --> 00:01:02.750
Daniel Ruby: cause. That's what matters right.

15
00:01:04.099 --> 00:01:05.579
Mandi Walls: Stay? Hydrated. Yeah.

16
00:01:05.760 --> 00:01:12.620
Daniel Ruby: Yeah gotta stay hydrated. I've been drinking nothing but coffee all morning, so I'm like shrivelling up like a prune here.

17
00:01:24.500 --> 00:01:26.180
Daniel Ruby: How's everybody's week going

18
00:01:29.070 --> 00:01:30.599
Daniel Ruby: quick thumbs up from now.

19
00:01:32.870 --> 00:01:33.949
Niall Murphy: It ain't rain. Them.

20
00:01:34.900 --> 00:01:36.780
Mandi Walls: Gorgeous weather here. So yeah.

21
00:01:36.780 --> 00:01:38.989
Daniel Ruby: It's nice here, too.

22
00:01:39.190 --> 00:01:40.180
Troy Koss: Clear skies.

23
00:01:41.320 --> 00:01:47.100
Daniel Ruby: Yeah, it's weird. We've we had started shifting my daughter's clothes over to winter clothes, and

24
00:01:47.190 --> 00:01:50.180
Daniel Ruby: all of a sudden we got 2 days in a row where it's

25
00:01:50.290 --> 00:01:54.260
Daniel Ruby: 78, 79 degrees, and she's like, I don't want to cook at school, Daddy.

26
00:01:55.900 --> 00:01:58.595
Daniel Ruby: Yeah. Well, I don't blame you, sweetheart

27
00:02:00.150 --> 00:02:02.320
Daniel Ruby: alright. So we got about 2 min

28
00:02:03.000 --> 00:02:05.809
Daniel Ruby: 2 more minutes. We're gonna let people

29
00:02:06.260 --> 00:02:08.520
Daniel Ruby: get into the zoom.

30
00:02:09.609 --> 00:02:16.260
Daniel Ruby: I noted earlier. We're going to be officially kicking off the conversation at 1 0, 4 Eastern time

31
00:02:18.230 --> 00:02:22.109
Daniel Ruby: you are in the right place. You are with the right people.

32
00:02:29.300 --> 00:02:32.879
Niall Murphy: Have you ever thought of doing affirmations, Daniel?

33
00:02:32.880 --> 00:02:35.010
Mandi Walls: A great voice for radio there, for sure.

34
00:02:35.010 --> 00:02:36.330
Daniel Ruby: I. Actually.

35
00:02:36.840 --> 00:02:39.449
Daniel Ruby: I actually was a TV journalist for a while.

36
00:02:39.450 --> 00:02:40.060
Mandi Walls: Wow!

37
00:02:40.780 --> 00:02:46.669
Troy Koss: It's just so impressive every time I watch the news, or I watch journalists, and I'm just

38
00:02:46.710 --> 00:02:51.989
Troy Koss: I'm always amazed. Just sounds so clean and concise and crisp.

39
00:02:52.480 --> 00:02:53.100
Troy Koss: I would

40
00:02:54.100 --> 00:02:55.090
Troy Koss: I would never make it.

41
00:02:55.280 --> 00:02:57.637
Daniel Ruby: They they beat it into you.

42
00:02:58.780 --> 00:03:04.040
Daniel Ruby: generic midwestern accent, enunciate all that fun, stuff!

43
00:03:05.010 --> 00:03:06.260
Daniel Ruby: Brutal career!

44
00:03:10.250 --> 00:03:11.250
Daniel Ruby: Oh, yes.

45
00:03:11.600 --> 00:03:15.999
Daniel Ruby: this is the right place. You are with the right people. We are going to

46
00:03:16.010 --> 00:03:20.059
Daniel Ruby: discuss the right things today on affirmations

47
00:03:20.590 --> 00:03:23.040
Daniel Ruby: with the reliability crew.

48
00:03:24.450 --> 00:03:31.040
Mandi Walls: Reliability affirmations actually sounds like a pretty good podcast series. I'm gonna have to like, maybe steal that from you, for sure.

49
00:03:31.040 --> 00:03:35.749
Daniel Ruby: Yeah, I if if you need somebody to record like the intro and stuff.

50
00:03:35.980 --> 00:03:36.500
Mandi Walls: Yeah.

51
00:03:36.500 --> 00:03:37.480
Daniel Ruby: Happy to do it.

52
00:03:39.410 --> 00:03:40.350
Daniel Ruby: I love your license.

53
00:03:40.350 --> 00:03:41.120
Alex Nauda: It's good.

54
00:03:42.390 --> 00:03:43.580
Mandi Walls: Oh, thanks!

55
00:03:43.910 --> 00:03:46.039
Mandi Walls: That's my handle everywhere, too. So.

56
00:03:46.040 --> 00:03:46.800
Daniel Ruby: Nice.

57
00:03:48.170 --> 00:03:49.199
Mandi Walls: You can always find me.

58
00:03:50.120 --> 00:03:51.220
Daniel Ruby: All right.

59
00:03:51.410 --> 00:03:52.320
Daniel Ruby: Well.

60
00:03:52.510 --> 00:03:54.660
Daniel Ruby: it is 1 0, 4

61
00:03:54.820 --> 00:03:57.189
Daniel Ruby: Eastern time, and as promised.

62
00:03:58.130 --> 00:04:00.429
Daniel Ruby: let's get this shindy kicked off

63
00:04:01.139 --> 00:04:05.730
Daniel Ruby: good morning. Afternoon evening wherever you are in the world.

64
00:04:06.320 --> 00:04:12.909
Daniel Ruby: Welcome to this roundtable discussion on implementing slos, the build versus buy debate.

65
00:04:13.280 --> 00:04:15.450
Daniel Ruby: My name is Dan Ruby.

66
00:04:15.750 --> 00:04:19.439
Daniel Ruby: I am the I'm the Vp. Of marketing at Noble 9,

67
00:04:19.510 --> 00:04:23.959
Daniel Ruby: where, you know, we've been bombarding you with emails about this

68
00:04:24.090 --> 00:04:26.459
Daniel Ruby: webinar. So you probably know what we do?

69
00:04:28.130 --> 00:04:30.519
Daniel Ruby: I'd like to thank

70
00:04:30.760 --> 00:04:37.090
Daniel Ruby: everybody who's joined, and especially thank our esteemed panel. Go around and introduce everybody.

71
00:04:37.516 --> 00:04:42.390
Daniel Ruby: We have Niall Murphy, who is the co-founder and CEO of stanza.

72
00:04:43.150 --> 00:04:47.199
Daniel Ruby: We have Mandy walls, devops advocate at pager duty.

73
00:04:47.971 --> 00:04:56.450
Daniel Ruby: Troy Koss, who is the director of Reliability software engineering at a major financial institution, and Alex Nada

74
00:04:56.510 --> 00:04:59.049
Daniel Ruby: Noble nine's own. CTO.

75
00:04:59.728 --> 00:05:05.109
Daniel Ruby: Thank you, everybody for joining us, and let's get it kicked off. So I'm gonna

76
00:05:06.590 --> 00:05:08.669
Daniel Ruby: the the baseline of this

77
00:05:08.790 --> 00:05:13.429
Daniel Ruby: whole conversation is that people are realizing more and more that slos

78
00:05:14.050 --> 00:05:18.989
Daniel Ruby: are not a nice to have. They're not a feature. They're kind of a core requirement

79
00:05:19.170 --> 00:05:25.330
Daniel Ruby: for a modern user centric reliability strategy.

80
00:05:26.370 --> 00:05:39.540
Daniel Ruby: So the discussion then becomes, well, how do we accomplish that? There's a few different ways to do it? You can, you can try, and you can build your own internally if you've got an engineer. If you've got engineering

81
00:05:40.360 --> 00:05:44.780
Daniel Ruby: staff that you can allocate to it. You can hire

82
00:05:45.080 --> 00:05:49.939
Daniel Ruby: software like Noble 9, a dedicated slo platform where you can use

83
00:05:50.646 --> 00:05:52.990
Daniel Ruby: potentially the like, a

84
00:05:53.330 --> 00:05:57.299
Daniel Ruby: a feature, an slo feature in one of your existing

85
00:05:57.420 --> 00:05:59.310
Daniel Ruby: telemetry

86
00:06:00.470 --> 00:06:02.030
Daniel Ruby: pieces of software?

87
00:06:02.940 --> 00:06:03.610
Daniel Ruby: But

88
00:06:04.510 --> 00:06:13.810
Daniel Ruby: how do you make that decision? How do you decide whether you're going to build it or you're going to buy it. And I'm going to go to Niall 1st with with our opening question, which is Niall.

89
00:06:14.130 --> 00:06:18.910
Daniel Ruby: when you start thinking about implementing slos in your organization.

90
00:06:19.180 --> 00:06:22.620
Daniel Ruby: what do you really need to consider 1st

91
00:06:22.680 --> 00:06:25.459
Daniel Ruby: in the build versus buy debate.

92
00:06:26.480 --> 00:06:37.670
Niall Murphy: Yeah, so build versus by theory. If I can start off with that phrase, kind of rests on a bunch of underlying generic concerns. So 1st of all, if you're gonna build.

93
00:06:37.680 --> 00:06:57.169
Niall Murphy: do you have the capability to build like, do you have software engineers in your organization who can tackle this problem. And there's a bunch of nuances, shall I say very carefully to the question of Slo software, not all of which are necessarily

94
00:06:57.170 --> 00:07:20.559
Niall Murphy: clear to people before they embark on the task of writing that software kind of a yes, I will write a Twitter competitor over the weekend. I'm not too busy kind of thing. And so but I mean, that's kind of the case with more or less any business problem that there's some question of how complicated it is versus the capabilities of your organization. But of course, also, what else could those engineers be doing versus

95
00:07:20.890 --> 00:07:26.729
Niall Murphy: working on implementing this particular thing, which is kind of the opportunity cost question

96
00:07:26.800 --> 00:07:27.960
Niall Murphy: to

97
00:07:27.990 --> 00:07:33.630
Niall Murphy: that tends to make its way into business discussions as is this core to your business

98
00:07:33.910 --> 00:07:50.710
Niall Murphy: is doing Slo software, something that you are going to noodle on, maybe to the exclusion of a bunch of other stuff, or in some sense going to be a core pillar to the things you do? Or is it something that actually, at the end, at the end of the day you would buy in?

99
00:07:50.900 --> 00:08:01.990
Niall Murphy: And for some folks, I suppose that discussion would end up on one side or the other side, depending. One analogy would be, do you like to buy your coffee from

100
00:08:02.230 --> 00:08:06.549
Niall Murphy: a vendor, or do you have your own coffee machine or something like that. Okay, fine.

101
00:08:06.630 --> 00:08:29.820
Niall Murphy: There's a bunch of other criteria as well. So I think business core is important. Also, the question of scale and growth, because it's 1 thing to hack together a bunch of scripts for. Okay, I've like 50 slos, maybe, and that's probably within the scope of a for loop in bash, or like python or something. Okay? Fine.

102
00:08:29.940 --> 00:08:38.489
Niall Murphy: But if you have potentially a growing set of services and a growing set of data. The answer at one stage of

103
00:08:38.530 --> 00:08:41.779
Niall Murphy: your evolution might be, might change

104
00:08:41.960 --> 00:08:46.999
Niall Murphy: at another stage of your evolution, and you have to keep that potential evolution in mind as well.

105
00:08:47.380 --> 00:09:06.139
Niall Murphy: And there's also kind of cost as well, because one of the interesting things about the build versus buy is that in most organizations the costs of build are not fully articulated because they are seen mostly under the heading of Software engineer salary

106
00:09:06.160 --> 00:09:14.310
Niall Murphy: rather than actual opportunity cost. As I said previously, or maintenance costs

107
00:09:14.410 --> 00:09:28.870
Niall Murphy: which are hugely important, and in some cases dominating over time, like they're the dominating term of what you do over time. But again, that depends on questions like surrounding growth, and so on. So those are some of the factors that go in towards it.

108
00:09:31.990 --> 00:09:36.720
Daniel Ruby: Anybody wanna add anything to that? Other factors that they think are

109
00:09:37.160 --> 00:09:40.049
Daniel Ruby: really need to be top of mind when you're making this decision.

110
00:09:42.730 --> 00:09:44.489
Alex Nauda: I can tack something on there.

111
00:09:45.300 --> 00:09:48.519
Alex Nauda: I'm kind of in a funny situation, because, on the one hand.

112
00:09:48.960 --> 00:09:56.199
Alex Nauda: I am CTO of a software company that's that's selling that would love our customers to buy the product right. But, on the other hand.

113
00:09:56.630 --> 00:09:59.350
Alex Nauda: I'm leading a team that's building one.

114
00:09:59.600 --> 00:10:02.929
Alex Nauda: So I myself am very much in the build camp

115
00:10:03.790 --> 00:10:05.779
Alex Nauda: for Slos in particular.

116
00:10:05.990 --> 00:10:07.130
Alex Nauda: and

117
00:10:07.400 --> 00:10:08.830
Alex Nauda: I want to point out

118
00:10:08.980 --> 00:10:10.679
Alex Nauda: some of the things

119
00:10:10.940 --> 00:10:13.679
Alex Nauda: about an Slo product

120
00:10:13.920 --> 00:10:14.960
Alex Nauda: that

121
00:10:15.050 --> 00:10:16.249
Alex Nauda: maybe you know.

122
00:10:16.310 --> 00:10:21.350
Alex Nauda: if I had known what this would look like 5 years ago, when we started, this company might have

123
00:10:21.780 --> 00:10:24.579
Alex Nauda: made me pause. One of those is

124
00:10:25.250 --> 00:10:26.820
Alex Nauda: the degree to which

125
00:10:27.520 --> 00:10:41.500
Alex Nauda: slos as a business practice, not calculating an slo. I think I can paraphrase Troy, who said, anyone can calculate some nines right? Anyone can divide 2 numbers and and get a fraction of one. That part's not the hard part.

126
00:10:41.550 --> 00:10:44.180
Alex Nauda: I think I underestimated the degree to which

127
00:10:45.020 --> 00:10:48.239
Alex Nauda: slos are a service provided to

128
00:10:48.360 --> 00:10:54.330
Alex Nauda: the org that's using them, whether it's an infrastructure, engineering or platform engineering.

129
00:10:54.640 --> 00:10:58.299
Alex Nauda: broad community of application development service, owning teams.

130
00:10:58.800 --> 00:11:05.930
Alex Nauda: we're constantly going in to fix things or adapt to things, and I think it's important to look at

131
00:11:06.010 --> 00:11:13.470
Alex Nauda: what it takes to build something, and then also what it, what it takes to run it also kind of harking back to what? What Niall just said.

132
00:11:13.930 --> 00:11:16.490
Alex Nauda: That's a big piece of it. When you're

133
00:11:16.560 --> 00:11:27.310
Alex Nauda: observability system goes down, or there's some breakage or discontinuity in your data. How do you actually go and repair it? And I've been consistently surprised at how much this is a service.

134
00:11:27.630 --> 00:11:30.619
Alex Nauda: This requires service customer service, or or

135
00:11:30.890 --> 00:11:33.490
Alex Nauda: sometimes data service.

136
00:11:34.227 --> 00:11:37.210
Alex Nauda: Hosting of the slos versus simply

137
00:11:37.270 --> 00:11:39.269
Alex Nauda: the functionality to create them.

138
00:11:41.090 --> 00:11:43.020
Alex Nauda: Yeah, it's real eye opener for me. It.

139
00:11:43.020 --> 00:11:48.229
Troy Koss: And I like that point, too. And and I, if I were to like overly summarize, I would say.

140
00:11:48.430 --> 00:12:03.359
Troy Koss: consider all factors right, not just the upfront like, build the tool, do the the math. It's it's all these other things that you're hearing like Alex. And I'll both mentioned. It's like a. It's a complete package. When you have to make the evaluation. It's it's everything from the

141
00:12:04.203 --> 00:12:17.229
Troy Koss: labor to do it. The infrastructure and and resources to actually consume and and build it and then the the support of it. And I think when you go back to the the core of it's like, why are we doing this at all? It's.

142
00:12:17.340 --> 00:12:19.440
Troy Koss: you know, in the case of Slos, it's still like

143
00:12:19.570 --> 00:12:49.220
Troy Koss: measure service reliability, and do that. It's more than just the math. It's like, Are we gonna actually achieve the outcome? And do you have the expertise to do that? And that's a lot a lot of times like, if you have, you have to consider. Do you have that resource? Not even just building the software like, do you have somebody that knows the domain enough to evangelize it, to push it, to educate teams, to support teams, to help deal with nuance edge cases and crazy data possibilities from source providers. In this case, you know. The the. It's a full ecosystem to consider.

144
00:12:49.220 --> 00:12:54.579
Mandi Walls: Yeah. And to that point, too, like a lot of places are sort of lacking the the discipline to have like

145
00:12:54.620 --> 00:12:56.979
Mandi Walls: what would be like an internal

146
00:12:57.020 --> 00:13:02.659
Mandi Walls: product for us, kind of product management. Right is you're committing to something that is going to be

147
00:13:03.290 --> 00:13:28.530
Mandi Walls: reliability for your production environment. That means it should hopefully be long lived, and take into account new and changing requirements based on the teams that you're onboarding to it. And all these other things, and like having the discipline to treat your internal production tools as a product and product, manage them effectively over time can be difficult, and some folks don't have the the personnel to do that as well.

148
00:13:31.750 --> 00:13:32.590
Daniel Ruby: So

149
00:13:33.580 --> 00:13:35.720
Daniel Ruby: when you do think about

150
00:13:36.410 --> 00:13:41.120
Daniel Ruby: when you're in this situation of making a build versus buy decision.

151
00:13:41.300 --> 00:13:45.680
Daniel Ruby: you're kicking off slos as this net new

152
00:13:45.770 --> 00:13:46.970
Daniel Ruby: initiative.

153
00:13:48.850 --> 00:13:51.130
Daniel Ruby: and I've heard

154
00:13:51.460 --> 00:13:52.880
Daniel Ruby: stories.

155
00:13:53.380 --> 00:14:01.749
Daniel Ruby: plenty of stories just being at Noble 9 for less than a year about. You know what we tried to do slos.

156
00:14:01.760 --> 00:14:03.539
Daniel Ruby: but it didn't work.

157
00:14:03.630 --> 00:14:06.959
Daniel Ruby: and we don't want to try it again. How much.

158
00:14:09.010 --> 00:14:10.959
Daniel Ruby: I guess. How much

159
00:14:11.250 --> 00:14:11.995
Daniel Ruby: is

160
00:14:13.800 --> 00:14:16.630
Daniel Ruby: the build versus buy discussion.

161
00:14:16.950 --> 00:14:19.275
Daniel Ruby: almost driven by

162
00:14:21.540 --> 00:14:27.509
Daniel Ruby: creating capacity, creating the ability to socialize and to get

163
00:14:27.680 --> 00:14:31.090
Daniel Ruby: to, to encourage adoption of slos.

164
00:14:33.270 --> 00:14:36.330
Troy Koss: Well, that's a full time, job. Oh, go ahead! Go ahead now.

165
00:14:36.330 --> 00:14:38.600
Niall Murphy: No, I didn't mean to interrupt at all.

166
00:14:38.600 --> 00:14:42.889
Troy Koss: I was. Gonna say, like that, that is the that goes back to like the

167
00:14:44.630 --> 00:15:12.539
Troy Koss: the, the outcome you're trying to achieve. Right? It's not to build software right? It's to use the software to achieve the means or an outcome. And and as long as like, you are thinking about, how does that actually work like in the case of Slos? It's a it's a big paradigm shift for people like in in most enterprises you're looking at. And the companies you're saying like, is it working or not working well, like? What about all the various degrees in between right and like? That's what the the metrics allow you to have

168
00:15:12.814 --> 00:15:17.749
Troy Koss: but in order to do that it's a big shift. It's a cultural shift. It's a mindset shift

169
00:15:17.750 --> 00:15:27.929
Troy Koss: for teams and to to adopt, and you have to have you know, a a body of folks that are willing to go out there and help teach and get people on board.

170
00:15:31.170 --> 00:15:31.910
Daniel Ruby: I don't.

171
00:15:32.400 --> 00:15:39.409
Niall Murphy: Yeah. I was just going to say, I'm I'm kind of in the devops camp on this one, which is to say

172
00:15:39.450 --> 00:15:45.429
Niall Murphy: that there is a complex relationship between people culture and tooling.

173
00:15:45.470 --> 00:16:11.600
Niall Murphy: and especially in the context of shall we say an slo implementation failure? You can often use new tools to kind of break people's previous historic assumption of failure or followed failure. If you see what I mean, and particularly if the tool comes from an outside place. I say that kind of with the vendor hat on, but also

174
00:16:11.940 --> 00:16:26.949
Niall Murphy: from the experience of being in a particular organization that failed to do X, and we bought an external tool to do X, and, as it happens, we failed in a different way. But the conversations changed

175
00:16:26.990 --> 00:16:34.940
Niall Murphy: such that the organization came to understand that it was actually not necessarily their failure or kind of a basic

176
00:16:35.020 --> 00:16:48.269
Niall Murphy: competence question, but was more kind of situational. And, in fact, the situation changed later on, and the tooling helped to kind of unlock people's mindset around things. So it can be a useful

177
00:16:48.630 --> 00:16:55.129
Niall Murphy: tactic when you're thinking about implementing, even if it doesn't actually end up solving the

178
00:16:55.330 --> 00:16:56.549
Niall Murphy: bulk of the problem.

179
00:16:57.480 --> 00:16:58.310
Daniel Ruby: Interesting.

180
00:16:58.310 --> 00:17:21.429
Mandi Walls: And I think there, too, there's a bit of a potential carrot and stick model that changes a little bit. If you have bought something. You've signed a contract. You've bought seats. You have some kind of impetus to ensure that those seats are being used since you're paying for them right? Or if not, then you're gonna cancel the whole project, maybe, and cancel the whole contract right or not renew it. So there is a certain amount of

181
00:17:21.490 --> 00:17:40.029
Mandi Walls: timeliness that comes into a purchase that folks don't necessarily feel when it's just. Oh, well, the guy in next cube writes this, and whenever we get to it, we get to it, and it's not maybe the same amount of urgency that comes into it rather than oh, we we bought this thing we paid for, we better use it.

182
00:17:40.360 --> 00:17:40.940
Daniel Ruby: Yeah.

183
00:17:41.880 --> 00:17:42.640
Daniel Ruby: it's

184
00:17:44.250 --> 00:17:46.040
Daniel Ruby: to to my mind.

185
00:17:46.070 --> 00:17:51.599
Daniel Ruby: I I feel like when when you're talking about slos, because they are still

186
00:17:52.740 --> 00:17:58.349
Daniel Ruby: somewhat of an unknown for a lot of organizations, you know you need them, but you don't necessarily know

187
00:17:59.360 --> 00:18:00.410
Daniel Ruby: what

188
00:18:00.580 --> 00:18:02.550
Daniel Ruby: they should look like.

189
00:18:03.050 --> 00:18:06.590
Daniel Ruby: How big of a challenge is it when building

190
00:18:06.710 --> 00:18:08.860
Daniel Ruby: when building out an slo

191
00:18:09.980 --> 00:18:10.920
Daniel Ruby: feature

192
00:18:11.140 --> 00:18:15.879
Daniel Ruby: to define what it should look like and what it should do, and how it should be.

193
00:18:17.640 --> 00:18:18.400
Daniel Ruby: kind of

194
00:18:18.990 --> 00:18:20.530
Daniel Ruby: engaged with.

195
00:18:23.310 --> 00:18:23.950
Troy Koss: It's really hard.

196
00:18:23.950 --> 00:18:24.969
Alex Nauda: Have a comment.

197
00:18:25.283 --> 00:18:27.270
Alex Nauda: I have a comment on this one.

198
00:18:29.450 --> 00:18:33.320
Alex Nauda: no look. We see this a lot, because often

199
00:18:34.750 --> 00:18:48.630
Alex Nauda: noble 9 is brought into, let's say, a proof of concept or a trial where a group has been working with slos, using some other product, or or often their own, their own internal tooling or their own internal mechanism for calculating.

200
00:18:48.990 --> 00:18:55.350
Alex Nauda: And one of the things I see over and over again is one of the things that can be challenging is

201
00:18:55.930 --> 00:19:11.290
Alex Nauda: if you're doing a custom build, or if you're starting trying to start small, one of the things you're gonna try to do is constrain the scope of what it is that you put in place whatever features you're going to build. So you might say, like, Oh, I'm only gonna use one or 2 data sources, and I'm gonna ignore all the other. You know.

202
00:19:11.790 --> 00:19:16.490
Alex Nauda: 5 or a dozen other data sources. Maybe some custom stuff.

203
00:19:16.550 --> 00:19:30.239
Alex Nauda: You might constrain the capabilities, you might say. Well, I'm gonna make all of my slos time based, like count good minutes versus bad minutes, or make them all occurrences based, which is sort of the straight ratio of good events versus bad events.

204
00:19:30.815 --> 00:19:34.839
Alex Nauda: You might make them all calendar, aligned like month by month.

205
00:19:34.940 --> 00:19:35.870
Alex Nauda: and

206
00:19:36.120 --> 00:19:44.819
Alex Nauda: not recognizing the fact that maybe some teams want a 2 week rolling slo or quarter long rolling slo, whatever it is. And

207
00:19:45.230 --> 00:19:54.659
Alex Nauda: I think it's a good valuable trick that if you can constrain the complexity and pick a smaller scope, you can make it work. But eventually that could hit some kind of a

208
00:19:55.280 --> 00:19:56.530
Alex Nauda: limitation.

209
00:19:57.590 --> 00:19:59.270
Alex Nauda: That's 1 of the

210
00:19:59.580 --> 00:20:05.680
Alex Nauda: challenges that I significantly see when when people get started. And I would say, I I felt this

211
00:20:06.000 --> 00:20:07.250
Alex Nauda: over the

212
00:20:07.310 --> 00:20:09.165
Alex Nauda: over the years like

213
00:20:09.810 --> 00:20:11.670
Alex Nauda: if I had realized

214
00:20:12.190 --> 00:20:28.379
Alex Nauda: the breadth of features you need just in the math. That's just in the basic calculating the nines kind of thing dealing with missing data, dealing with sparse data, dealing with extremely frequent data, trying to find the right way to aggregate it or histogram it, to get that to get a calculation that makes sense.

215
00:20:28.640 --> 00:20:36.210
Alex Nauda: But what those service owners are looking for. If I had realized the complexity at 1st I might have run away screaming, but

216
00:20:36.610 --> 00:20:50.070
Alex Nauda: but we at the same time we do see a lot of teams successful with slos, and then if they come to us, they know what they're looking for that, you know, it does serve to kinda crystallize what those requirements really are.

217
00:20:50.772 --> 00:20:56.330
Alex Nauda: Because they've tried to actually put it into play. And I think that's that's the other important consideration is.

218
00:20:56.560 --> 00:21:01.500
Alex Nauda: it's important to get started to clarify those requirements, especially with something as

219
00:21:01.620 --> 00:21:05.890
Alex Nauda: empirical, pragmatic as Slos. Right?

220
00:21:05.910 --> 00:21:10.090
Alex Nauda: There's a big difference between the theory and actually then calculating the slo. And

221
00:21:10.750 --> 00:21:17.380
Alex Nauda: oh, no! One of our data sources is just down or changed last month. And how do we roll with that? Right?

222
00:21:17.560 --> 00:21:20.170
Alex Nauda: That's that's what you discover along the journey.

223
00:21:20.900 --> 00:21:23.280
Troy Koss: Yeah, I agree. I I think, like, when you

224
00:21:23.570 --> 00:21:26.309
Troy Koss: we think about like slos in particular, it's

225
00:21:26.670 --> 00:21:55.589
Troy Koss: it's not that it's like an unknown or not really well defined, like body of of like, of knowledge or domain. But people people have been measuring success rate and putting targets on it for a long time. Right? This is this, there's no like a groundbreaking thing there. But when you start looking at how systems have success rates, and what are we measuring success rate of and, like you, mentioned all these nuanced things? Part of the benefit, though of like a build solution, is that you have the opportunity to

226
00:21:55.660 --> 00:22:19.710
Troy Koss: actually learn something, you know, like the saying, like, you learn more by doing like it. You can. You can read a million books, you can really. But when you start building your own, you really start understanding the math. And what's the difference between rolling versus not and calendar versus not? And what about exclusion windows? And what about, you know? Outage windows like these all become things that you learn about and have to build yourself to understand.

227
00:22:19.710 --> 00:22:26.589
Troy Koss: And I think what you alluded to is, you have a lot. You have a lot of customers that come to you and say, Oh, well, like, we know exactly what we want now, and

228
00:22:26.590 --> 00:22:31.939
Troy Koss: does it do this thing or not, and and and does right. I think the

229
00:22:32.160 --> 00:22:37.809
Troy Koss: the this, the what this is alluding to. And we talked about this before. And one of our things is that

230
00:22:38.130 --> 00:22:43.399
Troy Koss: the it's not a permanent decision either. Right like you can very well

231
00:22:43.420 --> 00:22:58.009
Troy Koss: proof of concept, and do a lot of R&D, and start building your own stuff to like, see it out and test the things out, to learn and to educate yourselves and then buy right like that's that's always a model. It's not a finite thing, or or, inversely.

232
00:22:58.030 --> 00:23:15.050
Troy Koss: you could use a vendor to get you bootstrapped and understood, understand the the domain, and then say, well, we actually know enough about it, and we have enough nuance. That in particular stuff for us as proprietary that the vendors are going to make. Therefore we can build our own solution, too. So there's many avenues and ways this could roll out.

233
00:23:18.190 --> 00:23:18.540
Daniel Ruby: So

234
00:23:19.800 --> 00:23:28.019
Daniel Ruby: I kind of want to go to to each of you for a moment and just ask you your own experience in implementing slos

235
00:23:29.490 --> 00:23:31.149
Daniel Ruby: when you made.

236
00:23:31.310 --> 00:23:34.369
Daniel Ruby: or we're involved in a build versus buy

237
00:23:34.510 --> 00:23:40.940
Daniel Ruby: discussion. You know. What were you looking for? What were the deciding factors? And do you

238
00:23:41.350 --> 00:23:46.850
Daniel Ruby: stick by your decision, or do you think you would have done it another way? I'll start with Mandy.

239
00:23:47.730 --> 00:24:01.649
Mandi Walls: And I was gonna say, oh, we should have invited our staff sre to this discussion because they were the ones who sort of put this together for us, and we are noble 9 customers now. And and these things are starting to roll out internally. And yeah, like.

240
00:24:02.310 --> 00:24:11.309
Mandi Walls: we have our own engineers in service to our product right? That is what we need them to be doing, and where we need that those

241
00:24:11.330 --> 00:24:32.789
Mandi Walls: precious, expensive resources to be working is on new features for our customers and focused on that. And we do buy a lot of other products that some places might do a little bit more of their own homegrown things. We do have some homegrown stuff that that has been around very long time. When the team was smaller, the product was smaller. The customer base was

242
00:24:32.790 --> 00:24:49.839
Mandi Walls: maybe not as demanding as some of our customers are now, but as the company matures, and you get to the point where, you know, you're a public company. There are things that we need to be doing in service to our customers rather than things that we'd like to

243
00:24:50.180 --> 00:24:58.099
Mandi Walls: build cool stuff internally, that isn't really in service to our product. So that is, that's kind of where we're sitting right now, with with a lot of our tooling.

244
00:24:59.680 --> 00:25:01.069
Daniel Ruby: Niall, how about yourself?

245
00:25:01.820 --> 00:25:19.579
Niall Murphy: So you must understand that prior to my most recent startup journey, I spent the bulk of my professional career in Amazon, Google, and Microsoft, all of whom, without exception, never saw a line of code they didn't want to rewrite. So when you say, Well, actually, I'll

246
00:25:19.640 --> 00:25:24.480
Niall Murphy: changed my position a little bit. Amazon did actually buy some software and had a very

247
00:25:24.630 --> 00:25:31.249
Niall Murphy: proud ethos around the. There shall be nothing except Rpc. Diktat

248
00:25:31.300 --> 00:25:33.199
Niall Murphy: that folks were very

249
00:25:33.972 --> 00:25:43.699
Niall Murphy: you. Ideological is kind of the wrong word, but ideological about being able to put some different implementation behind an Rpc. Layer, whatever

250
00:25:43.800 --> 00:26:04.540
Niall Murphy: kind. So. But I think it definitely holds for Google and Microsoft. And so you will be unsurprised to hear that the Slo Repository software in Google, which I think actually, if I recall correctly a long while ago originated in my org, or at least one bit of it did. And then the

251
00:26:04.570 --> 00:26:28.360
Niall Murphy: the Microsoft situation was, they have a tool, internally facing tool that helps to manage all of the azure services. And it's kind of a classic and corporate scorecard kind of management tool. So every service gets assessed under a bunch of headings, and they added slos as one of those headings, and so everyone had to

252
00:26:28.410 --> 00:26:37.039
Niall Murphy: put their slo in a specific place and report their slo according to that performance, and so on. So relentlessly build

253
00:26:37.270 --> 00:26:42.119
Niall Murphy: folks, is where where the bulk of my history is.

254
00:26:44.520 --> 00:26:46.649
Daniel Ruby: Troy. I know you've got an interesting

255
00:26:47.040 --> 00:26:49.109
Daniel Ruby: story around this, too.

256
00:26:49.500 --> 00:26:50.490
Daniel Ruby: Oh.

257
00:26:50.830 --> 00:26:56.889
Troy Koss: Yeah, I think there's a lot of factors that have come into like my analysis and and and just

258
00:26:56.960 --> 00:27:01.243
Troy Koss: the situation you're in. I think you know, when I was at

259
00:27:01.770 --> 00:27:24.690
Troy Koss: a telecom a long time ago? Well, feels like a long time ago. This wasn't that long ago. You know you. You look at some of the stuff that this kind of a full circle moment for what Nile mentioned. But what kind of staff do you have like, who who can build this? And when you're looking at a a vendor solution, you're saying, well, like, can can the like? Can it meet our particular needs. And typically, what I found is that

260
00:27:25.590 --> 00:27:53.340
Troy Koss: you think you're unique. And you think we have all these like crazy things that are, you know, proprietary to yourself, and and then you find and then you find out like, maybe we shouldn't be doing that. Maybe that's the problem. And then you try to evaluate you evaluate. So I but I think it's like, when you start making the decision to really understand what you're trying to solve first, st before you even do a build by versus the conversations like, what? What are we trying to solve with the thing we're trying to build or buy because you have to either build. You have to either write requirements

261
00:27:53.340 --> 00:28:16.720
Troy Koss: to build it yourself, or you have to have some sort of requirements that you're measuring. If you're doing an Rfp. Or something like that to get a product evaluation done right? So like, I guess, in a way, what I'm saying is like it really depends on the situation you're in at the time, and who's around you, and what you have to build, and maybe there's already something there that gives you a new opportunity or an edge. That's an existing

262
00:28:17.090 --> 00:28:44.310
Troy Koss: bit of software that you can build on top of or like in the example. And I mentioned, if you're building, and you have an existing scorecard mechanism like the the example he just mentioned, where plugging slos into it is really just the right thing to do. Maybe you maybe you can use a vendor tool to do the engine and then port the data into something else. Or maybe it makes sense to build it yourself because you already have a foundation of good software quality software that's there. You know, I think that.

263
00:28:44.887 --> 00:28:48.999
Troy Koss: It's it's pretty varied in my experience has been

264
00:28:49.620 --> 00:28:50.580
Troy Koss: that

265
00:28:50.710 --> 00:28:52.840
Troy Koss: exactly that varied.

266
00:28:54.030 --> 00:28:55.060
Daniel Ruby: That's interesting.

267
00:28:57.350 --> 00:29:00.661
Daniel Ruby: now, you you said something interesting there, Troy, about

268
00:29:01.640 --> 00:29:08.509
Daniel Ruby: kind of bringing in a vendor to do, to to be an engine to something that you build yourself. So

269
00:29:08.520 --> 00:29:12.460
Daniel Ruby: I mean to. To that point. The build versus buy is not a purely binary

270
00:29:12.910 --> 00:29:14.060
Daniel Ruby: decision.

271
00:29:14.290 --> 00:29:15.150
Daniel Ruby: You know.

272
00:29:15.950 --> 00:29:17.260
Daniel Ruby: I know that

273
00:29:17.430 --> 00:29:19.849
Daniel Ruby: at Noble 9 we've got clients who

274
00:29:20.680 --> 00:29:27.130
Daniel Ruby: purchase us for certain features, and then use those features to feed their own internally built tools.

275
00:29:32.270 --> 00:29:34.630
Troy Koss: Yeah, I think that's a i think that's a that's that's

276
00:29:34.690 --> 00:29:43.099
Troy Koss: I'm glad you called that out because I mentioned earlier, like how you don't. It's not one or the other. It could be both at some point in your journey, but it also could be both the whole time

277
00:29:43.100 --> 00:30:08.090
Troy Koss: right. There's very well. You could build around the tool or leverage its core strength. I think, when I'm looking, when you're looking in different solutions that are out there, right? Maybe there's a bundled offering, and you really have 20 things that they offer. But you really only need like one part of that thing. You know one is, can you? Can you only purchase that one thing, and not to pay for the whole thing that you're not going to use, but also can. Can you just use it behind the scenes of your

278
00:30:08.090 --> 00:30:13.400
Troy Koss: system? You know, at Microsoft? Could they have used the example that Niall use? Could could they have used.

279
00:30:13.933 --> 00:30:21.970
Troy Koss: noble 9 as the engine to compute the slos and adjust the data into that calculation and stuff, and then put that and read that Api out and put it on a scorecard.

280
00:30:22.150 --> 00:30:33.890
Troy Koss: Why not? Right? I mean that it really again it comes back to what you're trying to trying to accomplish, and maybe that saves a lot of your labor costs the hard part. The stuff that Alex mentioned, the all the math and stuff that you guys do.

281
00:30:37.560 --> 00:30:38.210
Daniel Ruby: Yeah.

282
00:30:39.370 --> 00:30:42.620
Daniel Ruby: we've we've had a lot of mentions of.

283
00:30:42.890 --> 00:30:43.800
Daniel Ruby: you know.

284
00:30:44.760 --> 00:30:48.860
Daniel Ruby: I I think all of you at 1 point have talked about

285
00:30:49.230 --> 00:30:50.610
Daniel Ruby: maintenance

286
00:30:50.850 --> 00:30:52.850
Daniel Ruby: of a build.

287
00:30:55.240 --> 00:30:56.660
Daniel Ruby: So there's

288
00:30:57.210 --> 00:30:58.810
Daniel Ruby: it's not like

289
00:30:58.820 --> 00:31:00.510
Daniel Ruby: you build it, and you're done

290
00:31:01.000 --> 00:31:04.100
Daniel Ruby: so. How much would you say?

291
00:31:04.420 --> 00:31:08.070
Daniel Ruby: Kind of the the investment

292
00:31:09.240 --> 00:31:23.989
Daniel Ruby: in building versus buying or well, not versus buying in building is that initial, you know, scoping and building out, creating the software itself. And how much of it is the maintenance and the the continued

293
00:31:24.821 --> 00:31:28.179
Daniel Ruby: evolution of your kind of

294
00:31:28.220 --> 00:31:30.650
Daniel Ruby: in-house built. Slo solution.

295
00:31:34.020 --> 00:31:35.270
Daniel Ruby: Yeah, Nile.

296
00:31:35.810 --> 00:31:40.330
Niall Murphy: So. The generic answer to this is, of course, that

297
00:31:40.510 --> 00:31:47.390
Niall Murphy: the long term costs of maintenance for software are completely dominant over the short term costs

298
00:31:47.610 --> 00:32:05.939
Niall Murphy: something like. And I mean, of course, one of the issues with actually coming up with a number for this is evidence-based software engineering is a terrible, terrible field, and like no one has any idea what the actual numbers are, and measurement is hard, and all of those kind of caveats. But basically, if anything lasts

299
00:32:06.160 --> 00:32:16.159
Niall Murphy: for a decent enough time. Basically, the startup costs go to 10%. Maybe 0 and maintenance cost is 90%.

300
00:32:16.270 --> 00:32:19.449
Niall Murphy: Now, maybe that overall cost

301
00:32:19.520 --> 00:32:35.869
Niall Murphy: is quite low, if it's a bunch of hacked together scripts, or whatever, or whatever, or, conversely, maybe the maintenance costs are quite high. If it's a bunch of hacked together. Stuff! And it's like there are so many variables. It's it's hard to

302
00:32:36.100 --> 00:32:45.619
Niall Murphy: hard to come up with a definitive answer, but the short one is, if it lasts any decent duration of time, or if it's in any way important.

303
00:32:45.620 --> 00:33:10.279
Niall Murphy: you'll want to keep it alive under conditions of changing scale. Apis cloud providers, like all of the stuff that's happening. Maybe you'll show generative AI in there somewhere, I don't know. But, broadly speaking, the maintenance cost will come to dominate any other thing, and most people make Bill versus buy decisions on the complement set of the maintenance cost.

304
00:33:13.740 --> 00:33:18.109
Alex Nauda: I have numbers on this, because that's what our.

305
00:33:18.200 --> 00:33:29.359
Alex Nauda: That's what our software does. I would say, look at a high level. There's really 3 categories of spend. Okay, there's feature development which could be initial, but it also could be ongoing

306
00:33:30.717 --> 00:33:36.059
Alex Nauda: and then there's like what it cost to run it. So, you know, pay for the infrastructure to run it.

307
00:33:36.574 --> 00:33:40.030
Alex Nauda: Do the support, which is both sort of like

308
00:33:40.550 --> 00:33:43.809
Alex Nauda: operations support, but also customer support.

309
00:33:44.140 --> 00:33:45.540
Alex Nauda: And then there's

310
00:33:45.910 --> 00:33:47.310
Alex Nauda: what I'll call

311
00:33:47.876 --> 00:33:53.769
Alex Nauda: the math aspect, which I think is its own category, partly because we see not only that

312
00:33:53.840 --> 00:34:11.430
Alex Nauda: it's more nuanced than it appears on the surface. But also questions come up as soon as you put this thing in the line of fire of a management review and executive review, maybe customer facing, depending on your use case, you're going to have questions. Some 75 or 80% of

313
00:34:11.679 --> 00:34:14.060
Alex Nauda: customer support situations. Aren't

314
00:34:14.260 --> 00:34:20.970
Alex Nauda: you know what's the right setting. How do I do this there? My data doesn't seem to match why? And

315
00:34:21.330 --> 00:34:22.070
Alex Nauda: it

316
00:34:22.310 --> 00:34:30.849
Alex Nauda: usually does somehow match, but requires explanation and investigation. Or sometimes it doesn't match and has to be fixed, and that and that's really where the

317
00:34:30.860 --> 00:34:35.620
Alex Nauda: nuance comes in. Now, when you break them down, you said like how much of it is maintenance.

318
00:34:36.040 --> 00:34:42.540
Alex Nauda: I would include sort of like what it costs to run it as well as what it costs to sort of maintain and support it.

319
00:34:43.650 --> 00:34:44.830
Alex Nauda: I think it's.

320
00:34:45.000 --> 00:34:53.929
Alex Nauda: sir, by my measurement, may not be up to the 90% that Niall was saying. But it's over half okay. It's over half of the total cost.

321
00:34:54.159 --> 00:34:58.509
Alex Nauda: keeping something running that's probably gonna have to be. If you're measuring systems that have

322
00:34:58.960 --> 00:35:07.489
Alex Nauda: 3 nines, 4 nines, maybe even more than that of reliability. This thing has to be that accurate. It has to be up all the time. If you're using it for alerting

323
00:35:07.790 --> 00:35:14.779
Alex Nauda: hits your learning mechanism, it's gonna have to wake people up when everything else goes down. So it has to have some independence in hosting.

324
00:35:14.870 --> 00:35:16.540
Alex Nauda: It's really not trivial.

325
00:35:16.640 --> 00:35:17.670
Alex Nauda: you know.

326
00:35:18.379 --> 00:35:19.239
Alex Nauda: I think

327
00:35:19.310 --> 00:35:23.010
Alex Nauda: a lot of times we like to focus on the product features.

328
00:35:23.990 --> 00:35:28.440
Alex Nauda: high level categories of product features. You got to get data into it from all the relevant places

329
00:35:29.530 --> 00:35:45.900
Alex Nauda: you have to make a bunch of pretty charts and reports that are kind of might have to be tweaked and customized to the audience, and you have a couple of different audiences like your overnight incident responders versus your executive team or your product team, or something like that. You might have to export the data somewhere.

330
00:35:46.305 --> 00:35:49.259
Alex Nauda: You have to be able to like. Leave yourself notes

331
00:35:49.280 --> 00:35:58.449
Alex Nauda: on what happened back then, maybe repair the data somewhere in the past. You might have to restate Nslo at some point. Someone might ask you to go and backfill

332
00:35:58.590 --> 00:36:06.259
Alex Nauda: correct data if there was an inaccuracy of data, or do. What if type of analysis and compare versions between things?

333
00:36:07.950 --> 00:36:12.719
Alex Nauda: you might have permissioning issues. And that's sort of like this whole role, role-based access control

334
00:36:12.830 --> 00:36:24.859
Alex Nauda: regime to build into the thing. And it just get, you know, all that stuff just gets really complex, really quickly. And and a lot of it doesn't really have to do with the core of

335
00:36:25.390 --> 00:36:42.449
Alex Nauda: measuring reliability, you know. Like, if you're if you're kind of shuffling around what your permissions are, and can I do a group sync back to my active directory, or whatever whatever the thing is you're working on, it's not really core to the Slo problem or the reliability question right? And I think more than half of it falls into that

336
00:36:42.510 --> 00:36:48.849
Alex Nauda: sort of maintenance category by by my current assessment of where we are 5 years in.

337
00:36:51.250 --> 00:36:52.399
Alex Nauda: That's my hunch.

338
00:36:55.050 --> 00:36:55.770
Daniel Ruby: Now

339
00:36:56.980 --> 00:36:59.470
Daniel Ruby: in in your experience.

340
00:37:00.810 --> 00:37:04.099
Daniel Ruby: when when an organization starts thinking about

341
00:37:04.510 --> 00:37:06.559
Daniel Ruby: slos as a strategy.

342
00:37:07.690 --> 00:37:09.680
Daniel Ruby: how near term

343
00:37:11.270 --> 00:37:16.210
Daniel Ruby: do do they feel that this needs to be like, how fast

344
00:37:16.450 --> 00:37:19.709
Daniel Ruby: do they want to go from? Okay? We've got.

345
00:37:19.930 --> 00:37:22.850
Daniel Ruby: We've decided that slos need to be.

346
00:37:22.940 --> 00:37:27.740
Daniel Ruby: you know, a core component of our reliability strategy to

347
00:37:28.360 --> 00:37:33.790
Daniel Ruby: I. Why aren't my slos up and running and and giving me business value?

348
00:37:35.780 --> 00:37:36.760
Troy Koss: Immediate.

349
00:37:37.300 --> 00:37:38.800
Troy Koss: They want it yesterday.

350
00:37:39.372 --> 00:37:53.819
Troy Koss: It's like, with anything right like you want it. And then, all of a sudden, it's like the priority. I I think slos in particular, it really depends on the current state of the of you know of things, our.

351
00:37:53.880 --> 00:37:57.980
Troy Koss: how are you measuring reliability today? Are you using?

352
00:37:58.310 --> 00:38:06.820
Troy Koss: Incident counts as a measure? Are you using Mttr like, what? What is your current strategy, right or wrong, right and and

353
00:38:06.830 --> 00:38:19.530
Troy Koss: how much different is it when you know to a 10th of a percentage how many failures you've had versus before, right where you didn't like? It's a very that that I think that the simplicity of what it is

354
00:38:19.640 --> 00:38:25.539
Troy Koss: you'd think it doesn't have such a big impact, or would take so much time to adoption and actually

355
00:38:25.860 --> 00:38:49.830
Troy Koss: mechanize and use. But it's an astronomical investment to to switch over to using them consistently. If you weren't born using them right? Smaller, newer modern startups companies have the opportunity to start by measuring those things. But a lot of times they don't even do it because they're compromising that for speed. So they're not really. There's like, is it working? It's still up. Let's keep going. Add more features. Right?

356
00:38:50.137 --> 00:39:03.979
Troy Koss: You know. But it's a it's a huge investment. I think the expectation is that it's a quick and easy thing, and the reality is that it's not. And that's why, when you go into these into the discussion of build for spy. It's like.

357
00:39:04.100 --> 00:39:06.760
Troy Koss: consider the not just the tool.

358
00:39:06.770 --> 00:39:12.349
Troy Koss: Consider the the resources to make it, of acting practice and to use the data

359
00:39:12.410 --> 00:39:13.910
Troy Koss: that the tools are outputting.

360
00:39:18.370 --> 00:39:19.299
Daniel Ruby: We've got some

361
00:39:19.670 --> 00:39:22.140
Daniel Ruby: questions coming in from the audience.

362
00:39:22.440 --> 00:39:23.310
Daniel Ruby: Oh, okay.

363
00:39:23.700 --> 00:39:32.589
Daniel Ruby: Kip, Malone asks, what are some strategies for driving slos with application owners that do not have strong opinions of how their applications operate.

364
00:39:32.760 --> 00:39:34.790
Daniel Ruby: IE. What does good look like.

365
00:39:37.080 --> 00:39:37.870
Niall Murphy: Me Trina.

366
00:39:37.870 --> 00:39:38.880
Alex Nauda: Then I'm I'm just.

367
00:39:40.330 --> 00:39:44.160
Alex Nauda: Yeah, yeah, I'm just typing some ideas in there that come up.

368
00:39:44.811 --> 00:39:49.450
Alex Nauda: You know, about sort of like identifying who the users are. So you can

369
00:39:49.840 --> 00:39:58.349
Alex Nauda: figure out what it is that they need, and sort of revisit. And then, starting with the basics like measuring, is it down? Is it slow, you know, like basic

370
00:39:58.640 --> 00:40:04.249
Alex Nauda: does the car start? Can it drive kind of stuff? And then also drawing them out in a workshop format?

371
00:40:06.060 --> 00:40:07.360
Alex Nauda: if you.

372
00:40:07.550 --> 00:40:10.959
Alex Nauda: Often when organizations are going about this.

373
00:40:11.190 --> 00:40:13.389
Alex Nauda: they'll start with

374
00:40:14.060 --> 00:40:22.319
Alex Nauda: either some central team or someone close to a central team who owns the tooling, maybe, or owns the observability stack, etc, or the platform

375
00:40:22.330 --> 00:40:37.739
Alex Nauda: and kind of work out from there. And yet the 100th user, or the 1,000th user in that company, is dealing with the same situation, which is like, yeah, I guess. Okay, I guess I was named the service owner for this application. Is it actually working? What does it do?

376
00:40:39.350 --> 00:40:44.059
Alex Nauda: it's still, you know, it's still kind of like this very fundamental question for many

377
00:40:44.080 --> 00:40:50.299
Alex Nauda: application Development Service owners to ask, What is this application actually supposed to do? Okay?

378
00:40:50.939 --> 00:40:55.440
Alex Nauda: But I think ultimately, what it comes down to is identifying

379
00:40:55.850 --> 00:40:59.639
Alex Nauda: what are the people or other systems that are dependent

380
00:40:59.810 --> 00:41:02.130
Alex Nauda: on that service.

381
00:41:02.190 --> 00:41:03.290
Alex Nauda: and

382
00:41:03.430 --> 00:41:07.140
Alex Nauda: and starting, measuring some basics and kind of working out from there.

383
00:41:07.470 --> 00:41:08.590
Alex Nauda: The other.

384
00:41:09.100 --> 00:41:10.620
Alex Nauda: The other thing I'll mention.

385
00:41:10.740 --> 00:41:14.130
Alex Nauda: and this is important, and it is kind of relevant to the build versus buy

386
00:41:14.920 --> 00:41:16.130
Alex Nauda: is

387
00:41:16.470 --> 00:41:18.380
Alex Nauda: slos are not

388
00:41:18.410 --> 00:41:23.939
Alex Nauda: sort of a 1 and done thing where it's like, Oh, I created my slo. Okay, I'm done. I'll just go back to my other work like.

389
00:41:24.330 --> 00:41:28.330
Alex Nauda: to a certain extent, that's true. The slo is gonna run and kind of monitor

390
00:41:28.720 --> 00:41:31.010
Alex Nauda: the end result of how you're actually doing.

391
00:41:31.260 --> 00:41:34.109
Alex Nauda: But you have to revisit that on a regular basis.

392
00:41:34.773 --> 00:41:52.140
Alex Nauda: And you want your tooling to also support that capability that you know. Like, okay? Well, maybe we should look at this in our weekly review or our sprint review every 2 weeks, or on a monthly basis with our management or with our stakeholders. Let's get in place something that

393
00:41:52.190 --> 00:41:55.270
Alex Nauda: that looks at that, and that comes down to, I think.

394
00:41:56.340 --> 00:42:00.399
Alex Nauda: 2 forms of review. The one is

395
00:42:00.540 --> 00:42:09.649
Alex Nauda: looking at the slos themselves and saying, Well, okay, this thing's completely green all the time, like it's not telling us anything. It's just a green dial tone. All it tells us is that it's up.

396
00:42:09.800 --> 00:42:15.010
Alex Nauda: But we know it wasn't. Actually, you know, there were issues. Why weren't they caught and expand from there.

397
00:42:15.250 --> 00:42:18.443
Alex Nauda: and the other. The other piece is

398
00:42:19.600 --> 00:42:22.099
Alex Nauda: including it in your incident retros

399
00:42:24.448 --> 00:42:28.160
Alex Nauda: anytime that you have. Let's say

400
00:42:28.803 --> 00:42:34.439
Alex Nauda: an incident where the system goes down. Maybe a botch release and a rollback that sort of thing.

401
00:42:34.550 --> 00:42:43.379
Alex Nauda: It's helpful to go back and look at the slos and make sure that. Okay, we did have an outage there. It cost us something. Did the slo actually turn red?

402
00:42:43.640 --> 00:42:58.680
Alex Nauda: If it did, how much you know how much error budget did we burn? Are we? Are we? Do we have it configured that there's the right amount of error budget. You need to be able to adjust the thing ideally, even backstate it and say, Okay, well, it looks like we set a goal that was too high and it was too tight.

403
00:42:59.310 --> 00:43:06.149
Alex Nauda: Maybe we want to dial that down a little bit. But let's go back and get a month of history and rerun it as if you know we

404
00:43:06.290 --> 00:43:12.679
Alex Nauda: had made that decision a month ago, and and see what the incident looked like. Oh, look it! It burned about half of our budget

405
00:43:13.570 --> 00:43:21.709
Alex Nauda: and the team that is, the users of the system is kind of half angry. Okay, it's about right, you know, and kind of enable that conversation.

406
00:43:22.650 --> 00:43:23.340
Daniel Ruby: Oh.

407
00:43:23.510 --> 00:43:25.310
Daniel Ruby: Nile's got his hand up.

408
00:43:26.050 --> 00:43:35.479
Niall Murphy: Yeah, I just while Alex was talking about history there, I remembered that I wrote a short piece on implicit slos a while ago.

409
00:43:35.820 --> 00:43:44.560
Niall Murphy: and one of the things that is an answer to how do you drive slos with application? Owners that don't have strong opinions?

410
00:43:44.580 --> 00:43:48.969
Niall Murphy: One of the answers is well, you have already been doing something

411
00:43:49.000 --> 00:43:57.099
Niall Murphy: for some time. If the application is not new, newly created or launched. Right, you have been offering service, and you

412
00:43:57.180 --> 00:44:02.129
Niall Murphy: measure how it's been performing, and you set the slo to thus.

413
00:44:02.210 --> 00:44:08.750
Niall Murphy: and at least then, you won't be wildly in conflict with the existing behavior today.

414
00:44:08.750 --> 00:44:09.940
Daniel Ruby: With reality.

415
00:44:10.290 --> 00:44:31.980
Niall Murphy: Exactly exactly, and the piece goes into some detail about what happens if your slo is up a wild variation with your kind of historical performance and the kind of weirdnesses that can create. And but I would I would definitely recommend measuring the historical performance if there is, in fact, a historical performance. And if it's new well, that's an opportunity to

416
00:44:32.180 --> 00:44:35.469
Niall Murphy: reflect on a number of larger issues.

417
00:44:36.350 --> 00:44:37.110
Niall Murphy: Alex, I told you.

418
00:44:37.110 --> 00:45:00.619
Troy Koss: I totally agree with that. I I mean that strategy in some places. I've read that there's like don't base it on that. But cause you don't want to. You want people to probe deeper and like, find out what is the customer impact. Really, if it's 3 nines versus 2 nines and the and I think the the struggle with that is in reality that doesn't pan out. And it's more like what Niall is mentioning is like.

419
00:45:00.630 --> 00:45:04.099
Troy Koss: Look, the world's not on fire today. Hopefully knock on wood.

420
00:45:04.210 --> 00:45:15.659
Troy Koss: Where are we at like? How are we doing right? And then put your error budget at a put your top objective that leaves you with an error budget that's 50% or something, and and or some somewhere where it's gonna be

421
00:45:16.020 --> 00:45:29.860
Troy Koss: used right? You don't want to put it so low that it's pegged at 100. You want to put it at so high that it's negative 10 million, but like, make sure it's in use, based on some recent historic traffic and do it. I do go back to what Alex was mentioning a bit on like

422
00:45:30.040 --> 00:45:35.649
Troy Koss: understanding your customer, I think. Like, if if I look at the I look at the question, it's like

423
00:45:35.730 --> 00:45:50.820
Troy Koss: application. Owners do not have a strong opinion. A lot of it is you have to remember the incentive structure here, like, are you? Is your organization, our teams incentivized to meet objectives for their application performance? Or is the is the bar that

424
00:45:50.820 --> 00:46:11.853
Troy Koss: it's working, or it's not working like as long as it's working. Everyone's happy, right? And that is what we talked about ruby before is like how long it takes to how long it can take to like, adopt like slos and use those. There's a lot of change in that right. If the incentive structures aren't in place. That requires a lot of change to your to your

425
00:46:12.580 --> 00:46:13.559
Troy Koss: your team.

426
00:46:13.730 --> 00:46:15.500
Mandi Walls: When I think there too like.

427
00:46:15.910 --> 00:46:23.510
Mandi Walls: if if your customers aren't telling you that they are expecting better performance out of the things that you're serving them like.

428
00:46:23.710 --> 00:46:36.800
Mandi Walls: maybe that's a thing you deprioritize like you're going to spend a lot of resources to build this out. Attach the slos, making sure you're getting all of the right data, and it's all coming in. And it's all flowing correctly. And then you're going to alert people on it and all these things.

429
00:46:36.990 --> 00:46:39.659
Mandi Walls: But if your customers are telling you me

430
00:46:39.790 --> 00:46:59.699
Mandi Walls: so really care one way or the other, and that's reflected in how the application owners, and maybe even the product owners kind of treat it like, maybe you're putting those resources in the wrong place. Maybe there's something else you could be doing with that stuff or other things you could by prioritizing if it really isn't something that's important to your customers, and that that's a lot of insight that

431
00:46:59.870 --> 00:47:14.070
Mandi Walls: maybe teams have, and maybe teams don't have, like, we see, a lot of over prioritization of certain services that maybe aren't your tier ones or tier 2 kind of services that you know you're getting people up in the middle of the night for something that people only use during the day, and things like that.

432
00:47:19.070 --> 00:47:21.960
Alex Nauda: Let me just tack 1 1 thing on there, which is.

433
00:47:22.270 --> 00:47:23.250
Alex Nauda: we see.

434
00:47:23.680 --> 00:47:26.480
Alex Nauda: we see people who have a lot of different kinds of slos

435
00:47:26.940 --> 00:47:34.729
Alex Nauda: and some of the slos are like experimental, like, someone decided to create something because they release a new service, and they want to see how it works, and

436
00:47:34.960 --> 00:47:43.469
Alex Nauda: you know, maybe it's red all the time. Maybe it's green all the time, because they may not have hit it right on the nose. But there are other other slos.

437
00:47:43.970 --> 00:47:47.820
Alex Nauda: the I, the the dream of how this works

438
00:47:48.190 --> 00:47:59.230
Alex Nauda: for this service owner right, is that they find something to measure which does map pretty well to what the user customer consuming system needs.

439
00:47:59.250 --> 00:48:03.949
Alex Nauda: And once they do that, if you have a way in your

440
00:48:04.360 --> 00:48:08.079
Alex Nauda: tooling, your reporting your review cycle whatever

441
00:48:08.100 --> 00:48:17.890
Alex Nauda: to nominate or sort of anoint those slos as the real ones, the sort of production slos. These are the ones we're actually going to watch.

442
00:48:18.090 --> 00:48:21.000
Alex Nauda: Now you have this advantage, which is.

443
00:48:21.040 --> 00:48:25.480
Alex Nauda: you know, you have a way of sort of marking that, you know

444
00:48:25.670 --> 00:48:38.810
Alex Nauda: this is one of the ones that really matter. It's sort of like our golden slo. We're gonna we're gonna defend that. We're gonna protect that. That's what gets you over that over that hump. But it's back and forth between the stakeholders and the service owners.

445
00:48:42.250 --> 00:48:44.593
Daniel Ruby: Great. Thank you, Alex.

446
00:48:45.510 --> 00:48:56.119
Daniel Ruby: we got another question of Prunita. Nike asks, how frequently would you recommend reviewing the slos? And what factors can we consider to decide if and when a review is needed.

447
00:48:56.920 --> 00:48:59.419
Daniel Ruby: I'll I'll ask Mandy that question.

448
00:48:59.520 --> 00:49:02.382
Mandi Walls: I think this is super interesting and and

449
00:49:03.710 --> 00:49:12.590
Mandi Walls: from our side, like it's probably something you want to talk about in an incident review, but also from a software delivery lifecycle kind of thing.

450
00:49:12.620 --> 00:49:28.250
Mandi Walls: It's probably something you want to look at when you're adding a lot more features to the service that you're working on as well as you're. You're now, maybe have different customer behaviors. You have different user expectations. You've added a whole bunch of new things onto this. So like

451
00:49:28.750 --> 00:49:50.579
Mandi Walls: probably not weekly, but definitely more than once a year, whatever your development sort of cycle is, and the behavior of that system in production. So I would weigh somewhere between. If you've had a major outage definitely, look at it, then, if you've done a significant amount of work or some periodic number of sprints. Look at it, then, maybe quarterly. If you don't have that kind of.

452
00:49:50.580 --> 00:50:17.090
Mandi Walls: you know rigor around this sort of things, but like like we mentioned earlier. These should be living sort of documents or living sort of metrics that you are constantly making sure they're in service to your actual goals, and if they're no longer working for you, then being able to change them so if you come to a place where they're no longer working for you after a major incident, or after some feature releases, then it's time to to clean out or or put something different in.

453
00:50:17.780 --> 00:50:19.250
Daniel Ruby: Niall, what do you got to say.

454
00:50:19.250 --> 00:50:24.430
Niall Murphy: Yeah, just super quickly. Let's distinguish between conformance and specification.

455
00:50:24.840 --> 00:50:35.959
Niall Murphy: So you should review your Slo conformance essentially, continually, or you know, at least in your production meetings weekly, or whatever you should review the specification

456
00:50:36.100 --> 00:50:56.820
Niall Murphy: either on an event triggered basis, ie. Turns out the system has a blah outages and blah amount of time, and maybe this slo is unsuitable for various reasons or kind of periodically. Quarterly, 6, monthly, yearly, or kind of ranges I have seen. So just be careful about those distinctions.

457
00:50:57.020 --> 00:50:59.790
Troy Koss: Yeah, I love that. I'm gonna use that.

458
00:50:59.960 --> 00:51:18.210
Troy Koss: We're using that in probably an hour. But yeah, yeah, that's such a true point like, how is it looking at how it's doing? You should probably more often, I would say, like, Oh, like it doesn't take much to glance at your slo and see its performance. And, like, you know, are we? You know, Manny, meeting a point like like during

459
00:51:18.210 --> 00:51:44.203
Troy Koss: during your releases like, Hey, let's observe it and see if the change caused some thing. But to to your point I'll like, are we measuring the right thing? Still, you know, a lot's happened in the past 6 months, like maybe we should add an slo for that new page that we did in this load time, or something, or maybe we should measure the availability of that new service or Api. We offer like like that. That's some stuff.

460
00:51:44.670 --> 00:51:45.510
Troy Koss: that

461
00:51:45.620 --> 00:51:50.640
Troy Koss: it's it's a very different thing you're trying to do so I like that. Call out on those 2.

462
00:51:51.350 --> 00:51:53.519
Alex Nauda: I have one thing to tack on.

463
00:51:54.667 --> 00:52:05.460
Alex Nauda: I did a informal survey a while back about what makes systems go down, and the top answer. Not that unexpected was, it's because we're working on them. You know, we're doing construction.

464
00:52:05.650 --> 00:52:18.800
Alex Nauda: We're making changes. We're, you know, we're upgrading. We're replatforming. We're migrating where we're simply releasing features. That's what makes this stuff go down. And so one of the important times to review is before

465
00:52:19.010 --> 00:52:24.110
Alex Nauda: you make a change, especially if if that change could risk the

466
00:52:24.500 --> 00:52:30.439
Alex Nauda: the one job or the multiple jobs that that service is supposed to do, and then also after right?

467
00:52:30.700 --> 00:52:34.320
Alex Nauda: And then certainly you want to have a conversation when

468
00:52:34.760 --> 00:52:37.750
Alex Nauda: an slo, when there's a burn event.

469
00:52:38.170 --> 00:52:45.339
Alex Nauda: Certainly, if you run out of error budget at the very least, this should be triggering a conversation. People should be looking at it, you know. Set an alert on the thing.

470
00:52:46.700 --> 00:52:50.679
Alex Nauda: create yourself a Jira ticket. Basically, somebody should be looking at it.

471
00:52:53.090 --> 00:53:02.349
Daniel Ruby: We got another question from the audience. Diego Sosa asks, what points make the adoption of Slos a challenge?

472
00:53:02.610 --> 00:53:07.770
Daniel Ruby: And what strategies can we use as a company that is starting to implement digital transformation.

473
00:53:15.250 --> 00:53:17.241
Troy Koss: I could take a crack at that one.

474
00:53:17.720 --> 00:53:18.780
Troy Koss: So

475
00:53:20.920 --> 00:53:36.059
Troy Koss: some of the stuff we've mentioned are challenges, I would say, like the incentives of people like like the fact that the metric needs to be used right, like some challenges I've seen in the past have been.

476
00:53:36.180 --> 00:53:54.020
Troy Koss: we measure the availability of a thing. And then it's just another measure. It's just a metric. It's just a thing. It's just there and not using it. And then having a plan for what you're going to do once you measure it right, like a lot of people, because things are working or not working. They live in that kind of world. If it's working.

477
00:53:54.020 --> 00:54:22.060
Troy Koss: they don't know how well it's working. And then when you say, Oh, it's actually 99% or 98.4%. And you're like, it's failing a percent or more of the time. They're like, no way, no possible way. We want to be 5 nines. We want to be available. And then this conflict arises that you have to prepare for which is like, actually, it's not. And here's the data that tells you it's not. And that becomes a really big challenge to overcome. I think, like the the incentivization and incentive structure it's like, been the

478
00:54:22.260 --> 00:54:25.389
Troy Koss: the thing I've learned this year. The most is like, you know.

479
00:54:26.000 --> 00:54:52.940
Troy Koss: everybody's driven by incentives like that's what it is. We come to work every day to get paid to do our job right. But like, if I come to work every day and I'm not incentivized to have a good performance application performance of my thing. Then I then there's no reason to so like making sure people that do really well with our Slo adoption. And they show that they've made. You know, they started at 98%. And they've improved at a percentage point because they've made investments to resilience and whatnot. That's a great thing to celebrate and and make sure.

480
00:54:52.940 --> 00:54:57.609
Troy Koss: You know you. You raise spirits to to get people to to want to follow suit.

481
00:54:58.090 --> 00:55:00.490
Troy Koss: But making sure just people

482
00:55:00.490 --> 00:55:10.859
Troy Koss: have the right incentives is is a big challenge, and and usually that requires you're gonna you need kind of that cross cutting kind of approach where you're gonna

483
00:55:10.950 --> 00:55:30.350
Troy Koss: work with engineers to teach them the value of knowing where they can find problems in their systems. And you're going to want to work with leadership, to say it's a tool, and and Mandy said it the best. It's a it's a resourcing tool for them to know where to spend their time and their money and their effort and their teams. You've got an organization of a bunch of people. Where should you put your people?

484
00:55:30.350 --> 00:55:45.450
Troy Koss: Not the thing that's just loud. It's the thing that actually, empirically needs the most help. So you want to show them all of the magical tools you're going to give them. But both the top and the bottom. So it kind of converges, and everyone's on the same page.

485
00:55:48.030 --> 00:55:48.650
Daniel Ruby: Right.

486
00:55:49.010 --> 00:55:54.358
Daniel Ruby: Well, we've only got about 4 min left. So I'm gonna go around

487
00:55:55.220 --> 00:55:56.680
Daniel Ruby: around the group

488
00:55:56.750 --> 00:55:59.060
Daniel Ruby: and ask you to fill in the blank

489
00:55:59.600 --> 00:56:05.669
Daniel Ruby: you should probably build if X you should probably buy. If y

490
00:56:06.350 --> 00:56:08.250
Daniel Ruby: I'm gonna start with Niall on this one.

491
00:56:08.940 --> 00:56:12.991
Niall Murphy: Sure you should probably build if you're Google, Amazon or Microsoft.

492
00:56:14.240 --> 00:56:22.660
Niall Murphy: or if you have sufficient software engineering expertise in house, or if blah blah, and you should probably buy if you're the complement set of that

493
00:56:23.895 --> 00:56:24.720
Niall Murphy: yep.

494
00:56:26.170 --> 00:56:27.040
Daniel Ruby: Alex.

495
00:56:28.220 --> 00:56:33.110
Alex Nauda: I would say you should probably build if your

496
00:56:33.440 --> 00:56:35.739
Alex Nauda: company, your organization

497
00:56:36.250 --> 00:56:38.899
Alex Nauda: really doesn't buy things.

498
00:56:38.980 --> 00:56:53.089
Alex Nauda: It, you know, there's sort of like a cultural. Maybe it has to do with security, or it has to do with the capabilities of the org. But there are some companies that really that really just bristle at bringing in 3rd party stuff.

499
00:56:55.140 --> 00:56:59.699
Alex Nauda: that is probably a longer term conversation, but that's 1 situation where I think you should

500
00:56:59.780 --> 00:57:01.060
Alex Nauda: consider building

501
00:57:01.461 --> 00:57:05.440
Alex Nauda: you should probably buy if you want to get results fast.

502
00:57:06.570 --> 00:57:09.810
Alex Nauda: The hole is a bigger hole to dig than

503
00:57:09.890 --> 00:57:13.670
Alex Nauda: I thought it was. If you want to get results fast.

504
00:57:14.280 --> 00:57:16.189
Alex Nauda: Bring something in, see if it works.

505
00:57:17.310 --> 00:57:18.090
Daniel Ruby: Mandy.

506
00:57:18.290 --> 00:57:23.039
Mandi Walls: Yeah, you should probably build if you want to compete with Noble 9, I mean.

507
00:57:23.040 --> 00:57:23.630
Daniel Ruby: Oh!

508
00:57:23.630 --> 00:57:25.200
Mandi Walls: For that.

509
00:57:25.300 --> 00:57:29.961
Mandi Walls: We don't want anybody to do that, because it's amazing product, anyway. But

510
00:57:30.690 --> 00:57:38.440
Mandi Walls: yeah, I for most cases buying is is going to get you the way where you want to be faster than

511
00:57:38.460 --> 00:57:42.040
Mandi Walls: then you are going to get there. If you try and build it yourself.

512
00:57:43.880 --> 00:57:44.530
Daniel Ruby: Roy.

513
00:57:44.950 --> 00:57:46.299
Troy Koss: And would it be difficult

514
00:57:46.550 --> 00:57:53.840
Troy Koss: you should probably should probably build if your evaluation tells you you should, and you should probably buy. If your evaluation tells you you should.

515
00:57:53.850 --> 00:58:16.410
Troy Koss: and really all the factors that were brought up today from the group here, I think you should consider, and that make that full educated, you know. Guess on where you go, and don't feel too concerned if you made the right or wrong decision, because I think what we've talked about here. Plenty of times is that you can change, and you can adopt, and it might not be one or the other. It might be both.

516
00:58:17.470 --> 00:58:18.160
Troy Koss: so.

517
00:58:19.330 --> 00:58:23.250
Daniel Ruby: And last thing I have a special question for Troy.

518
00:58:23.924 --> 00:58:27.969
Daniel Ruby: When we 1st met in, I think it was March

519
00:58:28.460 --> 00:58:29.679
Daniel Ruby: or February.

520
00:58:30.060 --> 00:58:31.799
Daniel Ruby: I asked you

521
00:58:31.840 --> 00:58:33.330
Daniel Ruby: what it was like

522
00:58:33.360 --> 00:58:39.760
Daniel Ruby: building an slo platform internally, and you answered me with a single word. Do you remember what that word was?

523
00:58:40.040 --> 00:58:41.060
Troy Koss: Pain. Yeah.

524
00:58:41.060 --> 00:58:42.480
Daniel Ruby: Yeah, pain.

525
00:58:43.190 --> 00:58:46.719
Troy Koss: It is, it is Alex. Alex will probably use the similar.

526
00:58:46.720 --> 00:58:49.449
Alex Nauda: I agree with you. I agree with you.

527
00:58:49.470 --> 00:58:53.370
Troy Koss: Who thought that simple division would be so hard.

528
00:58:53.370 --> 00:58:54.470
Daniel Ruby: Who knew.

529
00:58:54.470 --> 00:58:56.390
Mandi Walls: Vision at scale, yeah.

530
00:58:57.830 --> 00:59:00.720
Troy Koss: Dude that could be an s. 3. Talk s. Recon talk.

531
00:59:00.720 --> 00:59:01.569
Mandi Walls: There you go!

532
00:59:01.800 --> 00:59:03.040
Daniel Ruby: Vision at scale.

533
00:59:03.040 --> 00:59:04.909
Troy Koss: Fully. Open. Soon. Yeah.

534
00:59:04.910 --> 00:59:05.770
Daniel Ruby: Yeah, yeah, they're.

535
00:59:05.770 --> 00:59:07.530
Mandi Walls: Until November. First, st I think, yeah.

536
00:59:08.660 --> 00:59:12.390
Daniel Ruby: Well, thank you, Mandy Nile, Alex Troy.

537
00:59:12.980 --> 00:59:17.229
Daniel Ruby: all of you. This has been a a fascinating and enjoyable conversation.

538
00:59:17.661 --> 00:59:27.329
Daniel Ruby: I really wanna thank you for your time. If you could hear the construction happening in the room behind me, I apologize. They just decided to do that. Starting during the

539
00:59:27.610 --> 00:59:29.669
Daniel Ruby: during the discussion, apparently

540
00:59:31.400 --> 00:59:36.570
Daniel Ruby: to everybody who joined us today. Thank you for attending, and I

541
00:59:36.720 --> 00:59:38.490
Daniel Ruby: hope that this has been valuable.

542
00:59:40.430 --> 00:59:43.339
Daniel Ruby: Anybody want to say one last thing before we

543
00:59:43.870 --> 00:59:44.820
Daniel Ruby: head out.

544
00:59:45.850 --> 00:59:46.790
Daniel Ruby: Bye.

545
00:59:47.730 --> 00:59:48.700
Alex Nauda: Thanks, everybody.

546
00:59:49.240 --> 00:59:49.960
Daniel Ruby: Thanks. Y'all.

547
00:59:49.960 --> 00:59:50.699
Troy Koss: Yep, thank you.

548
00:59:50.700 --> 00:59:51.440
Mandi Walls: Thanks very much.

549
00:59:51.440 --> 00:59:52.799
Daniel Ruby: Enjoy the rest of your days.