Programming Throwdown talks cybersecurity with Dotan Nahum, CEO and Co-founder of Spectral. Dotan provides us with a high-level overview of the role of cybersecurity, its definition, evolution, and current challenges. He also shares tips for small- and medium-sized ventures on how to develop best practices.

The episode touches on the following key topics and ideas:

00:01:12 Evolution of modern cybersecurity 

00:06:06 When to integrate security in a design

00:11:54 Shadow IT

00:13:50 Hacker motives and motivations; SQL Injection explained

00:16:48 Firewalls and WAFs

00:20:29 Cybersecurity for small- and medium-sized companies 

00:23:52 “The last mile of developers”

00:26:47 dotfiles

00:32:23 Simple tools and good practices

00:40:42 Attack vectors, attack factors

00:44:16 Ransomware and phishing

00:48:19 Unsafe languages

00:50:02 Fuzzing

00:54:11 Rust programming language

00:55:54 Example security scenario with IntelliJ

00:59:42 More about Spectral, Dotan’s company

01:03:40 Staying virtual using Discord

Transcript:
Episode 110 Computer Security with Dotan Nahum

Jason Gauci: Programming Throwdown Episode 110, Security with Dotan Nahum. Take away, Patrick. 

[00:00:21] Patrick Wheeler: Hey everybody. We're here with a hundred and tenth episode, which is pretty exciting. And we have our guest to-- oh, yeah, go ahead. You want to... 

[00:00:30] Jason Gauci: I'm just saying, yeah!  (laugh) 

[00:00:32] Patrick Wheeler: So we're here with our guest today, Dotan, and you are CEO of Spectral. Why don't you go ahead and introduce yourself briefly, and then we'll get started.

[00:00:42] Dotan Nahum: Yep. So hi, guys. So I am Dotan, and by the way, 110 is binary, right? 

[00:00:48] Patrick Wheeler: Oh, there we go. That's right.  (laugh) 

[00:00:52] Dotan Nahum: So yeah, so I'm Dotan, CEO of Spectral. It's a cybersecurity company, geared towards developers. I mean, we like to say that we create tools for developers with security as a side effect. So yeah, so that's, that's, you know, that's what our focus is. 

[00:01:12] Patrick Wheeler: Awesome. Well, I mean, I guess that's a lot to unpack, so I think everybody would agree, security is very important, but maybe everyone doesn't understand what security is. So we were talking about this a little when we were doing, doing warmups. So if we talk about security, does that mean that you are developing antivirus for computers, for developers, or does it mean something more?

[00:01:35] Dotan Nahum: Yeah, I mean, I mean, it's kind of all goes back to, I guess, evolution of our, I guess it is our domain, our, our world, which is kind of a high-tech or softer, softer world? Time really gets compact with all these revolutions. We have a, we have evolution revolution. 

[00:01:57] So, I mean, if you go back to 2007, that was just before Facebook and just before iPhone, I guess. And if you go back to 2005, that that was before the rise of Microsoft, I guess the major rise of Microsoft as a .net shop, which really made, you know, made all the enterprise software come along and then kind of '98, 2000, the first bubble.

[00:02:27] So all these stages, they had, it's kind of a sprint to create technology. And, the focus is on creating technology that is supposed to give developers productivity, and supposed to make, you know, make companies very productive and create a very nice portfolio of products. 

[00:02:48] And almost always, I mean, maybe not intentionally, but almost always the security side of things, was kind of left behind. You know, I'm sure no one intended for it to be, but, there's a lot of more velocity under creating a great product at the time. Every, each and every step of this, like in the first bubble, and then in 2005, and then into 2007 and so on, rather than, okay, so let's create the technology and the product, and let's also make it, you know, kind of, dependent on making great security, be there for us. 

[00:03:35] So almost every time, security came after the revolution, after the evolution. So we had from, simple firewalls, to intrusion detection, which is, you know, the large kind of, systems that try, try their best to find anomalies in the, in the area of 2000, to the smarter firewalls. And even today, those like, this, mini kind of firewalls, of WAFs that you integrate as an SDK into your app. So yeah, so it's kind of come, it comes in waves, technology, and then, security comes in waves as well. 

[00:04:17] And yeah. So the latest, the latest we're seeing right now in terms of the evolution of software is that yeah, we know that software eats the world, but we are kind of feeling that it already ate the world? So, you know, you can do so much today that you couldn't have done, I mean, as little as three or four years ago, actually. You know, it can take a Lambda and you can pick up a bunch of SAS services and you're done. I mean, you build a product that used to be maybe three, four, five years ago, you know, used to take much more energy to build.

[00:04:58] So in that sense, as a developer, you have so much more power and so many more paths to get to the same end goal that... I'm not sure, I mean, I feel it for myself. I'm not sure the security world can even begin to realize, because they need, I mean, if we, if we think about them as they, then they need to understand how to develop as well as developers in order to give, to create great solutions for that developer, that glue stuff together, and, you know, invent stuff from existing, existing parts.

[00:05:37] Jason Gauci: Yeah, that that makes a bunch of sense. 

[00:05:39] Patrick Wheeler: I say, yeah, that covered, I mean, you, you went to the whole history of modern or last couple of decades of, computer software there, but I was going to say, so one of the interesting things I think before we get into the kind of specifics about, what needs to be secured, this, this kind of, thing you mentioned where people build a product first and then try to figure out security later.

[00:06:02] I guess that's an interesting balance where, if you're building something until it's built, maybe it doesn't really need security. Right? If this was a thought in my head, I don't need security. If people are going to start using it though, immediately, you need to start having some amounts of security. Do you have opinion on like, what is the balance there?

[00:06:19] So if you don't know yet what you're doing and what may be your risks, when is the right time to start considering security and what are some of the good, you know, first things to start considering? 

[00:06:30] Dotan Nahum: Yeah, so that, that's a great, great question. I mean, I think the balance is shifting towards really taking the time, in development time, in design time, and think about security on the security model.

[00:06:46] So, you know, this was kind of theoretical, yeah, everyone should do threat modeling and everyone should do secure by design and so on. And, and frankly, you know, you'll, you'll find these people who are extremely into security that are actually doing these things. But the thing is, it wasn't being done properly or, you know, by everyone as kind of a development workflow. You know, when you come to develop a feature, then you have the design and you have maybe a POC and, and you're supposed to have this small, or maybe large threat modeling box, but, you know, no one actually does it, or, you know, most people kind of, focus on the other or other areas of developing a new feature.

[00:07:35] So it is that way because we're used to, taking a product and pushing it into, you know, your traditional server farm, or you're really a secure and isolated cloud operation, whatever. And you're, you're pretty sure that within this closed garden, even if you didn't do the proper, you know, threat modeling as a developer, then things will be okay.

[00:08:04] However, this kind of understanding is changing because it's no longer pushing to a server. Or to a kind of a closed garden environment. It's, you know, taking your function and placing it somewhere. And now someone can ask a question, which is, I don't know, I didn't have the answer. If I push a function to whatever, you know, I don't want to name any, any service, but, you know, it's kind of a, any of the new hip cool services out there that really, make you productive.

[00:08:40] If you push that function, did the other side do everything they need to do in terms of their traditional threat modeling to keep you safe? Are they're obligated to do it? Do they have, you know, let's say a WAF to identify SQL injections for you maybe, or maybe to, drop someone who's attacking your service, and so on?

[00:09:04] I'm not sure actually. So, so it kind of shifts the responsibility to the developer. Because you're building a function, you're dropping it on a whatever cloud provider and your function is now live, you know, it's up to you, right? 

[00:09:22] Patrick Wheeler: Yeah. I guess you were talking about deploying these functions and applications to public-facing cloud, or do you think that the same applies to, internally deployed app, like an enterprise software that would just be used, sort of within your corporate firewall? I think you were sort of referring to this when you mentioned walled garden approaches. Right? 

[00:09:45] Dotan Nahum: Right. So, so I, I believe like eventually the enterprises, the closed enterprises really adopt whatever's happening on the open, let's say end up on wild world. So, you know, maybe we need to give one realistic example.

[00:10:03] So let's say I'm working at kind of an Acme Corp, some kind of corporation, doesn't matter. And I'm a developer and basically I have, you know, the service, small service to build, and I decide to build it on, I don't know, Heroku or I don't know, Vercel. I do that, no one is stopping me. I, I can do it.

[00:10:27] And then I can plug it into my existing infrastructure inside the corporation. And I don't know if, if, you know, if that would be something that is, you know, okay. I mean, as a developer, I'm just, you know, shipping software, but here's something that, you know, an ability or a possibility that wasn't there.

[00:10:52] Five years ago, you know, Heroku was there, but the culture of shipping things fast and, and being able to take things to the, you know, to the extreme, end to end, wasn't there. So here's one, you know, one path that is now open, you know, and, and now people can actually wake up tomorrow and figure out, scan their code and look for external services to, you know, exist in the code base and try to figure out how many are there that they know of.

[00:11:25] And how many are they didn't, don't know of. And that's just, you know, SAS services. now we can take the same analogy and try to think about what kind of libraries do you use. And everyone remembers left-pad, right? When it was just suddenly pulled out of NPM, breaking half the internet. That's, that's kind of the new world that's happening, in the last few years that I'm not sure, ah, everyone are ready for.

[00:11:54] Patrick Wheeler: So what would be an example of, like you mentioned, you know, building a Heroku based, you know, application and deploying it and scanning for, what services that you may not have been, may not have realized. Do you have examples there of like, what would be something that you may not intend to have exposed that got exposed?

[00:12:11] Dotan Nahum: Well, yeah, I mean... Well first, there's this kind of a, the cyberworld calls it, Shadow IT, where people basically, what they want to do is be more productive inside the organization. So it's kind of two sides of the coin. One is, one is positive. One is negative. And the positive side is do you have a team that thinks it can move quickly and adopts, you know, unvetted software, so to speak, and then ships it to production.

[00:12:44] And that creates a bunch of, you know, IT assets, I dunno, services, SAS services, whatever you, you, whatever you can think of that actually, no one knows exists, in production.

[00:12:58] On the other side, if from a cyber perspective, that is an unauthorized use of software, which is kind of giving it a kind of a warfare kind of name, like a Shadow IT, like shadow ops.

[00:13:14] So, so this is, this example, you know, if you guys even, check your, your stuff, then maybe you can find many examples of that, but, you know, it's, it's kind of a productivity thing. 

[00:13:27] Patrick Wheeler: Yeah, I think so. Maybe just, just stepping back a bit, like, like, it'd be really good to explain to folks like what, what are the different components of, of computer security, like what actually a firewall is and, and, and how to, protect and it's like, like, what is a SQL injection? You know, like what are the kinds of threats that you encounter, you know, and how do those things work? 

[00:13:50] Dotan Nahum: Oh, yeah. So I think first of all, let's, let's try to get the motives out there. Right? So there's hackers and there's the good people and bad people, right? So to speak.

[00:14:01] So I guess developers build software and they're trying their best to actually add value. And the hacker is trying to, to, I dunno, remove value or try to, gain the system and gain some profit really quickly. So basically when I build, for example, when I build a function, I don't know, that takes, a parameter from a URL from a website.

[00:14:27] And, you know, maybe it's a page number, you know, traditional paging, feature. Then I, I, take this parameter and I, you know, inject it into an SQL query that I have on my backend. And my goal is, is to just give you page number two. So that's, you know, that's my perspective as a developer, I see nothing, you know, no harm done.

[00:14:52] I mean, I'm taking a value and dropping it in a inside the string, which contains an SQL query and I'm done, like I pushed this feature, I go home and that's it. 

[00:15:02] But the other side of it is that, when a hacker look at, looks at it, then first of all, there's, "what is it for me"? Like what's there to gain, but first of all, the company needs to be really attractive in terms of, hacking anything.

[00:15:17] And there's has to be some kind of trophy on the other side. So if I'm looking at something, it's some company as a hacker, and I realized they might have sensitive data because they're, I don't know, healthcare, whatever. Then at least I have now the motive or incentive to actually, try to figure out where can I hack into.

[00:15:41] So looking at this naive, SQL thing that the developer just built. So I'm looking at the parameter and what I'm trying to do is take, instead of giving the parameter, what it expects, which is a number, I'll try as a hacker to, you know, try to inject some malicious SQL code. Maybe if I, if I'm in for doing some damage, maybe I'll try some drop tables instead of a number.

[00:16:08] If I have reason to believe that the backend will actually give me the results, as I wanted, then I, I'll try to inject an actual query into the number instead of a number. And what I'll hope for is for the developer not to actually be defensive, which means the developer forgot or didn't bother to actually sanitize the parameters and make sure that if the developer expects a number, there only should be a number there.

[00:16:38] So that, that is kind of the gist of. SQL injection. So this is, you know, one, one kind of attack. 

[00:16:47] So just a little bit about firewalls. So basically a firewall is something that sits between a machine and the outside internet, or maybe internally, it doesn't matter, the outside world. And what it tries to do is to monitor traffic and figure out which traffic is strange, and which traffic is normal.

[00:17:10] So it used to be, you know, it used to be very simple. It used to be basically looking at open ports and trying to block irregular ports on machines. That is like 20 years ago, and today it's a lot smarter. So today a firewall is maybe not the, you know, the correct name anymore, but it's, it's a system that looks at anomalies in your traffic.

[00:17:39] And that is, the acronym is WAF, which is web application firewall. So many cloud providers have that and you can actually flip a switch and have the, have it as a feature. And basically it looks at your traffic and it can recognize what is normal and what is not normal. And usually that is backed by some kind of machine learning.

[00:18:00] So, yeah, so these are two categories, I guess, of, of attacks. And basically the, the reality is that the amount of attacks always, you know, always grows. There are always new attacks because there's always new code and there's always new features and new products being launched. That makes sense?

[00:18:21] Patrick Wheeler: Yeah. So you mentioned this, the WAF is web application framework and trying to understand, like, what is questionable traffic.

[00:18:28] So if you're deploying, you know, some new new website and some new API, and it doesn't kind of know what to expect, how does it understand what is questionable and what is considered pretty normal traffic? 

[00:18:42] Dotan Nahum: Right. So, so there's that, I guess, like in every machine learning operation, there's the cold boot problem. So if you have something new, then obviously it hasn't yet learned enough traffic to tell you what's normal and what's not. But, luckily, you know, I guess if someone would look at most of the internet traffic, they realize there's like clusters of normal traffic and there's clusters of irregular traffic.

[00:19:11] So, and again, probably this is a, you know, the secret recipe of every different, vendor of, of such firewalls.

[00:19:19] Patrick Wheeler: Fair enough. 

[00:19:20] Dotan Nahum: But yeah, but, but you know, the generic cases, that there's, you know, there's normal behavior and you have so much traffic these days, public traffic, and so on that you can analyze and build on.

[00:19:32] And there's like, you know, irregular traffic that is very specific to a certain service. And I can tell you from experience that yeah, definitely. It takes time for this, for these technologies to actually learn what is normal. Then you get a small amount of false positives at the, at the beginning. But the good news is that if you have traffic, then it learns, and if you don't have traffic, then maybe this service is not that, you know, popular or risky because you don't have traffic.

[00:20:00] So yeah, so in this specific case, it's kind of, it kind of creates a nice, a nice closure on it. So there's no gaps.

[00:20:09] Patrick Wheeler: So I'm going to take a step back and come back to this in a second and, and correct me if my, I could be completely wrong here. So, if I'm thinking from the shoes of an individual developer, like myself or Jason or anybody, who's just, you know, developing software and I'm going to assume which you might have to help correct me.

[00:20:30] So if you're at a really big company, chances are, you're not able to deploy straight to the cloud, or at least that's been my experience, as they typically have it pretty locked down. There's like procedures and reviews to go through. And there's a whole organization kind of devoting to that. So if I flip to the other side, if you're a super, super small, like only a single developer or a couple of developers, then you probably are the whole entire stack.

[00:20:55] And then my guess is there's a, a sort of gradient in the middle where like, you kind of mentioned before this, you know, Shadow IT, where maybe there's people who are trying to do IT or monitor it, but they're not everywhere, you could get around them. And it's not that you intentionally or unintentionally meant to, it just sort of, kind of happens.

[00:21:12] Where along the spectrum, do you find that many of the developers kind of live in, like, I, I, I'm not clear from my side, like I've spent most of my life working at relatively large companies where these kinds of things you're talking about have always been of interest. They've always kind of been handled by someone else or handled by the platform.

[00:21:32] So if you're building something yourself or in a company that's, let's say sort of medium or small sized, how do you sort of figure out like who the right people to contact are? How do you kind of figure out for yourself, what is the best development approaches to make sure that you aren't accidentally going to, you know, expose all your data to the world.

[00:21:50] Dotan Nahum: Right. Right. So, so the thing is, is that, and that's why I kind of connect it to a kind of an evolution. So there used to be good news for this, you know, there used to be a good, a good answer for your question. And the good answer would be, Oh yeah. Once you have a few controls in your organization, basically 90% of the problem, of the risk, is gone, but that's no longer the answer, you know, for the big, big corporations, you know, that, I don't know, the Apples, Facebooks, Amazons, I believe it's so, it's so, you know, insanely sophisticated there that there's very little chances of, of a developer, adding risk in terms of, security, because the investment is probably huge.

[00:22:39] But for the medium-sized and the emerging companies that have to deliver and ship fast, which is probably the majority of the companies these days, there, isn't this kind of resources. So, and, and also the times of one solution to rule them all, this is gone. So you can't really buy, a system, put it in, in your network and now all of your security issues are gone, because ways to, ways to do things are just growing.

[00:23:13] So, so just for example, if you look at your machine right now, I mean, ignoring where you currently work at, if you look at your machine and, you know, have a, have a small thought: what's in your bash history, assuming you use the, you know, your terminal? What's... do you have a dotfiles? What's in your, if you use VIM what's in your VIM VMRC? Do you have any secrets there, any tokens, anything that is maybe issued by a company, organization, but you have it right there, and guess what's the first thing hackers want to steal, right?

[00:23:50] Patrick Wheeler: It has logins and passwords. 

[00:23:52] Dotan Nahum: Exactly. So, and you know, don't tell me, but, by all means, have a look after, after this, just, you know, you know, do a less on your, seashell history, and figure out what's in there. What's the material? How many times do you export a token? Maybe, even if a mirror token, something temporary, but still it gives clues. It gives clues to how systems work and, you know, how many of the tokens are temporary and how many of them are permanent and how does one look like, and then so on and so forth. 

[00:24:31] So all these things are kind of happening on the last mile, which is us, the developers. So, you can look at an organization and say, wow, that's, that's a fully secure organization, but, the last mile is probably us.

[00:24:46] We use all the, you know, in terms of R&D ,we use the assets, we connect them. We take on the risk of, yeah, I'm going to use this token and this password and this secret. And I'm going to connect this to this external service and to this external machine. And I'm going to store some stuff on my machines and all these things combined are just, you know, ways that I'm, I've adopted to be super productive.

[00:25:17] And as the world of software grows, the ways to be super productive also gets, get more sophisticated. And that creates an impossible, an impossible problem for one, you know, one cyber or security solution to solve. It, it's basically, you need to solve all of the habits of developers at once. 

[00:25:44] Patrick Wheeler: Yeah, I guess I don't, I security is not my, not my background or my forte, but I always hear security and depth and layers of security. And, and I guess this is what you're sort of mentioning, like the last mile of developers. And, we were talking at, at my company a little bit about social engineering and just, my takeaway was basically like if someone targets to social engineer, you're kind of hosed, like, the amounts of sophistication in some of the attacks that have been uncovered.

[00:26:12] It's just insane, that, no matter what you do that, someone could probably figure out your username and password without you, kind of knowing that you've turned it over. And I guess as you're mentioning, if you store stuff on your computer and your dotfiles, or even just in your history, and they get access to your computer, they're going to learn a lot.

[00:26:32] And, I guess that makes sense. That kind of brings to light to me the why you have to have so many layers of security. It's not just the firewall, the outside or the intrusion detection on the inside, but also like, even on individual computers, like having people have good habits and stuff. 

[00:26:47] Dotan Nahum: Right, right. And you can't not, not have your dotfiles. Right. Because first of all, dotfiles indicate, a kind of, progression, right? Because you're, you're using something that supports 12 Factor Apps and is considered as a best practice. Right? 

[00:27:08] But, and there's like a huge warning here that I think we all miss, is that somewhere we should have a dotfile that is not meant to be in a repo, but it's laying on our, on our computers. And what's in these dotfiles, I don't know, but I'm quite sure that a hacker learning these best practices, they also want to learn how to abuse these best practices. So I would, I dunno, I would build a script that searches for dot and dot production.

[00:27:44] Jason Gauci: Yeah. I think--

[00:27:45] Dotan Nahum: And grabbing it from your computer, right? 

[00:27:47] Jason Gauci: Oh yeah. Yeah. I think, I think I took, I also like Patrick don't have a strong background in cybersecurity, but I did take the, a, a course on it in university. And I remember at one point someone was, you know, the, the, the lecture was talking about, you know, the magnetic hard drives and how, when you just erase something, it's not actually gone.

[00:28:06] It's just, you know, some reference to the data's gone, you can still recover it. And someone started kind of challenging and saying, well, if I do all of these things in this course, then, you know, I'm totally untraceable. I'm totally secure. And the professor had a really insightful answer that always stuck with me.

[00:28:23] He said, You know, yes, he said, but if you're a criminal, and this was for like cyber forensics type stuff, if you're a criminal, you know, you have to, you know, make sure like in the world of atoms, you're totally, you know, untraceable and in the world of bits, you're untraceable and you didn't leave any footprints and you didn't, you know?

[00:28:41] And so when you, when you started accumulating all of this, it becomes harder and harder and harder to, to, to get away with something. And so, and so this is also true where there's so many different systems we work with, we, we, we use 10 different languages and we're, you know, having, we're on three different public clouds and there's four different machine type.

[00:29:00] When you start adding it all up, if you have a problem, even once, that could be enough to get access to everything else. 

[00:29:08] Dotan Nahum: Right. And that, that is, I think this is completely what's, what's happening right now. And, you know, unfortunately a lot of it is happening, in our domain, like, in the developer space, because, because the general sense is that if we get more power and, you know, I can look at the, Docker, just as something that happened that gives us developers more power.

[00:29:36] Because now I can, you know, I know previously I could use VMs, but now I can, you know, kind of juggle machines and  plug and play them and, you know, build so many things in a better way. So I have so much more power, using, using Docker and all these things kind of shift the power towards the developer and shift the responsibility also to the developer.

[00:30:02] And, and the big question is, did you, or did we know that these, that happen, that we have more responsibility now? Because as far as we care. you know, we have more, stuff to play with. I mean, it's, you know, I'm super productive and much more productive than 10 years ago. But did I know that I, I now have much more responsibilities in terms of security?

[00:30:29] So, so yeah, so I think that the answer for that is that, I'm seeing, we're seeing that not everyone realizes that, you know, with that extra, extra oomph that we got with all these technologies that we actually have now, we're taking more risk as developers. So that, that is, you know, that is the friction that, that we see.

[00:30:55] And the answer is to try to use the same, the same tools that we use to build features in products. And to figure out what do we, you know, what do we miss? so I just said, yeah, if I'm a hacker, let me just build a scanner that, you know, tries to find your dot and dot production. Right? So how about we use that and build a developer tool that actually can tell me that before the hacker knows it.

[00:31:28] So here's the goal. Here's an idea, build a tool that scans for all these kind of, of, of the, these files that you know, that you have in the back of your head, on your computer. And, and that is a tool that you build for yourself and put it in your own toolbox. And now you're, you feel much more secure and now you can work in a, in a safer, in a safer way.

[00:31:53] It's like, you know, it's like a carpenter working with a security goggles, right? So this is how you can actually build the tools for yourself. So that you can actually take on more responsibility. So, so that, that is, that was basically my thinking, I guess, throughout all of my career. So that every step you need to actually make your toolbox bigger so that you can actually take on more responsibility.

[00:32:23] Patrick Wheeler: Yeah. So when you talk about like, I guess your throughout your career, you're saying, you know, thinking about this, this toolbox, I guess it's probably bad, but I'll admit it. Like, I, I don't really spend much of my day thinking about what tools could I add to my toolbox to help make sure that I don't leak secrets.

[00:32:40] I'm a bad, bad engineer, I guess. But how does someone go from the mindset of like, my job is to sit here and as you were mentioning faster and faster to, to ship a product, to see what sticks, so what is the phrase to move fast and break things. Right. I didn't invent that. 

[00:32:56] And like, how do you, go from that mindset to sort of culturing a sense of like, Hey, wait a second, I'm taking these risks. And they're really easy to not take these risks. You know, these are the common dotfiles that people would, would scan for. 

[00:33:12] I mean, it makes sense when you told it to me, it seems obvious, but I won't admit that I probably ever thought about it before. How does someone go from my sitting here and just developing code to, I also need to have tools in my toolbox for not just developing my code, but for making sure my code is safe and secure.

[00:33:29] Dotan Nahum: Yeah. So, so actually it's, it's it can be very simple. I mean, to me it's, it's, it's very clear and I will try to give the same clarity in how I think about it. So I remember the day, when, we used to ship software and zero tests, that was around 2005-6, maybe. Yeah, no, no such thing as unit tests, no one even knew about that.

[00:34:02] And basically you would ship your software you've you wouldn't even test it, you know, properly and you'll figure out, yeah, we have a QA somewhere down the line, they'll do the work, tell me what's wrong. And I'll just, you know, wake up and fix this stuff they found. 

[00:34:17] That, that was life. So, and that was not long ago in terms of, you know, in normal professions, like 15 years in, I dunno, in car making, that's nothing. Right? 

[00:34:30] So that's, that's 15 years in software. So that, that was the reality. But since then, in terms of, quality and, QA, I mean, we almost obliterated that, that kind of workflow and we have unit tests, end to end tests, and so on and so forth, and everyone knows that you need to have coverage. So here's, that's one thing to think about.

[00:34:54] And another thing to think about is distributed systems, right? So there was a time where we built, you know, a server, you know, a server, a service doesn't matter, and we deployed it to one single server. And that again was at the same kind of, area of fears. and maybe we used views too. That was like amazing, you know, and the load balancer was kind of a hardware thing that you had to, you know, you had to put out from a box and install it somewhere.

[00:35:28] And, and, and, you know, we never thought about redundancy in that in the way that we do today. and today it's, it's insanely more involved and so much, you know, so much better. And, and again, today, what we do is we plan a service. And, I mean, almost before the first line of code, we think about how is it going to be deployed in how many instances and how would it fail and so on and so forth.

[00:35:58] And I think security will go through the same, same evolution. I mean, what changed? The only thing that changed is the responsibility. So on the QA story, there was a group of people who were responsible to test your code, but not you as a developer. And, on the, on the redundancy side, there was the group of people called it, not even ops, and they were supposed to make sure your service is always live, which is absurd these days, because how do, how would they even know back then?

[00:36:35] And I mean, today, you, I mean, you guys just said it, I mean, in, in some organizations, do you build your code in this. A bunch of people who are responsible to make sure you don't do a mistake or don't put the company in risk. So, I mean, just, you know, just, from a history point of view the story should repeat itself, right?

[00:36:59] Patrick Wheeler: Yeah. I mean, I think as you mentioned, like a single developer becoming more and more, I mean, in the beginning, there was only the developer and then there was the organization and then now we're going back to only the developer. So yeah, I would say it's a fair bet to say that it's going to go back to a combination of people and we'll see of it, I sometimes think that the, the roles can be embedded within the team.

[00:37:24] So, you know, we talk about like deploying on Docker or whatever. Like it's not that you have an ops team that is responsible for deploying everything, but maybe you have, you know, a person or a consultant on your team who helps you do it. And so the team's responsibility, but there's someone there to help it.

[00:37:40] So I guess if I hear what you're saying about security, you know, thinking, thinking similar to like having someone who isn't there to just, you know, send you emails when you've accidentally leaked your password. But actually, you know, help you guys develop good practices and sort of look over what you're doing and make good suggestions and is a shared goal with your team. Yeah. I think that can make a lot of sense. 

[00:38:05] Dotan Nahum: Yeah. And I mean, and you ask how, like how can actually, teams can improve and, you know, and take more, more of that ownership over security. So that's the first, that's the first step, realization, you know, history repeats itself. 

[00:38:22] And, I'm a huge fan of philosophy. And, what I see here is a pattern that repeats and I'm quite convinced it will repeat itself, because we are all people and humans and, we collaborate and work in fairly the same way. You know, what's the difference between quality and, and quality of service, which is actually kind of, you know, what distributed systems come to solve and then quality of your security.

[00:38:53] So it's all the same. so that's, that's one part. And the other part that I, I believe in personally is trying to instill the mentality of, you know, fresh perspectives and learning from mistakes. So if you have these kinds of processes in your, in your company, in your culture, then you can always use these processes to actually have, you know, plugin a space for, Hey, we're doing a pre-mortem or a post-mortem, let's talk about how security is a in this whole picture. 

[00:39:29] So we talked about redundancy and deployment and, how's the, you know, capacity planning. how about we talk a little bit about, security, which means sometimes you can either simulate attacks. Sometimes you can either prove that, you know, prove that a certain service is secure, or sometimes you can just, you know, list what your, what your risks are. And that, that is a great conversation opener. 

[00:39:57] So if you have a billing service, you know, then you can actually state your fears and you can just say in that same form, your fear is that someone can go in and, I don't know, steal what? Credit cards, transaction numbers and so on. So just by stating, what's valuable for you in that same forum of a pre-mortem retrospective, whatever. You know, you can actually start a great conversation, and you can discover that people didn't even realize that the service you're building is actually holding a lot of value for a potential attacker. And that is, you know, 50% of the job. 

[00:40:42] Patrick Wheeler: Yeah. You're kind of describing now. We've kind of covered a lot of ground already, but you're describing now you were mentioning, sort of thinking about the attack vectors and what is, you know, where could people attack?

[00:40:54] What would they think about evaluate sort of thinking about coming from someone on the outside, not just, Hey, I'm playing with your service, but I'm looking at it. I'm analyzing what I could get from it. And then I'm deciding how much time to spend, whether it's just to crash your site or to, to drop your data.

[00:41:10] Or if there's something here I could extract. And I guess that even in itself is a useful way to think about it is I'm deploying the service, which is valuable to end users. But in some ways like the more personal data or credit card, whatever those things that might be most valuable to delivering really awesome experiences to end users would also be very juicy targets for attackers.

[00:41:35] And I guess the juicier the target, the more concerned you have to be with, making sure your stuff is very secure. 

[00:41:42] Dotan Nahum: Right. And always, I try to keep it very, very simple. I mean, I mean, there's, you know, think about what you're doing in a day to day, you know, is your computer filled with, with sensitive material?

[00:41:56] Did you encrypt your drive? Because, you know, once, you know, at the times when we used to go to conferences, then I literally saw, I won't name the company, but it's a big cloud company, to, solution architects, just living their, laptop on, on the chair and going for the restroom. And I was kind of looking at two laptops with, obviously the company sticker on them, just telling me, steal me, right? I mean, so once that happens, if you didn't encrypt your hard drive, basically, it's just, you know, five minutes and I have all of your data. So these are the small, simple stuff that, that I like to think that these are the terms that I like to think in.

[00:42:45] So it's very, very simple. It's connect to the, you know, the stuff that you're doing on a day-to-day basis that you're feeling uncomfortable with and, try to figure out what that is. And on the other side is also, you know, in terms of organization and teams, it's also very, you know, very simple is connect to the things that you think you're fearing from, like your fears.

[00:43:13] So basically when you, when you're about to deploy a service, I'm sure as hell, like as a developer, your fear is that this service is going down, right. So you'll try to figure out how to keep it alive and, you know, avoid waking up at night because, you know, you, you didn't, I dunno, throttle something properly or you didn't think of an edge case.

[00:43:37] So, and so just add another layer of fear, which is, I think this service is storing sensitive data and I want to just state that. And that is just, you know, a simple way to start a conversation with other intelligent people and, you know, brainstorm and think what, what you can do. 

[00:44:02] And obviously when you do that a few times, you probably get to a more advanced levels of thinking of attack factors and, you know, kind of instilling this culture and it will, it'll just happen from itself.

[00:44:16] Patrick Wheeler: Yeah, I, I think you're right. I think realizing it's an issue thinking about it, these things are, you know, important. I feel like it's a, yeah, like you said, from philosophy, you know, applies over and over again, not just that things repeat itself, but that kind of the path to learning about it is first realizing there is something to learn.

[00:44:35] And I think the topics we're discussing here are interesting approaches. So we've talked about on a developer computer that, you know, having something that scans for maybe secrets that you wouldn't want pushed out, or that you wouldn't want someone to get on your computer. We talked about application firewalls and traditional firewalls.

[00:44:58] We talked about a little about SQL injection. So we talked about a little bit like someone getting on your computer or stealing your computer hard drive. We talked a little bit about like the end deployed app. What are other areas? I mean, there's a whole pipeline there, I guess, of. You know, not just developing the code, but then pushing the code, serving the code, you know, like the distributed systems themselves, like what are other areas that are important to think about security?

[00:45:23] Dotan Nahum: Yeah. So, so, so there's two, niche areas, I would say. I would be careful with calling these niche. But there's a reason I didn't mention these, first. So one is, I believe maybe the scariest of all, which is, you know, getting an email, clicking on something and then bam, your computer is now locked with all the data and you need to pay someone millions of dollars.

[00:45:52] Right? So this is like the whole ransomware thing. And, and yeah, as you mentioned, the best way to combat this is to actually, you know, be aware, so have awareness training, and make sure, you know, what phishing is. 

[00:46:07] And, you know, some, sometimes, email specifically, sometimes, in terms of emails, that there is some email providers that are very advanced, that can tell you, listen, this email came from outside of your organization. And, you know, I, I'm a fan of Gmail in that sense. you know, they can color the out of your context thing. I don't know if that exists in other clients. I hope it is. 

[00:46:34] But this is a good part of awareness that I thought should be there. And it wasn't there for a long while. I mean, if you're an email client and you have any working in, you know, you're giving a service to an enterprise and, obviously you can identify who's in the organization, who's outside. And then just give me a color. Tell me, listen, someone from outside the organization is now sending you an email. That's a great, great thing to know.

[00:47:02] Patrick Wheeler: And flagging hyperlinks that don't match the URL they point to as another one that shocks me, that email clients don't do today. 

[00:47:09] Dotan Nahum: Right, right, right. And obviously all the DNS typos, which is fun, you know, you have a company name and then someone changes a letter. Maybe it turns an "i" into a capital "I", which looks like an "L" and then you have something which looks familiar, but isn't, so that also, you know, that is also something. 

[00:47:33] I don't know if to call it funny, but it's, it's when you, when you realize that you can do it, then you go and buy the domain for like four bucks. And then, then you have something in your hand that, you know, someone can use for, for attacking. So basically by the way, this is something I do in almost every company I joined, they look at the domain and then I go and buy all the similar looking domains and hand them over to the ops and IT people to just have these domains, you know, just by them. So no one else can buy them and it's very cheap. So I really encourage everyone to do that for their companies. So yeah, so this is one, one category, which I, I would call niche of, of kind of attacks, because they are very, specific. 

[00:48:19] And the other one is, actually that would feel nice to people who works with embedded systems. Maybe more because in there, there is, there's a category of languages with are, which are considered unsafe, I guess that's, that's the category of languages that was kind of born these days. 

[00:48:41] And the idea is that, I mean, To me, it's strange because, I remember programming in C and C++ like it was yesterday and also assembly, I'm from this generation.

[00:48:59] And these days, these languages are kind of portrayed as unsafe. And, you know, basically because they are not providing the right checks, for your, memory usage, out of the box by default. So this class of languages, you have a whole, I dunno, domain of proper usage so you could actually build secure software, but it is secure in the sense that when you, I don't know, when you put this software on a pacemaker, then no hacker can actually hack that I guess, or a spaceship or whatever.

[00:49:44] So yeah, so there's this whole domain of security geared towards making sure these, these things are, are as safe as possible. And just to look at this from a developer point of view, then, when you build this kind of software. 

[00:50:02] So one of the most accessible tools to use is a fuzzer, which basically means it's a kind of a tool or a library that you can use. It's, I try to explain it in a simple term. If you have a function that is supposed to receive a string and an integer, for example, then this tool would look at this function, analyze it and try to inject kind of brute force all of the strings in the world, like strings, strings with binary characters in them, strings with Unicode characters in them, and we'll do it by brute forcing.

[00:50:41] It won't try to be smart about it. And if it's an integer, it, obviously it will try to overflow it, or give it some bad numbers. And basically it takes some time. And then it will come up with a combination which is bad for your, for your software. So basically you you'll be fuzzing this for weeks and weeks and weeks, and then you'll be able to see if there's any result.

[00:51:07] And this result would be gold for you because you just exchange time, with something which is very sensitive. If this, for example, if this was a spaceship, right? so that is a whole category of security and very geared towards the developer. And also, you know, in the realm of embedded or, or things that are, you know, in that sense are, can become very, very costly.

[00:51:33] Patrick Wheeler: Just to rip on that for a second, for people who may not realize. So if you run a fuzzer and your program crashes, there's a number of different reasons why it could be a problem. So the first reason is that obviously, like if you were running a spacecraft or a service, if it crashes it down and then you have, you know, an outage, right? And so someone could keep sending malformed data and crashing your system.

[00:51:59] And if they do that enough, then you have a sort of denial of service problem or, you know, even worse. And so in that way, you know, that's kind of like the first class of problems. The second class of problem though, is that sometimes it crashes because you are not checking the data properly, and the crash can actually lead to a buffer overflow that allows someone to exploit your code and then actually read contents of your computer or your memory, and actually, steal results, which is much worse than just, crashing it.

[00:52:30] And so running a fuzzer and finding those things is super useful, because in, many times it's an obvious, especially if you depend on, another library for doing something like json parsing or your, you know, any kind of data consumption, and someone can feed in there. And if there's a problem in that library, which you used, because you liked the fact that it had a lot of, convenient functions in it, you could be opening yourself to a whole host of problems that you didn't realize.

[00:52:57] Dotan Nahum: Right. You just said it much better than me, (laughs)  

[00:53:01] But yeah, I, you know, you can, you can, you can claim, I dunno, maybe, you know, as you know, we have Space X and all these, maybe there's many more developers that are being called in shipping, shipping into a spacecraft these days, then, you know, there was when there was only NASA. So I don't know, maybe fuzzers will come into fashion. 

[00:53:29] Patrick Wheeler: Yeah. So we talked about, okay. A couple other, other niche areas, and then one of the other things I know people kind of mentioned is, you see every so often is someone will publish a report where they scanned a GitHub and they found, you know, 10,392 places where someone had their database password stored.

[00:53:52] What are the, so obviously, like that's a huge problem. Like, is that something that you see people doing? Is pushing code itself up to repository where they put something in there? Unintentionally? 

[00:54:02] Dotan Nahum: Yeah, yeah. I mean, first of all, a quick disclaimer, this is part of what we do at Spectral. 

[00:54:08] Patrick Wheeler: Okay. Well, I guess I teed one up for you then.  (laugh) 

[00:54:11] Dotan Nahum: Yeah. And by the way, just to close a small circle, which, you know, it's, it's so much fun to recommend when I can recommend it. So I've been through, I don't know, since assembly, probably through almost every programming language. Including Haskell and Erlang and some wacky languages. And, at Spectral we use Rust, which on the, on the, on the, on the, fuzzing, unsafe languages, part of things, is, supposed to be safe.

[00:54:41] And, as far as I can tell it is. And, and there hasn't been a morning where I wake up and look at the code and say, this is an amazing program language that gives, you know, it gives me the Holy Grail. 

[00:54:56] So just to encourage, encourage people to just try out Rust, it's an amazing, language that, is, you know, for me it's as performant as C++ and as expressive, with none of the disadvantages.

[00:55:12] It's amazing. And, and back to your question, so yeah, so people do, tend to do these mistakes. And I mean, it all, it's actually a reflection of, of the situation right now, which is there's so many tools. So many technologies, so much is being asked from a developer these days, that there's, you know, there's so much opportunity to make a mistake, and making a mistake is yeah, is, is, let me give you an example.

[00:55:47] Is using an IDE, that you think, you know, let's, let's use a concrete example. So, you have IntelliJ and IntelliJ likes to save some, some settings in a folder called .idea. And, as far as I can tell, and anyone else can tell, this folder is clean, it's safe. 

[00:56:12] So if you like, you can actually push it to your GitHub repo, which is public and everything is great. because you get the guarantee that the common, the common sense thing to do from a intelligence point of view is, yeah, is make sure it's clean and, and let you publish your settings so that if someone else takes the project, then they will get a nicely configured environment for them in their own IntelliJ editor.

[00:56:40] So that's, that's the understanding that almost every developer has. But here's the twist, there's also plugins. And some developers would choose to install various plug-ins, unofficial ones. and I've seen in my own eyes, a plugin, that is actually doing something more than a search, like some hyper search something. And basically, I've seen a project where a developer pushed their own private settings. And in there they had this plugin installed. And what this plugin did is break the environment that I guess IntelliJ has that this folder has to be clean. So basically the plugins saved and cached the search terms of the developers inside this folder.

[00:57:35] Patrick Wheeler: Oh no.

[00:57:37] Dotan Nahum: And it's really funny because you could see how the developer cleaned up the project before publishing it. So you would see, the searches right there in a public repo. You would see, Amazon and then AWS and then Easy2, and then you'll see the actual Amazon key and secret. So basically what, what, what I've watched is the entire history of the developer, trying to clean up the project before publishing it and they search for the actual secrets, in order to remove them from the code, but the plugin re--

[00:58:16] Patrick Wheeler: Oh, they were trying--

[00:58:17] Dotan Nahum: Exactly. The plugin recorded everything and dropped it inside this .idea folder, which is supposed to be clean. That is that, you know, that is the understanding that every developer has a. So they pushed the project, published it.

[00:58:33] And basically, because it's a dotfile, I guess they all always, you know, they were all also on windows and by default on windows, you don't see hidden files or whatever. Yep. Completely didn't know about it. It was sitting there for at least six months until we reported it to the company that, that we saw this, exposing.

[00:58:56] And yeah, the story that we got after that was, basically, you know, they had minors and breaking into the cloud and whatever. So it was a mess for them. 

[00:59:07] Patrick Wheeler: Oh, no. 

[00:59:08] Dotan Nahum: So this is just one, one small example of, you know, so much, there's so much freedom, so much technology, so much to use and to experiment with.

[00:59:18] But you don't always get, you know, the best offering. Basically this one was down to adopting some plugin, which looks cool. but I guess the developer of the plugin didn't have security first, or didn't really understand, realize that, there's this kind of agreement that his folder needs to be clean. Yeah.

[00:59:42] Patrick Wheeler: I guess that serves as a, a pretty good transition as we, as we sort of wrap up here to, for you to talk a little bit about what Spectral is and sort of what it does. 

[00:59:51] Dotan Nahum: Yeah. So Spectral is a, is a tool for developers. It's a security scanner. And it tries to do kind of things that we talk about today, which is, follow the habits of a developer and, be the best buddy and make sure that, you know, everyone can use it for themselves and make sure they scan the work and scan their computer and make sure that they don't do mistakes.

[01:00:16] And we say mistakes because that, that is what it is. It's, you know, you can say vulnerability, that's a different term, but, basically at Spectral, we'll, we're 50% coming from cyber and 50% pure engineers. So we like to keep this balance of productivity and security. So we just call it, you know, mistakes. I think that that is what it is. 

[01:00:43] And a mistake is something that was made on a good intention. And, basically a whole class of mistakes is, you know, using and misplacing sensitive data, secrets, passwords, credentials, and so on, and, you know, placing different files somewhere you don't want them to be.

[01:01:04] And realizing we as a company are trying to realize the habits, and, and the workflows of a developer, and making sure there's always a great solution for that. So I think this week, or next week, we're going to release, an open source product. We're going to open source it. And what it does... so I want to, I don't want to jump the gun, but what it does is acknowledge that the whole, usage of vaults or dotfiles is not perfect.

[01:01:38] So it acknowledges it and gives you instead a nice tool, which is open source and you can build it and, you know, do whatever you want with it, which gives you a clean way to actually handle secrets, grab secrets from a vault or grab it from your disc, or from any other service and leave zero footprint on your hard drive.

[01:02:01] So basically everything is in memory. So this is something that we identify that, you know, developers are struggling with. So yeah, so this is just another example of something we, we do and we will launch next week. 

[01:02:16] Patrick Wheeler: Oh, nice. I'll have to keep an eye out for that. And then what about Spectral as a company? So you said you were 50% sort of cyber 50% engineers. Are you guys looking for interns? Are you hiring? What is it like to work at Spectral? 

[01:02:29] Dotan Nahum: Yeah, so, so first of all, we're hiring, basically we have kind of positions for, Rust engineers and go engineers and, no GS. And, I guess you could say we are always hiring.

[01:02:45] I mean, we're always, willing to meet great people. and in terms of the company, it's a, you know, the COVID thing, kind of caused us to be fully remote. But lucky enough, we're kind of, in terms of the experience in the company, then, some of us come from distributed companies by definition. So, we, we were able to create a really great infrastructure for being a nice, remote first or distributed company. 

[01:03:17] And, and yeah, so we're being super productive, making sure we have all of the advantages of distributed and zero of the disadvantages. and yeah. 

[01:03:30] Patrick Wheeler: Nice. Nice. So do you guys think you'll, I mean, you don't have the answer, but do you think you, will stay virtual, or are you guys, think you'll go back to a somewhat normal thing when this is all over?

[01:03:40] Dotan Nahum: Yeah, so, so, so we basically try to always think about the balance and think about the people first. so, you know, there's that, first of all, it's finding the right tools. So is Slack, the right tool is, email the right tools, right? So what we found is we use Discord, and, that was on a simple thesis, where if gamers are really lacking that platform, that coders properly would, and we found that it actually proved itself because when you share your screen and, and, share multiple screens and people can watch, you know, every everyone else's screens.

[01:04:20] And what I mean is editors, right, writing code, not games.  (laugh) So you have zero latency and you have really great audio and everything just works. So you realize that this whole thing was optimized for sharing games and it's, it's more than enough for sharing code and doing, you know, pair programming, all these things.

[01:04:44] So, and you also have rooms, which are, we actually made a few rooms, just a few, not too many, just like in a real office. So we have, we have rooms that everyone wants, but are taken, right, the same, same frustrations. So basically our discord resembles a real office. You can just drop in a room, drop into a conversation, and it's, it's really, really fun.

[01:05:09] So that is one thing we realized and experiment with, experimented with, and really works well for us. But yeah, we're trying to create a balance and also, you know, when, when possible meet physically, but yeah, we're always thinking about this as a problem. How do we solve it? Kind of like an engineer's, like an engineering problem.

[01:05:28] Patrick Wheeler: Nice. This is a great heuristic that, if gamers like it, engineers might too. Yeah, I think that's a, that's a great takeaway. 

[01:05:36] Dotan Nahum: Yeah, It's, it's great. 

[01:05:38] Patrick Wheeler: And then what about you personally? You have a, are you on social media? Do you have any, I think you might be doing some other stuff. Anything you want to share with people?

[01:05:47] Dotan Nahum: Yeah. So I'm, I'm on a GitHub, Medium and Twitter. I'm Jondot, J O N D O T. And yeah, I kind of, I'm trying to be on everything at the same time. It doesn't work (laugh) so I'm dividing my energy. So you'll see me active, you know, periods of times on GitHub and periods of times on Twitter. So I, I just load balance. I can't do everything in parallel, so yeah. 

[01:06:17] Patrick Wheeler: All right. Very cool. Well, thank you for your time, Dotan. We really appreciate it, that was really awesome. A high- level overview of security. I had a good time. Thank you for coming onto the show. 

[01:06:26] Dotan Nahum: Perfect. Me too. 

[01:06:36] VO: Music by Eric Barndollar. 

[01:06:41] Jason Gauci: Programming Throwdown is distributed under Creative Commons, Attribution ShareAlike 2.0 license. You're free to share, copy, distribute, transmit the work, to remix, and adapt the work, but you must provide attribution to Patrick and I, and sharealike in kind.

★ Support this podcast on Patreon ★