if you think about it like Engineers are not cheap right they're specialized so you know if we have a great algorithms engineer UI engineer whatever like that's what we want them to do is like solve problems in that domain and not worry about security so it doesn't mean like you have no responsibility like you still need to sort of participate but by and large we wanted to take care of the really difficult security problems for them welcome to a new episode of Olo the podcast where we talk to Industry experts and practitioners about how they're tackling the toughest problems in it in Security today I'm your host Alex bogy CEO and co-founder of conductor one in today's episode I have the pleasure of Hosting Jason Chan Jason has an illustrious career in security he led infosec and Netflix for over 10 years and created one of the most Forward Thinking security cultures in the valley in our conversation Jason shares how the Netflix security team kept up with the company's explosive growth and digital transformation we also dive deep on platform engineering Netflix's well-known open source involvement and Jason's advice for someone who's just getting started in security let's get into it Jason thanks for joining me today I appreciate you taking the time out of your schedule know you're a busy man uh but you know I was really um first of all we got a chance to get connected I guess it was a few weeks ago and I know that you know your background was it Netflix you ran information security there for a a good period of time had a great run there um and from my understanding from talking with you and then also just you're pretty well known out in the security industry is that you you guys did a lot of really interesting thing in things in Netflix particularly around platform engineering and security platform engineering and it was really excited to explore that topic with you and go a little bit deeper today can you maybe just give everyone listening like the five minute or or sort of like talk to me like I'm five uh explanation of what what platform engineering is and and maybe a little bit of flavor on that as well around security sure yeah thanks for having me Alex um yeah I love to talk about security and platform engineering um if I were going to try to explain it um to say at 5-year-old I I probably use a couple of analogies and I use a couple of playground analogies since we're talking five-year olds so so first I would say like if if you're a fiveyear old and you want to like make sand castles right that's kind of your thing right you want to do your your art you want to sort of build your thing and like you don't really care about building the sandbox and filling it with sand and making sure it's clean so I I kind of think of the platform engineering team is is the team that sort of creates the sandbox who kind of manages it for you and I think another kind of important analogy of course these AR perfect but they sort of give you a flavor for it um another kind of playground analogy I would say is a seesaw or a cheeter totter right it's kind of like if you're familiar with that it's I would say it's like if you've ever done that with your friend you see how easy it is maybe your friend weighs 100 pound or 200 lb but just by like sitting on this lever you can make them go up in the air pretty high whereas you couldn't do that without the Seesaw so I think of platform teams they're also kind of a unique way of creating leverage for an organization so basically like you get out of it more than you put into and that was the team that I joined when I joined Netflix in 2011 it was called cloud cloud platform engineering I think it was about a dozen people at the time and I sort of came in to think security and that's really what we were trying to do we we recognized you know Netflix was at kind of an important transition Point as it moved from being mostly like a DVD by mail business to streaming uh video streaming subscription video what we were trying to do was was unlock a lot of innovation um on top of the public Cloud at a time when those patterns of practices and mechanisms weren't really well out so we were in addition to kind of try to make it easy for the rest of the year audience we we were doing a fair amount of I guess what you'd call Path finding so trying to figure out you know what's the right way to kind of build and operate large scale distributed systems in the public cloud and you know at a time that was I think I think if you were going to do a start of now it would be it would probably be your natural approach is is to sort of launch in the public Cloud but um you know Netflix started moving there I think in 2008 so you know that's 15 years ago so there there weren't really any large companies doing it at the time even when I started 2011 it was still pretty early so yeah a lot of it was we're trying to figure out you know frankly just kind of like how to do it the right way yeah and can you can you click in a little bit I'm really curious because it sounds like it it was definitely a journey to mature that like organizational competency around what a platform the responsibilities of a platform engineering team and and what they were responsible for versus maybe like what individual developers are responsible for um presumably that that cloud native migration shift and embracing kind of cloud infrastructure helped drive a lot of the uh that thinking for Netflix specifically but can you click in a little bit to what did the team look like you said 11 people in 2011 what were some of the processes what did the tooling look like versus what does the tooling look like today or like what were some of those big shifts that you made either around principls or team or infrastructure automation yeah so I'm tell you know when I started yeah it was it was I think the overall company was and maybe five or 600 people so it wasn't wasn't huge it by the time I left in 2021 it was about I think about 10,000 yeah in terms of what we were doing so we were really like in the process of moving big parts of the service to AWS so a lot of what we were doing in the platform team was sort of building mechanisms to do to do like a lot of it was data management because we we still had our primary uh subscriber database was in the data center all the payments infrastructure was in the data center so a lot of it was like figuring out you know how to do sort of like dual rights in you to make sure things got committed to the data center as well as in the cloud uh we used to talk a lot about the phrase we would use was called Roman riding which was you know if you're like I guess back in the Gladiator days you ever see like somebody riding two horses they have like one foot on each horse and it's like it's very cool looking but like incredibly unstable right so you're like trying to like manage this process where you're doing some things in the data center some things in the public Cloud so that's what a lot of the tools were but yeah I mean in terms of the security side there there really was nothing because I was I was kind of like the first person to start on that side there were a couple of folks working in the IT team working on security so like kind of corporate security and then I sort of came in to think about Security in the public cloud and what are the mechanisms what's the infrastructure that we needed to kind of make that possible but I mean I think there were some things that were I would say like incredibly Advanced even if I look through today's lens so um like our use of IM mutable infrastructure and kind of ephemerality and auto scaling so it was quite Advanced even even when I started and then we really kind of built out from there especially things like deployment and a lot of what we were trying to do was we were trying to make things faster and safer right so it is we we knew we needed I mean if you if I think it was like Market dreon said in like 2011 right that software is eating this eating the world right and so if you think about that what does that really mean it mean well pretty much every company to some degree competes based on their ability to develop and operate software and we absolutely viewed that as a big differentiator for us so we put a lot of energy a lot of investment into making developing software easy and operating software easy so that that's that's a lot of a lot of what we spend time on and so it sounds like you you had some of the foundation when you joined and they were embracing everything you said right ephemerality um kind of immutability around infrastructure um and it that Journey was happening were there specific Investments or or things that you did along that Journey that you found really created that to use your metaphor uh before like that seesaw leverage effect with your internal customers and can you share maybe like one or two of those big wins or even stories examples sure yeah I mean I I think one of the things I I would give credit to to you know like my manager who hired me and the rest of the Y was they really viewed security as like fundamental and and integral to like their ability to operate so they didn't view security as like this add-on of like well let's do whatever we want to do and then you know Security will kind of patch things up it was a very very integrated mentality and and I think that really served us well and and even like as a team grew you know we were buil building and operating key parts of the streaming product ourselves the security team was so we we were on call we had we owned you know tier one services so I think that that was um pretty important just in terms of like setting the stage for how security was going to operate with the rest of engineering and and I think you know and I don't know that we were ever like super explicit about some of the like philosophies of some of the tools but we really thought about like developer experience and developer productivity and like how do you like what is the interface between the end developer and the infrastructure that they're building on top of we really wanted that to be you know really smooth and integrated so a lot of the tools that we built for the purpose of like whether it's deployment or administering systems they were cut I don't even want to say dual purpose but they were like a single tool that met the needs of multiple constituents so certainly the end users but also like me as the security person but you know folks working on observability or performance or efficiency or even things like Cloud cost so we didn't really build a single purpose tool that only you know one team used so all of those were were were dual or or multi-use systems and I think that that that's certainly something philosophically that I carried forward was hey what are the other sort of systems and techniques that adjacent teams to my own are using and what are they sort of how are they being successful and are there pieces of that that I can sort of pull into my own portfolio so I'm not rebuilding it's interesting because to hear you describe it sounds so similar to like a traditional software building process with like product managers and you're sort of working with like external customers and really thinking about uh how do I make it's like but the I guess the difference there is that internally those were your customers at Netflix but your sort of the philosophy was exactly the same like building this for multiple use cases multiple internal constituents making sure you're like really focused on their experience and making them super happy does that resonate with you and and like did you even internally did you think about the platform engineering team as kind of like serving the business and serving a bunch of internal customers even to the point of maybe like taking feedback on them from for road map and things like that oh yeah definitely I I would say that was very much the approach in those early days we didn't we didn't really have like traditional product managers but I I definitely I mean having been a project product manager earlier in my career I I carried that forward and yeah because when I started youan I started as an individual contributor at Netflix so I mean I I was just kind of building along side everybody else and and my scope was was much smaller I was really just focused on kind of cloud and product security I mean eventually you know we sort of grew and the scope grew but those early days it was um it was it was really about about focus and you know I I'll kind of use this story that I I've used a couple times before but I I mean it kind of maybe helps highlight sort of the way that we thought it was actually the very last social event I went to before lockdown right so it was a little bit over three years ago that we were at like happy hour security happy hour in San Francisco and I was I was talking with somebody from another you know another kind of like large Cloud forward company and she was telling me that you know their CTO really wanted every engineer to think Security First right and to be like that was like and and I thought that's that's that's a totally reasonable and probably pretty energizing approach especially for the security team because it makes you really clear what your priorities are but then I also told her I was like actually at Netflix we think pretty much like the exact opposite whereas we want developers to focus on what they're hired to do so if you think about it like Engineers are not cheap right they're specialized so you know if we have a great algorithms engineer UI engineer whatever like that's what we want them to do is like solve problems in that domain and not worry about security so it doesn't mean like you have no responsibility like you still need to sort of participate but by and large we wanted to take care of the really difficult security problems for them very much in like the philosophy that Amazon has like you know undifferentiated heavy lifting and because I also thought too with um I mean security is it's a it's a big big domain and um it's a high consequence domain and I don't want people having to make decisions in that kind of domain if they don't have the expertise so I would prefer to handle the security sensitive stuff um with via you know local domain expertise Within than the security team they let the engineering team sort of like innovate and iterate and whatever domain they specialize it I I personally think that that's a better use of company's resources so that's really like the way we structure that was like key to the overall philosophy of the Netflix security team and I would say more broadly the Netflix platform engineering that's really interesting it's like this concept of sort of secure by default or like transparent security and what you're doing I I have a feeling this is related to but um uh maybe you can correct me from wrong the the pave roads concept that Netflix I feel like really evangelized and talked a lot about uh very publicly around um well maybe uh maybe just stop there can you talk a little bit about that concept of paved roads I think you touched on it just now but maybe you go a little bit into detail kind of explain what that means and even what the metaphor means yeah sure I mean the metaphor it's like it's kind of like if you had if you had to get someplace like you know physically you had to sort of move your body throughout the world to get to some destination like there's a bunch of ways do that and so our idea of creating a paided road was like Hey we've already kind of built the infrastructure we have a nice smooth path for you to go down it's predictable it's it's well supported that doesn't mean you can't choose a different way maybe you want to take a machete and kind of hack your way through the jungle to get there um that's fine that's probably going to be like more unpredictable um and you're going to have to do a bunch of stuff yourself so we kind of thought of the paved Road and and you you'll hear other metaphors like you know golden path or well lit path I mean I think they're roughly the same um I I just I would kind of think of them is you you have a you have Central teams that are you know like platform or security they have well supported solutions for identity management whatever it is and um like that becomes a pave Road like you can use this and this will kind of let you get to your destination quicker and more solutely more predictably doesn't prevent you from going and do your own thing but I I I think part of that I was I kind of think of companies generally have a lot of different Central teams not just in technology so maybe it's facilities right maybe it's Human Resources maybe it's legal all of those Central teams are looking for ways where they can better serve their constituents in kind of a smooth way that will allow them to scale and paved Road was was was the sort of metaphor in the approach that to make that you know like a a super tactical um or like rockable example for the listeners do you is there like an example of payro that you could share that maybe you found particular success with at Netflix yeah yeah I think probably the most dramatic example I would have was um you know as Netflix went from becoming because just not to go into too much history of the business but you know Netflix started in streaming by taking content that other Studios had already created and then putting it on the service so they weren't actually creating content uh around 2013 we decided we we basically wanted to be a studio as well we create our own content so if you think about well the systems that we needed to build to make that happen to to serve all the the different Productions because because each production like each TV show or movie is kind of like its own little business right and they've got like systems they use and they've got wireless networks and printers um but they might only last six months right because however long it takes F and um as we started building applications to serve that population so to serve like production Crews there was more and more demand where people wanted to be able to access those from anywhere right they didn't want to be on a VPN because they're being accessed by Third parties not employees yeah so it was this idea of like how do I how do I as a developer get a system that I built onto the public internet and when when we started that Journey you know it was kind of embarrassing because we had like a 20page Google doc that said like here's all the things you have to do and it was like no this is just not acceptable because you know the developers they spent all their time on functionality and then they're like okay it's time to go live I should just hit a switch and we gave him this 20 pages of garbage so we eventually kind of over through iteration but we we ended up building a system called Wall-E um like I I think it was I I forget the movie but I think it was actually called Wall-E but um it was basically like an identity an identity aware proxy that I as a developer could just publish my app behind and by doing that I would get all like your SSO and I would get logging and I would get like anti-d do and sort of you know all that kind of stuff just by opting into it and I didn't have to worry about like how do I integrate that or make that happen so to me that was like a really dramatic example of pave Road where you go from your your original pave Road it was that Google do because this we're telling you this is the best we had so it was that pave Road it was probably had a bunch of gaps in it and it had some potholes it but it was still like the supported way and then we felt there was sufficient value in sort of continuing to invest so the when you came up with wall so you go from a 20 Page doc that you had to sort of manage manually to like I just opted at my deployment yeah it's like your you got the massive checklist the 20page checklist or like the technical solution you can work that solves the checklist for you that's uh that's pretty awesome can you talk for a s of Switching gears maybe a little bit but one of the things I've always really appreciated uh at least from outside looking in in Netflix is how much the the companies contributed to the open source Community you guys have a lot of popular projects in the open source space as it relates to security particularly bless lemur repo kid security monkey just to name a few I'm I'm really kind of curious what you didn't have to do that as a team and as a company and I'm I'm really curious what drove that like why did you guys publish those projects um to the open source Community was there any sort of philosophical thinking around that in particular yeah a few things I I think are are good to dig into so so first um you know really at the time as we were moving to AWS a lot of those um problems weren't really solved or they weren't really well understood so as we kind of came up with Solutions our idea was well let's show our work and like put it out there see if people agree with this like do you think this is a reasonable way to solve this problem if not what are some other ways to go about it so that's what a lot of the real early open source was about and so we actually created this thing called the Netflix Cloud prize in 2013 where we where we basic basically invited people to kind of innovate on top of our open source and we brought people to the AWS conference and gave away money so it was it was pretty neat and and I would say also um you know at the beginning where Netflix was a much smaller company five or 600 people um it wasn't really like really well well known as like a kind of a tech brand um if you think about fanging like that that wasn't really a thing back then so we were trying to establish ourselves as like a reasonable alternative to you know like Google or Amazon or other places like that for great Engineers to work so that was part of the open source um strategy and and um within security part of the reason we did it was certainly to align to that broader strategy but but also uh I strongly believe that we as as defenders in security should be sharing with each other whether it's going to conferences and talking about things you know talking about incident postmortem or open source so I was really big into that you know not not that I was um you know forcing folks to open source stuff but um my thought was a lot of these problems are not necessarily unique to Netflix so to the extent that we we have a reasonable solution to a problem let's go ahead and put it out there let's share it yeah and and you touched on something that I think maybe is one of the hardest things to do as a a leader and a manager which is that that kind of cultural aspect of building a team and clearly like the open source part of the open source strategy it sounds like to degree was a little bit of recruiting driven right so it's like we're working on really interesting problems you're getting that news out there you're you're letting people know that Netflix is more than just delivering DVDs in the mail I'm kind of really curious like taking that a step further were there things that you looked at or that you look at in general um specific considerations to culture skill sets uh whatever it might be when you're hiring someone on the security team who was going to be a part of that platform engineering team where there things that you looked at like orientations again kind of C skill sets yeah I mean that was uh that's a good question so because I think you know with security hiring it's it's always hard but I I would say it that um when I was first hiring like first I I think I first hired my first person like in 2012 so it was just me for about a year and a half and then as I started hiring I mean I would say broadly you know I was looking for people that could sort of complement my own skills and abilities but you know real broadly I think at the early stage of building a team you really more looking for uh more generalists than Specialists right like your your first hire in security is not going to be somebody who can like reverse engineer malware right that is just not is just not as broadly applicable so um I was really more looking for um just like a you know kind of experienced um generalist security Engineers that could also um build software I thought that that was pretty important so I think we were definitely sort of early in the trend of hiring software Engineers into security and then you know from there you you eventually become specialized but you're really looking for folks that had a decent background but I think you know maybe even more important for that probably like the number one I wouldn't even call it a skill but I would say mindset I would look for as like really pragmatism and practicality and and really people who were um had less of like a black and white view to security of like this is this is how it must be done and and who who could kind of understand ambiguity and and it because we were you know generally solving problems that didn't that didn't have great Solutions out there like I remember early days of having to like educate our external Auditors about you know cloud and you know continuous deployment and how does that factor the socks your PCI um so you know you had that people who were I think open-minded and really sort of valued I hate to say biased action but we're really just looking didn't really necessarily have preconceived notions about what a solution needed to look like and we're really just more driven to like hey let's get this let's get this thing done um we might have to do a little bit of inventing on the way to get there so I think because of that because we knew a lot of those solutions for what are the problems we had at the time they weren't available in the market we knew we were going to have to do some building on our own so I was also looking for people who had who had done some building and not just impl yeah so Builders uh comfortable and ambiguity generalists maybe a little bit of security background but yeah it sounds like people are just willing to to jump into the trenches get out the shovel and like start building things even if it wasn't wasn't perfect solve problems um can you talk I'm I'm really curious because you you had a really long run at Netflix and I'm sure you have a lot of War Stories of things that went well and and um things that didn't maybe go so so great um during the time as well if you could change or or do one or two things differently if you could rewind back in history and and kind of like learn from your mistakes and like prevent those footfalls anything you could share there maybe like an example of something you might do uh differently if you had a second go r on it yeah I I think probably at a high level and I think this is really just kind of speaks to my own leadership style like over the course of my career you you become to learn you you come to learn or hopefully you come to learn how important communication is and and a lot of it especially at an executive level is just kind of repeating yourself like a bunch of time so I think I would have spent more time you know communicating the philosophy is communicating the the strategy the vision the approach and not you know just assuming that people had read that one memo I wrote or had attended that one meeting where I talked about it because a lot of it is just that constant drum beat you know because we talked about pave the road earlier like that was something we just you just repeat it you like this is the philosophy and leverage and like you hear about abstraction and all these things over and over um but it's kind of like well if somebody just started at the company today well they weren't in last week meeting so you you have to kind of keep so I think that was something that I mean I I I probably back then I viewed it as a waste of time because I'm like hey if you want to know just go read that doc I wrote but you know that's just not how humans work you just got to keep keep communicating keep repeating yourself and and especially in multiple channels like don't just don't expect somebody saw your slack post or your your Google doc totally it's uh I've fallen in this trap as well it's like I I wrote it up I sent the email out everyone should get it right but the reality is it's like say it say it again say it a third time say it a part time that's very that's that's kind of uncomfortable to me generally like I I tend to not be like I mean you wouldn't maybe wouldn't know it me do at the podcast but I I tend to not be like the loudest person in the room I don't I don't necessarily talk a ton but yeah I realize like that is valuable it's important to do that um quick uh quick parting maybe tips for folks listening is there um you know if you were talking to someone maybe that was early on in the the platform um security platform engineering Journey you know they're just getting something going they're they're getting the competency set up maybe they're hiring their first hire any practical tips or you know they like gotas to look out for or I guess what would you what would you tell someone that was early on in that Journey yeah I think if you're getting started and I I mean I think this advice is relatively applicable to different kinds of situations is you might have a really expansive view like you might want to say hey I want to build the security paved Road where we take care of all these problems for our folks but then like you're the first person you have nothing but that's what you want so I would kind of view that is you really need to create strategic principles for how how you're going to build and so maybe one of those is like we want to make security easy for the developer so like that's a principle like you've just you've articulated that and now that's going to guide your your your sort of solutioning and your building and I always would like it's a kind of a Cheesy analogy but I don't know it just works for me I would always tell people it's kind of like if you're rowing a canoe from one one side of the lake to the other like when you're rowing a canoe you're actually facing like your back is facing your destination right so you have to you have to paddle but you also have to turn around and make sure you're going the right way so you got to figure out what's the ideal um mix of execution and kind of thinking so I would say if you're getting started like get some principles out there some long-term strategic principles so that as you're building you can always use those as like hey let me just make sure are we actually building and operating according to the Strategic principles we laid out and then I think by doing that you're going to have less chance that your work sort of goes off the rails or becomes misaligned with your with your strategy uh Switching gears last question before we wrap up 2023 is wild I I feel like the last three years have been absolutely insane from a startup standpoint uh just in you know funding new companies um you know horizontal platforms people going deep on probably just a lot of things happening do you have any uh any hot takes or predictions for the security market for 2023 D geez well I mean I feel like I think it's a good thing that that we're seeing a lot of funding in the security space because there's just a lot of unsolved problems um of course you know not everyone will be successful but I mean I generally I mean one of the things that I would encourage like people who are found who are founders are thinking about starting something is like you don't necessarily just have to go like incremental to what's out there you can actually like propose a completely different approach because where we are in security um we're we're still like pretty early like there's still like a lot of unknown so I generally encourage people like be comfortable like thinking differently and and don't feel like I just got to make this one solution like incrementally better and then that's going to be my differentiator like maybe you have a totally different approach I like it I like the philosophy of it I mean honestly I think that the to me the the biggest most interesting bets are always when people kind of flip things completely you know on their head and say like well this is just like the way that this works today in the world is just not a good way to for it to work and I'm going to build I'm going to paint this Future Vision of a better world I'm just going to make that happen in reality I think it's like way more interesting I mean don't get me wrong solving a play problem is great but like it's it's good to like have the big Vision too yeah we we've got so many of those yeah we got plenty plenty of work to do yeah I appreciate you joining in thanks so much for the the chat it was awesome I'm sure um listeners will love hearing about building out platform engineering um and uh thanks for the time yeah thank you Alex appreciate it thanks for tuning in this episode of All Aboard is brought to you by the team at conductor 1 we'd love to hear your feedback send us a message at all aboard conductor one.com till next time