Former GM of Snyk, Anton Drukh, recently became a mentor for VPs of Engineering and joined us on this week's episode to discuss his experiences leading a team split between two very different cultures. His journey as a leader working with teams in Israel and the UK is a lens into how diversity of mentalities can be an essential asset for tech companies and how it played a role in Snyk's success.
As the saying goes, "software is eating the world", but who's feeding this software machine? A lot of companies, and Snyk is no exception, have basically risen on top of the notion that developers are expected to do more, are expected to be more related to the business, and to measure their value in terms other than lines of code.
We should not read into this momentum as if suddenly in order to be called an engineer, you have to know so many things at once but to choose what you lean into and what you excel at, without segmenting too much. Because, if I define myself as "no, I'm just the backend engineer, if you have a front-end task, go to someone else", then I'm actually pushing myself away from the business impact I'm supposed to provide. And that's a very delicate dance, to choose your proficiencies while at the same time measuring the amount of business value you can provide with those choices, and making sure that you optimize for both.
As engineering leaders, our product is the organization that we build and things are never perfect in any organization, there's always friction. You move something under the lights and the shadow is cast somewhere else. It's a constant optimization problem. It feels like, if you just gave it some more thought and spent two more days and spoke to two more people about your thoughts, you will come up with a better solution to it. And what you often miss is that imperfection persists during these two days, and you would have been better off acting sooner on the level of confidence you had in the past. Because resilient organizations can also withstand these changes.
It really helped me once I realized in full that leadership in engineering, and management roles in engineering, are all about people. Technology is secondary to management positions, and the sooner you realize that, the better you embrace the real responsibility, the better you act on it. Suddenly the amount of confidence you have allows you to let go of the correct way of doing things. Because with technology it's either a zero or one, right. It's either correct or incorrect. With people, it's not the way it works, and that's okay.
Engineering probably doesn't have the same luxuries that other organizations have because it does feel like a black art to most people who've never engineered before built product. The other realization was how little actual information engineering has even about itself. We don't actually know how good we are planning or how good we are breaking projects down into small granular unions or how efficient we are internally half the time. Besides that, we also get all of the unknowns. So, when something pops up in the quarter, we've got to reprioritize from something coming in from sales hot. We have reprioritize, but how do we rebalance that tree by ourselves? Because we've already done the planning for the quarter that typically falls to engineering and engineering alone.
I think the ability to allow people to become professionals and to choose what they excel at, while at the same time holding teams accountable to a wide array of expectations, and to some extent, pushing against this definition of "well it's oil and water. It will be hard to mix", but instead say, "hey, but you're just four people. Can you show me a plan that covers both these aspects?", is basically rooted in the notion that delivery and success, and failure, by the way, is owned by a team, and never an individual. It is a team that sets out to achieve something. It is a team that celebrates the achievement.
I think it's one of the least satisfying things engineers can actually hear. Engineers or engineering managers tended to react to "I would like a checklist of once I've satisfied these things, I could become a Senior Engineer or a Staff Engineer" and, you know I blame Google for this, for what it's worth. Because I think Google does operate this way, but when you're 100,000 people or 200,000 people have to operate that way. But it's very different when you're 50 people growing to 150 growing to 300, there is no checklist of things that you can possibly put on paper to go do that.
In August 2011 Marc Andreessen coined the famous phrase “Software is eating the world”. In a Wall Street Journal op-ed, he suggested that “we are in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy.”
Source: Intercom
A decade later, this checks out, and Anton follows his reference to Adreesen's famous words with "who is feeding the software machine?", introducing one of the main themes of this episode around the ever demanding roles of software engineers.
Site reliability engineering is a set of principles and practices that incorporates aspects of software engineering and applies them to infrastructure and operations problems. The main goals are to create scalable and highly reliable software systems.rc a
Source: Wikipedia
Database administrators use specialized software to store and organize data. The role may include capacity planning, installation, configuration, database design, migration, performance monitoring, security, troubleshooting, as well as backup and data recovery.
Source: Wikipedia
A distributed system is a computing environment in which various components are spread across multiple computers (or other computing devices) on a network. These devices split up the work, coordinating their efforts to complete the job more efficiently than if a single device had been responsible for the task.
Source: Splunk
Kubernetes is an open-source container orchestration platform that enables the operation of an elastic web server framework for cloud applications. Kubernetes can support data center outsourcing to public cloud service providers or can be used for web hosting at scale.
Source: vmware
An Internal Developer Platform, or IDP, is a self-service layer that allows developers to interact independently with their organization’s delivery setup, allowing them to self-serve environments, deployments, databases, logs and anything else they need to run their applications.
Source: Humanitec
[00:00:00] Eiso: Welcome to developing leadership. The podcast where I, Eiso Kant and my cohost Jason Warner, talk about the lessons we've learned about Engineering Leadership throughout the years. In today's episode, we're joined by Anton Drukh, previously VP of Engineering at Snyk, and now dedicating his time to mentoring engineering leaders throughout the world.
[00:00:18] Anton joins us to talk about how the roles of engineers within companies and the skills that they Harbor have changed in the last decade. And what we think the future holds for the industry. If you want to dive deeper into some of the topics we mentioned today, check out the show notes for this episode at developingleadership.co
[00:00:39] Hi, everyone. We're back with developing leadership, Jason and I today are very excited because we have Anton Drukh our guest, for those of you who don't know Anton yet, Anton was the VP of Engineering at Snyk, the developer first security company. From literally the first engineer to about 150. And it's only been very recently that you've decided to take on your focus and experience to start mentoring other VPs of engineering. And so it's, that makes you the perfect guest for us and love to welcome you to the show, Anton.
[00:01:11] Anton: Thank you very much. A pleasure to be here. Hi Jason. Hi Eiso.
[00:01:15] Eiso: Welcome
[00:01:15] Jason: Anton, it's gonna be fun.
[00:01:16] Eiso: So Anton sent us, he had listened to the first episode, so by the time we're recording this, we have just released the first episode and Anton listened to it and he heard Jason and I talk about pendulum swings. Essentially the notion that throughout the history of software engineering, we've gone from one extreme to another. While Anton is not jumping in today with strong opinions about the JavaScript world. He does have some interesting thoughts to share with us about the way we work and the fact that engineers today are considered to be responsible, not just for writing and deploying the feature, but for business objectives, for security concerns, et cetera. And I'd love to kind of hear, you know, your strong opinions on this, Anton and then I'm sure Jason and I will dig in.
[00:01:57] Anton: Sure. Thank you very much. So, you know, the, as the saying goes, "software is eating the world", but, who's feeding this software machine, right? A lot of companies and Snyk is no exception have basically arisen on top of the notion that developers are expected to do more, are expected to be more related to the business, and to measure their value in terms other than lines of code. And I've been tooting this horn very much when I was weighing candidates to join us at Snyk saying, "Hey, remember the time when production outages were not a developer's problem, because why not just go to the IT guy, to the sysadmin person, and then alert them to the outage?"
[00:02:42] But the modern developer today is expected to be alerted, right? We all know what PagerDuty is about. And Snyk is actually riding this very wave by saying, "Hey, security should not be a third-party concern. This is not something to chase your developers with. This is something that you can actually expect your developers to own and to lead, let's give them the right tools." With all those additions of tools and new businesses comes a lot of responsibility. And I wonder, is this really the right direction? Are we expecting our developers to be Jacks of all Trades, Masters of None, with all of this? That's that's my main question.
[00:03:18] Jason: Do you think you have, uh, an answer to it?
[00:03:22] Anton: Right, I am the guest here, I'm supposed to be answering, not asking. Thank you for reminding me of that. I think that growth and expectations allows us to further distinguish between the different types of engineering, and to excel in specific areas. So we should not read into this momentum as if suddenly in order to be called an engineer, you have to know so many things at once but to choose what you, what you lean into, and what you excel at with the notion of not segmenting this world too much. Because if I define myself as "no, I'm just the backend engineer, if you have a front-end task, go to someone else", then I'm actually pushing myself away from the business impact I'm supposed to provide. And that's a very delicate dance, sort of, to choose your proficiencies while at the same time, measuring the amount of business value you can provide with those choices and making sure that you optimize for both.
[00:04:22] Jason: It's interesting. I would say that you and I would probably sit down and have a lot of fun talking about our shared experiences. I don't know if I actually have a great answer, but I can tell you what I'm current, how I currently think about it. And I'm curious once I say the following sets of things, what you viscerally react negatively to, or positively to.
[00:04:40] The way I, you know, when I started out my career, essentially, you worked on a component, you released it to IT. They released it in some Java or server farm somewhere or whatever, you know, you know, the drill. You even had database admins who would run SQL queries for you way back in the day, and that was a very different experience, obviously. Now, as you just said, a lot of people are saying engineer does all that top to bottom, everything there, including what might be the lowest level, way back in the day would be racking stacking servers, but this is provisioning systems or subsystems are understanding the security of the subsystems or networking layer or the IP tables as an example.
[00:05:22] And I can tell you, you know, as a distributed systems engineer, I didn't actually know about IP Tables. I didn't really that wasn't my world. But, once a couple levels up from that, was. And I think whatever we're doing to engineers a disservice right now, obviously, but what I think, I think that's because our tools have gotten abstracted and good to the point where you could become a full stack engineer on Rails and get 90% of the way there and do a lot of stuff in the database, but not all of the things.
[00:05:48] So we're saying, well, why don't you just do the rest of that in the database too? And we'll the reason why you don't want to just do the rest of the things in the database, in that particular example is because they're really f***ing hard with a lot of very specific information with a very long-tailed learning curve, if that makes any sense whatsoever. So my view of this is that we're actually gonna go towards a way that Full Stack Engineerings will have full stack again, but there'll be lines where other organizations and experts inside the company take over. So mine is security or infra or databases, but then application engineering becomes the practice of building the product, but all the other ancillary functions are about supporting all those engineers to do that. And again, like I I'll probably put SRE as an example of this as a special. SRE is a specialty in the world that we exist today and you don't want them necessarily being a full stack application engineer, but you're not going to ask the full stack application engineer, to jump into the SRE world, debug all what's happening on the edge.
[00:06:49] Anton: So I'm trying to, you know, map it against the, my mental models and how I see things shift from one side to the other, constantly coming back to this notion of a pendulum. I very much agree, that the shift towards this Full Stack Engineering motion is often misunderstood or was more frequently misunderstood in the past when people were looking down on, "oh, you're a full stack engineer, you know, a bit of everything".
[00:07:16] And there was another thing I don't know if you're familiar with. The definition of a full-stack engineer is an engineer that can mess up your system at any level of the stack. Okay. And with time, tools and practices rise to the occasion and you can see people basically saying, "well, I was sort of concerned to, you know, run a team, run a product, having no DBA next to me because the database was a pain in the ass in the past, and we need expertise in it. But as time went by, I found out that I'm fully capable of, you know, Googling something on Stack Overflow and figuring out "what this MySQL error message means and how to overcome it". And basically papering over all the issues they have, but with more sense of ownership and longer term success here, not just putting out fires.
[00:08:06] But then the pendulum shift again, and people do extract these proficiencies. And you brought the example of SRE, which is also still like, to some extent, a debated term, you know, two companies running two SRE teams won't necessarily expect the same of these individuals. And my personal view is that SRE is a very good way to extract the amount of practices that have to do with operational reliability and resilience of systems, especially distributed systems, and make sure that the people with their hands on the wheel of those systems are educated, are doing the right things or helping themselves become better at this.
[00:08:47] So an SRA team that would say, "no, no this is mine", would miss the mark, would act like that DDA who used to say, "I do not approve this SQL query. Go do it again until I approve it". But instead would say, "Hey, where are you now? What are your pain points? What hurts you the most in your deployment and scaling processes? Let me help you with that because there are practices."
[00:09:12] Jason: I think the big change in mentality in this approach, at least mine for my learning in the past is, in the past, each one of these functions was either a hurdle to jump over, to get approved. "I got to get the DBAs or Security to approve this." And so it basically, no matter how you organize yourself, it looked like a waterfall method. I will build this, then I'll get an approval, then I get another approval, then I get a release approval, then I get a stage and blah, blah, blah, blah. And now instead what we're, at least some of us in the industry are talking about is taking those and embedding them on the team. So if you're in the application engineering team, you actually have an SRE expert or a security expert or an ML expert, or any of those types of functions. And they're not reporting into that group, but they're dedicated to, or assigned to that team to help them as they build that.
[00:10:03] So for all practical purposes, they're a member of that team while building out those things. And you can get a lot of the pre-approval steps done earlier, but you're also using them to say "that's not how we think about the world over here. This is how we would do this sort of thing". And then you can have that back and forth really early in the design process or the development process, before it gets to the point where, "Oh crap, we just spent six months developing this. We're going to release it anyway and we'll fix it later, which means that all of us will have new jobs by the time that someone else finds that bug". That that's a big change. I think too, in mentality.
[00:10:38] Anton: Yeah. And it's interesting to see how it slips through the cracks differently in different areas. So SRE is like very hyped over the past several years, so a lot of progress and definition is developing in that area. Maybe to some extent, the notion of a full-stack engineer or application engineering preceeded that. So it's easier to take a team of engineers and say, "Hey, why don't you deliver user value through this product? And just tell us what you need. And we will support you?" But I still feel like areas such as Data ML, AI, those code worlds are still not as ripe in their ability to embed themselves into an incremental value delivery process. At least from what I've seen from my view on the world and I'm expecting very big jumps forward in that area of, our profession.
[00:11:30] Eiso: What you mentioned is something I see a lot. And I think what we're talking about is, overtime changes because of two different aspects. The first aspect of the actual ownership and context, right? I think what Jason's talking about, you know, when you take the notion of embedding someone in a team or squad who is, who comes from an expert background, if that's your optimizing SQL or applying Machine Learning into the earlier thinking of the design phases of a, of a new feature, that feels when it comes from, "Hey, this is an expert who can come in and can tell you how to do X, Y, or Zed", or comes in as a mercenary and helps in that team for a little while. But where I see it succeed is when this notion of the whole team has the full context and understanding of what you started with like the business value of what we're trying to deliver. Right. And I think over time, we're starting to see more and more of a shift from "engineering is here to build spec" to "this is what with product we believe to spec looks like, but this is also why we're doing it, what we're trying to achieve and the whole context. But I think then when you get a team together, you get a lot more value out of it, out of the interactions.
[00:12:36] But I think another part that we're talking about, and I try to kind of fast forward into the future and, you know, 10, 15, 20 years, is on the tooling side. Because a lot of the things that today we're discussing, if that's Machine Learning expertise or optimizing a database or SRE, those are all the kinds of things that we're actually looking to abstract away behind APIs or behind our cloud providers. So I'm very curious if we're going to end up in a world and this has a bit more my personal viewpoint, where in 10 years, or maybe faster, the vast majority of engineering organizations might not even have these, what maybe you and I would call it hardcore skills. Or very like, you know, expert-level skills, in their organization at all, because it's abstracted away behind an API to interface with or, or cloud service.
[00:13:22] Anton: Yeah. I agree with this very much. I think you can already see this in what used to be segregated as the domain of these experts, "don't touch them, they know what they're doing" and now becomes, "Hey, I'm no database expert, but I can run a postgres database at some scale by myself". So it's not even an API level obstruction. It's a, "I'm feeling comfortable with this box which I do not understand every bit of." Kubernetes is going through these phases as we speak, I think. There are changes where, uh, things are going in the direction of people relying more and more on managed services and redefining, "Hey, what's the, how deep do we really need to go into this? Yes, there is a lot of, you know, technical challenges involved and we can feel stronger in our Kubernetes strength, but do we really need that?"
[00:14:14] What I find very clarifying when these decisions are being made is constantly coming back to the business and saying, "remind me again, like who pays our salaries, our customers, are they buying Kubernetes knowledge from us or are they buying value that solves their problems? And they couldn't care less whether or not we run on top of Kubernetes?" This is very, very clarifying.
[00:14:37] Jason: I couldn't agree more. I think it's from a conceptual standpoint, what we basically have is we have this mass consolidation too, like oh, full stack engineers. And what we're basically seeing is, again, like we'll break some of those things up and we'll build other abstracted components around them or systems.
[00:14:53] So, good example here is that when most companies get to a certain size and scale, they actually develop an, what might be today be called an IDP. It's I think it's an emerging term, Internal Development Platform, where application, everyone kind of works together on, you know, for lack of a better term, almost like an internal Heroku, which looks to be like a cobbled together, Kubernetes, something, something, or whatever system that you're using. But you're saying "here's a set of standard practices. Some playbooks or paved paths or language, facts, or libraries, all that sort of thing that we would use together." And the goal there is to give a whole bunch of freedoms to one set of engineers, but a whole bunch of guarantees because of the way that you're putting those things together.
[00:15:32] And you layer on APIs or all those other things, and you start to, it starts to all come together again where, "yeah, I don't have to actually know how to build the ML system, but I can call the ML system and get those results back." And it's the ultimate promise, I think that what we wanted to go do when we were building out the services, microservices services, API constructs, it hasn't quite played out inside of organizations the way it has for building your own applications on AWS as an example. But I think it will in the future, as we build more sophisticated internally.
[00:16:03] Eiso: So Anton based on your experience at Snyk, and kind of, seeing these kinds of challenges, how did you guys do it there and what are some of the main lessons that, particularly in this context, that you feel that you've learned coming away from that experience?
[00:16:17] Anton: That's a very good question. I find myself talking very often recently about my experiences at Snyk and every time I do that, another realization comes to light. And so I'm always prefacing my responses with, with a disclaimer that the clarity I have gained is, is very much that of hindsight and that it's very hard to see these things as they, as they happen and lead them on.
[00:16:41] But that's, that's exactly our job. That's what we're trying to. A lot relies on fast iterations and experimentation, the best way to figure out. What's not what to do is to try out five things and decide which four are definitely wrong. And do so at a low cost to you, to the team, to your users. But remember that no one has the answers, really. No one really knows what will, you know, tick the box and be the right thing. So the muscle you need to practice the most is your ability to come up with theories with hypotheses. And prove or disprove them. Where do you do that? In production. That's the only thing that matters. You can run experiments, you know, in the background and crunch the same data over and over again, and glean out these amazing conclusions. But in the end, what matters is, so you put a button out there and users are either clicking it or not. Okay. It's like a single beat of information. This can be very intimidating and very, you know, demotivating because, "Hey, you don't understand how much of my brainpower went into the position and the size of this button and what happens once you click it."
[00:17:58] But I think it is the right way to look at it, is that it's very actually relieving to know that it's okay, you can do a lot of magic or just a bit of magic. And everything is measured in the terms of "did it, or did it not deliver user value in a way that made users vote with their feet and basically use or not use, what you've built.". No one has these answers, like another saying I often use goes, "there are no bad startups, only those that missed their timing." So you just need to experiment and to make most out of the time you have in front of , in front of your users.
[00:18:39] Jason: I get this question quite a bit. And I actually think about it myself often, too I'm curious, in your case. Let's say you're going to go back and do your time again, what would you most want to change about your own experience there? Not in terms of the way you interact with Snyk, but maybe something you believed or thought, or in hindsight would have done very drastically different.
[00:19:02] Anton: I don't think there is, anything that, you know, a regret I relieve. But I'll tell you this, the amount of back and forth that goes through my mind as I'm making decisions is, is often exaggerated and, you know, exactly what I just said about, you know, the, the product value and you put buttons out there and people either push them or not. As, as engineering leaders, our product is the organization that we built and things are never perfect in any organization, right, there's always friction. You move something under the lights, then the shadow is cast somewhere else. It's a, it's a constant, you know, optimization problem. It feels like if you just gave it some more thought and spent two more days and spoke to two more people about your thoughts, you would come up with a better solution to it. And what you often miss is that, things that imperfection persists during these two days, and you would have been better off acting sooner on the level of confidence you had in the past. Because resilient organizations can also withstand these changes and you don't like make a statement and then people want to see it roll out exactly as you said, across six months. You make a statement, you say, "Hey, look, here's what I'm seeing. Here's what we are seeing. Let me know if I'm missing something, but it feels to me like we need to tone down that part in our, you know, definition of success and dial up this part. Let's try for the next sprint and see what it's, what it's like and incrementally improve the organization that you're building."
[00:20:48] Jason: I couldn't agree more on this, but, and I'll go one level deeper because I think this has to do with obviously like the leadership in a lot of cases here. But I think this actually has to do with a lot of human elements that we, we lack inside companies from an empathy perspective and also fear, but also just a lack of understanding in engineering. So we say, "what is the correct way to go forward on one of these things," and no CEO or board ever wants to hear you say, "well, it depends," and it's like, "God, just give me an answer. Isn't it called? Computer science?" No, it's not actually, that's not the way it works in practice. But I couldn't agree more about making decisions faster, so the challenge I find that is, as a leader, people will say, well, you were wrong making that decision. So we start to backtrack and I think you have to, we all have to fight that hesitancy to do that. We want to, we'll couch our words more, or we'll seek more data on these things instead where we should be going more towards the experimental mindset. And then what we say is, "okay, if we go down this path, how might it go wrong and how do we mitigate? And when will we know how to do that"? So it's, it's a very different mindset in my opinion, to be able to pull that off and has less to do with the organizational structure and more to do with it. You mentioned resiliency and more to do with mindset and resiliency inside the organization to, to build that out.
[00:22:15] Anton: Two things come to mind when you say that. One is that it really helped me once I realized in full that leadership in engineering management roles in engineering are all about people. Technology is secondary to management positions. And the sooner you realize that the better you embrace the real responsibility, the better you act on it. Suddenly this amount of confidence you have letting go of the correct way of doing things. Because with technology it's either a zero or one, right. It's either correct or incorrect. With people it's not the way it works, and it's okay. Number two has to do with, you mentioned, you know, experimentation and going faster. That's basically it. That's basically, there is no number two.
[00:23:03] Eiso: So I want to mention something. As you guys know, I spend a lot of time with different engineering orgs and we discussed this a little bit in the last episode and you came back to it today, Jason, and you as well Anton. It's this notion between consensus building, and actually decisiveness as a leader. And those of you who listened to our last episode, we referenced this as kind of, you know, "the sales leader will say we're marching to point A and everybody marches and the engineering leader that we'll spend a lot more time trying to build consensus.". And I time and time again, come against this wall where, you know, senior leadership, a VP of Engineering at a high growth technology company with hundreds of engineers, in his gut, knows what needs to be done, has a lot of the people around him convinced that it needs to be done, but still will not be able to make that decision and say, "Hey, we are going to go march two X and we're going to go and improve this, and we're going to iterate and we're going to experiment along the way" because it isn't how, in many organizational cultures, and I think even in the engineering leadership role as itself, we've been trained to communicate.
[00:24:09] And so I'm very curious to hear your thoughts or both of your thoughts on this. And I also think there's probably some cultural differences from across the pond, you know, U.S. style versus you're based out of Israel, Anton, I spent a lot of time also in Western Europe and also happened to be able to see some differences on geography. So I'm curious to hear both of your opinions and also that in context of culture.
[00:24:31] Jason: Well, I'll start with one thing. I had this realization a long time ago when I was talking to Eiso you and I talked about this confidence competence spectrum a while back, I think it was. And there's a moment in time when this kind of dawned on me and maybe I'll save that for another episode again, but I was talking to a sales leader and they're talking about the quota for the quarter and all that sort of stuff. And I said, how do you have any confidence whatsoever that you said you're going to hit that quota? And to a degree, part of his answer. "Because we'll hit that number, come hell or high water. We will hit that number.". And I said, "but you, how do you know?" Then he started to walk me through all the various signal funnels that they have and they've got leads and they've got SALs and then they got NQLs and they've got SQLs and they've got the funnel from MQL to SQL and how these people are looking at it. By the time he gets down to where they will accept a quota number, they basically actually have, almost like, "Hey, here's the task list of people we're going to go after. And here's our confidence that we can convert some of these.". So they've done a whole bunch of pre-work over here. And in many cases, engineering never gets that opportunity.
[00:25:42] And so what happens is you're sitting there at an executive team offsite and somebody says, "Hey, here's what we're going to do for the quarter. How long do you think CTO, it'll take you to get this done?" "Well, I need to go back and talk to my teams to go do this." "Right. We need it by Friday." You're like, "Okay, well, hang on, sales has been doing this for five weeks to figure out their quota number, and now we're, we're figuring out from a priority station that we have four days from this offsite to go basically plan what we're going to be able to get done."
[00:26:08] So that's one thing. Engineering, one, probably doesn't have the same luxuries that other organizations have because it does feel like a black art to most people who've never engineered before built product. The other realization was how little actual information engineering has even about itself. And we've talked about this in the past too, Eiso, we don't actually know how good we are planning or how good we are breaking projects down into small granular unions or how efficient we are internally, half the time. And besides that too, we also get all of the unknowns. So, when something pops up in the quarter, we've got to reprioritize, from something coming in from sales hot, we have to reprioritize stuff. But how do we rebalance that tree by ourselves? Because we've already done the planning for the quarter that typically falls to engineering and engineering alone.
[00:26:55] Anton: Eiso I'll touch on the cultural aspect you brought out, because this is a great opportunity to tell all our listeners: Snyk's success is a hundred percent attributed to the team. And what made the team win in my mind, one major contributing factor to this was the fact that we started as a distributed team from day one. There was an office in Tel Aviv in Israel and office in London and the UK. And we set sail by basically saying " will not lead to this fact that we're a distributed company from day one, become a drag on us." Okay. "We will make the most out of it. And let's just find what is it there that makes the glass half full." This was a very big drive for me in joining the company saying "yes, I've never done that before. I want to see what it's like to build a distributed engineering team." And it turned out to be one of the biggest sources of my personal and professional satisfaction.
[00:27:54] We grew proportionately between Tel Aviv and London, making sure that no office becomes a headquarters and the other fields like a satellite, it imposed a lot of constraints on our hiring motion, on making sure we have, you know, an equal split in our engineering managers, product managers, directors of engineering later on, but it was all worthwhile because at any given point in time, and this is like, I'm spilling the beans here, and this is really a secret sauce of Snyk. At any point in time, I could step into a meeting and a team dealing with a business problem or a product requirement and building what needed to be built, always had people from both offices on every team. The first thing I would look for is trying to assess whether the team can benefit from dialing up it's Israeli mentality or dialing up it's British mentality. And it may sound like "okay, I'm now venturing into the non-PC territory", but I really, there's nothing to hide. This is something that needs to be on the table. Okay. It's a generalization. This is not about individuals. It's about mentalities and speculation, but it holds to some extent.
[00:29:05] A team that was to some extent, still sitting on the tracks, thinking where it wants to go, hashing through ideas without taking the first step would be a team where I would want to dial up the Israeli side. "Let's get moving. What can you show me?" Like let's not make it into a bigger problem than it should be. And I will see sometimes the exact opposite. A team will be running in circles, thinking they are making progress, not realizing they're always coming back to the same point and actually standing in place. They would say, "how about some longer term planning? Where do we want to be beyond next week? Okay. Where is the thing going on the longer term?" And dial up the British mentality so to speak? Uh, so I think it spans, Israeli doesn't have a monopoly on getting things done. The UK doesn't have a monopoly on thinking things through, but this combination was so unique compared to any, anything I've been on before and really contributes to my deep, deep appreciation of what diversity can do on a team. Diversity of many, many kinds, but specifically here, it was a diversity of mentality.
[00:30:22] Jason: So, I love your point, specifically because I think that as you, as organizations grow, I want to say the "default mode" is the way I would put it, of a set of individuals. You need many of those. So, good example, I'll just use I'll use app and infra as two examples because I, I intentionally used the default modes to GitHub's benefit in this specific example. But the default mode for application engineering tends to be "customer first, go fast", you know, build for the customer, go really fast. But the default mode for infrastructure was "up, stay up, don't break and be available". And if you think about those two things, if you want to make, if you want to optimize for each one of those, basically they would, it's oil and water. You take the system away from making changes and you basically just try to keep it as stable as possible. But the other is you go as fast as possible and you build all your own services into it, and you never even talked to infra, because you're going to just cobble something together off of AWS. And eventually now you're holding pager. But to put them together, you've got to scale. And we intentionally said, "we're going to ramp up X element of the development cycle now. So right now we're going to optimize for safety. Therefore, we're going to slow down on some sides to guarantee that we have more uptime or that some of the systems become more resilient or that we can suffer a catastrophic failure or DDoS or something like that over here.
[00:31:47] And everyone kind of understood that, but it allowed us to have that sort of back and forth and saying like, "we're going to take a little bit of all the good from each of our organizations and try to blend that mentality, but don't lose your soul X organization, you're you're the speed organization. That tends to be your mentality. But right now we're going to go a little bit more towards safety or security or something just for this period of time." And that wasn't super clean, people bocked against it. They're like, "why would I have to be concerned with that when that's their job and responsibility?" It's the customer, the customer is what we're after here. And as long as we can root back to that it allowed us to have that conversation though. But the ability to have those different viewpoints inside the organization, default modes was critical. And the homogeneous structure of everyone who is just fast or just secure or just stable, is not where you want to be. You want people that have all the different view- um, default modes, I'll call them.
[00:32:40] Anton: This brings us back to where we started with the question of, "are we expecting engineers to do too much?" And it actually highlights the point I wanted to clarify earlier. I think the ability to allow people to become professionals and to choose what they excel at, while at the same time holding teams accountable to a wide array of expectations, and to some extent, pushing against this definition of "well it's oil and water. It will be hard to mix." But instead say, "Hey, but you're just four people. Can you show me a plan that covers both these aspects?" Is basically rooted in the notion that delivery and success, and failure, by the way, is owned by a team. And never an individual. It is a team that sets out to achieve something. It is a team that celebrates the achievement.
[00:33:31] Jason: To put my language on this, this is going to sound rather, this has a potential to sound rather bad, and I recognize that upfront, but I will say it this way, from a mentality perspective. If I'm talking to engineers, break them down to three, three big buckets: An amateur, a pro-am, a professional amateur and then a professional. And just take those three buckets for right now. From a mentality perspective, an amateur,is almost unaware of a certain set of things that are outside their skillset. And when they're asked to do those things, they won't, out of a variety of different things. It could be fear, it could be lack of confidence. It could just be that they don't want to for, you know, any one of those reasons. Professional, pro-am, a professional amateur, I think goes into the more active phase of that, which is "that's not my responsibility, that should be somebody else's." And they're taking a "me first", as opposed to a, "we first" approach to that. So when you find a very selfish oriented team or individual, I would automatically kind of think that they're probably in their career arc where they're still figuring out that it's the team that's going to win, and then the company will win too. And the professional takes an attitude of "yeah, I don't know how to fix that, but we'll figure it out. And here's a way in which it would be done if it could be done", which is always the beginnings of the plan. So whenever I hear an engineer say, " can't do that". So the first question I ask is, "well, if it were to be done, how would it be done?" And immediately they string off five or six different things and it may involve more funding or new, a new data center, or, you know, a new team being up. And I'm like "cool, we have the beginnings of a plan". Well, when someone approaches their day job in that sort of, "well, if it were to be done, this is how it would be done way" they start to enter that superpowered professional category, in my opinion.
[00:35:18] Eiso: This is a super interesting way of looking at it. I want to come back to, to an earlier question, which is maybe very tactical, but it just, I just see it come up time and time again. And maybe if we start with you Anton, How do you empower your leaders, your directors, and your engineering managers to find that right balance between "we are getting that consensus across the team" versus, "Hey, we are going to go make decision X and we're going to go move forward". And at what levels do you see this differ?
[00:35:48] Anton: So, consensus more of it or less of it is it's more of an implementation detail of how the team wants to run. I will be involved in this level if I think that the team is so versed in, you know, flexing this muscle, they cannot consider flexing another and saying, Hey, how about we go a different way on this decision?
[00:36:10] What is empowering in my mind is constantly pulling the people reporting to you to your level and saying, "So here's what I see. I don't want to chew it down for you. I will explain. Please feel free to ask me questions", but I won't make concessions, I won't say "no, no, you will never understand what four nines of availability mean to our customers. Let me just walk you through the recent outages and ask you to fix these things." But instead to say, "so if this is the worldwe live in, how can you and I agree on what success would look like, without me spelling it out for you." Jason, you mentioned the amateurs professional-amateurs and pros, the same can be applied to engineering managers, and maybe I would use a different, a different scale of basically saying, well, how experienced are you? Do you know yourself well enough to be responsible for the success of your team and to support your team? Or do you have the experience to know what different practices look like and when to pull this or that card for your sleeve, because you have more than one approach when, when a challenge arises?
[00:37:20] And then I would expect a more experienced engineering manager to basically talk to me about alternatives. And to say, "I can get this far with this. I can get this far in the other direction with that". And then they will usually also state what is their preference, they would say, "but this seems more optimal for the organization". My job is done.
[00:37:43] Jason: That is a very succinct way to talk about that as well. And I think that what you've also shown is what engineering leadership really is too. It's not just take this task, take this task, take this task. Here's the reporting on this task. Let me do the one-on-one and make sure that they're feeling cared for. It's that part of it, and it's also guiding the teams through that conversation as well. And we get into this quite a bit inside organizations. And let me give you a story here too. When I first joined GitHub, I was going over the levels and trying to figure out what engineering manager meant at GitHub and the director and VP and all of that. And there was no real structure at GitHub, so that let's get that out of the way first, but they asked me what my philosophy was. And I said, well, I take a very strict one, and I start at the director level and will slowly work down then up. And I say, "it all comes down to philosophy. What do you think the director level is?"
And I think most of the industry thinks as director, as manager plus plus. You basically have, if you're a manager, you got one team, maybe two, but usually typically one team maybe, and then a director, you have several teams or whatever, but you're basically manager plus plus and I said, "I would rather take the approach that director is VP minus minus". And the distinction between those two is that in manager plus plus oversight is going to be needed somewhere, and you're going to be talking about just that conversation. At VP minus minus, it's almost like you can give them direct responsibility for a hundred million dollar budget at some point, because that's effectively what VPs would end up doing. You have that responsibility, you gotta manage it. And the difference in mentality of that changes how your, what you expect out of manager, director, and VP at that point. But what was really, really, really important in that conversation was from a philosophy perspective, what you expect from a sophistication of professionalism, a long-term long range thinking map back to short-term objectives and the ability to hold both of those things in context at the same time and get their teams to understand that as, as well as they possibly can.
[00:39:44] Anton: I think the career laddering is an amazing topic for a long string of podcasts. And it was definitely one of the, one of the few, most impactful things we've delivered as managers in the engineering organization at Snyk.. We did this once clarifying the distinction between Software Engineers and Senior Software Engineers dealing with the question of what seniority actually is, what does it mean for us? And then a year later we applied a similar approach to the distinction between engineering managers and senior engineering managers.
[00:40:18] What was always in the background of this process is the fact that we were scaling as an organization. Okay. That was 2019 was an amazing year. We nearly tripled the size of the engineering team, but before and after doubling the team size every year was, was, that's what we do. In this mindset, the importance of making people around you better, and focusing on the success of the ever-growing ever-splitting, ever-reinventing itself team, where like 80 to 90% of what seniority meant for us. And I remember very vivid chart I put out in front of the teams saying "seniority is a combination of technical competence and the desire to make others better". You can combine these two in various, you know, proportions. So there is no single definition of "you need to cross this box in order to be considered a Senior Engineer", but let's now talk about, how are we making progress? How do you assess yourself right now? Where are you moving? That was very, very clarifying and tapped into the desire of people joining a very highly scaling company to see that reflect on their professional growth. I think we tapped into something that was very strong of a motivation for every, every person on the engineering team. And then the company as a whole.
[00:41:39] Jason: I think it's one of the least satisfying things engineers can actually hear too, exactly what you just said. Engineers or engineering managers tended to react to "I would like a checklist of once I've satisfied these things, I could become a Senior Engineer or a Staff Engineer" and, you know I blame Google for this, for what it's worth. Because I think Google does operate this way, but when you're 100,000 people or 200,000 people have to operate that way. But it's very different when you're 50 people growing to 150 growing to 300, there is no checklist of things that you can possibly put on paper to go do that.
[00:42:15] And again, I think this goes to some of the seniority conversation as well is if you understand that too, actually shows that you probably understand the context of the company that you're joining or a member of, as opposed to maybe what Google or Facebook is today as well and what they were like earlier. But I also recognize that it's very unsatisfying answer for many people in the industry. But for what it's worth, I 100% agree that that is the right approach because when you're basically shedding your skin every several weeks in terms of size of company and how you're growing and how rapidly you are, that you have to take an approach that looks like that way.
[00:42:53] Anton: But also this conversation is not a zero sum game. It's not like, "I have these promotion cards and people want to grab them and they don't want to give them away.
[00:43:02] Jason: It's the very definition of a startup. It's not a zero sum game. The whole thing is to raise the entire pool available. So it's, it is interesting. It's all about mindset mentality.
[00:43:12] Anton: Abolutely
[00:43:13] Eiso: I think on that note, it's worth ending our episode today. It's all about mindset and mentality. I think underpins a lot of this podcast, not just today's episode, it's been an absolute pleasure having you with us Anton. It sounds like we have a lot more topics to discuss and you're going to have to come back in the future, but I think that's going to be a growing trend for all of our guests.
[00:43:33] So just wanted to thank you very much for today.
[00:43:35] Anton: Thank you very much for having me.
[00:43:36] Jason: Anton it was great to have you on, this was amazing.