AI Driven IT for Less Toil with Emanuele Sparvoli, Director of IT at Intercom

Emanuele Sparvoli [00:00:00]:

We need to redismantle that traditional idea of it and we need to see IT as not just a service provider, but an accelerator for business.

 

Arek Dreyer [00:00:18]:

Today's guest is Emanuele Sparvoli, director of IT at Intercom, where he's rethinking what IT can be not just as a service function but but as a strategic driver. He's led a shift toward customer centric IT building systems that are not only secure and compliant, but also intuitive, automated and designed for the way that people actually work. In this episode we'll dig into how he's rebalancing security and experience removing friction through smart automation and building systems that give time back to both users and IT teams. Emmanuele, welcome to Patch Me If You Can™.

 

Emanuele Sparvoli [00:01:00]:

Very nice to meet you Arek. Thanks for having me.

 

Arek Dreyer [00:01:02]:

Managing non human service accounts can be a real security puzzle. What made you realize that the industry standard approach wasn't cutting it anymore?

 

Emanuele Sparvoli [00:01:12]:

You know, non human service account are not a new thing. They've been there since forever, since at least we've been starting using systems and best practices come into place where not all privileged accounts are connected to a human for security and for best practices. But also with the advent of identity provider, when you have an identity provider, what you do is connecting systems to IT and then you create non human accounts or let's say you deploy an appliance, you create a non human account and as your stack grows and and your business grows, this number of non human accounts start to skyrocket. Like I think at Intercom right now we probably have something like 350 non human accounts between service accounts and robots and accounts that are created for automations. The issue is that as threat evolve, traditional username, password and MFA also are not a good solution for security as well. But while for humans the landscape evolved so you have like biometric factors, fingerprints, face, ID force, non human accounts and service accounts which are usually shared between people, between engineers, you can and don't belong to a human. You can't really use biometric factors at scale. Let's say I connect my biometric factor to a non human account, then it only works for me.

 

Emanuele Sparvoli [00:02:39]:

So most of those accounts in companies where you actually have best practice usually are in secure password managers where you have like username, a password and a DP token, like you know the classic scan the QR code and you get the code that changes every 30 seconds and if you are lucky also the password manager is behind the device trust, behind saml, single sign on and biometric factors. So you get that is the best level of security that you can have. But you may imagine it's not really that safe from extended threats because people may copy those passwords, may add tokens, may have like personal password managers or even your most secure password manager could be compromised. The endpoint could be compromised. So if you don't have a biometric factor connected to it, anybody that is an event password and the token OTP token gets generated every 30 seconds can access that account. We saw in the industry like okta themselves in 2023, in October they had some accounts leaked from their external customer support. The person was using a personal password manager. Google Password manager, got Comprom, malicious actors got access to that and got some data of Okta customers.

 

Emanuele Sparvoli [00:03:50]:

It's a public incident, so nothing secret there. And so Okta themselves are not immune to this. It's like it's really hard to secure them because by their nature they need to be shared.

 

Arek Dreyer [00:04:02]:

Why do you think securing non human identities is still such a blind spot for so many organizations, even those with a strong authentication system in place?

 

Emanuele Sparvoli [00:04:12]:

I think there's a, there's a combination of factors there. Like first of all there's a problem awareness. I think people think they're not used, they are secured in a password manager, it's fine. I think the security bar in a lot of companies not that high and a lot of business businesses are not that mature. So we need to accept that. On the other end, like there's really not much in terms of like products. There are no products to secure this. There are no really consolidated best practices.

 

Emanuele Sparvoli [00:04:43]:

So the thing that we did was develop the tool that actually keeps the accounts locked. So we have the accounts behind the policy. Okta is our idp. So we have those accounts behind the policy that doesn't allow login to the accounts. And I'm painfully aware of it because every few months I need to use a service account, log into the system to check some logs or some building details and oh, it's not working. I tried three or four times and I say it's not working, there's a problem. Then I remember I actually need to use the tool to unlock it. But what happens is the accounts are permanently, they're valid accounts.

 

Emanuele Sparvoli [00:05:18]:

They work. If you generate an API with it, it's going to keep working because the account exists. But Okta doesn't allow logging into it in any way. What you do, you go into this tool that we developed. You log in as a human with your biometric factors, very secure on your trusted device. You choose which accounts you want to unlock. You unlock it for 15 minutes, then you can use the credentials in your cluster manager to log into it. After those 15 minutes, it gets locked again.

 

Emanuele Sparvoli [00:05:46]:

So that's our solution. That's actually what we are trying to pitch to Okta as well. I do not understand why there is no will on their side to develop a solution. Like many, I don't think we are the only ones. So I think many are suggesting them, these kind of solutions, but for some reason they're not there yet. They could be close. We had a conversation on Friday, but so far I think one of the big issues that there's no easy tool that you can just deploy and remove the problem.

 

Arek Dreyer [00:06:16]:

At Intercom, you ran into growing pains with manual access management as the company scaled what were some of the biggest pain points for your team and your end users.

 

Emanuele Sparvoli [00:06:30]:

As we said before, as the business grows, your stack grows, so you grow in number of users, but you grow as a number of systems, right? So if you rely on manual process, where people request access to something, even if everything is documented and people know who to ask, when to ask, where to ask, there's going to be a lot of manual work behind the scenes to actually grant them the access they need, the access to the system, the role that they need for the job they need to do. That's a lot of manual work. There's a lot of toil. And in general, it's work that doesn't really bring much value. Like you're just basically giving people access, which is something that you need to do. But I don't think it's the most valuable type of work you can do. We tried to introduce some systems, some better than others. In reality, it's not an easy problem to solve because there's no solution, there's no size fits all.

 

Emanuele Sparvoli [00:07:22]:

Every system is different. Every system requires a different process. People want it to be easy and for a reason. You know, you. You don't want to create too much friction. You don't want to create like a process that is convoluted and takes a lot of approvals. And it's not, it's hard to understand. The customer experience could be terrible, you know, in many of the products that are on the market.

 

Emanuele Sparvoli [00:07:43]:

So it's not really an easy problem to solve.

 

Arek Dreyer [00:07:47]:

And it's interesting you said the customer experience. So be explicit about who you mean by the customer.

 

Emanuele Sparvoli [00:07:53]:

So the first thing that we did at Intercom when we created Acti Team nine years ago was like, we are a business inside the business, we need to consider the users of Intercom as our customers. It changes a bit the dynamic on how usually it teams think. Because once you consider the users your customers, then the customer needs to be satisfied in order for you to be successful at your job. Like if you have no customers, the customers are unhappy. If you lose your customers, I mean worst case scenario you lose a job. But for sure you're not doing a good job, you're not creating user satisfaction, you're not bringing value.

 

Arek Dreyer [00:08:31]:

What approach did you take to solve this? And how did you balance speed, automation and security along the way?

 

Emanuele Sparvoli [00:08:38]:

So the first thing that we wanted to do, we wanted to leverage all the possible technologies in our stack. So unfortunately not all systems in the world support SAML sync sign on and scheme. Many of them do. And you can automate provisioning up to certain level. If there's a poor scheme, you can even automate up to the specific role and permission. So we definitely want the system to leverage that. But on the other end we wanted the customer experience that was fast. Least friction possible.

 

Emanuele Sparvoli [00:09:06]:

It's impossible to not have any friction. But least friction possible is definitely the principle there. And we also wanted people to be able to. I'm in a browser, I want to, I like to work in my browser. I'm gonna have a web interface, I can ask for the access, I'm in Slack, I can ask for the access. And so like we don't even want that people that needed to approve the access needed to go maybe open a webpage. So it would be important, for example, if there is luck, they get the notification, they can approve it directly from there. But the other requirement is that it needs to be safe, it needs to be behind, it needs to be zero trust based, needs to be device trust based.

 

Emanuele Sparvoli [00:09:43]:

So you need to be on a trusted device and need to make sure that it's really you approving that access. Because some of these access can be very sensitive. So we opted for Lumos. We tried octi GA actually and we invested a lot in Oct iga. We tried to make it work for around a year, but the customer experience, the user experience just wasn't great. And so we decided to go for Lumos. And Lumos was a much younger company. They are now probably a bit less known, but they were really good.

 

Emanuele Sparvoli [00:10:10]:

They even developed some specific features for the Slack integration so that we can make sure that when you approve the access, you go through a device trust authentication, make sure that it's you and they are on a trusted device. They built that feature for us. And the other factor that we could customize the process for each system, for each role inside certain systems. And it's all based on how risky is the privilege. Right. So if the privilege is very risky, that could be more approvals. If the system doesn't contain sensitive data and the role is pretty basic, maybe read only, then it could be even automated. Also could be based on your role.

 

Emanuele Sparvoli [00:10:48]:

Like if you are in a certain role, we assume that you should have access to that should you ask for it. And so it's going to be automated. It's the flexibility is the secret there. We are definitely not perfect at this. We have a pretty ambitious target to be able to approve every access request within an hour. We're not near that, but that's what we strive for. Like the bar is super high and we strive for that and hopefully this year we're going to make even more of a dent to get closer to that target.

 

Arek Dreyer [00:11:15]:

Is that an hour, 24 hours a day globally, seven days a week?

 

Emanuele Sparvoli [00:11:20]:

No, it's working hours that we can cover. Between North America and emea, we are not following the sun. We actually do not have presence. We have a Sydney office, but we don't have presence there. So we're not following the sun. And it's not 24 hours in the sense that the weekend is not covered, but we are on call. So if you actually are in an incident and you need access to something, you can page somebody from it and they can approve your access in case of emergencies.

 

Arek Dreyer [00:11:46]:

When we talked about you coming on the show, you emphasized multi channel communication using Slack, the web, whatever tool your users were already using. Why was this such a critical piece?

 

Emanuele Sparvoli [00:12:00]:

I think that in order to provide the less friction possible in your processes, you need to meet the users where they are. And so multichannel is very important. If I'm in Slack, I should be able to reach out to it. If I'm on a web browser, I should be able to reach out to it. If I'm on a mobile device, I should be able to reach out to it. That removes a lot of friction because you can reach out to it wherever you are. And as I said, this is all we strive for this. We are not good at this everywhere.

 

Emanuele Sparvoli [00:12:30]:

We're pushing a lot on this specifically this year because I believe before Slack was kind of not really one of the channels we're pushing that much and we are actually making thanks to AI. So we are making strides there and we want it to be one of the channels that people can use because they spend a lot of their time there. But that's why we believe that multi channel is important. You need to meet the users where they are and every company is going to be different. Some channels are going to be more important than others for different businesses. But we want to go over one.

 

Arek Dreyer [00:13:02]:

Of our big themes is that the patch is just the start. Once you rolled out this new system, what changed for your team and what kind of work were you able to do that you couldn't get to before that you were finally able to get to?

 

Emanuele Sparvoli [00:13:16]:

Well, I think AI can help a lot. You know, AI agents can do a lot of the work of doing the first line of triage and sometimes can even just answer the questions, you know, like if you look at Fin, for example, like Fin is able to solve up to 60% for many customers and even more than that in some cases. So you can definitely leverage AI there. But in general, even if AI wasn't there, I do believe that that's the kind of work that you should take on. That's how you create value. You are there to, you're not there to gatekeep, you are there to facilitate the work that people needs to do for the business to be successful. And that's actually why you bring value. You need to make things faster for people.

 

Emanuele Sparvoli [00:13:57]:

And the reason why you're there is because you need to make them faster but still keep them secure. So there's always a balance, right? That's why we say least friction possible, absolutely no friction. Because if you wanted absolutely no friction, you just give access to everything, to everyone, and that's easy, right? So you need to strike a balance. And that's the difficult part of the job.

 

Arek Dreyer [00:14:17]:

One of our big themes is that the patch is just the start. Once you rolled out this new system, what changed for your team and what kind of work were you able to do that you couldn't get to before that you were finally able to get to?

 

Emanuele Sparvoli [00:14:31]:

Yes. In general, when we deploy processes, as you say, it's just the start and we need to see how it performs. And while we design it before deployment, we usually identify some metrics to see how successful it is. And if we don't meet those metrics, we need to make improvements. But the reality is that with all the automations that we create, with all the AI tools that we deploy, what we want to do is to remove toil not just for the business and the users, but for ourselves. And once you remove toil for yourself, you can, you can then focus on work that brings more Value, which is like improving your processes, make things faster, make things more efficient. That's actually where you bring value as an accelerator. You know, it is an accelerator of the business.

 

Emanuele Sparvoli [00:15:12]:

The value is brought by improving processes, making things faster, making things easier, saving time, and that's a lot. For example, we deploy, we deploy the Enterprise search at the start of the year, which is connected to a lot of our systems and makes the data that people already have access to available to them with a simple search. And that you, you can, you can't even imagine how much time people are saving by being able to actually search information that fast. That this is how you bring value. This is the things you focus on when your job is not just style, it's not just like moving things around, clicking buttons and doing repetitive tasks.

 

Arek Dreyer [00:15:48]:

Before the show, you mentioned also launching an IT engineering initiative focused on automation and AI. What kinds of capabilities has that opened up and where are you seeing the most impact?

 

Emanuele Sparvoli [00:16:02]:

Yeah, so as I said, at the start of the year, we deployed enterprise search and we did it through a platform called Glean. And they were already a leader in enterprise search, where you can basically ingest all your data across your systems, including permissioning, enables people to search that data really fast, really quickly. But they also have a lot of AI functionalities where you basically have a model of your choice. In our case, ChatGPT. Vorhoder, you can have a conversation with the AI, just imagine ChatGPT, but it has access to all the data that you have access to. And sometimes people even didn't know that they access to that data. And it changes things and accelerates things a lot. It's really a force multiplier in what you can achieve in a very short time.

 

Emanuele Sparvoli [00:16:51]:

But through that process, when we deployed that, we realized we are onto something here. There's definitely something we can do with AI tooling. And we started thinking we should probably just focus on that. Because the point is, a lot of companies are very aggressive in how they introduce AI tools and I believe they get a competitive advantage out of it. And so the slower you are implementing those technologies, the more you are at a disadvantage compared to this business. So the point in time to bet on these kind of things, it's not. And so we said, let's sit down. We want to find use cases for AI tooling and automation and see what we can build.

 

Emanuele Sparvoli [00:17:36]:

We started to talk to stakeholders across the business and we decided to move really, really fast and dedicate 90% of the resources of the team for six months to this. And so Far we didn't only just build some automation but we started to build the big thing that came out of this is building our own AI agent. It uses like a bunch of technologies, Lean for example, to retrieve the data. We have a combination of LLM, ChatGPT, even Fin, Interconf Fin is in there sometimes to validate the answers. And what we did is that we deployed those AI agents in slack channels where people used to come and ask for support. Like the HR channel, the legal channel, soon the IT channel where people used to come and ask to a human if they could help ask for a question and now the agent as in with all the ingested information that we have from the company. We develop these custom agents per channel where they are specific to the, to the team of the channel and they can answer a lot of the questions. Before they would just answer a question one off.

 

Emanuele Sparvoli [00:18:40]:

Now they became conversational. We created a feedback system where people can give feedback to the answer if the answer is good or not. Just so like we can understand if they're asking a question or they just want to talk in the channel so that we don't reply to things we shouldn't. We can also understand that if a human came in and he started to have conversation with the other human so the, the agent stopped responding because there's a human ear that took over. So there's no, there's no need for me to be there. I think that as we are seeing in general in the tech landscape, AI is going to revolutionize. It is revolutionizing customer support and I think that's what we need to do also internally for internal support we need to use AI tooling and automations to create self serve opportunities which allow people to get what they need faster, remove soil from the teams and they can use the time they save on work that brings more value to the business.

 

Arek Dreyer [00:19:40]:

Love it. And it's revolutionizing, not replacing.

 

Emanuele Sparvoli [00:19:47]:

Yep, I think so. Because there's always gonna be what I tell what I tell our tube teams that is the one owning support is like if your job was just answering questions, you don't have one anymore. That's easy. But because you don't have to do that, your job is changing. You are the engineer, the administrator behind the scene that build all these processes that make sure that everything behind the scene works correctly. I think it allows people to professionally grow and elevate themselves and doing work that brings more value while before they were just busy answering questions. I do not think that AI is fundamentally replacing humans there. It's really Helping humans, not doing the work that they don't want to do.

 

Emanuele Sparvoli [00:20:36]:

Let's be honest, very few people enjoy answering questions all day. Right. I don't think that's super enjoyable. So if there's an AI that can do that for you and that you can do work that brings more value, that makes you grow professionally and that's it's great value for the business.

 

Arek Dreyer [00:20:54]:

As you've made all these changes to optimize efficiency, what's one outdated mindset that you've had to leave behind and why?

 

Emanuele Sparvoli [00:21:02]:

I think that we need to really dismantle that traditional idea of it and that we, we need to see it as not just a service provider, but an accelerator for the business. If you are a forward looking leader in it, you are listening to the needs of the business and you're adapting your processes, your infrastructure, you're using AI tools, you are facilitating the introduction of AI tools, you're taking some risks, some calculated risks. As we said, it's always a fine balance. I don't think that we can really rely on the structures of the past. We cannot really rely on doing the work as we knew so far. Everything has changed and we need to change with the time. So we need to be flexible, we need to be AI first.

 

Arek Dreyer [00:21:51]:

Just to be explicit, what is the, the traditional definition of it that needs to be left behind?

 

Emanuele Sparvoli [00:21:59]:

It's just like the human offering a service which is there and does everything and controls everything. And it's basically the focal point of all the processes as a human there. And it's a shocking point. I don't think that we should accept that anymore. Before there was nothing else that could do that job. Now there are tools that can do that job. And by not being the choking point, you can accelerate the processes and the business with it. You can create innovation and you can spend time, as I said, doing things that bring more value, that create value for the business.

 

Emanuele Sparvoli [00:22:34]:

And in the meantime, you can also grow yourself professionally. I have kind of a positive outlook to this and I know that's probably not the most common that you hear when people talks about AI. I don't think that the AI is going to replace us. AI is going to allow us to do more valuable work, to learn more and to shed the work that nobody wants to do.

 

Arek Dreyer [00:22:57]:

If you could instantly patch one thing in your world, what would that be?

 

Emanuele Sparvoli [00:23:02]:

I think the biggest obstacle for us to automate everything and to make everything self serve is the fact that a lot of systems in the industry still do not support, fully support Samol and Scheme and you cannot really fully automate provisioning and the provisioning of access across the full stack. Unfortunately there's still a problem and I always tell Okta for example, you can launch all this, all the lifecycle innovations that you want but if the task companies that that create those systems and those apps don't adopt those standards, there's no point like it's just not going to work. And unfortunately if you look around there's still a lot, there's still a lot of system that maybe there's a proximal single sign on many of them don't but definitely don't support Scheme which fully support Scheme which would allow full automation of provisioning and the provisioning so if I could fetch something in the tech landscape I will make everyone support in a full spec of SAML and Scheme.

 

Arek Dreyer [00:24:07]:

Thank you Emmanuele for joining us on this episode of Patch Me If You Can™. If you like the episode, hit follow and share it with someone who's ready to lead it and security from the front. We'll see you next time.