You Can't Automate Buy-In

00:00 — Jamie Parker
IT automation is changing the world of tech. It's helping teams achieve more, be more consistent, and move faster, while also reducing risk, helping meet compliance requirements, and reducing long-term costs. That's what they say, anyways. It's not wrong, but it's also not the whole story. To make the most of automation, teams have to change how they work. And that's no small task. Inertia is a strong force. Learning new skills and setting new habits takes time. And that's all when the team is ready to put in the effort. I'm Jamie Parker. I've seen change in my 20 years in the tech industry from bare metal deployments to virtual machines, and then containers. And I've had to keep up. There's always more to learn, especially beyond the tech itself.

00:57 — Jamie Parker
Welcome to season two of Code Comments. We're going to explore what it takes to adjust to automation by talking to teams who've done the hard work. They're not just telling us about the tools they used and how they coded their scripts. They're sharing how they had to convince, compromise, and coordinate with each other to get their processes automated. And we get to hear how that helped them reach their goals and more. Getting your team's buy-in is crucial to success. That's a lesson World Wide Technology had to learn before they could start seeing positive results in their automation journey. Getting that buy-in took a lot of trial and error.

01:36 — Jason Kayser
20% of the engineers bought into doing it that way and 80% were kind of looking at it as like, this just doesn't make sense.

01:43 — Jamie Parker
As a technology reseller, it was crucial that they get it right. If they couldn't convince their own team to automate their work, how could they hope to do the same with their customers, and do it successfully? Jason Kayser is the director of infrastructure engineering at WWT. He leads the team that builds services the company sells, and they use those services every day. Back in 2016, automation became a priority for the company. They embarked on a six-year journey to find the tooling and solutions that would really work in practice.

02:17 — Jason Kayser
It was really driven as a management edict. If you look at some of the goals at an executive level, it's to double every five years. And when you start looking at that and you see that growth for the company, IT's got to scale along with that. And as you look at, again, how do you scale? Like most IT departments, there's not a desire to scale your people linearly with the way the business is scaling. There's a desire to scale. The business is going this fast, growing at 2X the speed. And IT, the goals from the business is to grow at 1.5X, not to grow at the same level. And the only way to really do that is to find ways to automate the task their engineers or analysts have been doing manually and make that go away.

02:59 — Jamie Parker
Corey Wanless is a technical solutions architect at WWT. And one of his first tasks was to figure out what this whole automation thing was going to look like from an engineering perspective. What were the problems that automation was actually going to solve?

03:14 — Corey Wanless
There's an aspect of, when you're doing things manually, especially in a larger scaling organization, you're going to find yourself where one engineer does a deployment within a lower environment. But then because of availability or different reasons, maybe a different engineer does it in the test environment or the production environment. And what happens organically because of that being manual, where having humans injected into the system, the manual way of doing that work essentially created a future problem, a tech debt, if you will, where further down the line, months or weeks later, there's an issue that occurs and nobody could figure out why that issue happened. But what it eventually was boiled down to was differences between the different environments. And so automation really comes in to help make things not only more efficient, drive to outcomes faster, but also makes the changes that happen within the environment more reliable and scalable.

04:10 — Jamie Parker
So far, these sound like typical motivations for IT automation. They started with Puppet to handle configuration management. It started out all right, but as we heard earlier, something went wrong. But what was it? What's not to like about efficiency, reliability, and consistency? Why did around 80% of the engineering team not take to Puppet?

04:32 — Jason Kayser
That was the tool that management and IT leadership was pushing at the time to drive, air quote, "Automation," in the environment. And again, it was really around configuration management. And unfortunately, we're trying to use it in, let's call it unnatural ways, or ways that really didn't fit the best way that Puppet was positioned. It seemed like it created some angst for our engineers. Puppet's got a learning curve with it. And when you push into unnatural spaces, it's even harder to learn and harder to manage and harder to maintain. And we saw that, over time, engineers weren't really buying into the thesis that we had out there of configure management to, again, drive streamlined automation.

05:11 — Jamie Parker
In the natural spaces, doing what it was designed to do, Puppet was a great tool. But WWT was looking to automate processes that weren't within the scope of Puppet's capabilities.

05:24 — Corey Wanless
I would akin it to an analogy of: I bought a minivan because I have three kids. And it's serving that purpose. It's helping haul our kids around from activity to activity and dealing with that logistic issue. It's solving that problem for us. But then we were trying to then use the minivan like a truck. Initially, it's like, okay, we're going to put eight-foot boards into it. It'll fit. Just take out all the seats and make it all work. But then every time I do that, I have to take the seats out, put the seats back in. It's very clumsy, doesn't fit well. It's not a very efficient way of working.

06:00 — Jamie Parker
Between the choices of doing things the same way they'd been doing it or trying to automate the task in a clumsy, inefficient way (words that send shivers down our engineering spines), the team chose the familiar. That didn't bode well for the company's goals. When the teams didn't adopt the tool, they weren't automating their repetitive work in ways that help the company scale. In fact, it ended up creating more work than it replaced. Luckily, not everyone gave up on the idea of automating their tasks. They decided to try something else and bring that to management as an alternative. That moment was the turning of the tide.

06:44 — Jamie Parker
WWT was stretching Puppet beyond its capabilities, but that wasn't the end of their attempts to automate their workflows.

06:52 — Jason Kayser
At that time, I was a product owner for our Windows and Linux application teams. And one of the guys on our Windows team had an idea of, hey, we're trying to automate more. We're not seeing a lot of success with the Puppet aspect of automation as a whole.

07:09 — Jamie Parker
It turned out a lot of engineers were already writing their own scripts to make their lives a bit easier. But the scripts were scattered and isolated. Those scripts were helping individuals do their jobs a little bit better. But without coordination and planning, they were nowhere near meeting automation's potential. One engineer took notice and had an idea.

07:31 — Jason Kayser
We've got all these things sitting out here. They're independently used by people. Are there ways to framework these things together?

07:37 — Jamie Parker
And just like that, everything was about to change.

07:42 — Jason Kayser
ServiceNow is one of our major platforms we use organizationally. We use that for ingestion of incidents, for all our change processes. We use it for requests from our stakeholders and our customers. So he looked at that as a entry point into automation and then, I don't want to say Frankenstein, but built something that he affectionately termed the ServiceNow automation platform.

08:05 — Jamie Parker
Also known as SNAP.

08:08 — Jason Kayser
It was pretty cool, actually, at the time. Team started using that and started seeing, all of a sudden they're like, "Hey, there's some adoption coming around this thing." The engineers were automating things that wasn't possible with the way we had Puppet deployed or wasn't even really possible with Puppet. But it was pretty cool to see they saw a problem that we weren't addressing and found a way to address at the time by creating a framework that they could consume and use and share amongst each other.

08:34 — Jamie Parker
With SNAP, WWT could incorporate existing scripts into a broader framework. That made them easier to share and use. And the expanded scope caught the interest of the larger team, creating a snowball effect of adoption. It was custom-built for their use. They were starting to accomplish their goals, but not without hitting a few snags.

08:56 — Corey Wanless
Well, it had a lot of successes. A downfall that we had within the solution was that it was built and managed by a single individual. He managed it, he supported it. We contributed to it. We were beneficiaries of it. But ultimately, if something was broken or if there needed to be a feature, he was the one that was the mastermind and the one to help implement that said feature.

09:18 — Jamie Parker
He became a bottleneck, and bottlenecks don't scale. It was also reliant on a single language that not everyone was familiar with.

09:27 — Corey Wanless
You had to know PowerShell to be able to automate the thing or be able to have PowerShell call something. And so while that worked well for a good part of the infrastructure teams, one of the teams that had a harder time with it was obviously the Linux side. At the time, there wasn't really much support for running PowerShell on top of Linux, if at all.

09:50 — Jamie Parker
As much of a wider success their SNAP platform was, it was still coming short of the company's goal to scale their operations faster than their teams. That's when they found another option.

10:01 — Corey Wanless
As we learned about Ansible and started to look at adopting it, one of the biggest things that attracted us to it, not only the fact that it could automate across multiple technology disciplines, but was the fact that we had a whole community behind us to support it.

10:17 — Jamie Parker
It was yet another tool to try, to learn, and to convince everyone to use. And that took some work. Corey went to management and asked for some help to stand up a proof of concept in the hopes of getting that all important buy-in.

10:31 — Corey Wanless
We took a corner of the office, quite literally the corner, and spent a whole month doing it. And the funny thing was is we brought along one of our consultants that was helping us with this transformation of our ERP solution. And she said, "Nobody's done this before." Just laughed it off in a way. And we said, "That's just ammo for us. Let's go. Let's try to do it." And she was on board, but she was skeptical of our ability to get to the end state that we were looking to.

10:58 — Jamie Parker
They got there in the end, and people noticed their success.

11:02 — Jason Kayser
You get these four to five engineers working on it, other engineers are seeing this, they're getting excited about what's happening with it. And all of a sudden you start seeing this snowball effect of automation for us organizationally. And when I say that snowball effect, it functionally was one engineer saw what they were doing like, hey, I can automate something similar to that. And then all of a sudden they would automate something. And then another engineer would say, "Well, they just automated that. Why can't I do this?"

11:26 — Jamie Parker
This time, they were off to the races. But what did that look like practically?

11:35 — Jason Kayser
A distribution list change we could estimate would be a half hour worth of IT time. And we could estimate that a wait time on the customer side would be 45 minutes or a day. You pick a number. And you take those things and you assign a dollar value to those. We ran 47 distribution list updates this week at half hour a pop. And you start doing math on that, 47 divided by whatever. You put a cost around that and start saying, "Here's what it looks like."

12:00 — Jamie Parker
They could then use those estimates to graph trends. Those trends showed a steadily increasing use of those automations, and they indicated really positive results.

12:11 — Jason Kayser
It doesn't scale up because we're always adding new automations. We are, but you're also seeing that scale as the organization grows. Again, going back to we're a organization, we want to double every five years. As more people on board, more requests come in. That request that used to service 40 a month is now serving 80 a month. And all the time we don't have engineers doing that work. That's just work that's gone away from us. That's doing more with less.

12:36 — Jamie Parker
Organizationally, they were seeing the benefits to scaling they were looking for. But the platform they adopted and built, the one that teams across the organization were using more and more, it also let them work with each other in ways they hadn't done previously. Classic divisions between infrastructure and development teams weren't gone entirely, but started fading away.

12:59 — Corey Wanless
Developer-minded people came in and said, "Hey, we really need to have a better flow for how we develop code," which then led us to an overall solution, which then helped us in the long-term create a flow for creating automation that allowed for experimentation. And created feedback loops so that we understood what was working and what wasn't working.

13:19 — Jamie Parker
Experimentation across teams is key for large-scale projects. We all talk about cross team collaboration being key to success. Not speaking the same language is a big hurdle, but it's far from insurmountable.

13:32 — Corey Wanless
Ansible as a framework for automation enables a common terminology when speaking about automation. So it didn't really matter that I was automating the storage and the network guy was automating the network. I barely know what BGP is. He barely knows what a LUN is. But in the middle, there's a terminology around the framework we're using, Ansible, to know what a playbook is, what a play is, what a task is, what a module is, what are arguments that go into that module. That created the framework to have collaboration and discussions around how we could collaborate together to create automation.

14:11 — Jamie Parker
What they've achieved over several years of trying automation solutions is a way to offload a lot of their repetitive tasks, establish consistency, and break down barriers between teams. That's letting them scale in accordance with their goals. And they've started helping their customers do the same. But that doesn't mean they've moved on beyond their automation projects. Now it's become part of the company culture.

14:34 — Corey Wanless
There's never a destination. It's just automation is a journey. And that same is true not just from a World Wide perspective--World Wide Technology perspective--but also within our customer base and even internally. Technology is not stagnant. Technology is ever-changing.

14:51 — Jamie Parker
Those automation projects weren't one-off efficiency boosters. It took over six years for World Wide Technology to get to the point where they felt they'd gotten a handle on and integrated automation into their processes. And that's helping them meet the challenges of an ever-changing tech landscape. But none of that would've changed much without the team's participation. After all, they're the ones automating away their own work. They're in the best position to make it work. We just heard WWT's story of getting their engineers on board with automation. And given their culture of curiosity, incorporating automation into their day-to-day operations wasn't too much of a stretch, but that's not always the case. Next time on Code Comments, we'll hear from Deloitte on how they help companies shift their mindset and make automation a habit.

15:49 — Jamie Parker
You can read more at redhat.com/codecommentspodcast or visit redhat.com to find out more about our automation solutions. Many thanks to Jason Kayser and Corey Wanless for being our guests. And thank you all for joining us today. This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead, and Brent Simoneaux. Our audio engineer is Robyn Edgar. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oats, Rachel Ertel, Carrie Da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, and Victoria Lawton. I'm Jamie Parker, and this has been Code Comments, an original podcast from Red Hat.