Modern DevOps Engineering: Multicloud, Security, and Solving Complex Infrastructure Problems

Listen to the full episode above or read the article below:

This article is available in English based on the John Meyer Podcast interview.

In this episode of the John Meyer Podcast, host John sits down with Jacek Marmuszewski, co-founder of Let's Go DevOps, to discuss his 15-year journey in DevOps, the evolution of the role, and some of the most complex infrastructure challenges his team has tackled—including accidentally shutting down a country's internet.

15 years in DevOps: How long is that really?

That's a tricky question because I feel old recently. Probably like 15-18 years. I would need to count it actually.

Did you naturally progress into that role or were you just "hey, I'm a DevOps engineer today"?

To be honest, I started as a Java developer, but it was like 15 years ago. We had to write a lot of boilerplate code to actually do anything with Java.

I quickly decided that it's not my thing. For a while I was a C++ guy, but let's be honest, it was not also my thing.

I always love tinkering with machines. So I moved to a more system admin position. At some point I noticed the world is actually looking into automating the stuff and actually building the DevOps culture.

This is when I hopped the train and became a DevOps from being a developer and a little bit of a system admin. I think this is how it clicked.

What has changed in DevOps over 15 years

I remember C++ and typing everything in Notepad, but we won't go there. You've been in DevOps for the last 15 years. Talk to me through it. What has changed from when you started to now?

I would like to say that not much, but yeah, 10 years or 15 years is quite a lot of time.

I remember the very beginnings of DevOps. All the companies, when they had any issues—like even if the coffee machine was broken—they were hiring DevOps to fix it. Everyone wanted to have a DevOps engineer.

It was so unclear what is the responsibility of the DevOps guy that we actually had everything under our wing. DBA, system admin, you name it. Everything was within the DevOps position.

The shift to specialization

What changed recently is first of all we are looking into more specialized people. We are talking more of platform engineers, infrastructure engineers, not necessarily DevOps engineers. It's not so commonly used term anymore.

But what I'm seeing that brings the biggest value is actually creating the platforms for the companies.

We are not only thinking about the automation, but we are also thinking about this developers highway to release software as fast as possible. It's not only about automating the infrastructure stuff.

What's missing from today's DevOps

When I was a server admin from your definition, I was technically a DevOps engineer where I not only did the servers, the databases, the infrastructure. We just didn't utilize that term and how our job roles actually functioned. What is missing from today's DevOps?

I think that it depends from company to company because we are still missing a unified definition of what DevOps or platform engineers should be doing.

This leads to weird situations where even when we are recruiting guys that have experience as DevOps engineers in their CV, they can be from an array of spectrums—from DBAs to system admins. Sometimes the guys don't even have a chance to code in any language.

So it's more like a click-ops DevOps part. It's really weird still in the field.

I think that we are missing unification. This makes it a lot harder when I approach a potential client to actually tell them that we are doing DevOps stuff, because we usually first need to make sure that the definition is okay and the conversation becomes a lot longer than it should be.

You're spot on with talking about each role or definition of DevOps is unique for every company. When they look at it and say "I need a DevOps engineer" and they go out and they try to hire somebody, it's very unique for them. It's a unicorn. Your definition as a DevOps engineer is "I do X, Y and Z."

What is Let's Go DevOps?

Let's jump over and talk about Let's Go DevOps. What is it?

Basically it's a group of DevOps engineers. The DevOps engineers that have a really unique skill set.

Because we are a team of over 20 DevOps engineers or infrastructure engineers, we have guys that are hardcore network engineers. We have guys that came from coding. There are more devs than system admins and plenty other skill sets that we have in our portfolio.

Although we are doing a team-based approach for every client and we are building the team that matches his organization, the fact that we are a team of people that really are excited about technology and have their own unique skills—we actually provide a variety of services.

If you need a DBA, we have DBAs in our squad. We can always pull in the guy just for consulting or doing a full-time job. We are a little bit more flexible.

We're a little bit like a cloud for DevOps services. You can just pick the services that you need and we will deliver high quality engineers.

Solving the unique team problem

The problem it sounds like you're solving is each group is unique for a company that they need to handle. Company A needs DevOps engineers and they talk to you and they're like "I need this service, this service or this skill set" and you're like "we've got you, here's the team for you."

Usually we focus on the long-term relationship. We grow with the company.

The best companies for us are startups that are just building. We are usually picking the commando unit that can do everything—the Swiss knives of DevOps world.

We put them into the startups. When the startups grow, we actually notice what kind of skill set they have. Usually we are recruiting explicitly to build the team within single client or single partner.

But we still have the skill set among the other members of the team. If at any moment you need help with database or something like this, you can always pull one of the guys from other projects. Tell him "okay, I have this issue, can you guide me or can you jump into the project and fix it?"

We are flexible when it comes to moving knowledge around and everyone is eager to learn new stuff.

What makes Let's Go DevOps different

Haven't other companies tried to do this and what has been missing from this type of methodology or work?

I think a lot of companies try to do it, especially when you look at software houses that usually try to have all the technologies available.

We have a little bit different approach. It's not that we will solve all the issues that you have.

We are specialized in cloud—mainly AWS and GCP. We have a lot of Linux knowledge. So there is a certain pillar how we build the company and the skill set.

We will not help you with every issue that you have. We are pretty specialized, but when it comes to the specialization, we actually have even deeper specialists.

It's a little bit more narrow than other companies, making sure that we always can deliver high quality stuff and have the guys that are actually knowledgeable about the certain aspects.

The biggest DevOps challenges today

Can you help me understand some of the biggest technical or strategic DevOps challenges you might see your clients facing today?

Well, when it comes to technical challenges, I think the biggest one is actually making the camera to work.

[Camera issues ensue]

Wait, wait, wait, Yazik, do you have a DevOps engineer for that?

Yes, we do. And we can solve those problems as well, as you can see.

The advanced AWS GPU infrastructure project

When it comes to technical challenges, we like solving big issues, because a lot of the guys that I have became a little bit bored with the companies that they worked in. They need some distractions and more complex challenges.

When it comes to the biggest stuff that we did over the last year or two:

We had a client that requested us to create something with graphics cards available on the internet or on the net with unified memory.

We created an advanced GPU computing infrastructure containing:

• 8 terabytes of memory

• 2 terabytes of graphic memory that is unified

• You can access it from any node that's working in the cluster

• 32 fiber optics cards that are trunked

• Theoretical 3.2 terabit per second capacity when it comes to the network

• We are saturating it through most of the lifecycle of the project

It was challenging because it was first in the world implementation. We worked deeply with AWS engineers who helped us troubleshoot the network virtualization that they have internally in AWS.

They mentioned that it's a first of its kind setup that we were doing. Those are the challenges that we like to tackle and like to solve.

How Krakow shaped Let's Go DevOps

You're based in Krakow, Poland, which some consider a thriving tech scene. How has it shaped Let's Go DevOps growth?

Krakow is a really interesting place because a lot of businesses actually came into Krakow a while ago.

The idea behind this is that we have plenty of universities in Krakow, plenty of technical universities. You have pretty much quite a pool of talent that comes to the market every year.

A lot of companies like Sabre, Oracle, Motorola, Google and so on came to Krakow a while ago and they started building a tech ecosystem here in Poland, in Krakow area.

This meant that we had a lot of really cool engineers working on cool stuff that were willing to share and build the entire ecosystem here in Krakow.

Having this in mind, when I finished college, I didn't have any problem with finding internship. I met a lot of people. Some of those relationships and trust that I built over the years actually resulted in landing some of our clients.

The multicloud reality

We know that multicloud exists and it takes a unique set of skills per cloud provider. How are you handling multicloud for customers though?

First of all, we had the episode of being truly multicloud—like being hot-hot when it comes to AWS and GCP.

The question is a little bit tricky because I usually discourage our clients from going truly multicloud and cloud agnostic.

Why multicloud is painful

Nobody talks about it, but there's so much pain in actually running your stack on two different cloud providers.

You're losing so many benefits of going into the cloud that in my opinion you need to have a really, really solid case and a really, really deep budget to actually decide to go hot when it comes to multiple clouds.

The truth is that from our perspective, we are mainly passionate about AWS. If you look at how many announcements at re:Invent there are for new services, being ahead of what's going on with AWS is basically a full-time job just reading the blogs and news about the new stuff.

Our cloud focus

We are focusing on AWS and decided that the secondary cloud that we are supporting is GCP. We are not expanding too much.

In many cases we can help out with our Linux and foundation knowledge that we have. The clouds are, to be honest, not that different. They have some caveats, but a lot of the concepts are actually similar. You just need to learn a little bit more about what the corner cases are in each of the configurations.

From our perspective, we are two major clouds: AWS and GCP. Plus, we sometimes help with smaller vendors that are usually more cost-optimized for smaller companies.

In there, we just have people that are specializing with this particular technology as it's not as broad as AWS or GCP.

How multicloud actually happens

I don't think too many people decide that they're going multicloud. It just happens. Companies get bought, sold. Somebody's playing around with a new service.

Going multicloud is very difficult. The old thought process behind it was that you don't want to be vendor locked in, you want to be able to move where you want. That was because it came from the traditional on-premise thing where they were locked into licenses and vendors.

When you go to cloud, each has their own specific niche of services and value to them and is really geared toward what is beneficial for your company.

Security: Shifting left and gamification

Security is top of mind for everybody and it should be job zero. How are you handling making security a top priority not only for you but for Let's Go DevOps and your customers?

First of all, what we do is we shift security left as much as possible.

Starting from our code, it's constantly scanned with stuff like tfsec and other code scanning tools. We make sure that the approach that we have for new services and cloud is actually least privilege approach.

Building security into modules

What we do for every company is when building modules, those blocks are pretty much constrained about the security and compliance restrictions that each of the companies have.

We are making sure that if you are a developer, you don't need to think too much about the infrastructure security. If you use predefined modules used in the company, they are actually secured.

There's a caveat because usually we cannot secure every module in exactly the same way. It differs from one customer to another. Sometimes you have geological restrictions, sometimes you have some other weird stuff.

I usually call it "the floor is lava"—you don't know why, but there is some kind of restriction from compliance and so on. We have a lot of those caveats and we don't want developers to think about them.

Everything that we do, the platform that we are building for them, actually keeps some kind of restrictions but without interfering their work too much.

What "shifting left" means

When you say shifting left, are you talking about inherited security from the cloud providers?

That's one of the cases. We also create the software development lifecycle in the way that from very beginning you have all the scanning tools, unit tests and so on run on the code bases that we are deploying.

This is something that will come to you automatically when you use our modules. Not only the infrastructure stuff but also the development stuff or the code is well secured and scanned regularly.

When we go to the cloud and we are launching the infrastructure, we try to enable as many security features as possible. It usually depends on the budget—they are not cheap, let's be honest—but we try to add as many features as possible so we get notified about any breach from our side, any misconfiguration and so on.

The gamified pentest challenge

We are partnering up with some of the security companies that are doing pentests. I love the challenge.

Every year we have this one week or two weeks actually where one of our partners comes in and he tries to beat the hell out of the configurations that we have. It's always like "who's on top?"

We really like the game. Having this in mind from the very beginning that there will be someone testing you at every single step on the way, it brings this approach that we need to make it secure from the very beginning.

The more we hardcode into the platform and enforce it as a part of the platform, the easier it is for everyone to actually move forward with the security stuff.

So you've integrated security into your processes and also the pentest on top of that, doing basically a hectic week of trying to break security, try to figure out some of the loopholes but keep everybody on their toes.

Exactly. We love the game so there is even a small prize for the winner of the competition.

Gamifying competitions is always helpful, keeps everybody engaged. It's always fun because we have a natural competitive nature amongst ourselves.

Vision for the future

What's your vision for the future?

To be honest, I think we never thought about a whole lot of the future in front of us.

We started the company because we saw stuff not working, especially when it comes to consulting and the DevOps consultancy.

Being as DevOps engineers hands-on in the companies, when we were offered with the help from consulting agencies, we were never too happy about the stuff that they delivered.

So we decided: "Okay, it's a moment to actually start something of our own, have our own team." The initial idea was to have like six people on board and have this great team of great people working together.

Currently we have over 20 guys. It's still first of all focused on having great team and great minds in our team.

The future is progress, but making sure that we don't lose this initial speed and initial integrity that we have within the team. Everything else is more like the opportunities that we like to explore.

If you have a really interesting challenge, we will probably be super hyped up about fixing it or solving it. If you come to us with a cool idea, we are like usually: Let's go. Let's hop on board and let's try to do something cool and something that will change the way technology is used somewhere.

Staying ahead of the technology curve

Technology is always evolving. Every six months is like 10 years in technology. DevOps has changed from the last 10 to 15 years. How do you plan to stay ahead of the curve?

I think it's all about the culture that we have and the cultural fit for everyone that's coming to Let's Go DevOps.

The guys are actually hyped by learning new stuff. I personally love learning new stuff and going through different tutorials, playing out with the stuff and so on.

It's okay for the team to actually gather new knowledge and it's not something that we need to enforce. Everyone has this internal driver.

Every week we have this session of showing off what we actually accomplished and something cool among the colleagues. If you would see the presentations, the guys are just blowing my mind with the stuff that they are doing on their home labs or that they are solving for our clients.

It's like we are always learning. We are trying to be on the very front of the curve—see where the stuff is going.

Greatest achievements: Stories from the field

Do you have any examples you can share with us that can showcase some of your achievements?

Well, I think there will be plenty of those.

The advanced AWS GPU infrastructure

I think the AWS case with the GPU infrastructure and working directly with AWS engineers—this is our biggest achievement of last couple of months because we're truly delivering something for the first time and actually influencing the way that AWS is making their services, making their documentation and running the stuff.

So this was really, really cool to work on it.

The hot potato project: Half a petabyte migration

The second thing I can think about is the stuff that we did for other company. I will call it a "hot potato" because this was the project where nobody in the company actually wanted to pick it up.

We thought it may be a really cool project. I actually went to C-level and told them: "I want to have your hot potato with us."

The project was that they had over half a petabyte of auditing data scattered across entire Europe.

First of all, the project was not documented at all. We needed to figure out:

• Why it was built

• How it was built

• What versions are up and running

• Where are the servers

Some of them were actually turned off. We needed to call support to come into server room and plug it in, turn it on.

The accidental internet shutdown

Some of them—this is really, really fun bit of it—we actually shut down internet for one of the countries in Europe.

It seems that our data was sitting in the government facility. Their way of limiting the internet speed was not smart. When we started pushing Golang code, we actually saturated the connectivity and most of the country actually went offline.

We got a call from one of the ministries to shut down the operation.

I can imagine that call where they called you and said "Hello, you just killed our internet. We shut you down. Stop doing it."

At first I thought it was phishing. But then it actually escalated.

"I don't believe this call. Sorry. Hanging up." They called you back. "Hello."

Imagine getting a call and somebody tells you "Hey, I'm the president of United States. You need to stop your process on the server."

Yeah, please give me your GoFundMe and social security.

The results: From hours to 11 seconds

That's one of the cases that we actually had. With the project itself, like half of petabyte of data, it was really hard for the guys to get any knowledge out of this data.

We moved it into GCP. We actually focused on the time and the complexity of the operations that are done on the dataset.

Now the guys can scan the half of petabyte of data within 11 seconds. You can ask any question for the data.

Looking how the organization changed over the months, when they noticed that now we can actually make educated decisions based on data that is available for everyone—that was astonishing.

Technically, the project of moving half of petabyte of data across Europe in different directions—it was really, really interesting stuff technically to do.

Closing thoughts

Today's topic is modern DevOps engineering, multicloud strategies, security, and solving complex infrastructure problems. Joining us was Yazik, who's the founder of Let's Go DevOps.

Yazik, thank you so much for joining me.

Thank you very much. It was a pleasure talking with you.

I hope everyone enjoyed it. See you in the comments.

This was the John Meyer podcast. Don't forget to hit that like, subscribe, and notify because guess what? We're out of here.

FAQ

How long has Jacek been in DevOps?
Approximately 15-18 years, starting from a Java developer background, moving through C++ and system administration before embracing DevOps culture when it emerged.

What changed in DevOps over 15 years?
Initially, DevOps engineers handled everything "including fixing coffee machines"—from DBA to system admin work. Today the focus shifted to specialized platform engineers building developer highways for faster software releases, not just infrastructure automation.

What is Let's Go DevOps?
A team of 20+ specialized DevOps/infrastructure engineers based in Krakow, Poland, offering flexible cloud engineering services primarily focused on AWS and GCP. They function like a "cloud for DevOps services" where clients can access specific skill sets as needed.

Why does Let's Go DevOps discourage multicloud?
Running on two different cloud providers causes significant pain and loses many cloud benefits. It requires a really solid business case and deep budget. Each cloud has unique services, and being truly cloud-agnostic means you can only use the features common to both platforms.

What was the advanced AWS GPU infrastructure project?
A first-in-the-world implementation featuring 8TB of memory, 2TB of GPU memory accessible from any node, 32 fiber optic cards delivering 3.2 terabit/sec theoretical capacity. Built in collaboration with AWS engineers who called it a first-of-its-kind setup.

What happened with the half petabyte migration?
The team migrated 500TB of audit data scattered across Europe (some servers were offline, had to be physically turned on). They accidentally saturated a government facility's internet connection, shutting down most of a country's internet and receiving calls from ministries to stop. Final result: Moved to GCP, reduced query time from hours to 11 seconds.

How does Let's Go DevOps approach security?
Shift security left with code scanning tools (tfsec), least privilege approach, building security into infrastructure modules so developers don't need to think about it. They also run annual gamified pentest competitions with prizes where security partners try to break their configurations.

Why is Krakow important for Let's Go DevOps?
Krakow has many technical universities producing talent, plus major tech companies (Google, Oracle, Motorola, Sabre) built an ecosystem there. The relationships and trust built in this community directly resulted in landing clients for Let's Go DevOps.

Podcast: The John Meyer Podcast
Host: John Meyer
Guest: Jacek Marmuszewski (Yazik), Founder of Let's Go DevOps
Company: Let's Go DevOps
Topics: DevOps evolution, platform engineering, AWS GPU infrastructure, multicloud challenges, security practices, cloud infrastructure, Krakow tech scene