Ole Lensmar (00:01.422)
Hello, everyone. Welcome to today's episode of the Cloud Native Testing Podcast. I am super happy to be joined today by Oleg Nenashev, who is a lot of things, champion for test containers, one of many. I'll hand it over to you, Oleg, to do the extended intro. Welcome.

Oleg Nenashev (00:18.019)
Yeah, thanks for inviting me. And definitely a lot of things in my portfolio. Sometimes when I do presentations, I say that I'm not a mother of dragons, but a father of many bugs, especially when it's a Jenkins presentation, because I used to wear so many Jenkins hats when I was actively working on the project.

Now things have been changed, so I still remain in the developer productivity area, in developer tooling area. I did some tours with Cloud Native, I'm a CNCF ambassador, and now I'm actually back to Java. So I work for Gradle, I contribute to various projects like Wiremog, Microx, test containers, and of course, Gradle itself. So yeah, there is quite a lot of stuff I'm working on. But for me, the key area is of course developer productivity.

Oleg Nenashev (01:10.129)
activity and the second area is community around developer tools which is also an exciting topic.

Ole Lensmar (01:13.646)
Okay, great. Awesome. Thank you. So let's dive straight in. So I think super interesting that you're involved like in, in microcs test containers and wiremock. think mocking in the context of testing in general, but also maybe specifically in cloud native testing is definitely, you know, something that people from my point of view should have in their, you know, bag of tricks. What are your thoughts around that? And how do you see people using mocking in testing?

Oleg Nenashev (01:44.685)
Definitely. One thing about my career is that I actually started in hardware and electronics engineering and integration testing of electronics because electronics normally, especially when I started, it had a lot of different chips that interconnect with each other, that talk to each other via special protocols that might have various interoperability issues. So integration testing is a key. And basically, since I started my career, it was around 2000.

eight each. Every company, every project I work on integration testing becomes one of my important things. And when we talk about cloud native environment, microservices, of course, it's also one of the critical points because you can write a lot of unit tests, but actually how your services talk to each other, how they react, how actually how to make the system robust against potential failures, breach of protocol by

external service provider is quite an important thing in Cloud Native World. So for me, when I started doing my first microservices, when I started working on Kubernetes clusters, having a proper tool chain for integration testing has became a priority. And it was slightly before test containers became a thing. So test containers, I believe they, for a long time, they appeared in 2018 or 2019.

So before that, yeah, it was quite difficult to develop things. You had to manage the environments on your own. Yes, there is a lot of special cloud native projects. yeah, for me, yeah, this area where I spent quite a lot of time and I'm happy to talk about it. I also know that Testkube invests a lot in various kinds of testing and environment management for cloud native. So yeah, that's the area.

Ole Lensmar (03:34.883)
Yeah, no, definitely. I mean, I'm just kind of going back to the days when I was many, years ago involved with specifically talking about mocking and hardware with SoapUI, which was a tool I wrote like 20 years ago. And I remember that actually we had a big Swedish telecom vendor. I'm not going to say which one.

that actually used the soap UI to mock out their, one of their hardware switches. So they, they took the whistle of that, of the API soap API at the time of that switch. And then they created a mock of the switch. And then instead of when people wanted to try their hardware, instead of sending them the hardware, they would actually send them this, this mock of that, piece of hardware. that obviously made it so much easier for them to try. And I think that's really where it's intriguing that mocking and

tools like Microx and Wiremock can be super efficient in like speeding up and making it easier for you to try both external dependencies, but also simulating internal dependencies and maybe what you were talking about, like simulating faulty behavior, slow behavior or unwanted behavior. But it's also seemed to me often that this is something that's easy to talk about, but it's hard to do in reality.

What's your experience? Do people actually do these things? And if they don't, what's holding them back?

Oleg Nenashev (05:02.755)
Yeah, so when cloud native started and when microservices started, actually it was in the time of the continuous delivery hype. And there was an assumption, in my opinion, assumption that if you can deliver fast, you probably do not need integration testing, performance testing or whatever, because you can just fix it and ship a new version.

To some extent, it makes sense if you have proper observability, proper platform operations, disaster recovery, etc. If you invested in all of that, maybe you can live this way. But actually it's much cheaper and more efficient to actually have integration testing and avoid problems before they get shipped even if it's continuous delivery.

Oleg Nenashev (05:44.815)
So in my case, I always thought that it's important to do it. When I was doing presentations on evolution of CI, CD, you can find them on the internet, I was saying that stay tuned, integration testing will be back in cloud native. And now, yeah, I think it's back because so many projects actually invest in that. And that is a lot of things. That's it. Thanks a lot for SOPUI because I used it a lot.

Ole Lensmar (06:00.942)
Mm-hmm.

Ole Lensmar (06:11.406)
Okay. didn't, I great. I didn't mean to steer in the discussion about this, but it was interesting to just the fact, just mocking like where I'm guessing like tools like Microx and a wire mock can be really helpful is when you're running your integration tests, you might have some dependency that it could be an external dependency, like, you know, Stripe or something, or like an, an, an dependency internally where you need to simulate certain behavior.

Oleg Nenashev (06:13.775)
It's hand-to-toe.


Ole Lensmar (06:39.63)
while during the execution of your integration tests. So that's where those tools really can come in and help. Is that correct?

Oleg Nenashev (06:48.397)
Yes, exactly. And one thing that always confuses people is term mocking, because they think that mocking is basically replacing a part of your code by something. And to be frank, firstly, I hate this approach because it means that you are not really testing your application. You test something instrumented. And secondly, just term mocking is misleading. For me, I prefer more academic term modeling.


Oleg Nenashev (07:15.981)
So the idea is that you create a model of your system. And all of us know that models are not accurate representation of the system, but depending on your use case, depending on your objectives, you can create a different model. So for example, you can have behavioral model, you can just have whatever interface model for testing, you can have a simple stop if you need something. And these are completely different types of testing.

Oleg Nenashev (07:44.663)
Also, for example, for performance evaluation, et cetera, you'll create different models of different complexity and different use cases. But to build a model, you need a tool. Of course, you can just write everything on your own as code to create a model, but probably it won't be so much different from actually creating an application. But there are different other approaches to create models. One of, and in my case, I distinguished two different...

Oleg Nenashev (08:13.295)
types of models. One model is basically a simplified version of your system. So let's say you have a complex database, have just a microservice based system, banking application service provider, you can take a simplified version of that, put it in a container or result container if you're brave enough, and just use it for your integration testing. So basically instead of using a whole external system,

Oleg Nenashev (08:42.169)
You have something local, simplified, but still that emulates your system. This is one of the approaches. And this is approach, I could name it test continuous approach because test containers is exactly tool that allows doing it. So putting smaller versions of your system, but still it's kind of accurate model in terms of functional behavior, in terms of sometimes architecture specifics. And there is another approach.


Oleg Nenashev (09:12.333)
when you actually don't have an implementation and you take, for example, a specification. So let's say open API specification or I think API specification. If you use protocols like Apache Kafka, et cetera, you have a specification that describes your service, or you might just have a bunch of human diagrams. If we take the previous era and then basically you put these

Oleg Nenashev (09:39.695)
specifications into a tool and magically it has a kind of working version of your system. You send a request, you get a response. So basically this is approach used by tools like Wiremog, Microx and zillions of other tools which basically convert a model definition in code or in various kind of specification to a working model that of course is less accurate than real system.

Oleg Nenashev (10:06.201)
But at the same time, it's fast, it represents common behaviors, for example, if you need stateful behavior, et cetera, and you can still test against it. So for me, these are two main kinds of modeling and two groups of basically modeling or mocking tools that are currently present on the market.

Ole Lensmar (10:25.902)
Yeah, no, so there's a lot of thoughts in my head. So one is on the ladder. I think I was very entrenched in the API space for a while. And this kind of, you know, contract driven development of APIs were to your point, you would have your open API definition and you could from that generate some kind of server stubs or mocks or whatever the term you might want to use. And then that could be used both to


Ole Lensmar (10:51.416)
build your clients, but also to create tests or at least like basic tests, scaffolds. So that's definitely an interesting use case for mocks, but then also to your point to actually model your system to be able to run integration tests for either specific components, something that, and I think something that I think in the industry or in the commercial industry was called like service virtual virtualization, which I personally find found to be a somewhat confusing term because

But I know that many commercial vendors use that as more dynamic mocking or maybe kind of, or modeling a system behavior. I think another one that I've seen is basically just recording traffic and trying to use that to model the interactions with an external system. So if you don't, say you're interacting with a third party API, there is no open API definition that you can use. So then,

just recording that interactions you have and playing that back. To me, that sounds very fragile, but it seems to be something that I know many commercial vendors adopted. What are your thoughts on that?

Oleg Nenashev (12:00.907)
Yes, so to start with recording and reproducing things, it's often the only thing you can do.

because you can create open API specification for your own server service. Whether the application actually follows the specification, it's a separate story. And actually even open API as a specification has a lot of deficiencies. used to be a member of the board for open API when I was working for WiMock. I deliver some presentations. There is also a new open API for standard in the works. So there is quite a lot of things. But...

Oleg Nenashev (12:37.175)
Long story short, you do not always have open API or whatever specification for the service you consume. And in this case, recording and reproducing the behavior or reverse engineering the protocol is the only thing you can do. It's a poor man's solution, but yeah, it works. And yeah, we use it quite a lot.

Oleg Nenashev (13:03.023)
and going back to actually integration testing. So I also agree with you that service virtualization is probably not the exact term.

But what is important to mention that this approach, when you actually use the existing system, it becomes much more popular these days. And I think there are a lot of reasons for that. First of all, co-development of service site and client site, continuous delivery. So you basically just having many scaffolding frameworks. So now it is possible to co-develop applications. And now it's also possible to actually put

bigger applications in a testable mode. Let's say 20 years ago when you needed a database like for example Oracle DB or whatever, good luck setting up it without a staging environment. These days, well even for Oracle there is actually a test continuous model for Oracle database. I won't touch it but spoiler alert, you can do it now. But what you can actually do now...

Oleg Nenashev (14:09.663)
If you do proper deployment of your application, you, let's say, in Cloud Native Vault, you have a Helm Chart, you can take the same Helm Chart and put it, for example, in a test container. So despite the name, so test container started as a simple container orchestrator, mostly for Docker. Now it also supports ContainerD, Podman, to some extent. But actually, it has models, let's say, for K3S.

Oleg Nenashev (14:38.331)
or for KinD. So kind is Kubernetes in Docker. And what it means is that you can actually deploy the whole cluster inside a test container, inside a single container, inside your test that would be managed by your solution. So I mentioned test container so many times, but just in case if you don't know it, the principle is very simple. It's a simple unit testing framework extension that

Oleg Nenashev (15:07.119)
basically encapsulates all the container management, et cetera. You just write a few definitions in your test. I need, let's say, a Redis container. I need a K3S container with the Helm chart to be deployed. And then the framework does all the magic and you just consume it in your test, like assuming that the system is provisioned and it's available on many technology stacks. So this way,

Oleg Nenashev (15:34.583)
I keep referring to test containers because for me, in terms of integration testing, it's actually an industry-changing moment because it completely changed how I develop integration tests.

Ole Lensmar (15:47.317)
It's very cool. I agree with that and actually have some examples on test kube on how to use test containers with test kube as well to just to spin up dependencies. But also like I really like what you mentioned about since you can use KinD to basically spin up your own Kubernetes cluster in Docker. You could even do that under test containers and then provision whatever you want inside that cluster, right? And then run tests against it. So it becomes an extremely powerful framework to

to instantiate whatever you might need to run your tests. I like that. Another thing I was intrigued by also in the context of mocking in Kubernetes is the possibility to kind of inject services into a cluster and using a service mesh, for example, to basically replace routes, right? So you could say, have service A, pod A talking to pod B, and now I want to replace pod B with a mock of pod B.

If you have a service mesh in place, you could basically do that transparently without having to reconfigure pod a, right? You just reroute the network. and so that big also that, cause what I remember, like if you go back 10, 15 years with mocking, that was always a challenge. you want to use a mock instead. You have to point your test at, you know, there are the comp, you have to reroute the, the, the end points. So it's talking to the mock and not to the actual component, but in Kubernetes and

with service meshes, seems like that could be done fully transparently. So you could have a framework that basically injects external dependencies or, I mean, this goes into this chaos testing, even goes into maybe replacing certain services and then making them behave in a certain way, basically, and in that way to see how the rest of your system reacts. Does that resonate with you or is it like, am I on a wild trip here?


Oleg Nenashev (17:42.479)
It resonates with me quite a lot, especially when you actually test our infrastructure.

Oleg Nenashev (17:49.263)
Test containers are cool, but they're basically limited to a cluster of containers. They don't really represent the system on the application level, they don't manage environments. And yes, if you take a proper environment management solution, for example, you can build on the top of Service Mesh, you can try to build on the top of Istio, or maybe just cross-playing other tools that actually manage the environment for you.

Oleg Nenashev (18:18.645)
And basically what you get, you get a dev environment or staging environment where particular bits can be replaced upon the request by mocks. It's a good approach. And this is actually an approach that you can use when developing something against the Kubernetes cluster. You want to take a version of the system, you take it out, replace it and do the testing. And I know that a lot of people do that.

Ole Lensmar (18:44.078)
How do they orchestrate that though? Do they do to kind of say you have a pipeline where you want to test some parts of your application with specific dependencies? it how I'm just curious kind of what's the mechanism of getting all that into place and working?

Oleg Nenashev (19:01.492)
So, there are basically...

Two answers, one what is often being used and another one is how I see that. So how it's often being used now, yeah, of course there is a lot of glue code. So for example, you create a branch, send a pull request and it creates, let's say an environment for you specifically for your need for development, et cetera, and you use it. There's a lot of automation, et cetera. So there is a pipeline that monitors something through CRDs, et cetera, that somehow connects to your local development environment.


Oleg Nenashev (19:33.725)
that basically deploys the system. Of course, it's a small matter of programming and if you're well versed in cloud native solutions and management, you can implement a pipeline like that. And there is a lot of examples. But how I actually see it, I see it as a kind of existing framework that actually does it for you.

Oleg Nenashev (19:57.071)
So we touched about, we already touched test containers as a service that brings it into your integration tests. What I didn't mention that there are also frameworks that actually do develop our environments as a part of the solution itself. In the Java world, I'm a big fan of Quarkus. So Quarkus is one of modern frameworks. It's actually quite popular for cloud native Java solutions. And it has a concept of dev services. So dev service, what it is?

Oleg Nenashev (20:26.307)
When you write a Quarkus application, can say that basically, my application depends on a service and you can have multiple implementations of the service just embedded in your definition. And then when you write, for example, integration tests, when you need to do a developer environment, you can supply another implementation. And many Quarkus dev services, you actually use test containers under the hood. So what it means that if I depend, let's say on a database provider,


Oleg Nenashev (20:54.223)
I can write my application that uses abstract definition, but the implementation might be different. It might be whatever pipeline being triggered and provisioning the cluster. It might be a production implementation and it be just a small test container for my local laptop. And the key point is that I can develop as an engineer without even caring about that because depending on my profile, depending on the configuration, on the use case, my framework.

I use can provision a different system. And I don't know whether there is a similar solution for Golang at the moment. Probably there should be. But if you ask me, this is how I see the evolution. So basically, environment management becomes a part of your framework, a part of your IDE. So you as developers don't have to worry about various pipelines, internal services, provisioning, et cetera.


Oleg Nenashev (21:49.283)
But basically everything becomes a part of your IDE and a part of your local or remote developer experience. So you just focus on developing code and these frameworks, et cetera, basically do all this legwork for providing you with proper development.

Ole Lensmar (22:02.286)

And how does that then permutate into your build pipelines and deployment?

Oleg Nenashev (22:11.343)
I think it should be basically the same framework doing that. So just to provide you an example, I'm a heavy user of dev containers. So dev containers is not test containers, it's basically a solution that was created. And for example, if you use IntelliJ IDE, if you use Golant, or if you use Visual Studio Code like me, you can provision a dev container with the environment included. I often have to develop a Python or Ruby code, and I can assure you that I couldn't develop that result.

Oleg Nenashev (22:40.911)
a dev container, but the principle is quite simple. There is a container that defines the environment or a set of containers. And what is important, I actually do my builds, deployment, testing in the same dev containers. So I normally use either GitHub Actions or Jenkins these days, sometimes Argo, but all the systems can also do cloud native and.

Oleg Nenashev (23:08.351)
container-based builds and integration testing. I can use services like Buildparks, etc. and basically use the services for both my local development environment and for CI-CD. And I can use test containers in the same way. So basically for me there is less and less difference between local environment and what CI-CD actually uses. well, firstly it helps. Secondly, it improves the developer experience.

Oleg Nenashev (23:37.933)
because I know that actually what I run, how it's built, it will be also built on CI, which is really good because it saves so much time on troubleshooting. And there are actually another things that people do not really notice, but that happens. Observability comes to local builds too, because the same observability tools like CI, CD, build observability we do on CI these days, you can also do locally to speed up your local developer experience.

Oleg Nenashev (24:05.875)
You can also have build caching, distribution, everything locally. And for me, basically, there is less and less boundary between local environment, CI-CD environment. For better or worse, firstly, yeah.

Ole Lensmar (24:07.852)
Hmm. All these things, yeah.

Ole Lensmar (24:20.066)
That's interesting. Yeah. That's interesting. Like evolution of how building. Awesome. So one thing I was thinking of, so in the context of cloud native testing, so if you're looking at people who are using test kube, there for them. They moved to cloud native and Kubernetes and they see that testing, you know, becomes more complicated for them. But what I'm actually hearing or thinking here is that you might get more complicated, but actually.

Thanks to test containers and MicroX and others, actually there are some things that are actually easier to do in a cloud native Kubernetes environment and that allow you to do more.

I know better testing for lack of a better word, things that might be hard to do in a non-cloud native environment. Would you agree with that? Or do you think it's still kind of overly complicated of testing in cloud native?

Oleg Nenashev (25:15.439)
Well, cloud native is complicated. Sometimes for a reason, because if you want to build microservices, if you want to build cloud native application with scalability, especially scaling to zero. In fact, scaling to zero for me is much worse than infinite scalability. But what it means when you build such applications, you can, you actually design for a lot of complexity and for a of features. 99 % of applications in the world probably don't actually need.

Oleg Nenashev (25:44.175)
So for me, using Kubernetes, using cloud native architecture for everything has always been a kind of rather cargo cult than necessity. But at the same time, I do feel that for many scalable applications, of course, Kubernetes is what it shines there. And my ability to manage environments. Yeah, there are plenty of tools for use cases like service mesh you mentioned, or just managing environment.

Oleg Nenashev (26:12.711)
being able to inject observability tools is super important. So if you operate on this level, you can actually build an awesome system. The problem with all awesome systems is that actually you take a framework. Kubernetes, some call it operating system, some call it a framework, but actually you need to take a bunch of tools and create a lot of code and tooling to have them working together.

Oleg Nenashev (26:41.763)
What I like about the industry that now you get less and less of this glue because the industry matures. There are more tools, more solutions that are rather designed for end-to-end consumption. Of course, I have to mention Testkube and other things there because they were designed to actually solve complexity for some of the problems. And there are so many other similar solutions on the market for different use cases. So the infrastructure for Kubernetes becomes easier, better. It's still quite complicated.

Ole Lensmar (27:10.966)
Yeah, I was gonna say, so I definitely agree that the fog is somewhat lifting, but it's slowly. I mean, there's so much there, everyone's getting into this space. And I agree that there's, know, it's evolving very quickly. And we see kind of efforts to standardize around different things, which I think is a good thing. But I think it's gonna take time just because there's so many people in there. And there's a lot of

Ole Lensmar (27:40.079)
Yeah, there's just a lot going on. So, but I mean, it's great to be in that space. It's a lot of fun. Yes. So finally, let me ask you, so you have a box behind you where it says Gradle. And so, I have to ask a little bit about Gradle and kind of how do you see Gradle fitting into this discussion that we've had now? Is there anything special going on with Gradle specifically that is interesting in the context of testing or cloud native that you could share?

Oleg Nenashev (27:44.099)
Yeah, exciting times for sure.

Oleg Nenashev (28:03.567)
Mm-hmm.

Yeah, so here we go into naming problem because Gradle means two different things. So first of all, it's a Gradle build tool, a very popular build tool for Java and not only Java applications. If you want, can build Python or Node.js with Gradle or native projects like C++. But there is also Gradle as a company and Gradle is actually a developer productivity company. So now our main product isn't even Gradle build tool, it's the Velocity.

Oleg Nenashev (28:36.299)
It's an integrated developer platform that was designed actually for supporting developers, whether they run CI or whether they run locally with a bunch of tools like Speed Builds. So Gradle is famous for build caching. But what people don't know that we basically bring build caching to Maven at the same level on Java and to other applications. We also bring local build observability with a lot of...

Oleg Nenashev (29:03.235)
monitoring capabilities, we add test monitoring and for example, flaky test detection because flaky tests, they time sync for many projects. Yeah, basically we try to discover them. We try to give some suggestions and there is a lot of other use cases basically. Develocity as a platform tries to resolve. It's still not very popular in cloud native world because I don't think we support GoLang officially at the moment and spoil.

Ole Lensmar (29:24.91)
Okay.

Ole Lensmar (29:32.312)
But yeah, but there's a big, I mean, there's a big Java community going into cloud native, isn't there?

Oleg Nenashev (29:33.145)
us.

Oleg Nenashev (29:38.753)
Yeah, it is. So there are many projects. Spring. So Spring is one of the most popular frameworks. Spring goes cloud native. Also modern frameworks like Quarkus, Micronaut. I don't mean to call Spring non-modern. Actually, they do awesome job in terms of adapting. But you know that basically, I'm a Ginkgoes maintainer. I know that maintaining a long multi-year project, it requires a lot of effort.

Ole Lensmar (29:44.181)
Yeah.

Ole Lensmar (29:56.418)
Let's, yeah.

Oleg Nenashev (30:07.823)
especially if you want to bring your user base and compatibility. But modern frameworks, in Java, they are designed to be cloud native. So there is a lot of presentations by Quarkus team on cloud native applications, on building microservices, on building AI applications with Quarkus. Java is not dead. It's a massive world. It's been actively used.

Ole Lensmar (30:17.964)
Mm-hmm.

Ole Lensmar (30:32.222)
Yeah.

Oleg Nenashev (30:35.599)
And it also becomes quite popular in the Cloud Native.

Ole Lensmar (30:39.15)
Yeah, I think it's impressive actually that, mean, Java was my big language 20 years ago. And so I think it's impressive to see that it's still relevant and that it's kind of evolving with the space. I'm sure people would argue that Go is more modern, et cetera, but a lot of people are still using Java and I don't have any big issues with that.

Oleg Nenashev (30:56.783)
Yeah, around 10 years ago it was stagnating, but now the velocity of Java community with a lot of languages like, for example, Kotlin, was a lot of new tools and frameworks. It completely changed the landscape. So exciting times for Java developers too.


Ole Lensmar (31:16.194)
Yeah, awesome. So Oleg, thank you so much. This was super interesting and we managed to do an entire show without talking about AI, which is a thumbs up. And so thank you so much for joining. Wish you all the best and hopefully I'll see you at a conference sometime soon. Thank you for joining. Of course.


Oleg Nenashev (31:30.851)
Yeah, thanks a lot for inviting me and yeah, hopefully you will be at KubeCon or other events.

Ole Lensmar (31:36.652)
I will be at KubeCon, so we should meet up there. And if you're there, looking forward to it. Okay. Okay.

Oleg Nenashev (31:41.057)
I'm missing this edition, but I'm on the program committee and definitely there is a lot of meetups going on around the world. And spoiler alert, there will be something in Geneva in December. We are yet to announce it, but yeah.

Ole Lensmar (31:51.432)
Okay, you signed me up, that's close to skiing. So that's probably the right time of year.

Oleg Nenashev (31:56.559)
Yeah, or latch hadron collider tours so depending on what you prefer

Ole Lensmar (32:00.391)
Okay, it depends on a skiing large heart and collider and also one or the other. Okay. Thank you. Take care Thanks

Oleg Nenashev (32:07.331)
Okay, thank you very much and see you.