Ole Lensmar (00:01.08)
Hello and welcome to this episode of the Cloud Native Testing Podcast. I am super honored and thrilled and amazed and happy to be joined today by Ramiro Berrelleza who is the CEO and co-founder of Okteto one of the first players in the ephemeral environment provisioning space for communities and Cloud Native, which is obviously super interesting from a testing perspective. Ramiro, welcome. Tell us about yourself.

Ramiro Berrelleza (00:25.583)
Hi, Ole Thank you. Nice meeting you. Very happy to be here. Honored to be part of the Cloud Native Testing Podcast. It's very excited. As we've discussed in the past, ephemeral and cloud native testing are two super important elements that often go together. So we're excited to talk more about that. Can I share what we're seeing the space and learn from you as well?

Ole Lensmar (00:48.888)
Yeah, so that's great. let me just, mean, how did this idea with Okteto and ephemeral environments get started? Was it driven by testing or by other needs or problems?

Ramiro Berrelleza (01:00.187)
Actually, now that you mention it, there is a testing component to this. So like all good dev tools, Okteto started out of a person on Vendetta that I and my co-founders had against the way we're building software. I'm a software engineer by trade. I've been lucky enough to have been working on the cloud native space from a long time when I date myself, but from way before it was called cloud native, I was part of the team that was working on the first version of Azure at Microsoft.

Then I was in one of the early DevOps startups here in California, then Atlassian. And in all of these places, one of the common challenges that every teammate was at is it was really hard to test our software. We were going through this transition where before Cloud Native, software was mainly a monolith that you would compile on your desktop, run some tests, show it in CI, and then go to production.

But the difference between local machine and production was not as big as today. There were of course, dependency challenges, version compilation, I used to work in Windows, so DLLs and all those wonderful things. Then after microservices and containers, that was no longer a problem. But then this problem of, hey, my laptop looks nothing like production, started to happen a lot more.

And that's something that we suffer at Azure. You know, as we were building Azure, we had to build a new class of DevTools. Back then, we built this thing called Azure on a Box, which was a VM that replicated all the Azure infrastructure for us to be able to test our services in a way that was more realistic than, hey, this one component works, but not everything. the same situation we saw at Azure, you know, like 10 years later, I saw at Atlassian as we were adopting

all these cloud native patterns, microservices, async communication, automatic scale. So in all of these places, testing was very hard. Just validating, like, hey, my code works. Will it work in production? And at some point, I was working on teams where, honestly, there was no other way than on production. So we're just kind of like, throw our code to production, use feature flags, cross your fingers that nobody gets affected, and spend a lot of time looking at locks to make sure that nothing broke.

Ramiro Berrelleza (03:22.139)
And that was not great from a moral perspective, not great from a dev experience perspective. So at some point when I was done with Atlassian, Pablo, my co-founder, Ramon and I, we're kind of talking, hey, we should do something together. And the three of us kind of at some point kind of complain about the same thing. And it was like, hey, that's something there. And something of, if everybody has this problem from large companies to small startups, maybe there's something there. And that's how we started with this idea of.

Back then, actually, we were calling it Cloud Native Development, which is if you peek in our GitHub repo, you'll see that it was called CND at the early, early days, which engineers make great marketers. And this was the vision was, hey, as we move to this world of Cloud Native, it is crucial for developers to have access to dev environments that look like production to minimize this huge gap. And our journey started by helping developers put their code

Ole Lensmar (04:00.437)
Okay.

Ramiro Berrelleza (04:21.857)
as fast as possible on these ephemeral environments. And over the year, it has evolved to also build hooks for things like testing, build, secret management, and all these other things that makes an ephemeral environment truly useful. And that's where we're seeing a lot of our customers going now is what you said is now that they have ephemeral environments, hey, testing in those environments becomes very important. And they're replacing the traditional CI environments of like a single box.

And it's fascinating to see the industry kind of move forward. And I'm very happy that we've been able to play a small part on that.

Ole Lensmar (04:56.942)
Do you, I mean, my understanding is that you're leveraging Kubernetes and namespaces specifically to provision e phemeral environments. How big does Kubernetes play into this? Like the capabilities that Kubernetes provides to dynamically provision namespaces, et cetera. It does seem to me like that's a huge opportunity for someone like you to.

to utilize that possibility to create e phemeral environment specifically.

Ramiro Berrelleza (05:27.311)
Yeah, no, like, like Kubernetes is, was crucial for us. Actually, Pablo and I have been our CTO and I have been pretty involved on the container ecosystem from pretty much day one. At some point, Docker was at, Pablo was at Docker. I was at Atlassian pushing for kind of early container experiments. And we were at a Docker con when Kubernetes was announced to the public. And I remember both were like, right. That was the missing piece. Why?

Ole Lensmar (05:40.492)
Okay.

Ramiro Berrelleza (05:57.275)
Because I think why we couldn't do a phemeral environment before is what you're describing. Kubernetes is really good at spinning up infrastructure fast. 10 years ago, if you wanted to do FEMMA environments, that meant spinning up a bunch of VMs, and that would take five, 10 minutes per VM. Kubernetes, if you're sharing the cluster, if you have namespaces for your different FEMMA environments, which is the core of how the Okteto commercial solution works, it's almost instant. We have customers who are deploying

very complex dev environments, and I'm talking hundreds of microservices, per environment, and maybe it takes five, 10 minutes to deploy the whole thing, which 10 years ago was unthinkable. And a lot of this has to do with Kubernetes' abilities to kind of spin up infra fast, give you this, which I think is a wonderful abstraction, the namespace, which most people don't need hard multi-tenancy, the namespace boundary.

is good enough and we see that with a lot of customers and users of open source how having this, hey, I can give every developer a namespace where they can deploy this full ephemeral environment. It's a huge accelerator and it's something that to your point, I don't think we could have done it before Kubernetes. You could do some things at Heroku, but Heroku is doing kind of like a proto Kubernetes thing, managing processes and containers. So same principle.

And that's to me, at least personally, is why we did this company now and not like 10 years ago. And that's why I think this whole ecosystem has grown so much is because that building block enables so much.

Ole Lensmar (07:40.45)
OK, cool. And do you see, I mean, me looking at it from a testing perspective, seems like that's why we have ephemeral environments, to be able to test intermediate versions, releases, builds, whatever. But are there other use cases that you see for ephemeral environments versus just testing? I'm sure there are. I'm the guy with the hammer, so everything looks like a nail. But maybe there are other options there.

Ramiro Berrelleza (07:56.379)
Yeah.

Ramiro Berrelleza (08:04.571)
No, it's very interesting that you mentioned it this way because I have my own hammer, which is a ephemeral environment, and I'm trying to push them everywhere. And I never thought until recently, and I think you and I had a conversation at KubeCon and I've kind of started to make open my eyes to this, is that, yeah, it's for testing. Like for us, testing was an afterthought. We're thinking of environment, environment in our initial primitive thinking was for fast.

Ole Lensmar (08:12.323)
Mm-hmm.

Ramiro Berrelleza (08:34.245)
feedback loops. I wanna write code, see the results. As I am saying, this group, that is testing. Because you are, hey, is my code working as I'm expecting it to do? So yeah, I think testing is a core use case for a emphemeral environment, whether it's manual testing, hey, I'm running my code, then I go to an endpoint and I check, hey, is this endpoint responding as expected? All the way to.

Right now we're having a customer, sorry I can't say in the name, but it's very well done company in the hiring somebody to do a gig kind of thing. And they're now spinning up hundreds of the environments in parallel to do all their end-to-end CI testing. They're looking at chaos engineering. It's one of my favorite scenarios because I'm also very passionate chaos engineering practitioner. So it's a cool mix. So we're seeing a lot of that. Now we're seeing other things emerging to your question and to not make this all about testing.

One of my favorite scenarios is customers using ephemeral environments for their own customer research. We have a bunch of customers who they will equip their product managers with the ability to like, hey, you can spin up an entire environment that has all the components for product X. But this could be out of a branch that hasn't been released to main. This could be out of a demo branch. And then they can take this.

go to an interview, go to the customer's office, on Zoom, whatever, and they can show them this product working end end that they can click, that they can react to. So that's one that I think is not obvious at first, very interesting. The other is, that just came out of one of our customers, support. You have a customer, they come to you with a problem, you kind of spin up an environment.

feed it all the data and all the state that mimics the customer's setup. And you can use that to reproduce the customer problem. then once you have our production, our reproducible state, you can take this environment, it as is to the dev team. Say, hey, we have this problem. Can you help me fix it? you, whether it's a product fix or a customer facing fix. So we're seeing a lot of that coming up. And the way I think about this is,

Ramiro Berrelleza (10:55.551)
Once the ephemeral environment becomes a unit on its own, it's no longer the microservices that hate the environment as a whole, gets deployed, gets destroyed, gets upgraded, all those things. I think all of these scenarios open up, from very sophisticated testing scenarios to mobile teams using ephemeral environment to test their applications on their demo devices all the way to this customer facing.

Once we had the CTO of one of our customers using the Okteto environments to demo the next version of their product in a confidence to their own kind of like stakeholders, which was very cool because they didn't tell us, but then I was watching the conference and I was like, hey, that link in the URL looks like not the company name. What's going on there? And then I asked them and they told me, oh yeah, no, we have this branch kind of like ready for the CEO and we just deploy the day before the conference

Ole Lensmar (11:35.788)
Okay.

Ramiro Berrelleza (11:52.571)
And the CEO was able to kind of show the full product running in front of like a thousand of their potential customers way before this was merged to production, way before this was like ready for, for, for, you know, for prime time. So those is not just to me, it's fascinating because I think it's something that in the past was really hard to do. And at the end of the day, it is about faster feedback loops. It is about testing because all of this is about helping us shift value faster, helping us.

kind of bring down some of the traditional gates that we had in the past. You know, like you have to wait for CI to go before you, before you merge it and you have to wait for it to be production before you can show it to a user. And that's one of things that I'm very excited to see as more and more people in our space adopt the environment as a core piece of their software development strategy. So it's very exciting. I'm curious how you're seeing it. You mentioned that a lot of our customers, do it. So have you seen anything else?

Ole Lensmar (12:46.69)
Yeah.

Well, yeah, no, I so to notice I have a bunch of questions in my head or thoughts. So one is around that we specifically in the context of testing, right? When you spin up an ephemeral environment, you usually want to prime that with some kind of data set, right? So like you said in the support case, but you might want to take a snapshot from a production environment, right? To say, want this. So you create the ephemeral environment maybe to reproduce an issue, right? Or a bug.

And then to be able to do that, you need to prime whatever persistent storage you have in that ephemeral environment that ephemeral environment is dependent on with that set of data. To me, seems like not always easy, but it's very important to be able to run reproducible tests on reproducible data sets. Is that something that...

is made easier with the ephemeral environments, or is it maybe a product feature of Okteto specifically? Or what are your thoughts and experiences around working with fixtures?

Ramiro Berrelleza (13:49.659)
No, I think it's both. This is one of my other personal vendettas. I don't like fixtures. I don't like this kind of having to run this seed script at the beginning of your setup because in my personal experience, they tend to be very flaky. They are very quickly out of date on those things. So in my view of FML environments, especially in development use cases, actually for everything.

Data is fundamental. And I think that is the piece that most people forget when thinking about their FM environment strategy. Because you know, level one, you want to have an environment on like a single command that you can deploy, that takes very little time to run, and that looks at production, or as close as you want it to be. That is kind of like stage one. But then you have to things with it. And that's where data becomes very fundamental.

part because in modern applications, most of the issues that we have, most of the serious bugs come because of the transactions. That's what really matters. Now, this joke that all we do is move bits from one database to another. And to a point, it's true because whatever we build, we're building solutions to enable customers to do something. And that something, at the end, tends to be state.

if you're making a bank transaction, if you're buying something online, if you're calling a car, that's what really matters. How you get that is what we enable. So data is fundamental because that's where a lot of things break. They'll break because they have incompatibility, because you didn't assume that, hey, there's some parts of the world where there's only, people have two last names, not one, or where names are not just as the characters. data is fundamental and it is hard because it requires kind of a strategic approach.

And at Okteto we ended up a couple of years ago building a feature to help kind of like developers bring data into their dev environments. And in our case, and going back to Kubernetes, we leverage a bunch of Kubernetes features like volume snapshots, PBCs, all those things. And through Okteto, you can build this kind of like supply chain, to speak, data supply chain, where you can have data production in a source environment.

Ramiro Berrelleza (16:11.567)
you can automate getting the data out of that source environment, save it in Kubernetes, on Okteto, on like volume snapshots. And then in our world, when a developer spins up an environment, all that data is automatically there. It's the same data because you're not seeding, you're actually cloning the data sources. It's more reliable. It's way, way faster. Like in our benchmarks, data cloning is 80 % faster than data seeding for like large data sets. And what's really cool is that you can...

Ole Lensmar (16:38.039)
Mm-hmm. Sure.

Ramiro Berrelleza (16:41.253)
fully automate this, we have one of our customers is this big e-commerce store. And they have the whole thing fully automated that every weekend they take a dump from production. They will clean the data, remove all PII, then they make that data available in Okteto so that Monday morning, if you deploy an environment, you have fresh data from that looks like production, you know, that it's for you to develop. And now they have this automated.

Ole Lensmar (17:04.494)
That's very cool.

Ramiro Berrelleza (17:08.491)
every week nobody has to think about it, which for me is the most important thing about this technologies is when you get to the point that nobody thinks about it. It's like, of course I have an environment with data. What else would I have? That to me is kind of like when you achieve success in an organization and we're seeing that across multiple places. I think not everybody is there yet. I think to me that is the next frontier of ephemeral environments is the final data. And you know, there's many ways to do this, but I like the get it from production, clean it.

make it available for developers. for your question... Go ahead.

Ole Lensmar (17:40.972)
Let me, yeah, I'm sorry. I was gonna ask, so like you, but for testing specifically, you probably want a reoccurring data set to be used, right? So you don't want a new data set from production. Well, or it depends on, either you build your tests to be not too dependent on data, which I think, I guess can be hard in many cases, or do you, you said you don't like fixtures and just kind of thinking of how does that play out in a?

in a testing environment where you need to make sure that your application can handle invalid data or certain data sets or blah, blah, blah, all of those kinds of things. how does those, and how would people, yeah, go ahead.

Ramiro Berrelleza (18:19.813)
Now that's a good question. Now that's a great question. So a couple of things where I feel like a ephemeral environment is really help here is you can have multiple data sets. So in this example that I described, when they create these data sets, they also keep the older ones. So in the case of Okteto but this is like when you deploy in a ephemeral environment, you can pick your data set. What I like about this approach and why I think it's superior to fixtures is because you can have that.

Run your tests. When you're done, destroy, launch another environment, load a different data set if needed, go out again. I don't think it's an either or. I think fixtures have a place, and especially for invalid data. At some point, you do want to inject data in your system that isn't supposed to be there, because that's real. In real world, not every data is clean. Yeah, it happens. So that's another aspect where I like the fact that you have an environment where you have a database, you have some data loaded. You can do those things.

Ole Lensmar (19:08.846)
happens there.

Ramiro Berrelleza (19:18.287)
And because it's ephemeral, one, if anything goes wrong, you destroy it, launch another one. You're not blocking other people like what happens on traditional shared staging environments. The other is you can then start to introduce all these tools to your dev toolkit, right? You can have fixtures for this kind of class of testing. You can have real data made for end-to-end testing for smoke. You can have data sets from...

weeks ago because you want to test migrations, like this opens all these doors to make it easier for developers to test because I think one of the reasons that a lot of developers tend to be test averse, especially for like the more complex testing, is because it's hard. Years ago I gave a talk with Uma from Chaos Native on chaos engineering and they did some research and one of the reasons why people were not adopting chaos engineering is this perceived complexity of like, oh my god I have set up

Ole Lensmar (20:02.158)
Mm.

Ramiro Berrelleza (20:15.395)
all these 25 different things manually before running, which meant most people will not do it unless they were forced to. And I think another kind of complex testing that cloud native testing requires can be made more palatable by automating all these things and giving developers all these tools because nobody likes to make production. Nobody likes to deploy to production and then have to sit there and look at tool and look at logs. It's annoying.

Ole Lensmar (20:39.278)
I'm not sure, without having to.

Ramiro Berrelleza (20:44.697)
always makes you feel bad when you break something. So I think there's a space there for all of us to kind of give the same way that, know, like, like, Testkube helps execute those tests in a way that before was very manual. So I think all of these tools together make this offering a lot more interesting, a lot more powerful. And I like that it doesn't have to be one or the other. I think that's a very cool thing about the whole Kubernetes space is how easy it is.

Ole Lensmar (20:46.094)
and

Ramiro Berrelleza (21:14.255)
plugging things together because we have this common layer that is Kubernetes and very powerful. Yeah, like you can use Testkube and Okteto and anything else together without like a lot of complexity.

Ole Lensmar (21:19.03)
model and yeah.

Ole Lensmar (21:28.462)
And I think we're seeing, mean, what I, impressive with customers, I'm sure you see it on your end, how much effort they spend to build their own kind of platform engineering or pipeline pipelines, or kind of how they build their own kind of support infrastructure for their specific needs. And there's so many tools to choose from in the cloud native landscape, but they're all built on the same fundamentals. So it's reasonably easy to plug them. I'm sure there's...

obviously the challenges, but it's reasonably easy to kind of plug them together because they have this, they're grounded in the Kubernetes space that kind of sets certain predicaments for how things work and how things interact. So I think that's really, gives a lot of flexibility to the ops and the platform engineering and developers to kind of really build a platform that works for them and maybe caters to how their business works. So I think that's super interesting, but it's also very complex to get to that level.

Ramiro Berrelleza (22:21.531)
Yeah, it is a complex tool, but I think that...

And we forget how much worse it was before. Like I always feel like, you know, every generation of tools solves a problem and then creates new problems. But the new problems are not as complex as before. know, integrating tools before Kubernetes was very complex. Like, unless you had something like Jenkins, it was just kind of like this open book of like run anything, but build everything. And nothing was compatible. Or you came from kind like the Microsoft world where everything was highly integrated.

Ole Lensmar (22:36.131)
Mm-hmm.

Ole Lensmar (22:48.494)
Thanks.

Ramiro Berrelleza (22:57.071)
but had to be everything inside the Microsoft world, which was also very pricey and had all these other challenges. I think we're in a good path there. And I think that a lot of things also seem more complex because our applications are way more complex than before. The expectation of the scope of an application, of the resiliency, of the quality has grown immensely. Our users are way more sophisticated than, you know.

Ole Lensmar (23:18.51)
change that.

Ramiro Berrelleza (23:24.955)
we always started building software where it was more of a, it kind of works, that's good enough. And I was like, no, no, there's no space for crappy software anymore.

Ole Lensmar (23:34.61)
Okay. So one other thing I wanted to ask you about. is, I mean, we see with our customers at least, the ephemeral environments often used kind of as a CD, post CI thing, right? So you build your artifacts, your Docker images, you push them to registry, and then you spin up in a ephemeral environment where you either with Argo or whatever mechanism you're using, and then you run your tests against that. But my understanding is that you can also use the ephemeral environments much earlier in the development.

while you're actually coding. Let's say I'm building a microservice that interacts with other microservices. I could spin up in a ephemeral environment that contains the dependencies and then either use like teleport or telepresence or something else mechanism to help me while I'm actually coding locally before I even commit and build. Is that correctly understood or do you see that use case or is it mostly more of a post CI practice?

Ramiro Berrelleza (24:30.243)
No, that is actually in my vision, that to me is the biggest accelerator for ephemeral environments. Like the core of our product, Okteto is two pieces. One is the platform to manage these environments at scale. And the other is a set of tools, this very comprehensive set of tools that we built to make ephemeral environments super easy to use for development. Like the core mission of Okteto.

is your run one command that command Okteto up is going to connect your local computer, your file system with your environment so that anytime you open a terminal, anytime you modify a file, all of this will be immediately reflected on this environment. And the idea for this is fast testing, fast feedback loop. I don't want developers to have to wait until CI to then run their code on that environment with Okteto

And telepresence does something similar to a point and there's a lot of few others in the spaces It's becoming a very which I love it's becoming a very rich space with different opinions, but all of us agree on the importance of You need to get your code to these environments as fast as possible if you want it to be part of the core dev experience you're going able to make it that it feels like it's local and That you can use your common tools. So a lot of our customers do that now where now they have Okteto

tele becomes the default dev, not only a phemeral but also the default environment for development. There's no local version anymore. Everything runs remote and the user tooling to give their developers this experience of like sync code, remote debugging, remote execution, and all of these things. Even for dev before CI, before merge, before any of those things.

Ole Lensmar (26:17.646)
I mean, we still see, I mean, I know it's not the preferred way to go, but we do still see a lot of organizations with QA teams, right? Who do like integration testing, security testing, compliance testing, et cetera. And for them, this must also be very helpful to be able to write tests against an ephemeral environment, right? So I need to spin up a ephemeral line so I can create my integration tests or whatever kind of tests I need to do. So it's a dev loop for

for test development because as much as we want to shift left and we talk about it, we still often see teams that are more dedicated to making sure that everything works as a whole as QA teams, which I think also there is value in that definitely. So not disputing the value of having QA teams. Have you seen that at all that QAs use specifically a federal environments to write tests and to validate that their tests are?

working in the sense that they seem to be testing the right thing.

Ramiro Berrelleza (27:18.703)
I have, yeah, that's another very interesting use case that I forgot to mention, which is traditionally a lot of testing only happens on CI because it's very hard to set up, because you didn't have this kind of like end-to-end environments. Typically there's a stress load, end-to-end testing, UI based, like pay rate kind of testing. We're seeing that a lot with our customers where they're now using pre...

environment for that reason as well. They will spin up the environment on a branch maybe, maybe in main, and then you have the QA teams iterate. And one of the really cool things that we're seeing is this is also helping kind of reduce the black boxiness of this class of testing. Part of the fast feedback loops that I see now with Octano is now you can run the same tests when you're developing that non-CI with the same command.

with the same configuration. And the value of this is that you don't have to wait until the CI cycle. So QA teams can develop this test very fast because when you're building something like on GitHub Actions, you have to write the action, push it, wait for it to run, then something fail, then you add some logs, and it takes forever. If you could run everything locally or everything as you write code, you go 10, 20 times faster. So you can do that with test execution.

FFM environments and your favorite, you know, like test execution framework. You can use test cube, can run, you know, play right locally if you want. For me, the value is in these three pieces is you have the code, you have the test executor and the environment. And if you can use those things before you even commit code, before you push, that allows that QA developer to write this test much faster. We're also seeing more sophisticated testing because they get to tinker more and.

can understand better than all the pieces. And also as an added bonus for developers. a test fails, now they have a way to run the test from their workstation. They don't have to wait for another CI cycle, which also speeds up solving the issue that this test uncovered. of our large customers, the reduction of CI cycles was one of their main targets for their developer experience program, which was very interesting.

Ramiro Berrelleza (29:45.243)
And they were investing on a friend environments and this kind of testing class of testing to reduce those cycles. Cause they know that each of the cycles adds hours to the day makes every developer goes lower, which means you know, they're missing their targets. Not all of these things. And it was fascinating to see it. And it's really cool when you see it working in concert, the QA team building a test developers running them, not just as this annoying thing that breaks every now and then, but as a tool to verify.

Ole Lensmar (29:56.844)
you

Ole Lensmar (30:01.954)
Yeah.

Ole Lensmar (30:10.542)
It's part of the...

Ramiro Berrelleza (30:13.133)
And, know, know it doesn't work all the time, but when you see it working this way, it's very powerful. And it really shows the, you know, why you have to invest in QA, why you need to write this kind of more comprehensive test, why they're not just this gate, but actually a tool to help you ensure that you're shipping high quality software as fast as you can possibly can. So to me, that's a fascinating kind of combination and it's really cool to see more and more players there.

Ole Lensmar (30:38.254)
Yeah. I agree. And it's amazing kind of to see that. Like one of the many advantages of cloud native development and kind of building Kubernetes that it allows you to do these kind of shifts in your development life cycle.

find these gains with the right tools. Ramiro thank you so much. This has been great. Amazing to hear about so many facets of ephemeral environments that I didn't know about. So I learned a lot. And thank you so much for joining and sharing. And thanks, everyone, for listening. And hope to see you again in another episode soon. Thanks, everyone. Bye-bye.

Ramiro Berrelleza (31:18.693)
Thanks for watching, bye.

Ole Lensmar (31:20.056)
Bye.