Ole Lensmar (00:00)
Hello and welcome to another episode of the Cloud Native Testing podcast. I'm your host Ole Lenzmar, CTO at TestCube and I'm thrilled to be joined here by ⁓ one of the latest people joining TestCube itself, Lakshmi. Welcome.

Lakshmi (00:13)
Thank you, Ola. And it's super exciting to be a part of TestCube. Again, I used to be the senior most person in my previous company and joining TestCube feels like joining a place where I've been mentored by veterans in this space. So super excited to be a part of something bigger.

Ole Lensmar (00:29)
Great. So I'd love to hear a little bit of a background view and how you kind of got into the space, but then you teased something just as we were preparing for this that I want to get back to. But before we go back to that specific topic, why not just give us a short background of where you are and kind of your thoughts on the space of cloud-native testing and things that are going on.

Lakshmi (00:51)
for sure. So a bit of background, I did my masters in electrical engineering and that was something very different from what software is currently. But again, worked a lot into the hardware architecture side from a memory perspective. And I got naturally attracted to performance engineering, which is again, working very closely on how your memory works, how is your application restoring the memory after your load test is done. And I got pulled into the

Performance space. That's how I got into testing. And while I got into testing, cloud was getting pretty much, was back in 2015. Again, that's when like people were adopting into cloud, moving away from physical servers. And I was working with Comcast then. So if we were moving out of slowly out into the cloud, cloud world. And while we are moving into the cloud world, what we notice is like people are thinking, okay, now we have auto scaling enabled. Do we actually need performance testing? Like, okay, the server can pretty much handle all the load.

I don't really need to have it tested. We did go to a couple of releases without the performance testing. We got a cloud build which was like...

insurmountable like pretty much beyond what we expected. And that's when they realized, okay, we need to catch memory leaks, we need to catch a lot of other things. So we went deeper into performance engineering, identifying these bottlenecks. And that's when we realized as technology evolves, testing is going to be a core component of it rather than being eliminated out of it. And with the AI evolution, I feel pretty much the same, right? I feel like

as and when people are deploying more applications or creating more applications or getting code pushed at pretty much unimaginable speeds. Testing should be in fact much faster and complementing the development cycle rather than being left behind.

Ole Lensmar (02:30)
I mean, as always, I would say it feels like always like the testing is easily left behind. And like you said with performance testing when the cloud kind of was a new, but also now there's there's I'm sure there's people think, well, if AI will just generate perfect code, we want to not going to need any tests. I don't think any one in the QA world or more senior believes that at least I'm skeptic. But I mean, you said something really interesting.

I'll get back to it. You said like, testers were the perfect vibe coders, right? Or something in the end. So please, I'd love to hear you kind of elaborate on that statement and let's see where that takes us.

Lakshmi (03:08)
for sure. So when, the term vibe coding was introduced, right again, even before vibe coding, as soon as we had the ability for the AI to write code, again, I'm not coming from a traditional coding background, no way closer to a developer, nor I've actually had the energy and effort to actually put my time into learning deeper programming skills, because I was more into the testing space where finding bugs were more fun, or pointing fingers were more fun than writing that piece of code and getting pointed at. But when the

AI

came in, I started building stuff on my own using prompting because I know exactly what I want to build. The biggest barrier was the language between me and the computer. Like it was not able to understand what I was trying to tell earlier. But with the AI evolution, that barrier was removed and Jensen Huang rightly told back in 2019 or 2020 in the early stages that English is going to be the new programming language. A lot of people laughed at it. But now we know that English is technically a new programming language. As a matter of fact,

don't even use my keyboard a lot at work right now. I've got my mic in and I use a software called Whisper Flow, which is pretty much talking to your computer. So it's like living in the future already for me. So that's how I got into wipe coding. So the first app which I built was primarily, my daughter was born in the North America. So she speaks only English and my parents back from India, they don't speak English. So I wanted a natural translator between

Ole Lensmar (04:12)
Mm-hmm.

Lakshmi (04:34)
both of them so they can actually share things and communicate to each other. So I wanted to build an app, if possible have a physical device again coming from an electrical engineering background use Raspberry Pi combined all of the programs together as a matter of fact I didn't even open my IDE while I was building this there were like so many bugs out of it. So

Ole Lensmar (04:39)
Mm-hmm.

OK.

Lakshmi (04:53)
But I got it to work, which was like the first biggest victory for me. Okay, I was able to have my daughter speak into it and it was converted to my native language Tamil. And my parents were able to understand and they were able to talk and it was converted to English. And initially it was going really good. like, as you see a tangible output without having this programming skills, I thought, okay, we are actually having English as the programming language, but there were gaps. And that's where I had to wear my testing hat.

Ole Lensmar (05:09)
Amazing.

Lakshmi (05:20)
and go through several iterations to test out different positive, negative, all of our edge cases to make sure things are working as expected. And I had written a blog at that point, why wipe coders or like why testers are actually good wipe coders or why testers would be good wipe coders because we always try to think where things could break. So that's the mindset which testers have and that's something which I firmly believe in because testing.

Ole Lensmar (05:39)
Mm-mm.

Lakshmi (05:45)
is by itself not just creating, it's making sure that what you've created is working as expected for all possible cases.

Ole Lensmar (05:49)
Mm-hmm.

Mm-hmm. Yeah, that's an amazing story and I'd love to see that thing in action. I mean, I have a couple of questions. One is just around, are we thinking of doing any automated testing for that, what you built, like where you would automatically put in some state language in English and you would get out in the language that you expected and vice versa, or was the technology not there?

Lakshmi (06:16)
there was that that technology wasn't there at the point. So what I did was again, I went into hugging phase. I got a model which was pre deployed.

A lot of people have actually explored this area to convert one language to another. I didn't, mean, it's good thing is like, if you start building everything from scratch, it's going to take forever. So try to leverage what's already pre-existing. So I went, had an opportunity to use those models from hugging face, have it deployed onto it. And good part is I did spend some tokens on the API keys.

Ole Lensmar (06:28)
Mm-hmm. Yeah.

Lakshmi (06:44)
So that was like, okay, that is the only part which I had to experiment. I always considered like, like when people say like, how much you spend on AI, spent about $400 to $500 per month in the last three years. And I don't consider it as a spend. I consider that as an investment. I mean, being in this field, it's like you, things are going to evolve. Either you travel along with it or you'll be left behind.

There's only two options. So I had this model deployed. I was able to get things done locally. Things were working seamlessly. And I, again, I, my whole goal was to make it like a hardware toy kind of a thing where you could pretty much give it to multiple people. Again, this is just for like, not just for English to Tamil. It could be for English to Spanish. It could be for any language. And the biggest thing for a grandparent.

is to make sure that they're able to talk to their grandchildren in the way they want to talk. So their communication, and that's where I feel that AI was able to do that. making it to productionize was something which I was not, I mean, it requires a different level of energy and time commitment. You you've built several companies before, so it's not easy to build a company overnight, but it solved the problem for me. So that's why I noticed like, okay, things like.

Ole Lensmar (07:34)
Yeah.

Lakshmi (07:56)
If AI could help me solve smaller problems where I don't need to rely on buying a third party software, I'd be more than happy to incorporate that into my real world.

Ole Lensmar (08:05)
Okay. Yeah. Okay. Great. Well, thanks for sharing that. I want to get back to something else you said about like corner cases for testing. And my experience, at least when using AI to create tests, is that it's actually pretty good at, if you tell it, you know, think of all the corner cases and all the things that could go wrong. Maybe specifically for a developer who maybe not is always good at that, right? Is thinking about all the variations of something. My experience is that AI is actually really good.

uh, aid in helping maybe developers create more comprehensive test suites that cover all the, you know, the, negative cases and all the paths that you didn't think of. Do you have a similar experience or do you, do you think that there's, mean, I'm sure testers can still go one step beyond maybe where AI is today, but ultimately there's a lot of material online, uh, which AI has been trained on that touches on this subject. So I would kind of expect AI to be pretty good at, at kind of.

covering those kind of cases? What are your thoughts in general on that maybe or specifically on that, but also in general just on using AI to to cover to improve your test coverage?

Lakshmi (09:10)
for sure. So this is something where I noticed while I was working with my previous company again, another Y Combinator backed startup, like I was hired as a third person into the company. So, know, I was like very close to the product as well as the customers. So that's when we realized, OK, things are moving fast. Of course, there are going to be testers. We're not trying to replace testers. We are trying to give them the superpower. Right. We're trying to make them.

No one is replacing developers. Developers are now using AI to create code at a faster pace to deploy it. Why not testers use AI to complement the faster code deployments which are going through? So that's when we realized, okay, if AI could go through the pull requests to understand, again, nowadays even pull requests are written by coding agents, right? And you've got enough elaborate PRs which are going to be giving you all of the details on.

what features were implemented, what's going on, you can get multiple factors out of it. The first thing is you'd be able to get clear understanding on what features were touched during this PR. So that's the key core aspect of it. So if you have like 2000 odd tests, regression tests which are running, you don't need to run all 2000 of them for your current feature deployment. So the first thing is you'd be able to filter out what tests are going to be run as part of this cycle.

Ole Lensmar (10:18)
Mm-hmm. Mm.

Lakshmi (10:25)
The second key part is if there are new functionalities which are developed, does my testing bed already have all of these tests pre-written? Of course not, because these are new features which are developed and this is tying back into the whole concept of shift left testing. So how do you bring testing into your, as close as possible to your development? So that is where you get an understanding of what features have been developed recently. Look at your existing framework.

and see how your testers have written this test. Let's not put that back into the drain, right? Let's try to make use of it. It might be in Catalon, Selenium, Playwrights, Ipress, name a tool, right? There are like numerous tools and there are more tools which are getting released every single day. So beat any framework, you try to understand what's the framework underlying behind it because you've got, the AI has got access to the repository. So you would be able to...

complement that or add additional tests based on what's already pre-existing. So your tester who's going to be reviewing these tests, not only knows what tests are being created, but also these are created based on the framework which he or she has built. So it's not like I'm reinventing the wheel. I'm actually building a vehicle on top of the wheel which you've already invented. It makes things easier and that's personally how I think we are moving towards, yeah.

Ole Lensmar (11:29)
Mm-hmm.

Yeah,

That's super interesting. think, I mean, I think there's been a lot of focus maybe now in the, in the community around using AI to generate tests, which is perfectly understandable. think one, one interesting aspect there is to your point, if, if let's say you open a pull request or you create a commit and you ask AI to analyze, okay, just do we have coverage for the changes that were introduced and which tests are there? And the next step.

for the things that are not being tested, can AI generate the tests, right? Or at least generate the scaffolds for the tests and then someone might have to post process them to make sure they test the right thing. And on that specific point, I've always been thinking, if you ask AI to generate tests, it's gonna generate tests for what the code does, right? And not what the code might actually be supposed to do, right? So if you've introduced bugs in your code,

Lakshmi (12:24)
Mm-hmm.

Ole Lensmar (12:30)
Obviously, AI will be smart and if it says like if the method is called multiply and it does a subtraction, it's probably going to tell you, hey, this seems wrong. But it could be much more subtle where AI doesn't really know what the actual requirements are. So that's also something that I think is super important to tie in into these analyses is to bring in requirements from your test management tool or your test planning tool or maybe if you're structured, you have your requirements.

part of the pull request or in the code base itself. to give AI actually a picture of what the code is supposed to do versus what it actually does. So it doesn't just use the code with any bugs it might have as a reference.

Lakshmi (13:07)
Totally context is the key right again without context the more the AI knows the better it is for the AI of course your tokens are going to be used left right and center that's how the world operates right like again I was just reading an article where Amazon actually kind of gives credits or basically appreciates

Ole Lensmar (13:12)
Mm.

Lakshmi (13:26)
their developers who are actually using more of tokens. So people are automating several things which necessarily doesn't need to be automated, but they still like to do it because they have the ability to do it now. So why not testers who actually know what to get tested, use AI's brain with the context which AI brings in to get all of that tied together. And that's something which is pretty fascinating for me from a test cubes perspective, right? Again, a little bit about

Coming from the industry where I worked with several tools which are like legacy tools, I invented about in 1998 and stuff and like same like tools which companies which we bought in the last six months. fortunate enough to work in both of these worlds. What I noticed is the testing concept still is the same. The whole goal of the tester is still trying to find out what's happening. It's just the methodology behind it is going to be changing. And with test cube, the key aspect is.

doesn't matter what it's pretty much the same concept right doesn't matter what tool you use we are not trying to optimize or we are not trying to change and bring a new solution to you right like

you might use a deterministic tool, you might use something like a locator based tool or you might use a non deterministic tool, something which is generic, complete the checkout flow, there are tools which actually does that, one of my previous companies does that where you kind of let the AI loose on your application and it pretty much goes all around the site and does different options.

Ole Lensmar (14:47)
Mm-hmm.

Lakshmi (14:48)
The tester here can make the best use of what both sides of the world. There is no right or wrong. There is no right or wrong in testing, right? It's just the output which is binary. It's either pass or fail.

Ole Lensmar (14:53)
Mm. Mm.

Yeah.

Lakshmi (15:00)
but

the method of testing can never be categorized as, okay, you're using Playwright, you're not doing it the right way, you're using Cypress. I've seen Selenium frameworks, again, Selenium has been in the field for quite some time, which are super robust, right? While working with Katalon, I've seen customers who actually customize Katalon, which is a wrapper around Selenium, in such a way that even the product team was like, wow, our product could be used in this way. So it's all about the creativity and the knowledge the tester brings in with the domain.

combining all of that together to bring it into the testing world.

Ole Lensmar (15:32)
Super interesting. I'm actually intrigued by what you just said, the non-deterministic testing approach, where if you say, here are the requirements, you know, validate the checkout process. I'm just curious there kind of how, just from a token perspective and from a performance perspective, that seems to me like it could take some while, it could take a while and could be expensive. But maybe if you weigh that against having to manually...

or having to create the tests in a more deterministic way, it kind of evens out or maybe it's better. And I'm guessing that's the case, what these companies are making. Do you have, it sounds like you have some experience with that. What are your thoughts?

Lakshmi (16:07)
Exactly. So it depends on who uses it, right? Like say for example, the ICP for the previous company, which I worked for was completely on the e-commerce perspective. So with the e-commerce perspective, if your checkout flow breaks, you're pretty much going to be like, you're losing direct revenue, right? It's like direct impact on something, which is like, it can never be.

Ole Lensmar (16:16)
Mm.

Lakshmi (16:27)
compensated and you okay with spending say a six figure on a testing tool which goes and does this from a non deterministic perspective where it goes and does all sorts of testing. It has to always compliment your deterministic part. Your checkout button should still work but possibly the agent would have not gone to a checkout flow from your

Ole Lensmar (16:45)
Mm.

Lakshmi (16:50)
main PLP page or PDP page, it could go to your checkout flow from any perspective. That is where you kind of complement both of these worlds. And I noticed from an e-comm perspective, they are totally okay with bringing in these new tools because customers are also like AI, right? Or in fact, customers are way more, what do call, First thing, secondly, on Amazon, if it takes a longer time to load, you're going to be clicking around.

Ole Lensmar (17:10)
Yeah.

Lakshmi (17:15)
You're not like going to be staying on Amazon or staying on any e-comm site per se. So that is where you try to bring in this non deterministic approach where you let you give instructions based off natural language, where you kind of tell the tool or you kind of tell the agent to go take a persona. It is an interesting use case again.

using this non deterministic piece, you can actually give an agent a persona. So you can let me I mean, in my previous company, again, you can actually let the agent take a specific persona like you can tell the agent that you are a 35 year old mother who's looking to buy a mattress for your kids. by default, the agent assumes this persona and looks for bunk beds in a furniture store.

Ole Lensmar (18:03)
Mm-hmm.

Lakshmi (18:05)
So

that knowledge, that brain behind it is something which the agent is bringing in, but you on the human in the loop, like you would still give the agent the persona. So I feel the tester should think on how they can best use the AI agent to get their testing done.

Ole Lensmar (18:21)
That's fascinating. Actually, I haven't thought of it. mean, I was just thinking of you'd give it like personas of you or someone who has never used an online shopping site before, right? Like, you know, when you're, I'm sure you've bumped into people.

that this is the first time they're doing something. And for you, it's super obvious, but for them, it's like, what is the button and these kinds of things. And often websites or applications aren't really optimized for those users, not understanding me. And so I'm sure that even if it's non-deterministic that...

approach using outlier personas can still help you improve the overall experience and track down usage scenarios that maybe you weren't really mindful of, but that could still be helpful. And of course, you want every customer to succeed regardless if they're new or not to your website or to your application. That's very cool. Another thing, something that we see

Lakshmi (19:10)
Exactly.

Ole Lensmar (19:14)
also like with our customers maybe, and also in the testing community is that a lot of talk has been about creating tests and maybe executing tests, like you said, non-deterministically, but it feels also to me, AI is moving downstream from those steps. Like one is around selecting which tests to run, where AI can definitely help because by gathering context from infrastructure and code and everything else, but then also.

once your tests have run, what do do with all those test results and how do you sift through them and identify, for example, if you think this test is flaky, is it actually flaky? And it feels that's where AI could also be super helpful because it just can so much more easily ingest more data, right? So on flakiness specifically, that...

What you might conceive as flaky is maybe not really flaky. It's just that there are certain things going on beyond what you see in the tests that were in the test results that are resulting in what you see. if you give AI all that context and through MCP now, you can give it access to, your source code, your APM tools, your logs, and your infrastructure, et cetera. And from there can really get a much more holistic picture and do a much more holistic analysis and summary.

So to me, it feels like that's also a place where AI could save a lot of time. And I'm sure it'll be wrong sometimes, right? But so are we. what are your thoughts there? Do you see that being a valid use case? And what do you think the adoption is of using AI for those kind of use cases?

Lakshmi (20:48)
totally right. Here is where again, I'm pretty new to test cube, but I kind of like one key feature which test cube has recently launched, right? More looking at from a analysis perspective. So from testing world perspective, right? Creating tests have now become pretty easy, right? Just now we established that, okay, that's not a pain point now. Like if I'm able to create an application and deploy it.

testing it can also be done at a pretty much faster pace. So it's not like that is not really a problem now. Now the bigger problem is what value I get out of the test. Writing the tests kind of gives your PM the confidence that, okay, I'm going to be testing like so much. My application is going to be subjected to test a lot and I'm going to go to production after the testing is done. But people often forget the next piece, which is the results.

What do I get out of it? Right? Analysis component. It can be performance. It can be functional. Any any test, right? Like is my memory getting released after my endurance test of eight hours is done? Would let you know like, OK, you're not having any sort of memory leaks. Things are working as expected. The same thing with functional perspective. OK, 10 tests have failed. That gives you a very, very vague reason. As a tester, the goal for me is to find out why things have failed.

most importantly what has failed? Is it my test which has failed or is it my application which has failed or is it my infrastructure? Say probably a service connection is not established properly. So probably that downstream system which is supposed to give me a response, it could be a stock picker, it could be any sort of response, it's not getting updated in real time or I'm not getting that rendered properly which fails an assertion for me in my test. So what is exactly causing a failure is a bigger problem right now and with TestCube what I noticed is

you would be able to spin up these failure analysis agents. And I think, correct me if I'm wrong, we are one of the first tools which actually gives the customer the ability to modify the agent. the customer can actually give more prompts to analyze the logs they wanted to be analyzed because they are the one who do it manually today. So they know how they analyze their logs. Is that fair assumption or is that fair?

Ole Lensmar (22:50)
No, definitely.

mean, especially when it comes to, so yes, TestTube comes with a built, like four built-in agents and then a bunch of templates that you can customize and you can obviously create your own. But I think ultimately we can't anticipate, you know, which MCP servers you might have, what components you might have. So you have to be able to.

Customize exactly how you want to prompt it and what you think is important and which you know tools to use and whatever so I think it's it's hard for we provide the framework for building the agents that react to your tests and analyze them and then we provide you with some templates and examples and out of the box so you can easily get started, but I think Everyone's gonna have very or will have their own use cases and their own

requirements regarding integrations and what tools they want to tile in and how they want to report on, know, do they want to notify on Slack or in Jira or whatever. And I think that all needs to be customizable. We can't we can't provide like out of the box solution that covers everything. We can make it easy for you. It's like it's making we want to make the the simple things easy. Right. So we have out of the box agents that can do many things for you, but also the advanced things possible so you can customize too. So definitely that's been a design point for us.

I mean, we're seeing a lot of interest here. think what's interesting on our end, since we have a lot of enterprise customers, they have questions around, you know, bringing their own LLM and running in air-gapped environments and those kinds of things, which we also support and we've built into the product. But I think that's also something we're going to see as people start being conscious of token costs, which I'm pretty sure. mean, I think we've already seen people like doing token maxing, et cetera, that it's actually not that cheap.

looking at ways, we use non-frontier models? Can we use local models? And especially in what we do, like troubleshooting and log analysis, that's something that even a, you definitely don't need a frontier model to do that, right? So you can get very far with an older, faster, or maybe even an open source model and get perfectly valid and usable results for analyzing logs or doing flakiness analysis. So that's...

think there's still a lot ahead there as people get their head around how to use AI in the most efficient way. And I mean, it's a super exciting space to be in, obviously.

Lakshmi (25:01)
for sure. one thing which I think would close this loop, right? Right now, the analysis is getting pretty stronger. So you know exactly why things are failing, where things are failing and what's happening out of these agents. The next default path would be, right? Say, for example, your login button is now changed into sign in and you have identified the locator based out of your ID specific to sign in. And now things are not working as expected. So what?

the agent could do is go into your test code repository, understand this failure and update your test script as well. So that could be kind of closing the loop where if in case it's not a genuine failure, but it's a failure on your test script side. So you can technically fix this test script, rerun it and get back into the loop and check whether the test script fixing has actually made your test to pass. In that way, you are kind of

Ole Lensmar (25:34)
Mm.

Lakshmi (25:54)
I mean, every tool calls self-healing, but I kind of feel that this is the true self-healing where you kind of heal based on your analysis from your, what do call it, from your existing test runs and you rerun it. So that way it kind of closes the loop. And when you do it a couple of times, when you report a bug,

Ole Lensmar (26:06)
Mm-hmm.

Lakshmi (26:14)
your developer by default wouldn't really blame the tool anymore. They would take this, okay, this has been going through this loop. Now I know that there is a fault on my end. So I'll go and look at it from my perspective. So it kind of makes the tester more reliable or make them more confident when reporting a bug.

Ole Lensmar (26:17)
Hmm.

Yeah, it's fascinating. To be honest, I'm always a little bit like self-healing. I'm still a little bit like, don't you want a human in the loop here? You never know what the AI decides is actually the cause of the error. But to your point, an AI could, you know, try to fix the error, open a creative branch and run the test and say, okay, the test passes and then open a pull request, right? Saying, Hey, this is what I did.

to fix the test and then there's the human in the loop to say you did the right thing or actually you misunderstood and then you have to go back and correct the error and teach it so it doesn't do the same error the next time. definitely I think that feels like the way we're going and as models become more deterministic and as people get better at prompting to get more deterministic results I think that feels like an obvious path forward.

Okay, awesome. Thank you so much for joining. It's been fantastic to talk to you and it's going to be amazing to work with you. So thank you so much. was a privilege and thanks to everyone listening and looking forward to the next episode. Bye Lakshmi. Thank you. Thank you.

Lakshmi (27:31)
Awesome. Thank you all.