Ole Lensmar: Hello, welcome to this episode of the Cloud Native Testing Podcast. I'm super thrilled to be joined by Sahana Nagapushan. Is that correctly pronounced? Reasonably, I hope. Great. Sahana, great to see you. How are you?

Sahana Nagabhushan: Yes, Sahana Nagboj. I'm good, how are you?

Ole Lensmar: Good, thank you. I'm looking forward to spring, although it's a slow start here in Sweden. Tell us a little bit about yourself. I'm obviously super excited to talk to you about testing and quality at scale, just in general, but also what you're up to at your venture, Dev Plaza, and how testing and quality fits into all of that. But I'm guessing many of our listeners aren't aware, so please go for it.

Sahana Nagabhushan: Sounds good. First of all, I'm so excited to be here on this podcast with you. I know we've been talking about this for so long, and finally we've made it, so I'm excited. A little bit about me. I used to be a developer in my past lives, and then an analyst, and then a product manager. I worked for Fidelity Investments and Workday.

In my developer and analyst days, I used to experience a whole lot of software lifecycle inefficiencies and pain points. And in my product management roles, was more the technical product management from the perspective of developer experience or developer productivity kind of products. It was a great space. It still is a great space, as you can imagine. Basically, the idea was to ensure that developer communities have everything they need to create their products. And as they're creating their products, they're not going through a whole lot of pain points. And so these products that I worked on were basically to resolve all those kinds of problems. And as a product manager, creating products for the developer communities, you tend to also see how it affects the business side and whether things slow down because of inefficiencies that developer communities experience and I saw a whole lot of that. Yeah, go ahead.

Ole Lensmar: That's super interesting. I'm guessing a lot of big enterprises have people who work on internal products to help develop productivity internally and to kind of get the maximum or kind of to track efficiency and velocity and all that kind of stuff. How were you tracking? Were there specific metrics that were applied or what was kind of the way to track the efforts that you were doing?

Sahana Nagabhushan: Yeah, we had all kinds of metrics. Of course, we had product metrics, meaning how does the functional aspect of the product work kind of metrics. But this is more so from the perspective of testing and quality, from the perspective of whether it will perform well and whether it's a stable enough product for customers to actually come and use it. The other things that we used to track internally were also surrounding the engineering metrics about cycle time and how many bugs, how many escape defects, those kinds of things. And that goes on forever, I guess. But in general, the platforms that we are products that we ended up creating were trying to help all these kinds of problems to get resolved. I don't want to say resolve because it's a big word. It's more to reduce because you potentially can never have perfect quality as such. You will never be at 100%. You'll probably be at 90%, maybe 95 % in the best case scenario. And so what I used to focus on at workday especially was to ensure better testing strategies and quality. And at Fidelity, I used to focus on having a developer portal, but also the focus shifted to having an API platform.

In all of these roles, we created all these solutions in-house for good reason, because I did a whole lot of research and of course others on the team also did a whole lot of research. And what we currently have missing in the market is a holistic enough solution that can give the developer communities everything that they need to have that, as I'm beginning to call it, production readiness.

Ole Lensmar: Okay. Yeah. And okay. And so that led you to what you're doing now, I guess, or? Okay, so tell me about that.

Sahana Nagabhushan: So I... Yes, exactly. Okay, cool. So as part of being in all those roles and creating these internal solutions, I was wondering why everywhere I go, I'm creating these kinds of solutions to different extents, focusing on different things, but everything related to software life cycles and efficiencies of how we can make things better for these developer communities.

And because I've been in the space long enough, I've also got some free usability testing from the small, medium, large size companies. And I realized that basically everybody else is also doing some form of the solution. And the most common ones that I have heard of is what is called internal developer portals. And even though those kinds of solutions come about in each of these companies in different forms. The intent is still the same and that is great. But we are doing this everywhere. The stats that I've heard of and done some analysis myself is that we are spending anywhere between three and 16 million dollars per year to create these kinds of solutions. We spend a minimum of two or three years to come up with something a little like we have enough return on investment, but it's not full return on investment. And at some point, even the developer communities and even the executives, they're like, where is my full return on investment? And so either they pull the investment or they continue to keep that stagnant.

So I was like, I understand what this space needs, what this persona needs. I've experienced it myself and I have also try to create these kinds of solutions to help the communities and also help the business. So that's how Dev Plaza was born. The idea of Dev Plaza is that it provides production readiness. And if you think about like a visual, think of all your wipe coding tools on your left side or your IDEs on your left side, and there is your live environment on your right. In between, there's a whole lot of stuff that needs to be done before it goes into those live environments that the customers use. We do it in traditional software development. We do it to certain extents. We end up skipping steps. Simple stuff, like we don't do unit tests until we have to learn it the hard way, or performance tests, or some other kinds of tests, right? It's mostly tests. We don't bring in governance. We don't, or we have manual processes. So these kinds of things take away so much time from core business development. So that's what Dev Plaza takes care of by making sure that you go through every stage of the software life cycle from your, I finished my coding stage to I wanna push to production stage and beyond for any other problems that come about. We call it the production readiness layer and the trust layer to go from code generated or code created to code in production. And yeah, it bridges that gap by integrating the tools and the processes and the governance that's actually required for daily software life cycles.

Ole Lensmar: Okay, I think I understand. So basically, like you said, from left to right, from going from code to production, helping you do that more efficiently, faster, I guess. And testing is, I'm sure, big part of that, but also building and deploying and everything in the middle of it. But you did mention IDPs. I just wanted to get back to that because in the community space, we're seeing a lot of uptake on backstage specifically, but I know there are others. How does that relate to what you're doing? I'm sure that's a common question.

Sahana Nagabhushan: Yes, it is a very common question. Yeah, Backstage is, I want to say a glorified plugin framework. It is managed by open source communities. It is not full lifecycle. It is not production grade. So usually in companies, we end up evaluating Backstage as a potential option for us to do the build versus buy. We think that it's not a good enough solution because it does not give you that enterprise grade quality that you want. But in certain other cases, I have seen that people try it out and they're not happy ultimately because the developer communities are not getting what they want out of it. And so at some point they ditch it and they go back and create their own internal solutions. And so it beats the purpose.

Ole Lensmar: But it also, yeah. Yeah, so I've heard similar, but I also think it feels to me like Backstage is a more generic framework. Like you said, it's a plugin framework and you could plug in whatever you want to visualize whatever you might want to surface inside your infrastructure. While yours is more purpose built or Dev Plaza has more purpose on bridging that gap from left to right. And you can of course use Backstage and others for that, but... You can use them for other things as well. I guess it's the common, you're maybe a little bit more focused on a specific problem domain, and I'm guessing that's why you're also better at it and kind of helping. If that's really the problem someone wants to solve instead of going with a more generic framework, it might be worth looking at something more specific. Does that make sense?

Sahana Nagabhushan: Yeah, that sounds pretty bang on. You're absolutely right. The fact that the whole point of Dev Plaza is to make sure that those software lifecycle inefficiencies are reduced such that you actually are production ready. And that ultimately ends up making sure that the customers, your customers end up getting good quality and good products that they are able to continuously use is what the goal of the product is.

And especially if you think about the age of AI, that's what we are in right now. AI is amongst the coolest technologies that we've currently got, I guess. But it comes with certain challenges. We don't know what the quality is because humans have not written it. It's AI that has written it. And we don't know whether it has satisfied all the functional requirements of what that product expects. We don't know if the trust to ship it to production is there. So I was doing this survey and I asked a few people. I was like, okay, you've web coded great. And but how confident are you to ship that to production? Do you feel okay to ship it to production? And people are like, no, not really, because I'm not sure of this. I'm not sure of that. Several different things that come to mind, especially with all the stats that we've been seeing. That's, I mean, come on, we had that even with traditional software development that we didn't have 100% confidence. Now imagine with AI, somebody else, some machine is writing code and humans are expected to make sure that it's in production with all that trust that it needs to have. So yeah, we are in between that whole situation to make sure that production readiness is actually there because of that reason.

Ole Lensmar: I definitely agree that, I while AI is a wonderful thing and it adds a lot of higher values, a big word, but still production readiness is still, I'm sure even maybe a bigger concern for many who are adopting AI at scale just because it does increase velocity and you're generating so much code, but how can you really trust that code? And of course it goes into testing and... quality and the concept of quality in the age of AI is super interesting because ultimately the models are getting so much better that I'm sure in a year or two the code they'll generate will be even better than what obviously much better than it is today and they'll probably be able to generate tests etc. So the concept of quality is still but I'm sure there's still going to be a production readiness concern and I wonder where that What does that barrier end up being and what are the resources you put into that? I have a hard time thinking that it's only going to be AI from idea to production. There still has to be humans in the loop somewhere to validate. And even if they are using AI as a tool to do that validation, I mean, it's super interesting. It's just really hard to prophesize about where this is going, because it's going so fast and it's going in so many directions. But I'm guessing it's something that you're also seeing a lot and people are asking you, I'm sure, what does production readiness mean in the age of AI?

Sahana Nagabhushan: Yeah, absolutely. You're absolutely right. And I am also a complete believer in the fact that it can't all be AI. It needs to be AI, the friend of the human that is actually helping with actually shipping that to production with that trust that you need. I mean, it makes you wonder, especially in your space, because we have the focus is to ensure that there is correct quality. So what does quality mean in this case? You know, do you want, I get asked this question about whether we write tests. No, we don't write tests. We get tools that write tests integrated within Dev Plaza. So our philosophy is not to recreate the tools that work well in the market. Our philosophy is to actually bring those tools together and create the process and create that governance and then enable the developers to ship. So if you think about it from the perspective of AI, even if AI writes the code, that's the first step. And that code that has been generated by AI will go into Dev Plaza, make sure that all those nuts and checks and bolts are tightened up before it actually ships to production is where we come into the picture. And even with AI, you cannot skip all those steps, technically. That's just an immediate hit for quality.

Ole Lensmar: No, sure. Do you think that AI can help with evaluating production readiness? I'm sure you could have agents that look at a bunch of metrics that you get from different steps in the pipeline. I think production readiness is always somewhat relative. I know that we've had in there our... discussions of like quality gates, which is maybe one way to express a production readiness where there is it's never a hundred percent, right? If people say, you know, if 90% of our tests pass, we're good to go. Right. And I'm guessing it's a similar thing here that production readiness is somewhat a it's a relative thing. And I could I'm sure help evaluate that and maybe gather data from a bunch of different data points and different types of tools to to give you that confidence.

Sahana Nagabhushan: That's exactly where we are headed as well, actually. We have workflows that we've actually already released for ensuring better quality. And honestly, this came because people said amongst the other workflows that we've already planned, we've done research and people feel like quality is their biggest problem. And everybody wants to have that high quality and they don't want their bugs to shift to production. And so we ended up focusing on that and we've actually created our first workflow and rolled it out as well, which is for early bug detection and resolution. And this is exactly helping with, first of all, figuring out what the bug is, why it happened, and how you could fix it. In the next stages, we are obviously also going to be having more agents that will become a part of this workflow and also several other workflows. And that is going to help get insights as well to figure out where there are opportunities for enhancing it. So we don't want to be blind to the problems that are there that customers have raised or something's about to go into production and then you find like a critical bug. And you just get the approval anyway and push it to production. We don't want all that. And so yes, that is where the insights and analytics that AI produces is where we are obviously also going to leverage that and give those insights back to the teams so that they are able to use that and ensure that they get that confidence and they have actually shipped something without a critical bug or a major bug into production. Yes, AI can be extremely helpful, but we need that AI as the friend and not the main person there.

Ole Lensmar: I agree. What do you think, mean, something that I've been, you know, encountering with once again our customers is around using AI to generate tests and to do testing. And it feels like a little bit ambivalent in the sense that when it comes to quality and testing, you want determinism, right? You don't want to, and flaky tests is like the worst thing that can happen because they're non-deterministic and AI is non-deterministic. So how does... How does that, and we can say sure the determinism is going to go up and up and up, but it'll never be 100%. And how, are your thoughts around just using AI to generate tests or to do like quality efforts? That doable or is it feasible or what do we draw the line or would you delegate, could you delegate some testing to AI, but there's some testing still has to be done by humans. What are your thoughts?

Sahana Nagabhushan: Yeah, it's a difficult question to be honest. But you're right, even I get the same kinds of questions, right? I'm not even in the space of writing, know, creating only testing solutions. Like you are directly in that space. I mean, in the business of trying to get tools like TestCube integrated with Dev Plaza and provided to the customers that want to use it. But they still ask me, hey, do we have tests that these tools can write? Or do you help with writing those tests? Because this is actually their biggest problem. Because they take so many cycles to actually write clean tests, good tests, relevant tests. And people write tests sometimes just for the sake of having a check on the automation testing box. And obviously it's useless. So where does AI come into the picture? There are so many tools that have begun to do that. In all the research that I have heard of, it's improving the way that AI writes these tests, but it's not there yet. I think that there's a lot of hallucinations and there's a lot of misses from the perspective of what needs to have a test that needs to be written for a block of code or a function or edge cases. Those kinds of things getting skipped is exactly the kind of quality problems to expect.

Ole Lensmar: Yeah, but I think it also puts a finger on your point if you're prompting AI to generate tests that, you know, whatever you put in is what kind of comes out, right? So you need to be really clear on your requirements, how you want things to work and what are the edge cases to your point. The AI is not good at guessing in the sense, right? So and so it's just interesting to see sometimes you could say that almost makes you sometimes makes me feel more honest about what I'm trying to create because I have to be really think through. If I'm talking to someone and say, hey, we need a test for this. And then I'm kind of thinking that that person will have the understanding and context of what I actually need and what I actually want. But you can't do that with AI. You have to be very, very clear. And which I think is a good thing, right? Because it does also force you to think about what are the actual requirements? How do I actually want it to work? I was paging. I don't know. I'm just making things up. So when it comes to creating tests with AI, most of the testing tools today are code-based. And AI is great generating code. So I'm sure it can do that. And it can definitely help you with scaffolding your tests and getting basic stuff into place. To really have valuable tests, you need to spend a lot of time giving the AI the context as just as you need exploratory testing in the real world, you're going to continue needing it, I think, in the world of AI-generated testing.

Sahana Nagabhushan: Absolutely. I think there's one other thing that I would want to add here. The fact that AI as a system, okay, or a machine that is writing some tests, there is the human aspect of it. You're creating products for hopefully humans. Sure, there are products for agents these days, but then ultimately, you know, it goes to the humans in some form. So how does AI know enough about what the customer wants? And that... In so many cases, this is about the sentiment and the emotion and the experience. And it's hard for a machine to figure all this out. And sure, maybe we'll get there someday. I'm not saying no to that, but I think that is still a little far-fetched, which is why we need the humans also telling the AI, here's the emotion, here's the experience. This is what I don't like. There was no test for this particular scenario. It caused a problem in production, so give me a test. So I think that's extremely critical when we are thinking of how to even train the models such that it is able to think that these kinds of things also exist. And so many things come attached to with feelings.

Ole Lensmar: Yeah, no, you're right. It makes me think of sometimes you want almost AI to be a little bit more skeptic to what you ask it. Because you ask it to build something, it'll say, OK, I'll build that instead of saying, is that really what you want? Is that really going to achieve what your users and users want to your point? And especially when it comes to testing and test creation, maybe AIs should be little bit more questioning. I'm sure they could be if you prompted them correctly or trained them correctly, et cetera. But there's currently, I almost sometimes feel like I want more pushback from these models because I know I'm not always thinking the right thing and asking the right stuff. And I'm sure it does have the right, it's been trained in a way that it could actually say, sometimes it does, right? What do you actually mean? But in many cases, and especially for testing, when it comes to evaluating quality, which can be somewhat subjective. I'm thinking there that AI could also actually be helpful in creating the right tests, but maybe we're not there yet, but I'm sure we'll make big advancements in that space as well. So what's next for Dev Plaza then? How are you kind of rolling out the product?

Sahana Nagabhushan: Exactly. Yeah, absolutely. So we are actually live and we have got a few early customers that have started using the product and they have started seeing some good benefits from it from the perspective of faster builds, reduced errors. TestCube is one of the things that we have integrated with by the way. So, you know, they because those, you know, we started off with this whole conversation with the this up to $16 million savings, two to three years of time saved. They are beginning to confirm that, you know, they don't need an extra solution that is piecemeal or fragmented. They are able to get what they need from Dev Plaza. And of course, it's not like it's a perfect product even today. Like there's always scope for bringing in more and obviously that's where we are getting this feedback from our customers and stakeholders as to what else they would want as part of the platform. And the basic edition is currently live. We are also working on the premium edition, which is next. And eventually we'll work on the comprehensive edition. But yes, we are excited to onboard customers and by customers, I mean over here. Any sized companies, the only criteria is that they're building something software related and they want to have some kind of monetization. Even if it's at a later point, that's fine. What we want is for them to make something out of that product. So basically people or companies of all sizes, even hackathon users that want to spin stuff off and do their own thing eventually are completely welcome. And so we've started seeing that as well at this point.

Ole Lensmar: Cool. Great. Okay, let me ask you one final question. I hope this doesn't put you on the spot. So to what extent are you using Dev Plaza to develop Dev Plaza itself?

Sahana Nagabhushan: Okay. Okay, I'm glad that was an easy question. We are using it. Yeah, so we are using it. Obviously, we want to drink our own champagne, and we want to make sure that it's all right and it's working from the perspective of what a developer or a team or even an executive would expect. And so we do use Dev Plaza for our own development and testing and ready cycles as well.

Ole Lensmar: Okay. Okay, great. Well, thank you so much for sharing that, Sahana. It's been a pleasure having you here and to talk about AI and quality and Dev Plaza and everything else. I wish you all the best and good luck going forward and thanks everyone for listening.

Sahana Nagabhushan: Thank you so much. It was such a pleasure talking to you again.

Ole Lensmar: Take care. Great to have you. Take care. Bye bye.