Video: An Architect's Guide to Finding Untested Risk in ServiceNow | Duration: 3180s | Summary: An Architect's Guide to Finding Untested Risk in ServiceNow | Chapters: Welcome and Introduction (2.24s), Speaker Introductions (60.63s), Session Overview (126.53s), Testing at Walmart (171.305s), ATF Coverage Challenges (276.57s), Automated Regression Testing (417.53s), Production Break Causes (550.355s), Platform Demo Overview (713.735s), Dependency Risk Analysis (930.63s), Platform Comparison (1232.115s), AI Production Readiness (1371.935s), Automated Test Generation (1515.425s), Test Case Generation (1679.625s), Workflow Testing Validation (1868.52s), Cross-Platform Integration Testing (2070s), Resilience Engine (2303.865s), Change Readiness Review (2540.27s), Risk Prioritization (2754.925s), Q&A and Wrap-up (2918.48s), Closing Remarks (3074.352s)
Transcript for "An Architect's Guide to Finding Untested Risk in ServiceNow": Hello everyone and welcome to another Now Then webinar. Thank you so much for joining us and please let us know in the chat where you're joining us from. It's really love to have you all here. Today's session is called An Architect's Guide to Finding Untested Risk in ServiceNow, and it's in partnership with Copado. A really big thank you to Copado for preparing this session with us. We're really excited to dig into this content. Brilliant. Just a bit of housekeeping before we get started. Obviously, we are now Ben. As you know, we are a ServiceNow publication. We publish a newsletter, we publish articles every week, and so much more content all about ServiceNow trends and best practices. Please make sure to follow us if you wanna learn more. Today's session will be recorded, and so it will be sent to everyone's inbox, later this week, along with some key takeaways. So hopefully, if you are missing us today, you can rewatch the recording on demand. Cool. Without further ado, I would like to welcome our speakers to the stage. Mike Black, Solution Architect. Thank you so much for joining the session. Hi, Would you like to give us a little bit of intro to yourself? Claudia. Yeah. Thank you so much, and thank you everyone that's joining. So my name is Michael Black. I'm the solutions architect for Capado. Joined from Walmart leading service not practices there as well. Went there for about ten years and then, of course, came over to Caputo to lead the ServiceNow practice specifically for their group. That's me. Ready? Amazing. Thank you, you. Thanks, Claudia. And my name is Hardit Singh, and I'm working as technical and platform architect in one of the financial organizations. And I have been working with the platform for over twelve years. Prior to this, I was working on HPSM, kind of the competitor of ServiceNow, but not anymore in the market. And I have worked across multiple modules of ServiceNow, ITSM, ITOM, CMDB, IRM. Currently, my focus is on AI. That's it from my side. Amazing. Thank you for the intros, Skye. So, what we'll cover today essentially includes, you know, where risk can build up, between major upgrades within your ServiceNow instance, where these risks can hide, how to kinda test what matters first, so where to start testing, and how to decide, you know, where to start this process. And also what I wanna share today is Mike will be, sharing examples from his own instance three times during today's session to make sure it's really practical and useful for everyone. We also have some polls, so please do respond to the polls as the conversation goes on because we would love to hear your point of view. Brilliant. Without further ado, I wanna leave leave it to our speakers. I'll jump off stage. But, yeah, we'll guys, and thank you so much. Claudia, thank you so much. I appreciate it. Thank, you. Argi Ardi, I don't know how we wanna kick this off, but I think the the the big thing for me is gonna be kind of the conversation around what is ServiceNow risk and kind of where what areas are we gonna be focused on today and kind of what. topics. I know you've had a lot of experience. correct. So before you start, actually, I heard in your conversation, in your introduction that you worked in, Walmart. before this. So how did testing look like in there? What were you doing? What was your role? And was it a pain like us? Yeah. So testing was large. As you can imagine, Walmart's a fairly large organization. The testing practice, of course, fit that same model. Primarily, of course, we have a lot of good talent there. So the testing was a lot of manual regression. We had a lot of selenium scripts. And I would say pain wise for scale, certainly, and for customizations, it was definitely an issue, for a long you. time. Yep. Right. So, one quick thing. So, you said you use Selenium as well. And were you guys using ATF also or it was just Selenium? Yeah. We adopted ATF, I think roughly for a little bit to kind of fill some of the gaps, and it was, of course, kind of bleeding edge. We had migrated from Legacy Remedy somewhere close to 2018 and then adopted platform. So that was still kind of a new practice for us. And when it first came out, we were using test management prior to that, which was kind of the legacy version of of what ATF. then became. And then we started. using that kind of in tandem with, with other tools like Playwright, Selenium, Tosca. I think the, when you were migrating, it was the London release and testmanagement2.o was released at the same. year. Right? Yeah. Yep. Yeah. Right. Correct. It was the test management two point o and the London release. That's special place for sure. Yes. It is special. Right? Okay. And, you know, one, other question which I have is, usually, when I am working with certain consultants or certain teams, they usually say, you know, we have the ATF coverage. What do you think or what do you think as an architect when they say that, you know, we have all the idea of coverage and everything? So I think for me, the biggest portion was and it's funny. This infographic that we have up actually kinda describes that full life cycle, and it's been more specifically where ATF and other tools are complementing each other. Right? So ATF being the kind of very granular version of that, of course, when it first came out covering the large scale enhancements and the large scale regression testing and some of the more application features. For us though, I think the struggle was primarily in how do we maintain those at a long term? How do we maintain them from a customization perspective for sure? And then also, how do we keep the more smaller changes, from breaking, you know, at an equal blast radius as the larger upgrades and upgrade kind of mid term upgrade cycles like the security patching and everything. Because we had a lot of federated developers that, of course, is very difficult to produce ETF scripts that are really as proactive as they need to be, and then more kind of intentionally, and we can also cover the customizations that were difficult to manage in ETF kind of as an overall. And that's that's kind of our primary focus was figuring out. For me, it it signals that they've done a lot of testing and you're probably doing a lot of regression testing, but you're also probably doing a lot of work to maintain those over time. So in other words, what, I hear from you is, you know, there are a lot of, configuration changes happening, lot of enhancements happening. Right? And then people are not able to keep up with those, and that is, where the ATF kind of breaks in between. And, you're not able to catch those small changes, those configurations. Right? That's exactly right. Yeah. And I I think we've got a pull up as well. I'd be interested. to kinda hear from the audience on. some of that. But as we're answering those, I think a big portion to answer your question is that's exactly right. Those smaller changes can be equally or if not, you know, bigger problem because they do create silent risk within the platform that have the you know, can have a large effect on the overall health, which is why we have so many practices around CICD and federated development and, you know, independent changes within the platform itself from a at a global scale. Okay. So I think that is, the question which is on the poll. Right? If you upgraded this weekend, what share of critical business processes have automated regression test you addressed? Kind of interesting questions for the audience. Right? So let's see what they have to say. No. Certainly. Can we share I the report? Yeah. yeah. And I think it'd be really interesting as well, one of the topics here of of what is regression, what is automated, is a big portion of that as well. And I think that's kind of a changing landscape. Right? The the word automated testing is kind of a been a hot topic, I think, for a lot of our conversations of what is true automation testing look look like is that we trigger it once and then we rebuild it in the next cycle, or is it we build it once and then it stays forever. Right? And that's kind of our definition of of a true automated test is we can leave it on the weekend. We can let it run. Yeah. We can trust the output of that. Right? Yeah. I think that's the dream. Right? Okay. And can we post the results here quickly? Yeah. I'd love to see those. Claudia? Jumping on stage very quickly, guys, to let you know that while we have this slide, we're not able to share the, poll. results screen, but I've just popped them in the chat and also in the. in the chat. I can see a few I don't know responses, actually, but I'm wondering if anyone else has any any different point of view here. But yeah. This is this seems to be the consensus. I think that's the honest answer. That's what we have. Right? We are not really sure what would happen. Will those give us the correct result or not? Having said that, I think we should move up to the second poll. So think about your last unplanned production break. What caused it? Okay. So I think, a lot of times, your production breaks for some reason. Right? And, you know, people are not able to access, the service. No instance. And it has been a few times for me. I have really great stories. But, what about you, Mike? Have you seen this happening? Production is down. Yeah, certainly. And I think, I think the, the, if I can tell a horror story for a moment, I think the biggest change in a production instance that I ever saw was a workflow breakage caused by kind of some independent developers that were working on some of the smaller, ITSM, catalog flows for ticketing and user assignment. It ended up causing a huge catastrophic break in the overall workflows of the platform and caused all of user management to go down for six hours. Right. So, you know but I would always say see when, whenever the production break, it's always a customization on some configuration. Right? Because out of the box platform is working really great. Right? And then the things mess up when we have those Yes. customizations. Right? So it is important to track those customizations. Okay. I is. think Claudia would have yeah. Sorry. Yeah. No. No. You're fine. I think I think you're you're spot on. Customizations are kind of a touchy point, especially for a lot of developers. Right? There's always a business need for customizations. We've all wanna be on, you know, the golden image out of the box. But I think the reality, especially in, like, enterprise market, is not that. We need a lot of customizations. We need a lot of modifications to the platform that kinda fit our business needs, and that that poses its own kind of special risk in its own little bucket. Right. Amazing. I think, I'm not able to see the results on the chat even. Can you see the results? Or, Claudia, maybe you'll have to help us out again. Welcome worries. back, Claudia. Maybe we're having couple of issues. You see me a lot more a lot more than me than than normal on this. But that's all good. So I can see that configuration changes that nobody flagged as risky is really leading the charge there. And then following closely, it's like normal enhancement or change requests. And then I would say the least risky are the integration and third party changes or not determining the cause. So, Right. yeah, what we're seeing right now. Thank you. so much, Claudia. I I think, one thing to note here, Mike, is there was no problem. There were a few problems during the platform upgrade, but others are only the customizations and configurations. So that is the part, I think, which we should focus on. Yes. No. I absolutely agree. And I think that kinda leads us pretty cleanly into let's show a little bit of, you know, Right. the platform and kind of what we're. holding, what we're finding, and talk, through kind of these four domains. right. So, I think, since we are on this slide, I have a question for you again. Sure. So if you inherited an instance just now, okay, and had one week to fix that and all the untested risks, where do you start? So I think that's a great question. And let me kinda lead in if we don't mind handing over. I think I wanna show some of the demo for this portion to kinda lead through some of the report and scoring. If we could go ahead and allow me to do that. Thank you. Yeah. Here we go. Perfect. So I think the first kind of portion that I would go for and for us now, Caputo is essentially producing kind of this this that same question. Right? We wanted to look at this from top down and and from my time kind of working with the platform. The initial question is, where do you start? And I think the common answer amongst most people when you ask any platform owner or any architect that is, I don't have enough data to know. So we wanted to capture that that kind of mentality of we don't have enough data because I think there's a lot of surface down reports. There's a lot of different admin tools that are out there. And we kinda wanted to capsulize that into what we're calling an IQ report, which is essentially and I've already pre ran it here just so everyone everyone knows against kind of our instance internally. But this is really the idea of encapsulating the entire snapshot, the entire full picture of that instance going fresh into it. And even if I'm not familiar with the instance, I think pretty much everyone or at least every platform number generally knows kind of what to look for, but they don't necessarily have all of the data that they need. And that's kinda where we go into the IQ analysis. The IQ analysis, when we run this, will pull all of the information, all of the metadata from your instance and gather a lot of that foundational data for you and get some of that fundamental, you know, kind of, individual pieces. The place that I would start typically is gonna be looking more specifically at the dependencies of the actual instance. And this is gonna be more of the, you know, how many fields are dependent, are there any that are custom, As well as looking at the customization ratios for that actual instance. Right? So as you can see in the top, we have this idea of customization rate looking at the overall instance, how customized is that instance, and then this report specifically targets those areas around dependency, integrations, upgrade coverage, plugins and applications, and so forth. Right. So, Mike, what we are showing on the screen, it's it's the Copado. Right? Correct. Yeah. This is the Capata. platform. So just for everyone's information, guys, we are looking not at a ServiceNow screen. It's Copado screen, which is pulling up information from the ServiceNow platform. And like Mike was showing, it shows you how much your instance is customized and shows all kind of matrices and how much are you ready for making those changes in the future. Those upgrades and everything is visible right here. All those risks are flagged here. Right? Absolutely. Having said that, you know, I whenever I am always, involved in the upgrades, Most of the issues I used to get on portals, but nowadays, we are using workspaces. Those are the latest things. Right? So we are seeing. a lot of issues there. You know, they don't work really well. What do you think? Like, what's the reason? So I I think in my experience, one of the big troubles that we've had in in, you know, even working with other ServiceNow practitioners, the biggest question they've had is how do we how do we effectively interact with workspaces, right, is kind of the big question. I don't really know how to test it. We don't really have coverage for those. You know, it's sometimes slow even from that perspective of usability from a user perspective. That causes challenges with testing specifically. So for us, one of the big concerns around customizations was that secondary piece of how do we then also trigger on understanding the workspaces as they're built, even custom ones, for example, but more primarily the out of the box ones that come kind of baked into the instance. And then how do we understand that from a shape perspective? And I think every, you know, ServiceNow practitioner generally understands that the the ServiceNow platform is kind of a data table. It's table of tables. Right? Very interactive and ServiceNow does a very great job of interacting with that data and then presenting it in a way that's very usable. Workspaces being the newer iteration of that for sure. But it for us, the big question was how do we understand kind of what that data is, where it lives, what the shape of it is, and what interacts with it directly. And that's kinda where we go into the upgrade risk and test coverage section, of the report as well as the customizations and configurations more specifically. Going over each one of those business rules, all of the ACLs, all of the workspace load times. Right? For example, the customer health, and so forth. Right. I've see I'm seeing a question in the chat. The question is, where are these risks flagged? How can we see those particular risks? Can you show maybe an example here quickly? Yeah. Absolutely. So in the risk section of the report, when you go through this, you're gonna get a very brief overview, and that's gonna be kind of a common denominator. We do have a full explanation. I'm not gonna bore everyone with the math. I I think this is actually recorded. So if you wanna review, feel free. But we'll go through kind of the actual custom risk elements in some of those pieces. So we're we're calculating risk on a couple of places. There is a dependency graph that we generate from the instance itself that is infinitely important to the risk of the platform. Right? If you have something that's very dependent or very subdependent, has a lot of different dependencies, if you make configuration changes, for example, like we saw in the polls, even though that may not look risky on the back end of that, it's very risky. Right? There's a huge blast radius, and that's kind of where some of my horror story earlier came from. That same conversation around there was a general configuration change, but that configuration was managed by so many other processes in the environment that it became a cascading issue. And we see that across the platform. Right? So when we look at the. dependencies yeah. We do see those. when Yeah. and we need to so one of the the questions we have to answer, right, as as practitioners, especially from an architecture perspective is, do we none do we know and actually understand the depth of dependency per per field, per table, per group, per customization? And do we do we know how to kind of accurately apply that when our risk calculation is right? So, one of the thing which I'm able to see so the sys user table inbound is 107. So what's that count? So that's gonna be the inbound dependency. So this is and, of course, everyone's familiar with this as user table. Right? This has been a staple. for, sys user is the largest cross dependent field across. So the inbound. dependencies are the inbound connections directly to the sys user table. So anything that's gonna be a lookup, anything that's gonna be a, you know, a reference point or a reference field, for example, that's gonna be what the inbound, outbound children in-depth is. So depth zero, by the way, just so for everybody on the call, when you see that, if you see that right in the Caputo platform, depth zero is a we'll call that a p zero, or a high impact field. Right. It is actually here. It's quite interesting that, you know, we can see it table wise and other platform wise. So it I think this is really great. You can see with respect to the integrations, configuration. So are these different zones we are talking about? Yes. Correct. So and this is more of a holistic view. So instead of taking the zoning approach, which we've definitely, I think, tried and I'm I'm sure you've actually seen as well from some of the ServiceNow portions that are really helpful. Instead of going that direction, we went in the more holistic approach of let's kind of introspect the whole platform on itself and determine what what zones we need to create for ourselves. Right? What areas do we need to touch based off of? Is it dependent? Does it have a lot of configuration items? Does it have a lot of business scripts, for example? And then kind of putting those into their own respective testing buckets more kinda more specifically. Okay. Makes sense. Okay. One one. question which comes to my mind here is, you know, as an architect, how does your, tool compares it, with the out of the box, ServiceNow platform? So I think that's a really good question. And I think we've actually been asked a couple of times of, like, hey, does this complement? Does this compete? Does this, you know, work against. whatever? I would say I would say for us and I would say for most people, the Capata platform is a very good complement to all of those tools. Of course, every goal is to unify. Right? Everyone's goal is to kind of unify some of those functions and myself and my goal as an architect was to build a solution that would help enable developers with whatever tooling their hat to kind of meet them where they are in their journey, to give them some leverage points, right, essentially to be able to build what they need to build, how they wanna build it in in a more effective way, even that if if that is with ATF, that is with the, you know, report tool. Right. So having, so on this slide, I can see there are five questions and 1.8. So what do you think on this? So I think going through some of these, if we wanna just kinda cover. the topics, Right. I think yeah. Is it, Pick anything. Go ahead. yeah. Is it modified out of the box record rather than some something you built new? So all these questions. Right? So these are interesting. Does anything, outside service now read it or write it? Has anything changed in last ninety days? Right? Yep. So I I would think mhmm. go ahead. Yep. It's great. Yeah. I was gonna say, I think any any one of these. Right? If we if we tally these up, and and I'd be interested from a chat perspective to kinda go through this this checklist, of what you know about your instance. Right? What you know about each independent field, each independent table. Do you have these answers? Right? And I think we all kind of generally do. But, again, data is key. Data is kind of the, you know, the master of our own destiny at this point. If you wanna go through these of, you know, do you check all of these boxes when you're doing testing, when you're generating those tests? And then, you know, are you able to make those in a way that cover these consistently across upgrades, across changes in independent developers? Right. So if we want to answer these questions, I think we do have answers to these questions generally, but I think there is not not a hard core, answer. Right? And that is where we struggle. And bringing that to the next poll, which brings us to the next poll, do you have any AI enabled capabilities running in production today? Quite interesting question. Right? Everybody's chasing AI, and everything should have AI. You know, I see lot of memes on my Instagram and all. You know, everything should have an AI now. It's mandatory. Right? So let's. see what the audience has to say. No. And I I think this is gonna be a really fun pull for us as well from the topic. The I I think the AI push for sure, right, creates a new dynamic that we, as, you know, engineers haven't really had to tackle up to this point because it is so new. It is so fresh. Even service now adopting a lot more AI on the Right. testing perspective, you know, the question then becomes kind of how do we how do we guardrail the output of that. Right? How do we really gate the testing. around? Are we making sure that, you know, I can test what I changed, but do I know how to test everything AI changed? And that was a big a big question for us in our, you know, kind of in our hearts and our minds when we were building some of this as well. I wanted to jump on stage, guys, to make sure that you guys didn't miss out on the poll results, but I can see lead in the charge here is in pilot or proof of concept, which is encouraging. And then we have no plans right now. Actually, a few response responses there as well. And then a couple of nots not yet, but plan within twelve months. So that's where we're at right now. Oh, and I can see. another one that just come through, which is yes, and we test them like any other change. So there is there is someone in the audience who is who is being really zealous about this, but I think it's very mixed. I think I think, you know, I'll there are people still, discussing that we want to go to production with AI. But I think the real, question is, are you ready to go to, the production with AI? Because like you said, are there any card rails already there to protect your instance, to protect your data? That becomes really important. I think we can. move on to the next slide. Yeah. Yeah. Please. Yeah. Mike, go ahead. Yeah. And, Hardee, I just wanted to jump back because I know we were talking about kind of, you know, speed of AI, speed of innovation, and speed of testing more specifically and more importantly. I I think we actually, had one kind of live look at how the Capada platform is taking on some of that test. generation specifically if we if we wouldn't mind go ahead and transitioning to that just super quickly. Yes, please. Yes. I would love. to see that. Yeah. Let's take a look. So we also the kind of takeaway our board now. Every reporting solution is great. Every reporting solution though with a test profile for me is is more close to our heart. Right? The one question we have to ask is how quickly can we get you kind of onboarded, ready to go with tests that will cover a large portion of your environment? So I wanted to take a moment, and I'm actually just gonna generate these live, and we'll call this, you know, we'll call this demo team. I like that. Import that. So what we're doing is building a a couple of things, and it's really important to note. One of the hard portions of generating testing, and I'm sure every Selenium tester will find this, you know, a a really fun topic is generating test data for one to be able to generate a large portfolio of testing. We're doing that automatically kinda through our platform based on your instance. And then also generating a test job, which the test jobs are these large scale suites. Right? You can think of them as your playwright libraries or your selenium libraries. We're we're doing a couple of things more specifically. Now we saw the the run that takes about twenty seconds in the overall report and queue editor kind of portions of this, we're generating those tests almost instantaneously from a, you know, programmatic standpoint to cover, let's say, 40% of your instance. Right? And kind of looking at a more foundational approach of how do we get your developer started and running faster. Right? We want to meet you where you are, but we also wanna give you the tools to excel even while you're implementing. I think we had one chatter in common, and the and the comments say that, you know, they're testing AI just like they're testing any other change. And I would say that I think the largest thing is is volume at that point. How do you generate the volume of tests? And that's kinda what we're handling here is we're generating these very large scale kind of macro tests, right, essentially to provide a leverage tool for most developers to say, here's a base set of 50 tests against your instance using your test data, in the shape of your environment and some of those more critical aspects of, hey. We looked at what was dependent and we're applying them to testing so that you can feel really confident by the time you go in and generate these tests, run them live. You know, you can go to sleep at night and feel safe. Right. So I have two questions here for you, Mike. The first question is you said the data is automatically generated. So is the data actually created on the instance, or if does it stay only in. Copado? So that's a really good question. And there's there's two pieces. So the data stays in Capata. So we're taking the snapshot of the instance. And then we have a kind of opt in, opt out feature for specific data points. So if we need, like, example, ITSM records. Right? Generally, we're working in lower environments anyway, so that's not really a secure, you know, security concern from that end. Data stays on ours. It's all kind of, you know, pseudonymized. It all goes through its security practices. And then we're applying essentially that shape of your data and the shape of your instance to our testing practice, which is based out of our product framework language. Right. So, the second question which I have is these test cases you are talking about, are these the ATF test cases which are created on ServiceNow, or is Copado creating those in this platform? Great question. So now we're actually creating those in our platform. So this is a little separate from the ATF portion. I know ATF has kind of their own creation and it goes kind of very granular. Right? Generates couple thousand tests kind of per per unit, per item. What we did is we took a little bit more of a different approach in terms of one, we're using kind of a different platform portfolio from a robotic framework language, of course, you know, ATF is all written natively. And then we're also creating kind of these we're calling its keywords. Right? So these large scale kind of macro workflow tests, that you can kind of get super granular into if you want to, if you're more of a, you know, power developer, right, in most cases, or if you're more of, you know, just the general, like, QA testing, for example, and you don't really need to get that super complicated into it. You have these large scale create an incident submit. Right? So that's gonna vary. Right. So, how these test cases are generated? So based on your instance, the customizations, the dependency risk that we come up with, we're essentially taking the data out of your instance and we're applying that to a testing portfolio. So the same way a developer would go in and they would say, hey. I know I need to test every time we do a change to, let's say, a user table, for example. I need to check to make sure that certain workflows function correctly. Right? We can submit incidents. We can reference that caller field, that user field, for example. So we're taking all of that data and then we're applying it to the robotic framework language kind of as a package, to then test those overall workflows for you. Does Alright. So just to reiterate, that we all discussed here is, make Copado understands your platform, what customizations? you have done. It creates some use ADF. I can set test cases for you in Copado and then runs them and then flags the risks and the results to you. Is that correct? That is correct. Yep. Absolutely. Right. Yeah. And one quick question. When somebody is using Copado, they have, you know, they are, taken this Copado platform. They have the ServiceNow platform. Who is the user who will actually be looking after Copado? What what would be their persona? So I I would? love Yeah. yeah. And I I would I would think primarily the testers. Right? I would even say there's a home for, let's say, the portfolio owners. Right? In some cases to kinda look for overall health of testing, how much has been tested, the QA teams, for example, or even the partnership teams, right, to get an understanding. You know, if you're going into, like, we asked in the beginning of this conversation, if you're starting with a fresh environment that you've never worked in. Right? If you're the the client working on that customer environment, you just need a quick snapshot. That would also we have a home for you as, you know, in the IQ report as well to give you that kind of quick jump start on the work. Great. No. It makes sense. Yes. So, when, so since we are on this slide, right, one departure, three systems, one gap. Right? You know, whenever, we are, saying in the actual environment, we have tested everything. We are done. You know, everything is covered. We have minimum customization, minimum, configurations and all. But still, we see some of the workflows fail. Right? And having said that, the the workflows can fail, we can see that. But if the workflows are also this is a follow-up question. Okay? So sorry for two questions. No. No. But how do we know the workflow which is running behind the scenes? Is it generating the correct output we need or not? Are these two things, possible here? Yeah. No. And I think that's that's exactly what we need to be asking. Right? Is one, do we understand the workflow intent to begin with? Right? We have a workflow, we know what's in there, we know what data should be flowing through that. And those are very deterministic. Right? We know that certain things should look certain ways. Unfortunately, for us as as humans, there's a lot of it, right, that we have to kind of sift through The snapshot of that initial environment is kind of approach that approach to understanding where those workflows live, what data is in there, and then the testing portfolio is specifically designed to look for is the output as expected. Right? And those are things that you can kind of configure within your platform that we kinda handle a lot for you from a workload perspective to answer some of those questions of, hey. If this ticket is closed, is it closing the right way? Does it have the right data? Is it closing the way that we want it to? Right? Oh, perfect. So what I'm assuming is when testers are using Copado platform, the tests are automatically generated, of course. But they can go and put in, you know, what are the expected outputs we are expecting, and, they can then go and check, you know, whether it has, the workflows, they have generated the expected outputs or not. Right? Yeah. That's exactly right. And they're all. very modular. Everything's very flexible. So we've given enough wiggle room from a developer and design perspective that if you have more specific use cases, it's certainly able to be adjusted for those. But if you have kind of the more out of the box approach, of course, we're we're covering that as well. So everything in between. Perfect. So what's the most common cross system failures you see? So I would say both in it, and I'll have to agree with the polls that we had earlier. Configuration changes, of course, are probably one of the biggest failure points. Right? And that typically comes down to it doesn't look risky. Right? That's the big thing. You look at it on the surface. Everyone goes, yes. That looks okay. It doesn't look like it's got a huge blast radius. And then, unfortunately, for most people involved, it is. And then the second point I would say is is certainly cross platform integrations. I know a big part of the market. Right? Most ServiceNow practitioners are working with Workday. They're working with Salesforce. They're working with other platforms that come into the ServiceNow ecosystem that they're kind of managing through that operation either operationally or business wise. But I would say that's a big portion as well, that we wanna make sure that the output of that is captured. Right. So, you know, I understand all of this and everything. So if I want to start this, how can I do that? What's the step by step approach for this? So, if somebody wants, to understand these cross platforms, Copado, and all, how do you think. it should approach? So I think I think the first level of of kind of detail that we need in the first step is gonna be, let's look at the IQ report. Let's take a look at the actual instances. Let's pull the metadata right for context because that that I this idea of context, I think, is one that we're definitely seeing grow in the market, from the ServiceNow perspective, a couple of other things. And then understanding the dependencies and the integration points specifically within that ecosystem, what integrates with what, what data is flowing between which ones, and then how are we actually covering those from a testing perspective, to make sure that, like you said earlier, we have an output that's expected on each platform. We have an understanding of each platform at a very deep and rich level, enough so that we can cover that confidently. Right. Okay. And, can we go and see that, actually? Yeah. Absolutely. You wanna get a live look? Yep. Alright. So we're gonna pop a little back over to the integration portion specifically because we didn't I think we touched on this a little bit, but I think it's gonna be really important to cover. One of the big portions of, you know, cross platform of, you know, if you have a Workday instance, for example, feeding into your user tables, right, which I know a lot of people do, they use other platforms. You know, we used kind of a Workday portfolio as well even on top of that. We need to understand a couple of things. We need to understand the integrations that are specific to the platform. Right? These could be your OAuth integrations. They could be your Microsoft integrations or data tables. We also need to understand the the actual data points that are being generated. Right? If you have a SIS user interaction specifically with Workday, what happens when those fail? So we wanted to take an approach of looking at those independent sections of you have an HR portion, you have these dependency sections, you're able to search for those, right, if we look under the SIS user, for example. And being able to take a look at the groups, anything that routes through those, all of those dependencies, and generate tests specifically for that instance. Right? And that's kind of what we see on the robotic testing. And I think I actually shared the wrong tab, so everyone didn't see what I was just doing. But, let's go back to this. Let me go down to the IQ report. Just for everyone's information, we have been talking about Cobado Cobado. I'm just I know you would have a doubt. You see tango there. So, just for your information, tango has been renamed to Cobado. Is that right, Mike? Yes. That's correct. So, yeah, the tango is the, it's the internal kind of project name of for Caputo, working with this group. Right. So, when we are talking to you, we should we say Tango or Copado, Mike? Let's go ahead and then skip scapata for now. Okay. Perfect. So and just to cover what I was just talking about, off screen, actually, unfortunately, the the integration points that we're touching with the reports, as well as the data flowing in between those. Right? We're generating kind of two things. The data tables are gonna cover your data driven testing. Right? And this is the idea that you have, let's say, Workday feeding into your authorization system or your user accounts, for example, to feed kind of the user business information or the emails. And we're pulling those integrations from an authorization perspective, and then we're also pulling the cross dependencies and plugins for each individual thing, looking at the licensing. Right? If you have these built into a catalog or if it's an installed modular custom application, for example, that you're running out of, or even business includes. Right? And if you have business scripts that are generating some of those. And we're looking at all of those. And then in the actual test jobs, for example, when you go to run these, we have coverage for kind of all of that opportunity and all of that data flows in between. Okay. So what what if, these other teams make some configuration changes? That's a great question. And I think that goes to how do we how do we run this on a weekend. Right? It's more of the conversation. You know, I wanna go to bed over the weekend. I wanna feel confident when there's an upgrade patch. You know, patching's moving to weekly. I think we're all aware of that. Right? That's been a fairly large announcement for most people. Two things. So I think it's really important that for me as an architect and I know working at the group and our wonderful engineers here, one of the big questions was, how do we solve this problem of the regression pain for testing? Because I know I've written, you know, a couple thousand selenium scripts in my life and had to rewrite them at least a couple times a week. Oh, man. I'm sure everybody else has. Yes. How do we how do we kind of solution for some of that pain? So what we came up with was a this ideology of a resilience engine, which is kind of coupled on a couple of things. Right? We are plugging into the ServiceNow instance, and we also understand the shape of that data. So one of the big portions for us was we came up with essentially the we'll call them keywords and labels for your actual instance. And what this is is kind of a corpus library pulled from your instance in the instance history, for example, so the drift of that instance to look for things like, hey. I have the caller field that's referenced on, let's say, 60 other fields. What are the variants of those caller fields so that I don't have to go hand key those and hand look them up every time? So when we do the initial ingestion, we're pulling some of those variants as well as their independent sections and then actually using those in practice in the test scripts themselves. Right? So you don't have to manually do this. Even better, if you wanted to, you could click on the sync variables, which is gonna fetch the actual live instance for you. It's gonna pull all of those variable lists that we already generated for you and it's gonna double check them and make sure that, hey, If anything is broken, which I didn't change anything this time. But if I want to, you know, surface something has changed and I wanna know whether it's intentional or not, I can hit apply. It'll update those automatically for you so you're not having to go through. I only have six tests here, but let's say a 100, right, for example, and hand key all of those back in. Right. So one more, question on this then, from me is if ServiceNow isn't system of record for that process, can the tools still model it? Yes. Is is is the kind of brief answer, and I think that that question's got a couple of things. And I think for most of us, the system of record is kind of service now. Right? And I think this goes back to some of the ATF conversation as well. We want to kind of meet everyone where they are and give them leverage tools to make sure that we're, you know, interacting effectively with the platform itself while also maintaining that system of record kind of on top of that. Right? And using it as a central focus. But we're doing it more at a system level than we are at kind of the user level. Right? So there that takes a little bit of that out of it, so that we can be a little bit more deterministic with the testing. That's great. I think now we can switch to our another slide, which I assume is the most interesting one for an architect just like me. So I'll just wait for it to come up. Yeah. And, Annie, if you wanna pull that up. Just will you just give it a second? Yep. Here is it. Okay. Thank you. So now, you know, before we make any changes next, I don't know. And, what are the questions we should ask before the next change window? Right? If everyone on this call took a look of these five questions into their next change review, which one should it be? I think I think all of them. Right? Let's go through the checklist. I think I think the bigger question here is how much, you know, are you prepared? I think it's the right question to ask here. And all of these five kind of go into that of we know that patching is gonna be weekly. We know that these larger cycles, Australia, of course, is is still fresh in everybody's mind and I think some people are still migrating. Right. We have Brazil coming up very quickly on the horizon. I you go. through this checklist to kind of identify, are you prepared? Right? Do you have the testing portfolio that you need? Do you have the regression And how much work is it gonna be to actually update those tests going into those larger release cycles? And kind of when those tests do break, you know, how much work is that gonna be to fix? Right. I think, most of the people will have these questions. Right? So they would have, like, when when when a test breaks or to tell, real defect is from the intended change, or is it a bug or not? I think, definitely, those, questions are really, critical. We are on this slide, but before that, you know, I have a question for you, which I saved for this slide, actually. Can we connect Copado to our personal developer instances? So, you know, from my heart, I'm a developer as in I'm working as an architect. But this is my question because I if I want to test this, your Copado tool, can I connect it with my PDA? Yeah. No. I think that's a good question. And for us internally, right, at at being developers, kind of developer driven products, yes. The answer is overwhelmingly less. The conversation, of course. Right. And I think we've got kind of the the QR code up for us. You know, we'd love to chat. We'd love to get an idea of kind of what your interest is and kind of where that is. But for a PDI, absolutely. Yeah. That's part of what our internal testing looks like. Right? So, basically, if somebody wants to use it in their PDIs or anything, they can just scan the code and then they can, get in touch with you. Right? Yeah. No. I would I would love for them, and I think this is actually for us to kinda get get in touch. I'd love to, you know, start that dialogue and definitely see kind of what the interest is. Maybe get a first peek for everybody, you know. And, Annie, I'm trying not to speak too far out of turn, but maybe even trials, right, of of getting you in the platform, getting you kind of acquainted with it to get a good idea of of what we're gonna be providing from a Kibana perspective. Right. And, guys, who's ever is on the live screen right now, go and please drop your questions in the q and a session, section, and we would be happy to reply to those. But I have a few interesting questions here. So I'll start, Mike, if you don't mind. So what are the parameters. to flag these risks? I think this was a question asked at the start. So what are these parameters? How can we flag the risks, which are really important? How to prioritize the very interesting question because there could be 1,000 risks coming up. Yes. How. do you know which one to take up? Right? The age old question of how do we know what's most important with everything. is important. No. And that's a really good question. So, and I think the the big thing is sorting out kind of a couple of things. And and risk does mean different things, I think, in different environments. Right? But the the kind of broader idea of how do we identify those important risk comes down to a few things. Right? Dependencies being kind of top of mind. Right? How dependent is the thing that we're interacting with? Does it touch 12 things or does it touch the entire platform? Right? If I break this or if I accidentally if something glitches out, is it gonna break the whole platform? Is it gonna break a whole business critical process? So Caputo took kind of great effort to identify specifically those and rank them based off of dependency analysis as well as the actual data importance, to the business perspective. Right? And some of that's defined by you. Some of that's defined by the platform itself, right, from an integrity perspective. So we have the freedom to actually, move those, risks up and down, change the priority as well. Right? Yeah. And it it's bigger from our end. And, of course, we didn't get into some of the comparison. But the the question there is you can tailor that report kind of based on your own priorities. Right? We're giving you the snapshot of the system. So the system of record is saying, hey, you know, I've got some areas in x, y, and z that are important, right, from a from a health perspective. And then we're also giving you some ideas on where to touch on from a the perspective of what is important to you and your business. Right. So the next question is, does inbound includes document ID fields where potentially potentially field can point to and I think I just point to sys underscore user. I think we were talking about sys underscore user. Right? mhmm. So they're saying, the question is, does that include the document ID fields also? Yes. So and that's a really good question and we probably should touch on that. The the metadata analysis will pull out any field that you've already configured in the platform. So if you're trying to reference a specific data field or you're trying to specifically look at the configuration fields, either one of those, and we can target both of those independently. Right. Okay. So how can, the next question, how can this support teams inheriting a platform with no source and truth and disorganized development data and operational processes. How. Wonderful questions. So just like we talked. about kind of in the beginning. Right? Hey. I'm going in blind to this instance. I don't really know what's going on underneath the hood. I need a good understanding. So Caputo is certainly a resource for that. Right? We put kinda great emphasis on making sure that when we go into an environment for testing, we get that initial snapshot and that initial pull of the metadata to really, really understand what platform you're going into, how it interacts, what are the dependencies in all of those different areas so that you can get testing fairly, you know, fairly quickly within fifteen or twenty minutes. Right. So, you know, there is a question from, Paramjit, which I had written down, but I forgot to ask you. Great question, Paramjit, by the way. So is it possible to compare various service instances, dev, pre prod, staging? And if there is a way, how can we sync all of them to one source? Absolutely. So and we didn't show this either, but I think it it for time, I won't show this. But the so there's a couple of things. The report can go across any amount of ServiceNow analysis, whether that be internal tooling or your different environments, as well as even some of the, cross platform more solutioning. Right? So if you have a dev prod stage instance, for example, that you wanna test all three and look at a comparison against, we actually do generate the comparison report to tell you what plugins each one has respectively, what risk each one is showing, you know, kind of independently. And we go through all of those same metrics and compare each one in a few different ways against, you know, the primary or against each other, so that we can give you kind of a road map of what would need to be updated at that point to. Okay. I think we have answered all the questions. them I, I don't know if I've missed anything, but I think we are good there, parity. actually. Any anything, my view would like to share, about, you know, people reaching out to you and what they should expect and, any any best practices you can think of as of now you want to share with that community. Yeah. I don't think so. I'm very excited to be here, of course. And Claudia and team, thank you and Hardeep. Thank you for, you know, engaging the dialogue. I think talking about kind of from a from a community perspective, that pain that we've kind of generally felt, you know, I think over the years for sure is is infinitely important to the conversation. And I know for myself, I was very passionate about bringing a solution that would kind of remedy some of that pain, right, or at least give us the tools to be able to manage it over time. So I appreciate the time. Thank you, Mike. We have Claudia back. Hi, Yes. guys. Thank you so much for that amazing session. It's been really weird to see all the questions in the Q and A and the engagement in the chat. I can see everyone who has joined the session is still watching, which is amazing. So thank you guys for sharing all these really you. useful best practices demos. So yeah, obviously, as I said at the start, everyone will receive the recording of the session in their inbox whether they have joined or not. And please, you know, if you haven't had a chance to do so already, scan the QR code on the screen if you want to reach out to Mike Accapado if you want to get support with your ServiceNow testing. Again, thanks so much for joining. Have a great rest of your day. Thanks, Thank. you, you, everybody. everyone. Have a great day. Bye bye. Bye bye.