Building large, complex, interlinked, cyber-physical systems the lean way (#16)

"Experiment like a scientist, not a mad scientist" - Mark Richards

Have you ever pondered, "How can a lean approach work wonders for massive systems?" That’s exactly where we dive into this episode of SPCs Unleashed. Hosted by Swiss SPCTs Stephan and Niko, Dutch SPCT Ali, and Aussie SAFE Fellow Mark Richards, the show continues to unravel the complexities of enterprise solution delivery, this time with a focus on lean systems engineering.

Ali kicks off by revisiting the less glamorous but crucial core competency of lean systems engineering, which aims to trim the fat from system development by integrating and evolving systems incrementally. The discussion reveals how managing large systems can be challenging, but incremental improvements, like those from Mark's Australian Post story and Niko’s Swiss bank example, can lead to agile success.

Stephan draws parallels with his army experience, emphasizing the need for clarity and cohesion in large, complex operations. Mark and Niko share insights on navigating legacy systems and adapting roadmaps, while Ali highlights the importance of aligning long-term goals with agile, iterative development. Each story showcases how even the most intricate systems can benefit from lean principles, turning vast, monolithic challenges into manageable, incremental deliverables.

References:

Refactoring Databases: Evolutionary Database Design - Scott W. Ambler

SAFe Solution Intent Article

Cast:

Ali Hajou(Moderator)

Niko Kaintantzis

Stephan Neck

Mark Richards

Transcript

Mark Richards: 0:01

You're listening to SPCs unleashed a shaping agility project that emerged from the 2023 Prague safe summit. The show is hosted by Swiss SPC, T Stephan Nick and Niko kaintances, Dutch, spct, ali hajou and Aussie safe fellow, Mark Richards. We're committed to helping SPCs grow their impact and move beyond the foundations taught during implementing safe each week, we explore a dimension from the frameworks, competencies. We share stories about our journeys, the secrets we've found and the lessons we've learned the hard way. And welcome back to SPCs unleashed. We're continuing in the land of enterprise solution delivery today, and we have Ali hosting, and we're digging into lean systems engineering takes away Ali.

Ali Hajou: 0:49

Oh yes. Thank you very much. Hello everyone. Welcome back to yet another episode about a core competency that is not really used that much. We started this, let's say talking about this core competency last week under the watchful eye of Niko, when Niko guided us through episode 20, which was about coordinating trains and suppliers. When things become very big, typically, we need to find ways to collaborate a little bit more. So we're going to continue the core competency this episode as well, where we talk about lean systems engineering. Lean systems engineering, which is just like meet it, is trying to engineer a system with the minimum amount of fat. So in this in this core competency, and more specifically, this specific dimension, we're going to dive into the practices to align and coordinate activities necessary to specify, architect, design, implement, test, deploy, evolve, and ultimately decommission these systems in the language, nomenclature and specifically the article about this dimension, we talk about five different practices. So in this episode, we're going to make sure that these five practices are coming back as what we call our triggers. We have specified, specify the solution, incrementally, using multiple planning horizons, designing for change, frequently integrating into an end to end solution, and continuously address compliance issues and concerns. But more specifically, in this show, we're going to figure out that the bigger the system becomes, the more complex it becomes to manage it. So we'll hear of all kind of difficulties and opportunities to move away from a monolith system and thinking as seeing it as a monolith, and how to deliver in small increments. In from from stories from Mark about whatever he has experienced at Australian post, and from Niko at an a large private bank in Switzerland, that's going to be interesting. So Swiss bank and how they do things slightly differently. We're going to hear more about the difficulties of emerging long term planning from with short term planning, which we know from agile, from Stephan in the pep I'm not even going to pronounce the what P, E, P stands for. I think Stephan can do that way better than I do, and I'll briefly talk about the wrong incentive structures that are introduced, introduced by traditional project management, and how that sometimes conflict with agile teams. So I'm going to move the microphone to we're going to do that first to Niko, to talk about the true passion with respect to this dimension. Niko take it away

Nikolaos Kaintantzis: 4:16

next to eating meat, and thank you for making me hungry. Where my passion is in this dimension is something you said wrong in the beginning. It's not, it's not not used so much. It's used very much this core competency. The problem is more that people think I can use it only if I have large solutions. So using this core competency outside the large solutions, that's where my passion is, because my passion is one thing with roadmaps. So seeing that the roadmaps are some kind of a feedback loop and not just a given order. So a roadmap is one thing, and I use this the whole time, even if I have not, even if I don't have large solutions. And the other thing is, it doesn't sound funny, but it's compliance. So compliance is something that. Cool, if you do it regularly, if you do it nightly, or if you do this, integrate it so all these sleepless weeks before somebody comes and checks you, if everything is compliant, they are now not there anymore. And compliance is fun. So that's where my my passion is. Sounds dry, but it's cool.

Ali Hajou: 5:21

Compliance, being fun, additional documentation and checks being fun. Okay, I'm very curious about your stories. Thank you. Niko, moving on to Stephan, how about you? What is your passion with respect to this specific core competency

Stephan Neck: 5:41

one can't hide the background he or she has. And I've learned in the army that large and complex operations, they start with briefings on the intent. If people are not on the same page, it's pretty difficult to start planning, right? So after those briefings, we always had that was sand pit models. We really created models, walking through those models, talking about, do we really understand what to tackle? Do we understand the bigger intent, and do we have the same information coherence after walking through a model? And I tie that, and we will talk about that for sure, the solution, intent, how? How do you create clarity about the thinking and the action? That's where my passion is, nice,

Ali Hajou: 6:30

nice, nice, nice. Thank you, Stephan. And of course, we have Stephan talking or giving us good examples and analogies from the army you know never, never feels too impressed. Thank you, Stephan, how about you mark?

Mark Richards: 6:46

Well, I've got to jump in and say, if you listen to any chef, they'll say fat is flavor, so we

Stephan Neck: 6:56

need some fat

Mark Richards: 7:00

now. Look, to be honest, I'm I'm with Niko when it comes to my passion. Here it's, you know, if you listen to me last week, you you heard me say, Whatever you do, don't use a solution trade unless you absolutely, completely need it. But I think when you look at the ideas and the tools that are in the toolbox of lean systems engineering. It's one of the first places you go to beyond essential safe. There's so much rich thinking there that you could literally, you don't even need to be a whole release trainer. You just be a team. And you benefit from reaching into the toolbox of lean systems engineering.

Ali Hajou: 7:37

Yeah, yeah. Indeed, it's, it's, I think what we're going to cover quite a bit is that the, let's say the nuggets of knowledge in specifically this dimension, they extend far beyond this notion that we only need it whenever we are very large, whenever we have Huge systems, whenever an endless amount of agile release trains. As a matter of fact, the nuggets of knowledge apply pretty much everywhere at all times, a waste of collaboration. And I think for me, that's, that's where that's, that's where my true passion is. It is in the end, you know, we have smart people working together, and they're creating very complex, interesting systems that are helping us in day to day life. And we need to find a way to have them be able to just do their work, but built in sort of a forced check, if I can call it like that, that we're still, still in line with each other on a two weekly basis. So we're going going TO to have, I think, quite a lot of really interesting examples and stories from from our audience today. So I'll, I'll just directly move into our first trigger. The trigger that we've, that we've prepared nicely, is, is quite, quite focused on examples. Just so really interested to hear your stories. Trigger number one is that as an SPC, you're going to encounter a lot of obstacles that inhibit technical agility. So I mean, we're talking about team agility. We've done that before, but we're also talking about technical agility, so making sure that the technology, the system, the whatever you built, also is agile. So agility, agility is more than just replanning. Is the ability to deliver increments. It's the ability to deliver increments of the system and get feedback as soon as possible. So the first question that I'm going to ask will go to Niko, and the question. Days, do you have any example of a monolith system that eventually had to be built by agile teams and had to be, let's say, built in small increments, small increments that are observe, observable and that generated verifiable releases? Niko, would you be able to give us an example?

Nikolaos Kaintantzis: 10:22

The one is for the for the private bank you mentioned before. And just while listening, just a second, just came up, popped up in my brain. The one from the bank was, I was really a young developer these days, and we really had to replace the host. Was those days when you have to replace a PL two system with something modern, in our case, was Java. Because I was a Java developer, what we did there was really in every step people needed, the old host tried to step in with the new solution and defining clear interfaces, because the old host were still working while the new software only had parts of it, and you had to take care that things have been done twice. So they went to the old system, but also partly wise to the new system via connections to the old system. And so you tried to to to make clearly handovers handshakes in every step, or define these handshakes and also introduce them into the old host. Was also fun to see. We touch also the old system by replacing a new one. And it was really a cool endeavor. Wasn't a large solution, because the bank was a small bank founded by smugglers, by the way, great history, yeah, private banks now. So this was really cool to be part of it, because it was smooth for the for the people using it. And from the outside, it looked like two years. Nothing really happened, because the system, the new system, just were attached to it by defined interfaces. And at some point, the new system was there and the old was more kind of the backup. And at the end, the new system was there, and with many, many replaceable components, comparing to the old yeah monolith, as the PO two hosts were programmed. So a really cool story to be part of it. And this was also replicated, and in my in my next mandate, which we also had to replace the system. And this was much more difficult because we weren't allowed to touch the old system, because the compiler wouldn't work if it touched it. This was really a computer not not a network anymore, because they knew if they get got a patch, it will not be able to run the compiler again. So it was really a computer running under the desk of the last guy who was able to do po two. So there we had to do different things. But also the new software was a lot of small components. Also an international standard for data exchange was also integrated. So it's really many, many applications in a big one at the end

Ali Hajou: 12:57

the building a large system with small increments, and more specifically, trying to not really touch the existing thing. Because my, oh, my, if that stops working, then the company stops working, I guess, exactly.

Nikolaos Kaintantzis: 13:17

And this was in the area before the micro services, where modern it was really a kind of way to developing small, small, little solutions and together a big solution.

Ali Hajou: 13:27

Nice, nice, nice. Thank you. Thank you. Niko, bridging it to bridging this to the story of another, let's say, player in critical infrastructure. Mark,

Mark Richards: 13:39

well, I'm actually, I'm going to take a slight detour. I haven't inspired listening to you guys, because I think that notion of delivering things incrementally, so much is about creativity. And I listened to Niko and I had flash tax to my first Agile Project 25 years ago. It was very similar, but I also thought, what's an increment? An increment is something that gives you real feedback. And I know the first hardware customer that I worked deeply with, and you know, there was a lot of what's an increment there, and one of the things that became very clear to them was an increment was a version of the product that could generate a design win, right? And rethinking what an increment could be, allowed them to have laser focus, that allowed them to pull out a lot of the big, long lead times when they went to mass manufacturing of things. But it was very much something that generated feedback. So I think sometimes it's just about creativity. But yeah, the biggest story for me was Australia Post years ago, not 25 years ago. It's probably 10 years ago. They were rebuilding their main website for interaction with consumers, right? So it was this whole, how does the postal organization be relevant in the digital age initiative? Safe and building this, this very rich web interface, but the back end to all the important stuff for it was SAP, and SAP only deployed every six months. And so when they got this thing going, of course, it was, well, we can't update our website more than every six months, because to update the website, you had to light up to an SAP deployment. And initially that was kind of okay, because they had a lot of stuff to build before they put the new version of it live. But once they got past that hurdle, I was like, That's ridiculous. We can't have a website that can only change every six months. And so this is actually before, you know, safe introduced the idea of streamlets, but it was very much that strategy. They went, you know what? We've got changes which are just cosmetic, no impact to SAP what about we consider the fact that we can go actually quite regularly with web updates. And they had a pretty mature CICD set up. So they started to split and go, Okay, we've got releases which can go live web only. We can do those 20 times a day, releases that need something changed in SAP. Well, that's going to be less regular. And they worked with the SAP guys, and they convinced them to go to quarterly releases. So every pi would have a series of small changes multiple times a day, and then a big set of changes bundled together, and life was a little bit better, but they still went that's terrible, right? Because, yes, you can change things about the experience with cosmetic changes, but the truth is, we want to be able to change our true business and value offerings to our customer more regularly than once a quarter. So they had some very smart architects who started to go, well, let's put a micro service layer in, and we'll move some of our core business behavior logic up into the micro service layer, so that we get a, you know, rapid cycle with meaningful change. And so they wound up having, you know, 20 plus times a day deployment of actually, in the end, something like 70% of their, you know, behavior change for customers didn't need an SAP deployment could go out multiple times a day, and every now and again, that was something that relied on an SAP update and got stuck waiting for the big batch. So it's getting out of that habit of thinking, just because some of our stuff needs huge endeavors to put an increment in the market doesn't mean all of it has to and, you know, different situations, different techniques, but clever technical people are pretty good when you give them a challenge, yeah,

Ali Hajou: 17:36

yeah. Nice, nice. So it's, I think I've I hear some parallels in the between the story of Niko and your story mark, which is essentially, you know, a huge system can be fixed and updated and, let's say, improved in small steps, small bunches, small increments. However, provide the teams with that challenge, give them the space to develop small, little improvements or increments to the big monolith, to the big system, and just trust them to get your job done. I like it. How about you? Stephan,

Stephan Neck: 18:19

listening to mark and Niko. Let me open a third lane on that motorway. It's a core banking system. You do have the old guys right four decades being at that in that bank, working on that monolith system, even specializing. And now you have the young guys, the young devs, working on whatever it is. And Ali, you mentioned that principle number two, system thinking is not only about technicality and the practices, it's about social system. And if you look at that core banking system, it might look like a strangler approach, right? The new takes over the old. You break down the monolith into several parts, and from the outside, you look at that tree, and it's full of other plants growing around that tree. You don't see the tree anymore, but it's still a tree, and it's it's supporting all the other stuff. And what I've learned, and what I've seen in this example is the old guys not only supporting the old monolith, or part of it, they learned to become agile, like Mark said, right, you can do it in a better way. And by the way, you become a consultant for those who hang on you, like those plants going around the tree, and I think it's interesting. I like monoliths breaking it down, but not for the sake of breaking it down, because it creates something in the social system, because you still you have to produce, generate, and the. Value, as Mark said, and the context will will tell you how fast it is, because the core banking system, once or twice a year gets an update. Still the agile teams, they deploy that often, so they really can check, does it work and and the context is, is the constraint on that. So breaking down the monolith for the sake of breaking it down, is not the objective. It's still, how do we generate and deliver value in a very good way, so that we can learn as fast as possible? And as Mark said, ingenious people. They do consulting. They inform each other, and they come up with ingenious ideas how to solve the problems in the past, today and even tomorrow. Yeah,

Ali Hajou: 20:51

so, the the, the call it, the we should not let ourselves be limited by the slowest, let's say learning cycle, or, in this case, the updating every quarter, or let's say having a new release every quarter. We should find ways to be able to get feedback quicker, to deliver something quicker, exactly, to receive feedback. I think I have quite a similar experience where this was in the chip making industry, where normally designing a new chip might take up to nine months, or typically all the way at the beginning of a of one of those projects. The project is sort of designed so you have a product architect who, who identifies all the different elements that you would find into a microchip, and then splits that up into all kind of small little pieces. And every, every engineer, we get like a small little piece, what they call, which they call, a functional block to design. And, you know, they come back to each other a half a year later. So after half a year, each engineer will have its own functional block, which will be the most beautifully designed low noise amplifier or the most beautifully designed RF antenna to radio frequency antenna. But then whenever things come together, it is a huge disaster. So the idea that they came up with was, you know, what? What if we don't want to have that huge integration mess, what if we try to get feedback quicker? So they thought of a an approach where they had a three week iteration. And after three weeks they would, each engineer would, you know, add their little design into one bigger overall design that is simulatable, and then try to get feedback on the simulation so the chip would be incomplete. Every three weeks, it would be incomplete but simulatable. And I found that a fantastic idea of tackling, you know, what is usually a very long learning cycle by just breaking it down into smaller pieces. I found that fantastic. And I think I hear that coming back in the stories of all three of you.

Nikolaos Kaintantzis: 23:19

For me, it sounds like, like, like, like, having a date. So every three weeks, you come together and bring your stuff together and simulate it, and then you go apart and come, come again together.

Ali Hajou: 23:30

Yeah, yeah, to get to get to get some feedback, a three weekly date, yeah, it's like, have a date night every three weeks, it sounds pretty good. I think it's really healthy as well, to maintain a good relationship, to at least frequently, schedule in a date night.

Nikolaos Kaintantzis: 23:49

But I want more. Isn't it possible to make it faster, or is it just no not the date, not the date, the integration points. So why three weeks? Is this a hardware thing, or isn't, isn't it possible faster, daily, or whatever? I'm just wondering, sorry if I destroy the flow.

Ali Hajou: 24:08

No, no, no, that's a good one. That's a good one. In the end, you want to get feedback earlier, and one way, one ways to get feedback, is to integrate one component with another component, even if it's not a physical component, but if it's a design that you can integrate with each other, you know, it surfaces, whether you've missed anything. So it's just it provides earlier feedback by forcing integration essentially. Then, you know, it helps, it helps planning and and talking about planning these type of large systems, they typically tend to have large road maps and large milestones. So I'm wondering what your examples are of trying to, you know, work with. Small, little iterative, let's say, iterative development. So it's small, agile teams that develop in short iterations while simultaneously still having a long term roadmap. So would you be able to give an example of a system that was developed using some form of a long term roadmap, something that is, you know, takes maybe even longer than a year, but, but is developed by small, little agile teams. Mark, I'm giving the microphone to

Mark Richards: 25:33

you. Look, I've been there a bunch of times, both in hardware and software. I think probably the one that was most complex for me. It was a multi billion dollar program of work. One of the end goals of it was turning off a very long standing mainframe system. But that was not the core goal, and it was very much a, to an extent, a strangler Ivy approach. But it was five years and it was 1000 plus people, and there were a whole bunch of elements. There was a replatforming of their all their digital stuff that was going on at the same time, all kinds of things happening. And what we knew in the end was there are things where you have to signal intent a long way out, and there are some things that can change fast, and something that's that can change slow. You've got to know what's fixed and flexible, and you've got to have a view into the future. You know, sometimes we'd look, and we'd say, look, there's a real opportunity for a bunch of features on this bit of legacy kit, but we know we plan to replace it a year from now. What's the payback of doing these kind of quick fixes now, and knowing that we're going to have to, you know, we're going to be completely redeveloping that area on a new platform 12 months from now, and sometimes it's stacked up, right? Because we knew we'd learn something, we knew we'd get some benefit in the intervening time, but if you didn't know where you're going, you weren't in a position to have that conversation. And you know, the other aspect was just people might be counting on you, right? You might have something that triggers a process, if you're in hardware, something that triggers a process that takes nine months, or in my big software program, a nine month commercial procurement process. And there were things you had to know in order to trigger the procurement process, and you know, nothing was going to speed that procurement process up. It was just a fixed block. So when you start to look at those longer term roadmaps, recognize the things that are pretty fixed, the things that have hard to move lead times and the things that are flexible. And the more flexible things you've got, the more ability you have to move around and still maintain your heart commitment, because you know that quite often it's catastrophic if you move a hard commitment with a big, long lead time. But you want the resilience of having flexible things that have the benefit of being flexible, so know your lead times, know what's upstream and what's downstream. Know what drives the lead times. And then sometimes be creative about ways of unpacking it a little bit.

Nikolaos Kaintantzis: 28:33

Yeah, it sounds like we can make now detour to the solution intent, but I stopped doing detours. I completely agree, is an S, C, T or SPC. If you never heard about solution intent, it's now the time to read about it. If you never got what road maps are, please read again the idea what the road maps are. Because I'm quite shocked when I talk to Agile coaches about the road maps, they're thinking, Oh, they were evil. We are doing Agile. I know only for two weeks what is going on, and I have no idea what comes after two weeks. That's smoking pot and that's ad hoc, and it's not agile. Agile has road maps and road maps with feedback. So the big difference road maps in our world is not a plan which you have to obey. And say, this the way to go for the next 10 years, we know each day where you are. It's just a forecast, an idea which has feedbacks. So if the team teams find something out, they inform up, and you adjust the bigger road map. So there's really a cascade of road maps. And one of my stories is a family owned company who develops physical products for European government agencies or near government agencies. And in Europe, we have a lot of government a lot of countries, so each country has its own regulations, even if they have the EU so they had the really proper. To adjust to. They had their road maps. They need their road maps to commit on dates. Like Mark said, some dates are fixed. If you, if you say, I will bring you some features. You need this kind of fixed date things, but also being open to for feedback and realize which, which, which features, which capabilities to to adjust or replace, or whatever. So you need the solution intent to find out what we have, what we have agreed on, what is fixed, what is open. We need the road maps to at least have a chance to communicate with all the agencies around you. The very important topic as an SPC, please read this stuff on the web page.

Stephan Neck: 30:39

Yeah, I can really. I can Ali, I can second what Niko and Mark said, right? I'm in the automotive industry, aircraft industry, kitchen appliances, where it's not only software, it's always hardware. It's mechatronics, electronics, software, and what these guys have, and I think it's fabulous. In Europe, we call it the pep the product engineer spot test the product development process. They know more or less how much time does it do you use or need to come up with a new series, a new line of models, right? I have those five steps, like you do some basic application research, pre p then you have a definition phase, you have a concept development, you get the assurance you're on the right path. Then comes serious development, and then the serious preparation and ramp up. The conflict. For me, most of the time appears because you have some and I'm being mean now you have some agile evangelists telling them, You have to be faster, you have to be more agile. You're not agile. You have a four year plan. This is this is evil, right? And what I've learned is you come into this context or environment, I ask for the pep and I ask these guys, where are the integration points? And they can show me those integration points. And then the next question will be, have you talked to all those partners? Let's see last episode, right? How do you coordinate partners in a large solution, or in any solution, and then it becomes pretty quiet in the room, because they say, We are the program. We are the leading stream. They have to be compliant. They have to fit into this. And that's when Leon Ali comes in, right? How do you foster again, this conversation about intent, Niko mentioned it the solution intent. If you have different intents, it doesn't match, right? And you have to solve the mismatch of roadmaps. There's a huge amount of roadmaps. How do you lay them over each other and bring people together to talk about that, and then let's get rid of the blame game, because the blame game is is zero. It doesn't help in any way, right? So talking about the integration points, and I don't care if it's a pap, if it's a four year process, if it's a long term or midterm roadmap, what I'm looking for is, is there the mismatch? And now bring the people together to talk about and those are fabulous engineers. They have much more intelligence than I have. Their brain cells work 10 times faster than what mine do. Let them figure out how to solve it. And I'm with you guys, road maps, solution, intent, awesome stuff. One of the first things I ask in any transformation, and if it's not there, let's create it step by step, yeah,

Ali Hajou: 33:49

yeah. So understanding the the integration points certain certain tasks just take long Mark has mentioned that we use these milestones or these integration points from pep or from, let's say, our long term roadmap, to synchronize our developments with other departments or other trains or other suppliers. So we need that. But then in between, you'd like to maintain that flexibility in between you would like to make maintain that flexibility. Jumping on to, I think there's a nice bridge to the next, the next index trigger, which is whether the whether you have an example of a system that has been designed as such that it becomes easier to change later, so that there is, in a way, flexibility built into the system. And therefore how we call that, it's designed for change. Mark, would you be able to share an example with.

Mark Richards: 35:00

Us, yeah, look, I think for me, because, of course, my roots are all software. And if you go back to, you know, the land of the Martin Fowler, the Kent Beck, the uncle Bob, what they were driving was designed for change, instead of designing to prevent the need for change, right? So much of the techniques that they were giving us were Change is inevitable. It's not about measuring your design by whether or not it needs to change. It's about measuring your design by how easily it changes. And it was quite confronting to us, because it asked us to do things that felt very inefficient. I as a software guy, I was very honored towards back end databases. And, you know, changing data models was never fun, and so we used to get very angsty about people suggesting we make big changes to our data models. And took a lot of work to figure it out. And but once we figured it out, it was easy, but in that first little while, it's like, feels so inefficient, but then once we had it was so much easier to be nimble and responsive. And I remember when Scott Ambler, I was just talking to Eric a couple of weeks ago, Scott ambul put out a book in the late noughties on refactoring databases, and I was quite excited when it came out. And then I read it, I went, I've been doing everything he's talking about for last five years because we'd had to. And so people start out going, Oh, you can, yeah, it's easy to refactor a website. Well, no, actually, you can refactor a database. And when you start to live now in the land of, you know, cyber, physical there's been so much advance in that world of being able to do things like that. I remember my first big experience was actually teaching a leading safe to a group of buyers who built fiber networks. And I was talking a lot about set based design and some of the techniques and going, please God, be able to give me an idea that you recognize from the world where you know what you're talking about. And one of the senior guys in the room, he's like, Well, he's not wrong. We're increasingly going virtual. The amount of stuff in our switches that software driven rather than hardware driven now is ridiculous. And they were actually playing with it was some pylon which acted as a hub in the fiber network at the time that had, you know, hundreds of ins and hundreds of outs, and they were going through a whole bunch of engineering tests on it. And he shared a story. He said, Look, we had a we built a prototype, and then we got text to try working on routine maintenance for it, and they exposed the fact that it was going to be almost impossible to work on, because it was really hard to get angles, to get your tools in. And we never would have found that out if we hadn't built the prototype. But then he went on, he's like 90% of our work these days in terms of ongoing change is actually going to be software, firmware driven. And same thing. If I look at my sort of audio based cyber physical class, they love FPGAs because that lets them kick start their hardware manufacturing process while they're still in development and irrational on the product. And you just take smart people who know the technology they're working on, and you go, if we want to change this more easily in the future, and we're valuing the ability to change over the efficiency and the elegance of the initial concept. What does that change about your equations? Yeah, there's an overhead and a cost to an FPGA. There's an overhead and a cost towards some of the software techniques we built. But to be honest, the world we're living in, the pace at which technology is changing, those overheads are becoming much less obvious than they were 20 and 30 years ago.

Ali Hajou: 38:59

Yeah, it's just the, I don't know, I don't know about about your experiences, gentlemen, but this constant discussion about efficiency, you know, it's not efficient to design something right now and then change it later. You know, it's a waste of our resources. It's just, you know, it poses the idea that it needs to be perfect, perfectly designed the first time. I mean, I understand that upon delivery, whenever the product is being sold or being used and out there in the market, it needs to be, you know, of good quality, compliant and all of that stuff. But during design, I think, I think value is in inefficiency, to just do it again and fix it. How about you? How about you? Stephan, anything that you can share about designing for change.

Stephan Neck: 39:55

I love the curveball you just threw. Ali, do. It's about flow efficiency and not resource efficiency. And when it comes to design for change, let me pick from the aircraft industry, the domain of structural engineering. Structural Engineering two challenges, new materials being used now and included into the design process, into the production process. And as you mentioned, Design for Change right, usually as a flying box, and the flying box has some ingredients that also have some software in it. And now this, this change of of thinking about the product. It's a flying software box, or it's a flying box that should be designed for change. And now the structural engineers, and that's a huge shift, are no longer just suppliers of their box or some structural elements, they are integrated in as a production partner, and they start at the production thinking about the end product. And how do you integrate those who do design as early as possible, viewing from the production point, so that you are a partner and you're more closely integrated also. And you mentioned that, like prototypes, like modeling, you use data networks, and you use production logistics, right? And that means GEMBA, go where it is and also interact with those data engineers, AI, whatever it is. So this creates, all of a sudden, a different discussion. And again, in this initiative, at an early stage, we found out it's not about implementing safe, it's about the production and we started at the production line, and really had a gambar, and we found out it's not even the production line. It's a supply chain, which is now the challenge, okay? And in that initiative, the VP of supply chain is a huge player beside the CIO, the CTO, and the guy who oversees the business unit of structural engineering. And now we talk leaner structures, faster response time, the right people talking about the right stuff, right? And that comes back to also the solution, intent. What do you want to achieve is verification and validation, right? Sometimes it's more about, are we doing it the right way? Is it more feasible? Is it about practices and then also thinking from the end product, like four or five years ahead? What do we do to introduce new ideas and bring the people together? And that's the fascinating job of a change agent, right? Finding out. Where do you start with the thinking, bringing together the people, there's lots of data, there's existing production, there's existing practices, and then finding the potential how you can improve over the time. And I would see Stephan,

Ali Hajou: 43:14

that means we're introducing all kind of extra meetings, and we're introducing all kind of other inefficiencies in the system,

Stephan Neck: 43:24

yeah. But again, if you listen carefully to mark, it's, it's about, it's about the collaboration, and you have the swarming principle around the flow unit, right? And the flow unit is the product, the end product, but getting there multiple year period at the beginning, you might only have a model, and you need, you need the swarming principle. I'm not talking about resource efficiency. I'm talking about the flow

Nikolaos Kaintantzis: 43:53

efficiency. And that, by the way, was my aha moment of this episode, because the whole week I had discussions about efficiency and effectiveness and so on and so on. And I was trapped in the wrong discussion. And by you saying it's, yeah, it's about the flow efficiency, not efficiency, about about about the time, or whatever, that really opened the new discussion. Because many leaders, and if you're an SPC outside, you have these discussions, I'm sure for that I want to have Agile to be more more efficient, and then you say, oh, with agile, you're effective, not efficient. They don't get it because you're starting a very academic discussion. And now with the flow efficiency sentence, it really helps staying in the vocabulary of efficiency and saying, Yeah, we are efficient, but about flow, not about about doing the things fast, wrong? There was really a cool epiphany for me, and maybe the quote of the episode. So thank you, Stephan, for your story. Really saved my week. And I know what if I do Monday?

Ali Hajou: 44:54

Cool. Nice, nice, nice, nice. So, I mean, so how about all. Those additional activities that you need to do in order to be able to even deliver a product, so in either even to have flow activities, such as performing these huge, elaborate but extremely necessary compliancy tests, and, you know, making sure that whatever is being produced is actually safe or complies to functional safety standards. You know this is there's another, how do you call it? Sort of a marsh pit that we can jump into. But just, I would like to pick your brain very briefly, of how to do that in a context where we try to get feedback as soon as possible, where we try to build a large system incrementally. So how do we make sure that we're building something that doesn't catch a fire and at least is compliant to, you know, regulation and international legislation? Moving on to Stephan,

Stephan Neck: 46:02

I once had a lady from life science in one of my P O, P M courses, and she told us the story they learned over the time to build in automated quality assurance as early as possible, like principal general approvals as early as possible. But we just talked about design for change. If things get added, bring it in Step by Step. Bring it in incrementally, right? And she said changing the way we work with compliance and QA saved her life, because the final approval of a remedy is almost a non event. The final signature right, and she said, but that needed a shift, or a shift of paradigm. How we do QA, you don't do it at the end. You do it consistently and again, you need the road maps. You need a midterm, long term view to see what you can shift left and what has to be done spot on in each increment,

Ali Hajou: 47:07

which is extremely tough. But I need, I need more I need. I need more tips. I need. Give me some more ideas. Mark Niko, give me some more.

Mark Richards: 47:19

I'm going to jump in and I'm going to take it in a slightly different direction an awful lot. An awful lot of compliance is documentation. And I think if there's one most under consulted article in the whole of safe, it's solution, intent, particularly this notion of future state versus current state artifacts. And, you know, working a lot of organizations coming out of a project metaphor, they've got a world full of project artifacts, and they often struggle to go, you know, do we keep doing the legacy artifacts? Because the auditors need them, and this needs them, and that needs them, versus, you know, the epics and features and stories and those things while they're kind of second rate, or they write them as though they're going to use them the way they did the past ones, getting to that mindset of going, I have things that specify an intent to change the system, and things that specify the way the system is meant to behave. And that as I move from an intent to change into a delivered change I might actually use different artifacts and put my focus into the kind of living as built documentation to free me up for an awful lot of flexibility in the way that I approach my things like epics, features and stories and still satisfy auditors and quality standard auditors and CMMI compliance assessments, you can still do it all, but the prompts from solution intent to just go back and go, What is our full system? And what do we need to satisfy with our artifacts? And how do we recognize this notion of the fact that it's going to be different in a system that's continuously changing so much good guidance in there.

Nikolaos Kaintantzis: 49:09

I personally think we need to. We need the fourth episode just about solution, intent, maybe a coffee club, discussions or whatever. There's so much stories in it, and it doesn't really fit somewhere. I really love to do that, I would like to say something about compliance. What I don't get is, why do people think compliance is their enemy and the fight they have to do just because we have a shelf full of documents which we have to read and and to to fill out and forms? It's just anything, just self implied. It's not a law. It's just the companies decided that we need these binders. But take, take the compliance people, as stakeholders, as friends, try to make them their lives easy, and then your life will be easy, too. And had this experience twice, where we really ask the compliance people, what? How can we make your life easier? And it ended up with automated tests, with compliance tests, tests marked as compliance running every night. And then we had one compliance server which run a night. They built every night. In the morning, it was green. You can imagine how it looks like when even external people came to do an audit. It was so easy. Everything was green for weeks, for months, and comparing to the old old days where, the week before, the architects started running around and cleaning up all the mess and finding carpets to put the trash underneath that nobody sees it and and praying and hoping nobody gonna look in this corner of the application. Because it really sucks. So treat the compliance people as friends, as stakeholders, and you will have an easy life with compliance and ignore the old things you just wrote about yourself. And we have to do this and this and this. If there is no law, you don't have to do it. Do it differently, ask the people what they need to be compliant, and then you are compliant to Yeah, it's

Ali Hajou: 50:56

what we've known in software development forever. Do not do the tests all the way at the end, but blend testing in all shapes and forms throughout development. It also applies in the development of large systems, which is weird. We're just it's this the same simple thoughts, but then applied on large, complex, large, complex environments, interesting, yes, sort of all kind of a weird ideas pop into my mind. Okay, I'll stop right there. I'll stop right there. I'd like to sort of summarize, summarize what we've been talking about in this episode. So if you would be able to summarize the topics and the learnings with respect to especially this dimension, into just one sentence, what sentence would that be? And I'm going to give the microphone to mark

Mark Richards: 51:56

what feels inefficient when you first do it becomes extremely efficient once you master it, you can teach an old dog new tricks. Oh, nice. Can

Nikolaos Kaintantzis: 52:05

Teach Yes, thank you, yes.

Stephan Neck: 52:10

You have to teach.

Ali Hajou: 52:13

You have to teach an old dog new tricks. Nice. It's like, if you scramble the words a little bit, you'd have such a Yoda sentence. There's a lot of wisdom in this sentence. That's a nice one. How about you? Niko,

Nikolaos Kaintantzis: 52:36

old dogs, new tricks you teach. Have to no feedback. Nothing more than feedback. Yeah, feedback. Really

Ali Hajou: 52:44

nice. Adding a little bit of musical notes in the summary. Very nice. Niko, thank you. How about you? Stephan,

Stephan Neck: 52:52

I want sentences more dry. Verification and validation is based on the solution intent, and it requires a lot of transparency and discipline, and

Mark Richards: 53:01

what song would you sing that too?

Stephan Neck: 53:05

At the moment, I'm on four, I have to think. I can't imagine what that

Ali Hajou: 53:17

Michael Jackson instead of beat V and V hit do? That was pretty terrible. All right, moving on to measure and grow, because I saw that all of us had a nice one. All of us had chose a completely different mode, let's say measurements to you. Selected a different measurement, if we would like to figure out how we are scoring in terms of our lean systems engineering. I selected, I'll just start with myself. I selected with models simulations and test doubles. That's why I selected Stephan. It's something else. And why did you select yours

Stephan Neck: 54:10

based on the recent experience in some industries? Verification, validation activities? So you you perform them continuously based on your cadence. And as Niko said, it's included in your evolving definition of done, because at the end, you want the production process that creates the value. And for me, this is, this is the trigger I use quite often to foster the discussions. Mark,

Mark Richards: 54:41

it was a tough choice this week. So many of them were important. But coming back to my theme of fun, smart people who know what they're doing, and they're the best people to figure out a different way of doing it, I voted for compliance. Personnel are trained in lean, agile principles and safe and a part of the value stream if you want to build compliance. In, get your compliance. People are thinking about how to build it in, and part of the continuous conversation,

Nikolaos Kaintantzis: 55:08

oh, that's better than mine. I'm also with compliance, but I asked the developers to perform small batches, automate continuous delivery pipeline to make compliance also incremental. But I like yours now more, because it's about also train the other side, or train your partners, your stakeholders, to be on the same side. I like this, and I also hope somehow that somebody choose my second one. This was just about the road maps, and it's a feedback circle or a learning circle, but at the end, it's just a statement. Nothing to measure. You have to do it. Don't measure it. Just do it.

Ali Hajou: 55:41

But you could only make one choice, Niko, but

Nikolaos Kaintantzis: 55:45

that's a compliance. That

Mark Richards: 55:46

was an excuse for another song, because he could do, just do it to the tune of, just beat it exactly.

Ali Hajou: 55:59

Alright, then, and any you know, we typically have one additional, let's say, last part in every episode, which is, you know, are we missing some form of a measurement? And I see that actually next to me, only Stephan has filled this one in. So, Stephan, what are you missing?

Stephan Neck: 56:20

Yeah, we had similar thoughts. Ali, right? We're talking about integration points. That's probably the crucial element I'm looking for, no matter if it's a traditional approach, if it's a lean, agile approach, that's, for me, the basis to then find out, where do I start, and how do you evolve into the process. How do you foster that swarming principle around exploration, creation and delivery of a product or value? So planning and feasibility of integration points, not only planning, I've seen a lot of integration points not being feasible. If it's on a roadmap, it's it's on paper, but is it feasible? Have we talked about it, and do we evolve those integration points like Mark mentioned that? Right? That's an evolution over the time, and you get better at what you do over the time. Yeah,

Ali Hajou: 57:15

to making sure, or, I don't know, counting, no, that's not right. At least figuring out the existence of frequent integration points, just because, in the end, it's risk reduction, which nicely summarizes exactly this dimension. Because we've talked about building large, complex systems in small pieces, we've talked about trying to do that, let's say with a typically with a stringent, long term planning, but then building in flexibility in while going through that planning. And in order to do that, we need to build our system as such, so that we could change them later. So in a way, building that technical agility and integrate frequently for more risk reduction early on, interfacing any potential issues, and doing all of that with compliancy in mind, or compliancy built in, which sort of nicely summarized, summarized this entire typically very complex and often ignored. It's not the right word. How did we call it that? We typically don't really talk that much about this core competency because it's so large, complex, difficult, you know, we don't really want to have we much rather want to forget it, which is such a waste, right? There's, there's too much goodness that can help us, even if we don't have a large solution layer in place. So thanks a lot, gentlemen, for sharing your ideas and your examples. Which brings us to the end of the show. It does.

Mark Richards: 59:04

Thank you, Ali, and next week is a week off looking at our calendar, because this you know, we all know the Swiss love to have holidays, and

Nikolaos Kaintantzis: 59:17

we're just playing our money somehow. We have to spend it somehow,

Mark Richards: 59:22

at private banks founded by smugglers, obviously. But when we return in two weeks, we'll be rounding out our journey through enterprise solution delivery and talking about continuously evolving live systems. Yeah.