Author: Bernd Ruecker
MMS • Bernd Ruecker

How Did You Become An Architect? [00:46]
Michael Stiefel: Welcome to the Architects Podcast, where we discuss what it means to be an architect and how architects actually do their job. Today’s guest is Bernd Ruecker, who is the founder and chief technologist at Camunda, a company involved with process orchestration. It now has about 500 people, and your clients include banks, insurance companies, and telcos. Bernd has contributed to various open source workflow engines over the past 20 years and has written three books, and is writing a fourth. And I’ve written two books, and I know how time-consuming book writing is. It’s great to have you here on the podcast, and I’d like to start out by asking you, were you trained as an architect? How’d you become an architect? Because it’s not something you decided one morning, you woke up and said, “Today I’m going to be an architect”.
Bernd Ruecker: Yes. Thanks for having me, Michael. No, it’s probably not like that. For me, honestly, I mean, I have probably a very typical thing that I started to do things like programming with a computer super early on, when I was like 13 or 14 or whatever. And then I think one of the key moments for me was really when a friend of mine during high school basically started to start his own business. He was selling a kind of modded graphic card, it’s a very specific thing back then, but over the internet. And he had a lot of trouble because he got successful with that, and he had a lot of piles of everything, piles of hardware lying there, piles of emails lying there, piles of invoices lying there, piles of stuff lying everywhere, and basically asked me if I could help him.
And I started to, naive as I was, I basically, from the perspective I have now, I would say I started to write an ERP system for that, and that was really by accident. And I dived into that, and that was kind of coexisting with, let’s say, my interest about a kind of software architecture in a way. I tried to get books around that. I tried to understand how to structure a system, and I could try that out. And I got into Enterprise JavaBeans back then. I got into how to structure different components, how to build them that I can run them, and I wrote a system that is, really weird, still up-and-running today, to some extent.
And that taught me a lot of things about, and also taught me that I find it more interesting to understand that kind of bigger picture around that system instead of diving into one thing very, very deep, and I think that was kind of the roots of it. And I did study computer science. I did a master’s degree for software engineering, so I really focused on that in my university, as well, but I got started, I would say, even earlier than that.
Michael Stiefel: So, in other words, it was sort of an accidental discovery of necessity that got you into architecture?
Bernd Ruecker: Probably, yes. Yes.
Michael Stiefel: In other words, the mess it is and you figured out how I’m going to organize this mess that I’m in, and that led you to sort of architecture. And then you somehow, you mentioned that you were working on essentially an ERP system. Was that sort of what got you thinking about orchestration?
Bernd Ruecker: Oh yes. 100%. So I mean, part of what the system has to do is basically looking at the data address, kind of what orders do I have, what hardware do I have lying around and stuff? But at the same time, you also have to look at the processes, and that’s where we started, like the workflows, like the order fulfillment, like the returns good handling. These were basically the problems where we started, like how can we manage that process that it runs efficiently?
And also back then I started to look into that, I found it fascinating, and probably we can discuss that later on, as well, but I’m a big fan of visuals. I also was always a big fan of UML Class diagrams, for example, for structuring, and I looked into visuals for workflows. And back then that was not yet very well-developed, I would say. There were a couple of weird things happening. There was BPEL-
Michael Stiefel: I remember BPEL.
Bernd Ruecker: Okay, yes. Hopefully a lot of people forgot about that. But there was not really a good solution for these kinds of workflows, and that got me hooked into the whole question, but there should be. And even back then, there were two options I found. The one was going with super expensive enterprise software on a really complex tech stack, which was obviously not an option for that at all. And I looked at open source projects, and there were not many back then and they were super technical. But I looked into one, I got a contributor there, and I worked with that, actually, to implement the workflow within that system, basically auto-fulfillment and returns good handling. And that got me into the whole process orchestration thing, first of all, thinking about it, also in the community. Then I started giving a talk about that. That brought me into freelancing around workflow engines. And in a way that’s also the story how I co-founded a company that does process orchestration as a product. I stumbled into that, as well, if you will.
What is Process Orchestration? [05:40]
Michael Stiefel: Well, look, life, sometimes some people know where they’re going from day one and some people just stumble their way to success. But you said two things that were kind of interesting, and I want to sort of focus in on them. You mentioned process orchestration, and process orchestration is a type of distributed system, and distributed systems are becoming increasingly common in our world. We see them all over the place, but people still really do not understand when people are involved in the distributed system. And process orchestration is all about that. And why do you think people have so much trouble dealing with process orchestration?
Bernd Ruecker: It’s a very good question. To be honest-
Michael Stiefel: And you might want to define, just for our listeners who are not familiar with this, you might want to also define what process orchestration is.
Bernd Ruecker: Probably that’s a good first step. So process orchestration is basically getting order in certain steps you have to do. So you orchestrate those steps, you coordinate those steps, that’s kind of the wording behind that. And normally, then you have technology that can do that. And the way it works is you define models, like nowadays, it’s BPMN. I’m a big fan of BPMN, business process model and notation. It’s a graphical model. You define where you say, “I have those, for example, three steps I have to do in a sequence”. So I do the first step first. When that’s done, the next step is happening. And then you can run such a workflow engine or process orchestrator that can interpret those models and can run instances of that, like the orders I talked about.
And then they run through basically all those steps. And then the workflow engine makes sure that, if there are decisions on the way, there are complex patterns, like if you are waiting for too long, you should do something else or push something to a human and whatever. So that’s one thing, process orchestrators can interpret those models so you get a visual. And the second big thing is it’s also about long-running processes. So the process orchestrator can wait for things to happen, that’s very often a human, but it might also be an asynchronous answer from a system that takes longer or a response from a customer. So there are a lot of things where you might need to wait, and that’s what a process orchestrator can do. And then it’s normally like persisted and you have all the tooling around that, where you can see what’s currently happening, where are you waiting, and so on and so forth.
What Is So Difficult About Understanding Orchestration? [08:01]
Michael Stiefel: So what do people find so difficult about this?
Bernd Ruecker: That’s the question that also puzzles me for a decade now, I think. I think that there are a couple of perceptions around that. It’s a lot about perception, I think, because I started with visual models, for example. If you ask developers about visual models, very often you get a negative feeling around that. That’s true for UML Class diagrams, I never understood that. That’s also true for process models. And I make the case with UML first, because that’s probably what most people out there know, and a lot of people probably learned that in university and it’s a drawing, it’s like dead paper. Or you even had model-driven architecture quite a while back, where we tried to generate software out of these models, which never worked very well.
But I also recall when I started there was a tool, which was actually super great that where you wrote Java code and then you could just look at the code you wrote as an UML diagram, because it knows if I have a collection typed of that type, I can just make two boxes in the right arrow. So it was kind of a automatically documenting things, it was live documentation, and it worked the other way around, as well. You could add attributes there and it was adding that to the code. I found that so helpful to get an overview of what I was doing, and the same is true for these process models. They are so helpful at their executable code, so they are not a wishful thinking document, they are executable code, but they are readable by every human. And that does not actually have to be a computer scientist, it can be a business person, as well, it can be whatever. And that’s so valuable, but somehow developers are scared of these process models. That’s the first perception I see everywhere.
Michael Stiefel: I mean, some of that is sort of historical. After all, the original idea behind COBOL was that even a manager could program. I don’t know if you remember COBOL, but that was one of the first computer languages developed, and of course, that was an utter failure, from that point of view. I mean, COBOL is used all over the place from the idea that that would be obvious. I think some of the fear programmers have, and I remember from when I used some of those visual tools, is sort of the paradox of programming.
Normal engineers, or a civil engineer, can essentially spend their entire career conceptually building the same bridge over and over again. But in software, if we want to build something new, we reprogram it all from scratch, because if you want another copy of Microsoft Word, you just copy the bits and you’re done. You only undertake a programming project if there’s something new to do. But the technology that we’ve automated is based on what we knew in the past. So there’s a disconnect between what we could automate in the past, but the fact that we’re doing something new that we never did before and that the tools are not capable of doing that. And that’s the frustration I had when I used some of those visual modeling tools.
Bernd Ruecker: Yes. But I would look at it from a different lens. For me, it’s really a little bit of a different view on the same thing. I mean, you can write code in whatever, Java, Node, or C#, it’s also an abstraction. You don’t write it on whatever assembler level anymore, right? And the same is for BPMN, for example. It’s a language that’s defined exactly how it should behave, and then you express problems with that language on a different abstraction level, but it’s still general purpose. You can model whatever you want to model there. And the other thing people forget then very often is you’re not, especially if you could use good tooling, that’s probably, again, something a lot of vendors did pretty bad in the past, but if that’s flexible enough or developer-friendly enough, you can say, “There is an SPI, there is a hook where it can hook in code, if I don’t get any further with BPMN, then I simply just code it”.
But there are certain elements in BPMN which are great for expressing long-running problems, and I simply should use them because that’s what I see on a weekly basis, at least. Otherwise, people start implementing their own homegrown, bespoke workflow engines without knowing it. It’s the, “Oh, we just need that order status column in that table over there”, but we have to monitor that. We write a small schedule looking at that every day and then we do this and that, and they start to build workflow engines. I see that so often, and it always starts with, “Oh, we don’t want to have the complexity of an additional component”, but you have the complexity of that mess you wrote.
Michael Stiefel: Plus, you’re rethinking something from scratch and not getting the benefit of other people’s experience. So that raised something also interesting that makes me think about the low-code movement. So in other words, when you use a system such as this, you’re trying to minimize the amount of code that a developer has to write, which is generally a good thing.
However, when you’re trying to figure out why the system is not working, do these orchestration engines, or whatever you want to call them, get in the way of understanding what’s happening? Are they too much of a black box?
Bernd Ruecker: Let’s say the core engine itself should be a black box. I mean, even if you have the source available, you could look into that, nobody wants to do that. I mean, I recall times when I debugged into Hibernate, that’s nothing you ever want to do just because you can.
Michael Stiefel: Yes, I’ve done that. Yes.
Bernd Ruecker: So I would say overall, that it should a black box, and that’s good as it is. But it’s, again, very general purpose, so if you have a good product there, the core should work and it’s not… I mean, it’s still complex to get that right, especially at scale, for example. But what it does, from looking at it from the outside, is not that complicated. So it basically triggers certain activities, and that’s either connector, which is code, or it’s your code, so it’s normally not that big of a problem. So you still have the possibility to debug into things, for example, that you just don’t debug into the core engine, you debug into what you made out of that, basically the activities you put into the process. You have the possibility to write automated unit tests, for example. So that’s not gone because you used such an orchestrator.
Michael Stiefel: I guess what you’re sort of saying is that we write stuff on top of the operating system, and we don’t step into the operating system.
Bernd Ruecker: Yes. Yes. Yes, you’re right.
Michael Stiefel: And there are bugs in the operating system, but I guess what you’re saying is we look at them sort of as a, this system is supposed to X, it doesn’t do X. We have to call up the vendor and understand why that is the case.
Bernd Ruecker: And I mean, overall, it’s a bit like also the cloud discussion, “Oh, we can’t put that in the cloud. Is that secure?” I’m 100% sure it’s more secure than what most companies do at home.
Michael Stiefel: Yes, yes, yes. The only interesting thing about the cloud, just as a side issue, as you mentioned it, it just builds a more tempting target for someone to hack.
Bernd Ruecker: Probably, yes. Yes.
Michael Stiefel: Yes. Willie Sutton, who was a very famous bank robber, and he was asked why he robbed banks and he said, “Well, that’s where the money is”.
Bernd Ruecker: True, it makes sense. But at the same time, if you compare using a product that’s built for that-
Michael Stiefel: Obviously. Yes, yes, yes.
Bernd Ruecker: And used in a lot of organizations, it’s probably better than just code it yourself. Yes.
Michael Stiefel: Yes, yes. But again, when you’re dealing with human fear-
Orchestration vs. Event-Driven Architectures [15:44]
Bernd Ruecker: Yes. But honestly, the typical objections are not about, “Ooh, is my software still stable?” It’s very often about, “I don’t need that component”. It’s still that belief, because it’s so simple, my problem is so simple. It’s not. And probably because people are not really looking into, let’s say, the full scope of process orchestration, like on an enterprise level, that’s a different topic we should probably look into. And the one thing, and that kept me actually busy for a couple of years, discussing orchestration versus choreography, basically. The whole thing, do I want to control what’s happening, or do I want to let it evolve by pushing events around, for example?
And there was a belief for quite a while, I think it changed at that moment, but there was a belief that event-driven is so much easier to do. It’s so much more decoupled. It’s better, or probably more beautiful or modern or whatever you want to call it, architecture. And that led a lot of projects into we don’t need orchestration. Orchestration is kind of bad. They brought that together with, oh, centralized bottlenecks and whatever, and that’s 100% not true. And that’s another perception we saw there where people deliberately didn’t want to go into the orchestration route, especially out of IT, which is kind of weird.
Michael Stiefel: Yes, it is. But you’re not arguing against event-driven architectures altogether?
Bernd Ruecker: No. No. So my point is there is a place for event-driven and there is a place for orchestration. Basically, both are typically about different components or services communicating their way back to the distributed system. Because in my whatever, order fulfillment process, I need to talk to the CRM system, to the ERP system, to the stock, to the whatever, logistic system, so I need to get all those pieces together. So the question is how do those components interoperate? And with orchestration, basically I have one component that has the duty, the responsibility, of orchestrating the end-to-end process and saying, “Okay, the first thing I have to do is, for example, getting the stuff from stock or retrieving the payment”, and so on and so forth. It can control that, to some extent. With event-driven, with the choreography, you basically distribute that responsibility, saying, “There’s an event, the order just came in”, and then some component knows, okay, then I have to collect money.
And that’s an architecture question, actually, one of the core architecture questions, how do you want to distribute responsibilities amongst the components? And that’s a design choice you can make. And there are sometimes good arguments where you want to be event-driven, and there are sometimes good arguments where you want to be orchestrated. And especially if I look at these typical business processes, like order fulfillment or whatever, I want to get a new bank account, I want to get a claim settled or all these kind of business processes, then you very often, from a business perspective, you want to have somebody responsible for the end-to-end, because the customer is only happy if the parcel arrives at my door. It doesn’t look at how beautiful my events are flowing within, just if I get my stuff here or if I get my money paid out or whatever, then the customer is happy. So there is a big emphasis on the end-to-end and understanding certain KPIs (Key Performance Indicators) and bottlenecks and everything on the end-to-end level.
And because of that you want to have one owner, and because of that you should have also on the IT side probably one personal, one component being responsible for that. And if you want to help that component responsible for something, it needs to control certain aspects, like the sequence of things for example, or the SLA. What happens if the logistic provider doesn’t talk back to me within two hours? Is that okay or not? And deciding all these kind of factors, that’s an architectural decision, and that’s… To make a very long answer short, it depends on how you want to lay out the system, and in most bigger systems I see a coexistence of both event-driven and orchestration.
Michael Stiefel: Where do you see, ideally speaking, event-driven should predominate in your architecture choice?
Bernd Ruecker: Every time when the component that triggers something is not really responsible for doing it. I’ll give you one example, I like examples. I find that much easier to understand, and it’s actually one thing I discussed with Sam Newman regularly. He wrote these microservices books, and he has an example also of an account opening in his book and I have an account opening in my book, so that’s a good collision of examples. And basically, what I look at from an orchestration perspective is the whole the customer applies to the bank account, for example, is opened because that’s a responsibility of one component, one microservice, if you will, or however you want to implement that, and that should be orchestrated.
He looks at, okay, the customer needs to be put into the loyalty points bank, he needs an account there, he probably needs a whatever letter, and certain things that are not in the responsibility of that core opening process. And then events are great because then you could just push an event on the bus saying, “Hey, the customer is created”. And then it could build different components reacting to that saying, “Okay, then we do this, but if we don’t do that, nothing happens”. The bank account is still open. Your COO will not come to you basically asking you, “Why is it not on the loyalty points bank?” It’s a different responsibility, and this kind of trade-off, I think, is very important to think about.
The Real Problem Is Most Systems Are Architectural Spaghetti [21:19]
Michael Stiefel: So, if I could summarize what you said in sort of a sentence or two, and tell me what I’ve got it right about, it’s basically, from a business perspective, where the responsibility lies?
Bernd Ruecker: Yes. Interesting side note here, while this is one of the things, I think, on a conference level I discussed the most over the last years, it’s still, when I look into our customer base, it’s not the predominant problem. It might be discussed, but the predominant problem is that they have an integration spaghetti there, where systems are talking to each other in a totally uncontrolled manner and you have no way of really extracting the end-to-end business process. It’s already hard to understand it, which makes it almost impossible to adjust it, and that’s a big, big problem at the moment because there is so much change. You might want to drive customer experience, you might want to add new services, you might want to infuse AI everywhere, and they have mostly no idea how the process is currently running and they can’t touch it. And I think that’s a much bigger problem at the moment, to be honest.
The Public Does Not Understand How Fragile These Systems Are [22:26]
Michael Stiefel: Yes. Well, it’s also a big problem from the public’s point of view because they don’t understand how fragile these software systems actually are. I don’t know if you’re familiar, for example, in the United States there was this, the security vendor for some Microsoft systems had a bug, and this caused tremendous outages in the airlines and, especially Delta, took days to recover, and there’s all kinds of finger pointing going back and forth. But the public really doesn’t understand how hodgepodge or fragile these software systems are, and you seem to, when you talk about this from the IT perspective, I don’t think upper management sometimes realizes how fragile these systems are. Do you have any feeling about this one way or the other, how to explain this to people, whether upper management or the public?
Bernd Ruecker: But I think that’s, honestly, one of the, let’s say, core areas where we also look at when we talk about business processes and process orchestration, because it’s exactly that connection between business and IT. The IT side is implementing the process, okay, but the business side needs to understand the end-to-end process. And if I talk to the business, they have different language. They talk about whatever business model canvas, customer journeys, value streams, and these kind of things. We can get also the business to level where we talk about an end-to-end business process and probably business capabilities. And then we want to link that to implementation, and the process, for me, is one of the key elements to do that because that’s also what a business person understands. And then as soon as you have that running, you can even link data to that, yes.
The Architect is the Translator Between Business and IT [24:12]
Michael Stiefel: You hit on one of what I think is a key role of an architect, because the architect is the person who can talk to the business people about business and the technical people about technology. They’re sort of the interpreter of the two different worlds.
Bernd Ruecker: Yes, 100%. When you ask about architecture, what always comes to my mind first is I think there are still different levels to an architect. So I started, I would say, as a solution architect, looking at a single solution only, which is a bit different to looking on an enterprise architect or in the business architecture level. But I totally agree, especially the latter two should be able to, I like the Gregor Hohpe metaphor, actually, of the architect elevator, that you have to ride the elevator up and down, from the engine to the penthouse and you have to be able to talk to all of them.
Michael Stiefel: Which means that one of the skills of an architect is, even if because of their generalization they lose some of their technical chops, they still have to keep their intuition about what is technically right and wrong.
Bernd Ruecker: I agree, and I’ am still a big fan, I mean, I also do a lot of discussions nowadays about business value, for example, express business value, and I understand why it’s so important to do that, right? At the same time, I try to program, probably not on a daily basis anymore, but at least a weekly basis. I need to touch code very regularly to, exactly, keep that feeling what you’re talking about, so yes, 100%
Michael Stiefel: Yes, yes. Good. So you mentioned about processes in the enterprise and you wanted to get back to that. Is there anything that comes to mind when you think about that?
Bernd Ruecker: I think I touched on that briefly. So what we are currently seeing is a lot of change in organizations. That’s probably not news. AI or agentic AI, or whatever the buzzword of the day is, puts a lot of pressure, and we see organizations struggle with that. And what we also see them doing, if you look at it from an enterprise perspective, very often they try to optimize things in local areas, like in that one application or I use a, whatever, robotic process automation bot over here, I want to use that AI tool over here. But very often that’s local because that’s how the teams are siloed in that organization, very often the business teams. And then they start to do projects which, on a local level, might make sense because they have an ROI, oh, we automated that, we saved manual labor, there’s a ROI.
But zooming out, and that’s what a lot of companies miss, zooming out, looking at these end-to-end processes, it doesn’t have to be necessarily good what they are doing. Either it doesn’t have any influence on the overall end-to-end process, it can even cement certain bad behaviors, bad patterns, like making it harder to change that overall process, and it definitely not looks at improving it end-to-end. There are tons of examples. So we have seen, for example, AI to do support requests you could save if you make the overall application process much smoother, for example. And then, of course, you should invest there and not making the support better, even if that looks good on an isolated basis.
And all these kind of local optimizations, I think, are a huge problem. So we need to elevate that discussion to understand that value stream and the business process, and for me, process orchestration is kind of the key element to make that happen because it can actually look at the process and make the connection to really executable software and then orchestrate different pieces, different tasks, together. And that’s super powerful if you look at that on an enterprise level. So not so much in my local project, how can I orchestrate the process better? We have that, as well, but really look at it more holistically. And that’s what we are currently doing with quite a couple of customers, which is super interesting.
Michael Stiefel: I don’t know if you’re aware of this or not, but this is actually a broad economic phenomenon. I don’t know if you’re familiar with something called Pareto optimization, but the argument there is very much the one that you mentioned, is that if you want to optimize something and you can’t do local optimizations, you generally have to reoptimize the whole system to get an incremental optimization. And what you’re describing actually fits into a larger economic system, and you seem to say that process orchestration is a way of forcing that issue to the front.
Bernd Ruecker: Yes. And it’s a way of probably solving the riddle, at least to some extent. I mean, that’s not the only piece, but I think it’s a centerpiece. And it’s something, again, which you can make a CIO or a CEO or a COO understand because business process is not-
Michael Stiefel: A foreign language.
Bernd Ruecker: I think that’s one problem of the microservice community, for example, it’s discussed in IT, IT only. If I go to a business process, business capability level, I can talk to business, as well. And that’s a different thing because you need them. I mean, that’s actually why the business exists, so you need them, and that’s saying me as kind of an engineer by heart, you need them, of course, on board to make transformation happen.
The Technology Is Fun Trap [29:24]
Michael Stiefel: I think, in my career, I’ve had the same experience, that it’s amazing how many of the technology people think in terms only of technology and don’t understand the business reason for doing something. After all, it’s the business that’s paying, it’s the business that’s driven. I mean, yes, I’m fascinated with technology, too, but the idea is in service of a goal.
Bernd Ruecker: Yes. I totally agree. I thought about that quite a bit, as well, because for me it was kind of a transformation, a journey, I would say, personally because I was super nerdy and enthusiastic about technology for a long time.
Michael Stiefel: It’s fun. Technology is fun.
Bernd Ruecker: It is, and I loved it. I mean, we did a cross-orchestration engine that can scale on geographically-distributed data centers without adding latency, but we still have the same throughput, for example, so that’s obviously awesome and exciting.
Michael Stiefel: But you tell that to a business person and they go, “Duh”.
Bernd Ruecker: Yes, it’s like, “Duh”.
Michael Stiefel: Right.
Bernd Ruecker: On georedundancy, they might understand, but not how we did it.
Michael Stiefel: Right.
Bernd Ruecker: But I still had that progression, I would say, to understand why business value is important, and then also starting to understand how you can explain that better. Because very often it’s an indirect story you have to tell why this technology enables this, what it can do in a software application, and that will basically increase whatever customer experience, and then you can talk to them. And that’s so important, and probably, I would say, it’s a typical journey a lot of people run through, and I think it’s probably a good thing. I want to keep all those enthusiastic 20-years-old people that love technology. They are awesome, and we need them.
Architecture and Development are Two Equal Roles [31:05]
Michael Stiefel: And maybe some of them will stay that way for 40 years.
Bernd Ruecker: Yes, it’s also fine. Yes, yes, yes, true.
Michael Stiefel: I think it’s a mistake to think that everybody has to become an architect. To be a super developer is great, too.
Bernd Ruecker: It is. It is. And for me, for example, I did a very conscious decision not to go into people management. And for me, that architect, for example, is also a path you could take where you say, “I don’t want a people manage, but I want to get to a broader view”, and have kind of whatever, probably making bigger decisions, if you will.
Michael Stiefel: That must be tough running a company.
Bernd Ruecker: You have to find the right co-founder. Again, a piece of luck I had in my life.
Michael Stiefel: But also, it’s important to explain that story to the technology people, why they can’t fall in love with this piece of technology, and they have to do it maybe the old-fashioned way because that’s better for the business.
Bernd Ruecker: Yes. It’s not always easy. You probably know that there’s a blog post called Choose Boring Technology, or something like that, and that’s what it’s very often about.
How Do You Train Developers? [32:06]
Michael Stiefel: Yes. So two questions that sort of come to mind, and they’re separate questions, but I think they’re things that you’ve touched on that are important, how do you train developers? If developers have trouble understanding when to use events and choreograph and when to orchestrate and distributed systems and asynchronous systems, how do you train new developers to understand this?
Bernd Ruecker: There is, as always, nuance to that. So there are things where I just think people have to be made aware just by trying things out, and that’s, I would say, for the smaller problems, for example, mastering asynchronous communication, having probably consistency problem, like transactional problems, long-running things, those you can occasionally train by doing examples, probably let them writing things, probably review stuff, so that-
Michael Stiefel: And wonder why their system’s hanging.
Bernd Ruecker: Yes. I even personally learned the trouble of two-phase commit early on in my life in a real-life scenario. I mean, that’s something that sticks with you afterwards. So that’s okay. I would say the bit more architectural questions, the bigger, hairy ones like event-driven or orchestration, which might influence not only one communication link, but probably how you lay out the whole architecture, that’s probably not for a newbie. That’s probably something where it’s good to have some experienced people on board. We very often do coachings, but when I talk to people, it’s very often architects that already have seen something, otherwise it might be hairy.
Michael Stiefel: So sort of what you’re saying is that gradual exposure to increasingly more complex situations is basically the way people learn?
Bernd Ruecker: Yes, probably. I would say so.
The Importance of Visual Tools [33:51]
Michael Stiefel: And the other thing you talked about is sort of your love, if that’s too strong a word, but your appreciation of visual communication and visual tools.
Bernd Ruecker: Oh, I would say love is a good word.
Michael Stiefel: Okay. I think, given that people very often don’t appreciate this, and I know from the work that I’ve done with workflow and the little work that I’ve done with process orchestration, visuals are very, very important, especially with long-running processes that you can’t necessarily get in your head completely.
Bernd Ruecker: Yes. It’s even a problem that you can express very well with visuals. It’s a flow of activities, probably some different path, then there are events happening where you go somewhere else, so it’s a problem that fits well on a visual. And to be honest, I don’t get it why people are so against it. And every time, when I discuss a process with people that don’t like visuals, what they do first is take a whiteboard and draw it.
Michael Stiefel: Yes.
Bernd Ruecker: It’s like, because otherwise you can’t follow. So I personally don’t get it. And then sometimes get arguments like, “But how can I do diffing?” Yes, you can do visual diffing. I’m not sure if that’s the most important question, but you could do it. So no, I just don’t get it.
Michael Stiefel: There’s a limit to the amount of complexity the human mind can conceive of, and they’re not linear processes, necessarily, and there’s feedback. It seems almost perfect for a diagram.
Bernd Ruecker: Yes, I agree. And same thing, and probably you need different levels of abstractions, for example, you can do subprocesses or certain things that might happen you might want to handle in a different process or some things you might still want to handle in code, so you can play with that. It’s one tool in your tool belt, and if it fits, it simply should be used.
The Architect’s Questionnaire [35:39]
Michael Stiefel: Yes. So one of the things I like to do to ask all the architects that appear on the podcast is the architect’s questionnaire, which gets into the more human side of being an architect. So what’s your favorite part of being an architect?
Bernd Ruecker: I think I said that relatively in the beginning, seeing the big picture. Not going super deep in one element, but understanding the different pieces and how they are connected. Probably also how they are connected to business strategy, how they are connected to the technical implementation, so I think that’s my favorite part, I would say.
Michael Stiefel: So what is your least favorite part of being an architect?
Bernd Ruecker: To some extent, that’s probably also connected to my current role, which means I’m very often talking to a lot of companies, but always as an external. And I think that’s also true for architects in organizations, so you’re not part of the team typically. You’re very often an outsider, which I can live with, but it’s probably the one my least favorite thing of it.
Michael Stiefel: Is there anything creatively, spiritually, or emotionally about architecture or being an architect?
Bernd Ruecker: I love that question. I mean, the creativity part, I think it’s clear. I think it’s super creative. You have to very often find elegant solutions for very complex problems and make or communicate them in a way that they’re easy to understand. That’s very creative, and I love doing that. It always makes you a little bit happy if you see that this seems to be understood and works. And the spiritual or emotional element, it’s probably harder to answer, but for me it’s, again, I would say happy is kind of the thing. If you see that it’s working out, if you see that people are understanding it, if you see that whatever you draw helps people understand things or implement that, it’s just a good feeling.
Michael Stiefel: The world is a little better off incrementally. What turns you off about architecture or being an architect?
Bernd Ruecker: I would say it’s the, there is a typical perception around architects also being kind of the naysayers, the no-sayers, the governance entity, the police, if you will, which I think is not true, at least not for good architects at all, but it’s kind of the one thing very often I hear.
Michael Stiefel: Do you have any favorite technologies?
Bernd Ruecker: That’s easy. Of course, BPMN. Of course, Camunda. I contributed to a couple of other workflow engines in the past, as well. So there it’s great technology. Otherwise, I would say simply also programming. So I am a Java person, I started with Java and I like Java. I like the type languages, to be honest. I like the dependency management, I like the tooling, the ecosystems, the open source around it.
Michael Stiefel: What about architecture do you love?
Bernd Ruecker: Probably repetition. I would say breaking things down into manageable pieces, solving a complex problem. I would say that’s more or less it.
Michael Stiefel: And what about architecture do you hate?
Bernd Ruecker: Discussions. Too long discussions, too many discussions, not fruitful discussions. I know it’s part of the thing, and to some extent I like it and it makes things better, but at some point in time it’s too much.
Michael Stiefel: What profession, other than being an architect, would you like to attempt?
Bernd Ruecker: I haven’t found any so far. I mean, co-founding the company, I had a couple of roles in the past, but it always was architecture related, to be honest. I think my absolute favorite was kind of what we now call a solution engineer doing proof of concepts, like flying into the customer three days, get something to work in their environment, and leave. You don’t have to finish it, but you can draw the boxes to sketch the idea and prove that it really works, so that was always my favorite.
Michael Stiefel: Do you ever see yourself as not being an architect anymore?
Bernd Ruecker: Probably, let’s see. I mean, what I said early on, what I experienced at the moment is that I had that phase of my life that, I would say 10 years, where I was so into enthusiastic about technology. When I was at a conference I picked up something, some keyword I had to Google that, I could try it out. I was sitting in the evening in the hotel room hacking something to understand it. To some extent, that changed a little bit, probably also with family and kids and whatever, not so much time, and seeing that generation is still there doing that, probably now with AI and stuff. So I kind of foresee to probably do more mentoring than architecture actively myself, but not yet.
Michael Stiefel: And when a project is done, what do you like to hear from the clients or your team?
Bernd Ruecker: When I read the question, my first thought was, it’s not really an answer, but my first thought was, don’t make any more projects. Go into the product thinking world, that’s where you should be, actually. So a product is never done. If you do software, it should be kind of a product, it should be maintained, it should be a life cycle, it should keep doing things. So that’s one thing. But I totally get the question.
So the other thing, and that’s probably, again, because of my time in my life I’m currently, I would like to hear from them what’s the value of what we just did? What’s the story that connects it to the business value? What could we tell everybody in the company why we did what we just did, and why is this great? I think if they achieve that, for me, that would be even better than, quote unquote, just having a great technical solution.
Michael Stiefel: Well, thank you very much. I found this conversation fascinating, and I think you’ve illuminated many dark corners for people about both orchestration and how to think about architecture, so thank you very much.
Bernd Ruecker: Thanks for having me, Michael.
Mentioned:
.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.
Presentation: Are You Done Yet? Mastering Long-running Processes in Modern Architectures
MMS • Bernd Ruecker

Transcript
Ruecker: I talk about long running, not so much about exercise, actually. We want to start talking about food first, probably more enjoyable. If you want to order pizza, there are a couple of ways of ordering pizza. You probably have ordered a pizza in the past. If you live in a very small city like I do, if you order pizza, what you do is actually you call the pizza place. That’s a phone call. If I do a phone call, that’s synchronous blocking communication. Because I’m blocked, I pick up the phone, I have to wait for the other person to answer it. I’m blocked until I got my message and whatever, whatnot.
If then the person answers me, I get a direct feedback loop. Normally, that person either tells me they make my pizza or they don’t. They can reject it. I get a direct feedback. I’m also temporarily coupled to the availability of the other side. If the person is currently not available to pick up the phone, if they’re already talking on another line, they might not be able to take my call. Then it’s on me to fix that. I have to probably call them again in 5 minutes, or really stay on the line to do that. Synchronous blocking communication. What would be an alternative? I know you could probably use the app. Again, I can’t do that where I live. You could send an email. An email puts basically a queue in between. It’s asynchronous non-blocking communication, and there’s no temporal coupling.
I can send the email, even if the peer is not available, even if they take other orders. How does it make you feel if you send an email to your pizza place? Exactly that, because there is no feedback loop at all. Do they read my email? I pick up the phone to call them. It could be. It’s not a technical restriction that there is no feedback loop. They could simply answer the email saying, we got your order, and you get your pizza within whatever, 30 minutes. You can do a feedback loop again, asynchronously. It’s not really the focus of today. I have another talk also talking about that this is not the same. You can have those interaction patterns decoupled basically from the technology you’re using for doing that. Synchronous blocking communication, asynchronous non-blocking.
The most important thing is on the next slide. Even if I do that independent of email, or phone, it’s important to distinguish that the feedback loop is not the result. I’m still hungry. They told me they send a pizza, I’m probably even more hungry than before the result of the pizza. The task of pizza making is long running, so it probably goes into a queue for being baked. It goes into the oven. They hopefully take the right time to do that.
They can’t do that in a rush. Then the pizza is ready, and it needs to be delivered to me. It’s always long running, it takes time. It’s inherently there. That’s actually a pattern we see in a lot of interactions, not only for pizza, but for a lot of other things. We have a first step, that synchronous blocking, but we have an asynchronous result later on.
Could you do synchronous blocking behavior for the result, in that case? Probably not such a good idea. If you take the example not of pizza but of coffee. If you go to a small bakery and order coffee, what happens is that the person behind the counter takes your offer, takes your money, basically turns around, going for the coffee machine, presses a button, waits for the coffee to come out of that. Going back to you, give you the cup. It’s synchronous blocking. They can’t do anything else. I can’t do anything else.
We’re waiting for the coffee to get ready. If you have a queue behind you, and if you’re in a good mood to make friends, you probably order 10 coffees. It takes a while. It’s not a good idea. It’s not a good user experience here and it doesn’t scale very well. The coffee making is relatively quick compared to the pizza making and other things. It doesn’t have to be that way. There’s a great article from Gregor Hohpe. He called it, “Starbucks Doesn’t Use Two-Phase Commit.” He talked about scalable coffee making at Starbucks, where you also separate the two things. The first step is the synchronous blocking thing. I go to the counter, order, pay. Then they basically ask for my names or correlation identifier. Then they put me in a queue, saying, to the baristas, make that coffee for Bernd. Then these baristas are scaled independently.
There might be more than one, for example, doing the coffee, and then I get the coffee later on. That scales much better. That’s another thing you can recognize here, it also makes it easier to change the experience of the whole process. A lot of the fast-food chains have started to replace, not fully replaced, but replace some of the counters or the humans behind the counter with simply ordering by the app. Because that’s very easy for the first step, but not so easy for the coffee making. There’s robotics also for that. There are videos on the internet, how you can do that, but it’s not on a big scale. Normally, the baristas are still there, the coffee making itself. We want to distribute those two steps.
With that in mind, if I come back to long running, when I say long running, I don’t refer to any AI algorithm that runs for ages until I get a result. No, I’m basically simply referring to waiting. Long running for me is waiting because I have to wait for certain things, that could be human work, the human in the loop, like we just heard, because somebody has to prove something. Somebody has to decide something that are typically things, or somebody has to do something. Waiting for a response, I sent whatever inquiry to the customer, and they have to give me certain data.
They have to make their decision. They have to sign the document, whatever it is, so I have to wait for that. Both of those things are not within seconds, they can be within hours, days, or even weeks, sometimes even longer. Or I simply want to let some time pass. The pizza baking is one example, but I had a lot of other examples in the past. One of my favorites was a startup. They did a manufactured service, which was completely automated, but they wanted to make the impression to the customer that it’s like a human does it. They waited for a random time, between 10 and 50 minutes, for example, to process a response. There are also more serious examples.
Why Is Waiting a Pain?
Why is waiting a pain? It basically boils down to because we have to remember that we are waiting. It’s important to not forget about waiting. That involves persistent state. Because if I have to wait not only for seconds, but minutes, hours, days, or weeks, or a month, I have to persist it somewhere to still remember it when somebody comes back. Persistent state. Is that a problem? We have databases? We do. There are a lot of subsequent requirements, if you look at that.
For example, you have to have an understanding what you’re waiting for. You probably have to escalate if you’re waiting for too long. You have versioning problems, like if I have a process that runs for a month, and I start at like every day a couple of times, I always have processes in flux. If I want to change the process, I have to think about already running ones, and probably do something different for them than for newer ones, for example. I have to run that at scale. I want to see where I’m at, and a lot of those things.
The big question is, how do I do that? How do I solve those technical challenges without adding accidental complexity? That’s what I’m seeing, actually, quite often. I wrote a blog post, I think, more than 10 years ago, where I said, I don’t want to see any more homegrown workflow engines. Because people stumble into that, like we simply have to write a status flag in the database. Then we wait, that’s fine. Then they start, “We have to remember that we have to have a scheduler. We have to have an escalation. People want to see that.” They start adding stuff. That’s not a good idea to do.
Background
I’m working on workflow engines, process engines, orchestration engines, however you want to call them, for almost all my life, at least my professional life. I co-founded Camunda, a process orchestration company, and wrote a lot of things in the past about it. I’ve worked on a couple of different open source workflow engines as well in the past.
Workflow Engine (Customer Onboarding)
One of the components that can solve these long running issues is a workflow engine. We’re currently more going towards naming it an orchestration engine, some call it a process engine. It’s all fine. The idea is that you define workflows, which you can run instances off, and then you have all these requirements being settled. I wanted to give you a 2-minutes demo, not because I want to show the tool, that’s a nice side effect. There are other tools doing the same thing. I want to get everybody to the same page of, what is that? What’s a workflow engine? If you want to play around with that yourself, there’s a link.
It’s all on GitHub, so you can just run it yourself. What I use as an example is an onboarding process. We see that in basically every company to some extent. You want to open up a new bank account, you go through an onboarding process, as a bank. You want to have a new mobile phone contract, you go through onboarding. If you want to have new insurance contract, onboarding. It’s always the same. This is how it could look like. What I’m using here, it’s called BPMN, it’s an ISO standard, how to define those processes.
You do that graphically. In the background, it’s simply an XML file basically describing that. It’s standardized, ISO standard. That’s not a proprietary thing here. Then you can do things like, I score the customer, then I approve the order. That’s a manual thing. I always like adding things live with the risk of breaking down. We could say, that takes too long. We want to basically escalate that. Let’s just say, escalate. Yes, we keep it like that. We have to say what too long is. That’s a duration with a period, time, 10 seconds should be enough for a person to review it. I just save that.
What I have in the background is a Java application, in this case. It doesn’t have to be Java, but I’m a Java person. It’s a Java Spring Boot application basically that connects to the workflow engine, in this case also deploys the process. Then also provides a small web UI. I can open a new bank account. I don’t even have to type in data because it does know everything. I submit the application. It triggers a REST call basically. The REST call goes into the Spring Boot application. That kicks off a process instance within the workflow engine. I’m using our SaaS service, so you have tools like Operate, where it can look into what’s going on.
There it can see that I have processes running. You see the versioning. I have a new version. I have that instance running. If I kick off another one, I get a second one in a second. I’m currently waiting for approval. I also already have escalated it, at the same time. Then you have tasks list, because I’m now waiting for a human, for example. I have UI stuff. I could also do that via chatbot or teams’ integration, for example. Yes, to automatic processing, please. Complete the task. Then this moves on. I’m seeing that here as well. I’m seeing that this moves on, and also sends an email. I have that one.
Process instance finish, for example. It runs through a couple of steps. Those couple of steps then basically connect to either the last two things I want to show, for example, create customer in CRM system, is, in this case, tied to a bit of Java code where it can do whatever you want to. That’s custom glue code, you simply can program it. Or if you want to send a welcome email, you already see that. That’s a pre-built connector. For example, for SendGrid, I can simply configure. That means in the background, also, my email was sent, which I can also show you hopefully here. Proof done, “Hello, QCon,” in email. We’re good.
That’s a workflow engine running in the background. We are having a workflow model. We have instances running through. We have code attached, or UIs attached to either connect to systems or to the human. Technically, I was using Camunda as a SaaS service here, and I had a Spring Boot application. Sometimes I’m being asked, ok, workflow, isn’t that for these like, I do 10 approvals a day things? No. We’re having customers running that at a huge scale. There’s a link for a blog post where we go into the thousands of process instances per second.
We run that in geographically distributed data centers in the U.S. and UK, for example, and this adds latency, but it doesn’t bring throughput down, for example. We are also working to reduce the latency of certain steps. What I’m trying to say is that that’s not only for I run five workflows a day, you can run that at a huge scale for core things.
When Do Services Need to Wait? (Technical Reasons)
So far, I looked at some business reasons why we want to wait. There are also a lot of technical reasons why you want to wait for things, why things get long running. That could be, first of all, asynchronous communication. If you send a message, you might not know when you get a message back. It might be within seconds in the happy case or milliseconds. What if not, then you have to do something. If you have a failure scenario, you don’t get a message back, you want to probably just stop where you are, and then wait for it to happen.
Then probably you can also notify an operator to resolve that. Or the peer service is not available, so especially if you go into microservices, or generally distributed systems, the peer might not be available, so you probably have to do something about it. You have to wait for that peer to become available. That’s a problem you should solve. Because otherwise, yes, you get to that. You get chain reactions, basically.
The example I always like to use is this one. If you use an airplane, you get an email invitation to check in a day before, 24 hours before that normally. Then you click a link, and you should check in. I did that for a flight actually to London. I think that was pre-pandemic, 2019, or something like that. I flew to London with Eurowings. I wanted to check in, and what it said to me was, “There was an error while sending you your boarding pass.” I couldn’t check in. That’s it. What would you do? Try it again. Yes, of course. I try it again. That’s what I did. Didn’t work. I tried it again 5 minutes later, didn’t work.
What was the next thing I did? I made a calendar entry in my Outlook, to remind me of trying it again in a couple of hours. Because there was still time. It wasn’t the next day. I just wanted to make sure not to forget to check in. That’s what I call a stateful retry. I want to retry but in a long running form, like 4 hours from now because it actually doesn’t work. It doesn’t matter because I don’t need it yet now.
The situation I envision is that, in the background, they had their web interface, they probably had a check-in microservice. They probably had some components downstream required for that to work, for example, the barcode generation, or document output management, or whatever. One of those components did fail. The barcode generation, for example, didn’t work, so they couldn’t check me in. The thing is that the more we distribute our systems into a lot of smaller services, the more we have to accept that certain parts are always broken, or that network to certain parts are always broken.
That’s the whole resiliency thing we’re discussing about. The only thing that we have to make sure, which is really important, that it doesn’t bring down our whole system. In other words, just that the 3D barcode generation, which is probably needed for my PDF boarding pass, I need to print out later, is not working, shouldn’t prevent my check-in. That’s a bad design. That’s not resilient. Because then you get a chain reaction here. The barcode generation is not working, probably not a big deal. It gets to a big deal because nobody can check in anymore. They make it my problem.
They transport the error all the way up to me, for me to resolve because I’m the last one in the chain. Everybody throws the hot potato ones further, I’m the last part in the chain as a user. That makes me responsible for the Outlook entry. The funny part about that story was really, the onwards flight, same trip from London, easyJet, “We are sorry.” Same problem, I couldn’t check in, but they give you the work instruction. They are better with that. “We’re having some technical difficulties, log on again, retry. If that doesn’t work, please try again in 5 minutes.” I like that, increase the interval. That makes a lot of sense. You could automate that probably.
The next thing, and I love that, “We do actively monitor our site. We’ll be working to resolve the issue. There’s no need to call.” It’s your problem, leave us alone. In this case, it’s very obvious because it’s facing the user. It’s an attitude I’m seeing in a lot of organizations, even internally to other services, their problem, which is, throw an error, we’re good.
The much better situation would be the check-in should probably handle that. They should check me in. They could say, you’re checked in, but we can’t issue the boarding pass right now, we’re sorry, but we send it on time. Or, you get it in the app anyway. I don’t want to print it out, don’t need a PDF. They could handle it in a much more local scope. That’s a better design. It gives you a better designed system. The responsibilities are much cleaner defined, but the thing is now you need long running capabilities within the check-in service. If you don’t have them, that’s why a lot of teams are rethrowing the error.
Otherwise, we have to keep state, we want to be stateless. That’s the other part, which I was discussing with a lot of customers over the last 5 years. The customer wants a synchronous response. They want to see a response in the website where it says you’re checked in, here’s your boarding pass, here’s the PDF, and whatever. We need that. People are used to that experience. I wouldn’t say so. If my decision as the customer is either I get a synchronous error message and have to retry myself or I get some result later on. I know what I’d pick. It’s still a better customer experience. It needs a little bit of rethink, but I find it important.
Let’s extend the example a little bit and add a bit more flavor on a couple of those things. Let’s say you’re still doing flight bookings, but maybe you also want to collect payments for it. That would make sense as a company. The payments might need credit card handling, so they want to take money from the credit card. Let’s look at that. The same thing could happen. You want to charge the credit card. The credit card service at least internally but maybe also on that level will use some SaaS whatever service in the internet. You will probably not do credit card handling yourself unless you’re very big, but normally, you use some Stripe-like mechanism to do that.
You will call an external API, REST, typically, to make the credit card charge. Then you have that availability thing. That service might not be available when you want to charge a credit card. You probably also then have the same thing, you want to charge it and want to probably wait for availability of the credit card service, because you don’t want to tell your customers, we can’t book your flight because our credit card service is currently not available. You probably want to find other ways. That’s not where it stops. It normally then goes beyond that, which is very interesting if you look into all the corner cases.
Let’s say you give up after some time, which makes sense. You don’t want to try to book the flight for tomorrow, for the next 48 hours. It does make sense. You give up at some point in time. You probably say the payment failed, and we probably can’t book your flight, or whatever it is that you do. There’s one interesting thing about distributed systems, if you do a remote call, and you get an exception out of that, you can’t differentiate those three situations. Probably the network was broken, you have not reached the service provider.
Maybe the network was great but the service provider, the thread exploded while you were doing it. It didn’t process it. Did it commit its transaction or not? You have no idea. Or everything worked fine and the response got lost in the network. You can’t know what just happened. That makes it hard in that scenario, because even if you get an exception, you might have charged the credit card, actually.
It might be a corner case, but it’s possible. Depending on what you do, you might not want to ignore it. Maybe you can. If that’s a conscious decision, that’s fine. Maybe you can’t, then you have to do something about that. You also can do that in a workflow way. You could also run monthly conciliation jobs, probably also a good solution. It always depends. If you want to do it in a workflow way, you might even have to check if it was charged and refunded, so it gets more complicated. That’s what I’m trying to say.
In order to do these kinds of things, again, embrace asynchronous thinking. Make an API that’s ready to probably not deliver a synchronous result. That’s saying, we try our best, maybe you get something in a good case, but maybe you don’t. Then, that’s HTTP codes. I like to think in HTTP codes, like 202 means we got your request, that’s the feedback loop, we got it, but the result will be later. Now you can make it long running, and that extends your options, what it can do. Speaking of that, one of the core thoughts there is also, if you make APIs like that, make it asynchronous, make it be able to handle long running.
Within your services, you’re more free to implement requirements the way you want. Let’s say you extend the whole payment thing, not only to credit cards, but probably to also have customer credits on their account. Some companies allow that. If you return goods, for example, you get credits on your account, which you can use for other things, or PayPal has that. If you get money sent via PayPal, it’s on your PayPal account, you can use that first before they deduct it from your bank account, for example. Then you could add that where you say, I first deduct credit and then I charge the credit card, and you get more options of doing that also long running. That poses interesting new problems around really consistency. For example, now we have a situation where we talk to different services, probably for credit handling, or for credit card charging.
All of them have their transactions internally, probably, but you don’t have a technical transaction spawning all of those steps. Where you say, if the credit card charging fails, I also didn’t deduct the customer credit, I just say payment failed. I need to think about these scenarios where a deducted customer credit card charge doesn’t work. I want to fail the payment. Then I have to basically rebook the customer credit. That’s, for example, also what you can do with these kinds of workflows. That’s called compensation. Where you say, I have compensating, like undo activities for activities if something failed. The only thing I’m trying to say here is, it gets more complex very quickly if you think about all the implications of distributed systems here.
Long Running Capabilities (Example)
Going back to the long running capabilities. My view on that is, you need long running capabilities to design good services, good service boundaries. That’s a technical capability you should have in your architecture. I made another example to probably also make it easier to grasp. Let’s say the booking service basically tells the payment service via method via REST call, saying, retrieve payment. I won’t discuss orchestration versus choreography, because that could be something you’re also interested in. Why doesn’t it just emit an event? Booking says, payment, retrieve payment for that flight, for example. Payment chose the credit card. Now let’s say the credit card is rejected. Service is available, but the credit card is rejected. That very often happens in scenarios where I store the credit card in my profile, it’s expired, and then it gets rejected.
Now the next question is what to do with that. Typically, a requirement could be, if the credit card is rejected, the customer can provide new details. They hopefully still book their flight. We want them to do that. They need to provide new credit card details. You can also think about other scenarios. Somewhere I have the example of GitHub subscriptions, because there, it’s a fully automated process that renews my subscription, uses my credit card. It doesn’t work, they send you an email, “Update your credit card.”
The question is where to implement that requirement. One of the typical reactions I’m seeing in a lot of scenarios is that, as a payment, we’re stateless again. We want to be simple. We can’t do that, because then we have to send the customer an email. We have to wait for the customer to update the credit card details. We have to control that whole process. It gets long running.
They understand it adds complexity, they don’t want to do that. Just hot potato forward to the booking, because the booking is long running anyway, for a couple of reasons. They also have that. They can handle that requirement better, so let’s just throw it over the fence over there. I’m seeing that very often, actually. If you make the same example with order fulfillment, or other things where it’s very clear that that component, like booking, order fulfillment has a workflow mechanism, then this happens. The problem is now you’re leaking a lot of domain concepts, out of payment into booking, because booking shouldn’t know about credit card at all. They want to get money. They want to have the payment. They shouldn’t care about the way of payment. Because that probably also changes over time, and you don’t always want to adjust the booking, just because there’s a new payment method.
It’s a better design to separate that. That’s questionable. If you go into DDD, for example, it also leaks domain language, like, credit card rejected. I don’t care, I wanted to retrieve payment. Either you got my payment or you didn’t. That’s the two results I care about as booking. You want to really put it into the payment service. That makes more sense. Then, get a proper response, like the final thing. In order to do that, you have to deal with long running requirements within payment. That’s the thing. You should make that easy for the teams to do that.
I added potentially on the slide. In such a situation, payment in 99% of the cases might be really super-fast, and could be synchronous. Then there are all these edge cases where it might not be and it’s good to be able to handle that. Then you can still design, for example, an API versus say, in the happy case I get a synchronous result. It’s not an exceptional case. It’s just one case. The other case could be, I don’t get that. I get an HTTP 202, and an asynchronous response. Make your architecture ready for that. Then you could use probably also workflows for implementing that.
Just because there’s a workflow orchestration doesn’t mean it’s a monolithic thing. I would even say, the other way round, if you have long running capabilities available in the different services you might want to do, it gets easier to put the right parts of the process in the right microservices, for example, and it’s not monolithic at all. It gets monolithic if, for example, payment doesn’t have long running capabilities, and you move that logic into the booking service, just because that booking service has the possibility to do long running. I find that important. It’s not that having orchestration, or long running capabilities adds the monolithic thing. It’s the other way round, because not all the services have them at their disposal. Normally, what they do is they push all the long running stuff towards that one service that does, and then this gets monolithic. From my perspective, having long running at the disposal for everybody avoids these, what Sam Newman once called, god services.
Successful Process Orchestration (Centers of Excellence)
Long running capabilities are essential. It makes it easier to distribute all the responsibilities correctly. Also, it makes it easier to embrace asynchronous, non-blocking stuff. You need a process orchestration capability. That’s what I’m convinced of. Otherwise, probably, I wouldn’t do it for all my life. That’s also easy to get as a team. Nowadays, that means as a service, either internally or probably also externally, to create a good architecture. I’m really convinced by that. Looking into that, how can I do that? How can I get that into the organization better? What we’re seeing very successful, all organizations I talk with that use process orchestration to a bigger extent, very successfully, they have some Center of Excellence, organizationally. They not always call it Center of Excellence. Sometimes it’s a digital enabler, or even process ninjas. It might be named very differently. That depends a little bit on company culture and things.
It’s a dedicated team within the organization that cares about long running, if you phrase it more technically, or process orchestration, process automation, these kinds of things. This is the link, https://camunda.com/process-orchestration/automation-center-of-excellence/, for a 40-page article where we collected a lot of the information about Center of Excellence: how to build them, what are best practices to design them, and so on. One of the core ideas there is, a Center of Excellence should focus on enablement, and probably providing a platform.
They should not create solutions. Because sometimes people ask me, but we did that BPM, where we had these central teams doing an ESB and very complicated technology and didn’t work. It didn’t work, because at that time, a lot of those central teams had to be involved in the solution creation. They had to build workflows. It was not possible without them. That’s a very different model nowadays. You normally have a central team that focuses on enabling others that then build the things. Enabling means probably consulting, helping them, building a community, but also providing technology where they can do that.
What I’m discussing very often within the last two or three years is, but we stopped doing central things. We want to be more autonomous. We have the teams, they should be free in their decisions. We don’t want to put too much guardrails on them. Isn’t a central CoE the path? Why do you do that? I discuss that with a lot of organizations actually. I was so happy about the Team Topologies book. That’s definitely a recommendation to look into. The core ideas are very crisp, actually. In order to be very efficient in your development, you have different types of teams. That’s the stream-aligned team that does business logic, that implements business logic, basically. They provide value. That’s very often also value streams and whatever. You want to make them as productive as possible to remove as much friction as possible so they can really provide value, provide features. In order to do that you have other types of teams.
The two important ones are the enabling team, a consulting function, like hopping through the different projects, and the platform team, providing all the technology they need, so they don’t have to figure out everything themselves. The complicated subsystem team is something we don’t focus on too much. It can be some fraud check AI thing somebody does, and then provides an internal as a service thing. You can map that very well. Our customers do that actually very well to having a Center of Excellence around process orchestration, automation, for example.
Where you say they provide the technology. In our case, that’s very often Camunda, but it could be something else. Very often, they also own adjacent tools like RPA tools, robotic process automation, and others. They provide the technology and also the enablement: project templates, and whatnot. That’s very efficient, actually. It frees the teams of figuring out that themselves, because that’s so hard. As a team, if you don’t have an idea how you build your stack, you can go into evaluation mode for two or three months, and you don’t deliver any business value there. That’s actually not new. There are a couple of recommendable blog posts out there also talking about that. One is the thing from Spotify. Spotify published about Golden Path, 2020, where they basically said, we want to have certain defined ways of building a certain solution type. If we build a customer facing web app, this is normally how we do it.
If we build a long running workflow, this is how we do it. They have these kinds of solution templates. The name is good, actually, they name it Golden Path, because it’s golden. They make it so easy to be used. They don’t force teams to use it. That’s the autonomy thing. They don’t force it upon people. They make it desirable to be used. They make it easy. It’s not your fault if it’s not working. Then it’s golden. I like the blog post, actually, I love that quote, because they found that rumor-driven development simply wasn’t scalable. “I heard they do it like that, probably you should do that as well.” Then you end up with quite a slew of technology that doesn’t work. I find this really important that you want to consolidate on certain technologies. You want to make it easy to use them across the whole organization. That makes you efficient. Don’t force it upon the people.
They also have a tool. That’s a big company, they do open source on the side. They made backstage.io. I have no idea if the tool is good. I have not used it at all. I love the starting page of their website, The Speed Paradox, where they said, “At Spotify, we’ve always believed in the speed and ingenuity that comes from having autonomous development teams, but as we learn firsthand, the faster you grow, the more fragmented and complex your software ecosystems become, and then everything slows down again.” The Standards Paradox, “By centralizing services and standardizing your tooling, Backstage streamlines your development environment.
Instead of restricting autonomy, standardization frees your engineers from infrastructure complexity.” I think that’s an important thought. They’re not alone. If you search the internet, you find a couple of other places, for example, Twilio, but also others. Same thing. We’re offering paved path, mature services, pull off the shelf, get up and running super quickly. What you do is create the incentive structure for teams to take the paved path, because it’s a lot easier. If they really have to go a different route, you make it possible. It’s not restricting autonomy, simply helping them. That’s important. I think it’s also important to discuss that internally.
Graphical Models
Last thing, graphical models. That’s the other thing I discuss regularly. Center of Excellence, yes, probably makes sense. Process orchestration, yes, I understand why we have to do that. Graphical models? We’re developers. We write code. Thing is, BPMN, that’s what I showed. It’s an ISO standard. It’s worldwide adopted. It can do pretty complex things. I just scratched the surface. It can express a lot of complex things in relatively simple model, so it’s powerful. It’s living documentation. It’s not a picture that’s requirement, but it’s running code. That’s the model you put into production. It’s running code. That’s so powerful.
This is an example where it’s used for test cases. That’s what the test case tests, for example. You can leverage that as a visual. Or it can use it in operations like, where is it stuck, or what is the typical way it’s going through, or where are typical bottlenecks, and so on? You can use that to discuss that also with different kinds of stakeholders, not only developers, but all of them.
If you discuss a complex algorithm, like a longer process or workflow, you normally go to the whiteboard and sketch it because we’re visual as a human. Just because I’m a programmer doesn’t make me less visual. I want to see it. Very powerful. It’s even more important, because I think a lot of the decisions about long running behavior needs to be elevated to the business level.
They need to understand, why we want to get asynchronous. Why this might take longer. Why we need to change, also customer experience to leverage the architecture. The only way of doing that is to really make it transparent, to make it visual. I think it was a former marketing colleague that worked with me, phrased it like that. What you’re trying to say is that in order to leverage your hipster architecture, you need to redesign the customer journey. That’s exactly that. That’s important to keep in mind.
Example (Customer Experience)
I want to quickly close that with another flight story. The first thing it’s happening, so you get everything asynchronous. They did change the customer experience a lot. Now I’m working on train companies. That’s the same thing. Mobile. You get automatically checked in for flights. You don’t even have to do that. Why should I do that? My flight to London was delayed by an hour. Ok, that’s delayed. That was canceled. That’s not so nice. Then I got a relatively quickly and automated email, that’s the only one in German, which I don’t get why. Did I get that one in German? It wasn’t German.
I got the link to book my hotel at Frankfurt airport. Why? I don’t want to get a hotel in Frankfurt, I want to get to London. Everything automated, everything pushed. Nice. Then I got, via the app not via email, a link to a chatbot where I should chat about my flight. It says, we rebooked you for tomorrow morning. It didn’t do that completely because it’s not Lufthansa, so you have to see a human colleague. I don’t want to get to London tomorrow, I want to get there today. I basically visit a counter.
The end of the story is they could rebook me to a very late flight to London, Heathrow, which was very late. I hated that. What I still like, everything was asynchronously. I got notification of everything in the app via email. I think there’s some good things on the horizon there. The customer experience for airlines at least changed quite a bit over the last 5 years. Funny enough, last anecdote, I read an article about the bad NPS score of Lufthansa, and I probably understand why.
Recap
You need long running capabilities for a lot of reasons. Process orchestration platforms, workflow engines, great technology. You should definitely use that for those, because it allows you to design better service boundaries, implement quicker, less accidental complexity. You can embrace asynchronicity better. Provide a better customer experience. We haven’t even talked about the other stuff like increased operational efficiency, automation, reduce risk, be more compliant, document the process, and so on. In order to do that successfully across the organization, you should organize some central enablement. I’m a big advocate for that, to really adopt that at scale.
See more presentations with transcripts