What is a Workflow Engine?

Workflow engines are used to simplify, expedite, and optimize all sorts of internal tasks and processes.

By codifying and digitalizing business logic, an effective workflow engine enables us to achieve a high degree of efficiency, repeatability, accuracy, and productivity within daily operations.

They’re also a powerful basis for business process automation, integration, real-time analytics, and other digital transformation efforts across your business.

Today, we’re exploring everything you need to know.

Specifically, we’re going to start by checking out what workflow engines are and what they’re used for. Then, we’re going to take a look under the hood at how they work and the kinds of solutions that are out there.

We’ll wrap up by thinking about what you’ll want to consider when it comes time to build a workflow engine of your own.

First, though, let’s think about the basics.

What is a workflow engine?

A workflow engine is a solution for centrally storing and enforcing the logic that underpins regular workflows within business operations. Depending on the scope, this could be a single task - or a whole suite of related processes.

The goal is - where possible - to replace human decision-making with computational logic.

When used as part of your automation efforts, workflow engines can then trigger and coordinate defined follow-on actions when certain decisions are made for business process management.

For example, say you used a workflow engine that’s designed to help you manage financial tasks.

Our engine might store rules so that when a user submits an expense claim, it can be automatically approved or declined based on the value, vendor, or category of the invoice.

A successful approval might then trigger other tasks and automated workflows or processes - like notifying the original user of the outcome.

So, at the most basic level, a workflow engine does a couple of things:

1. Storing computable rules to govern workflows.
2. Managing triggers and actions for automation events.
3. Facilitating user inputs and interactions.

Besides this, workflow engines have an important role to play in your integration efforts. Rather than separately configuring the same rules across multiple nodes and platforms, we can connect disparate tools to a single solution to achieve the same ends.

Before we get any deeper into how workflow engines work, it’s worth taking a step back and thinking about what we really mean by a workflow.

So…

What is a workflow?

A workflow is a defined sequence of steps that are required to complete a particular business task.

In other words, it’s made up of three components:

1. The goal we’re trying to achieve.
2. The constituent actions that we need to take to achieve this goal.
3. The logic that governs when these actions are taken.

Check out our guide to workflow analysis to learn more about how you can outline these steps and the logic underpinning them.

Strictly speaking, a workflow can concern just about anything - from basic admin tasks to top-secret research and development. That is, assuming that it’s a regular task, requiring the same or similar steps each time it’s completed.

To make things easier, we can point to a few distinct types of workflows.

These include approval and request workflows, data management tasks, numerical and analytical tasks, human interactions, compliance tasks, document/information management, and general admin.

We don’t need to worry about these in-depth for now. Rather, the point is simply to illustrate how we can think about workflows in more abstract terms - without getting bogged down in the very specific details relating to particular tasks.

But, our focus today isn’t the theory behind workflows, so let’s get back to the main topic at hand.

MIT

Where are workflow engines used?

Workflow engines are only one solution out of many for managing internal tasks. Therefore, we need to understand the scenarios where they’re most helpful - and moreover, the specific use cases where they’re appropriate.

To understand this, we need a strong picture of what alternatives are out there.

Very broadly, this includes any non-centralized solutions for implementing business logic in workflows. For instance, storing rules separately in different SaaS platforms involved within a process.

So, when do we need centralization?

Cross-platform workflows

There are several factors to consider here. First, there’s the complexity of our workflows - both in and of themselves and in terms of the technology we use to implement them.

For instance, the greater the number of distinct tools and actors involved in a workflow, the more difficult it’s going to be to consistently and manageably enforce business rules without some degree of centralization.

Following on from this, consider what happens when the business rules that underpin a workflow change - say for example that we raised the cost threshold for automatically approving expenses.

This is easier when we use a single orchestration engine to enforce business rules.

When these are stored and enforced from one central location - managing this is relatively easy - as we’ll only need to do it once, potentially without ever touching other platforms that are involved in the task at hand.

By contrast, if multiple platforms enforced this rule independently of each other - we’d need to action the relevant changes in each of these.

However, highly integrated processes aren’t the only scenario where workflow engines can provide value.

Interoperability and implementing similar logic in multiple workflows

Workflow engines are also commonly leveraged in scenarios where we have the same kinds of logic across many distinct workflows.

The best example of this is the case of request and approval workflows.

In purely computational terms, these kinds of workflows are made up of a relatively small number of elements:

• The information required for a particular request.
• The various states that a request can be in at any given time.
• The conditions that need to be met for each possible transition between states.

As we’ll see a little bit later, one of the options for implementing this kind of logic is to treat each of these elements as distinct objects within a workflow management database system.

Since we’d be handling all of our approval workflows in a single platform in this scenario, this is a comparatively cost-effective and robust approach to implementing a whole range of automation and digital transformation efforts.

Operational control

Another key impetus for implementing workflow engines is a need for a high degree of operational centralization and control - particularly as this relates to oversight from particular departments, like IT or ops.

We can contrast this with workflows that are controlled and implemented by the individual departments where they’re actioned.

To this end, operations or IT colleagues might build or configure centralized workflow engines as a more viable alternative to managing business rules within discrete tools - as their on-the-ground counterparts might do.

This offers a number of important benefits, including increased oversight, ease of maintenance, and cost-effectiveness.

McKinsey

We’ll consider each of these factors more deeply a little bit later.

How do workflow engines work?

That’s enough theory for now. It’s time to start thinking about the technical details of how workflow engines actually work.

It won’t surprise you to learn that there are a few distinct models here.

A traditional workflow engine is based on what’s known in computer science as a state machine. This is a relatively simple piece of software that stores all of the possible states a task can be in, alongside how it can transition from one state to another.

Examples of generic states might include:

• Requested.
• Approved.
• Declined.
• In-progress.
• Queried.

Or, you could have any number of other, more task-specific states.

Transitions are made up of two elements:

• Each of the possible states that a task can move to from its current state.
• The trigger that’s required to initiate this.

So, at any point in time, there might be several states that a request can transfer to next.

Triggers can be data conditions, external events, or even manual actions.

This is where a workflow engine departs from being a strict state machine - in the sense that a typical state machine would only store these transitions - without necessarily offering the capability to enact them.

To manage workflows with a state machine, we’d need some additional way to trigger transitions through external events.

A functional workflow engine provides the additional functionality to actually trigger and handle transitions.

In terms of architecture, we have a few options for achieving this.

3 types of workflow engines

We’re going to consider three distinct types of workflow engines - that can be implemented as application back-ends, process layers, or middleware.

Let’s take each in turn.

1. Workflow management databases

First, we have workflow engines that are implemented by creating dedicated databases - effectively state machines - and then using some combination of stored procedures, automations, purpose-built UIs, or third-party integrations to handle transitions.

So, we might have a set of interrelated data tables for requests, transitions, states, and any other objects that are required to implement or internal business logic at a data level.

This is an effective and comparatively cost-effective method for building a workflow engine with a high degree of centralized control.

However, the downside is that we need a large degree of database management skills in order to create, manage, and maintain these kinds of solutions.

There’s also a tradeoff in the sense that this kind of workflow engine becomes very complex when there’s a higher degree of variety between the different kinds of workflows involved.

Check out our in-depth guide to workflow management database design to learn more.

2. API-based & hard-coded solutions

We also have the option of creating fully customized solutions based on existing data sources.

So, rather than creating an entirely new database schema to store and manage business rules, this approach means connecting to existing databases that are already used within workflows and creating hard-coded scripts to enact business rules.

We’d then connect each of the relevant tools within a workflow to this via API requests.

For example, we might create a piece of middleware that monitors changes to our inventory management database and then performs defined actions when certain conditions are met using asynchronous programming - based on established business logic.

The clear upside to this approach is the huge amount of flexibility and customization available to us.

The trade-offs, however, are fairly major.

Since we’d effectively need to hard code each individual piece of business logic involved in every workflow we want to manage in this way, the relative development costs would be huge - alongside increased product development times.

3. Low-code workflow engines

In the past few years, more and more IT and operations teams are turning to low-code development in order to systematize, manage, and automate internal workflows.

The idea behind low-code is to empower specialists and non-developers alike to create custom tools, automations, and other solutions in a fraction of the time.

For building a workflow engine, this can include creating data management solutions, flow-based automations, custom API calls, user interfaces, and cross-platform integrations to enforce defined business rules.

Effectively, we can achieve the same levels of functionality, flexibility, and customization as we would with a hard-coded solution, with far reduced development costs and lead times.

Check out our in-depth guide to open-source low-code development platforms to learn more.

Workflow engines: benefits and challenges

Before we wrap up, it’s important to note that workflow engines are powerful tools - but they aren’t necessarily the right option in every context.

To get a realistic view of the key decision points and trade-offs here, it’s important to thoroughly assess the pros and cons of implementing a workflow engine to manage internal business rules.

Here goes!

Benefits of workflow engines

First, let’s think about how workflow engines provide positive value. Workflow engines can improve profitability and reduce costs by increasing:

• Efficiency - Reducing and eliminating the need for manual human actions.
• Accountability - Rationalizing responsibilities within workflows and helping to create a paper trail of how processes are adhered to.
• Accuracy - Cutting the scope for human error or actions outside of formal workflows.
• Standardization - Ensuring that the same outcomes are arrived at no matter which users are involved.
• Morale - Empowering colleagues to focus on more challenging, cognate tasks.

Of course, these are only the high-level business benefits. In more technical and IT ops terms, they also offer benefits including:

• Ease-of-maintenance - Compared to enforcing business rules in a range of disparate platforms.
• Connectivity - Facilitating cross-platform integrations and automation rules.
• Risk management - Improving our readiness to respond to issues.
• Low vendor dependence - By eliminating the need to manage business rules in third-party SaaS tools.
• Customization - We’re not limited by capabilities provided by existing platforms.
• Scalability and adaptability - Centralized solutions make it easier to respond as our needs evolve and change.
• Reducing development workloads in the longer term - Avoiding the need to configure and manage business rules in multiple, separate locations.

In other words, workflow engines have the potential to provide superior, streamlined experiences for on-the-ground users and IT teams alike.

Challenges of workflow engines

However, nothing is ever all plain sailing.

Indeed, workflow engines come along with their own trade-offs, challenges, and other potential downsides. It’s important to be cognizant of these in order to make an informed decision about the right workflow management strategy for your specific needs.

Specific issues to keep in mind include:

• Up-front-costs - We may incur relatively high up-front costs compared to alternative strategies.
• Maintenance and lifecycle management - When we build fully custom solutions, we’re liable for maintenance and lifecycle management costs.
• Variability within or across workflows - Workflow engines aren’t always the appropriate solution when our processes lack homogeneity.
• Stakeholder buy-in - We need to convince stakeholders at all levels that a workflow engine is the most viable solution.
• Centralization risks - Centralization can leave us more vulnerable to certain threats.
• Redundancy - If something goes wrong with our workflow engine, we’ll need some kind of backup plan to avoid service interruptions.
• High technical barriers - Traditional developments will be beyond the ability of less technical colleagues - whereas SaaS-based solutions might not be.
• Hiddens costs - We may incur additional costs from the tools we need to connect to our workflow engine - for instance, on a per-API call basis.
• Implementation and change management - We’ll need to plan and resource our roll-out.

So, implementing a workflow engine won’t be the right option in every scenario. Broadly speaking, we can point to two strands here.

On the one hand, there are times when a workflow engine is just not the right solution. On the other, there are situations where internal resources inhibit our ability to leverage workflow engines - even if they are a viable solution from a technical point of view.

However, that latter point is becoming increasingly alleviated with the rise of low-code development.

How to build a workflow engine in 7 steps

Let’s wrap up by thinking about the specific sequence of steps you can follow to plan, implement, and optimize a workflow engine project.

Here’s what this looks like in practice:

1. Analyze workflows - First, we need to analyze the workflows that will be handled by our engine, to determine the constituent actions that go into completing each task.
2. Create a data model - Next, we need to map out the requisite data objects we need to properly represent our resources, requests, and transitions.
3. Define transition rules - Determining the specific conditions and triggers that will govern when and how requests transition between states.
4. Choose your tools - Making a cost/benefit decision around the type of workflow engine to build and the development tools to use.
5. Implementation - Creating our workflow engine.
6. Testing - Testing our workflow engine against defined requirements - as well as fringe cases and user testing.
7. Roll-out and monitoring - Planning and implementing our changeover - alongside continuous monitoring to find further opportunities for improvement and optimization.

To learn more about choosing the right tools to support your workflow management efforts, check out our in-depth guide to digital transformation platforms .