Background jobs in cloud-native applications
04.04Maarten Ghijsens is one of our application architects specialising in cloud-native solutions for Microsoft’s Azure platform. He recently built an application for one of our clients that accumulates their Office 365 workspaces (multiple SharePoint-sites and Teams) into a convenient overview. As part of this project, Maarten researched ways of running background jobs (or long-running operations) in a cloud-native way on Azure. In this blog post, we will explain our decision-making process and how you can get started yourself.
The application Maarten built is mainly responsible for lifecycle management and user management of the client’s workspaces. It provides users a personalised overview of their workspaces and the possibility to request different types of workpaces. Additionally, the application unifies and centralizes the creation of SharePoint sites and Teams, monitors inactive workspaces and reminds users when archiving is recommended. Because users can still create workspaces outside of the application as well, the application’s database needed to synchronise with the client’s Office 365 environment regularly.
The table below summarises Maarten’s research. It sums up different possibilities in Azure and the elements that may influence your decision in designing the background job. Keep in mind that it is not exhaustive, and should therefore be treated as a starting point as opposed to a complete guide.
Note: Maarten deliberately left out Azure Batch from this table, since it focuses on virtual machines instead of containers. Because our solution is serverless and cloud-native, Azure Batch was not a feasible choice.
| Azure Web Jobs | Azure App Service + Hangfire | Azure Functions | Azure Logic Apps | Azure Container Instances | ||||
| Regular | Durable | Premium | ||||||
| Serverless | No | No | Yes | Yes | No | Yes | Yes | |
| Duration limit | 230 seconds | 230 seconds | Max. 10 minutes | Yes | Max. 60 minutes | No | No | |
| Cost | High | High | Low | Low | High | Low | Low | |
| Scalability | Scale-up | Manual | Manual | Auto | Auto | Auto | Auto | Auto / Manual |
| Scale-out |
 |
|||||||
| Purpose | Existing background jobs that need to be moved to the cloud quickly in a lift-and-shift scenario. | Short/small tasks that are triggered by user actions in the web application, but need to run asynchronous to avoid delays in the user experience. | Short/small tasks that are triggered by events or other applications. | Long-running operations that can be split up in short/small, parallel tasks. | Tasks that are triggered by events or other applications. | Integration and low-code workflows. | Run event-driven applications, quickly deploy from your container development pipelines, run data processing or build jobs. | |
| Important notes | Should not be used for greenfield applications. | Fire-and-forget tasks are affected by time-outs of the Azure App Service and the storage where tasks are persisted. Recurrent background jobs may fail to trigger if the Azure App Service is not always on. | Cold-starts after long inactivity affect response times. | Orchestration of work is required. Hard to debug due to parallel task execution. | Not suited for intensive or complex workloads. | |||
We eventually decided on a .NET 6 C# console application, which we put into a Linux-based Docker container image. We uploaded that container to an Azure Container Registry, which allowed us to run instances of this console application in Azure Container Instances. Once the background job was running correctly inside the Azure Container Instance, we used an Azure Logic App with a recurrence trigger to start the Azure Container Group periodically.
By using a serverless model and a consumption-based plan, the background job is cheap to run compared to alternatives. Because we coded the application in .NET 6 C#, we can use it with all platforms and operating systems. The Azure Container Instance provides us with plenty of scaling possibilities, while avoiding dependencies on packages like Hangfire — an approach often used in fire-and-forget background jobs for .NET applications.
Interested in running your own background job in a container? Download our step-by-step plan to get started, or contact us to discuss the possibilities together!
