Skip to main content

Background Jobs

The background jobs module is responsible for handling the execution of async processes or long running tasks in the system. These jobs can be triggered in the following situations:

  • From the Rails Application (web server) as a response to user actions e.g. generate a PDF from the form submission
  • Scheduled to run at specific times e.g. import data from a payer's FTP server every night

Rails Infrastructure​

One of Rails' main features is the ability to run background jobs. Rails provides a way to run background jobs using ActiveJob. ActiveJob is a framework for declaring jobs and making them run on a variety of queuing backends.

Main features​

  • Job Definition: Define a job by creating a class that inherits from ActiveJob::Base and implements a perform method. The job definition lives in the application codebase and has access to all the application's models and services.
  • Queues: Jobs are enqueued in queues. Queues are a way to group jobs by priority or type.
  • Priority: Jobs can be enqueued with a priority. The priority is used by the queue system to determine which job to run next.
  • Retry: Jobs can be retried if they fail. The number of retries and the delay between retries can be configured.

Other features (backend dependent)​

  • Scheduled Jobs: Some backends support scheduling jobs to run at a specific time.
  • Concurrency: Some backends support limiting the number of jobs that can run at the same time.
  • Batch Processing: Some backends support processing multiple jobs in a batch.
  • Monitoring: Some backends provide monitoring tools / web console to track the progress of jobs.

Backends​

While ActiveJob defines a common interface for working with background jobs, the actual execution of jobs is handled by a queueing backend. Rails provides a way to use different queueing backends by configuring the ActiveJob adapter.

See ActiveJob::QueueAdapters for a list of available adapters.

Different with traditional queue systems

Although most backend adapters use a queue system to store jobs, background job systems build higher level features on top of it as described above in the features section.

Testing​

WIP

  • inline execution
  • unable to test retries

Resque​

Resque reating background jobs, placing those jobs on multiple queues, and processing them later. It serves as the backend for ActiveJob.

Main components​

  • Worker: A worker is a process that listens to a queue and processes jobs. A worker can process multiple queues or be dedicated to a subset of them.
  • Scheduler: A scheduler is a process that schedules jobs to run at a specific time. It requires an extra process to manage and schedule jobs: resque-scheduler
  • Web UI: Resque provides a web interface to monitor the status of jobs, workers, and queues.

AWS Infrastructure​

Best Practices​

  • comparison to sidequick