---
title: "Logarithmic auto-scaling for Laravel Horizon"
description: "Why I added a logarithmic queue-size strategy to Laravel Horizon after large scheduled spikes kept starving smaller realtime queues."
canonical: "https://gummibeer.dev/blog/2026/logarithmic-auto-scaling-laravel-horizon"
---

# Logarithmic auto-scaling for Laravel Horizon

Why I added a logarithmic queue-size strategy to Laravel Horizon after large scheduled spikes kept starving smaller realtime queues.

This started with one of my side projects doing exactly what Laravel Horizon is supposed to handle: a bunch of queues, a bunch of workers and very different load patterns.

Some sources constantly add new jobs. Others have hourly or daily peaks. And some can suddenly dispatch tens or hundreds of thousands of jobs within a few seconds after a specific action.

At peak Horizon manages a little over 100 workers across five servers and processes around 4,500 jobs per minute.

Not the most impressive numbers when it comes to scale. That's a lazy morning at [Hospitable](https://hospitable.com/) for me. 😂

But the load profile is annoying in a very specific way.

A queue getting 100,000 jobs at midnight doesn't automatically mean I want it drained as fast as physically possible. That queue has roughly 24 hours until the next batch. Meanwhile another queue might get 10,000 jobs every hour and another one gets a constant stream of about one job per second.

Those smaller queues are much more sensitive to being starved for 20 minutes.

And that's exactly what my current Horizon setup does.

## The problem with size scaling

The jobs have roughly the same runtime profile, so splitting them into separate supervisors just because their arrival patterns differ always felt wrong to me.

They belong into the same worker pool.

So the supervisor currently uses Horizon's `size` auto-scaling strategy.

```php
'autoScalingStrategy' => 'size',
```

That strategy is very literal: if one queue contains 100,000 jobs and another contains 10,000, the first queue gets roughly ten times the weight.

Which sounds absolutely reasonable.

Until the 100,000 jobs arrived five seconds ago and the 10,000-job queue is the one that has new work arriving constantly.

At that point the huge queue immediately takes nearly all available workers. The other queues sit at their configured minimum - one worker in my case - and start growing.

Eventually they grow enough to deserve a second worker according to the size ratio. Then a third one. And so on.

But that's backwards for my workload.

The realtime-ish queue first has to become a problem before Horizon reacts to it.

Exaggerated, size scaling behaves a bit like this:

1. process the 100k queue until it gets close to the 10k queue;
2. process both 10k queues until they get close to the 2k queue;
3. finally process all three at a similar rate.

For a queue where waiting 24 hours is fine, that's a lot of unnecessary urgency.

For the queue getting new jobs every second, the temporary starvation is the actual problem.

## I already tried other solutions

This wasn't my first attempt at fixing it.

Over time I tried three different approaches. None of them were something I wanted to keep in production. 🙈

The last one went way too far and essentially replaced Horizon's balancing logic with my own implementation.

But it contained one idea I really liked: normalize queue sizes logarithmically before calculating their worker share.

That was the part worth stealing from myself.

## Logarithms without the university flashbacks

For everyone who doesn't have the math definition of a [logarithm](https://en.wikipedia.org/wiki/Logarithm) ready in their head: the bigger a number gets, the logarithm still gets bigger, just much more slowly.

I would probably, and mathematically not entirely correctly, describe it as kind of the inverse direction of power.

The important part here is what it does to large differences.

Using base-10 logarithms just because the numbers are easy to read:

| Queue size | `log10(size)` |
| ---------: | ------------: |
|      2,000 |           3.3 |
|     10,000 |           4.0 |
|    100,000 |           5.0 |

The 100,000-job queue is **50 times bigger** than the 2,000-job queue.

Its logarithmic weight isn't even twice as large.

That's exactly the property I want.

If I idealize the calculation with 13 fractional workers, the difference becomes pretty obvious:

| Queue |    Jobs | Size target | Log target |
| ----: | ------: | ----------: | ---------: |
|     A |   2,000 |        0.23 |       3.49 |
|     B |  10,000 |        1.16 |       4.23 |
|     C | 100,000 |       11.61 |       5.28 |

So instead of saying "C is 50 times larger, give it basically everything", logarithmic scaling says "C is clearly the largest queue and should get more workers, but A and B are still real backlogs that deserve capacity".

In human terms I think about that as roughly **4 / 4 / 5 workers** instead of **1 / 1-2 / almost everything else**.

Horizon doesn't assign fractional workers and its scaler applies pool changes sequentially, so the exact integer result depends on the current state, minimum processes and scaling order. The table is about the weighting, not a promise that every rebalance will result in those exact integers.

## It almost looks like predicting future jobs

Logarithmic scaling obviously doesn't know what will happen in five minutes.

But for my workload the result can look a little like it does.

With size scaling, the smaller queue first has to pile up enough jobs to become relevant compared to the huge spike.

With log scaling, a queue with 2,000 jobs already has enough weight to get a meaningful number of workers while another queue sits at 100,000.

That means the constant queue can keep up with incoming jobs instead of growing until Horizon eventually decides it became large enough to care about.

The hourly queue can get through its batch in maybe 20 minutes and then release those workers again.

The 100k daily queue gets a more irregular processing pattern. It gets a decent share while everything else is busy, then speeds up significantly once the smaller queues are empty.

Which is fine.

It has all day.

## Adding it to Horizon

Once I reduced the problem to "I want one more weighting function", replacing Horizon suddenly looked pretty stupid.

Horizon already has the concept I need.

The `autoScalingStrategy` option was added with [`time` and `size`](https://github.com/laravel/horizon/pull/1254). So instead of introducing another scaler, another abstraction or a configurable strategy class, I added a third value to the existing option.

```php
'autoScalingStrategy' => 'log',
```

The implementation in [laravel/horizon#1818](https://github.com/laravel/horizon/pull/1818) is intentionally small.

First calculate the total logarithmic weight:

```php
$totalLogJobs = $supervisor->options->autoScaleLogarithmically()
    ? $queues->sum(fn ($queue) => log1p($queue['size']))
    : 0;
```

Then use that weight instead of the linear queue size:

```php
if ($supervisor->options->autoScaleByNumberOfJobs()) {
    $numberOfProcesses = $timeToClear['size'] / $totalJobs;
} elseif ($supervisor->options->autoScaleLogarithmically()) {
    $numberOfProcesses = log1p($timeToClear['size']) / $totalLogJobs;
} else {
    $numberOfProcesses = $timeToClear['time'] / $timeToClearAll;
}
```

That's basically the feature.

No custom worker allocator.

No strategy interface.

No new service to resolve from the container.

Just another way to calculate the weight before Horizon does what it already does.

## Why `log1p`?

The obvious mathematical version would be:

```php
log($queue['size'])
```

Except `log(0)` doesn't exist in a useful way for this calculation.

Empty queues are normal, so that would immediately need a special case.

PHP's [`log1p`](https://www.php.net/manual/en/function.log1p.php) calculates `log(1 + x)` and gives exactly the behavior I want:

```text
0 jobs   -> log(1) -> 0
1 job    -> log(2)
100 jobs -> log(101)
```

An empty queue has zero weight, positive queues have positive weight, done.

The code uses the natural logarithm rather than `log10` from the examples above. That doesn't matter after normalization. Changing the logarithm base multiplies every queue weight by the same constant, which cancels out when each weight is divided by the total.

## Why job count and not time-to-clear?

This was another decision I wanted to keep very explicit.

Horizon already has the `time` strategy. It multiplies queue size by the measured runtime and allocates workers by the total amount of queued work.

That's useful when runtime is the meaningful difference between queues.

It isn't my problem.

My jobs are close enough in runtime that I care primarily about queue pressure and starvation.

Logging the time-to-clear would also introduce a weird property: logarithms care about the scale of their input. `log(5000ms)` and `log(5s)` are not related by a constant multiplier in the final normalized weighting because converting units adds a constant inside the logarithm.

Job count has a natural unit here. Five thousand jobs are five thousand jobs.

So `log` is deliberately a **logarithmic size strategy**, not a logarithmic version of `time`.

## Testing the actual problem

I wanted the tests to prove more than "PHP can calculate a logarithm".

The first test uses queue sizes `999`, `99`, `99`, `99` and `9` with ten workers.

Those numbers are a little artificial, but useful because `log1p` turns them into exact powers of ten:

```text
log(1000) : log(100) : log(100) : log(100) : log(10)
       3  :        2 :        2 :        2 :       1
```

So the expected allocation is a clean `3 / 2 / 2 / 2 / 1`.

Another test gives the smaller queue an absurdly higher runtime and verifies that `log` still follows queue size rather than accidentally becoming another time strategy.

And then there is the test based on the situation that originally annoyed me.

Two busy queues and one empty queue. 25 workers. Both busy queues start with one worker:

```text
A:    946 jobs
B: 13,702 jobs
C:      0 jobs
```

With normal `size` scaling, one pass ends up at:

```text
A:  2 workers
B: 22 workers
C:  1 worker
```

The queue with almost 1,000 jobs gets one additional worker while B gets basically the entire pool.

With `log`:

```text
A: 11 workers
B: 13 workers
C:  1 worker
```

B still gets more capacity because it is much larger.

It just can't erase A from the worker pool anymore.

That's the behavior I was after.

## `log` isn't the better strategy

I deliberately don't want to sell this as a better Horizon auto-scaler.

It isn't.

The PR description uses a media processing pipeline as the counterexample. Imagine one import job creating WebP conversion, thumbnail and fingerprint jobs with very different runtimes.

In that setup, `time` can be exactly the right strategy because the backlog represents real amounts of work. A fingerprint job taking five times longer than an import job should influence worker allocation heavily.

The import queue might even be useful as natural backpressure. Giving it fewer workers prevents it from flooding already overloaded downstream queues.

Logarithmic scaling can be actively worse there because its entire purpose is to compress large differences.

That's why I like it as a third strategy rather than a replacement for either existing one:

- `time` balances by estimated queued work;
- `size` balances by raw number of queued jobs;
- `log` balances by queue size while deliberately reducing the dominance of huge backlogs.

Different workload, different goal.

For my side project I don't want maximum throughput on whichever queue happened to receive the largest batch most recently.

I want all five queues to keep making useful progress, especially the ones that continue receiving new jobs while another queue works through a batch that has hours left on its deadline.

That's what the logarithm gives me.

The [Horizon PR is open](https://github.com/laravel/horizon/pull/1818) now.

And even if it doesn't make it upstream, at least I no longer need an almost-custom Horizon installation just to change one division. 😅