Hangfire.InMemory 0.8.0
Hangfire.InMemory
This is an attempt to create an efficient transactional in-memory storage for Hangfire with data structures close to their optimal representation. The result of this attempt should enable production-ready usage of this storage implementation and handle particular properties of in-memory processing like avoiding OutOfMemoryException
at any cost and moderate load on garbage collection. Of course we can't avoid them entirely, but at least can control them somehow.
Read and write queries are processed by a single thread to avoid additional synchronization between threads and keep everything as simple as possible (keeping future async-based implementation in mind). Monitoring API also uses that dedicated thread, but its future implementation can be changed by using concurrent data structures and immutability, but I expect this will increase load on garbage collection.
Distributed locks (heh, in an in-process storage), queue fetch logic (even from multiple queues) and transactional queries are implemented as blocking operations, so there is no active polling in these cases. Every data returned by storage can be safely changed without causing underlying storage state to be changed with bypassing required transactional processing logic, so everything is safe (but increase load on GC). Every data structure, including indexes and their records, is removed when empty to avoid memory leaks.
TODO
- Control
OutOfMemoryException
by providing some kind of limits which can be established easily, e.g. total number of jobs. - Avoid unnecessary object allocations without sacrificing the safety property (as described above).
- Add integration (for public API) and unit tests (for internal API).
- Force expiration when memory pressure is high to avoid
OutOfMemoryException
. - Add overridden default for expiration time for jobs and batches?
- Can avoid synchronization in some read-only methods in the
MemoryConnection
class. - Avoid using large object heap at any cost by large collections – millions of jobs can be created.
Installation
Hangfire.InMemory is available on NuGet so we can install it as usual using your favorite package manager.
> dotnet add package Hangfire.InMemory
Configuration
After the package is installed we can use the new UseMemoryStorage
method for the IGlobalConfiguration
interface to register the storage.
GlobalConfiguration.Configuration.UseInMemoryStorage();
Maximum Expiration Time
Starting from version 0.7.0, the package controls the maximum expiration time for storage entries and sets it to 2 hours when a higher expiration time is passed. By default, the expiration time for background jobs is 24 hours, and for batch jobs and their contents is 7 days which can be too big for in-memory storage that runs side-by-side with the application.
We can control this behavior or even disable it with the MaxExpirationTime
option available in the InMemoryStorageOptions
class in the following way.
GlobalConfiguration.Configuration.UseInMemoryStorage(new InMemoryStorageOptions
{
MaxExpirationTime = TimeSpan.FromHours(6) // Or set to `null` to disable
});
No packages depend on Hangfire.InMemory.
Version | Downloads | Last updated |
---|---|---|
1.0.0 | 0 | 09/18/2024 |
0.11.0 | 0 | 09/18/2024 |
0.10.4 | 0 | 09/10/2024 |
0.10.3 | 2 | 09/28/2024 |
0.10.2 | 0 | 06/17/2024 |
0.10.0 | 0 | 06/03/2024 |
0.9.0 | 0 | 04/23/2024 |
0.8.1 | 0 | 03/21/2024 |
0.8.0 | 2 | 06/27/2024 |
0.7.0 | 0 | 01/10/2024 |
0.6.0 | 0 | 10/23/2023 |
0.5.1 | 0 | 06/28/2023 |
0.5.0 | 0 | 06/27/2023 |
0.4.1 | 0 | 05/24/2023 |
0.4.0 | 0 | 04/28/2023 |
0.3.7 | 0 | 03/22/2023 |
0.3.6 | 0 | 02/08/2023 |
0.3.5 | 1 | 12/02/2022 |
0.3.4 | 0 | 11/06/2020 |
0.3.3 | 0 | 11/06/2020 |
0.3.2 | 0 | 10/29/2020 |
0.3.1 | 0 | 10/22/2020 |
0.3.0 | 0 | 10/22/2020 |
0.2.0 | 0 | 10/12/2020 |
0.1.0 | 0 | 09/29/2020 |