Hangfire.InMemory 0.8.0

Hangfire.InMemory

Build status

This is an attempt to create an efficient transactional in-memory storage for Hangfire with data structures close to their optimal representation. The result of this attempt should enable production-ready usage of this storage implementation and handle particular properties of in-memory processing like avoiding OutOfMemoryException at any cost and moderate load on garbage collection. Of course we can't avoid them entirely, but at least can control them somehow.

Read and write queries are processed by a single thread to avoid additional synchronization between threads and keep everything as simple as possible (keeping future async-based implementation in mind). Monitoring API also uses that dedicated thread, but its future implementation can be changed by using concurrent data structures and immutability, but I expect this will increase load on garbage collection.

Distributed locks (heh, in an in-process storage), queue fetch logic (even from multiple queues) and transactional queries are implemented as blocking operations, so there is no active polling in these cases. Every data returned by storage can be safely changed without causing underlying storage state to be changed with bypassing required transactional processing logic, so everything is safe (but increase load on GC). Every data structure, including indexes and their records, is removed when empty to avoid memory leaks.

TODO

  • Control OutOfMemoryException by providing some kind of limits which can be established easily, e.g. total number of jobs.
  • Avoid unnecessary object allocations without sacrificing the safety property (as described above).
  • Add integration (for public API) and unit tests (for internal API).
  • Force expiration when memory pressure is high to avoid OutOfMemoryException.
  • Add overridden default for expiration time for jobs and batches?
  • Can avoid synchronization in some read-only methods in the MemoryConnectionclass.
  • Avoid using large object heap at any cost by large collections – millions of jobs can be created.

Installation

Hangfire.InMemory is available on NuGet so we can install it as usual using your favorite package manager.

> dotnet add package Hangfire.InMemory

Configuration

After the package is installed we can use the new UseMemoryStorage method for the IGlobalConfiguration interface to register the storage.

GlobalConfiguration.Configuration.UseInMemoryStorage();

Maximum Expiration Time

Starting from version 0.7.0, the package controls the maximum expiration time for storage entries and sets it to 2 hours when a higher expiration time is passed. By default, the expiration time for background jobs is 24 hours, and for batch jobs and their contents is 7 days which can be too big for in-memory storage that runs side-by-side with the application.

We can control this behavior or even disable it with the MaxExpirationTime option available in the InMemoryStorageOptions class in the following way.

GlobalConfiguration.Configuration.UseInMemoryStorage(new InMemoryStorageOptions
{
    MaxExpirationTime = TimeSpan.FromHours(6) // Or set to `null` to disable
});

No packages depend on Hangfire.InMemory.

https://github.com/HangfireIO/Hangfire.InMemory/releases 0.8.0 • Project – Sign NuGet package and .NET assemblies on build with a company's own certificate. • Project – Require package signature validation when restoring dependencies. • Project – Add HangfireIO as an owner for the NuGet package. • Project – Add readme file and icon to the NuGet package. • Project – Fix Git repository URL in the NuGet package metadata. 0.7.0 • Added – `InMemoryStorageOptions.MaxExpirationTime` option to control the maximum expiration time. • Changed – The default value for maximum expiration time is 2 hours now, not days. • Fixed – Populate `ParametersSnapshot` and `InvocationData` properties in `IMonitoringApi.JobDetails`. • Fixed – The "Awaiting Jobs" page now includes the state name of an antecedent background job. • Fixed – The "Scheduled Jobs" page now has correct identifiers for jobs with explicit queues defined. • Fixed – Unify job ordering in Monitoring API to be the same as in other storages. • Project – Enable source link support with embedded symbols for simplified debugging. • Project – Refactored internals and added even more unit tests. • Project – Enable NuGet package restore with lock file and locked mode. • Project – Project and Release Notes URLs in the NuGet package now point to the repository. • Project – Enable tests running on the `net6.0` platform and Ubuntu on AppVeyor. 0.6.0 • Added – `InMemoryStorageOptions.MaxStateHistoryLength` option to control state entries. • Changed – Always use monotonic clock when working with time. • Changed – Release distributed locks when their connection is disposed. • Changed – Pass dispatcher fault exceptions to a caller thread. • Project – Refactor internal types to have a cleaner project structure and avoid mistakes. • Project – Enable static analysis by the Microsoft.CodeAnalysis.NetAnalyzers package. • Project – Enable portable PDBs for the .NET Framework 4.5.1 platform. 0.5.1 • Fixed – Infinite loop in recurring job scheduler consuming 100% CPU regression after 0.5.0. 0.5.0 • Added – `InMemoryStorageOptions.StringComparer` as a central option for key and index comparisons. 0.4.1 • Fixed – "Awaiting Jobs" metric is now correctly populated with `Version180` compatibility level. 0.4.0 • Breaking – Package now depends on Hangfire.Core version 1.8.0. • Breaking – Replace the `net45` target with `net451` one as the former is not supported. • Changed – Improve `GetFirstByLowestScoreFromSet` operations. • Changed – Implement the `Job.Queue` feature. • Changed – Implement the `Connection.GetUtcDateTime` feature. • Changed – Implement the `Connection.GetSetContains` feature. • Changed – Implement the `Connection.GetSetCount.Limited` feature. • Changed – Implement the `Connection.BatchedGetFirstByLowestScoreFromSet` feature for the storage. • Changed – Implement the `Transaction.AcquireDistributedLock` feature. • Changed – Implement the `Transaction.CreateJob` feature. • Changed – Implement the `Transaction.SetJobParameter` feature. • Changed – Implement the new monitoring features. • Changed – Populate the new properties in Monitoring API. • Changed – Populate the `Retries` metric in the `GetStatistics` method. 0.3.7 • Fixed – Throw `BackgroundJobServerGoneException` outside of dispatcher thread. 0.3.6 • Fixed – Ensure lock entries are eventually removed from their collection. • Fixed – Ensure lock entries are always updated under a monitor. 0.3.5 • Fixed – Ensure entries are expired even during constant storage pressure. 0.3.4 • Fixed – Reverse state list instead of sorting it by date in the `JobDetails` method. • Fixed – Better sorting for state indexes, take into account job creation date. • Fixed – Reverse succeeded and deleted job lists to match Redis implementation. 0.3.3 • Fixed – Sort queues and servers when returning them from monitoring api and in the Dashboard UI. 0.3.2 • Fixed – Enqueued jobs may become invisible when adding a lot of jobs simultaneously to a new queue. • Fixed – Some workers are waiting for background jobs forever when several jobs added at once. • Fixed – Workers are able to detect new background jobs only after another background job is processed. • Fixed – Workers don't see background jobs when multiple queues are used with minimal workload. 0.3.1 • Fixed – `NullReferenceException` in the `SignalOneQueueWaitNode` method when using multiple queues. 0.3.0 • Added – `InMemoryStorageOptions.DisableJobSerialization` option. • Fixed – `ObjectDisposedException` on semaphore when committing a transaction. • Fixed – Gracefully handle `ObjectDisposedException` when signaling for query completion. • Fixed – Avoid killing the whole process in case of an exception in dispatcher, stop it instead. • Project – Add a lot of new unit tests for `InMemoryMonitoringApi` class. 0.2.0 • Fixed – A lot of corner cases revealed by unit tests. • Project – Added a ton of unit tests for the top-level classes to ensure behavior is consistent. 0.1.0 – Initial release

.NET Framework 4.5.1

.NET Standard 2.0

Version Downloads Last updated
1.0.0 0 09/18/2024
0.11.0 0 09/18/2024
0.10.4 0 09/10/2024
0.10.3 2 09/28/2024
0.10.2 0 06/17/2024
0.10.0 0 06/03/2024
0.9.0 0 04/23/2024
0.8.1 0 03/21/2024
0.8.0 2 06/27/2024
0.7.0 0 01/10/2024
0.6.0 0 10/23/2023
0.5.1 0 06/28/2023
0.5.0 0 06/27/2023
0.4.1 0 05/24/2023
0.4.0 0 04/28/2023
0.3.7 0 03/22/2023
0.3.6 0 02/08/2023
0.3.5 1 12/02/2022
0.3.4 0 11/06/2020
0.3.3 0 11/06/2020
0.3.2 0 10/29/2020
0.3.1 0 10/22/2020
0.3.0 0 10/22/2020
0.2.0 0 10/12/2020
0.1.0 0 09/29/2020