Table of Contents

Expiring resources

Coming in 4.0

Uploads can delete themselves: exports, previews, temporary shares. Give an upload an expiry, and turn on the background service that deletes what has expired:

builder.Services.AddResourcefull(options => options
    .UseSqlServer(connectionString)
    .UseExpiry());   // looks for expired versions every minute; pass a TimeSpan to change that

await container.UploadAsync("exports/report.zip", stream, new ResourceUploadOptions
{
    ExpiresAt = DateTimeOffset.UtcNow.AddHours(24)
});

How it works

  • The expiry belongs to the version, and ResourceInfo.ExpiresAt reports it. An expired version is deleted as a delete of that version would delete it: a name whose only version expires is gone, and when the current version expires, with versioning, the one stored before it becomes current again. Without versioning, the versions it replaced go with it.
  • A version is gone within one interval of expiring. Until the service gets to it, it can still be read. Any time works, from seconds to years away, or in the past, which the next sweep deletes. For expiries of a few seconds, use a shorter interval, such as UseExpiry(TimeSpan.FromSeconds(5)): with nothing expired, a sweep reads one row of the index of versions that expire.
  • Every instance of the application can run it. Deleting a version takes its name's lock, so instances don't conflict, and a version two instances find is deleted once.
  • One version that fails doesn't hold up the rest. If a version can't be deleted, the sweep carries on with the others and reports the failure; that version is tried again next time.
  • Read-only containers don't expire. A read-only container keeps its expired versions, unreported, until it's writable again, and its expiries can't be changed. Freezing a container keeps everything in it. Each sweep still goes past the expired versions it keeps, a seek on that index for each batch of 100.
  • It also deletes what stopped uploads left behind 4.0, a day after they last stored anything: an upload session nothing was appended to, with any provider, and, with SQL Server, PostgreSQL and SQLite, the staged content of an upload whose process died partway.
  • Retained and held versions wait 4.0. A version that's retained or under a legal hold is left until both its expiry and its retention date have passed, and its hold is lifted.
  • Expired versions skip the bin 4.0. In a container with a bin, an expired version is still deleted for good, since an expiry is often set to erase data on time. A version already in the bin doesn't expire there.
  • It also purges the bin 4.0: after deleting expired versions, each sweep purges, for good, the versions whose time in a bin is up. KeepDeletedFor needs UseExpiry for that reason.
  • Without UseExpiry, expiries are stored, and IResourceServiceClient.DeleteExpiredAsync() deletes what has expired, and what stopped uploads left behind, whenever you call it, such as from a scheduled job.

Changing an expiry

SetExpiryAsync moves a version's expiry, such as to extend a share, or with null keeps it for good:

await container.SetExpiryAsync("exports/report.zip", versionId, DateTimeOffset.UtcNow.AddDays(7));

A null version ID changes the current version's expiry. Only the expiry changes: the content, LastModified and ETag stay as they are. It returns false when the version is gone, as it is if the service deleted it first. A read-only container refuses it.

Copies, moves and restores

  • A copy doesn't expire unless ResourceCopyOptions.ExpiresAt says so.
  • A move keeps each version's expiry.
  • A restore doesn't expire: it brings back a version to keep.

Auditing what's deleted

To record every version Resourcefull deletes by itself, such as in an audit system, register a cleanup handler. It hears about expired versions, about older versions pruned past MaxVersions, and 4.0 about versions the sweep purges from a bin, with the reason Purged, before each is deleted. In a container with a bin, a pruned version goes to the bin at once, and the handler hears about it then, with Pruned. Otherwise, with expiry on, pruning gives a version five minutes before the sweep deletes it, so the handler hears about it then, with the reason Expired, or with Pruned if, in a container that keeps one version, it's deleted sooner along with the version that replaced it:

public sealed class AuditCleanup(AuditClient audit) : IResourceCleanupHandler
{
    public async ValueTask OnCleanupAsync(ResourceCleanup cleanup, CancellationToken cancellationToken) =>
        await audit.RecordAsync(cleanup.Version.ContainerName, cleanup.Version.Name, cleanup.Version.VersionId,
            cleanup.Reason, cleanup.At, cancellationToken);
}

builder.Services.AddResourcefull(options => options.UseSqlServer(connectionString).UseExpiry()
    .UseCleanupHandler<AuditCleanup>());
  • No deletion goes unreported. A version is deleted only after the handler returns. If it throws, as when the audit system is down, the version stays, and the next sweep, or the name's next prune, tries again.
  • A report means "about to be deleted", not "deleted". A version can be reported more than once, by each instance or prune that finds it, or again when its delete failed, and occasionally it isn't deleted after all, when its expiry was extended or something else deleted it first. Record reports by version ID to ignore repeats, and check ExistsAsync if you need to know it's gone.
  • Your own deletes aren't reported: audit those where your code makes them. That includes putting a version in a bin, and purging one early with PurgeDeletedAsync.
  • It's a scoped service, so it can use your other services.

Seeing what expires

4.0

To see what a container loses next, list the versions that expire soonest. To check that the sweep is keeping up, list the expired versions it hasn't deleted yet, in every container:

IReadOnlyList<ResourceInfo> next = await container.GetTopVersionsAsync(ResourceVersionOrder.ExpiringFirst, 20, cancellationToken);
IReadOnlyList<ResourceInfo> waiting = await resources.GetExpiredVersionsAsync(100, cancellationToken);

The first lists current and older versions alike, and leaves out versions without an expiry. The second lists them in the order they expired, oldest first, so if the first one expired long ago, the sweep isn't running or isn't keeping up. It includes a read-only container's expired versions, which the sweep keeps until the container is writable again. The dashboard shows both.

With a provider of your own

Expiry needs a provider that stores each version's ExpiresAt and can list and delete expired versions. Every built-in provider does. A provider of your own says whether it does through SupportsExpiry, which is false unless the provider overrides it. While it's false, every write that sets an expiry throws NotSupportedException and stores nothing, so no version has an expiry that never happens. UseExpiry also stops the application from starting, with an InvalidOperationException that names the provider.

Telemetry

Each sweep that finds expired versions records an expire operation, with resourcefull.expired counting the ones it deleted, and resourcefull.abandoned_uploads the stopped uploads whose leftovers it deleted, and a sweep that finds no expired versions records nothing. A read-only container's expired versions count as found, so while one keeps some, every sweep records an operation, deleting none of those. SetExpiryAsync records set_expiry. Counters add up the versions sweeps delete (resourcefull.client.expired) and fail to delete (resourcefull.client.expire.failures), and the cleanup handler's failures; see Work that happens out of sight.

Edit this page