Table of Contents

S3-compatible storage

Coming in 4.0

Resourcefull.S3 stores resources in a bucket of an S3-compatible object store: Amazon S3 itself, or any store that speaks its API, hosted or self-hosted. Files stream in and out with flat memory use, and every name keeps its versions, content hashes and metadata, as with the database providers. It's tested on every change against SeaweedFS, a self-hosted store.

dotnet add package Resourcefull.S3

using Resourcefull;
using Resourcefull.S3;

builder.Services.AddResourcefull(options => options.UseS3("my-company-files"));

The bucket must exist; Resourcefull doesn't create it. Without a client of your own, Resourcefull uses the IAmazonS3 registered in your application's services, or else creates an AmazonS3Client that finds its region and credentials the way the AWS SDK always does: environment variables, the shared credentials file, or the role of the machine or container it runs on.

What it supports

Uploads, downloads (with ranges), listing, versions, create-only and conditional writes, conditional deletes, copies, moves and renames, container and version metadata, container policies, export and import, expiry and upload sessions. It passes the same contract tests as the other providers, so switching to it, or from it, takes only the line that picks the provider. Provider features compares the providers.

Connecting to a store

Amazon S3. The defaults work: give the bucket's name, and run with credentials that can reach it.

Another store. Give it a client pointed at the store's endpoint:

using Amazon.Runtime;
using Amazon.S3;

var client = new AmazonS3Client(new BasicAWSCredentials(accessKey, secretKey), new AmazonS3Config
{
    ServiceURL = "https://s3.example.com",
    ForcePathStyle = true,  // most self-hosted stores want the bucket in the path, not the host name
    // Recent AWS SDKs send checksums some stores don't accept; these send them only where S3 requires them.
    RequestChecksumCalculation = RequestChecksumCalculation.WHEN_REQUIRED,
    ResponseChecksumValidation = ResponseChecksumValidation.WHEN_REQUIRED
});

builder.Services.AddResourcefull(options => options.UseS3("files", s3 => s3.Client = client));

Keep the keys out of source control, in user secrets, environment variables or a secrets store.

What the store must do

To keep one current version per name, and to decide create-only writes and conditions, Resourcefull writes an object only if it hasn't changed since it was read. That needs the store to honor two headers on PutObject: If-None-Match: * (write only if the object doesn't exist) and If-Match (write only if the object still has the ETag that was read), answering 412 Precondition Failed otherwise. Amazon S3 has done both since late 2024, and SeaweedFS does too. A store that ignores them would let two writers both think they won, so check yours before relying on it: upload the same new name twice at once with ResourceWriteMode.Create, and exactly one upload must succeed. The contract tests do this, among much else; running them against your store, as Writing a provider shows, is the thorough check. If a store refuses 100 conditional writes of the same object in a row, the operation fails with InvalidOperationException rather than trying forever.

Resourcefull doesn't need bucket versioning, and is better off without it: it rewrites a small object for every change to a name, and with versioning on, the store keeps every old copy of it.

Sharing a bucket

Every key Resourcefull writes starts with resourcefull/, so the bucket can hold other things too. To keep several stores in one bucket, such as one per environment, give each a prefix of its own:

options.UseS3("company-files", s3 => s3.KeyPrefix = "staging/resourcefull/");

Don't let anything else write, change or delete objects under Resourcefull's prefix, and give every instance of the application the same bucket and prefix. A prefix can be up to 190 bytes of UTF-8 long; an empty one puts Resourcefull's objects at the top of the bucket.

Permissions

The credentials need these actions on the bucket's objects under the prefix: s3:GetObject, s3:PutObject, s3:DeleteObject and s3:AbortMultipartUpload, and s3:ListBucket on the bucket itself. They don't need to create or delete buckets, or change their settings.

How it stores things

You don't need this to use the provider, but it explains its costs and what you'll see in the bucket.

  • Each container has a record, {prefix}c/{name}, holding a random ID and the container's metadata. Everything the container holds is kept under that ID, at {prefix}d/{id}/.
  • Each name has a head, a small JSON object listing its versions' attributes in the order they were stored; the last is current. Every change to a name (an upload, a delete or a metadata change) reads the head and writes it back only if nobody changed it meanwhile. When somebody did, it reads it again and retries. Each write carries a random ID, so when an answer is lost and the store refuses the resent write, Resourcefull can tell its own write landed.
  • Each version that expires has an entry in an index under {prefix}x/, keyed by when it expires, so the expiry sweep lists only what's due rather than reading every head. The entry goes in just before the head write that sets the expiry, so a version never has an expiry the index doesn't know about. Each entry's key holds a random token that the version records, so no two writes share a key. Changing an expiry writes a new entry, then deletes the old one once the head no longer names it; deleting a version deletes its entry with its content. An entry those deletes miss, such as one a failed write left, is dropped by a sweep that finds it due and over an hour old; a younger one may belong to a write still in progress.
  • Each version in a bin 4.0 stays in its name's head, separately from the name's versions, and its content stays where it was. It has two entries, written just before the head write that puts it in the bin: one under {prefix}d/{id}/t/, newest deletion first, whose key holds its content's length, for listings of the bin and the usage figures, and one under {prefix}t/, keyed by when it's purged, for the sweep. Both are checked against the head when read, and one whose version is no longer in the bin is dropped once it's over an hour old, as expiry entries are; the daily scan for content no head names also drops stale bin entries that no listing reaches. A name left with only a bin has a mark under {prefix}d/{id}/e/, so the usage figures can leave it out of the names, and the same scan fixes any mark a failed write left wrong.
  • Each version's content is an object of its own, uploaded before the head names it, and deleted after the head stops naming it.
  • A move writes an entry for it under {prefix}m/, marks both names' heads with its ID, then commits that entry with one conditional write: until then, every read finds the versions at the source, and from then at the destination, so they're never in both places or in neither. Across containers, the store first copies each version's content under the destination container's ID, in parts of up to 1 GiB, so the destination's usage counts it and deleting the source container doesn't take it; within a container, nothing is copied. Each version that expires gets an expiry entry for its new name. A write to either name while the move is still pending abandons it, so the mover's commit is refused and it starts again. Once a move is decided, the mover clears the marks, deletes the entry and, across containers, the content left behind.
  • Each upload session has a record, {prefix}u/{container ID}/{upload ID}!/r, holding the version's attributes and the list of its segments, and one object per segment beside it. Sessions are kept apart from the containers' data, so they don't count toward a container's usage. An append stores its content in segments of up to 64 MiB, adding each to the record as soon as it's stored, so an append that fails partway keeps what it stored. Each change to the record is a conditional write, as for heads, so two appends from the same offset can't both land. A last segment shorter than 8 MiB is read back and stored again with the start of the next append, so every segment but the last is at least 8 MiB, above the 5 MiB S3 requires of the parts it joins. Completing marks the record sealed, so nothing more can be appended, then has the store join the segments into the version's content object with a multipart upload whose parts are copied from them (UploadPartCopy); the bytes don't pass through your application. Then it adds the version to the head and deletes the session.

In keys, the space, the punctuation from ! to /, [ and \ are escaped, and a ! marks where each name ends, so a name such as ../report.pdf or a//b is kept exactly as given, even by stores that keep keys as files. The escaping keeps the names' order, so listings come straight from the store's key order: names are listed in Unicode code point order, as with PostgreSQL. That differs from .NET's ordinal order only between characters beyond U+FFFF, such as emoji, and those from U+E000 to U+FFFF.

What stays behind

  • A deleted container keeps its record, marked deleted, and a name whose versions are all deleted keeps an empty head. Removing them safely would need conditional deletes, which few stores have yet. Both are a few bytes each, and are reused if the name is created again.
  • Deleting a container hides everything in it at once, then deletes its objects a thousand at a time. If that stops partway, deleting the container again finishes it, and so does creating a container of that name again. Its versions' expiry entries are kept apart from it, under {prefix}x/, so they stay until they're due, and a sweep then drops them.
  • A write interrupted after its content was uploaded, such as by a crash, leaves that content named by no head, and so does a delete that fails to delete its content. It counts toward the container's stored size until the expiry sweep deletes it: once a day, a sweep reads every head and deletes the content that it and the sweep the day before both found named by no head and over a day old. Two scans are needed because one reads the heads one after another, so a version renamed meanwhile could look unnamed to it. The same daily scan also finishes purging any container whose delete stopped partway.
  • An upload of 8 MiB or more that fails partway, and whose abort also fails, leaves its parts in the store. Amazon S3 keeps them until a lifecycle rule that aborts incomplete multipart uploads removes them, so consider adding one to the bucket. The same goes for a completion whose join fails.
  • A move that stops partway, such as by a crash, leaves its marks and its entry. Every read still finds the versions in one place, at the cost of one more read for a name a move marked, and the first expiry sweep once the move has made no progress for an hour finishes or undoes it. Until then, the resource counts under both names, in each one's container's number of resources. A move across containers writes its entry again every 15 minutes while it copies, so a long copy doesn't make the move look stopped.
  • A deleted container's upload sessions stay under {prefix}u/ until the next expiry sweep, which deletes them. So does a session's segment whose delete failed, once it's an hour old.

Costs

  • An upload smaller than 8 MiB reads the container's record, sends the content in one request, reads and writes the head, then reads the container's record again to catch a container deleted meanwhile. A create-only or conditional upload, which includes the default mode, also reads the head before sending the content. From 8 MiB up, content goes up in 8 MiB parts, each read into a buffer first, so each upload in progress holds 8 MiB of memory. Content larger than 78 GiB, the most one upload's parts hold, goes up in pieces of 78 GiB. The store then joins them, one copy request per GiB, and the pieces are deleted. Until they're joined, a small record under {prefix}p/ lists them, written again every 15 minutes, so the daily scan for unnamed content leaves them alone. An upload holds up to 5 TiB, the most S3 keeps in one object; a larger one throws ResourceUploadLimitException, which ASP.NET Core's handler answers with 413 Content Too Large.
  • A download reads the container's record and the head, then streams the content; a range asks the store for only that range.
  • A listing page reads the container's record, lists the heads, then reads each one, several at a time, since the attributes are inside them.
  • Usage, which container policies check before each write they limit, comes from listings alone: the content objects' sizes, and the number of non-empty heads. That's one list request per thousand objects in the container.
  • An expiry sweep with nothing due is three list requests (of expiry entries, upload sessions and moves) and one read, of the record of the last scan for unnamed content. Once a day, the sweep runs that scan: it lists every container's record, head and content object, and reads every move's entry, record and head, several at a time. Each expired version it finds costs three reads to find it (its index entry, its container's record and its head), two more to delete it (the container's record and the head again), a write of the head, and one delete request for its content and its entry. A version with an expiry costs one more small object, and an upload with an expiry one more write, for its entry. Changing an expiry reads the container's record and the head, writes the new entry (none when clearing it), reads and writes the head, then deletes the old entry.
  • A rename reads both containers' records and both heads, writes the move's entry, marks the source head, writes the destination's, and commits the entry; then it reads and writes each head again to clear the marks, deletes the entry, and reads the destination container's record: about eight reads, six writes and two deletes. A version with an expiry costs one more write, for its new entry. A move to another container also has the store copy each version's content, one request per GiB, and writes the move's entry again every 15 minutes while it copies and after each version; then it deletes the old objects. While a move is unfinished, a read of either name costs one more read, of the move's entry.
  • An upload session costs, to start, a read of the container's record and a write of the session's record. Each append reads both, then for each segment it stores (one per 64 MiB) uploads the segment and reads and writes the record. An append that starts after a segment shorter than 8 MiB first reads that segment back, so many small appends each cost up to 8 MiB more, each way. Completing reads the content once to hash it, as with every provider; then it reads and writes the record to seal it, reads the head, asks the store to copy each segment into place (several at once), and writes the head; then it deletes the session's objects. A session holds up to 5 TiB, the most S3 keeps in one object. The store joins at most 10,000 parts, so once a session's segments take that many, the next append first has the store join those stored since the last such join into one segment. That copies them, though the bytes stay in the store, and each byte is copied about once. With appends in multiples of 64 MiB, the first join comes after about 625 GiB, and later ones sooner, since a joined segment counts as one part for each GiB it holds. An append past 5 TiB throws ResourceUploadLimitException, which ASP.NET Core's handler answers with 413 Content Too Large. The expiry sweep lists every session, and reads the record of each, to find those that nothing was appended to for a day.

Health checks

The generic health check, AddResourcefullCheck(), makes one read through the provider, so it reports whether the store can be reached and the bucket exists and is readable with the credentials given.

Telemetry

Besides the operations every provider records (see Telemetry), the provider counts resourcefull.s3.retries: writes to a name's head, or to an upload session's record, retried because another writer changed it first. A few are normal when writers share names; a steady rise means many writers are contending for the same names. It counts resourcefull.s3.moves_recovered: moves the expiry sweep finished or undid because their mover stopped first, such as after a crash or a lost connection. It counts resourcefull.s3.session_joins: joins of an upload session's segments, made once they took as many parts as the store joins, each of which delays an append while the store copies what was appended since the last join. It also counts resourcefull.s3.stale_expiries: expiry entries a sweep dropped because their version no longer existed or had another expiry; resourcefull.s3.stale_bin_entries: bin and purge entries dropped because their version was no longer in its bin; and resourcefull.s3.abandoned_uploads: upload sessions nothing was appended to for a day, or whose container was deleted, and content no head names, which the expiry sweep deleted. The AWS SDK can also trace its own requests; see its documentation for OpenTelemetry.

Edit this page