Store¶

Overview¶

The high-level Store API implementation transparently deals with nesting and soft deletion, so the caller doesn’t need to care much about that, and the backend API can be much simpler:

create/destroy: initialize or remove the whole store.
list: flat list of the items in the given namespace (by default, only non-deleted items; optionally, only soft-deleted items).
store: write a new item into the store (providing its key/value pair).
load: read a value from the store (given its key); partial loads specifying an offset and/or size are supported.
info: get information about an item via its key (exists, size, …).
hash: computes the hexdigest for the content of an item (given its key). Supported algorithms are all algorithms supported by hashlib (e.g. “sha256”, the default) and also “blake3” (requires the optional blake3 package, see Installation).
delete: immediately remove an item from the store (given its key).
move: implements renaming, soft delete/undelete, and moving to the current nesting level.
defrag: general purpose defragmentation helper (copies blocks to new items). If the target name is computed from the content, the same algorithms as for hash are supported.
quota: return quota limit and usage (-1 if quotas not enabled or not supported)
stats: API call counters, time spent in API methods, data volume/throughput.
latency/bandwidth emulator: see Latency and bandwidth emulator.

Store operations (and per-op timing and volume) are logged at DEBUG log level.

See also Store caching for optional Store-level caching with a secondary backend.

Initialization¶

The Store is the main entry point to the API. It is initialized with a backend URL and an optional configuration dictionary:

from borgstore import Store

store = Store(
    url="file:///abs/path",
    config={
        "data": {"levels": [1]},
        "config": {"levels": [0]},
    },
)

The config dictionary maps namespace names to their configuration. The following keys are supported in each namespace configuration:

levels (required): A list of integers specifying the nesting depths supported for the namespace.

See Store caching for details on caching.

Latency and bandwidth emulator¶

The Store can emulate slower backend behavior using environment variables:

BORGSTORE_LATENCY: per-primary-call latency in microseconds ([us]).
BORGSTORE_BANDWIDTH: primary-call bandwidth limit in bits per second ([bit/s]).

Current behavior with Store caching enabled:

Emulation applies to primary backend operations.
Emulation does not apply to cache backend operations.

This means:

On cache miss paths (for example writethrough/mirror reads that load from the primary backend), emulation affects the primary backend calls.
On cache hit paths, cached reads avoid primary backend load operations and therefore do not incur emulated bandwidth delay for the cache backend read.
Name resolution for Store operations still uses primary backend lookups, therefore configured latency can still be visible even when data comes from cache.

Keys¶

A key (str) can look like:

0123456789abcdef… (usually a long, hex-encoded hash value)
Any other pure ASCII string without ‘/’, ‘..’, or spaces.

Namespaces¶

To keep things separate, keys should be prefixed with a namespace, such as:

config/settings
meta/0123456789abcdef…
data/0123456789abcdef…

Please note:

You should always use namespaces.
Nested namespaces like namespace1/namespace2/key are not supported.
The code can work without a namespace (empty namespace “”), but then you can’t add another namespace later, because that would create nested namespaces.

Values¶

Values can be any arbitrary binary data.

store(name, value) accepts the value either as bytes or as a memoryview of bytes, so a caller that already has the data inside a bigger buffer can avoid copying it:

buffer = bytearray(...)  # e.g. a big buffer, filled with multiple objects
store.store("data/0123456789abcdef", memoryview(buffer)[offset : offset + size])

Notes about memoryview values:

The view must be C-contiguous (a plain slice of a buffer is). Otherwise, a ValueError is raised - the value must be storable as one consecutive sequence of bytes.
If the view has an itemsize > 1 (e.g. a view of an array.array("I", ...)), the bytes it consists of are stored, so the stored size is len(value) * value.itemsize.
The value is only used while store runs. Afterwards, the caller may reuse or release the underlying buffer.
Not all backends can hand a memoryview to the library they use: posixfs, rest and rclone store it without copying, sftp and s3 internally create a bytes object (one copy) first.

Loading always gives bytes.

Automatic Nesting¶

For the Store user, items have names such as:

namespace/0123456789abcdef…
namespace/abcdef0123456789…

If there are very many items in the namespace, this could lead to scalability issues in the backend. The Store implementation therefore offers transparent nesting, so that internally the backend API is called with names such as:

namespace/01/23/45/0123456789abcdef…
namespace/ab/cd/ef/abcdef0123456789…

The nesting depth can be configured from 0 (= no nesting) to N levels and there can be different nesting configurations depending on the namespace.

The Store supports operating at different nesting levels in the same namespace at the same time.

When using nesting depth > 0, the backends assume that keys are hashes (contain hex digits) because some backends pre-create the nesting directories at initialization time to optimize backend performance.

Soft deletion¶

To soft-delete an item (so its value can still be read or it can be undeleted), the store just renames the item, appending “.del” to its name.

Undelete reverses this by removing the “.del” suffix from the name.

Some store operations provide a boolean flag “deleted” to control whether they consider soft-deleted items.

Scalability¶

Count of key/value pairs stored in a namespace: automatic nesting is provided for keys to address common scalability issues.
Key size: there are no special provisions for extremely long keys (e.g., exceeding backend limitations). Usually this is not a problem, though.
Value size: there are no special provisions for dealing with large value sizes (e.g., more than available memory, more than backend storage limitations, etc.). If one deals with very large values, one usually cuts them into chunks before storing them in the store.
Partial loads improve performance by avoiding a full load if only part of the value is needed (e.g., a header with metadata).