minimalkv package¶

Submodules¶

Module contents¶

class minimalkv.CopyMixin¶

Bases: object

Mixin to expose a copy operation supported by the backend.

copy(source: str, dest: str) → str¶

Copy data at key source to key dest.

The destination is overwritten if it does exist.

Parameters¶

sourcestr: The source key of data to copy.
deststr: The destination for the copy.

Raises¶

ValueError: If source is not a valid key.
ValueError: If dest is not a valid key.

Returns¶

keystr: The destination key.

move(source: str, dest: str) → str¶

Move data from key source to key dest.

The destination is overwritten if it does exist.

Parameters¶

sourcestr: The source key of data to be moved.
deststr: The destination for the data.

Raises¶

ValueError: If source is not a valid key.
ValueError: If dest is not a valid key.

Returns¶

keystr: The destination key.

class minimalkv.KeyValueStore¶

Bases: object

Class to access a key-value store.

Supported keys are ascii-strings containing alphanumeric characters or symbols out of minimalkv._constants.VALID_NON_NUM of length not greater than 250. Values (or records) are stored as raw bytes.

close()¶

Clean up all open resources in child classes.

Specific store implementations might require teardown methods (dangling ports, unclosed files). This allows calling close also for stores, which do not require this.

delete(key: str) → str | None¶

Delete data at key.

Does not raise an error if the key does not exist.

Parameters¶

key: str: The key of data to be deleted.

Raises¶

ValueError: If the key is not valid.
IOError: If there was an error deleting.

get(key: str) → bytes¶

Return data at key as a bytestring.

Parameters¶

keystr: The key to be read.

Returns¶

datastr: Value associated with the key as a bytes object.

Raises¶

ValueError: If the key is not valid.
IOError: If the file could not be read.
KeyError: If the key was not found.

get_file(key: str, file: str | BinaryIO) → str¶

Write data at key to file.

Like put_file(), this method allows backends to implement a specialized function if data needs to be written to disk or streamed.

If file is a string, contents of key are written to a newly created file with the filename file. Otherwise, the data will be written using the write method of file.

Parameters¶

keystr: The key to be read.
fileBinaryIO or str: Output filename or file-like object with a write method.

Raises¶

ValueError: If the key is not valid.
IOError: If there was a problem reading or writing data.
KeyError: If the key was not found.

iter_keys(prefix: str = '') → Iterator[str]¶

Iterate over all keys in the store starting with prefix.

Parameters¶

prefixstr, optional, default = ‘’: Only iterate over keys starting with prefix. Iterate over all keys if empty.

Raises¶

IOError: If there was an error accessing the store.

iter_prefixes(delimiter: str, prefix: str = '') → Iterator[str]¶

Iterate over unique prefixes in the store up to delimiter, starting with prefix.

If prefix contains delimiter, return the prefix up to the first occurence of delimiter after the prefix.

The default uses an naive key iteration. Some backends may implement more efficient methods.

Parameters¶

delimiterstr, optional, default = ‘’: Delimiter up to which to iterate over prefixes.
prefixstr, optional, default = ‘’: Only iterate over prefixes starting with prefix.

Raises¶

IOError: If there was an error accessing the store.

keys(prefix: str = '') → List[str]¶

List all keys in the store starting with prefix.

Parameters¶

prefixstr, optional, default = ‘’: Only list keys starting with prefix. List all keys if empty.

Raises¶

IOError: If there was an error accessing the store.

open(key: str) → BinaryIO¶

Open record at key.

Parameters¶

keystr: Key to open.

Returns¶

file: BinaryIO: Read-only file-like object for reading data at key.

Raises¶

ValueError: If the key is not valid.
IOError: If the file could not be read.
KeyError: If the key was not found.

put(key: str, data: bytes) → str¶

Store bytestring data at key.

Parameters¶

keystr: The key under which the data is to be stored.
databytes: Data to be stored at key, must be of type bytes.

Returns¶

str: The key under which data was stored.

Raises¶

ValueError: If the key is not valid.
IOError: If storing failed or the file could not be read.

put_file(key: str, file: str | BinaryIO) → str¶

Store contents of file at key.

Store data from a file into key. file can be a string, which will be interpreted as a filename, or an object with a read() method.

If file is a filename, the file might be removed while storing to avoid unnecessary copies. To prevent this, pass the opened file instead.

Parameters¶

keystr: Key where to store data in file.
fileBinaryIO or str: A filename or a file-like object with a read method.

Returns¶

key: str: The key under which data was stored.

Raises¶

ValueError: If the key is not valid.
IOError: If there was a problem moving the file in.

class minimalkv.TimeToLiveMixin¶

Bases: object

Mixin to allow keys to expire after a certain amount of time.

This mixin overrides some of the signatures of the api of KeyValueStore, albeit in a backwards compatible manner.

Any value given for a time-to-live parameter must be one of the following:

A positive int or float, representing seconds,
minimalkv._constants.FOREVER, meaning no expiration
minimalkv._constants.NOT_SET, meaning that no TTL configuration will be done at all or
None representing the default (see TimeToLiveMixin’s default_ttl_secs).

Note

When deriving from TimeToLiveMixin, the same default implementations for _put, _put_file and _put_filename are provided, except that they all take an additional ttl_secs argument. For more information on how to implement backends, see Implementing a new backend.

default_ttl_secs = 'not_set'¶

put(key: str, data: bytes, ttl_secs: float | int | str | None = None) → str¶

Store bytestring data at key.

If ttl_secs is a positive number, the key will expire after ttl_secs. Other possible values for ttl_secs are minimalkv._constants.FOREVER (no expiration) and minimalkv._constants.NOT_SET (no TTL configuration). None will be replaced with minimalkv._mixins.TimeToLiveMixin.default_ttl_secs.

Parameters¶

keystr: The key under which the data is to be stored.
databytes: Data to be stored at key, must be of types bytes.
ttl_secsnumeric or str: Number of seconds until the key expires.

Returns¶

key: str: The key under which data was stored.

Raises¶

ValueError: If the key is not valid.
IOError: If storing failed or the file could not be read.
ValueError: If ttl_secs is invalid.

put_file(key: str, file: str | BinaryIO, ttl_secs: float | int | str | None = None) → str¶

Store contents of file at key.

Store data from a file into key. file can be a string, which will be interpreted as a filename, or an object with a read() method.

If file is a filename, the file might be removed while storing to avoid unnecessary copies. To prevent this, pass the opened file instead.

If ttl_secs is a positive number, the key will expire after ttl_secs. Other possible values for ttl_secs are minimalkv._constants.FOREVER (no expiration) and minimalkv._constants.NOT_SET (no TTL configuration). None will be replaced with minimalkv._mixins.TimeToLiveMixin.default_ttl_secs.

Parameters¶

keystr: Key where to store data in file.
fileBinaryIO or str: A filename or an object with a read method.
ttl_secsstr or numeric or None, optional, default = None: Number of seconds until the key expires.

Returns¶

key: str: The key under which data was stored.

Raises¶

ValueError: If the key is not valid.
IOError: If there was a problem moving the file in.
ValueError: If ttl_secs is invalid.

ttl_support = True¶

Indicates that a key-value store supports time-to-live features. This allows users of stores to test for support using:

getattr(store, 'ttl_support', False)

class minimalkv.UrlKeyValueStore¶

Bases: UrlMixin, KeyValueStore

Class is deprecated. Use the UrlMixin instead.

Deprecated since version 0.9.

class minimalkv.UrlMixin¶

Bases: object

Mixin to support getting a download URL for keys.

url_for(key: str) → str¶

Return a full external URL that can be used to retrieve data at key.

Only checks whether key is valid.

Parameters¶

keystr: The key for which to generate a url for.

Returns¶

url: str: A string containing a URL to access data at key.

Raises¶

ValueError: If the key is not valid.

minimalkv.create_store(type: str, params: Dict[str, Any]) → KeyValueStore¶: Create store of type type with params.

minimalkv.decorate_store(store, decoratorname)¶

minimalkv.get_store(type: str, create_if_missing: bool = True, **params: Any) → KeyValueStore¶

Return a storage object according to the type and additional parameters.

The type must be one of the types below, where each allows requires different parameters:

"azure": Returns a minimalkv.azure.AzureBlockBlobStorage. Parameters are "account_name", "account_key", "container", "use_sas" and "create_if_missing" (default: True). "create_if_missing" has to be False if "use_sas" is set. When "use_sas" is set, "account_key" is interpreted as Shared Access Signature (SAS) token.FIRE "max_connections": Maximum number of network connections used by one store (default: 2). "socket_timeout": maximum timeout value in seconds (socket_timeout: 200). "max_single_put_size": max_single_put_size is the largest size upload supported in a single put call. "max_block_size": maximum block size is maximum size of the blocks(maximum size is <= 100MB)
"s3": Returns a plain minimalkv.net.botostore.BotoStore. Parameters must include "host", "bucket", "access_key", "secret_key". Optional parameters are
- "force_bucket_suffix" (default: True). If set, it is ensured that the bucket name ends with -<access_key> by appending this string if necessary; If False, the bucket name is used as-is.
- "create_if_missing" (default: True ). If set, creates the bucket if it does not exist; otherwise, try to retrieve the bucket and fail with an IOError.
"hs3" returns a variant of minimalkv.net.botostore.BotoStore that allows “/” in the key name. The parameters are the same as for "s3"
"gcs": Returns a minimalkv.net.gcstore.GoogleCloudStore. Parameters are "credentials", "bucket_name", "bucket_creation_location", "project" and "create_if_missing" (default: True).
- "credentials": either the path to a credentials.json file or a google.auth.credentials.Credentials object
- "bucket_name": Name of the bucket the blobs are stored in.
- "project": The name of the GCStorage project. If a credentials JSON is passed then it contains the project name and this parameter will be ignored.
- "create_if_missing": [optional] Create new bucket to store blobs in if "bucket_name" doesn’t exist yet. (default: True).
- "bucket_creation_location": [optional] If a new bucket is created (create_if_missing=True), the location it will be created in. If None then GCloud uses a default location.
"hgcs": Like "gcs" but “/” are allowed in the keynames.
"fs": Returns a minimalkv.fs.FilesystemStore. Specify the base path as “path” parameter.
"hfs" returns a variant of minimalkv.fs.FilesystemStore that allows “/” in the key name. The parameters are the same as for "file".
"memory": Returns a DictStore. Doesn’t take any parameters
"redis": Returns a RedisStore. Constructs a StrictRedis using params as kwargs. See StrictRedis documentation for details.

Parameters¶

typestr: Type of storage to open, with optional storage decorators.
create_if_missingbool, optional, default = True: Create the “root” of the storage (Azure container, parent directory, S3 bucket, etc.). Has no effect for stores where this makes no sense, like redis or memory.
kwargs: Parameters specific to the store type.

Returns¶

store: KeyValueStore: Key value store of type type as described in kwargs parameters.

minimalkv.get_store_from_url(url: str, store_cls: Type[KeyValueStore] | None = None) → KeyValueStore¶

Take a URL and return a minimalkv store according to the parameters in the URL.

Parameters¶

urlstr: Access-URL, see below for supported formats.
store_clsOptional[Type[KeyValueStore]]: The class of the store to create. If the URL scheme doesn’t match the class, a ValueError is raised.

Returns¶

storeKeyValueStore: KeyValueStore as described in url.

Notes¶

User credentials like secret keys have to be percent-encoded before they can be used in a URL (see azure and s3 store types), since they can contain characters that are not valid in this part of a URL, like forward-slashes.

You can use Python to percent-encode your secret key on the commandline like so:

$ python -c "import urllib; print urllib.quote_plus('''dead/beef''')"
dead%2Fbeef

Store types and URL forms:

DictStore: memory://

RedisStore: redis://[[password@]host[:port]][/db]

FilesystemStore: fs://path

BotoStore s3://access_key:secret_key@endpoint/bucket[?create_if_missing=true]

AzureBlockBlockStorage: azure://account_name:account_key@container[?create_if_missing=true]

AzureBlockBlockStorage (SAS): azure://account_name:shared_access_signature@container?use_sas&create_if_missing=false

AzureBlockBlockStorage (SAS): azure://account_name:shared_access_signature@container?use_sas&create_if_missing=false[?max_connections=2&socket_timeout=(20,100)]

AzureBlockBlockStorage (SAS): azure://account_name:shared_access_signature@container?use_sas&create_if_missing=false[?max_connections=2&socket_timeout=(20,100)][?max_block_size=4*1024*1024&max_single_put_size=64*1024*1024]

GoogleCloudStorage: gcs://<base64 encoded credentials JSON>@bucket_name[?create_if_missing=true][&bucket_creation_location=EUROPE-WEST1]

S3FSStore s3://access_key:secret_key@endpoint/bucket[?create_if_missing=true][&region_name=...]

See the respective store’s _from_parsed_url() function for more details.

minimalkv.url2dict(url: str, raise_on_extra_params: bool = False) → Dict[str, Any]¶

Create dictionary with parameters from url.

Parameters¶

urlstr: Access-URL, see below for supported forms.
raise_on_extra_paramsbool, optional, default = False: Whether to raise on unexpected params.

Returns¶

paramsdict: Parameter dictionary suitable for get_store()

Note¶

Supported formats:: memory:// redis://[[password@]host[:port]][/db] fs://path s3://access_key:secret_key@endpoint/bucket[?create_if_missing=true] azure://account_name:account_key@container[?create_if_missing=true][?max_connections=2] azure://account_name:shared_access_signature@container?use_sas&create_if_missing=false[?max_connections=2&socket_timeout=(20,100)] azure://account_name:shared_access_signature@container?use_sas&create_if_missing=false[?max_connections=2&socket_timeout=(20,100)][?max_block_size=4*1024*1024&max_single_put_size=64*1024*1024] gcs://<base64 encoded credentialsJSON>@bucket_name[?create_if_missing=true][?bucket_creation_location=EUROPE-WEST1]

minimalkv package¶

Subpackages¶

Submodules¶

Module contents¶

Parameters¶

Raises¶

Returns¶

Parameters¶

Raises¶

Returns¶

Parameters¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Raises¶

Parameters¶

Raises¶

Parameters¶

Raises¶

Parameters¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Raises¶

Parameters¶

Returns¶

Parameters¶

Returns¶

Notes¶

Parameters¶

Returns¶

Note¶