Google Cloud Storage

This backend is for storing data in Google Cloud Storage by using the google-cloud-storage library.

google-cloud-storage is only available for Python 3. Minimalkv also provides access to Google Cloud Storage through BotoStore using the boto library which is available for Python 2.

Note that google-cloud-storage is not a dependency for minimalkv. You need to install it manually, otherwise you will see an ImportError.

Here is a short example:

from minimalkv.net.gcstore import GoogleCloudStore

credentials_path = "/path/to/credentials.json"

store = GoogleCloudStore(credentials=credentials_path, bucket_name="test_bucket")

# store some data in the store
store.put("first-key", b"Hello Google Cloud!")

# print out what's behind first-key. You should now see
# the key in the bucket as well
print store.get("first-key")

Testing

The tests for the google cloud storage backend either

  • use a real google cloud storage account

  • use the Fake GCS Server storage emulator

The travis tests use the second method. Caution: Some methods (deleting buckets, copying blobs, …) aren’t implemented in the emulator and should therefore be tested using a real cloud storage account.

To test with a real blob store account, edit the file google_cloud_credentials.ini s.t. the first config section contains the path to the credentials.json of your test account.

To test against a locally running Fake GCS Server instance make sure to start the docker container:

docker run -d --name fake-gcs-server -p 4443:4443 fsouza/fake-gcs-server -scheme http

before running the tests.

class minimalkv.net.gcstore.GoogleCloudStore(credentials, bucket_name: str, create_if_missing: bool = True, bucket_creation_location: str = 'EUROPE-WEST3', project=None)

A store using Google Cloud storage as a backend.

See https://cloud.google.com/storage.

__init__(credentials, bucket_name: str, create_if_missing: bool = True, bucket_creation_location: str = 'EUROPE-WEST3', project=None)

Initialize an FSSpecStore.

The underlying fsspec FileSystem is created when the store is used for the first time.

Parameters

prefix: str, optional

The prefix to use on the FSSpecStore when storing keys.

mkdir_prefix: bool, optional

If True, the prefix will be created if it does not exist. Analogous to the create_if_missing parameter in AzureBlockBlobStore or GoogleCloudStore.

write_kwargs: dict, optional

Additional keyword arguments to pass to the fsspec FileSystem when writing files.

custom_fs: AbstractFileSystem, optional

If given, use this fsspec FileSystem instead of creating a new one.