SecureStore: the open secrets container format

It’s been a while since we first released our SecureStore.NET library for C# and ASP.NET developers back in 2017, as a solution for developers looking for an uncomplicated way of safely and securely storing secrets without needing to build and maintain an entire infrastructure catering to that end. Originally built way back in 2015 to support secrets storage in legacy ASP.NET applications, SecureStore.NET has been since updated for ASP.NET Core and UWP desktop application development, and now we’re proud to announce the release of SecureStore 1.0 with multi-platform and cross-framework support, with an updated schema making a few more features possible and official implementations in C#/.NET and Rust.

What is SecureStore?

SecureStore is a KISS solution for secrets management. We’ve found that the complexity involved in most solutions for secrets management tends to either scare away developers from Doing the Right Thing™ for small- to medium-scale projects or else cause developers to over-complicate things and end up with brittle systems with unnecessary runtime dependencies and points-of-failure.

At one end of the spectrum, it’s unfortunately all-too-common to look through code and find things like this, committed to git repositories (or copy-and-pasted to the code smb share) for the world to see:

GetExpringS3Url("s3://bucket/bar", "aws access id", "aws access key");

At the other end of the spectrum, you have developers deploying simple CRUD webapps with only one ACL per service spinning up one or more EC2 or Google Compute Engine instances and relying on half-a-dozen cloud services to get what amounts to a single S3 access key and a database username and password. Changes to dependencies requiring secret-based authentication mandate synchronizing changes between the code and the secrets server.

We designed SecureStore to be as simple and flexible a means of securing – and deploying – passwords encrypted at rest. SecureStore doesn’t bother trying to separate your passwords from your code, and as a result, doesn’t require you to build massive abstractions and a separate service just to dole out secrets at runtime. Your passwords live in the same repository as your code and are versioned alongside it – just encrypted. At some point, you may outgrow this and need to go full-on secrets-as-a-service but we don’t believe you should pay such a high price until that day comes.

SecureStore was simultaneously designed with two polar opposite use cases in mind: on one hand, providing easy, interactive access for quickly and painlessly setting up a secrets store and adding/removing/updating user secrets; on the other hand, easily and securely decrypting those secrets on servers (or in apps) for production use with as little maintenance and intervention as possible.

As such, in addition to having a choice between using a password or a keyfile to a secure a SecureStore secrets vault, you can also export a keyfile corresponding to your password: password-based encryption/decryption is used when interactively modifying vaults during testing/development and to configure production secrets, while the password-equivalent keyfile can be securely moved out-of-band to the production server(s) where it is used for passwordless decryption of secrets (and the local copy of the keyfile can shredded altogether).¹ This keyfile deployment step is a one-time process: unlike the other secrets that will come, change, and go as your codebase and dependencies evolve, this one keyfile is static and can be deployed just once during your initial infrastructure rollout, however you so choose.

The VCS-friendly SecureStore file format

A SecureStore secrets file is “just” a JSON file containing a mapping of secret names to their encrypted values, but is especially crafted to be version control friendly. It’s always pretty-printed with each secret on a separate line and kept sorted alphabetically (by secret names, which are not considered sensitive data and shouldn’t have any meaning besides being identifiers). The encrypted values don’t change when the secrets are accessed, and each secret is encrypted separately so that adding/removing/changing a secret only affects the lines in question – meaning you can treat your secrets file just like the rest of your code when it comes to your version control system. This should both drastically diminish the need for any conflict resolution when merging changes across branches and make resolving any conflicts that do occur a straightforward and intuitive process.

The secrets themselves are encrypted in as-universally-feasible a format as any out there, with the intention that the secrets file schema should be stable over time and secrets files shareable between clients written in any language for any framework or platform (presently defined as AES-128-CBC after out-of-band HMAC-SHA1 authentication, using two separate keys derived via 256,000 rounds of PBKDF2),² but with all the gory encryption details completely hidden away. Here’s what a sample secrets.json file looks like³ – this is what you’d commit (and possibly deploy) alongside your code, completely safe to include in version control⁴ and probably even OK to publish in a full-page ad in The New York Times (but we wouldn’t recommend you do that, tbh, and it wouldn’t take too much guessing to crack any of the secrets in this particular sample file).

A brand-spanking-new SecureStore crate for the rustaceans…

The biggest news we have today is that we’re going all-in when it comes to cross-platform and multi-language support. SecureStore was very carefully designed from the start to make it easy to port to other platforms – it was important for us that different SecureStore implementations not only share the same conceptual basis, but that a single, shared secrets store should be readable/writeable by clients written in any language for any platform. At NeoSmart Technologies, we develop software that covers virtually the entire spectrum, from bootloaders to kernel drivers and from desktop applications to distributed cloud webservers. For the high level languages, C# and rust are currently our preferred langues du jour for greenfield applications, and so it is only natural that the next SecureStore implementation comes in the form of a rust implementation of both a crate and an easy-to-use CLI utility for managing stores and secrets.

Up first: ssclient, the SecureStore CLI. ssclient (short for SecureStore Client) has been published as a binary crate to crates.io and is installable by executing the following:

cargo install ssclient

After which it can be executed by running ssclient in your shell. While the SecureStore API supports everything that the client does and more, ssclient is a more natural way of interactively creating secrets stores and managing their secrets from the command line. Full documentation for ssclient is available online, but we’ll walk through the process of creating a new store and adding a secret to it below.

The first step in the process is to create a store. There are a few configurable options available but the only thing we need to decide at this point is whether we’re going to use a password or not.⁵ Here’s how you would create a new store (default: secrets.json) in the default password-based encryption mode, while exporting an equivalent keyfile for subsequent passwordless use:

~> ssclient create --export-key secret.key
Password: ***********
Confirm password: ***********

Now we can interact with the store in either password-based or keyfile-based modes to add some secrets, and the secret values themselves can either be provided directly as command line arguments or entered interactively for greater security:

~> ssclient set aws:sns:accessId AKIAIOSFODNN7
Password: ***********
~> ssclient set --key secret.key aws:sns:accessKey
Value: wJalrXUtnFEMI/K7MDENG/bPxRfiCY

While we created one secret with the password and the other with the password-derived keyfile, either of the two can be used to decrypt any (or even export all) of the secrets:

~> ssclient get aws:sns:accessId -k secret.key
AKIAIOSFODNN7
~> ssclient get aws:sns:accessKey
Password: ***********
wJalrXUtnFEMI/K7MDENG/bPxRfiCY

Next we’ll continue with examples on exporting all secrets in multiple formats and updating or deleting individual secrets with the C# and .NET Core client to demonstrate the cross-platform capabilities of SecureStore and the benefit of an open, human-readable file format for secrets management. If you’re not interested in that and just want the rust goodies, you can refer to the ssclient documentation or even watch an asciicast demo of its use.

… a rust crate for decrypting secrets vaults in code

Of course creating the secrets file and interactively adding secrets to it is only half the puzzle – there’s no point to storing secrets if we’re not going to use them. The securestore crate provides the answer to the second half of that puzzle.

After first adding securestore to Cargo.toml, we’ll be ready to begin:

[dependencies]
securestore = "0.99"
…

Presuming we want to use the keyfile secret.key we exported above with the client to interact with the secrets.json user secrets file in a passwordless fashion and that we already have access to both on the local filesystem, nothing could be easier:

use securestore::{KeySource, SecretsManager};

fn get_api_key() -> String {
    // It's probably a good idea to unwrap and panic if
    // we're not able to access the access keys in production..
    // or at least for the sake of this example, anyway.
    let sman = SecretsManager::load("secrets.json",
                KeySource::File("secret.key")).unwrap();
    let sns_id = sman.get("aws:sns:accessId").unwrap();

    assert_eq!("AKIAIOSFODNN7", sns_id.as_str());
    return sns_id;
}

The API is extensively documented and also self-documenting,⁶ it’s also generally stable as it was modeled around the original C# implementation of SecureStore which has been stabilized over the past five years.⁷ The crate also has full support for serializing/deserializing other secret types including &[u8] and Vec (think binary client or server certificates), or even your own custom types (via the BinarySerializable and BinaryDeserializable traits), should you need to.

Plus a new .NET Core tool for manage secret stores from the command line

When SecureStore was first introduced in 2017, it was actually released without a frontend of any sort – just a library. That was a gross oversight that was remedied shortly thereafter, but with this release we have a much-improved SecureStore that’s even easier to get started with so as to remove the last remaining bit of friction left in the secrets management process. With today’s release, SecureStore not only has a CLI interface that can be used to quickly and interactively create stores, export keys, and manage secrets, but also a client that integrates well with the .NET Core and ASP.NET Core ecosystem as a whole.

The SecureStore command line client is packaged and released for use as a dotnet tool, meaning you no longer have download and build or install the ssclient utility manually. Getting started with SecureStore for .NET is as easy and natural as using dotnet itself in your terminal:

> dotnet tool install --global SecureStore.Client
You can invoke the tool using the following command: SecureStore
Tool 'securestore.client' (version '1.0.3') was successfully installed.

After which you can call up the client at any time by just executing SecureStore in your shell; e.g. here’s an example on creating a new store at the command line:

> SecureStore create secrets2.json --password --keyfile secret.key
Password: ***********
Confirm password: ***********

By using both --password and --keyfile we are creating a dual-decryptable store where we can use either the password we entered interactively or the keyfile exported to the path specified (secret.key, in this case) interchangeably to manage or retrieve secrets.⁸ But our primary demonstration will actually pick up where our previous rust ssclient example left off, showing how the SecureStore open secrets format makes it easy to centralize your secrets between the various components of the same project, effortlessly consuming secret stores created from one in the other, or even modifying stores back-and-forth between the different implementations with full binary compatibility.⁹

Here we’ll show how the dotnet client can open the store previously created with ssclient to retrieve and decrypt a single secret or even export them all:¹⁰

> SecureStore --store secrets.json get aws:sns:accessId
Password: ***********
AKIAIOSFODNN7

Next, let’s export the entire contents of the store. Why? Perhaps a better secrets format was created and you want to move to greener pastures, or perhaps it’s time to move to a full-blown secrets-as-a-service infrastructure and all the headache entailed. Regardless, it’s your right to be able to do so:

~> SecureStore get --all -k secret.key

{
  "aws:s3:accessId": "AKIAIOSFODNN7",
  "aws:s3:accessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCY",
}

Now, let’s delete the accessId (decrypting with the keyfile this time):

~> SecureStore delete -k secret.key aws:s3:accessId
~> SecureStore get aws:s3:accessId -k secret.key
Key "aws:s3:accessId" not found in secrets store!

Afterwards, let’s take a peek at the results, first by looking at the secrets.json file itself then by exporting the contents again:

> cat secrets.json

{
  "version": 2,
  "iv": "5ib8ZQCvPJMRIphD5cK96w==",
  "sentinel": {
    "iv": "mOvY3SWH05FpVyIeFq4Ayw==",
    "hmac": "dKI0lWZ2IiHs20JE/X+iYcgzbd0=",
    "payload": "PxiXZLHvrX8gZtxwvy6yjb7V4ZjW8jCxtQzZRpJzK5L6ojOB14jehf+pY+966YMF"
  },
  "secrets": {
    "aws:s3:accessKey": {
      "iv": "0ohSrOLPVaywPqTYZv4jyQ==",
      "hmac": "GlIGDPw8a/9Znbx+OdzZnVR/4ko=",
      "payload": "oFGCMNp8qmuhy3Ql0GHiz9ZManEK2Pei+b4jOeAA/4o="
    }
  }
}

> SecureStore get -k secret.key --all --output-format text
aws:s3:accessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCY

By default, SecureStore always exports to pretty-printed json because it’s a human-readable format with clear escaping rules. Yes, you need a dependency on a JSON parser if the coding language you’re writing the importer in doesn’t come with one, but on the flip side, you won’t have to figure out what’s part of a secret and what’s not. There are no constraints on what can or can’t be a valid secret value in the real world, so the client won’t artificially constrain you, either.¹¹

And finally, an updated SecureStore.NET nuget library

The SecureStore.NET implementation for C# and other dotnet targets has also been updated (nuget link). In recognition of the fact that the majority of secrets are not arbitrary types and to improve cross-platform compatibility, secrets themselves are no longer serialized to JSON prior to encryption; instead, a custom serializer is implemented for the basic types (strings, numbers, and binary enumerables). The option to use a custom serializer is still available by setting the DefaultSerializer property on the SecretsManager object to an instance of ISecretSerializer, and a Utf8JsonSerializer class is included that implements the old behavior.¹²

Although it is explicitly not a goal for the library to provide any form of safety in the event the machine itself is compromised (either physically or with root/admin privileges), the C# implementation of SecureStore has also been updated to use SecureBuffer, our binary variant of SecureString, for storing binary secrets in memory; SecureBuffer is basically a GC-pinned byte[] implementing IDisposable. It can be instantiated in one of two modes: in the secure mode a new byte[] is first allocated and pinned before any secure content is written to it, in the second variant an existing byte[] that already contains sensitive data can be passed in to the SecureBuffer constructor.¹³ When the object is disposed, the memory is first overwritten with random data then unpinned. This is strictly a best-effort experiment that may be removed in the future and should not be relied upon to provide any meaningful security.

Core usage of the actual SecureStore.NET API has not changed from the last release:

using NeoSmart.SecureStore;

string GetSecret()
{
    // Presuming a secrets file has already been created with the CLI utility:
    using var sman = SecretsManager.LoadStore("secrets.json");
    sman.LoadKeyFromFile("secure/secrets.key");

    if (!sman.TryGetValue("secretName", out string secretValue))
    {
        throw new Exception("Cannot find crucial secret value!!!");
    }

    // Or we could have just done this, if we were going to throw anyway:
    secretValue = sman.Get("secretName");

    return secretValue;
}

Please note that this release does increment the schema version for the store; in addition to updating the format of secrets.json, if you were previously using a keyfile derived from a password please make sure to re-export the key for continued passwordless access. If managing secrets programmatically, stores are automatically upgraded in memory when and where possible and SecretsManager.Upgraded can be used to determine if it is necessary to write these changes back to disk (and you should do this even if you’re performing read-only operations).

That just about covers the most important talking points for this release. We’ll be continuing to improve the documentation for the SecureStore.NET library and client in the repository as well as updating the aliases for command line arguments to the SecureStore client to provide parity with the rust-based ssclient, but that is wholly independent from the API and store formats. As for the rust crate, we will wait to hear back from the community regarding the API design and hopefully push out a 1.0 soon.

We hope that together with the community we can grow SecureStore to even more languages and frameworks and help improve security for developers everywhere. If you have any feedback or suggestions, feel free to comment below or open an issue on one of the two GitHub repos (rust and .NET). If you have any questions or comments, please leave a comment below so users of both libraries can benefit.

The libraries are released under the MIT and Apache 2.0 public licenses – hopefully you will find them useful! We’re actively accepting improvements to the documentation and the code over at GitHub. If you’re interested in porting SecureStore to another language or framework, let us know if we can help and so we can send some traffic and karma your way! If you’re a security professional with feedback or an audit, we are humbled and open to constructive criticism!

One last thing: we’re experimenting with setting up dedicated, topic-specific mailing lists rather than a catch-all mailing list with all our posts. Currently we have one for rust dev and one for .NET dev, so if you’re interested in getting a ping when we have new dev resources to share, consider putting your email in one of the two forms below. You can also (please!) follow us on twitter @neosmart (and the author of this article @mqudsi).

1. You can also bring-your-own-keys or have SecureStore generate them for you via a CSPRNG. ↩

2. Yes, there are more modern options such as AES-256-GCM and scrypt, but for the specific use cases SecureStore is designed for, a separate round of HMAC-SHA1 authentication suffices given the expected maximum length of individual secrets, and the output of the key stretching function is not used as a hash as in the common case, but rather as a full-on encryption/decryption key to be as closely guarded as the secret itself, meaning there’s nothing to brute force compare it against, at least not directly. We do not want to make it too onerous to access SecureStore files on different platforms or from different frameworks/languages, where access to the latest and greatest key-stretching or authenticated encryption routines may not be available. ↩

3. Don’t worry, you can name it whatever you like ↩

4. But make sure to revoke secrets when they’re not used or any SecureStore keyfiles our passwords are compromised, as your repo history lives forever! ↩

5. A keyfile can always be generated later from a password, but not the other way around. ↩

6. In the sense that “if it compiles, you’re probably using it right;” with the exception of a particular edge case that is still technically correct although with the undesirable side effect of never having the right decryption key, as KeySource is intentionally not split into ExistingKeySource and NewKeySource traits as it was judged that the obvious incongruence of using a newly generated key to decrypt an existing store was not worth doing away with the simplicity, clarity, and ease of static dispatch.. at least until RFC 2593 gets figured out and rust has a way to treat individual enum variants as types in their own right. ↩

7. The only reason we are not committing to a 1.0 just yet is rust’s lack of an answer for default types for generic functions.. in some contexts you’d have to explicitly use sman.get::<String>("secretName") if the compiler can’t deduce the type, which isn’t the prettiest. ↩

8. The command-line syntax for the rust ssclient and the .NET SecureStore is a bit different, but we’ll be working on making them compatible with a future release. It’s just a matter of brevity/verbosity, inspired by the languages themselves. ↩

9. i.e. even the whitespace is preserved across the implementations, so that there are minimal diffable changes as a result of cross-client store manipulation. ↩

10. We don’t believe in lock-in, which is why the secrets container format is purposely extremely simple and parseable and why all client implementations offer full text and json export support, but please be careful when exporting your secrets! ↩

11. Technically the name could at least be constrained to exclude certain characters like " or : but somewhat along the lines of having your SSL certificates expire early and often and using UTF-8 instead of UTF-16, this forces you to deal with the fact that you can’t just arbitrarily split the string and call it a day. ↩

12. Compiling the library with JSON_SERIALIZER defined will restore the old behavior by default. A future release may drop the dependency on Newtonsoft.Json on platforms with the in-box System.Text.Json to bring down the dependency count further. ↩

13. In the latter case, you must combine this with plenty of hopes and prayers or else you may have multiple copies of the sensitive data floating around the memory from various GC actions, only the most recent of which will be sanitized. ↩