Add persisted parameters to CLI applications in .NET

August 26th, 2022
Console applications with Spectre.Console dotnet cli console
Use System.CommandLine to write .NET global tools. Copy-paste-driven development with copy-paster. dotnet cli console
Develop Clean Command Line Applications with System.CommandLine. Clean CLI. dotnet cli console

TL;DR

See how to implement persisted parameters feature in a CLI/console application. For example, Azure CLI offers persisted parameters that enable you to store parameter values for continued use. You will learn how to use System.CommandLine. It provides building blocks that make functionality composable and reusable.

Source code: https://github.com/NikiforovAll/cli-persistent-parameters-example

Introduction

Persisted parameters improve the overall developer experience. Since the previous values are stored and are ready to be re-used during the next command run. It is easy to understand the benefit of this feature by looking at how az does it:

# Turn persisted parameters on.
az config param-persist on

# Create a resource group.
az group create --name RGName --location westeurope

# Create an Azure storage account in the resource group omitting "--location" and "--resource-group" parameters.
az storage account create \
  --name sa3fortutorial \
  --sku Standard_LRS

As you might assume, there is some sort of state involved. So before we take a look at persisted parameters implementation let’s look at how to implement a configuration in general.

Both Azure CLI and AWS CLI utilize a dedicated file created in a well-known directory. The format of the file is intended to be lightweight and human-readable. Ini markup language will do.

For example:

  • ~/.azure/config is used by Azure CLI
  • ~/.aws/config and ~/.aws/credentials are used by AWS CLI

Here is what ~/.azure/config can look like:

───────┬────────────────────────────────────────────────────────────────────────
       │ File: /home/oleksii_nikiforov/.azure/config
───────┼────────────────────────────────────────────────────────────────────────
   1   │ [cloud]
   2   │ name = AzureCloud
   3   │ 
   4   │ [core]
   5   │ first_run = yes
   6   │ output = jsonc
   7   │ only_show_errors = false
   8   │ error_recommendation = on
   9   │ no_color = True
  10   │ disable_progress_bar = false
  11   │ collect_telemetry = no
───────┴────────────────────────────────────────────────────────────────────────

Implement configuration storage

My goal is to replicate the behavior of az CLI and make the implementation reusable, so you can just grab the code and add it to your application.

Let’s look at commands provided by az CLI to work with the configuration.

> az config -h

Group
    az config : Manage Azure CLI configuration.
        Available since Azure CLI 2.10.0.
        WARNING: This command group is experimental and under development. Reference and support
        levels: https://aka.ms/CLI_refstatus

Subgroups:
    param-persist : Manage parameter persistence.

Commands:
    get           : Get a configuration.
    set           : Set a configuration.
    unset         : Unset a configuration.

At the high level, the structure could be defined as follows:

// Program.cs
var root = new RootCommand();
root.Name = "clistore";

root.AddConfigCommands();

// ConfigCommands.cs
public static RootCommand AddConfigCommands(this RootCommand root)
{
    var command = new Command("config", "Manage CLI configuration");

    command.AddCommand(BuildGetCommand());
    command.AddCommand(BuildSetCommand());
    command.AddCommand(BuildUnsetCommand());

    root.AddCommand(command);
    return root;
}

private static Command BuildGetCommand()
{
    var get = new Command("get", "Get a configuration");
    var getpath = new Argument<string?>(
        "key",
        () => default,
        @"The configuration to get. If not provided, all sections and configurations
will be listed. If `section` is provided, all configurations under the
specified section will be listed. If `<section>.<key>` is provided, only
the corresponding configuration is shown.");
    get.AddArgument(getpath);

    return get;
}

private static Command BuildSetCommand(CliConfigurationProvider configurationProvider)
{
    var set = new Command("set", "Set a configuration");
    var setpath = new Argument<string[]>(
        "key",
        "Space-separated configurations in the form of <section>.<key>=<value>.");
    set.AddArgument(setpath);

    return set;
}

private static Command BuildUnsetCommand(CliConfigurationProvider configurationProvider)
{
    var unset = new Command("unset", "Unset a configuration");

    var unsetpath = new Argument<string[]>(
        "key",
        "The configuration to unset, in the form of <section>.<key>.");
    unset.AddArgument(unsetpath);

    return unset;
}

Now, we want the actual code that handles commands. But before that, we need to grab the configuration from someplace. In my opinion, we already have the required and de-facto standard way of working with configuration namely - IConfiguration. The only this is left is to use Dependency Injection capabilities of System.CommandLine and define BinderBase<IConfiguration>. See https://docs.microsoft.com/en-us/dotnet/standard/commandline/dependency-injection for more details.

public class CliConfigurationProvider : BinderBase<IConfiguration>
{
    public static CliConfigurationProvider Create(string storeName = "clistore") =>
        new(storeName);

    public CliConfigurationProvider(string storeName) => StoreName = storeName;

    public string StoreName { get; }

    public string ConfigLocationDir => Path.Combine(
        Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
        $".{StoreName.TrimStart('.')}");

    public string ConfigLocation => Path.Combine(ConfigLocationDir, "config");

    protected override IConfiguration GetBoundValue(BindingContext bindingContext) => GetConfiguration();

    public IConfiguration GetConfiguration()
    {
        var configuration = new ConfigurationBuilder()
            .AddIniFile(ConfigLocation, optional: true)
            .AddEnvironmentVariables(StoreName.ToUpperInvariant())
            .Build();

        return configuration;
    }
}

As you may notice, we are using ConfigurationBuilder to compose a configuration not only from a configuration file but also from prefix environment variables. As result, we can override parameters, e.g.: CLISTORE_MyConfigKey.

The config get command handler. Note, the IConfiguration is injected as parameter:

var get = new Command("get", "Get a configuration");
var getpath = new Argument<string?>("key");
get.AddArgument(getpath);

get.SetHandler((string? path, IConfiguration configuration) =>
{
    var output = new Dictionary<string, object[]>();

    foreach (var config in configuration.GetChildren())
    {
        output[config.Key] = config.GetChildren()
            .Select(x => new { Name = x.Key, x.Value })
            .ToArray();
    }

    if (output.Any())
    {
        Console.WriteLine(JsonSerializer.Serialize(output, new JsonSerializerOptions()
        {
            WriteIndented = true,
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        }));
    }

    return Task.CompletedTask;
}, getpath, CliConfigurationProvider.Create(root.Name));

return get;

On the write side, we need to be able to manipulate config file. I will skip parsing details. Basically, we load the file from known location and can call Save(path) when we are ready.

public class CliConfigurationProvider : BinderBase<IConfiguration>
{
    // ... skipped
    public IniFile LoadIniFile()
    {
        var ini = new IniFile();

        Directory.CreateDirectory(ConfigLocationDir);

        if (File.Exists(ConfigLocation))
        {
            ini.Load(ConfigLocation);
        }

        return ini;
    }
}

The config set command handler.

var configurationProvider = CliConfigurationProvider.Create(root.Name);
var set = new Command("set", "Set a configuration");
var setpath = new Argument<string[]>("key");
set.AddArgument(setpath);

set.SetHandler((string[] path) =>
{
    var ini = configurationProvider.LoadIniFile();

    foreach (var p in path)
    {
        var keyvalue = p.Split('=');
        var (key, value) = (keyvalue[0], keyvalue[^1]);

        var sectionKey = key[..key.IndexOf('.')];
        var configKey = key[(key.IndexOf('.') + 1)..];
        ini[sectionKey][configKey] = value;
    }
    ini.Save(configurationProvider.ConfigLocation);
    return Task.CompletedTask;
}, setpath);

return set;

Test configuration commands

We can use Verify to perform snapshot testing and check for the correct output of the program. In order to make things easier and simplify working with process output capturing and invocation, I used CliWrap.

Here are some tests. I included *.verfied.txt content as comments. For more details please refer to Tests.

[UsesVerify]
public class StoreCommands_Specs
{
    // kinda ugly, but I can live with it
    private const string relativeSourcePath = "../../../../../src";

    public StoreCommands_Specs()
    {
        // cleanup, runs every test, concurrent test execution is disabled
        EnsureDeletedConfigFolder();
    }

    [Fact]
    public async Task Help_text_is_displayed_for_config()
    {
        var stdOutBuffer = Execute("config", "--help");

        await Verify(stdOutBuffer.ToString());
    }
    // Description:
    // Manage CLI configuration
    // Usage:
    // clistore config [command] [options]
    // Options:
    // -?, -h, --help  Show help and usage information
    // Commands:
    // get <key>    Get a configuration []
    // set <key>    Set a configuration
    // unset <key>  Unset a configuration

    [Fact]
    public async Task Get_config_is_performed_on_populated_config()
    {
        Execute("config", "set", "core.target=my_value");
        Execute("config", "set", "core.is_populated=true");
        Execute("config", "set", "extra.another_section=false");
        var stdOutBuffer = Execute("config", "get");

        await Verify(stdOutBuffer.ToString());
    }
    // {
    // "core": [
    //     {
    //     "name": "is_populated",
    //     "value": "true"
    //     },
    //     {
    //     "name": "target",
    //     "value": "my_value"
    //     }
    // ],
    // "extra": [
    //     {
    //     "name": "another_section",
    //     "value": "false"
    //     }
    // ]
    // }

    private static (CommandResult, StringBuilder, StringBuilder) Execute(params string[] command)
    {
        var stdOutBuffer = new StringBuilder();
        var stdErrBuffer = new StringBuilder();

        var result = Cli.Wrap("dotnet")
            .WithStandardOutputPipe(PipeTarget.ToStringBuilder(stdOutBuffer))
            .WithStandardErrorPipe(PipeTarget.ToStringBuilder(stdErrBuffer))
            .WithArguments(args => args
                .Add("run")
                .Add("--project")
                .Add(relativeSourcePath)
                .Add("--")
                .Add(command))
            .WithValidation(CommandResultValidation.None)
            .ExecuteAsync().ConfigureAwait(false).GetAwaiter().GetResult();

        return stdOutBuffer;
    }

    private static void EnsureDeletedConfigFolder()
    {
        var path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
            ".clistore",
            "config");
        if (File.Exists(path))
        {
            File.Delete(path);
        }
    }
}

Here is the content of ~/.clistore/config file after test run:


       │ File: /home/oleksii_nikiforov/.clistore/config
───────┼────────────────────────────────────────────────────────────────────────
   1   │ [core]
   2   │ target=my_value
   3   │ is_populated=true
   4   │ 
   5   │ [extra]
   6   │ another_section=false
   7   │ 
───────┴────────────────────────────────────────────────────────────────────────

Use configuration in business logic

Let’s say we want to print out “Hello, {target}” and take {target} as an option/argument. If an option is not provided we should check configuration storage for default values.

The first approach is to directly use BinderBase<IConfiguration> and read from IConfiguration.

// Program.csharp
root.AddConfigCommands(out var configProvider);
root.AddCommand(GreetFromConfigCommand(configProvider));
// GreetCommandsFactory.cs
static void Greet(string target) => Console.WriteLine($"Hello, {target}");

public static Command GreetFromConfigCommand(CliConfigurationProvider configProvider)
{
    var command = new Command("greet-from-config", "Demonstrates how to use IConfiguration from DI container");
    var targetOption = new Option<string?>("--target");
    targetOption.IsRequired = false;
    command.AddOption(targetOption);
    command.SetHandler((string? target, IConfiguration configuration) =>
        Greet(target ?? configuration["core:target"]), targetOption, configProvider);

    return command;
}

The second one is to provide the default value factory getDefaultValue. Note, it might slow down built-in suggestions:

// Program.csharp
root.AddConfigCommands(out var configProvider);
root.AddCommand(GreetFromDefaultValueCommand(configProvider));
// GreetCommandsFactory.cs
public static Command GreetFromDefaultValueCommand(CliConfigurationProvider configProvider)
{
    var command = new Command("greet-from-default-value", "Demonstrates how to provide default value to an option");
    var targetOptionWithDefault = new Option<string>("--target", getDefaultValue: () =>
    {
        // note, this is evaluate preemptively which may slow down autocompletion
        var configuration = configProvider.GetConfiguration();
        return configuration["core:target"];
    });
    command.AddOption(targetOptionWithDefault);
    command.SetHandler((string target) => Greet(target), targetOptionWithDefault);

    return command;
}

The benefit of the approach above is that you don’t need to worry about configuration in the handler code.

And handler could be simplified to command.SetHandler(Greet, targetOptionWithDefault) by using C# method groups.

The tests:

[Fact]
public async Task GreetFromConfig_greets_with_populated_target_config()
{
    Execute("config", "set", "core.target=World");
    var stdOutBuffer = Execute("greet-from-config");

    await Verify(stdOutBuffer.ToString());
}
// Hello, World


[Fact]
public async Task GreetFromDefaultValue_greets_with_default_value_from_populated_target_config()
{
    Execute("config", "set", "core.target=World");
    var stdOutBuffer = Execute("greet-from-default-value");

    await Verify(stdOutBuffer.ToString());
}
// Hello, World

Persisted parameters

Now, we have a good grasp of how to implement stateful CLI applications. We can see how to add the persistent parameters feature. Let’s start with expected behavior and the corresponding test:

[Fact]
public async Task GreetFromPersisted_greets_with_persisted_params_value()
{
    Execute("greet-from-persisted", "--target", "World");
    var stdOutBuffer = Execute("greet-from-persisted");

    await Verify(stdOutBuffer.ToString());
}
// Hello, World

The --target parameter is stored as a result of a previous successful invocation. We can organize a dedicated section in configuration to store previously used values.

To define persisted options we can use BinderBase<T> as a decorator to Option<T>. Like following:

public class PersistedOptionProvider<T> : BinderBase<T?>
{
    // skipped 

    protected override T? GetBoundValue(BindingContext bindingContext)
    {
        if (!bindingContext.ParseResult.HasOption(_option))
        {
            var ini = _configProvider.LoadIniFile();
            string text = ini[CliConfigurationProvider.PersistedParamsSection][_option.Name].ToString();
            var value = (T)TypeDescriptor.GetConverter(typeof(T))
                .ConvertFromString(text)!;

            return value;
        }

        return bindingContext.ParseResult.GetValueForOption(_option);
    }
}

So, if option is not provided, we check the config file for a potential fallback value.

Here is the handler for the persisted parameters greeter variation:

public static Command GreetFromPersistedCommand(CliConfigurationProvider configProvider)
{
    var command = new Command("greet-from-persisted", "Demonstrates how to use persisted parameters");
    var targetOption = new Option<string>("--target");
    targetOption.IsRequired = false;
    command.AddOption(targetOption);

    command.SetHandler(Greet, new PersistedOptionProvider<string>(targetOption, configProvider));

    return command;
}

That solves the read side of the task. What about the write side? We want to store options upon successful completion after all.

Thankfully, System.CommandLine provides a handy abstraction - middleware. We can use it on CommandLineBuilder.AddMiddleware.

var root = new RootCommand();
root.Name = "clistore";

var commandLineBuilder = new CommandLineBuilder(root);

commandLineBuilder.AddMiddleware(async (context, next) => {/*implementation goes here*/});
commandLineBuilder.UseDefaults();

var parser = commandLineBuilder.Build();
await parser.InvokeAsync(args);

It fairly common practice to pack middleware registration in extension methods. Here is the how to write persisted parameters by using middleware approach.

///<summary>
/// Stores provided options registered with CliConfigurationProvider.RegisterPersistedOption
///</summary>
public static CommandLineBuilder AddPersistedParametersMiddleware(
    this CommandLineBuilder builder, CliConfigurationProvider configProvider)
{
    return builder.AddMiddleware(async (context, next) =>
    {
        Lazy<IniFile> config = new(() => configProvider.LoadIniFile());
        var parseResult = context.ParseResult;

        await next(context);

        bool newValuesAdded = false;
        foreach (var option in configProvider.PersistedOptions)
        {
            if (parseResult.HasOption(option))
            {
                config.Value[CliConfigurationProvider.PersistedParamsSection][option.Name] =
                    parseResult.GetValueForOption(option)?.ToString();
                newValuesAdded = true;
            }
        }

        if (newValuesAdded)
        {
            config.Value.Save(configProvider.ConfigLocation);
        }
    });
}

Demo

17:32 $ dotnet run -- -h
Description:
  Shows proof of concept of how to store persistent configuration in a CLI apps

Usage:
  clistore [command] [options]

Options:
  --version       Show version information
  -?, -h, --help  Show help and usage information

Commands:
  config                    Manage CLI configuration
  greet-from-config         Demonstrates how to use IConfiguration from DI container
  greet-from-default-value  Demonstrates how to provide default value to an option
  greet-from-persisted      Demonstrates how to use persisted parameters

✘-1 ~/projects/configuration-builder/src [main|✔] 
17:32 $ dotnet run -- greet-from-persisted --target foo
Hello, foo
✘-1 ~/projects/configuration-builder/src [main|✔] 
17:32 $ dotnet run -- greet-from-persisted 
Hello, foo
✘-1 ~/projects/configuration-builder/src [main|✔] 
17:32 $ dotnet run -- greet-from-persisted --target bar
Hello, bar

Summary

I’ve shown you how to use System.CommandLine and how to implement persisted parameters feature in your CLI/console application. The suggested code is not production-ready but is a decent starting point.

Reference

Oleksii Nikiforov

Jibber-jabbering about programming and IT.