io7m | single-page | multi-page | epub | Looseleaf User Manual

Looseleaf User Manual 3.0.0

CREATOR Mark Raynsford
DATE 2022-06-28T14:43:07+00:00
DESCRIPTION Documentation for Looseleaf server and APIs.
IDENTIFIER ceb1f5c4-e436-4480-9243-dc3662cde5a1
LANGUAGE en
RIGHTS Public Domain
TITLE Looseleaf User Manual 3.0.0

Table Of Contents

1. Introduction

1
The looseleaf server is an HTTP-accessible key/value store with a focus on minimalism, a small footprint, and reliability. The server has the following notable features:

1.2. Features

  • ACID semantics.
  • Atomic reads and updates for arbitrary sets of keys. An unlimited number of keys can be read, updated, and/or deleted in a single operation that is atomic with respect to all other database operations.
  • Fine-grained role-based access control.
  • A trivial HTTP interface for easy access from shell scripts.
  • A strictly defined JSON protocol with a full schema.
  • Convenient endpoints for use with command-line tools such as curl.
  • A small, easily auditable codebase with a heavy use of modularity for correctness.
  • An extensive automated test suite with high coverage.
  • A small footprint; the server is designed to run in tiny 16-32mb JVM heap configurations.
  • Platform independence. No platform-dependent code is included in any form, and installations can largely be carried between platforms without changes. The database file format is also platform-independent.
  • Security-conscious engineering. All requests require authentication, extensive validation is performed on all requests, and careful use is made of the Java type system to enforce invariants throughout the codebase.
  • Fully instrumented with OpenTelemetry for reliable service monitoring.
  • Configurable fault injection for testing monitoring.
  • OSGi-ready
  • JPMS-ready
  • ISC license
3
The server does not and is unlikely to ever have the following features:

1.4. Non-Features

  • The server cannot store arbitrary binary data; keys and values are UTF-8 encoded strings.
  • The server's database is not a distributed database; it is a simple local store based on the SQLite database.
  • Adding/removing users and roles is not dynamic. The users and roles are defined ahead of time in a single configuration file, and the server must be restarted for changes to the configuration file to take effect. The server is intended to be provisioned according to the principles of immutable infrastructure and, as such, a change to the security policy is considered to be a critical infrastructure change. This has the benefit that the server's current security policy is trivially observable in a single location, as opposed to being part of the database's mutable state.
  • TLS support. Use a reverse proxy such as nginx to provide TLS if required.

2. Model

2.1. Overview

1
The looseleaf server provides a persistent key/value database with ACID semantics. Keys and values are arbitrary UTF-8 strings, and a fine-grained role-based access control system is provided to control how users are allowed to perform operations on the database.

2.2. Value

1
A value is an arbitrary UTF-8 encoded string.

2.3. Key

1
A key is a unique name for a value. The core functionality of the looseleaf server is a persistent database of keys and values that can be updated atomically. A key is a UTF-8 encoded string with an associated UTF-8 string value. Key names must begin with / and consist of an arbitrarily long sequence of segments separated with / characters. Repeated sequences of / characters are automatically reduced to a single / character. Key names, effectively, resemble absolute UNIX paths. Key names are ordered lexicographically, so the path /a is considered to be less than /b.

2.4. Key Expression

1
A key expression describes a set of keys. The syntax of key expressions is identical to keys, except that a key expression is allowed to end with a wildcard character *. A key expression /x/y/z matches exactly one key: /x/y/z, whilst the key expression /x/y/z* matches all keys that begin with /x/y/z.

2.5. Action

1
An action defines an operation that may be performed upon a key. An action may be READ or WRITE, where READ means that the value of a given key may be retrieved, and WRITE means that the value of a given key may be modified, or the key created and/or removed as desired.

2.6. Grant

1
A grant combines a key expression and an action, and effectively declares that the given action is allowed to be performed on the keys described by the key expression. For example, a grant with action READ and key expression /x/y/* allows reading the values of any keys that begin with /x/y/. A grant with action WRITE and key expression /a/b/c allows the key /a/b/c to be created, modified, and/or deleted.
2
Any action not explicitly allowed by a grant is denied.

2.7. Role

1
A role is a uniquely named (with respect to other roles) container that defines a set of grants. A user may have zero or more roles.

2.8. User

1
A user is a uniquely named (with respect to other users) container that combines a set of roles with a password. All operations in the looseleaf server are conducted on behalf of authenticated users; no operations are anonymous.

2.9. Read-Update-Delete

1
All operations that occur in the looseleaf server occur in terms of single atomic read-update-delete operations, or RUDs. An RUD operation specifies a set of keys that will be read, followed by a set of keys that will be updated, followed by a set of keys that will be deleted. The reads, updates, and deletes are specified to occur in that order, and the operation as whole will occur atomically; either all read/update/delete operations succeed, or it is as if none of them took place at all. More formally, an RUD can be specified as a 3-tuple (R,U,D), where R is the set of keys to read, U is the set of keys to update, and D is the set of keys to delete. If any of the keys specified in U do not exist, then they will be created by the update operation. It follows that:

2.9.2. Operations

  • Reading a single key k can be expressed as the operation (k,∅,∅).
  • Updating a single key k can be expressed as the operation (∅,k,∅).
  • Deleting a single key k can be expressed as the operation (∅,∅,k).
3
As mentioned previously, operations are specified to occur in R → U → D order. This means that, for example, the operation (∅,k,k) for any k is a no-op.

3. Installation

3.1. Overview

1
The looseleaf package is available from several sources:

3.1.2. Sources

3
Regardless of the distribution method, the looseleaf package will contain a command named looseleaf that acts as the main entrypoint to all of the package's functionality. The looseleaf command expects an environment variable named LOOSELEAF_HOME to be defined that points to the installation directory. See the documentation for the installation methods below for details.

3.2. OCI Images

1
The looseleaf package can be installed from quay.io using docker or podman.

3.2.2. Docker/Podman Pull

$ docker pull quay.io/io7mcom/looseleaf:3.0.0

$ podman pull quay.io/io7mcom/looseleaf:3.0.0
3
The image is configured such that LOOSELEAF_HOME=/looseleaf, with the looseleaf command existing at /looseleaf/bin/looseleaf. The image is configured with the expectation that users will mount a volume at /looseleaf/etc containing a configuration file and space for the looseleaf database file. The container can otherwise be run without any privileges, and with a read-only root filesystem.

3.2.4. Run

$ podman run --read-only --volume /x/y/z:/looseleaf/etc:Z -i -t quay.io/io7mcom/looseleaf:3.0.0 looseleaf
info: Usage: looseleaf [options] [command] [command options]
...

3.3. Maven Central

1
A distribution package can be found at Maven Central.
2
The looseleaf command requires that a Java 17+ compatible JVM be accessible via /usr/bin/env java.
3
Verify the integrity of the distribution zip file:

3.3.4. Verify

$ gpg --verify com.io7m.looseleaf.cmdline-3.0.0-distribution.zip.asc
gpg: assuming signed data in 'com.io7m.looseleaf.cmdline-3.0.0-distribution.zip.asc'
gpg: Signature made Tue 28 Jun 2022 15:01:56 GMT
gpg:                using RSA key 3CCE59428B30462D10459909C5607DA146E128B8
gpg:                issuer "contact@io7m.com"
gpg: using pgp trust model
gpg: Good signature from "io7m.com (2022 maven-rsa-key) <contact@io7m.com>" [unknown]
5
Unzip the zip file, and set LOOSELEAF_HOME appropriately:

3.3.6. Extract

$ unzip com.io7m.looseleaf.cmdline-3.0.0-distribution.zip
$ export LOOSELEAF_HOME=$(realpath looseleaf)
$ ./looseleaf/bin/looseleaf
info: Usage: looseleaf [options] [command] [command options]
...

3.4. Upgrading

3.4.1. 2.0.0 → 3.0.0

1
In 3.0.0, the looseleaf package added support for SQLite as the new default database implementation underlying the key/value store. Support for MVStore is still present, but deprecated.
2
Data should be migrated out of existing databases and into new SQLite databases using the migrate-database command.

4. Configuration

4.1. Overview

1
The looseleaf server accepts a JSON configuration file with a very strictly-defined format. The configuration file is a JSON object with the following properties:

4.1.2. Properties

4.2. %schema

1
The %schema property must be present and set to the value "https://www.io7m.com/software/looseleaf/looseleaf-config-1.json".

4.3. addresses

1
The addresses property specifies an array of addresses to which the server will bind. An address is a JSON object with a mandatory string-typed host and mandatory integer-typed port property. For example, for a server that listens on 172.17.0.1 port 20000 and fe80::42:31ff:fe0a:119a port 20001, the following definitions would be used:

4.3.2. Example Addresses

"addresses": [
  { "host": "172.17.0.1", "port": 20000 },
  { "host": "fe80::42:31ff:fe0a:119a", "port": 20001 },
]
3
The looseleaf server does not support TLS and it is expected that the server will be configured to listen on localhost behind a reverse proxy that provides TLS such as nginx.

4.4. databaseFile

1
The databaseFile property specifies the path of the database that the server will use to store key/value data. The file will be created if it does not already exist.
2
The optional databaseKind property specifies type of the database. One of the following values must be used:

4.4.3. Database Kind

  • SQLITE
  • MVSTORE
4
If no value is provided, SQLITE is assumed.

4.5. roles

1
The roles property is an array-typed property that defines a set of roles.
2
The following example defines a role read-xy that allows any user that has the role to read keys that begin with /x/y/:

4.5.3. read-xy

"roles": [
  {
    "name": "read-xy",
    "grants": [
      {
        "action": "READ",
        "keys": "/x/y/*"
      }
    ]
  }
]
4
Similarly, the following write-xy definition allows any user that has the write-xy role to create, update, and/or delete keys that begin with /x/y/:

4.5.5. read-xy

"roles": [
  {
    "name": "write-xy",
    "grants": [
      {
        "action": "WRITE",
        "keys": "/x/y/*"
      }
    ]
  }
]

4.6. users

1
The users property is an array-typed property that defines a set of users. Each element of the array is a JSON object that specifies a user name, a hashed password, and a set of role names that reference roles declared in the roles property.
2
A hashed password declares an algorithm identifier, an uppercase hex-encoded salt value and an uppercase hex-encoded hash value. Currently, the only supported algorithm identifier is PBKDF2WithHmacSHA256:n:256, which states that passwords are hashed with PBKDF2 using a SHA-256 HMAC, with n rounds of hashing, using a 256 bit key.
3
The create-password command can be used to hash a password suitable for use in a configuration file.
4
The following example defines a hashed password for a user:

4.6.5. Hashed Password

"password": {
  "algorithm": "PBKDF2WithHmacSHA256:10000:256",
  "hash": "7706A5A86FEA0CE2BAC511FD0C3C10B3432D247CF28B8B9BD9CD99234D80B738",
  "salt": "4B4057CD69190E6D41898F9E793824D6"
},
6
The following example defines a user called someone, with a hashed password and a set of roles:

4.6.7. User Example

"users": [
  {
    "name": "someone",
    "password": {
      "algorithm": "PBKDF2WithHmacSHA256:10000:256",
      "hash": "7706A5A86FEA0CE2BAC511FD0C3C10B3432D247CF28B8B9BD9CD99234D80B738",
      "salt": "4B4057CD69190E6D41898F9E793824D6"
    },
    "roles": [
      "read-xy",
      "write-xy"
    ]
  }
]

4.7. Telemetry

1
The telemetry property is an optional object-typed property that configures telemetry. It expects three optional properties metrics, traces, and logs, that each specify an endpoint and protocol for telemetry. If any of the three properties are absent, data will not be sent for that class of telemetry. The telemetry property also requires a property named logicalServiceName property which, unsurprisingly, configures the logical service name to be included in telemetry; this is used to distinguish between multiple running instances of the looseleaf server in telemetry. The protocol name may either be HTTP or GRPC.

4.7.2. Telemetry Example

"telemetry": {
  "logicalServiceName": "looseleaf",
  "metrics": {
    "endpoint": "http://metrics.example.com:4317",
    "protocol": "GRPC"
  },
  "logs": {
    "endpoint": "http://metrics.example.com:4317",
    "protocol": "GRPC"
  },
  "traces": {
    "endpoint": "http://traces.example.com:4317",
    "protocol": "GRPC"
  }
}

4.8. Fault Injection

1
In order to test that your monitoring system is working correctly, it can be desirable to be able to inject faults into the server in order to verify that the monitoring system picks them up. The faultInjection property is an optional object-typed property that specifies the probability that various types of faults will be injected. If the property is not present, no fault injection will occur. All probabilities are in the range [0, 1] where 1 indicates that faults will occur on every operation unconditionally.
2
Currently, the only supported property is databaseCrashProbability, which specifies the probability that database accesses will raise exceptions.

4.8.3. Telemetry Example

"faultInjection": {
  "databaseCrashProbability": 0.005
}

4.9. Example

1
A full configuration file example is as follows:

4.9.2. Example

{
  "%schema": "https://www.io7m.com/software/looseleaf/looseleaf-config-1.json",

  "addresses": [
    {
      "host": "localhost",
      "port": 20000
    }
  ],

  "databaseFile": "/looseleaf/etc/data.db",

  "roles": [
    {
      "name": "read-xy",
      "grants": [
        {
          "action": "READ",
          "keys": "/x/y/*"
        }
      ]
    },
    {
      "name": "write-xy",
      "grants": [
        {
          "action": "WRITE",
          "keys": "/x/y/*"
        }
      ]
    }
  ],

  "users": [
    {
      "name": "someone",
      "password": {
        "algorithm": "PBKDF2WithHmacSHA256:10000:256",
        "hash": "7706A5A86FEA0CE2BAC511FD0C3C10B3432D247CF28B8B9BD9CD99234D80B738",
        "salt": "4B4057CD69190E6D41898F9E793824D6"
      },
      "roles": [
        "read-xy",
        "write-xy"
      ]
    }
  ],

  "telemetry": {
    "logicalServiceName": "looseleaf",
    "metrics": {
      "endpoint": "http://metrics.example.com:4317",
      "protocol": "GRPC"
    },
    "logs": {
      "endpoint": "http://metrics.example.com:4317",
      "protocol": "GRPC"
    },
    "traces": {
      "endpoint": "http://traces.example.com:4317",
      "protocol": "GRPC"
    }
  },

  "faultInjection": {
    "databaseCrashProbability": 0.005
  }
}
3
The configuration defines two roles read-xy and write-xy which allow reading of keys /x/y/* and writing of keys /x/y/*, respectively. The configuration defines a single user named someone with both roles assigned. The configuration specifies that the local database will be created a /looseleaf/etc/data.db, and that the server will listen on http://localhost:20000.

4.10. Schema

1
The JSON schema that defines the configuration file format is as follows:

4.10.2. Schema

{
  "$id": "https://www.io7m.com/software/looseleaf/looseleaf-config-1.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$defs": {
    "SchemaIdentifier": {
      "type": "string",
      "const": "https://www.io7m.com/software/looseleaf/looseleaf-config-1.json"
    },

    "Protocol": {
      "type": "string",
      "enum": [
        "HTTP",
        "GRPC"
      ]
    },

    "TelemetryEndpoint": {
      "type": "object",
      "properties": {
        "endpoint": {
          "type": "string"
        },
        "protocol": {
          "$ref": "#/$defs/TelemetryEndpoint"
        }
      },
      "required": [
        "endpoint",
        "protocol"
      ]
    },

    "Telemetry": {
      "type": "object",
      "properties": {
        "logicalServiceName": {
          "type": "string"
        },
        "metrics": {
          "$ref": "#/$defs/TelemetryEndpoint"
        },
        "logs": {
          "$ref": "#/$defs/TelemetryEndpoint"
        },
        "traces": {
          "$ref": "#/$defs/TelemetryEndpoint"
        }
      },
      "required": [
        "logicalServiceName"
      ]
    },

    "FaultInjection": {
      "type": "object",
      "properties": {
        "databaseCrashProbability": {
          "type": "double"
        }
      },
      "required": [
        "databaseCrashProbability"
      ]
    },

    "BindAddress": {
      "type": "object",
      "properties": {
        "host": {
          "type": "string"
        },
        "port": {
          "type": "number"
        }
      },
      "additionalProperties": false,
      "required": [
        "host",
        "port"
      ]
    },

    "Action": {
      "type": "string",
      "enum": [
        "READ",
        "WRITE"
      ]
    },

    "Grant": {
      "type": "object",
      "properties": {
        "action": {
          "$ref": "#/$defs/Action"
        },
        "keys": {
          "type": "string"
        }
      },
      "additionalProperties": false,
      "required": [
        "action",
        "keys"
      ]
    },

    "Role": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "grants": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Grant"
          }
        }
      },
      "additionalProperties": false,
      "required": [
        "name",
        "grants"
      ]
    },

    "Password": {
      "type": "object",
      "properties": {
        "algorithm": {
          "type": "string"
        },
        "hash": {
          "type": "string",
          "format": "[A-F0-9]+"
        },
        "salt": {
          "type": "string",
          "format": "[A-F0-9]+"
        }
      },
      "additionalProperties": false,
      "required": [
        "algorithm",
        "hash",
        "salt"
      ]
    },

    "User": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "password": {
          "$ref": "#/$defs/Password"
        },
        "roles": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false,
      "required": [
        "name",
        "password",
        "roles"
      ]
    },

    "Configuration": {
      "type": "object",
      "properties": {
        "%schema": {
          "$ref": "#/$defs/SchemaIdentifier"
        },
        "addresses": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/BindAddress"
          }
        },
        "databaseFile": {
          "type": "string"
        },
        "databaseKind": {
          "type": "string"
        },
        "roles": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Role"
          }
        },
        "users": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/User"
          }
        },
        "telemetry": {
          "$ref": "#/$defs/Telemetry"
        },
        "faultInjection": {
          "$ref": "#/$defs/FaultInjection"
        }
      },
      "additionalProperties": false,
      "required": [
        "%schema",
        "addresses",
        "databaseFile",
        "faultInjection",
        "roles",
        "users"
      ]
    }
  },

  "$ref": "#/$defs/Configuration"
}

5. Usage

5.1. Overview

1
This section of the documentation describes the usage of the looseleaf server.

5.2. Running The Server

1
Given a correctly defined configuration file in config.json, the server can be started with the following command:

5.2.2. Run

$ looseleaf server --file config.json
3
The looseleaf command does not fork into the background, and is designed to run under process supervision.

5.3. Endpoints

1
The looseleaf server exposes a number of HTTP endpoints. For the sake of example, we will assume throughout the rest of this documentation that the server was configured to listen on localhost on port 20000. We will also assume that a user exists with name grouch and password 12345678. The looseleaf server requires basic authentication for all requests except for the root endpoint and the health endpoint. The API versions exposed by the server can be inspected directly:

5.3.2. Versions

$ curl http://localhost:20000
[{"name":"com.io7m.looseleaf.v1","base":"/v1"}]
3
The above output indicates that the server only supports protocol v1, with all the protocol's endpoints based at http://localhost:20000/v1. Please see the reference pages for the endpoints for precise descriptions of inputs and outputs, and usage examples.

6. Monitoring

6.1. Overview

1
The looseleaf package is extensively instrumented with OpenTelemetry in order to allow for the database to be continually monitored. The package publishes metrics, logs, and traces, all of which can be independently enabled or disabled. Most installations will only want to enable metrics or logs in production; traces are more useful when trying to diagnose performance problems, or for doing actual development on the looseleaf package.

6.2. Metrics

6.2.1. Names

1
The package publishes the following metrics that can be used for monitoring:

6.2.1.2. Metrics

Name Description
looseleaf_db_reads A counter that is incremented every time a key is successfully read from the database.
looseleaf_db_writes A counter that is incremented every time a key is successfully written in the database.
looseleaf_db_errors A counter that is incremented every time an error occurs when trying to read or write the database. This can occur on database transaction conflicts.
looseleaf_auth_errors A counter that is incremented every time a user fails to authenticate properly.
looseleaf_db_deletes A counter that is incremented every time a key is successfully deleted from the database.
looseleaf_db_time A gauge that is periodically updated with the time the most recent database access took in nanoseconds. This can give a general view of how well the database is performing.
looseleaf_db_keys A gauge that is periodically updated with the approximate number of keys present in the database.
looseleaf_db_size A gauge that is periodically updated with the approximate size of the database in bytes.
looseleaf_http_request_time A gauge that is periodically updated with the duration of the most recently processed HTTP request. This is an important metric for showing the overall latency of the service.
looseleaf_http_error_400 A counter that is incremented every time an HTTP request results in an HTTP response with a 4xx status code.
looseleaf_http_error_500 A counter that is incremented every time an HTTP request results in an HTTP response with a 5xx status code.
looseleaf_http_request A counter that is incremented every time an HTTP request is made. This is an important metric for showing the overall traffic level to the service.
looseleaf_up A guage that provides a fixed 1 value whenever the service is up.
3
The package may produce other metrics, however these are undocumented and should not be relied upon.

6.3. Logs

1
The looseleaf package is conservative in the amount of logging output it produces by default. The package is written to publish only a specific set of log messages to telemetry logs in order to increase the signal-to-noise ratio. There are very few kinds of errors that can even occur; the database might fail on reading or writing, and users might fail to authenticate properly. The looseleaf package logs errors at ERROR severity with at least the following attributes:

6.3.2. Message Attributes

Name Description
looseleaf.user The user that triggered the error.
looseleaf.key The database key associated with the error.
looseleaf.operation The operation being performed, such as "read", "update", etc.
looseleaf.remoteAddress The address of the client that triggered the error.

6.4. Traces

1
The looseleaf package publishes traces for all internal operations. No specific documentation is provided on the structures of the traces as they are effectively tied to the internal structure of the code and are subject to change.

6.4.2. Trace Display

Telemetry displayed in Grafana.

6.5. Grafana Dashboard

1
The looseleaf package provides a basic Grafana dashboard. The dashboard can be imported from https://raw.githubusercontent.com/io7m/looseleaf/develop/com.io7m.looseleaf.grafana/src/main/resources/com/io7m/looseleaf/grafana/dashboard.json.

6.5.2. Dashboard

Dashboard

7. Base Endpoints

7.1. /

7.1.1. Name

1
/ - The root endpoint

7.1.2. Description

1
The / endpoint displays the protocols supported by the server.

7.1.3. Examples

1
The following command displays the supported protocols:

7.1.3.2. Example Update

$ curl http://localhost:20000/
[{"name":"com.io7m.looseleaf.v1","base":"/v1"}]

7.2. /health

7.2.1. Name

1
/health - The health endpoint

7.2.2. Description

1
The /health endpoint is intended to be used as a health check by load balancers. The endpoint returns HTTP status 200, and a small JSON object containing a few statistics.

7.2.3. Examples

1
The following command displays the health of the server:

7.2.3.2. Example Update

$ curl http://localhost:20000/health
{"reads":18,"writes":117}

8. Protocol V1

8.1. Generalities

1
On failure conditions that are unrelated to problems on the server (such as hardware or software failure, bugs, and etc), all endpoints return a 4** HTTP status code, and a JSON object of type Errors. An Errors object contains an array-typed errors property such that each element is of type Error. An Error object contains an errorCode property and a human-readable message property.

8.1.2. Errors

{
  "errors": [
    { "errorCode": "operation-not-permitted", "message": "Operation READ not permitted on key /x/y/z" },
    { "errorCode": "not-found", "message": "Key not found: /q" },
  ]
}
3
See the schema for the exact types of all objects that can be returned by the endpoints.

8.2. /v1/rud

8.2.1. Name

1
/v1/rud - Atomically read, update, and delete sets of keys

8.2.2. Description

1
The /v1/rud endpoint performs a read-update-delete operation. The endpoint requires a JSON object containing three properties: read, update, and delete. The read property is an array-typed property containing a list of key names that will be read. The update property is a JSON object where the properties names are key names, and the property values are key values; the key names mentioned will be set to the corresponding values. The delete property is an array-typed property containing a list of key names to be deleted.

8.2.2.2. Example Object

{
  "read": [
    "/x/y/z"
  ],
  "update": {
    "/x/y/z": "24"
  },
  "delete": []
}
3
On success, the endpoint returns a JSON object with a values property that contains the values of the keys specified by the original read property.

8.2.2.4. Example Values

{
  "values": {
    "/x/y/z": "before"
  }
}

8.2.3. Examples

1
The following command reads the value of /x/y/z, then sets the value of /x/y/z to "after":

8.2.3.2. Example Read/Update

$ curl -u grouch:12345678 -d '{"read":["/x/y/z"],"update":{"/x/y/z":"after"},"delete":[]}' http://localhost:20000/v1/rud
{"values":{"/x/y/z":"before"}}
3
The following command deletes /x/y/z:

8.2.3.4. Example Read/Update

$ curl -u grouch:12345678 -d '{"read":[],"update":{},"delete":["/x/y/z"]}' http://localhost:20000/v1/rud
{"values":{}}
5
The following command updates/creates all of /x/y/a, /x/y/b, and /x/y/c atomically:

8.2.3.6. Example Read/Update

$ curl -u grouch:12345678 -d '{"read":[],"update":{"/x/y/a":"q","/x/y/b":"w","/x/y/c":"x"},"delete":[]}' http://localhost:20000/v1/rud
{"values":{}}

8.3. /v1/read

8.3.1. Name

1
/v1/read - Directly read the value of a single key

8.3.2. Description

1
The /v1/read endpoint is a convenience endpoint that directly returns the value of a key, assuming the key exists. It is equivalent to calling the /v1/rud endpoint with empty update and delete sets, and a read set that contains a single key, and then extracting the key value from the resulting values property and returning it directly. The endpoint is suffixed with the name of the target key, so to read key /x/y/z, the endpoint is called as /v1/read/x/y/z. The content type of the returned data is text/plain.

8.3.3. Examples

1
The following command reads the value of /x/y/z:

8.3.3.2. Example Read

$ curl -u grouch:12345678 http://localhost:20000/v1/read/x/y/z
/x/y/z

8.4. /v1/update

8.4.1. Name

1
/v1/update - Directly update the value of a single key

8.4.2. Description

1
The /v1/update endpoint is a convenience endpoint that directly updates the value of a key, assuming the key exists. It is equivalent to calling the /v1/rud endpoint with empty read and delete sets, and an update set that contains a single key. The endpoint is suffixed with the name of the target key, so to update key /x/y/z, the endpoint is called as /v1/update/x/y/z.

8.4.3. Examples

1
The following command updates the value of /x/y/z to "a new value":

8.4.3.2. Example Update

$ curl -u grouch:12345678 -d 'a new value' http://localhost:20000/v1/update/x/y/z

$ curl -u grouch:12345678 http://localhost:20000/v1/get/x/y/z
a new value

8.5. /v1/delete

8.5.1. Name

1
/v1/delete - Directly update the value of a single key

8.5.2. Description

1
The /v1/delete endpoint is a convenience endpoint that directly updates the value of a key, assuming the key exists. It is equivalent to calling the /v1/rud endpoint with empty update and read sets, and a delete set that contains a single key. The endpoint is suffixed with the name of the target key, so to update key /x/y/z, the endpoint is called as /v1/delete/x/y/z.

8.5.3. Examples

1
The following command deletes the key /x/y/z:

8.5.3.2. Example Update

$ curl -u grouch:12345678 http://localhost:20000/v1/delete/x/y/z

$ curl -u grouch:12345678 http://localhost:20000/v1/get/x/y/z
{"errors":[{"errorCode":"not-found","message":"Key not found: /x/y/z"}]}

8.6. Schema

1
The JSON schema that defines all JSON messages exchanged between the client and server in this version of the protocol is as follows:

8.6.2. Schema

{
  "$id": "https://www.io7m.com/software/looseleaf/looseleaf-1.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$defs": {
    "SchemaIdentifier": {
      "type": "string",
      "const": "https://www.io7m.com/software/looseleaf/looseleaf-1.json"
    },

    "Error": {
      "type": "object",
      "properties": {
        "%schema": {
          "$ref": "#/$defs/SchemaIdentifier"
        },
        "errorCode": {
          "type": "string"
        },
        "message": {
          "type": "string"
        }
      },
      "additionalProperties": false,
      "required": [
        "errorCode",
        "message"
      ]
    },

    "Errors": {
      "type": "object",
      "properties": {
        "%schema": {
          "$ref": "#/$defs/SchemaIdentifier"
        },
        "errors": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Error"
          }
        }
      },
      "additionalProperties": false,
      "required": [
        "errors"
      ]
    },

    "RUD": {
      "type": "object",
      "properties": {
        "%schema": {
          "$ref": "#/$defs/SchemaIdentifier"
        },
        "read": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "update": {
          "type": "object",
          "additionalProperties": {
            "type": "string"
          }
        },
        "delete": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false,
      "required": [
        "read",
        "update",
        "delete"
      ]
    },

    "Result": {
      "type": "object",
      "properties": {
        "%schema": {
          "$ref": "#/$defs/SchemaIdentifier"
        },
        "values": {
          "type": "object",
          "additionalProperties": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false,
      "required": [
        "values"
      ]
    }
  },

  "anyOf": [
    {
      "$ref": "#/$defs/Error"
    },
    {
      "$ref": "#/$defs/Errors"
    },
    {
      "$ref": "#/$defs/RUD"
    },
    {
      "$ref": "#/$defs/Result"
    }
  ]
}

9. Command-Line Interface

9.1. Overview

1
The looseleaf package provides a command-line interface for performing tasks such as starting the server, checking configuration files, hashing passwords, and etc. The base looseleaf command is broken into a number of subcommands which are documented over the following sections.

9.1.2. Command-Line Overview

info: Usage: looseleaf [options] [command] [command options]

  Options:
    --verbose
      Set the minimum logging verbosity level.
      Default: info
      Possible Values: [trace, debug, info, warn, error]

  Use the "help" command to examine specific commands:

    $ looseleaf help help.

  Command-line arguments can be placed one per line into a file, and the file
  can be referenced using the @ symbol:

    $ echo help > file.txt
    $ echo help >> file.txt
    $ looseleaf @file.txt

  Commands:
    check-configuration     Check configuration file.
    create-password         Create a hashed password.
    help                    Show detailed help messages for commands.
    server                  Start a server.
    version                 Show the package version.

  Documentation:
    https://www.io7m.com/software/looseleaf/documentation/
3
All of the command-line functionality is implemented using the standard looseleaf APIs.
4
All subcommands accept a --verbose parameter that may be set to one of trace, debug, info, warn, or error. This parameter sets the lower bound for the severity of messages that will be logged. For example, at debug verbosity, only messages of severity debug and above will be logged. Setting the verbosity to trace level effectively causes everything to be logged, and will produce large volumes of debugging output.
5
The looseleaf command-line tool uses quarrel to parse command-line arguments, and therefore supports placing command-line arguments into a file, one argument per line, and then referencing that file with @. For example:

9.1.6. @ Syntax

$ looseleaf server --file config.json
info: [localhost:20000] server started

$ (cat <<EOF
server
--file
config.json
EOF
) > args.txt

$ looseleaf @args.txt
info: [localhost:20000] server started
7
All subcommands, unless otherwise specified, yield an exit code of 0 on success, and a non-zero exit code on failure.

9.2. check-configuration

9.2.1. Name

1
check-configuration - Check configuration file.

9.2.2. Description

1
The check-configuration command will validate the configuration file specified with --file.
2
If the command encounters no errors or warnings, it will not print anything.

9.2.3. Parameters

9.2.3.1. --file

Attribute Value
Name --file
Type java.nio.file.Path
Default Value
Cardinality [1, 1]
Description The configuration file.

9.2.3.2. --verbose

Attribute Value
Name --verbose
Type com.io7m.quarrel.ext.logback.QLogLevel
Default Value info
Cardinality [1, 1]
Description Set the logging level of the application.

9.2.4. Examples

9.2.4.1. Example

$ looseleaf check-configuration --file server.conf
$ echo $?
0

$ looseleaf check-configuration --file bad.conf
error: Nonexistent role 'oops'
$ echo $?
1

$ looseleaf check-configuration --file bad2.conf
error: Duplicate user 'someone'
$ echo $?
1

9.3. create-password

9.3.1. Name

1
create-password - Create a hashed password.

9.3.2. Description

1
The create-password command creates a hashed password suitable for placing into the configuration file.

9.3.3. Parameters

9.3.3.1. --password

Attribute Value
Name --password
Type java.lang.String
Default Value
Cardinality [1, 1]
Description The password.

9.3.3.2. --verbose

Attribute Value
Name --verbose
Type com.io7m.quarrel.ext.logback.QLogLevel
Default Value info
Cardinality [1, 1]
Description Set the logging level of the application.

9.3.4. Security

1
Note that, as the command accepts a password directly on the command line, the command should not be executed on systems where hostile users may try to read the process environments of other users on the system. This would allow the hostile users to capture the unhashed passwords being created. If this is of concern, the password can be placed into a file temporarily using @ syntax.

9.3.5. Examples

9.3.5.1. Example

$ looseleaf create-password --password 12345678
{
  "algorithm" : "PBKDF2WithHmacSHA256:10000:256",
  "hash" : "D53F53E4777177AEF56746914970D10314958570A8293FEB2C6C88371CF65AB1",
  "salt" : "9E7DFDE9DA90D8473F7920B57B79B6F2"
}

9.4. migrate-database

9.4.1. Name

1
migrate-database - Migrate data between databases.

9.4.2. Description

1
The migrate-database command migrates data between databases.

9.4.3. Parameters

9.4.3.1. --database-source

Attribute Value
Name --database-source
Type java.nio.file.Path
Default Value
Cardinality [1, 1]
Description The source database.

9.4.3.2. --database-source-kind

Attribute Value
Name --database-source-kind
Type java.lang.String
Default Value
Cardinality [1, 1]
Description The source database kind.

9.4.3.3. --database-target

Attribute Value
Name --database-target
Type java.nio.file.Path
Default Value
Cardinality [1, 1]
Description The target database.

9.4.3.4. --database-target-kind

Attribute Value
Name --database-target-kind
Type java.lang.String
Default Value
Cardinality [1, 1]
Description The target database kind.

9.4.3.5. --verbose

Attribute Value
Name --verbose
Type com.io7m.quarrel.ext.logback.QLogLevel
Default Value info
Cardinality [1, 1]
Description Set the logging level of the application.

9.4.4. Examples

9.4.4.1. Example

$ looseleaf migrate-database \
--database-source source.mvstore \
--database-source-kind MVSTORE \
--database-target target.mvstore \
--database-target-kind SQLITE

9.5. server

9.5.1. Name

1
server - Start a server.

9.5.2. Description

1
The server command will start the server based on the configuration given in the configuration file specified with --file. The command does not fork into the background, and is suitable for use under process supervision.

9.5.3. Parameters

9.5.3.1. --file

Attribute Value
Name --file
Type java.nio.file.Path
Default Value
Cardinality [1, 1]
Description The configuration file.

9.5.3.2. --verbose

Attribute Value
Name --verbose
Type com.io7m.quarrel.ext.logback.QLogLevel
Default Value info
Cardinality [1, 1]
Description Set the logging level of the application.

9.5.4. Examples

9.5.4.1. Example

$ looseleaf server --file config.json
info: [localhost:20000] server started

9.6. version

9.6.1. Name

1
version - Show the application version.

9.6.2. Description

1
The version command shows the application version.

9.6.3. Examples

9.6.3.1. Example

$ looseleaf version
com.io7m.looseleaf 3.0.0 29e3fae04b84c117b3bfd21fc26a89ec73ef53c6

10. API

1
Documentation for the looseleaf APIs are provided in the form of JavaDoc.
io7m | single-page | multi-page | epub | Looseleaf User Manual