Sunburst User Manual 0.0.6

Mark Raynsford

Book Metadata

CREATOR | Mark Raynsford
DATE | 2022-11-15T16:26:35+00:00
DESCRIPTION | User documentation for the com.io7m.sunburst package.
IDENTIFIER | f8b0c868-8df6-42b1-a50e-2ad93f842eb6
LANGUAGE | en
RIGHTS | Public Domain
TITLE | Sunburst User Manual 0.0.6

License

Copyright © 2022 Mark Raynsford <code@io7m.com> https://www.io7m.com

This book is placed into the public domain for free use by anyone for any purpose. It may be freely used, modified, and distributed.

In jurisdictions that do not recognise the public domain this book may be freely used, modified, and distributed without restriction.

This book comes with absolutely no warranty.

Table Of Contents

Cover

Colophon

Table Of Contents

1. Design & Implementation

1.1. Overview

1.2. Motivation

1.3. Structure

1.3.1. Blob

1.3.2. Package

1.3.3. Peer

1.3.4. Inventory

1.3.5. Relationships

2. Maven Plugin

2.1. Overview

3. API

3.1. JavaDoc

4. Schemas

4.1. Peers

4.2. Packages

5. License

1. Design & Implementation

1.1. Overview

1.2. Motivation

1.3. Structure

1.3.1. Blob

1.3.2. Package

1.3.3. Peer

1.3.4. Inventory

1.3.5. Relationships

1.1. Overview

The sunburst package provides a strongly-versioned, transactional package system for applications that include large assets such as texture images, audio files, and etc.

1.2. Motivation

The Java runtime environment includes the concept of resources. A resource is a file that can be referenced from within programs, and is typically delivered inside one of the jar files that comprise an application. The runtime environment provides convenient access to resources via the Class.getResource() method, and this has the desirable property that the application effectively does not need to know its own installation location to locate data files, and conceptually doesn't even need access to a filesystem; it simply calls into the resource API and gets data in return from the runtime. However, resources have a few limitations that are problematic for certain kinds of applications and deployment scenarios:

1.2.2. Limitations Of Resources

As resources are tightly bound to the application's jar files, the resources must be re-delivered to users every time code in the same jar file is updated, even if none of the resources changed. If a jar file contains a gigabyte of resource files, and a single line of code in a class in the jar file is changed, then users must be sent a new copy of the gigabyte-sized jar file when the application is updated.

As the Java runtime environment allows for classes to be loaded over a network connection, resources and classes alike are limited to being accessed in a stream-like fashion, unless the application introspectively tries various hacks to locate its own jar files (which may or may not be present in the filesystem). This precludes accessing large resource files such as audio files in a random-access fashion. It also precludes memory mapping resources for efficient uploading to GPUs and other external hardware.

Prior to the introduction of the module system in the Java runtime, any class could access the resources of any other class, intentionally or unintentionally. This can introduce accidental implicit dependencies across codebases, which can be broken unexpectedly when a (transitive) dependency happens to be removed from the set of packages that comprise an application.

For many applications, the above limitations don't constitute practical problems. For small files, and files that do not require random access, Java resources should continue to be the go-to method for including content with applications. For applications where any of the above points are a problem, the sunburst package exists to provide the following benefits:

1.2.4. Design Points

Resources in sunburst are simple files in the filesystem. They can be accessed in a random-access fashion, memory-mapped, or subjected to any of the other operations supported by the host filesystem.

Resources in sunburst are located using a simple URI-like mechanism that insulates the application from having to know where to find resources in the filesystem. This preserves most of the convenience of the Class.getResource() API.

Resources in sunburst are integrity-checked. Resources are inserted into a content-addressed directory structure indexed by cryptographic hashes. Individual corrupted or missing resources can be automatically re-downloaded without requiring the re-download of all resources. This saves greatly on network bandwidth during updates.

Resources in sunburst are deduplicated. Resources are indexed by their cryptographic hash, so if many different packages reference a resource, the resource is only stored in the filesystem once. For large files, this can entail significant space savings.

Resources in sunburst are grouped into strongly-versioned packages, with non-snapshot versions of packages being treated as immutable. Packages must be explicitly imported by Java code in order to be visible to it; accidental implicit dependencies are prevented. APIs are provided to check that all of the application's required sunburst packages are present and installed on application startup.

Packages in sunburst are backed by transactional storage, meaning that packages can be safely and atomically installed/uninstalled/updated, even from multiple processes concurrently. Multiple versions of the same package can be installed concurrently, and an unlimited number of applications can share the same sunburst database if desired.

1.3. Structure

1.3.1. Blob

A blob is the most fundamental object in sunburst. It consists of binary data representing the content of the blob, and a purely informative media type. A blob is uniquely identified by the cryptographic hash of its content. In sunburst, resources are stored as blobs, one blob per file, with no leading or trailing data in the file.

1.3.2. Package

A package in sunburst lists a set of blobs, associating each blob with a path, and is assigned a unique package identifier (in reverse DNS form), and a semantic version number.

A package with a version number that does not have a SNAPSHOT qualifier is considered to be immutable once published. It is an error to attempt to add or remove resources to/from a non-SNAPSHOT package once published. Packages with a SNAPSHOT qualifier are considered mutable, should be used during application development, and should be assigned a non- SNAPSHOT version when a version of the application is to be released.

1.3.3. Peer

A peer in sunburst associates a Java package (such as java.net.http) with the sunburst packages it explicitly imports. When a Java class requests a resource from the sunburst runtime, the runtime checks that the package to which the Java class belongs has an import that names the sunburst package that contains the requested resource. This ensures that code cannot accidentally access resources in packages that it did not explicitly import.

Peers are declared both in static XML metadata that is included in a standardized location inside the META-INF directories of jar files, and also exist as instances of an SBPeer class that are loaded by the sunburst runtime using the standard Java ServiceLoader mechanism. In order to relieve programmers of having to keep metadata and implementations in sync, APIs and a Maven plugin are provided that will generate both the required metadata and code at build-time from a few trivial import declarations given in build scripts.

It is an error for a Java package to import two different versions of the same sunburst package, but it is acceptable for two different Java packages in the same application to import two different versions of the same sunburst package. Imports are not "inherited" in any form by Java packages; if a Java package x.y.z imports a sunburst package a.b.c, this does not imply that classes in a hypothetical x.y.z.w package also have access to a.b.c.

The peers present in a given jar file must be declared in a file at META-INF/Sunburst/Peers.xml inside the jar. The XML data must conform to the published peers schema.

1.3.4. Inventory

The inventory in sunburst is the local store containing packages and blobs. The inventory is backed by transactional storage, and can be safely accessed from any number of threads and/or processes concurrently. The intention is to allow for resources to be updated in realtime during application development without requiring the application to restart, and to allow installer programs to install resources safely and atomically during deployment.

1.3.5. Relationships

The relationships between the objects in sunburst are illustrated by the following diagram:

1.3.5.2. Relationships

In the above image, a Java program named installer downloads packages and blobs from an arbitrary location on the internet, and installs them into the local inventory. In this case, it has installed sunburst packages x.y:1.0.0, x.y:1.2.0, and x.z:1.0.3. The package x.y:1.0.0 references blobs 93BEF670 and 6C10D993. The package x.y:1.2.0 references blobs 3D05CF39 and 6C10D993. This indicates that whilst both packages are installed into the inventory, some content within the packages is the same and therefore isn't stored twice on disk. The package x.z:1.0.3 only references blob 64508EDA.

A second Java program named consumer imports packages x.y:1.2.0 and x.z:1.0.3 and so can read the contents of 6C10D993, 3D05CF39, and 64508EDA by specifying the paths associated with the blobs in the respective packages.

2. Maven Plugin

2.1. Overview

2.1. Overview

The sunburst package provides a Maven plugin for inserting metadata into jar files.

3. API

3.1. JavaDoc

3.1. JavaDoc

Please see the included JavaDoc for API reference documentation.

4. Schemas

4.1. Peers

4.2. Packages

4.1. Peers

The schema for serialized peers is as follows:

4.1.2. Peers

<?xml version="1.0" encoding="UTF-8" ?>

<!--
  Copyright © 2022 Mark Raynsford <code@io7m.com> https://www.io7m.com

  Permission to use, copy, modify, and/or distribute this software for any
  purpose with or without fee is hereby granted, provided that the above
  copyright notice and this permission notice appear in all copies.

  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
  SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
  IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-->

<schema xmlns="http://www.w3.org/2001/XMLSchema"
        targetNamespace="urn:com.io7m.sunburst:peer:1"
        xmlns:p="urn:com.io7m.sunburst:peer:1">

  <simpleType name="PackageVersionQualifierT">
    <annotation>
      <documentation>
        The type of package version qualifiers. See com.io7m.sunburst.model.SBPackageVersion.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="[A-Za-z0-9\-]+(\.[A-Za-z0-9\-]+)*"/>
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <simpleType name="PackageNameT">
    <annotation>
      <documentation>
        The type of package names. See com.io7m.sunburst.model.SBPackageIdentifier.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="[a-z][a-z0-9_-]*(\.[a-z][a-z0-9_-]*)*"/>
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <element name="Import">
    <annotation>
      <documentation>
        A unique package import.
      </documentation>
    </annotation>
    <complexType>
      <attribute name="Name"
                 type="p:PackageNameT"
                 use="required"/>
      <attribute name="Major"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Minor"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Patch"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Qualifier"
                 type="p:PackageVersionQualifierT"
                 use="optional"/>
    </complexType>
  </element>

  <element name="Peer">
    <annotation>
      <documentation>
        A peer declaration.
      </documentation>
    </annotation>
    <complexType>
      <sequence minOccurs="0" maxOccurs="unbounded">
        <element ref="p:Import"/>
      </sequence>
      <attribute name="Name"
                 type="p:PackageNameT"
                 use="required"/>
    </complexType>
    <key name="ImportPackageKey">
      <selector xpath="p:Import"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="Peers">
    <annotation>
      <documentation>
        The top-level peers declaration.
      </documentation>
    </annotation>
    <complexType>
      <sequence minOccurs="0" maxOccurs="unbounded">
        <element ref="p:Peer"/>
      </sequence>
    </complexType>
    <key name="PeerNameKey">
      <selector xpath="p:Peer"/>
      <field xpath="@Name"/>
    </key>
  </element>

</schema>

4.2. Packages

The schema for serialized packages is as follows:

4.2.2. Peers

<?xml version="1.0" encoding="UTF-8" ?>

<!--
  Copyright © 2022 Mark Raynsford <code@io7m.com> https://www.io7m.com

  Permission to use, copy, modify, and/or distribute this software for any
  purpose with or without fee is hereby granted, provided that the above
  copyright notice and this permission notice appear in all copies.

  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
  SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
  IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-->

<schema xmlns="http://www.w3.org/2001/XMLSchema"
        targetNamespace="urn:com.io7m.sunburst:package:1"
        xmlns:p="urn:com.io7m.sunburst:package:1">

  <simpleType name="NameT">
    <annotation>
      <documentation>
        The type of package names. See com.io7m.sunburst.model.SBPackageIdentifier.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="([a-z][a-z0-9_-]{0,63})(\.[a-z][a-z0-9_-]{0,62}){0,15}"/>
      <maxLength value="1024"/>
    </restriction>
  </simpleType>

  <simpleType name="QualifierT">
    <annotation>
      <documentation>
        The type of package version qualifiers. See com.io7m.verona.core.VersionQualifier.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="[A-Za-z0-9\-]+(\.[A-Za-z0-9\-]+)*"/>
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <simpleType name="HashAlgorithmT">
    <annotation>
      <documentation>
        An index for a hash algorithm.
      </documentation>
    </annotation>
    <restriction base="string">
      <enumeration value="SHA2_256">
        <annotation>
          <documentation>
            256-bit SHA-2.
          </documentation>
        </annotation>
      </enumeration>
    </restriction>
  </simpleType>

  <simpleType name="PathT">
    <annotation>
      <documentation>
        A normalized path. Paths can be used to refer to objects inside packages without having to refer to them by
        their hash value.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="/([a-z0-9_\\.-]+)(/[a-z0-9_\\.-]+)*"/>
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <simpleType name="HashT">
    <annotation>
      <documentation>
        The base 16 representation of a hash value.
      </documentation>
    </annotation>
    <restriction base="string">
      <pattern value="[A-F0-9]+"/>
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <simpleType name="ContentTypeT">
    <annotation>
      <documentation>
        An IANA media type. Purely informative, and not checked for validity.
      </documentation>
    </annotation>
    <restriction base="string">
      <maxLength value="255"/>
    </restriction>
  </simpleType>

  <element name="Version">
    <annotation>
      <documentation>
        A (semantic) package version. The major, minor, and patch version numbers correspond to the version number
        components defined in the Semantic Versioning 2.0 specification. The optional qualifier can be used to indicate
        alpha/beta/snapshot versions.
      </documentation>
    </annotation>
    <complexType>
      <attribute name="Major"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Minor"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Patch"
                 type="unsignedInt"
                 use="required"/>
      <attribute name="Qualifier"
                 type="p:QualifierT"
                 use="optional"/>
    </complexType>
  </element>

  <element name="Identifier">
    <annotation>
      <documentation>
        A unique package identifier.
      </documentation>
    </annotation>
    <complexType>
      <sequence>
        <element ref="p:Version"/>
      </sequence>
      <attribute name="Name"
                 type="p:NameT"
                 use="required"/>
    </complexType>
  </element>

  <element name="Meta">
    <annotation>
      <documentation>
        A metadata value.
      </documentation>
    </annotation>
    <complexType>
      <attribute name="Key"
                 type="string"
                 use="required"/>
      <attribute name="Value"
                 type="string"
                 use="required"/>
    </complexType>
  </element>

  <element name="Metadata">
    <annotation>
      <documentation>
        The set of optional metadata values.
      </documentation>
    </annotation>
    <complexType>
      <sequence minOccurs="1"
                maxOccurs="unbounded">
        <element ref="p:Meta"/>
      </sequence>
    </complexType>
    <key name="MetaKey">
      <selector xpath="p:Meta"/>
      <field xpath="@Key"/>
    </key>
  </element>

  <element name="Entry">
    <annotation>
      <documentation>
        An entry within a package. This is an opaque blob of data that can be uniquely identified by its hash, and
        assigned a path that can be used to refer to it.
      </documentation>
    </annotation>
    <complexType>
      <attribute name="Path"
                 type="p:PathT"
                 use="required"/>
      <attribute name="HashAlgorithm"
                 type="p:HashAlgorithmT"
                 use="required"/>
      <attribute name="HashValue"
                 type="p:HashT"
                 use="required"/>
      <attribute name="Size"
                 type="unsignedLong"
                 use="required"/>
      <attribute name="ContentType"
                 type="p:ContentTypeT"
                 use="required"/>
    </complexType>
  </element>

  <element name="Entries">
    <annotation>
      <documentation>
        The entries within a package.
      </documentation>
    </annotation>
    <complexType>
      <sequence>
        <element ref="p:Entry"
                 minOccurs="1"
                 maxOccurs="unbounded"/>
      </sequence>
    </complexType>
    <key name="EntryKey">
      <selector xpath="p:Entry"/>
      <field xpath="@Path"/>
    </key>
  </element>

  <element name="Package">
    <annotation>
      <documentation>
        The top-level package declaration.
      </documentation>
    </annotation>
    <complexType>
      <sequence>
        <element ref="p:Identifier"/>
        <element ref="p:Metadata"
                 minOccurs="0"
                 maxOccurs="1"/>
        <element ref="p:Entries"
                 minOccurs="0"
                 maxOccurs="1"/>
      </sequence>
    </complexType>
  </element>

</schema>

5. License

This documentation is placed into the public domain.

5.2. License

Copyright © 2021 Mark Raynsford <code@io7m.com> https://www.io7m.com

This book is placed into the public domain for free use by anyone
for any purpose. It may be freely used, modified, and distributed.

In jurisdictions that do not recognise the public domain this book
may be freely used, modified, and distributed without restriction.

This book comes with absolutely no warranty.