The sunburst package provides a strongly-versioned, transactional package system for
applications that include large assets such as texture images, audio files, and etc.
The Java runtime environment includes the concept of resources. A resource is a file that
can be referenced from within programs, and is typically delivered inside one of the
jar
files that comprise an application. The runtime environment provides convenient access to resources via the
Class.getResource()
method, and this has the desirable property that the application effectively does not need to know its own
installation location to locate data files, and conceptually doesn't even need access to a filesystem; it simply
calls into the resource API and gets data in return from the runtime. However, resources have a few limitations
that are problematic for certain kinds of applications and deployment scenarios:
1.2.2. Limitations Of Resources
- As resources are tightly bound to the application's jar files, the resources must be re-delivered to users every time code in the same jar file is updated, even if none of the resources changed. If a jar file contains a gigabyte of resource files, and a single line of code in a class in the jar file is changed, then users must be sent a new copy of the gigabyte-sized jar file when the application is updated.
- As the Java runtime environment allows for classes to be loaded over a network connection, resources and classes alike are limited to being accessed in a stream-like fashion, unless the application introspectively tries various hacks to locate its own jar files (which may or may not be present in the filesystem). This precludes accessing large resource files such as audio files in a random-access fashion. It also precludes memory mapping resources for efficient uploading to GPUs and other external hardware.
- Prior to the introduction of the module system in the Java runtime, any class could access the resources of any other class, intentionally or unintentionally. This can introduce accidental implicit dependencies across codebases, which can be broken unexpectedly when a (transitive) dependency happens to be removed from the set of packages that comprise an application.
For many applications, the above limitations don't constitute practical problems. For small files, and files that
do not require random access, Java resources should continue to be the go-to method for including content with
applications. For applications where any of the above points are a problem, the sunburst
package exists to provide the following benefits:
1.2.4. Design Points
- Resources in sunburst are simple files in the filesystem. They can be accessed in a random-access fashion, memory-mapped, or subjected to any of the other operations supported by the host filesystem.
- Resources in sunburst are located using a simple URI-like mechanism that insulates the application from having to know where to find resources in the filesystem. This preserves most of the convenience of the Class.getResource() API.
- Resources in sunburst are integrity-checked. Resources are inserted into a content-addressed directory structure indexed by cryptographic hashes. Individual corrupted or missing resources can be automatically re-downloaded without requiring the re-download of all resources. This saves greatly on network bandwidth during updates.
- Resources in sunburst are deduplicated. Resources are indexed by their cryptographic hash, so if many different packages reference a resource, the resource is only stored in the filesystem once. For large files, this can entail significant space savings.
- Resources in sunburst are grouped into strongly-versioned packages, with non-snapshot versions of packages being treated as immutable. Packages must be explicitly imported by Java code in order to be visible to it; accidental implicit dependencies are prevented. APIs are provided to check that all of the application's required sunburst packages are present and installed on application startup.
- Packages in sunburst are backed by transactional storage, meaning that packages can be safely and atomically installed/uninstalled/updated, even from multiple processes concurrently. Multiple versions of the same package can be installed concurrently, and an unlimited number of applications can share the same sunburst database if desired.
A blob is the most fundamental object in sunburst. It
consists of binary data representing the content of the blob, and a purely informative
media type. A blob is uniquely
identified by the cryptographic hash of its content. In sunburst, resources are
stored as blobs, one blob per file, with no leading or trailing data in the file.
A package in sunburst lists a set of
blobs, associating each blob with
a path, and is assigned a unique package identifier (in reverse DNS form), and a
semantic version
number.
A package with a version number that does not have a
SNAPSHOT
qualifier is considered to be immutable once published. It is an error to attempt to add or remove resources
to/from a non-SNAPSHOT package once published. Packages with a
SNAPSHOT
qualifier are considered mutable, should be used during application development, and should be assigned a non-
SNAPSHOT
version when a version of the application is to be released.
A peer in sunburst associates a Java package (such
as java.net.http) with the
sunburst
packages it explicitly imports. When a Java class requests a resource from the
sunburst
runtime, the runtime checks that the package to which the Java class belongs has an
import
that names the
sunburst
package that contains the requested resource. This ensures that code cannot accidentally access resources in
packages that it did not explicitly import.
Peers are declared both in static XML metadata that is included in a standardized location inside the
META-INF
directories of jar files, and also exist as instances of an
SBPeer
class that are loaded by the sunburst runtime using the standard Java
ServiceLoader
mechanism. In order to relieve programmers of having to keep metadata and implementations in sync, APIs and a
Maven plugin
are provided that will generate both the required metadata and code at build-time from a few trivial import
declarations given in build scripts.
It is an error for a Java package to import two different versions of the same
sunburst
package, but it is acceptable for two different Java packages in the same application to import two different
versions of the same
sunburst
package. Imports are not "inherited" in any form by Java packages; if a Java package
x.y.z
imports a sunburst package a.b.c, this does not imply
that classes in a hypothetical x.y.z.w package also have access to
a.b.c.
The peers present in a given jar file must be declared in
a file at META-INF/Sunburst/Peers.xml inside the jar.
The XML data must conform to the published
peers schema.
The inventory in sunburst is the local store containing
packages
and
blobs. The inventory is backed by transactional
storage, and can be safely accessed from any number of threads and/or processes concurrently. The intention is
to allow for resources to be updated in realtime during application development without requiring the
application to restart, and to allow installer programs to install resources safely and atomically during
deployment.
The relationships between the objects in sunburst are illustrated by the following
diagram:
In the above image, a Java program named installer downloads packages and blobs from
an arbitrary location on the internet, and installs them into the local inventory. In this case, it has
installed sunburst packages x.y:1.0.0,
x.y:1.2.0, and x.z:1.0.3. The package
x.y:1.0.0
references blobs
93BEF670
and 6C10D993. The package
x.y:1.2.0
references blobs
3D05CF39
and 6C10D993. This indicates that whilst both packages are installed into the
inventory, some content within the packages is the same and therefore isn't stored twice on disk. The package
x.z:1.0.3
only references blob 64508EDA.
A second Java program named consumer imports packages
x.y:1.2.0
and x.z:1.0.3 and so can read the contents of
6C10D993, 3D05CF39, and
64508EDA
by specifying the paths associated with the blobs in the respective packages.