com.io7m.gtyrell

gtyrell

OCI images are available from quay.io.

$ podman pull quay.io/io7mcom/gtyrell:${version}

Usage

Create a server configuration file (server.conf):

com.io7m.gtyrell.server.directory = /tmp/gh com.io7m.gtyrell.server.git_executable = /usr/bin/git com.io7m.gtyrell.server.pause_duration = 0h 15m 0s com.io7m.gtyrell.server.repository_sources = github0 com.io7m.gtyrell.server.repository_source.github0.type = github com.io7m.gtyrell.server.repository_source.github0.user = yourgithubusername com.io7m.gtyrell.server.repository_source.github0.password = yourgithubpassword

The above will attempt to sync repositories from GitHub to /tmp/gh every 15 minutes.

Run the server:

$ java -jar com.io7m.gtyrell.server-1.0.0-main.jar server.conf

The server will not fork into the background and can be safely used under a process supervision system such as s6 without issues.

Credentials

The server will use the system's installed git executable to clone repositories, and therefore will implicitly use the $HOME/.git-credentials data of the user running the server. It is important that the credentials in that file are a superset of the credentials specified in the server's configuration file (otherwise the server might list repositories with which git cannot interact).

When running under an OCI container such as podman, the server executable will be running as root, and therefore a git-credentials file should be mounted in the container at /root/.git-credentials. It is highly recommended that containers be run rootless; the server does not require any special privileges, and does not do any dropping of privileges on its own.

Pause Durations

The server executes in a loop where it will take the current time t, sync all of the repositories, and then pause until t + d, where d is the pause duration specified. This implies that you should configure the pause duration such that it is long enough for the server to actually complete repository syncing. If the server takes a duration k to complete syncing, and t + k >= t + d, then the server will immediately start syncing again. This will almost certainly annoy the administrators of the remote repositories that gtyrell is syncing. Please set d to a sensible value (such as 30 minutes or greater).

Inclusion And Exclusion filters

It's possible to explicitly include and exclude repositories from being cloned and/or updated. This is achieved by specifying a filter program:

com.io7m.gtyrell.server.repository_source.github0.type = github com.io7m.gtyrell.server.repository_source.github0.user = yourgithubusername com.io7m.gtyrell.server.repository_source.github0.password = yourgithubpassword com.io7m.gtyrell.server.repository_source.github0.filter = /etc/filter.conf

The file referenced by com.io7m.gtyrell.server.repository_source.github0.filter must contain a filter program. Filter programs are specified with a line-based format consisting of rules evaluated from top to bottom. An example filter file:

include ^.*$ exclude ^x/.*$ include-and-halt ^x/y$ exclude-and-halt ^z/.*$

A filter rule must be one of include, exclude, include-and-halt, or exclude-and-halt. The incoming repository names are matched against the patterns given in the filter rules. A repository is included or excluded based on the result of the last rule that matched the repository.

The include command marks a repository as included if the pattern matches the repository name. Evaluation of other rules continues if the pattern matches.

The exclude command marks a repository as excluded if the pattern matches the repository name. Evaluation of other rules continues if the pattern matches.

The include-and-halt command marks a repository as included if the pattern matches the repository name. Evaluation of other rules halts if the pattern matches.

The exclude-and-halt command marks a repository as excluded if the pattern matches the repository name. Evaluation of other rules halts if the pattern matches.

If no rules are specified at all, no repositories are included. If no rules match at all for a given repository, the repository is not included.

Patterns are given in Java regular expression syntax and are matched against the incoming repository owner and name separated by a slash.

Given this example:

include ^.*$ exclude ^x/.*$ include-and-halt ^x/y$ exclude-and-halt ^z/.*$ include ^z/a$

A repository a/b:

1. Will match rule 1 and therefore will, currently, be included
2. Will not match rule 2 and therefore its inclusion status will not change
3. Will not match rule 3 and therefore its inclusion status will not change
4. Will not match rule 4 and therefore its inclusion status will not change
5. Will not match rule 5 and therefore its inclusion status will not change

As a result, a/b will be included as rule 1 included it and no other subsequent rules changed this.

A repository x/q:

1. Will match rule 1 and therefore will, currently, be included
2. Will match rule 2 and therefore its inclusion status will be changed to excluded
3. Will not match rule 3 and therefore its inclusion status will not change
4. Will not match rule 4 and therefore its inclusion status will not change
5. Will not match rule 5 and therefore its inclusion status will not change

As a result, x/q will be not included as rule 2 excluded it and no other subsequent rules changed this.

A repository x/y:

1. Will match rule 1 and therefore will, currently, be included
2. Will not match rule 2 and therefore its inclusion status will not change
3. Will match rule 3 and therefore its inclusion status will be changed to included and evaluation will be halted here
4. Rule 4 is not evaluated because rule 3 matched and halted evaluation
5. Rule 5 is not evaluated because rule 3 matched and halted evaluation

As a result, x/y will be included as rule 3 included it and evaluation was halted at that point.

A repository z/a:

1. Will match rule 1 and therefore will, currently, be included
2. Will not match rule 2 and therefore its inclusion status will not change
3. Will not match rule 3 and therefore its inclusion status will not change
4. Will match rule 4 and therefore its inclusion status will be changed to excluded and evaluation will be halted here
5. Rule 5 is not evaluated because rule 4 matched and halted evaluation

As a result, z/a will be excluded as rule 4 excluded it and evaluation was halted at that point. Note that rule 5 would have matched the input exactly, but rule 4 prevented that rule from being evaluated.

The filter rules are inspired by OpenBSD's pf packet filter.

Metrics

As of 2.1.0, the server publishes JMX metrics. Use the following invocation to enable JMX access on localhost on port 9999:

$ java \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.host=127.0.0.1 \ -Dcom.sun.management.jmxremote.local.only=true \ -Dcom.sun.management.jmxremote.port=9999 \ -Dcom.sun.management.jmxremote.ssl=false \ -jar com.io7m.gtyrell.server-2.1.0-main.jar server.conf

The JMX interface can be inspected with tools such as visualvm, or can be plugged into a monitoring solution such as prometheus using the Prometheus JMX Exporter.

The exposed metrics class is com.io7m.gtyrell.Metrics.

The RepositoryGroupFailures and RepositorySyncsFailedTotal attributes are useful for monitoring purposes; they will only ever increase until gtyrell is restarted and indicate a failure to contact a remote repository or group of repositories.

The RepositoryCount attribute is also useful for catching configuration issues. If the number is zero, then there may be some kind of misconfiguration.

The RepositorySyncShortPauses attribute indicates that the server began a pause period after syncing with less than a minute to perform the pause. If the pause time is set to a sensible value such as 15 minutes, this attribute can indicate that repository syncing is taking too long, and that the sync is completing with very little time to spare until the next sync attempt.

Releases & Development Snapshots

Releases

You can subscribe to the atom feed to be notified of project releases.

The most recently released version of the package is 4.0.1.

4.0.1 Release (2025-05-22Z)

• Update container Alpine and JDK versions.

The compiled artifacts for the release (and all previous releases) are available on Maven Central.

Maven Modules

<dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.core</artifactId> <version>4.0.1</version> </dependency><dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.filter</artifactId> <version>4.0.1</version> </dependency><dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.github</artifactId> <version>4.0.1</version> </dependency><dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.oci</artifactId> <version>4.0.1</version> </dependency><dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.server</artifactId> <version>4.0.1</version> </dependency><dependency> <group>com.io7m.gtyrell</group> <artifactId>com.io7m.gtyrell.tests</artifactId> <version>4.0.1</version> </dependency>

Previous Releases

The changelogs for the most recent previous releases are as follows:

4.0.0 Release (2024-05-27Z)

• Update org.immutables:value:2.10.0 → 2.10.1.
• Update org.slf4j:slf4j-api:2.0.10 → 2.0.13.
• Update junit.version:5.10.1 → 5.10.2.
• Update com.github.marschall:memoryfilesystem:2.7.0 → 2.8.0.
• Update commons-io:commons-io:2.15.1 → 2.16.1.
• Update org.kohsuke:github-api:1.318 → 1.321.
• Update com.fasterxml.jackson.core:jackson-databind:2.16.1 → 2.17.0.
• Update com.fasterxml.jackson.core:jackson-core:2.16.1 → 2.17.0.
• Update com.fasterxml.jackson.core:jackson-databind:2.17.0 → 2.17.1.
• Update com.io7m.jaffirm:com.io7m.jaffirm.core:4.0.0 → 4.0.1.
• Update com.fasterxml.jackson.core:jackson-core:2.17.0 → 2.17.1.
• Update com.io7m.jlexing:com.io7m.jlexing.core:3.1.0 → 3.2.0.
• Require JDK 21.
• Repository names may now being with dots. (Tickets: 32)

3.2.0 Release (2023-05-14Z)

• Use ENTRYPOINT in the OCI image for more convenient execution.
• Include a git package in the OCI image.

3.1.0 Release (2023-05-14Z)

• Update org.slf4j:slf4j-api 2.0.6 → 2.0.7.
• Update ch.qos.logback:logback-classic 1.4.5 → 1.4.7.
• Update org.kohsuke:github-api 1.313 → 1.314.
• Update com.fasterxml.jackson.core:jackson-core 2.14.1 → 2.15.0.
• Update com.fasterxml.jackson.core:jackson-databind 2.14.1 → 2.15.0.
• Update com.github.marschall:memoryfilesystem 2.4.0 → 2.5.1.
• Add OCI images at quay.io.

3.0.0 Release (2022-04-10Z)

• Require JDK 17 (Backwards incompatible)

Development Snapshots

At the time of writing, the current unstable development version of the package is 4.0.1-SNAPSHOT.

Development snapshots may be available in the Central Portal Snapshots repository. Snapshots are published to this repository every time the project is built by the project's continuous integration system, but snapshots do expire after around ninety days and so may or may not be available depending on when a build of the package was last triggered.

Manual

This project does not have any user manuals or other documentation beyond what might be present on the page above.

Sources

This project uses Git to manage source code.

Repository: https://www.github.com/io7m-com/gtyrell

$ git clone --recursive https://www.github.com/io7m-com/gtyrell

Issues

This project uses GitHub Issues to track issues.

License

Copyright © 2023 Mark Raynsford <code@io7m.com> https://www.io7m.com Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.