Reference

The colcon-distro project is a client-server architecture, with mostly separate components between the frontend and backend.

Backend

The backend is the Sanic-based asynchronous server process which may be queried for distro snapshot information at the /get endpoint. It runs continuously and contains the logic to download packages by tarball, inspect them for discovered packages, and save the discovered information to a SQLite database. When a new snapshot request partially overlaps with one that is already cached, the backend is smart enough to only fetch the repo states it doesn’t know about yet.

server

This module has the main() that is the server process’s entry point as well as argument parsing and the rest of the Sanic interface, like route handlers.

model

This module provides the Model class, which is the main entry point to the backend of colcon-distro.

class colcon_distro.model.Model(config, db)

This class provides the high level interface which may be queried for repo sets.

Under the hood, it manages persistence via the database, but also sends work to be done and pauses requests for which the work is already in progress.

coroutine get_repo_state(repository_descriptor: RepositoryDescriptor)

Populates the passed repository_descriptor with PackageDescriptor instances in the packages set, and a repo_state_id field in the metadata dict. This may happen because all of the information was in the cache, or some or all of it may have to have been pulled from the original source.

Parameters:: repository_descriptor – The descriptor object to populate with information about its state. Must include name, type, url, and version fields.

coroutine get_set(dist_name, ref)

Returns a set of repository descriptors, by fetching them from the database if possible, and falling back to building up manually if required, after which it is saved in the database and returned.

Parameters:

dist_name – name of the distribution (eg, noetic)
ref – version control reference to fetch (currently only frozen tags and snapshots are supported).

repository_augmentation

The purpose of this module is to expose a colcon extension point for augmenting colcon_distro.repository_descriptor.RepositoryDescriptor objects with additional metadata. Some possible uses for this could include hashes of all or parts of the repository’s contents, information about whether the repository includes things like docs, tests, etc.

class colcon_distro.repository_augmentation.RepositoryAugmentationExtensionPoint

The interface for repository augmentation extensions, which work similarly to package augmentation, but for repository-level metadata. Colcon doesn’t normally concern itself with any source unit other than the package, which is why this lives here in colcon-distro (where repositories very much matter!) rather than in colcon-core with most of the other extension interfaces.

EXTENSION_POINT_VERSION = '1.0': The default priority of repository augmentation extensions.

augment_repository(path, metadata: dict, *, additional_argument_names=None): Augment the metadata dict with additional fields. The method is intended to be overridden in a subclass. :param path: Path to the repository. :param metadata: Dict on which to set metadata.

colcon_distro.repository_augmentation.augment_repository(repository_descriptor: RepositoryDescriptor): Augment the passed repository, populating its metadata dict according to available plugins.

colcon_distro.repository_augmentation.get_repository_augmentation_extensions(): Get the repository augmentation extensions in priority order.

download

This module provides classes related to efficiently accessing content from remote repositories, including whole tarball downloads, individual files, etc. Because there is no standard for this type of access, individual implementations are present for Github, GitLab, and the local filesystem; more could easily be added.

class colcon_distro.download.GitRev(repository_descriptor: RepositoryDescriptor): This class supplies asynchronous methods to download/access the contents of a remote or local git repo at a specific ref, choosing an appropriate backend depending on the url field of the repository descriptor that is passed into the constructor. The options at present are GitLab and Github, with some limited support for a local git clone (enough to use it as the rosdistro repo).

class colcon_distro.download.GitDownloader

This abstract class supplies the interface for the host-specific implementations that are used by GitRev.

coroutine download_all_to(path: Path, limit_paths: Optional[Iterable[Path]] = None) → None

Download contents of repository at the specified ref.

Parameters:

path – location on filesystem to extract/copy/clone contents to.
limit_paths – if supplied, only these paths within the repository will be extracted. This may be used to only copy certain packages for a workspace.

coroutine get_file(path: Path) → bytes

Returns contents of a single file from a git remote, using if possible a host-specific API that is faster than simply cloning the repo.

Parameters:: path – path of file within repo to fetch.

discovery

Helper functions for colcon_distro.model.Model which assist with calling into the discovery mechanisms in colcon-core.

colcon_distro.discovery.discover_augmented_packages(repo_dir): A lightweight wrapper around the colcon upstream operations of discovering packages in a particular filesystem path, and then augumenting them according to available plugins.

database

This module provides classes which are the interface to the underlying SQLite database, including multiplexing the connection across the different concurrent async activities, and also wrapping queries with a basic method interface.

class colcon_distro.database.Connection(filepath, connect_fn=None)

This manager class provides a few important capabilities to our sqlite connection. First, it mutexes it, so that commits from one coroutine don’t get interleaved with queries from another, since sqlite has no built in concept of there being multiple clients or concurrent transactions going on. Second, it starts a background task which on cancelation closes the connection.

A common instance of this class is used as a context manager. It yields the database handle and the caller must not store or continue to use it when the context has exited.

class colcon_distro.database.Database(config)

This class is a low-level wrapper on the SQLite interface, supplying function wrappers for all needed queries, with some limited processing such as converting to results between RepositoryDescriptor objects and calling the methods on that class which handle JSON serialization around the packages field.

coroutine connect_fn(db) → None: Callback function for anything we’d like to execute on a newly-opened database connection.

coroutine fetch_repo_state(desc: RepositoryDescriptor) → None: Uses the identity fields in the passed-in descriptor to search for it in the database. If found, the descriptor is populated with the row id and parsed PackageDescriptors; it not found RepositoryNotFound is raised.

coroutine fetch_set(dist_name: str, ref: str) → Iterable[RepositoryDescriptor]: Return either an iterable of RepositoryDescriptor objects if the set is in the database, or raise RepositorySetNotFound if it is not.

initialize(filepath: str) → None: Initializes a new empty database; this only ever happens at startup, so we just do it synchronously.

coroutine insert_repo_state(desc: RepositoryDescriptor) → None: Insert a repo state, setting the repo_state_id in the descriptor’s metadata dict. If the row already exists, this query will fail due to db constraints.

coroutine insert_set(dist_name: str, ref: str, repo_state_ids: Iterable[int]) → None: Insert a new set row from dist_name, name, and set of ids, all of which must exist in the repo states table or this query will fail due to db constraints.

Frontend

The frontend is colcon_distro.generate.Generator, which is synchronous and pulls a snapshot’s JSON from a running backend instance, transforming it into a spec file that can be consumed by something like vcstool. A command line interface to the generator is supplied as a colcon verb colcon generate, as well as a separate colcon download, which provides an alternative downloader that leverages the same download system as the backend (rather than using an external tool for this function).

class colcon_distro.generate.Generator(repositories_dict)

dependencies_from_descriptors(descriptors)

descriptor_set(*pkg_names, deps=False)

classmethod from_url_cache(cache_url: str, rosdistro: str, ref: str)

repositories_spec_from_descriptors(descriptors)

Common

repository_descriptor

This module provides the RepositoryDescriptor class, which is a data container for repository-level data.

class colcon_distro.repository_descriptor.RepositoryDescriptor

A descriptor for a repository at a moment in time. The identification items are the name, type (eg git), url, and version (a tag or hash), all of which typically come from a rosdistro source entry. The remaining items are path, which may be populated if the repo’s contents are available on the filesystem (via checkout or tarball extraction); packages, which is a list of PackageDescriptor objects; and metadata, which is a dict that may be used for storing additional information.

Similar to PackageDescriptor from colcon-core, this class is intentionally light on implementation. It is meant to be a data container that is passed around and acted on by other modules, rather than supplying its own suite of methods.

classmethod from_distro(name: str, source_dict: dict): Construct a descriptor from the name and source dict information that are typically in a rosdistro’s distribution.yaml.

identity(): An identification tuple used for hashing and equality checks. Returns None if any of the required fields are unset.

packages_dicts(metadata_inclusions=None) → List[dict]: Returns the packages list as a list of package dicts, ready to be serialized either to database or in a JSON HTTP response.

parse_packages_dicts(packages_dicts: List[dict]): Parses the passed-in list of package dicts, and sets the packages list to the corresponding PackageDescriptor objects.