API reference

Utilities

process.util.wrap(string)

Formats a long string as a help message, and returns it.

process.util.walk(paths)

process.util.get_publisher()

process.util.consume(*args, **kwargs)

process.util.decorator(decode, callback, state, channel, method, properties, body)

Close the database connections opened by the callback, before returning.

If the callback raises an exception, shut down the client in the main thread, without acknowledgment. For some exceptions, assume that the same message was delivered twice, log an error, and nack the message.

process.util.get_or_create(model, data)

process.util.create_note(collection, code, note, **kwargs)

process.util.create_step(name, collection_id, **kwargs)

process.util.delete_step(*args, **kwargs)

Delete the named step and run any finish callback only if successful or if the error is expected.

process.util.create_warnings_note(collection, category)

process.util.create_logger_note(collection, name)

Models

class process.models.Default

class process.models.Collection(*args, **kwargs)

A collection of data from a source.

There should be at most one collection of a given source (source_id) at a given time (data_version) of a given scope (sample or not). A unique constraint therefore covers these fields.

A collection can be a sample of a source. For example, an analyst can load a sample of a bulk download, run manual queries to check whether it serves their needs, and then load the full file. To avoid the overhead of deleting the sample, we instead make sample part of the unique constraint, along with source_id and data_version.

class Transform(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

clean_fields(exclude=None)

Clean all fields and raise a ValidationError containing a dict of all validation errors if any occur.

get_upgraded_collection()

Returns existing upgraded collection or None.

Returns:

upgraded collection

Return type:

Collection

get_compiled_collection()

Returns existing compiled collection or None.

Returns:

compiled collection

Return type:

Collection

get_root_parent()

Returns “root” parent of collection. Basically traverses the tree to the top.

Returns:

root collection

Return type:

Collection

class process.models.CollectionNote(*args, **kwargs)

A note an analyst made about the collection.

class Level(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

class process.models.CollectionFile(*args, **kwargs)

A file within the collection.

class process.models.ProcessingStep(*args, **kwargs)

A step in the lifecycle of collection file.

class Name(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

class process.models.CollectionFileItem(*args, **kwargs)

An item within a file in the collection.

class process.models.Data(*args, **kwargs)

The contents of a release, record or compiled release.

class process.models.PackageData(*args, **kwargs)

The contents of a package, excluding the releases or records.

class process.models.Release(*args, **kwargs)

A release.

class process.models.Record(*args, **kwargs)

A record.

class process.models.CompiledRelease(*args, **kwargs)

A compiled release.

class process.models.ReleaseCheck(*args, **kwargs)

The result of checking a release.

class process.models.RecordCheck(*args, **kwargs)

The result of checking a record.

Loader

process.processors.loader.file_or_directory(string)

Checks whether the path is existing file or directory. Raises an exception if not

process.processors.loader.create_collection_file(collection, filename=None, url=None, errors=None)

Creates file for a collection and steps for this file.

Parameters:

• collection (Collection) – collection

• filename (str) – path to file data

• errors (json) – errors to be stored

Returns:

created collection file

Return type:

CollectionFile

Raises:

InvalidFormError – if there is a validation error

process.processors.loader.create_collections(source_id, data_version, sample=False, upgrade=False, compile=False, check=False, scrapyd_job='', note='', force=False)

Creates main collection, note, upgraded collection, compiled collection etc. based on provided data

Parameters:

• source_id (str) – collection source

• data_version (str) – data version in ISO format

• sample (boolean) – is this sample only

• upgrade (boolean) – whether to plan collection upgrade

• compile (boolean) – whether to plan collection compile

• check (boolean) – whether to plan schema-based checks

• scrapyd_job (str) – Scrapyd job ID

• note (str) – text description

• force (boolean) – skip validation of the source_id against the Scrapyd project

Returns:

created main collection, upgraded collection, compiled_collection

Return type:

Collection, Collection, Collection

Scrapyd

process.scrapyd.configured()

Returns:

whether the connection to Scrapyd is configured

Return type:

bool

process.scrapyd.spiders()

Returns:

the names of the spiders in the Scrapyd project

Return type:

list

Command-line interface

class process.cli.CollectionCommand(stdout=None, stderr=None, no_color=False, force_color=False)

add_arguments(parser)

Adds default arguments to the command.

add_collection_arguments(parser)

Adds arguments specific to this command.

handle(*args, **options)

Gets the collection.

handle_collection(collection, *args, **options)

Runs the command.