Writing your own transformation functions¶

The single_input decorator¶

If you are writing a transformation function that takes a single dataset as its input, wrap it with the single_input decorator:

from ocelot.transformations.utils import single_input

@single_input
def doubler_variation(dataset):
    for exc in dataset['exchanges']:
        exc['amount'] *= 2
return [dataset]  # Note that this is a list

If you use the single_input decorator, be aware of the following rules:

• The input will be a single dataset
• The function should return a list of datasets, as transformation functions can split datasets

Report logging¶

Report logging is data which will go into the HTML report which is produced at the end of each Ocelot model run. Most of the time you will want to provide logging data that will be turned into tables in this report. To tell the report generator how to define these tables, define the function attribute __table__ as follows:

def some_transformation(data):
    return data

some_transformation.__table__ = {
    'title': 'Name of title to put in report',
    'columns': ["names", "of", "columns"]
}

__table__ should define:

• title: The title of the function data provided
• columns: A list of column headings

There should also be logging inside the transformation function. You need to retrieve the ocelot logger, and log using the log level info. So, a more complete example would actually look something like this:

logger = logging.getLogger('ocelot')  # Very important

def count_exchanges(data):
    """Function that counts things.

    Does not change any data."""
    for ds in data:
        logger.info({
            'type': 'table element',
            'data': [ds['name'], len(ds['exchanges'])]
        })
    return data

count_exchanges.__table__ = {
    'title': 'Count the number of exchanges in each dataset',
    'columns': ["Name", "# of exchanges"]
}

Log messages should be a dictionary, with the key type (and value table element for tabular data). The key data should give a list of data in the same order as columns.

If tables don’t work for your transformation function, you can skip the __table__ attribute, and just log list element log messages:

logger = logging.getLogger('ocelot')

def count_exchanges(data):
    """Function that counts things.

    Does not change any data."""
    for ds in data:
        logger.info({
            'type': 'list element',
            'data': "Dataset <b>{}</b> has <i>{}</i> exchanges".format(
                ds['name'], len(ds['exchanges'])
            )
        })
    return data

Messages that have the type list element can be HTML.

Detailed logging¶

Transformation functions should also write detailed logs that can be used afterwards to debug exactly what happened to a dataset. This logging uses a different data format and logger.

First, retrieve the detailed logger:

detailed = logging.getLogger('ocelot-detailed')

Then, inside the body of a function, call this logger (again using the info log level):

def count_exchanges(data):
    """Function that counts things.

    Does not change any data."""
    for ds in data:
        detailed.info({
            'ds': ds,
            'message': 'Some message about number of exchanges',
            'function': 'count_exchanges'
        })
    return data

In the detailed log messages, the following keys are required:

• ds: The dataset object being changed. If you don’t have access or can’t provide this dataset, don’t emit a detailed log message.
• function: The name of the function (as a string)
• message: The message to log (as a string)