Installation Manual - Service and Plugins

Introduction

The DQ0 Service provides a catalog and launch service for DQ0 Plugins. Plugins are executable jobs that perform analysis, machine learning or privacy preserving tasks.

Prerequisites

The DQ0 Service ships pre-built as a binary executable for the following platforms:

• Linux (tested on CentOS 7)
• Mac OS, Version 10.15 or higher
• Windows 10

Plugins

Plugins can be added or removed to/from the service at any time - even at runtime. A configuration file is available in the service where all services are listed with all necessary parameters. An entry of a plugin contains:

• Plugin Identifier (name)
• Environment Variables
• Dependencies to other packages / plugins
• Container information (name of the images, repository, cloud vendor, etc)

This information is required to start a new independent instance either locally or remotely (as a container instance) when using a plugin.

Example Service plugin configuration (snippet):

# content of config.yaml (Service)
...
plugins:
    ...
    # sample python plugin with dependency to source code version of dependency 'dq0-sdk' executing locally
    -   plugin: 'dq0.plugin_1'
        executor: 'local'
        command: 'python'
        arguments: ['-m', 'dq0.plugin_1']
        environment: 'PYTHONPATH=$PYTHONPATH:/path/to/dq0-sdk/:/path/to/dq0-plugin_1'

    # sample plugin without external dependencies
    -   plugin: 'my_external_plugin'
        executor: 'remote'
        command: 'python'
        arguments: ['-m', 'my_external_2']
        environment: 'ENV1=123'

After the plugins have been entered, they can be used in Service and addressed with “dq0.plugin_1” and “my_external_plugin” from outside. This means that dq0-platform can call the service with this plugin name.

Example Plugin (written in Python):

• Define plugin configuration (my_plugin.yaml)

version: 3.6.2
name: My Plugin
description: Description of my plugin
author: Me
author_email: me@here.com

dockerfile: /path/to/Dockerfile

requirements: /path/to/requirements.txt

entry_point:
    parameters:
      param1: {type: string, default: 'test'}
      param2: {type: float, default: 123.66}
    command: python my_plugin.py {param1} {param2}

• Define sample plugin in Python (my_plugin.py)

import argparse

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--param1', required=False,
                        type=str, dest='param1')
    parser.add_argument('--param2', required=False,
                        type=float, dest='param2')                    
def run():
    args, unknown = parse_args()

    # print hello world
    print('hello world')

    print(args.param1)    
    print(args.param2)    

if __name__ == "__main__":
    run()

• Provide Dockerfile

FROM python:3.8
WORKDIR /code
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY my_plugin.py .
CMD [ "python", "./my_plugin.py", "--param1=test" ]