Overview

This sample containerizes a basic image classification model. It uses the Modzy Python Template repository as a starting point.

The sample covers:

1. Template adjustments to fit a model and the Dockerfile addition.

2. Model script completion.

3. Build and completion a YAML file.

4. Unit tests creation and container validation.

🚧

Requirements

This process requires the installation of Docker, Python and the Model Project Template Repository.

The Modzy Python template repository

This repository serves as a starting point to package a Python model in a Docker container that meets the Modzy API specifications. It adds a run method to the model to run inference jobs.

Repository contents

The template comes with a Web micro-framework, an HTTP object generator code, Gunicorn, and preset Dockerfile files required to run the model.

Here is the full content list:

Edit template scripts

This guide goes over template adjustments and the Dockerfile addition.

Add a Dockerfile

Base Dockerfile

Select a base Docker image from the template Dockerfile that fits the different machine learning framework or dependencies the model requires to run. This ensures that the environment in the model container can run the model because it has the correct version of frameworks, Python, CUDA, and others.

Docker Hub holds thousands of pre-built Docker images to choose from as the base image in the Dockerfile. This template uses a vanilla Python 3.6 image.

In this sample, we start our Dockerfile with a base Docker image that contains the correct versions of Python, Pytorch, and CUDA that our model requires to run.

Add your dependencies

Navigate to the requirements.txt file and add any additional model dependencies. The Dockerfile uses the pip package-management system to install all libraries in the requirements.txt file. Keep all existing requirements:

click==7.0
flask==1.1.1
gunicorn==19.9.0
humanfriendly==4.18
itsdangerous==1.1.0       # via flask
jinja2==2.10.1            # via flask
markupsafe==1.1.1         # via jinja2
Pillow==6.0.0
pyreadline==2.1           # via humanfriendly
python-box==3.4.2
werkzeug==0.15.4          # via flask

In this sample, we only require one package that does not come pre-installed in our base docker image: Pillow.

Build the Dockerfile

Open a command-line interface to build the Docker image.

Use the -t argument and tag the Docker image with an intuitive and representative model name. The last argument to this build command specifies the file path to the Dockerfile.

docker build -t image-classification .

In this sample, we navigate in the command line to the same directory as our Dockerfile. We specify our current working directory path with a period.

Run the Dockerfile

Run the container to ensure the Dockerfile correctly satisfies the model requirements.

Use a docker run command to spin up the container:

docker run --rm -it --runtime=nvidia -v $(pwd):/work -w /work image-classification bash

In this sample, we run a command that opens an interactive container where we can test our model code in the environment to ensure it executes properly.

This command contains the following arguments:

Because the current working directory is mounted to the container, all changes performed to the code carry over to the container’s /work directory. This allows you to make code changes and test without having to rebuild the image and run the container after every change.

Complete the model script

The model.py script holds all the model code including the loading of weights and dependencies, variable instantiation, and inference code.

Add requirements

Paste in any library requirements the model code needs to successfully execute. Do not delete the existing import statements. Edit the THIS_DIR directory path if necessary and add any other relevant paths to the model weights, helper scripts, lookup files, etc.

import json
import os

import sys
import ast
import torch
from PIL import Image
from torchvision import models, transforms

from flask_psc_model import ModelBase, load_metadata

# create data directory
root = os.path.dirname(os.path.abspath(__file__))
labels_path = os.path.join(root, 'imagenet_classes.txt')

In this sample, we import Pytorch and a few other dependencies. We also specify the path to our class labels, which exist in a file we created in our directory called imagenet_classes.txt.

Initialize

Use the __init__ function to load any model weights and initialize model object class attributes:

def __init__(self):
    """Load the model files and do any initialization.

    A single instance of this model class will be reused multiple times to perform inference
    on multiple input files so any slow initialization steps such as reading in a data
    files or loading an inference graph to GPU should be done here.

    This function should require no arguments, or provide appropriate defaults for all arguments.

    NOTE: The `__init__` function and `run` function may not be called from the same thread so extra
    care may be needed if using frameworks such as Tensorflow that make use of thread locals.
    """

    self.model = models.resnet101(pretrained=True)

    # labels
    with open(labels_path, 'r') as f:
        self.labels = ast.literal_eval(f.read())

    # define data transform
    self.transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

In this sample, we download a pre-trained image classification model available via Pytorch’s torchvision library. We also use the __init__ function to read in our class labels and define a data transformation.

Add any helper functions to the model class if necessary:

def preprocess(self, image):

    # do data transformation
    img_t = self.transform(image)
    batch_t = torch.unsqueeze(img_t, 0)

    return batch_t

def postprocess(self, predictions):

    percentage = torch.nn.functional.softmax(predictions, dim=1)[0]

    _, indices = torch.sort(predictions, descending=True)
    top5_preds = [(self.labels[idx.item()], percentage[idx].item()) for idx in indices[0][:5]]

    return top5_preds

In this sample, we create a data preprocessing function and a predictions post-processing function to make our inference process easier to follow.

Add inference code

Insert any required inference code into the run method:

def run(self, input_path, output_path):
    """Run the model on the given input file paths and write to the given output file paths.

    The input files paths followed by the output file paths will be passed into this function as
    positional arguments in the same order as specified in `input_filenames` and `output_filenames`.
    """
    # read in data
    image = Image.open(input_path)

    # data preprocessing
    img = self.preprocess(image)

    # perform inference
    self.model.eval()
    output = self.model(img)

    # post process
    results = self.postprocess(output)

    # save output
    results = {'results': results}

    with open(output_path, 'w') as out:
        json.dump(results, out)

In this sample, we include five steps: data ingest, data preprocessing, inference, prediction post-processing, and output saving.

Update the model name

Adjust the template to fit your model.

Edit the code below the if __name__ == “__main__” line at the bottom of the script.

Open model.py and update the ModelName class to a preferred model name:

class ModelName(ModelBase):

    #: load the `model.yaml` metadata file from up the filesystem hierarchy;
    #: this will be used to avoid hard-coding the below filenames in this file
    metadata = load_metadata(__file__)

    #: a list of input filenames; specifying the `input_filenames` attribute is required to configure the model app
    input_filenames = list(metadata.inputs)

    #: a list of output filenames; specifying the `output_filenames` attribute is required to configure the model app
    output_filenames = list(metadata.outputs)

In the same file, update any other ModelName instances to the name of the base class defined in model.py:

if __name__ == '__main__':
    # run the model independently from the full application; can be useful for testing
    #
    # to run from the repository root:
    #     python -m model_lib.model /path/to/input.txt /path/to/output.json
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('input', help='the input data filepath')
    parser.add_argument('output', help='the output results filepath')
    args = parser.parse_args()

    model = ModelName()
    model.run(args.input, args.output)

In the app.py script in the parent directory of the repository, update any other ModelName instances to the name of the base class defined in model.py:

#!/usr/bin/env python3
"""The HTTP API application.

The model class is used to configure the Flask application that implements the
required HTTP routes.
"""

from flask_psc_model import create_app
from model_lib.model import ModelName

app = create_app(ModelName)

if __name__ == '__main__':
    app.main()

In the tests/test_model.py script, update the two instances of ModelName as above:

import unittest

from model_lib.model import ModelName
from flask_psc_model import ModelBase


class TestModel(unittest.TestCase):

    def setUp(self):
        self.model = ModelName()

In the container spun up at the end of the Dockerfile step, test the model.py script. Use a command-line Python command. Ensure this command abides by the specifications in your if __name__ == “__main__” code:

python -m model-lib.model ./data/dog.jpg ./results.json

In this sample, we run our model code script as a module (-m argument), specify the input file path to dog.jpg, and indicate we want our output written to a file called results.json.

The YAML file

The YAML file contains model metadata.

The top portion of this file allows you to document any important information about the model. It includes a description of the task it performs, what data it trained on, how it performed on validation datasets, etc.

The bottom portion of this file specifies the inputs to the model, hardware and memory requirements to run it, and runtime timeout thresholds.

Build a YAML file

Go to the model.yaml file in the template repository.

YAML is a human-readable data serialization language commonly used for configuration files.

In this case, the model.yaml configuration file contains the model metadata that the Modzy API requires to spin up the model container and run inference.

Inputs

Set the input file name

To run the model container via Modzy, the input file must match the name specified here.

In this sample, we call the input file image. We omit the file extension because our model supports different filetypes.

Set the input file types

Select the supported input file types included as MIME types. Add them under the acceptedMediaTypes section.

In this sample, the model accepts JPEG and PNG encoded images.

Set the input max size

Set the maximum amount of data the model can process per input item.

In this sample, the model accepts an image up to 1 megabyte in size.

Set an input description

Add a short input description in the description section. Include input item details such as options, dependencies, requirements, and other special considerations.

If the model requires multiple inputs to run, each input must hold its own description section under the inputs section:

# Please indicate the names and kinds of input(s) that your model
# expects. The names and types you specify here will be used to
# validate inputs supplied by inference job requests.
inputs:
  # The value of this key will be the name of the file that is
  # supplied to your model for processing
  image:
    # The expected media types of this file. For more information
    # on media types, see:
    # https://www.iana.org/assignments/media-types/media-types.xhtml
    acceptedMediaTypes:
    - image/jpeg
    - image/png
    # The maximum size that this file is expected to be.
    maxSize: 1M
    # A human readable description of what this file is expected to
    # be. This value supports content in Markdown format for including
    # rich text, links, images, etc.
    description: Image file to be classified with model.
    # Accepted image types: jpeg or png encoded images.
    
    config.json
    # filename
    # The expected media types of this file. For more information
    # on media types, see:
    # https://www.iana.org/assignments/media-types/media-types.xhtml
    acceptedMediaTypes:
    - application/json
    # The maximum size that this file is expected to be.
    maxSize: 1M
    # A human readable description of what this file is expected to
    # be. This value supports content in Markdown format for including
    # rich text, links, images, etc.
    description: Configuration file that tells the model which classification to execute.

In this sample, the model requires an image and a configuration file.

Outputs

Repeat the input steps above, this time for the outputs section:

outputs:
  results.json:
    # The expected media types of this file. For more information
    # on media types, see:
    # https://www.iana.org/assignments/media-types/media-types.xhtml
    mediaType: application/json
    # The maximum size that this file is expected to be.
    maxSize: 1M
    # A human readable description of what this file is expected to
    # be. This value supports content in Markdown format for including
    # rich text, links, images, etc.
    description: Top five classifications with their respective prediction probabilities.

In this sample, we output a JSON file called results.json. This file contains the top five image class predictions along with their respective prediction probabilities.

Resources

Lastly, complete the resources section of the YAML file. Include the memory, CPU, and GPU amounts required to run the model:

# The resources section indicates what resources are required by your model
# in order to run efficiently. Keep in mind that there may be many instances
# of your model running at any given time so please be conservative with the
# values you specify here.
resources:
  memory:
    # The amount of RAM required by your model.
    size: 512M
  cpu:
    # CPU count should be specified as the number of fractional CPUs that
    # are needed. For example, 1 == one CPU core.
    count: 1
  gpu:
    # GPU count must be an integer.
    count: 0
# Please specify a timeout value that indicates a time at which
# requests to your model should be canceled. If you are using a
# webserver with built in timeouts within your container such as
# gunicorn make sure to adjust those timeouts accordingly.
timeout:
  # Status timeout indicates the timeout threshold for calls to your
  # model's `/status` route.
  status: 20s
  # Run timeout indicates the timeout threshold for files submitted
  # to your model for processing.
  run: 20s

In this sample, the model container runs with 512 megabytes of memory on 1 CPU. It has timeout thresholds for 20 seconds each.

If after 20 seconds of spinning up this model container, there is not a successful response from the GET /status route, the model container shuts down. Similarly, if after 20 seconds of calling the POST /run route, the container does not return an output, the model container shuts down.

Unit tests and validation

Unit tests

Unit tests validate if the containerized model works as intended.

🚧

This step is not required but is strongly encouraged.

Create a subdirectory called /data under the tests/ directory.

In /data, create two types of validation tests: one for successful data examples and one for validation error tests. In the tests/data directory, create two more subdirectories: /example and /validation-error.

mkdir data
cd data/
mkdir example
mkdir validation-error

Within each subdirectory, create folders titled by the name of the test.

cd example/
mkdir 001 002 003
cd ../validation-error
mkdir invalid-file invalid-size empty-file

In this sample, we include three correct example tests (0001/, 0002/, and 0003/). We also include three validation error tests (invalid-file/, invalid-size/, and empty-file/).

The template repository contains code that allows the Python unittest module to execute.

This unittest module iterates over the different subdirectories inside the tests/data/ directory. Each test subdirectory within the tests/data/example/ directory must contain:

1. the sample data input or inputs the model expects, named according to the YAML file.

2. the expected model output for the respective input(s).

Upon execution, the unittest module runs the model against the input, take the run’s actual output, and compares it to the expected output file located in each example test folder. If the actual and expected output match, then the unittest module returns an OK message. If they do not match, it returns a FAILURE message.

The unittest module behaves similarly when it iterates over the tests within the tests/data/validation-error/ directory. Here, each test folder must contain:

1. the sample data input or inputs the model expects, named according to the YAML file.

2. a file called message.txt that holds the expected error message for the respective input(s).

Upon execution, the unittest module runs each input per folder against the model and compares the actual output (in this case an error message) with the expected error message. If they match, then the unittest module returns an OK message. If they do not match, it returns a FAILURE message.

In the running container, test the unittest module via a Python command-line command.

python -m unittest

If this module returns an OK message, the unit tests have run successfully. Continue to the final validation test to ensure the model container works correctly.

Container validation

The validation test ensures the model container works correctly.

Rebuild the Docker image

Exit the running container and remove the Docker image built at the beginning:

docker image rm image-classification

Up until this point, the unit tests that run interactively inside the container confirm the code changes work. However, since Docker images are immutable, the Docker image did not capture these changes. When the Modzy API spins up the model container, no interactive shell environment exists.

Rebuild the Docker image to capture all the changes:

docker build -t image-classification-final .

Test the container

To test the container, simulate the Modzy API spinning it up.

Run the container on a local available port:

docker run --runtime=nvidia --name Image-Classification -e PSC_MODEL_PORT=8080 -p 8080:8080 -v /data:/data -d image-classification-final

In this sample, we run on port 8080.

This command contains the following arguments:

Check the container’s status endpoint:

curl -s "http://localhost:8080/status"

This should return a 200 OK status:

{
 "message":"ready",
 "status":"OK",
 "statusCode":200
}

Run an inference test providing the input file path. The input file name needs to match the YAML file:

curl -s -X POST -H "Content-Type: application/json" \
    --data "{\"type\":\"file\",\"input\":\"/data\",\"output\":\"/data\"}" \
    "http://localhost:8080/status"

In this sample, we have an image file in the /data.

This should also return a 200 OK status.

{
 "message":"ready",
 "status":"OK",
 "statusCode":200
}

Validate that an output file was written to the /data directory and shut down the container:

curl -s -X POST "http://localhost:8080/shutdown"

Clean up the environment and remove the container:

docker rm <insert container name here>