This page will guide you through the process of importing your model container into Modzy. Throughout this process, Modzy will perform security scans on your container to check for any vulnerabilities, run functional tests to ensure your container meets the proper specification, and you will have the opportunity to document your model in detail.
To follow this guide, verify that your model meets the Open Model Interface (OMI) specification. You can either manually build your container or leverage Chassis to automatically containerize your model code into an OMI-compliant container.
Begin importing your container by navigating to the models page within your Modzy account and select Add New Model.
Next, click the Import Container option to launch the model deployment process.
Give your model a name and version. Note: After deploying your model, you can add new versions incrementally according to the semantic versioning system.
Begin importing your model container by starting the Choose Import Method step.
Here, you will have two choices for importing your container:
- Docker Hub: import from hub.docker.com
- Docker: import from another Docker registry
- Upload .tar file: import from saved tar archive version of Docker container
If you select the Upload .tar file option, ensure you created your .tar file using the following command:
docker save -o my-container.tar your-container-image
If you select the Docker option, fill in the three fields as follows:
Docker registry URL
You must convert your docker container image URL into the Docker Registry HTTP API v2 protocol. Regardless of the registry you are using (Dockerhub, AWS ECR, Azure or GCP Container Registry, etc.), the URL Modzy expects will follow this general structure:
For example, Modzy's Echo Model hosted on Dockerhub (https://hub.docker.com/r/modzy/grpc-echo-model) converts to
https://registry.hub.docker.com/v2/modzy/grpc-echo-model/manifests/1.0.0 in HTTP API v2 protocol format.
If your container image repository is public, then you can uncheck the "Docker registry requires authentication" box and continue. Otherwise, fill in the Username and Password fields with your registry credentials. For Dockerhub, these values are your username and password, but for cloud registries, these values correspond to your cloud provider Access Key (Username) and Secret Access Key (Password).
If you specify a valid Docker registry link and credentials (if needed), the Import model progress bar on the right of the screen will display the status of the model import.
While Modzy is downloading your model container file, you can start step two: Select hardware requirements. On this page, select the hardware option that best suits your model. As a general rule of thumb, select an option with RAM that exceeds the size of your Docker container. Further, if your model is enabled to parallelize the processing of data while making predictions, select the number of CPU cores and/or GPUs that most appropriately fit your model's intended use.
The Timeouts section allows you to specify timeouts Modzy will use as validation checks during two phases of your model's inference lifecycle in production:
- Status timeout: corresponds to your model spinning up and performing any required model instantiation
- Run timeout: corresponds to a single inference run (includes real-time or batch inference calls)
As a general rule of thumb, set these values high enough to account for large batches of data (if relevant for your model), but not too high where a user will wait an unreasonable amount of time. Click Save and Continue when complete to kick off the first of two functionality tests Modzy will execute.
After the "Import Model" progress bar completes, Modzy will move onto the next step in the process: spinning up your model container and checking the
Status() remote procedure call. As this test is occurring, you will notice the second progress bar for the "Verify Status Endpoint" will start to complete.
Pending a successful status message, start the Test model on sample input data step and upload a sample data file. This file should be representative of the data you would expect end users to run through your model. After uploading this file, select Start Job, which will kick off the second functionality test Modzy executes. For this test, Modzy will pass the sample input file through your model for predictions and ensure the container can return the model output through Modzy's APIs.
As Modzy is executing this test a progress bar will indicate the status of the job. Note that inference times can vary between models and depend entirely on the size of the model, optimization of the model, and the size of the input data. Pending a successful test job, you will see the full sample API request and responses.
Lastly, complete steps four and five - "Add Documentation" and "Assign Tags and Categories" - to fully document your model. In these steps, you can add a descriptions, performance metrics, detailed technical information, and anything else relevant an end user would need to know before using your model. Additionally, you can assign tags to your model to promote the discoverability of your model across your entreprise.
You are now ready to publish your model and begin submitting production inferences! After clicking Publish, view your model's details page and let your team know they can begin leveraging it in their applications.
Updated 3 months ago