AWS

Deployment Guide for Launching Modzy on AWS

Overview

This guide provides step-by-step instructions for deploying Modzy into an existing AWS VPC using the Modzy AWS Terraform module. Note that the VPC must meet the requirements for an AWS EKS cluster as outlined in the AWS documentation.

The guide assumes some familiarity with Terraform by Hashicorp/use of Terraform modules to provision infrastructure in the cloud.

Architecture

With the VPC and subnets created prior, executing the steps outlined in the guide builds the following Modzy environment in the AWS Cloud:

  • A highly available architecture that spans three Availability Zones.

AWS Services

AWS Services used by Modzy


Deployment Cost

The overall cost of a Modzy environment will vary based on the type, quantity and purchase method of the underlying AWS resources. Prices are subject to change; for current pricing, visit each AWS service home page.

Modzy Platform

The Modzy Platform resources are the minimum required set of resources to be able to run models but not including any of the inference nodes used to run models. The pricing list is smaller than the services used list as some of the costs below, like Simple Storage Service (S3), could grow depending on the amount of customer data in a deployment. Other services like Simple Email Service give you the first 62k emails per month for free and are unlikely to ever add a significant cost to a deployment. If the service is going to cost less than $1/month and is unlikely to grow, it’s not been included in the pricing list.

Table 1. Modzy core platform monthly component costs w/default parameters and on-demand pricing in us-east-1

ResourceQuantityUnit Price/MonthCost
Amazon Elastic Kubernetes Service (EKS) Control Plane1$73.00$73.00
Amazon Elastic Compute Cloud (EC2) - Modzy Platform3 x m5.2xlarge (8 vCPU + 32GiB RAM) + 100GB SSD Storage$280.32 Platform Server + $10.00 storage volume$870.96
Load Balancer1$16.42/month + bandwidth processed$20
NAT Gateway3$32.85/month$98.55
AWS Simple Storage Service (S3)100GB in an empty environment (1TB recommended)$0.023/GB/month$2.30 to $23
RDS Postgres Database (Multi-AZ H/A Deployment)1 Multi-AZ db.r5.xlarge (recommended) + 100GB SSD Storage$211.70 (db.t3.large)
$730.00 (db.r5.xlarge)
$10 storage volume
$221.70 (small install)
$740 (recommended production)
Subtotal$1,307.21

Modzy Inference Nodes

Modzy inference nodes run your AI/ML jobs. The type, size, and quantity of inference nodes are determined by you/configurable parameters for the deployment. EC2 pricing is by the second with a minimum run time of 10 minutes as the Modzy platform has a scheduling optimization to reduce result latency on a busy system by ensuring no other jobs need to be run for up to 10 minutes before terminating an inference node.

Table 2. Sample Modzy inference nodes hourly cost at on-demand pricing rates in us-east-1

Inference Node TypeGPU(s)vCPU(s)RAM (GiB)Hourly Cost
Small CPU m5.large-28$0.096
Medium CPU - m5.xlarge-416$0.192
Large CPU - m5.2xlarge-832$0.384

Deployment Cost Examples

Monthly cost of 10 concurrent processing engines running 24x7 using small CPU inference nodes
$2000/month = ($0.096/hr x 730 hrs x 10 nodes) CPU inference nodes + $1300 Modzy platform

Monthly cost of 10 concurrent processing engines running 24x7 using small GPU inference nodes
$7870/month = ($0.90 x 730 hrs x 10 nodes) GPU inference nodes + $1300 Modzy platform

These estimates assume 24x7 operation of each inference node, but the platform will scale down any unused inference nodes that are not pre-provisioned as part of the model settings after 10 minutes of inactivity. Another option to reduce costs if the workload has at least a partial steady-state is to pre-purchase EC2 reserved instances where you pay upfront for future utilization in return for a discount.

Planning the Deployment

AWS Account

If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad.

Technical Requirements

Before provisioning the Modzy infrastructure as outlined in the steps that follow, check your account service quota limits at https://console.aws.amazon.com/servicequotas/home and ensure you have sufficient capacity available before launching the deployment, otherwise, the deployment will fail.

Table 3. Service Quota Utilization

ResourceDefault QuotaModzy Default Configuration
VPCs5 per region1
VPC Elastic IPs5 per region4
VPC security groups2,500 per region6
IAM roles1,000 per account3
Auto Scaling Groups200 per regionMore than 3
(3 for the platform + number of inference node groups configured)
m5.2xlarge ec2 instances3
Other ec2 instancesNumber and types configured for inference node groups

Table 4. Other Technical Requirements

RequirementDescription
RegionsAmazon EKS and Amazon EFS aren’t currently supported in all AWS Regions. For a current list of supported Regions, see Service Endpoints and Quotas in the AWS documentation.
Key pairMake sure that at least one Amazon EC2 key pair exists in your AWS account in the region for the deployment. Make a note of the key pair name as it’s a required parameter in the deployment launch screen. To create a key pair, follow the instructions in the AWS documentation. If the deployment is for testing or proof-of-concept purposes, we recommend that you create a new key pair instead of specifying a key pair that’s already being used by a production instance.
IAM permissionsTo deploy Modzy, you must have credentials with IAM permissions for the resources and actions the templates will deploy. The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions.

Deployment Steps

The following steps deploy Modzy in your existing AWS infrastructure, into a VPC that must be configured prior to the deployment. This VPC must meet the minimum requirement of having 3 public and 3 private subnets provisioned and configured.

Prerequisites

Domain and SSL Certificate

  • Control of an internet domain. A fully qualified domain name (FQDN) for the planned Modzy instance is required for provisioning.
  • Use a service (such as Let’s Encrypt) to request a certificate for the FQDN with the domain name and wildcard (Example: modzy.mycompany.com and *.modzy.mycompany.com). The certificate and key files are required for platform configuration after installation. This certificate will be used by the API and the image registry.
  • Ensure your domain has been verified in AWS Simple Email Service (SES) in the region in which you deploy
    • SES is used to send user account registration emails

SAML Identity Provider Client Configuration

The following items are needed to configure the Modzy client in the SAML Identity Provider:

  • SSL URL: https://FQDN/saml/SSO

  • Audience URI (SP Entity ID): com.modzy.saml.sp

    Where FQDN is the fully qualified domain name for the Modzy instance (Ex.: mymodzy.mydomain.com) and must match the FQDN input parameter the Modzy instance is to be provisioned with.

Client Tools
The following tools are required on the client machine from which provisioning of Modzy environment will be executed:

The following files are needed to configure the Modzy instance :

  • A Modzy license file

  • The metadata (file) for the SAML Identity Provider to be used

Additionally, networking information (VPC Id, Subnet Ids) for the VPC to provision the Modzy environment into must be determined prior to execution. These and other parameter values will be needed to set up variables for the Modzy terraform module.

Provisioning Modzy Environment

Provisioning a Modzy Environment is comprised of the following steps:

  • Provision Infrastructure
    • Setup Modzy module with input variables
    • Run Terraform Init, Plan and Apply to create the infrastructure

  • Deploy Modzy Platform
    • Install Modzy Platform
    • Upload license, configure and deploy


Provision Infrastructure

  1. Setup Modzy module with input variables

    • Clone the repository as shown below for sample files. The example creates the folder modzy (this can be any valid name for a directory) and clones the contents of the repository :

      git clone https://github.com/modzy/terraform-aws-modzy-sample modzy
      

      This should create the directory with contents as shown below:

      modzy
      │- providers.tf
      │- modzy.tf
      
    • Edit the modzy.tf template file to specify appropriate input variables for the Modzy environment to provision

      Note: Each variable is described in detail in the Modzy Terraform module documentation

  2. In a command-line/shell, from within modzy directory,

    • Set the AWS credentials profile for the command-line/shell window

      export AWS_PROFILE=your-aws-profile-name
      
    • Execute the following in turn, reviewing the output from each :

      • Init
        terraform init
        
      • Plan
        terraform plan
        
      • Apply
        terraform apply
        

      On successful completion of Terraform Apply, two local files will be created which will be used in the next section to deploy/configure the Modzy platform :

    • kubeconfig_installation-identifier

    • replicatedconfig__installation-identifier.yaml

      (where installation-identifier is the value supplied for the installation-identifier variable and a postfix of randomly generated string of characters)

      Note: In addition, secrets used by the installation are created in Secrets Manager in the form /modzy/{installation-identifier}/secret-type

Deploy Modzy Platform

  • Install Modzy platform

    • Set the AWS credentials profile for the command-line/shell window

      export AWS_PROFILE=your-aws-profile-name
      
    • Use the kubeconfig file generated above to access the cluster context as shown below :

      export KUBECONFIG=$(pwd)/kubeconfig_<installation-identifier>
      
    • Verify you are in the desired kubernetes cluster context by executing

      kubectl config current-context
      

      Expected output: eks_installation-identifier

    • Export install command from Terraform output with :

      KOTS_CMD=$(terraform output -raw kots_install_cmd)
      
    • Execute the installation :

      sh -c $KOTS_CMD
      

      (Note: There may be a need to execute the above with sudo for sufficient permissions)

      The following output should be displayed on completion of execution :

      kubectl kots install modzy/stable \
      --namespace modzy \
      --shared-password "<password-value>" \
      --config-values ./replicated_config_<installation-identifier>.yaml \
      --port-forward=true
      • Deploying Admin Console
          • Creating namespace ✓
          • Waiting for datastore to be ready ✓
      • Waiting for Admin Console to be ready ✓
      
      • Press Ctrl+C to exit
      • Go to http://localhost:8800 to access the Admin Console
      

      (Note: The Admin Console Url and password-value for use in next steps. If not displayed, the password is also available in secrets manager /modzy/{installation-identifier}/app as config-password)

  • Log in to Admin Console and Upload license

    • In a browser, navigate to the Admin Console URL as indicated at the bottom of the above output
      Log in to the Admin Console with the password-value displayed in the output and upload the license file

  • Ensure the KOTS version indicated at the bottom of the page is >= 1.75.0

  • After successfully validating the license, a configuration page with various sections will be presented. Specify the indicated items in the sections listed:

    • General: Choose AWS as the Installation Profile
    • Security: Choose between SAML and Cognito Identity Provider
    • Ingress Settings: Certificate and Private Key
    • Database Settings: Choose between embedded and your own database
    • Storage Settings: Leave defaults to use AWS S3
    • SSO Settings: Identity Provider Metadata
    • SMTP Settings: From Address (Ex. [email protected])
  • On completion, click the Save button at the bottom of the page. This brings up the preflight checks. Wait for completion of the checks and click Continue

1121
  • A status page such as the example shown below should be displayed indicating successful initiation of deployment and configuration of containers in the cluster
1151
  • Configure DNS
    After the above configuration steps, you may end port-forwarding to the admin-console.

    • Obtain load balancer address
      While still in the kubernetes context for the cluster, execute the following to see an ingress listing for the Modzy environment
    kubectl get ingress -n modzy
    

    Note the load balancer indicated in the Address property of the resulting listed ingresses - use this value to create DNS entries for

    • Fully qualified domain name (FQDN) for the Modzy Environment
    • registry.[FQDN for the Modzy Environment]

Deprovisioning a Modzy Environment

S3

The Modzy installation creates three S3 buckets:

  • [installation-identifier]-model-assets-data
  • [installation-identifier]-job-data
  • [installation-identifier]-results-data

To successfully run terraform destroy to deprovision the Modzy Infrastructure, the buckets must be emptied.


Tip:
Using the AWS CLI, an S3 bucket can be emptied by executing:

aws s3 rm --recursive s3://[bucket name]

Alternatively, if there is a desire to preserve any of the buckets (and contents), you may remove the bucket from the Terraform state by executing the following

terraform state rm aws_s3_bucket.<bucket name>

RDS

The Modzy RDS Database is created with the Deletion Protection property enabled. This must be disabled in the AWS console in order for terraform to successfully remove the database if so desired.

You may also exclude the database from Terraform state management by executing the following

terraform state rm aws_db_instance.modzy

Deprovision Infrastructure

To remove the resources created by Terraform, execute the following

terraform destroy