Modzy Edge provides a way for machine learning models to run "at the edge" on hardware such as remove servers and edge devices.

How does Modzy Edge Work?

When you connect your edge device to Modzy, the following things happen:

  1. It downloads a copy of Modzy Core to your edge device. This small executable provides an API interface for any models you deploy to the edge. You can learn more about the Modzy Edge API here.
  2. It registers your device with Modzy and associates it with a device group
  3. It automatically downloads any models you've added to your device group and starts running them on your edge device

Edge Hardware Reference

To use Modzy Edge on an edge device, that device must meet some minimum hardware requirements. These requirements are typically easy to meet in the cloud, or on-prem, but Modzy Edge also works with many small form-factor devices.

Minimum Hardware Requirements

Operating SystemLinux
CPURequires an ARM 32-bit, ARM 64-bit, or x86 (AMD or Intel) 64-bit processor, with at least 2 cores
RAM2+ GiB of RAM is recommended, but more may be required for running larger models or multiple models at once
GPUNot required, but Modzy Edge supports NVIDIA GPUs
TPUNot required and not yet supported (though models will still run on a CPU)
Device Storage16+ GiB of storage is recommended, but more may be required for storing model results
DependenciesDocker v20.10.x or newer

Validated Small Form-factor Devices

Modzy Edge has been tested on the following small form-factor edge devices

DeviceCPURAMGPUStorage
NVIDIA® Jetson Nano™ Developer KitQuad-core ARM A57 64-bit SoC @ 1.43 GHz4 GB LPDDR4128-core Maxwell16 GB MicroSD
Raspberry Pi 3 Model B+Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz1 GB LPDDR2None16 GB MicroSD
Up BoardIntel® ATOM™ x5-Z8350 Processor 64-bit SoC @ 1.92GHz2 GB DDR3LNone16 GB eMMC

Hardware Resource Guidance

When using Modzy Edge, an important consideration to make in addition to the edge device meeting the list of minimum requirements is the model's resource footprint. In this context, resource footprint refers to the amount of RAM, CPU cores, and perhaps GPU(s) your model needs to run inferences. This consideration is especially critical when using Modzy Edge on small form-factor edge devices with limited resources (i.e., your device's resources must be able to handle your model's footprint).

Since determining a model's resource footprint is difficult, highly variable, and depends on many factors, we put together a table that provides examples of models and their corresponding footprint details as a reference.

Programming LanguageML FrameworkModel TypeBase Container ImageOSModel Weight(s) SizeContainer SizeMemory Allocated
PythonPyTorchText Classificationbalenalib/aarch64-ubuntu-python:3-latest-build-20220513Debian Linux20 MB1.86 GB175 MB

Performance Benchmarks

Modzy Core v1.5

ModelModel TypeThroughput
EchoN/A65 inferences/second
TinyBERTText Classification20 inferences/second

Modzy Core v1.6

ModelModel TypeThroughput
EchoN/A1600 inferences/second
TinyBERTText Classification300 inferences/second