Edge
Modzy Edge provides a way for machine learning models to run "at the edge" on hardware such as remove servers and edge devices.
How does Modzy Edge Work?
When you connect your edge device to Modzy, the following things happen:
- It downloads a copy of
Modzy Core
to your edge device. This small executable provides an API interface for any models you deploy to the edge. You can learn more about the Modzy Edge API here. - It registers your device with Modzy and associates it with a device group
- It automatically downloads any models you've added to your device group and starts running them on your edge device
Edge Hardware Reference
To use Modzy Edge on an edge device, that device must meet some minimum hardware requirements. These requirements are typically easy to meet in the cloud, or on-prem, but Modzy Edge also works with many small form-factor devices.
Minimum Hardware Requirements
Operating System | Linux |
CPU | Requires an ARM 32-bit, ARM 64-bit, or x86 (AMD or Intel) 64-bit processor, with at least 2 cores |
RAM | 2+ GiB of RAM is recommended, but more may be required for running larger models or multiple models at once |
GPU | Not required, but Modzy Edge supports NVIDIA GPUs |
TPU | Not required and not yet supported (though models will still run on a CPU) |
Device Storage | 16+ GiB of storage is recommended, but more may be required for storing model results |
Dependencies | Docker v20.10.x or newer |
Validated Small Form-factor Devices
Modzy Edge has been tested on the following small form-factor edge devices
Device | CPU | RAM | GPU | Storage | |
---|---|---|---|---|---|
✅ | NVIDIA® Jetson Nano™ Developer Kit | Quad-core ARM A57 64-bit SoC @ 1.43 GHz | 4 GB LPDDR4 | 128-core Maxwell | 16 GB MicroSD |
✅ | Raspberry Pi 3 Model B+ | Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz | 1 GB LPDDR2 | None | 16 GB MicroSD |
✅ | Up Board | Intel® ATOM™ x5-Z8350 Processor 64-bit SoC @ 1.92GHz | 2 GB DDR3L | None | 16 GB eMMC |
Hardware Resource Guidance
When using Modzy Edge, an important consideration to make in addition to the edge device meeting the list of minimum requirements is the model's resource footprint. In this context, resource footprint refers to the amount of RAM, CPU cores, and perhaps GPU(s) your model needs to run inferences. This consideration is especially critical when using Modzy Edge on small form-factor edge devices with limited resources (i.e., your device's resources must be able to handle your model's footprint).
Since determining a model's resource footprint is difficult, highly variable, and depends on many factors, we put together a table that provides examples of models and their corresponding footprint details as a reference.
Programming Language | ML Framework | Model Type | Base Container Image | OS | Model Weight(s) Size | Container Size | Memory Allocated |
---|---|---|---|---|---|---|---|
Python | PyTorch | Text Classification | balenalib/aarch64-ubuntu-python:3-latest-build-20220513 | Debian Linux | 20 MB | 1.86 GB | 175 MB |
Performance Benchmarks
Modzy Core v1.5
Model | Model Type | Throughput |
---|---|---|
Echo | N/A | 65 inferences/second |
TinyBERT | Text Classification | 20 inferences/second |
Modzy Core v1.6
Model | Model Type | Throughput |
---|---|---|
Echo | N/A | 1600 inferences/second |
TinyBERT | Text Classification | 300 inferences/second |
Updated 10 months ago