Edge

Modzy Edge provides a way for machine learning models to run "at the edge" on hardware such as remove servers and edge devices.

How does Modzy Edge Work?

When you connect your edge device to Modzy, the following things happen:

It downloads a copy of Modzy Core to your edge device. This small executable provides an API interface for any models you deploy to the edge. You can learn more about the Modzy Edge API here.
It registers your device with Modzy and associates it with a device group
It automatically downloads any models you've added to your device group and starts running them on your edge device

Edge Hardware Reference

To use Modzy Edge on an edge device, that device must meet some minimum hardware requirements. These requirements are typically easy to meet in the cloud, or on-prem, but Modzy Edge also works with many small form-factor devices.

Minimum Hardware Requirements


Operating System	Linux
CPU	Requires an ARM 32-bit, ARM 64-bit, or x86 (AMD or Intel) 64-bit processor, with at least 2 cores
RAM	2+ GiB of RAM is recommended, but more may be required for running larger models or multiple models at once
GPU	Not required, but Modzy Edge supports NVIDIA GPUs
TPU	Not required and not yet supported (though models will still run on a CPU)
Device Storage	16+ GiB of storage is recommended, but more may be required for storing model results
Dependencies	Docker v20.10.x or newer

Validated Small Form-factor Devices

Modzy Edge has been tested on the following small form-factor edge devices

	Device	CPU	RAM	GPU	Storage
✅	NVIDIA® Jetson Nano™ Developer Kit	Quad-core ARM A57 64-bit SoC @ 1.43 GHz	4 GB LPDDR4	128-core Maxwell	16 GB MicroSD
✅	Raspberry Pi 3 Model B+	Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz	1 GB LPDDR2	None	16 GB MicroSD
✅	Up Board	Intel® ATOM™ x5-Z8350 Processor 64-bit SoC @ 1.92GHz	2 GB DDR3L	None	16 GB eMMC

Hardware Resource Guidance

When using Modzy Edge, an important consideration to make in addition to the edge device meeting the list of minimum requirements is the model's resource footprint. In this context, resource footprint refers to the amount of RAM, CPU cores, and perhaps GPU(s) your model needs to run inferences. This consideration is especially critical when using Modzy Edge on small form-factor edge devices with limited resources (i.e., your device's resources must be able to handle your model's footprint).

Since determining a model's resource footprint is difficult, highly variable, and depends on many factors, we put together a table that provides examples of models and their corresponding footprint details as a reference.

Programming Language	ML Framework	Model Type	Base Container Image	OS	Model Weight(s) Size	Container Size	Memory Allocated
Python	PyTorch	Text Classification	balenalib/aarch64-ubuntu-python:3-latest-build-20220513	Debian Linux	20 MB	1.86 GB	175 MB

Performance Benchmarks

Modzy Core v1.5

Model	Model Type	Throughput
Echo	N/A	65 inferences/second
TinyBERT	Text Classification	20 inferences/second

Modzy Core v1.6

Model	Model Type	Throughput
Echo	N/A	1600 inferences/second
TinyBERT	Text Classification	300 inferences/second