./modzy-core direct

$ modzy-core direct [flags] sends all inferences directly to the running model and returns the model response in a single request. Nothing is queued or persisted, and there is no observability in this mode.

🚧
./modzy-core direct only supports inferences API
Since direct mode does not store any inference results, it can only be used with the inferences API . Both HTTP and gRPC inferences for this route are supported.

Usage

Enter the following code into the terminal:

$ ./modzy-core direct

The output is similar to the following:

2022-11-17T04:00:04.363Z	INFO	logger/logger.go:108	acquiring lock
2022-11-17T04:00:04.363Z	INFO	modzy/registration.go:49	connecting to Modzy...
2022-11-17T04:00:04.366Z	INFO	modzy/registration.go:271	Device Group is valid -- Verifying Certificates
2022-11-17T04:00:04.366Z	INFO	modzy/registration.go:300	valid device registration files found...bypassing registration and starting Core
2022-11-17T04:00:04.404Z	INFO	runner/direct.go:81	starting model: k8cjaqdcgz 1.0.0
2022-11-17T04:00:04.404Z	INFO	runtime/docker.go:89	Checking if container is running...
2022-11-17T04:00:04.410Z	INFO	runtime/docker.go:123	creating container...
2022-11-17T04:00:05.440Z	INFO	runtime/docker.go:182	container is ready: running
2022-11-17T04:00:16.257Z	INFO	apiserver/server.go:161	Modzy Core server is starting...
2022-11-17T04:00:16.258Z	INFO	apiserver/server.go:228	Server is starting...
2022-11-17T04:00:16.258Z	INFO	apiserver/server.go:270	Server is listening at :55000

Direct Flags

Flag	Type	Description
`--cert`	`string`	TLS cert to use
`-h, --help`		Help for direct mode
`--http.port`	`string`	Http port exposed for core server (default "55000")
`--key`	`string`	TLS key to use
`--model.runtime`	`string`	The model container runtime. Only recommended to use when container requires GPU to run (i.e., `nvidia`). (default "docker")
`--model.timeout`	`float`	Length of time (in seconds) to wait while attempting to connect to model container. (default 10)
`--model.memory`	`int`	Model container Memory usage limit (in MB). Applies to all models running on Core
`--modzy.token`	`string`	Registration token
`--modzy.url`	`string`	Url of Modzy Platform
`--resume`		Resume running after registering.