Model Serving made Efficient in the Cloud.
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.
Mosec requires Python 3.7 or above. Install the latest PyPI package with:
```shell
pip install -U mosec ```
Import the libraries and set up a basic logger to better observe what happens.
```python import logging
from mosec import Server, Worker from mosec.errors import ValidationError
logger = logging.getLogger() logger.setLevel(logging.DEBUG) formatter = logging.Formatter( "%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s" ) sh = logging.StreamHandler() sh.setFormatter(formatter) logger.addHandler(sh) ```
Then, we build an API to calculate the exponential with base e for a given number. To achieve that, we simply inherit the Worker class and override the forward method. Note that the input req is by default a JSON-decoded object, e.g., a dictionary here (wishfully it receives data like {"x": 1}). We also enclose the input parsing part with a try...except... block to reject invalid input (e.g., no key named "x" or field "x" cannot be converted to float).
```python import math
class CalculateExp(Worker): def forward(self, req: dict) -> dict: try: x = float(req["x"]) except KeyError: raise ValidationError("cannot find key 'x'") except ValueError: raise ValidationError("cannot convert 'x' value to float") y = math.exp(x) # f(x) = e ^ x logger.debug(f"e ^ {x} = {y}") return {"y": y} ```
Finally, we append the worker to the server to construct a single-stage workflow, and we specify the number of processes we want it to run in parallel. Then we run the server.
```python if name == "main": server = Server() server.append_worker( CalculateExp, num=2 ) # we spawn two processes for parallel computing server.run()
```
After merging the snippets above into a file named server.py, we can first have a look at the command line arguments:
```shell
python server.py --help ```
Then let's start the server...
```shell
python server.py ```
and in another terminal, test it:
```console
curl -X POST http://127.0.0.1:8000/inference -d '{"x": 2}' { "y": 7.38905609893065 }
curl -X POST http://127.0.0.1:8000/inference -d '{"input": 2}' # wrong schema validation error: cannot find key 'x' ```
or check the metrics:
```shell
curl http://127.0.0.1:8000/metrics ```
For more debug logs, you can enable it by changing the Python & Rust log level:
python
logger.setLevel(logging.DEBUG)
```shell
RUST_LOG=debug python server.py ```
That's it! You have just hosted your exponential-computing model as a server! 😉
More ready-to-use examples can be found in the Example section. It includes:
| | Batcher | Pipeline | Parallel | I/O Format(1) | Framework(2) | Backend | Activity |
| ----------------------------------------------------------- | :-----: | :------: | :------: | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------- | ----------------------------------------------------------------------------- |
| TF Serving | ✅ | ✅ | ✅ | Limited(a) | Heavily TF | C++ | 
*As accessed on 08 Oct 2021. By no means is this comparison showing that other frameworks are inferior, but rather it is used to illustrate the trade-off. The information is not guaranteed to be absolutely accurate. Please let us know if you find anything that may be incorrect.
(1): Data format of the service's request and response. "Limited" in the sense that the framework has pre-defined requirements on the format. (2): Supported machine learning frameworks. "Heavily" means the serving framework is designed towards a specific ML framework. Thus it is hard, if not impossible, to adapt to others. "Multiple" means the serving framework provides adaptation to several existing ML frameworks. "Agnostic" means the serving framework does not necessarily care about the ML framework. Hence it supports all ML frameworks (in Python). (3): Flask is a representative of general purpose web frameworks to host ML models.
We welcome any kind of contribution. Please give us feedback by raising issues or discussing on Discord. You could also directly contribute your code and pull request!