Model Serving made Efficient in the Cloud.

Introduction

Mosec is a high-performance and flexible model serving framework for building ML model-enabled backends and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.

• Highly performant: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
• Ease of use: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
• Dynamic batching: aggregate requests from different users for batched inference and distribute results back
• Pipelined stages: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads

Installation

Mosec requires Python 3.6 or above. Install the latest PyPI package with:

pip install -U mosec

Usage

Write the server

Import the libraries and setup a basic logger to better observe what happens: ```python import logging

from pydantic import BaseModel # we need this to define our input/output schemas

from mosec import Server, Worker

logger = logging.getLogger() logger.setLevel(logging.DEBUG) formatter = logging.Formatter( "%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s" ) sh = logging.StreamHandler() sh.setFormatter(formatter) logger.addHandler(sh) ```

Define our service schemas for both input and output. These schemas will help us for data validation: ```python class Request(BaseModel): x: float

class Response(BaseModel): y: float ```

Now, we are going to build an API to calculate the exponential with base e for a given number. To achieve that, we simply inherit the Worker class and override the forward method: ```python import math

class CalculateExp(Worker): def forward(self, req: Request): y = math.exp(req.x) # f(x) = e ^ x logger.debug(f"e ^ {req.x} = {y}") return Response(y=y) ```

Finally, we run the server when the file is executed: ```python if name == "main": server = Server(Request, Response) server.append_worker( CalculateExp, num=2 ) # we spawn two processes for our calculator server.run()

```

Run the server

After merging the snippets above into a file named server.py, we can first have a look at the supported arguments:

python server.py --help

Then let's start the server...

python server.py

and test it:

curl -X POST http://127.0.0.1:8000/inference -d '{"x": 2}'

That's it! You have just hosted your exponential-computing model as a server! 😉

Example

More ready-to-use examples can be found in the Example section.

Contributing

We welcome any kind of contributions. Please give us feedback by raising issues or directly contribute your code and pull request!