Worker Service

Instance management process

IPC socket protocol

Metalcore implements a simple TLV protocol to send tasks and receive results to and from the worker service instances. The byte format on the wire looks like this:

type	status	length	payload
uint8	uint8	uint32	0 to 4,294,967,295 arbitrary bytes

Six bytes in total for the header fields type, status, and length, which are using unsigned integers in big endianness / network byte order, plus n bytes for the payload itself.

type

Currently, the protocol implements the following values for the type field in the header:

value	interpretation
0	task
5	shared data
10	result

The type field is used to indicate between metalcore and the service instance if we send task data, shared data, or a result. The full range of uint8 values (0-255) is available to extend the protocol in the future.

status

Currently, the protocol implements the following values for the status field in the header:

value	interpretation
0	all OK
>= 1	any form of error

The status field is evaluated by metalcore to see if a task's result (or a full batch) should be re-queued. A worker service might also implement a status check after transmission, but that's optional. The full range of uint8 values (0-255) is available to extend the protocol in the future.

length

This will be automatically set depending on the actual length of the payload. Make sure to implement it in this way for any other language as well to ensure interoperability.

payload

Your actual task and result payload. How you serialize and deserialize this is completely up to you and your worker service, and fully transparent to metalcore itself.

The actual payload can theoretically hold up to 4,294,967,295 bytes (4 GiB, max range of uint32 indicated by the length field) per transmission. Keep in mind higher level limits of the metalcore stack or lower limits of your language's socket implementation. E.g. RabbitMQ should ideally use < 128 MB per message, and these messages might be batches of tasks and results!