jina.executors.compound

class jina.executors.compound.CompoundExecutor(routes=None, resolve_all=True, *args, **kwargs)[source]

Bases: jina.executors.BaseExecutor

A CompoundExecutor is a set of multiple executors. The most common usage is chaining a pipeline of executors, where the input of the current is the output of the former.

A common use case of CompoundExecutor is to glue multiple BaseExecutor together, instead of breaking them into different Pods.

Parameters
  • routes (Optional[Dict[str, Dict]]) –

    a map of function routes. The key is the function name, the value is a tuple of two pieces, where the first element is the name of the referred component (metas.name) and the second element is the name of the referred function.

    See also

    add_route()

  • resolve_all (bool) – universally add *_all() to all functions that have the identical name

Example 1: a compound Chunk Indexer that does vector indexing and key-value index

!CompoundExecutor
components:
  - !NumpyIndexer
    with:
      index_filename: vec.gz
    metas:
      name: vecidx_exec  # a customized name
      workspace: ${{TEST_WORKDIR}}
  - !BinaryPbIndexer
    with:
      index_filename: chunk.gz
    metas:
      name: chunkidx_exec
      workspace: ${{TEST_WORKDIR}}
metas:
  name: chunk_compound_indexer
  workspace: ${{TEST_WORKDIR}}
requests:
  on:
    SearchRequest:
      - !VectorSearchDriver
        with:
          executor: vecidx_exec
    IndexRequest:
      - !VectorIndexDriver
        with:
          executor: vecidx_exec
    ControlRequest:
      - !ControlReqDriver {}

**Example 2: a compound crafter that first craft the doc and then segment **

!CompoundExecutor
components:
  - !GifNameRawSplit
    metas:
      name: name_split  # a customized name
      workspace: ${{TEST_WORKDIR}}
  - !GifPreprocessor
    with:
      every_k_frame: 2
      from_buffer: true
    metas:
      name: gif2chunk_preprocessor  # a customized name
metas:
  name: compound_crafter
  workspace: ${{TEST_WORKDIR}}
  py_modules: gif2chunk.py
requests:
  on:
    IndexRequest:
      - !DocCraftDriver
        with:
          executor: name_split
      - !SegmentDriver
        with:
          executor: gif2chunk_preprocessor
    ControlRequest:
      - !ControlReqDriver {}

Create a new CompoundExecutor object

**Example 3: **

We have two dummy executors as follows:

class dummyA(BaseExecutor):
    def say(self):
        return 'a'

    def sayA(self):
        print('A: im A')


class dummyB(BaseExecutor):
    def say(self):
        return 'b'

    def sayB(self):
        print('B: im B')

and we create a CompoundExecutor consisting of these two via

da, db = dummyA(), dummyB()
ce = CompoundExecutor()
ce.components = lambda: [da, db]

Now the new executor ce have two new methods, i.e ce.sayA() and ce.sayB(). They point to the original dummyA.sayA() and dummyB.sayB() respectively. One can say ce has inherited these two methods.

The interesting part is say(), as this function name is shared between dummyA and dummyB. It requires some resolution. When resolve_all=True, then a new function say_all() is add to ce. ce.say_all works as if you call dummyA.sayA() and dummyB.sayB() in a row. This makes sense in some cases such as training, saving. In other cases, it may require a more sophisticated resolution, where one can use add_route() to achieve that. For example,

ce.add_route('say', db.name, 'say')
assert b.say() == 'b'

Such resolution is what we call routes here, and it can be specified in advance with the arguments routes in __init__(), or using YAML.

!CompoundExecutor
components: ...
with:
  resolve_all: true
  routes:
    say:
    - dummyB-e3acc910
    - say

Warning

When setting inner executors in components the workspace configuration will not be used and will be overridden by a workspace extracted considering the name of the CompoundExecutor, the name of each internal Component and the pea_id

One can access the component of a CompoundExecutor via index, e.g.

c = BaseExecutor.load_config('compound-example.yaml')
assertTrue(c[0] == c['dummyA-1ef90ea8'])
c[0].add(obj)

Note

Component workspace and pea_id are overridden by their CompoundExecutor counterparts.

Warning

When sub-component is external, py_modules must be given at root level metas not at the sub-level.

property is_updated

Return True if any components is updated.

Return type

bool

Returns

only true if all components are updated or if the compound is updated

save(filename=None)[source]

Serialize this compound executor along with all components in it to binary files. It uses pickle for dumping.

Parameters

filename (Optional[str]) – file path of the serialized file, if not given then save_abspath is used

property components

Return all component executors as a list. The list follows the order as defined in the YAML config or the pre-given order when calling the setter.

Return type

List[~AnyExecutor]

Returns

components

static get_component_workspace_from_compound_workspace(compound_workspace, compound_name, pea_id)[source]

Get the name of workspace.

Parameters
  • compound_workspace (str) – Workspace of the compound executor.

  • compound_name (str) – Name of the compound executor.

  • pea_id (int) – Id of the pea.

Return type

str

Returns

The name of workspace.

add_route(fn_name, comp_name, comp_fn_name, is_stored=False)[source]

Create a new function for this executor which refers to the component’s function

This will create a new function fn_name() which actually refers to components[comp_name].comp_fn_name. It is useful when two components have a function with duplicated name and one wants to resolve this duplication.

Parameters
  • fn_name (str) – the name of the new function

  • comp_name (str) – the name of the referred component, defined in metas.name

  • comp_fn_name (str) – the name of the referred function of comp_name

  • is_stored (bool) – if True then this change will be stored in the config and affects future save() and save_config()

Return type

None

close()[source]

Close all components and release the resources

Return type

None