Skip to content

Commit

Permalink
Article updated: Spryker Middleware
Browse files Browse the repository at this point in the history
  • Loading branch information
Vadym Sachenko committed Feb 26, 2021
1 parent fa2bbc0 commit 1fb1cd1
Showing 1 changed file with 40 additions and 40 deletions.
80 changes: 40 additions & 40 deletions Code releases 05.20/cc6aaccf-32c7-48b2-b3d4-100891d75db5.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
## Overview
Spryker Middleware is a constructor that allows you to set up a linear data processing flow, also referred to as pipeline, for import/export of data from some system to shop, or from shop to some system. For example, it can be used for importing products to a shop, or exporting orders from a shop.
Spryker Middleware is a constructor that allows you to set up a linear data processing flow, also referred to as a pipeline, for import/export of data from some system to shop or from shop to some system. For example, it can be used for importing products to a shop, or exporting orders from a shop.

### Pipeline Structure
### Pipeline structure
The Middleware applies the pipeline pattern allowing to connect different stages of data processing together and inverting dependencies between them. The imported/exported items are processed one by one and go through a set of specific steps called “stages”.

The pipeline contains 5 standard stages: reader, validator, mapper, translator, and writer. However, you can use them or define any number of stages.

First of all, a source item is **read**. Then, it is **validated** to make sure that all attributes etc. are correct and all the necessary data is available. Having passed the validation, the item is **mapped**, i.e. keys of the source system are mapped onto the target system. This being done, the items go through a **translator** which processes the values and translates them into a respective format (for example, the price value is a decimal value, but should be init - it’s translator’s responsibility to change it to the required value). After that, the item is **written** to the target system (to the database, in case of import, to a file, if it’s export etc.).
First of all, a source item is **read**. Then, it is **validated** to make sure that all attributes, etc., are correct and all the necessary data is available. Having passed the validation, the item is **mapped**, i.e., keys of the source system are mapped onto the target system. This being done, the items go through a **translator** which processes the values and translates them into a respective format (for example, the price value is a decimal value, but should be initit’s translator’s responsibility to change it to the required value). After that, the item is **written** to the target system (to the database, in case of import, to a file, if it’s export, etc.).
![Pipeline stages](https://spryker.s3.eu-central-1.amazonaws.com/docs/Developer+Guide/Spryker+Middleware/stages.png){height="" width=""}

Each stage can be abstracted as having:

1. Inputitem is received from the previous stage, exception might be a reader, which receives nothing.
2. Outputitem is provided for the next stage, exception might be a writer, which persists data and sends back nothing.
3. Configurationconfiguration of the stage, e.g. validation rules for the validator.
4. Loggingused by any stage to leave some artefacts of processing data.
1. Inputitem is received from the previous stage; exception might be a reader, which receives nothing.
2. Outputitem is provided for the next stage; exception might be a writer, which persists data and sends back nothing.
3. Configuration—the configuration of the stage; e.g., validation rules for the validator.
4. Loggingused by any stage to leave some artefacts of processing data.
![Input output](https://spryker.s3.eu-central-1.amazonaws.com/docs/Developer+Guide/Spryker+Middleware/input-output.png){height="" width=""}

The incoming data is taken from the stream - the Middleware does not care about the source of the data, whether it comes from a file, from an API etc. The middleware provides its own interface, so that the source of data does not really matter.
The incoming data is taken from the streamthe Middleware does not care about the source of the data, whether it comes from a file, from an API etc. The Middleware provides its own interface so that the source of data does not really matter.

### How the Middleware Works
### How the Middleware works

The Middleware provides a console interface to allow job triggering and Jenkins integration. It is evoked by running the `middleware:process: run` command. The main parameter of the command is *-p* (process name) which defines the process to be started.
The Middleware provides a console interface to allow job triggering and Jenkins integration. It is evoked by running the `middleware:process: run` command. The main parameter of the command is *-p* (process name), which defines the process to be started.

The default implementation of the middleware constructor includes the interface, reading/writing from/to JSON, .csv, .xml formats, business logic of mapping, translation and validation (you can add your own translators and validators).
The default implementation of the middleware constructor includes the interface, reading/writing from/to JSON, .csv, .xml formats, the business logic of mapping, translation, and validation (you can add your own translators and validators).

There are two main plugin interfaces, which should be implemented to configure Middleware Process: `ConfigurationProfilePluginInterface` and `ProccessConfigurationPluginInterface`.
There are two main plugin interfaces, which should be implemented to configure the Middleware process: `ConfigurationProfilePluginInterface` and `ProccessConfigurationPluginInterface`.

The `ConfigurationProfilePluginInterface` registers the processes (like import/export) and the list of custom translators/validators (if any) implemented at the project level. The interface can be implemented in any module under `\Spryker\Zed\[MODULE]\Communication\Plugin\Configuration`.

Expand Down Expand Up @@ -64,21 +64,21 @@ class AkeneoPimConfigurationProfilePlugin extends AbstractPlugin implements Conf

Each process is a separate plugin that consists of the following methods (ProcessConfigurationPluginInterface):

**getProcessName** - returns the process name which is used to find necessary process with the parameter (transferred with -p option).
**getProcessName**returns the process name which is used to find the necessary process with the parameter (transferred with -p option).

**getInputStreamPlugin** - configures the source where the data is read from.
**getInputStreamPlugin**configures the source where the data is read from.

**getOutputStreamPlugin** - configures the target where the data is written.
**getOutputStreamPlugin**configures the target where the data is written.

**getIteratorPlugin** - either does nothing and releases the input stream for processing as is, or alters the data for further processing. For example, if the input stream is just a file, the iterator does nothing and lets the data be processed further. If the input stream is, for example, a file catalog, `getInputStreamPlugin` returns the file name, the iterator goes through all the files, and if, say each file is in the JSON format, the iterator returns each JSON file of the catalog for processing to pipeline.
**getIteratorPlugin**either does nothing and releases the input stream for processing as is, or alters the data for further processing. For example, if the input stream is just a file, the iterator does nothing and lets the data be processed further. If the input stream is, for example, a file catalog, `getInputStreamPlugin` returns the file name, the iterator goes through all the files, and if, say each file is in the JSON format, the iterator returns each JSON file of the catalog for processing to the pipeline.

You can use one of two iterators that are provided out of the box (NullIterator, JsonDirectoryIterator) or implement your own iterator.

**getStagePlugins** - contains a list of all stages the items go through (reader, validator, mapper, translator, writer) and makes sure each item passes each stage one by one.
**getStagePlugins**contains a list of all stages the items go through (reader, validator, mapper, translator, writer) and makes sure each item passes each stage one by one.

**getLoggerPlugin** - defines the way logging happens. The default Middleware logger logs to the PHP standard error stream (php://stderr) (this can be changed as needed). Detalization of the logging is fully customizable, which means you can configure it as you wish.
**getLoggerPlugin**defines the way logging happens. The default Middleware logger logs to the PHP standard error stream (php://stderr) (this can be changed as needed). Detalization of the logging is fully customizable, which means you can configure it as you wish.

**getPreProcessorHookPlugins** and **getPostProcessorHookPlugins** - define what should be done prior to or after a process. For example, it might be necessary to download a file with the categories prior to the categories import: this would be specified in `getPreProcessHookPlugins`.
**getPreProcessorHookPlugins** and **getPostProcessorHookPlugins**define what should be done prior to or after a process. For example, it might be necessary to download a file with the categories prior to the categories import: this would be specified in `getPreProcessHookPlugins`.

```php
class CategoryImportConfigurationPlugin extends AbstractPlugin implements ProcessConfigurationPluginInterface
Expand Down Expand Up @@ -158,18 +158,18 @@ class CategoryImportConfigurationPlugin extends AbstractPlugin implements Proces
}
```

### Code Organization
### Code organization

The Middleware is a set of modules in the Middleware namespace allowing to group common functionalities together. The middleware cannot provide readers and writers for all systems, these should be implemented in scope of respective modules and namespaces.
The Middleware is a set of modules in the Middleware namespace allowing to group common functionalities together. The middleware cannot provide readers and writers for all systems. These should be implemented in the scope of respective modules and namespaces.

Here is an example of code organization for a project:
![Code organization](https://spryker.s3.eu-central-1.amazonaws.com/docs/Developer+Guide/Spryker+Middleware/code-organization.png){height="" width=""}

### Middleware Integration
### Middleware integration

The core of the Spryker Middleware is implemented in the Process module. This module collects all process plugins and creates processes out of them.

To install Process module, run this command in console:
To install the Process module, run this command in the console:

```
composer require spryker-middleware/process
Expand Down Expand Up @@ -224,7 +224,7 @@ See [this example](https://github.com/spryker-eco/akeneo-pim-middleware-connecto

### Middleware Reports

You can view the results of the Spryker Middleware processes in the Middleware *Reports* section under the *Maintenance* menu of the Administration interface. This *Middleware Reports* section provides an overview of all the processes run with Middleware, overview of the process results (start time, duration, item count, and status of each process), as well as the detailed information on each process. The detailed information includes:
You can view the results of the Spryker Middleware processes in the Middleware *Reports* section under the *Maintenance* menu of the Administration interface. This *Middleware Reports* section provides an overview of all the processes run with Middleware, an overview of the process results (start time, duration, item count, and status of each process), as well as the detailed information on each process. The detailed information includes:

**Process details:**

Expand Down Expand Up @@ -256,8 +256,8 @@ You can view the results of the Spryker Middleware processes in the Middleware *
* total execution time
* average item execution time

### Reports Integration
To install Report module, run this command in console:
### Reports integration
To install the Report module, run this command in the console:

```
composer require spryker-middleware/report
Expand All @@ -280,17 +280,17 @@ public function getPostProcessorHookPlugins(): array

After that, you will be able to see the result of your process runs in the Admin UI (Maintenance\ Middleware Reports).

### OmsMiddlewareConnector Module
### OmsMiddlewareConnector module

The `OmsMiddlewareConnector` module provides `TriggerOrderExportProcessCommand` which enables triggering of a Middleware process from OMS. Also, this module provides `OrderReadStreamPlugin` that provides input stream for reading orders and pass them to next stages of Middleware process.
The `OmsMiddlewareConnector` module provides `TriggerOrderExportProcessCommand`, which enables triggering of a Middleware process from OMS. Also, this module provides `OrderReadStreamPlugin` that provides an input stream for reading orders and pass them to the next stages of the Middleware process.

To install `OmsMiddlewareConnector` module, run this command in console:

```bash
composer require spryker-middleware/oms-middleware-connector
```

Please refer to config/Shared/config.dist.php for example of module configuration. To set up the order export process which should be triggered from the OMS command, add configuration of its name to your project’s config:
Refer to config/Shared/config.dist.php for example of module configuration. To set up the order export process which should be triggered from the OMS command, add configuration of its name to your project’s config:

```php
$config[OmsMiddlewareConnectorConstants::ORDER_EXPORT_PROCESS_NAME] = OrderExportProcessConfigurationPlugin::PROCESS_NAME;
Expand Down Expand Up @@ -334,11 +334,11 @@ By default, Middleware supports two strategies:

There are 5 different ways to set mapper rules:

* ArrayMapRule - this rule allows using the given payload as an array with a recursive call;
* DynamicMapRule - this rule allows using the value from the payload as a key;
* ClosureMapRule - this rule allows using a custom function for the payload array;
* DynamicArrayMapRule - this rule allows using the value from the payload as a key and works with the payload as with an array with recursive calls;
* KeyMapeRule - the simplest rule for the mapper that gets the value via the key from the payload. You can use . symbol as a key separator for getting value from the payload.
* ArrayMapRulethis rule allows using the given payload as an array with a recursive call;
* DynamicMapRulethis rule allows using the value from the payload as a key;
* ClosureMapRulethis rule allows using a custom function for the payload array;
* DynamicArrayMapRulethis rule allows using the value from the payload as a key and works with the payload as with an array with recursive calls;
* KeyMapeRulethe simplest rule for the mapper that gets the value via the key from the payload. You can use the `.` symbol as a key separator for getting value from the payload.

You can check the examples of each rule in the following snippet. It's a final mapper example with examples of payload and their result.

Expand Down Expand Up @@ -498,7 +498,7 @@ Use the following format to define validation rules:
}
```
#### Default Validators
#### Default validators
There are many predefined validators that can be used in `ValidationRuleSet`.
| Validator Name | Description | Options |
Expand Down Expand Up @@ -557,7 +557,7 @@ A translator is a way to update the values from the mapped payload using strict

At first, you should implement `SprykerMiddleware\Zed\Process\Business\Translator\Dictionary\AbstractDictionary`. The abstract method is necessary for the `getDictionary()` implementation. This method returns an array with translation rules for mapped payload.

You can apply translator function for value with the type array. You can use the `*` symbol to apply translator function to each value in the array like `mapped_key.*` or use the `mapped_key.*`.subkey syntax to apply translator function to the certain key in the array.
You can apply the translator function for value with the type array. You can use the `*` symbol to apply the translator function to each value in the array like `mapped_key.*` or use the `mapped_key.*`.subkey syntax to apply the translator function to a certain key in the array.

Use the following format to define translation rules:

Expand Down Expand Up @@ -818,7 +818,7 @@ First of all, you need to create a business model to import data to the database
### Prepare Publisher and datasetStepBroker
As an example, we can create `DataImporter` for categories.
Importer business model expects 3 parameters in the constructor, so let's create it.
The Importer business model expects 3 parameters in the constructor, so let's create it.
Firstly, you need to update the business factory with the following methods:
Expand Down Expand Up @@ -872,7 +872,7 @@ Firstly, you need to update the business factory with the following methods:
...
```
So, now we can create a facade method which uses Importer.
So, now we can create a facade method that uses Importer.
```php
<?php
Expand Down Expand Up @@ -937,7 +937,7 @@ Then, you need to add `CategoryDataImporterPlugin` to communication dependencies
### Prepare WriteStream
To save the categories into the database you need to create your own `WriteStream`. `SprykerMiddleware\Shared\Process\Stream\WriteStreamInterface` should be implemented.
To save the categories into the database, you need to create your own `WriteStream`. `SprykerMiddleware\Shared\Process\Stream\WriteStreamInterface` should be implemented.
```php
class DataImportWriteStream implements WriteStreamInterface
Expand Down Expand Up @@ -1034,7 +1034,7 @@ As the parameter for `DataImportWriteStream`, we should use `CategoryDataImporte
}
```
### Update Process Plugins
### Update process plugins
Finally, you are ready to update process plugins:
Expand Down

0 comments on commit 1fb1cd1

Please sign in to comment.