[DSIP-38] Add plugins directory in binary package

[ruanwenjun]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

In the previous binary package, the plugins are put at server/libs.

- dolphinscheduler-bin
   - api-server
     - bin
     - conf
     - libs 
   - master-server
     - bin
     - conf
     - libs 
   - worker-server
     - bin
     - conf
     - libs
   - alert-server
     - bin
     - conf
     - libs

This will bring below problems:

• Multiple copies of the plugin exist within different services, cause the binary package is very big.
• Multiple copies of the plugin exist within different services, cause if we want to replace a plugin we need to replace it at different service.
• It's difficult to support different version plugins, e.g. hive2, hive3

Design Detail

Firstly, we need to add a plugin directory, the binary package will be

- dolphinscheduler-bin
  - api-server
  - master-server
  - alert-server
  - worker-server
  - plugins
    - task-plugin
    - alert-plugin
    - storage-plugin
    - registry-plugin
    - xx

Then we need to shade the plugins and put them in target plugins.

The server should only rely on plugin-api, cannot rely on plugins, and use child first classloader to load plugins, each plugin will have a classloader.

Finally, we need to optimize the dependency tree.

Compatibility, Deprecation, and Migration Plan

Compatibility with the previous version

Test Plan

Test by manual/e2e/UT

Code of Conduct

SubTasks

• [Improvement] Shade plugin jar #16096
• [Improvement] Move plugin to plugin directory #16097
• [Improvement] Optimize dependency tree #16098
• [Improvement] Support upload/download plugins from maven repo #16645
• [Improvement] Load plugins using a separate class loader #16646

• I agree to follow this project's Code of Conduct