Problem with time-consuming operations on OM protocol side

[vtutrinov]

Preamble

There was quite a large load for the OM Leader through writing keys by S3G MultipartUpload operations: 14 100Gb files were written to the cluster from 14 clients/hosts (aws s3 cp)

Performance problems

During the writing operations, OM Leader's performance was awkward:

• The AllocateBlock operation could take up to ~30 sec.
  

• Almost 99% of the time was consumed by OM protocol messages handling
  
  
  
  

Does anybody know how to try to work the issue around?
I would suggest the following steps:

• try to reduce message size (e.g. cut out optional fields)
• muti-raft? At the moment only one OM instance handles the transactions (the other two instances do nothing)

[ivandika3]

There are some existing tickets to optimize the space for Multipart Upload including things like reducing the message size:

• https://issues.apache.org/jira/browse/HDDS-8238
• https://issues.apache.org/jira/browse/HDDS-10611

As I understand it, the main issue is that currently multipartInfoTable stores OmMultipartKeyInfo which contains all the previously committed parts for a particular upload ID. This means that as more parts are committed, the serde cost would increase considerably as we need to load and write the existing parts. As suggested by HDDS-8238, we can change the structure of the multipart info table (we probably need to introduce a new table) to "flatten" the structure such that each DB entry only contains a single part (e.g. maybe with the DB key "{keyName}/{uploadId}/{partNumber}" so that we can reduce the serde cost. However, this requires some significant change from OM and possibly client side.

As you suggested, removing unnecessary fields might also help to reduce the serde cost. As seen from the profile OzoneAclInfo might be removed since it's already in the OpenKeyTable. Maybe we can just set the PartKeyInfo ACL to empty list before committing it.

As for the multi-raft solution, this probably requires more engineering effort since this means sharding the OMs which requires more deep design decisions. Even if for example we shard by bucket, if there are many concurrent multipart uploads in the bucket, the issue would happen again.

@vtutrinov As for now, do we know how large is the multipart uploads part size? Smaller part size might cause increase in the number of MPU part generated which also adds the serde cost, so maybe you can increase the size of the parts for large MPU key.

For the long AllocateBlock calls, I'm not really sure since part upload will specify the size and will use CreateKey instead. Unless your
part size is very large and exceed preallocated block max ozone.key.preallocation.max.blocks, which I doubt will happen. Maybe there are some performance issues for SCM block allocation process or some datanodes issue that require additional blocks from AllocateBlocks.

Feel free to pick the tickets or file your own if needed.

[errose28]

+1. This is similar to the previous discussion here. The MPU code could definitely use some streamlining to better handle smaller part sizes, which are the default with most S3 clients.

[vtutrinov]

@ivandika3 @errose28 As OzoneAclInfo and DatanodeDetailProto serialization consumes most of the time, and the pipeline field of the OmKeyLocationInfo is an optional one (the DatanodeDetailsProto message is a part of the pipeline), we've tried to set these field to null:

      //Set the data size and location info list
      omKeyInfo.setDataSize(keyArgs.getDataSize());
      omKeyInfo.setAcls(null);
      omKeyInfo.updateLocationInfoList(keyArgs.getKeyLocationsList().stream()
          .map(keyLocation -> {
            OmKeyLocationInfo omKeyLocationInfo = OmKeyLocationInfo.getFromProtobuf(keyLocation);
            omKeyLocationInfo.setPipeline(null); // attempt to reduce response size
            return omKeyLocationInfo;
          }).collect(Collectors.toList()), true);

and the speed of the file uploading (14 clients from 14 hosts tried to upload 100G files) had sped up to 10x.

Question:

Are there any cons or ineligible effects of such changes? We know that the ACLs are in another RDB table, what about the pipeline? (FYI: tests of writing, reading, and deleting the keys committed with the changes above were passed successfully)

[vtutrinov]

@ivandika3, thanks for the quick response! A workaround involving increasing the part size during MPU works well. It seems the fix for the HDDS-8238 jira ticket would be a great way to resolve the problem