feat: Implement MPT changes

[shawnxie999]

Spec is here https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0033d-multi-purpose-tokens

Builds rippled on https://github.com/gregtatcam/rippled/tree/mpt-direct-pay

• refactor Issue and STAmount related changes (will need to change if rippled has further changes)
• introduces new table mp_token_holders
• ETL that extracts holders into mp_token_holders table whenever a new MPTokenAuthorize tx is observed
• new mpt_holders API
• inject synthetic mpt_issuance_id field into MPTokenIssuanceCreate transaction metadata for APIs (eg, tx, ledger, subscribe, account_tx)
• support new MPTokenIssuance and MPToken objects types for ledger_entry
• inject synthetic mpt_issuance_id field into MPTokenIssuance object for ledger_data and account_objects APIs

[cindyyan317]

Actually , Only this part needs to be virtual. The filter logic is same for different DB. By reducing the scope of virtual functions, you can unittest more of the logic without real DB.

[shawnxie999]

could you clarify what you mean? inside this function, there is another call to the database auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

[cindyyan317]

If we create a function in CassandraBackend.hpp to just select MPT holder, then all the filter logic can move to BackendInterface. If we want to add a new database, you don't need to copy all the logic to new database implementation. How to filter the valid holder is database unrelated. Instead, new database only needs to provide how to select MPT holder.

[shawnxie999]

But aren't existing functions in CassandraBackend.hpp also do something similar? BTW fetchMPTHolders doesn't only have filter logic, it does make another call to the objects table on line 576:

auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

[cindyyan317]

I understood the fetchMPTHolders does not only have filter logic. So I suggested that we can add the pure virtual function doFetchMPTHolders in BackendInterface, which only obtains the data from selectMPTHolders table. Then the filer logic can move to BackendInterface normal function.

override doFetchfetchMPTHolders in CassandraBackend.hpp

    std::pair<std::vector<ripple::uint256>, std::optional<ripple::AccountID>>
    doFetchfetchMPTHolders(
        ripple::uint192 const& mptID,
        std::uint32_t const limit,
        std::optional<ripple::AccountID> const& cursorIn,
        boost::asio::yield_context yield
    ) const override
    {
        auto const holderEntries = executor_.read(
            yield, schema_->selectMPTHolders, mptID, cursorIn.value_or(ripple::AccountID(0)), Limit{limit}
        );

        auto const& holderResults = holderEntries.value();
        if (not holderResults.hasRows()) {
            LOG(log_.debug()) << "No rows returned";
            return {};
        }

        std::vector<ripple::uint256> mptKeys;
        std::optional<ripple::AccountID> cursor;
        for (auto const [holder] : extract<ripple::AccountID>(holderResults)) {
            mptKeys.push_back(ripple::keylet::mptoken(mptID, holder).key);
            cursor = holder;
        }
        return {mptKeys, cursor};
    }

Then your fetchMPTHolders will be like:

MPTHoldersAndCursor
BackendInterface::fetchMPTHolders(
    ripple::uint192 const& mptID,
    std::uint32_t const limit,
    std::optional<ripple::AccountID> const& cursorIn,
    std::uint32_t const ledgerSequence,
    boost::asio::yield_context yield
) const
{
    auto [mptKeys, cursor] = doFetchfetchMPTHolders(mptID, limit, cursorIn, yield);

    auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

    auto it = std::remove_if(mptObjects.begin(), mptObjects.end(), [](Blob const& mpt) { return mpt.empty(); });

    mptObjects.erase(it, mptObjects.end());

    ASSERT(mptKeys.size() <= limit, "Number of keys can't exceed the limit");
    if (mptKeys.size() == limit)
        return {mptObjects, cursor};

    return {mptObjects, {}};
}

I admit that current CassandraBackend.hpp has similar implementation which is not correct. We will refactor them eventually. Feel free to leave it. We can refactor it in future.

[cindyyan317]

💯 All comments are addressed. Looks like ready to go. 👍

[shawnxie999]

updated to rippled v2.3.0 in commit 4863a9f

@cindyyan317 @godexsoft current build passes. i think would be ready to merge unless you guys have more feedbacks