[#3348] feat(core,server): Add the list operation of the user #4055

jerqi · 2024-07-03T08:53:43Z

What changes were proposed in this pull request?

Add the list operation of the user

Why are the changes needed?

Fix: #3348

Does this PR introduce any user-facing change?

I will add the document later.

How was this patch tested?

Add the new ut.

lw-yang · 2024-07-03T08:59:23Z

core/src/main/java/com/datastrato/gravitino/authorization/AccessControlManager.java

+  public String[] listUsers(String metalake) throws NoSuchMetalakeException {
+    return doWithNonAdminLock(() -> userGroupManager.listUsers(metalake));
+  }
+
+  public User[] listUsersInfo(String metalake) throws NoSuchMetalakeException {
+    return doWithNonAdminLock(() -> userGroupManager.listUsersInfo(metalake));
+  }


I think listUsers -> listUserNames, listUsersInfo -> listUsers would be more appropriate

jerqi · 2024-07-04T03:02:38Z

@lw-yang Could you take an another look?

lw-yang · 2024-07-04T03:58:45Z

core/src/main/java/com/datastrato/gravitino/storage/relational/service/UserMetaService.java

+    for (UserPO userPO : userPOs) {
+      List<RolePO> rolePOs = RoleMetaService.getInstance().listRolesByUserId(userPO.getUserId());
+      userEntities.add(POConverters.fromUserPO(userPO, rolePOs, namespace));
+    }


the num of users > 1w is common situation, this may execute for a long time, maybe we should consider using cache in listUsers

cc @yuqi1129 Maybe we can have a common cache layer for this.

I think it is better to use a paging query.

jerryshao · 2024-07-12T08:53:05Z

@jerqi can you please update the PR.

jerqi · 2024-07-16T07:49:30Z

@lw-yang @jerryshao Do you have any other suggestion?

jerryshao · 2024-07-17T12:55:38Z

core/src/main/java/org/apache/gravitino/authorization/UserGroupManager.java

+      LOG.error("Listing user under metalake {} failed due to storage issues", metalake, ioe);
+      throw new RuntimeException(ioe);
+    }
+  }


I think you need to use tree lock here.

listUserNames can be implemented based on listUsers, don't need to duplicate the codes.

I think you need to use tree lock here.

I have used tree lock in the UserOperations.

listUserNames can be implemented based on listUsers, don't need to duplicate the codes.

OK, I have refactored the code.

jerryshao · 2024-07-17T12:59:53Z

core/src/main/java/org/apache/gravitino/storage/relational/service/UserMetaService.java

+            userPOs.addAll(
+                SessionUtils.doWithoutCommitAndFetchResult(
+                    UserMetaMapper.class,
+                    mapper -> mapper.listUserPOsByMetalakeId(metalakeId.get()))));


I think we don't have to use transaction here, simply because we don't use select for update, so the transaction is not useful and will lead to dead lock.

I modified. But it shouldn't bring dead lock using the transaction.

jerryshao · 2024-07-18T07:34:50Z

common/src/main/java/org/apache/gravitino/dto/responses/NameListResponse.java

+@Getter
+@ToString
+@EqualsAndHashCode(callSuper = true)
+public class NameListResponse extends BaseResponse {


You should implement validate() method.

CatalogListResponse doesn't implement validate, too. I created an issue #4197 to track it.

jerryshao · 2024-07-18T07:35:05Z

common/src/main/java/org/apache/gravitino/dto/responses/UserListResponse.java

+    super(0);
+    this.users = null;
+  }
+}


jerryshao · 2024-07-18T07:40:50Z

core/src/main/java/org/apache/gravitino/storage/relational/service/UserMetaService.java

+    for (UserPO userPO : userPOs) {
+      List<RolePO> rolePOs = RoleMetaService.getInstance().listRolesByUserId(userPO.getUserId());
+      userEntities.add(POConverters.fromUserPO(userPO, rolePOs, namespace));
+    }


Do you really want to return all the roles for all the users? This will be extremely cost, is that really the frontend needs?

Now, we adopts the lazily loading information about users/roles.

jerryshao · 2024-07-18T07:42:36Z

core/src/main/java/org/apache/gravitino/storage/relational/utils/SessionUtils.java

+    } catch (Throwable t) {
+      throw t;
+    } finally {
+      SqlSessions.closeSqlSession();


Why do you need to add this?

jerqi · 2024-09-09T03:27:26Z

After discussing with @shaofengshi offline, we need the ability to list all the role information of the user. It will be convenient for users. I have supported the user lazily loading. So if we only want to load the user names, we don't need to load unnecessary information.
In the future, we have many users, we can support to page. It's an improvement. We can support it in an another pull request.

xunliu · 2024-09-13T03:38:19Z

Query all data at once or cache buffering in the service also has the problem of dirty data. I think paged queries are better.
@jerryshao @jerqi What do you think?

jerryshao · 2024-09-14T00:17:35Z

core/src/main/java/org/apache/gravitino/authorization/UserGroupManager.java

+    try {
+      Namespace namespace = AuthorizationUtils.ofUserNamespace(metalake);
+      return store.list(namespace, UserEntity.class, Entity.EntityType.USER).stream()
+          .map(entity -> (User) entity)


Why do we need this line?

So you only get the user list here, right? Will you also fetch the roles for each user, I saw that User and UserDTO have roles fields, will this be empty?

Yes, I only get the user list here. I won't fetch roles unless they are needed. Now the roles is empty for every user.

You can see the pull request. #4690
UserEntity roles field don't be fetched directly. We will put a supplier method field instead. So we don't fetch the values directly if we only need the names.
If we need all the information of entities, we can also have the ability to put them. You can see the list details. Because the front side need the all the users information, so I provide the interface.

I mean here why do we need to do a type conversion entity -> (User) entity?

Sorry, it's unnecessary. I ever designed other interface listEntitiesByRelation. It need me to use type cast. For this place, we don't need. I have removed it.

jerryshao · 2024-09-14T00:24:20Z

core/src/main/java/org/apache/gravitino/storage/relational/mapper/UserMetaBaseSQLProvider.java

+        + " mt ON ut.metalake_id = mt.metalake_id"
+        + " WHERE mt.metalake_name = #{metalakeName}"
+        + " AND ut.deleted_at = 0 AND mt.deleted_at = 0";
+  }


This SQL will not throw an exception when metalake is not existed, it will only return an empty list. In your definition, you have NoSuchMetalakeException defined, to throw an such exception, you have to add one more check to see if metalake is existed or not.

Added the checkMetalakeExists.

jerryshao · 2024-09-14T00:25:27Z

server/src/test/java/org/apache/gravitino/server/web/rest/TestUserOperations.java

+    ErrorResponse errorResponse2 = resp3.readEntity(ErrorResponse.class);
+    Assertions.assertEquals(ErrorConstants.INTERNAL_ERROR_CODE, errorResponse2.getCode());
+    Assertions.assertEquals(RuntimeException.class.getSimpleName(), errorResponse2.getType());
+  }


I think you also need to add integration tests for this.

I have added IT in the AccessControlId.java.

jerryshao · 2024-09-20T08:43:53Z

core/src/main/java/org/apache/gravitino/storage/relational/RelationalEntityStore.java

+      Namespace namespace,
+      Class<E> type,
+      Entity.EntityType entityType,
+      List<Field> allowMissingFields)


Is this design used for skipping some unnecessary fields? @yuqi1129 please also take a look about this design.

Yes, because different upper layers may ignore different fields. For example, list roles
Some upper layers may ignore objects and objects count.
Some upper layers may only ignore objects.
Some upper layers may not ignore any field.

jerryshao · 2024-09-20T09:10:10Z

core/src/main/java/org/apache/gravitino/EntityStore.java

   */
  <E extends Entity & HasIdentifier> List<E> list(
-      Namespace namespace, Class<E> type, EntityType entityType) throws IOException;
+      Namespace namespace, Class<E> type, EntityType entityType, List<Field> allowMissingFields)
+      throws IOException;


Maybe we can add a another inteface with default to fetch all the fields, then you don't have to change the code above for different catalogs.

OK, I preferring adding another interface, too. cc @yuqi1129 @xunliu

Also, I would suggest to rename the parameter to like: List<Field> skippingFields

jerryshao · 2024-09-20T11:35:19Z

core/src/main/java/org/apache/gravitino/EntityStore.java

   * @return the list of entities
+   * @throws IOException if the list operation fails
   */
  <E extends Entity & HasIdentifier> List<E> list(
      Namespace namespace, Class<E> type, EntityType entityType) throws IOException;


What I mean is that these interface can be implemented like:

default XX list(xxxx) { return list(xx, xx, xx, Lists.emptyList) }

jerryshao · 2024-09-20T11:40:45Z

core/src/main/java/org/apache/gravitino/storage/relational/mapper/h2/UserMetaH2Provider.java

+        + " '[' || GROUP_CONCAT('\"' || rot.role_name || '\"') || ']' as roleNames,"
+        + " '[' || GROUP_CONCAT('\"' || rot.role_id || '\"') || ']' as roleIds"


What is the usage of these code, can you explain it?

H2 doesn't support json_agg function. This uses GROUP_CONCAT to implement similar feature.

jerryshao · 2024-09-20T11:42:28Z

core/src/main/java/org/apache/gravitino/storage/relational/po/CombinedUserPO.java

+        && Objects.equal(getLastVersion(), combinedUserPO.getLastVersion())
+        && Objects.equal(getDeletedAt(), combinedUserPO.getDeletedAt())
+        && Objects.equal(getRoleIds(), combinedUserPO.getRoleIds())
+        && Objects.equal(getRoleNames(), combinedUserPO.getRoleNames());


I think you can use like super.equals(xxxx) and super.hashCode() to simplify the code here and below.

OK, I will do.

jerryshao · 2024-09-20T11:47:57Z

core/src/main/java/org/apache/gravitino/storage/relational/service/UserMetaService.java

+    String metalakeName = namespace.level(0);
+
+    if (skippingFields.contains(UserEntity.ROLE_IDS)
+        && skippingFields.contains(UserEntity.ROLE_NAMES)) {


It seems that skippingFields is not flexible enough, you still need to explicitly check these fields and execute by different branches.

After considering twice, I extract an interface SupportsSkippingFields to express the list user namespace handler. I also wrap the logic which we select the handler according to skipping fields into SupportsSKippingFieldsHandlers.
Now, we can match the skipping fields according to the handler which defined. It reduce the branches.
Moreover, we ask the developer must add high priority handler first.(You can see the code SupportsSkippingFieldsHandlers for details). I can add the priority method for the SupportsSkippingFields. But for now, this is not a blocker. So just add a comment to notify others the correct usage for this interface.

lw-yang reviewed Jul 3, 2024

View reviewed changes

jerqi force-pushed the ISSUE-3348 branch 4 times, most recently from a124b10 to 2f1476c Compare July 3, 2024 11:45

jerqi closed this Jul 4, 2024

jerqi reopened this Jul 4, 2024

lw-yang reviewed Jul 4, 2024

View reviewed changes

jerqi force-pushed the ISSUE-3348 branch 2 times, most recently from 9d07ea0 to a57e4e3 Compare July 16, 2024 06:56

jerryshao reviewed Jul 17, 2024

View reviewed changes

jerqi force-pushed the ISSUE-3348 branch from 88e3a3b to fba57b2 Compare July 18, 2024 07:25

jerryshao reviewed Jul 18, 2024

View reviewed changes

jerqi force-pushed the ISSUE-3348 branch 3 times, most recently from 62c5484 to 8118c63 Compare September 9, 2024 03:05

jerqi force-pushed the ISSUE-3348 branch 3 times, most recently from 4b728a5 to b2d206d Compare September 9, 2024 08:25

jerqi self-assigned this Sep 9, 2024

jerqi added the branch-0.6 Automatically cherry-pick commit to branch-0.6 label Sep 9, 2024

jerryshao reviewed Sep 14, 2024

View reviewed changes

jerqi requested a review from jerryshao September 14, 2024 02:08

jerqi added 2 commits September 18, 2024 14:03

Add ListRoles

4ec29a7

Add the metalake check

4b333a7

jerqi force-pushed the ISSUE-3348 branch from 056bf1e to af587e2 Compare September 18, 2024 09:55

Use join

92c3032

jerqi force-pushed the ISSUE-3348 branch from af587e2 to 92c3032 Compare September 18, 2024 10:07

jerqi added 6 commits September 18, 2024 19:03

fix test

81cd6cb

fix npe

15144d2

fix style

0e6b040

Modify PG SQL

0a0d6b3

fix comment

9d0821d

refactor the interface

feaa46f

jerryshao reviewed Sep 20, 2024

View reviewed changes

jerqi added 2 commits September 20, 2024 18:30

rename interface

91ac43a

fix test code errors

d371b97

jerryshao reviewed Sep 20, 2024

View reviewed changes

jerqi added 4 commits September 20, 2024 21:47

Add a new interface

93d7d38

make default for list

1707153

rename

ff1640f

Remove rebudant cast

fdc9133

		+ " '[' \|\| GROUP_CONCAT('\"' \|\| rot.role_name \|\| '\"') \|\| ']' as roleNames,"
		+ " '[' \|\| GROUP_CONCAT('\"' \|\| rot.role_id \|\| '\"') \|\| ']' as roleIds"

[#3348] feat(core,server): Add the list operation of the user #4055

Are you sure you want to change the base?

[#3348] feat(core,server): Add the list operation of the user #4055

Conversation

jerqi commented Jul 3, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi commented Jul 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao commented Jul 12, 2024

jerqi commented Jul 16, 2024

Choose a reason for hiding this comment

jerqi Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi commented Sep 9, 2024

xunliu commented Sep 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi Sep 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

jerryshao Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerqi Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

jerqi Jul 17, 2024 •

edited

Loading

jerqi Sep 14, 2024 •

edited

Loading

jerqi Sep 20, 2024 •

edited

Loading

jerryshao Sep 20, 2024 •

edited

Loading

jerqi Sep 20, 2024 •

edited

Loading

jerqi Sep 20, 2024 •

edited

Loading