AIP-157 Partial response implementation #717

sayan-biswas · 2024-02-21T13:03:14Z

In the present implementation, a user cannot filter the data in the response message, a feature provided by modern API implementations. This is particularly needed for the List APIs as these APIs return the whole object in the list. In scenario where the list and object definition are big, there's a lot network bandwidth wasted because the response of the List API is mostly used to display a list of resources and the Get API is then used to fetch the individual resource.
This feature provides an easy and basic way to filter the response message across all APIs, which reduces the size of the payload and hence allows faster transfer over network.

Changes

This feature add capabilities to filter the response message from across all APIs.
AIP detail:
https://google.aip.dev/157

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Tested your changes locally (if this is a code change)
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

API: Request message allows an additional parameter `fields` to filter the response message.

tekton-robot · 2024-02-21T13:03:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

sayan-biswas · 2024-02-28T20:18:37Z

/test all

tekton-robot · 2024-02-28T20:22:55Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

sayan-biswas · 2024-02-28T20:45:47Z

/test all

tekton-robot · 2024-02-28T20:49:52Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

sayan-biswas · 2024-02-29T12:14:24Z

/test pull-tekton-results-integration-tests

sayan-biswas · 2024-03-02T07:40:30Z

/test pull-tekton-results-integration-tests

gabemontero

finally got around to taking a pass at this @sayan-biswas

of course, being brand new / a novice to all this, consider that perspective when reviewing my feedback ;-)

docs/api/README.md

gabemontero · 2024-04-12T14:18:48Z

internal/fieldmask/fieldmask.go

+	mask := make(FieldMask)
+	for _, path := range paths {
+		current := mask
+		fields := strings.Split(path, ".")


put a comment above this split with some sample paths to illustrate why we are splitting on .

let's not make the coder have to do up to your README change to undersatand what is going on

Added reference and an example in godoc format.

awesome thanks

internal/fieldmask/fieldmask.go

internal/fieldmask/fieldmask_test.go

internal/fieldmask/fieldmask.go

gabemontero · 2024-04-12T15:26:25Z

cmd/api/main.go

@@ -169,6 +171,7 @@ func main() {
 			grpc_zap.UnaryServerInterceptor(grpcLogger, zapOpts...),
 			grpc_auth.UnaryServerInterceptor(determineAuth),
 			prometheus.UnaryServerInterceptor,
+			fieldmask.UnaryServerInterceptor(),


shouldn't we have a feature enablement check of some sort ? presumably a client could send the extra headers or encoded data and it is OK to ignore ?

I have added a feature flag implementation to handle all features control.

feature gate looks good @sayan-biswas good think .... in looking at it though I raised some questions around changes you made to my recently added tuning options

I'll submit a separate review for those

tekton-robot · 2024-04-20T09:46:00Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

tekton-robot · 2024-05-02T22:46:03Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	9.3%	9.1%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	70.1%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/features/features.go	Do not exist	85.3%

tekton-robot · 2025-02-18T19:52:08Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/features/features.go	Do not exist	85.3%

khrm · 2025-02-19T06:50:46Z

/assign @enarha

tekton-robot · 2025-02-19T06:53:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/features/features.go	Do not exist	85.3%

tekton-robot · 2025-02-19T06:59:24Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/features/features.go	Do not exist	85.3%

tekton-robot · 2025-02-19T07:20:08Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/features/features.go	Do not exist	85.3%

khrm · 2025-02-19T09:06:19Z

/test pull-tekton-results-unit-tests

tekton-robot · 2025-02-19T11:03:34Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/features/features.go	Do not exist	85.3%

khrm · 2025-02-20T11:03:56Z

/hold retesting this after rebase.

Co-authored-by: Sayan Biswas <[email protected]>

This feature add capabilities to filter the response message from all the APIs. AIP detail: https://google.aip.dev/157 Co-authored-by: Sayan Biswas <[email protected]>

This sets the protojson marshaller to omit the empty fields.

tekton-robot · 2025-02-20T11:08:52Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.4%	8.3%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	67.9%
pkg/api/server/features/features.go	Do not exist	85.3%

khrm · 2025-02-20T12:14:21Z

/hold cancel

Tested this. It's working fine.

enarha · 2025-02-20T13:05:50Z

/approve

tekton-robot · 2025-02-20T13:06:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enarha

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [enarha]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

khrm

/lgtm

sayan-biswas added the kind/feature Categorizes issue or PR as related to a new feature. label Feb 21, 2024

sayan-biswas self-assigned this Feb 21, 2024

tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 21, 2024

tekton-robot requested review from avinal and enarha February 21, 2024 13:03

tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 21, 2024

sayan-biswas force-pushed the partial-response branch from f603652 to 50dd2b9 Compare February 28, 2024 18:18

tekton-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 28, 2024

sayan-biswas force-pushed the partial-response branch 2 times, most recently from 32027d4 to 08b524b Compare February 28, 2024 19:57

sayan-biswas requested review from adambkaplan and khrm February 28, 2024 20:17

sayan-biswas force-pushed the partial-response branch from 08b524b to 4ea8c19 Compare February 28, 2024 20:44

sayan-biswas marked this pull request as ready for review February 28, 2024 20:45

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 28, 2024

gabemontero reviewed Apr 12, 2024

View reviewed changes

sayan-biswas force-pushed the partial-response branch 2 times, most recently from 0acbbac to 840e4cc Compare May 2, 2024 22:40

sayan-biswas force-pushed the partial-response branch from 840e4cc to 6b8e353 Compare May 2, 2024 22:57

khrm force-pushed the partial-response branch 2 times, most recently from f383686 to ee9cd28 Compare February 19, 2025 06:50

tekton-robot assigned enarha Feb 19, 2025

khrm force-pushed the partial-response branch from ee9cd28 to 625797a Compare February 19, 2025 06:55

khrm force-pushed the partial-response branch from cfb931f to d47fa10 Compare February 19, 2025 07:16

khrm force-pushed the partial-response branch from d47fa10 to 6ff5cf2 Compare February 19, 2025 10:58

aThorp96 mentioned this pull request Feb 19, 2025

Results API should return Partial Response #974

Open

tekton-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 20, 2025

khrm and others added 3 commits February 20, 2025 16:34

Add Feature Gates for Results API server

d411158

Co-authored-by: Sayan Biswas <[email protected]>

AIP-157 Partial response implementation

3c22aeb

This feature add capabilities to filter the response message from all the APIs. AIP detail: https://google.aip.dev/157 Co-authored-by: Sayan Biswas <[email protected]>

Omit Fields which are empty from protobuff

0be5e2a

This sets the protojson marshaller to omit the empty fields.

khrm force-pushed the partial-response branch from 6ff5cf2 to 0be5e2a Compare February 20, 2025 11:04

tekton-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 20, 2025

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 20, 2025

khrm reviewed Feb 20, 2025

View reviewed changes

tekton-robot assigned khrm Feb 20, 2025

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 20, 2025

tekton-robot merged commit f4a98d7 into tektoncd:main Feb 20, 2025
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AIP-157 Partial response implementation #717

AIP-157 Partial response implementation #717

sayan-biswas commented Feb 21, 2024 •

edited

Loading

tekton-robot commented Feb 21, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 29, 2024

sayan-biswas commented Mar 2, 2024

gabemontero left a comment

gabemontero Apr 12, 2024

gabemontero Apr 12, 2024

sayan-biswas May 3, 2024

gabemontero May 3, 2024

gabemontero Apr 12, 2024

sayan-biswas May 3, 2024

gabemontero May 3, 2024

tekton-robot commented Apr 20, 2024

tekton-robot commented May 2, 2024

tekton-robot commented Feb 18, 2025

khrm commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

khrm commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

khrm commented Feb 20, 2025

tekton-robot commented Feb 20, 2025

khrm commented Feb 20, 2025

enarha commented Feb 20, 2025

tekton-robot commented Feb 20, 2025

khrm left a comment

AIP-157 Partial response implementation #717

AIP-157 Partial response implementation #717

Conversation

sayan-biswas commented Feb 21, 2024 • edited Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented Feb 21, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 29, 2024

sayan-biswas commented Mar 2, 2024

gabemontero left a comment

Choose a reason for hiding this comment

gabemontero Apr 12, 2024

Choose a reason for hiding this comment

gabemontero Apr 12, 2024

Choose a reason for hiding this comment

sayan-biswas May 3, 2024

Choose a reason for hiding this comment

gabemontero May 3, 2024

Choose a reason for hiding this comment

gabemontero Apr 12, 2024

Choose a reason for hiding this comment

sayan-biswas May 3, 2024

Choose a reason for hiding this comment

gabemontero May 3, 2024

Choose a reason for hiding this comment

tekton-robot commented Apr 20, 2024

tekton-robot commented May 2, 2024

tekton-robot commented Feb 18, 2025

khrm commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

khrm commented Feb 19, 2025

tekton-robot commented Feb 19, 2025

khrm commented Feb 20, 2025

tekton-robot commented Feb 20, 2025

khrm commented Feb 20, 2025

enarha commented Feb 20, 2025

tekton-robot commented Feb 20, 2025

khrm left a comment

Choose a reason for hiding this comment

sayan-biswas commented Feb 21, 2024 •

edited

Loading