-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support fetching logs & artifacts from CircleCI #164
Conversation
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## master #164 +/- ##
==========================================
- Coverage 66.96% 65.23% -1.73%
==========================================
Files 9 10 +1
Lines 1002 1260 +258
Branches 210 253 +43
==========================================
+ Hits 671 822 +151
- Misses 279 381 +102
- Partials 52 57 +5
|
I am trying this branch with the following tuned up config for bids-specification: (tinuous-dev-dev) datalad@smaug:/mnt/datasets/datalad/ci/bids-specification-trycircle$ cat tinuous.yaml
repo: bids-standard/bids-specification
vars:
path_prefix: '{year}//{month}//{day}/{type}/{type_id}/{commit[:7]}/{ci}-'
common_path: '{path_prefix}{wf_name}-{number}-{common_status}/'
ci:
github:
paths:
logs: '{common_path}'
artifacts: '{common_path}'
releases: 'releases/{release_tag}'
circleci:
paths:
logs: '{common_path}/{job_name}/{step}-{index}.txt'
# logs: '{prefix}/{wf_name}/{number}/{job_name}/{step}-{index}.txt'
# artifacts: '{prefix}/{wf_name}/{number}/{job_name}/artifacts/'
# since: 2022-07-18T00:00:00Z
since: 2023-04-26T00:00:00Z
types: [cron, pr, push, manual]
datalad:
enabled: true
cfg_proc: text2git which fails with requiring `job_number` somewherecreate(ok): 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete (dataset)
2023-04-27T14:27:56-0400 [INFO ] datalad: Refresh progress bars
2023-04-27T14:27:56-0400 [INFO ] datalad: Clear progress bars
action summary:
add (ok: 6)
create (ok: 1)
run (ok: 1)
save (ok: 3)
2023-04-27T14:27:56-0400 [INFO ] datalad: Refresh progress bars
2023-04-27T14:27:56-0400 [INFO ] tinuous: Downloading logs for job 18197, step 103-0 to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete/build_docs/103-0.txt
2023-04-27T14:27:56-0400 [INFO ] tinuous: Downloading logs for job 18197, step 1023-0 to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete/build_docs/1023-0.txt
Traceback (most recent call last):
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/bin/tinuous", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/__main__.py", line 117, in fetch
for obj in ci.get_build_assets(
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/circleci.py", line 135, in get_build_assets
for job in self.get_jobs(wf.id):
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/circleci.py", line 78, in get_jobs
yield Job.parse_obj(item)
^^^^^^^^^^^^^^^^^^^
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Job
job_number
field required (type=value_error.missing)
|
BTW, feel welcome to troubleshoot on smaug
|
@yarikoptic Please rerun with |
here is what I see at the end2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v1.1/project/gh/bids-standard/bids-specification/18199 HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 0-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/0-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 99-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/99-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 101-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/101-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 102-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/102-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 103-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/103-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 104-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/104-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18199, step 105-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/build_docs/105-0.txt; skipping
2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v1.1/project/gh/bids-standard/bids-specification/18201 HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 0-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/0-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 99-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/99-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 101-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/101-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 102-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/102-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 103-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/103-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18201, step 104-0 already downloaded to 2023/04/26/push/pull%2f1476/e69f942/circleci-search_build-6227-success/check_links/104-0.txt; skipping
2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v2/pipeline/f443550d-c2c3-4682-8664-1c1f5a96370f/workflow HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [INFO ] tinuous: Found pipeline 6226 with 1 workflow(s)
2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v2/workflow/55053b83-660e-4b06-8e62-e30be2aed688/job HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v1.1/project/gh/bids-standard/bids-specification/18198 HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18198, step 0-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs_pdf/0-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18198, step 99-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs_pdf/99-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18198, step 101-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs_pdf/101-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18198, step 102-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-failed/build_docs_pdf/102-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18198, step 103-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs_pdf/103-0.txt; skipping
2023-04-27T14:47:38-0400 [DEBUG ] urllib3.connectionpool: https://circleci.com:443 "GET /api/v1.1/project/gh/bids-standard/bids-specification/18197 HTTP/1.1" 200 None
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 0-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs/0-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 99-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs/99-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 101-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs/101-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 102-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-success/build_docs/102-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 103-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete/build_docs/103-0.txt; skipping
2023-04-27T14:47:38-0400 [INFO ] tinuous: Logs for job 18197, step 1023-0 already downloaded to 2023/04/26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete/build_docs/1023-0.txt; skipping
Traceback (most recent call last):
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/bin/tinuous", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datalad/miniconda3/envs/tinuous-dev-dev/lib/python3.11/site-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/__main__.py", line 117, in fetch
for obj in ci.get_build_assets(
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/circleci.py", line 135, in get_build_assets
for job in self.get_jobs(wf.id):
File "/mnt/datasets/datalad/ci/tinuous-dev/src/tinuous/circleci.py", line 78, in get_jobs
yield Job.parse_obj(item)
^^^^^^^^^^^^^^^^^^^
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Job
job_number
field required (type=value_error.missing)
|
@yarikoptic Try now. |
that works! with the above config it seems to incorrectly organize -- places PRs as e.g. `push/pull%2f1476` instead of under `pr`(tinuous-dev-dev) datalad@smaug:/mnt/datasets/datalad/ci/bids-specification-trycircle/2023/04$ find -iname circle*
./26/push/master/77066b3/circleci-search_build-6230-success
./26/push/master/9210f6e/circleci-search_build-6223-success
./26/push/master/9210f6e/circleci-search_build-6223-failed
./26/push/master/cf1175d/circleci-search_build-6229-success
./26/push/pull%2f1476/b3d8927/circleci-search_build-6228-success
./26/push/pull%2f1476/e69f942/circleci-search_build-6227-success
./26/push/pull%2f1476/e69f942/circleci-search_build-6227-failed
./26/push/pull%2f1476/5168e13/circleci-search_build-6226-success
./26/push/pull%2f1476/5168e13/circleci-search_build-6226-failed
./26/push/pull%2f1476/5168e13/circleci-search_build-6226-incomplete
./26/push/pull%2f1476/0df6f6d/circleci-search_build-6225-success
./26/push/pull%2f1476/0df6f6d/circleci-search_build-6225-failed
./26/push/pull%2f1476/0df6f6d/circleci-search_build-6225-incomplete
./26/push/pull%2f1476/baf6517/circleci-search_build-6224-success
./26/push/pull%2f1476/baf6517/circleci-search_build-6224-failed
./26/push/pull%2f1476/baf6517/circleci-search_build-6224-incomplete
./26/push/pull%2f1333/b4cccf4/circleci-search_build-6222-success
./26/push/pull%2f1333/b4cccf4/circleci-search_build-6222-failed
./27/push/pull%2f919/91b39fd/circleci-search_build-6231-success
./27/push/pull%2f919/91b39fd/circleci-search_build-6231-failed so I guess |
@yarikoptic I suspect those runs were on a branch whose name CircleCI reported as |
no need to suspect -- easy to check that it was a PR:
$> ls -l ./26/pr/1476
total 0
drwxr-sr-x 1 datalad datalad 130 Apr 27 13:31 0df6f6d/
drwxr-sr-x 1 datalad datalad 130 Apr 27 13:31 5168e13/
drwxr-sr-x 1 datalad datalad 124 Apr 27 13:31 b3d8927/
drwxr-sr-x 1 datalad datalad 130 Apr 27 13:31 baf6517/
drwxr-sr-x 1 datalad datalad 130 Apr 27 13:31 e69f942/ |
@yarikoptic So do you want tinuous to do anything different here? |
yes -- as I have mentioned above for the freshly added circle ci seems to not correctly assign |
@yarikoptic So do you want me to implement my idea of treating pushes to any |
ah, gotcha. I think there should be some way to gather that information more explicitly. E.g. in v1 I see that for "pull_requests": [{
"head_sha": "8139060f4d1f6ff617ac49f8afb2273c4fee2343",
"url": "https://github.com/circleci/circleci.com/pull/2347"
}], chatgpt was not helpful (dreaming), found them also be listed in in the https://circleci.com/docs/api/v2/index.html#operation/triggerPipeline I found branch | string so may be the suggestion you have is based on this , but then I wonder how would it differ if there is indeed a branch with that name!? Also within v2 in https://circleci.com/docs/api/v2/index.html#operation/getPipelineById it seems that
so may be trigger itself isn't enough and indeed should/could be combined with data from other fields to decide if a PR? |
@yarikoptic Do you know how the BIDS repository set up running CircleCI on PRs? The only option I can find for PR builds on CircleCI is the "Only build pull requests" option, but when I try it, the API still lists the PR head branch rather than |
@yarikoptic I suspect that bids-standard/bids-specification#1476 got " |
bids-standard/bids-specification#1477 comes from (tinuous-dev-dev) datalad@smaug:/mnt/datasets/datalad/ci/bids-specification-trycircle$ ls -l 2023/04/27/pr/1477/7e72709/
total 0
drwxr-sr-x 1 datalad datalad 74 May 1 14:51 'github-Label PRs-17-success'
drwxr-sr-x 1 datalad datalad 1374 May 1 14:52 github-schemacode_ci-2288-success
drwxr-sr-x 1 datalad datalad 416 May 1 14:52 github-Validation-818-success
(tinuous-dev-dev) datalad@smaug:/mnt/datasets/datalad/ci/bids-specification-trycircle$ ls -l 2023/04/27/push/rwblair-patch-1/7e72709/
total 0
drwxr-sr-x 1 datalad datalad 122 May 1 14:54 circleci-search_build-6232-success so we can't rely on branch name to be sufficient to tell that it was for a PR, although circle ci does report back to that PR in bids-standard/bids-specification#1477 . Might be worth checking if it appears as a PR within recent builds and how far "recent builds" go. May be it would be sufficient (even though v1)? |
@yarikoptic Seeing as CircleCI build commits on PRs the same way as it builds pushes — i.e., by building the commit from the repository rather than by autogenerating a merge commit — I do not believe CircleCI's "PR builds" should count as PR builds. They're just "push" builds that happen to belong to a PR branch. |
ok, probably you are right and indeed it is what it is... would then may be nice to have a note on that aspect somewhere in https://github.com/con/tinuous/pull/164/files#diff-7b3ed02bc73dc06b7db906cf97aa91dec2b2eb21f2d92bc5caa761df5bbc168fR242 |
@yarikoptic Note added: 47daabc |
Closes #124.
TO DO:
vcs
field is present in pipelines for scheduled & manually-triggered runsvcs
field reported by the API.vcs
field in the API.Properly handle expired artifacts