Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle previously failed indexing jobs #206

Merged
merged 14 commits into from
Dec 30, 2024
Merged

Handle previously failed indexing jobs #206

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
dd9e0da
Add option to use existing ingress controller
clarked-msft Aug 15, 2024
3d6c812
Merge branch 'main' into main
jgbradley1 Aug 15, 2024
307de72
Update AKS_NAMESPACE env when creating indexing manager
clarked-msft Aug 16, 2024
96775f5
Merge branch 'main' of https://github.com/clarked-msft/graphrag-accel…
clarked-msft Aug 16, 2024
0e64131
Merge branch 'Azure-Samples:main' into main
clarked-msft Aug 16, 2024
9dce010
Update index namespace reference
clarked-msft Aug 19, 2024
116c8a2
Merge branch 'main' into main
jgbradley1 Sep 13, 2024
23a304a
Merge main
clarked-msft Nov 19, 2024
74738cc
Update manage index job script to handle failed indexing jobs
clarked-msft Nov 19, 2024
608d988
Lower indexing job memory requirement
clarked-msft Dec 6, 2024
6ca0f06
Merge branch 'main' of https://github.com/Azure-Samples/graphrag-acce…
clarked-msft Dec 6, 2024
d83b1f1
Add TTL for indexing job
clarked-msft Dec 6, 2024
bf296ca
Remove indexing managed TTL
clarked-msft Dec 6, 2024
a8a7b30
Merge branch 'main' into main
jgbradley1 Dec 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion backend/indexing-job-manager-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
ttlSecondsAfterFinished: 30
template:
metadata:
labels:
Expand Down
4 changes: 2 additions & 2 deletions backend/indexing-job-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ spec:
imagePullPolicy: Always
resources:
requests:
cpu: "6"
memory: "56Gi"
cpu: "5"
memory: "36Gi"
limits:
cpu: "8"
memory: "64Gi"
Expand Down
5 changes: 3 additions & 2 deletions backend/manage-indexing-jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ def list_k8s_jobs(namespace: str) -> list[str]:
jobs = batch_v1.list_namespaced_job(namespace=namespace)
job_list = []
for job in jobs.items:
job_list.append(job.metadata.name)
if job.metadata.name.startswith("indexing-job-") and job.status.active:
job_list.append(job.metadata.name)
return job_list


Expand Down Expand Up @@ -124,7 +125,7 @@ def main():
)
pipelinejob = PipelineJob()
pipeline_job = pipelinejob.load_item(item["sanitized_index_name"])
pipeline_job["status"] = PipelineJobState.FAILED.value
pipeline_job.status = PipelineJobState.FAILED
else:
print(
f"Indexing job for '{item['human_readable_index_name']}' already running. Will not schedule another. Exiting..."
Expand Down
Loading