Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Built-in scheduler should consider changes to missing intervals due to signals #3094

Closed
plaflamme opened this issue Sep 5, 2024 · 0 comments
Assignees
Labels
Bug Something isn't working Scheduler: Built-in

Comments

@plaflamme
Copy link
Contributor

Assuming the following:

MODEL(name full_with_signal, kind FULL, cron '@daily', signals=[(kind=SomeSignal)]);
select * from external_table

and

MODEL(name other_full, kind FULL, cron '@daily');
select * from full_with_signal

When the scheduler runs and both models have a missing interval, then they will correctly execute in sequence, allowing other_full to take into account the data in full_with_signal.

If the signal on full_with_signal determines that the interval is not yet ready, it will correctly be skipped by the scheduler, but the interval for other_full will still be computed in that run. In a subsequent run, the signal becomes ready and the interval for full_with_signal is computed, but now we're in a situation where they have run in the reverse order, making other_full not take data from full_with_signal into account.

The built-in scheduler needs to consider changes to missing intervals made by signals.

@izeigerman izeigerman added Bug Something isn't working Scheduler: Built-in labels Sep 17, 2024
@tobymao tobymao self-assigned this Oct 13, 2024
tobymao added a commit that referenced this issue Oct 17, 2024
if a signal marks an interval as 'unready', then all downstream jobs
should also consider than interval as unready.
tobymao added a commit that referenced this issue Oct 18, 2024
if a signal marks an interval as 'unready', then all downstream jobs
should also consider than interval as unready.
tobymao added a commit that referenced this issue Oct 18, 2024
if a signal marks an interval as 'unready', then all downstream jobs
should also consider than interval as unready.
tobymao added a commit that referenced this issue Oct 18, 2024
if a signal marks an interval as 'unready', then all downstream jobs
should also consider than interval as unready.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Scheduler: Built-in
Projects
None yet
Development

No branches or pull requests

3 participants