If materialization is kicked off from the variant view UI, and page is refreshes, materialization happens again and they collide.

[dbgoodman]

You get this error:

ERROR Internal Server Error: /_/variants/refresh_materialized_variant_table
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "/home/ubuntu/millstone/genome_designer/main/xhr_handlers.py", line 1000, in refresh_materialized_variant_table
    mvmvm.create()
  File "/home/ubuntu/millstone/genome_designer/variants/materialized_view_manager.py", line 38, in create
    self.create_internal()
  File "/home/ubuntu/millstone/genome_designer/variants/materialized_view_manager.py", line 208, in create_internal
    self.cursor.execute(create_sql_statement)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 41, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
    six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
    return self.cursor.execute(query, args)
DatabaseError: relation "materialized_melted_variant_c534fa08" already exists

Also a corresponding popup dialog error in the browser.

[dbgoodman]

The solution is to set up some kind of database lock once materialization starts so that it doesn't start twice. This could be done with a materializing status in addition to a valid/invalid status.

[dbgoodman]

One other potential complication:

Say the view is invalidated to start with.
User A loads the variant UI and kicks off a materialization.
User B then starts calling SV variants, adding variants to the DB, which then re-invalidates the view before User A's materialization finishes.

Should we restart materialization after user B re-invalidates? We would need to make sure that User B's new variants aren't forgotten in the new materialized view. A dumb lock that ignores User B's re-invalidation would result in a materialized view that was missing the new variants but was marked as 'valid'.

[glebkuznetsov]

The lock is not a good idea. For example, if the machine goes off during
materialization.

If anything we should store a "latest_valid_uid" flag that will prevent
clashes.

Even better is if upgrading Postgres fixes this.

On Thu, Jun 16, 2016, 15:47 Daniel Bryan Goodman [email protected]
wrote:

One other potential complication:

Say the view is invalidated to start with.
User A loads the variant UI and kicks off a materialization.
User B then starts calling SV variants, adding variants to the DB, which
then re-invalidates the view before User A's materialization finishes.

Should we restart materialization after user B re-invalidate? We would
need to make sure to make sure to that User B's new variants aren't
forgotten in the new materialized view, which could happen with a dumb lock
that ignores User B's re-invalidation.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#677 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAORu7dAvwrGooTDdXflwdp59KdZEyzaks5qMahegaJpZM4IyaBu
.