Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve compatibility with different visibility formats #622

Open
7 tasks
mpluess opened this issue Oct 9, 2024 · 1 comment
Open
7 tasks

Improve compatibility with different visibility formats #622

mpluess opened this issue Oct 9, 2024 · 1 comment
Labels
prio-med Medium priority

Comments

@mpluess
Copy link
Member

mpluess commented Oct 9, 2024

#620 changed the default visibility format to MS and improved / unified how visibilities are handled within Karabo.
However, the compatibility of simulators and imagers with the different visibility formats (currently two) needs to be improved or Karabo needs to be able to convert incompatible formats on-the-fly. See also this comment: #595 (comment)
Conversion of large visibility files can take a long time and potentially use a lot of storage. There are two options: do conversions only when specifically requested, e.g. via a flag, and raise exceptions by default when encountering an incompatible format, or automatically convert if needed and issue a warning to the user.
As Simon mentioned, conversion would ideally by implemented in a streaming fashion, i.e. without touching the disk.
When auto-converting, if the same file has been converted before, there should be a mechanism in place to reuse it and don't do unnecessary work. Requires checking if there have been changes to the original file since the last conversion.

  • imager_rascil: support VIS to MS conversion @ create_dirty_image, create_cleaned_image, create_cleaned_image_variants
  • imager_wsclean: support VIS to MS conversion @ create_dirty_image, create_cleaned_image
  • InterferometerSimulation.__run_simulation_rascil: support MS to VIS conversion
  • InterferometerSimulation.__run_simulation_long: support VIS output
  • Visibility.combine_vis: support MS input
  • Add newly supported combos to test_sim_format_imager_combos
  • Remove temporary function karabo.test.util.get_compatible_dirty_imager
@sfiruch
Copy link
Member

sfiruch commented Oct 10, 2024

1 vote for "automatically convert if needed and issue a warning to the user."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prio-med Medium priority
Projects
None yet
Development

No branches or pull requests

2 participants