filter_skymodel

Filter and group a sky model based on image data.

This module provides functionality to filter and refine sky models using image data and source finders like PyBDSF or SoFiA-2. It allows for the creation of both apparent and true sky models, taking into account primary beam correction.

There are two submodules: bdsf and sofia, implementing the sky model filtering with pyBDSF and SoFiA-2 respectively.

lsmtool.filter_skymodel.filter_skymodel(
flat_noise_image: str | Path,
true_sky_image: str | Path | None,
input_true_skymodel: str | Path | None,
input_apparent_skymodel: str | Path | None,
output_apparent_sky: str | Path,
output_true_sky: str | Path,
beam_ms: str | Path | None | Sequence[str | Path],
input_bright_skymodel: str | Path | None = None,
*,
source_finder: str = 'bdsf',
**kws,
)

Filters a sky model based on a source finder.

This function filters a sky model using either SoFiA-2 or PyBDSF, based on the source_finder parameter. It applies the chosen source finder to generate a filtered sky model.

Parameters:

  • flat_noise_image (str or pathlib.Path) – Filename of input image to use to detect sources for filtering. It should be a flat-noise / apparent sky image (without primary-beam correction).

  • true_sky_image (str or pathlib.Path or None) –

    Filename of input image to use to determine the true flux of sources. It should be a true flux image (with primary-beam correction).

    • If beam_ms is given and exists, this parameter is ignored and the flat_noise_image is used instead.

    • If beam_ms is None or an empty string, this argument must be supplied.

  • input_true_skymodel (str or pathlib.Path or None) –

    Filename of input makesourcedb sky model, with primary-beam correction.

    • If this file exists, and input_bright_skymodel exists, they are concatenated and used as the input_true_skymodel.

    • If this file does not exist, and the input_bright_skymodel does not exist either, source filtering is still done on the input_apparent_skymodel if that exists.

  • input_apparent_skymodel (str or pathlib.Path or None) –

    Filename of input makesourcedb sky model, without primary-beam correction.

    • If this file exists, and input_true_skymodel exists, it is filtered and grouped to match the sources and patches of the input_true_skymodel.

    • If this file does not exist it is generated from the input_true_skymodel by applying the beam attenuation. If beam_ms is not given, the generated output_apparent_sky file will be identical to the output_true_sky file.

  • output_apparent_sky (str or pathlib.Path or None) –

    Output file name for the generated apparent sky model, without primary-beam correction.

    • If this file exists, it will be overwritten.

  • output_true_sky (str or pathlib.Path) –

    Output file name for the generated true sky model, with primary-beam correction.

    • If this file exists, it will be overwritten.

  • beam_ms (str or pathlib.Path or list of str or list of Path, default None) –

    The filename of the MS for deriving the beam attenuation and theoretical image noise.

    • If None (the default), or an empty string, the generated apparent and true sky models will be equal.

  • input_bright_skymodel (str or pathlib.Path, optional) –

    Filename of input makesourcedb sky model of bright sources only. This parameter can be used to add back bright sources to the sky model that were peeled before imaging. This should be a true sky model, with primary-beam correction.

    • If input_true_skymodel exists, sources in input_bright_skymodel will be added to the sky model from input_true_skymodel.

    • If input_true_skymodel does not exist, the input_bright_skymodel will be used as the input_true_skymodel.

  • source_finder (str, optional) – The source finder to use, either “sofia” or “bdsf”. Defaults to “bdsf”.

  • **kws – Additional keyword arguments to pass to the source finder function.

Returns:

n_sources (int or None) – The number of sources detected (only returned when source_finder is “bdsf”; otherwise None is returned).

See also

lsmtool.filter_skymodel.bdsf.filter_skymodel

Skymodel filtering implementation using pyBDSF.

lsmtool.filter_skymodel.sofia.filter_skymodel

Skymodel filtering implementation using SoFia-2.

lsmtool.filter_skymodel.resolve_source_finder(name: str) → str

Resolve which source finder to use based on input string.

This function checks the given source finder name against valid options, Currently supported options are: “bdsf” and “sofia” (if installed). The currently supported source finders are listed in the module variable: KNOWN_SOURCE_FINDERS.

Parameters:

name (str) – The source finder name to resolve. Input is converted to lower case. If the result is not in the list of known source finders, an exception is raised.

Returns:

str – The resolved source finder name.

Raises:

ValueError – If the input name is not a string matching one of the known source finders.

bdsf

Module for filtering and grouping a sky model based on sources found by pyBDSF in an image. This file was originally copied from the rapthor repository. Also includes substantial changes introduced in the SKA self calibration pipeline.

lsmtool.filter_skymodel.bdsf.add_bright_sources(
input_skymodel: SkyModel,
input_bright_skymodel: str | Path,
)

Add bright sources to the input sky model.

Loads the bright source sky model, renames the sources to remove any previous sector prefixes, and appends it to the input sky model.

Parameters:

  • input_skymodel (SkyModel) – The input sky model to which bright sources will be added.

  • input_bright_skymodel (str or pathlib.Path) – The filename of the bright source sky model.

lsmtool.filter_skymodel.bdsf.create_dummy_skymodel(
img_true_sky: bdsf.image.Image,
output_true_sky: str | Path,
output_apparent_sky: str | Path,
)

Create a dummy sky model if no islands of emission are detected.

Parameters:

  • img_true_sky (bdsf.image.Image) – The PyBDSF image object.

  • output_true_sky (str or pathlib.Path) – Output file name for the true sky model.

  • output_apparent_sky (str or pathlib.Path) – Output file name for the apparent sky model.

lsmtool.filter_skymodel.bdsf.filter_skymodel(
flat_noise_image: str | Path,
true_sky_image: str | Path | None,
input_true_skymodel: str | Path | None,
input_apparent_skymodel: str | Path | None,
output_apparent_sky: str | Path,
output_true_sky: str | Path,
vertices_file: str | Path,
beam_ms: str | Path | None | Sequence[str | Path] = None,
input_bright_skymodel: str | Path | None = None,
*,
thresh_isl: Real = 5.0,
thresh_pix: Real = 7.5,
rmsbox: Tuple[Integral] = (150, 50),
rmsbox_bright: Tuple[Integral] = (35, 7),
adaptive_rmsbox: bool = True,
adaptive_thresh: Real = 75.0,
filter_by_mask: bool = True,
keep_mask: bool = False,
output_catalog: str | Path | None = '',
output_flat_noise_rms: str | Path | None = '',
output_true_rms: str | Path | None = '',
ncores: int = 8,
) → int

Filters the input sky model using PyBDSF.

If no islands of emission are detected in the input image, a blank sky model is made. If any islands are detected in the input image, filtered true-sky and apparent-sky models are made, as well as a FITS clean mask (with the filename input_image+’.mask’).

Parameters:

  • flat_noise_image – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • true_sky_image – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • input_true_skymodel – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • input_apparent_skymodel – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • output_apparent_sky – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • output_true_sky – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • beam_ms – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • vertices_file – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • input_bright_skymodel – See lsmtool.filter_skymodel.filter_skymodel() for the meaning of the positional parameters.

  • thresh_isl (float, optional) – Value of thresh_isl PyBDSF parameter.

  • thresh_pix (float, optional) – Value of thresh_pix PyBDSF parameter.

  • rmsbox (tuple of float, optional) – Value of rms_box PyBDSF parameter.

  • rmsbox_bright (tuple of float, optional) – Value of rms_box_bright PyBDSF parameter.

  • adaptive_rmsbox (bool, optional) – Value of adaptive_rms_box PyBDSF parameter.

  • adaptive_thresh (float, optional) – If adaptive_rmsbox is True, this value sets the threshold above which a source will use the small rms box.

  • filter_by_mask (bool, optional) – If True, filter the input sky model by the PyBDSF-derived mask, removing sources that lie in unmasked regions.

  • keep_mask (bool, optional) – If True, keep the PyBDSF-derived mask file instead of deleting it.

  • output_catalog (str or pathlib.Path, optional) – The filename for source catalog. If not provided, do not create it.

  • output_flat_noise_rms (str or pathlib.Path, optional) – The filename for the flat noise root-mean-square (RMS) image. If not provied, do not create it. Creating this image may require an additional PyBDSF call and thereby slow down this function significantly.

  • output_true_rms (str or pathlib.Path, optional) – The filename for the true sky RMS image. If not provied, do not create it.

  • ncores (int) – Specify the number of cores that BDSF should use. Defaults to 8.

Raises:

FileNotFoundError – If none of the following input files is defined: input_true_skymodel, input_apparent_skymodel or input_bright_skymodel.

Returns:

n_sources (int) – The number of sources detected by pyBDSF.

lsmtool.filter_skymodel.bdsf.filter_sources(
img_true_sky: str | Path,
vertices_file: str | Path,
input_true_skymodel: str | Path | None,
input_apparent_skymodel: str | Path | None,
input_bright_skymodel: str | Path | None,
beam_ms: str | Path | None | Sequence[str | Path],
filter_by_mask: bool,
keep_mask: bool,
output_true_sky: str | Path | None,
output_apparent_sky: str | Path | None,
) → SkyModel

Filter and group sources based on a mask and other criteria.

This function filters the input sky model based on a mask generated from the true sky image. It also handles adding bright sources, removing negative components, and grouping sources by mask islands.

Parameters:

  • img_true_sky (bdsf.image.Image) – The PyBDSF image object.

  • vertices_file (str or pathlib.Path) – Filename of file with vertices, which determine the imaging field.

  • input_true_skymodel (str or pathlib.Path, optional) – Filename of input makesourcedb sky model with true fluxes.

  • input_apparent_skymodel (str or Path, optional) – Filename of input makesourcedb sky model with apparent fluxes.

  • input_bright_skymodel (str or pathlib.Path, optional) – Filename of input makesourcedb sky model of bright sources only.

  • beam_ms (str or pathlib.Path, optional) – The filename of the MS for deriving the beam attenuation.

  • filter_by_mask (bool, optional) – If True, filter the input sky model by the PyBDSF-derived mask.

  • keep_mask (bool, optional) – If True, keep the PyBDSF-derived mask file instead of deleting it.

  • output_true_sky (str or pathlib.Path) – Output file name for the generated true sky model.

  • output_apparent_sky (str or pathlib.Path) – Output file name for the generated apparent sky model.

lsmtool.filter_skymodel.bdsf.parse_rmsbox(rmsbox: str | None)

Parses the rmsbox parameter.

lsmtool.filter_skymodel.bdsf.process_images(
flat_noise_image: str | Path,
true_sky_image: str | Path,
beam_ms: str | Path | None | Sequence[str | Path],
output_catalog: str | Path | None,
output_flat_noise_rms: str | Path | None,
output_true_rms: str | Path | None,
**config,
)

Processes images using PyBDSF and generates output files.

This function runs PyBDSF on either the true sky image or the flat noise image, depending on whether a beam measurement set is provided. It then optionally generates a source catalog and RMS maps.

Parameters:

  • flat_noise_image (str or pathlib.Path) – Path to the flat noise image.

  • true_sky_image (str or pathlib.Path) – Path to the true sky image.

  • beam_ms (str or pathlib.Path or list of str or list of Path or None) – Path to the beam measurement set.

  • output_catalog (str or pathlib.Path, optional) – The filename for source catalog. If empty, do not create it.

  • output_flat_noise_rms (str or pathlib.Path, optional) – The filename for the flat noise RMS image. If empty, do not create it. Creating this image may require an additional PyBDSF call and thereby slow down this function significantly.

  • output_true_rms (str or pathlib.Path, optional) – The filename for the true sky RMS image. If empty, do not create it.

  • **config – Additional keyword arguments passed to the bdsf.process_image call.

lsmtool.filter_skymodel.bdsf.select_midpoint(beam_ms: Sequence[str | Path]) → str

Select the best measurement set for beam attenuation.

Selects the measurement set (MS) that is closest to the median time of all provided data. This is intended to choose a representative beam for attenuation calculations.

Parameters:

beam_ms (list of str) – List of measurement set filenames.

Returns:

str – The filename of the selected measurement set.

sofia

lsmtool.filter_skymodel.sofia.filter_skymodel(
flat_noise_image: str | Path,
true_sky_image: str | Path | None,
output_apparent_sky: str | Path,
output_true_sky: str | Path,
beam_ms: str | Path | None | Sequence[str | Path] = None,
output_dir: str | Path | None = None,
output_prefix: str = 'sofia',
ncores: Integral = 0,
**kws,
)

Filter the sources in a FITS image using SoFiA-2.

In order to measure the source parameters (size and orientation), SoFiA does a spatial moment analysis of the image data. This method of obtaining the source parameters is typically much faster fitting gaussians to the image data. It is advised that users assess the quality of the source parameterisation when choosing SoFia for filtering sources from the sky model.

Note: SoFiA-2 only supports input beam in FITS format (BDSF uses Measurement Set format).

Note: This function produces the following output files:

  • SoFiA source catalogue: “<output_prefix>_cat.xml”.

  • output_apparent_sky: Generated apparent sky model in CSV format, without primary-beam correction.

  • output_true_sky: Generated true sky model in CSV format, with primary-beam correction. Will be identical to the output_apparent_sky if no beam measurement set is provided.

Please refer to the SoFiA-2 documentation (pdf) for more information.

Parameters:

  • flat_noise_image (str or pathlib.Path) – Filename of input image to use to detect sources for filtering. It should be a flat-noise / apparent sky image (without primary-beam correction).

  • true_sky_image (str or pathlib.Path, optional) – Filename of input image to use to determine the true flux of sources. It should be a true flux image (with primary-beam correction). If beam_ms is not given or is None, this argument must be supplied. Otherwise, filter_skymodel ignores it and uses the flat_noise_image instead.

  • output_apparent_sky (str or pathlib.Path) – Output file name for the generated apparent sky model, without primary-beam correction.

  • output_true_sky (str or pathlib.Path) – Output file name for the generated true sky model, with primary-beam correction.

  • beam_ms (str or pathlib.Path, optional) – The filename of the MS for deriving the beam attenuation and theoretical image noise. If empty, the generated apparent and true sky models will be equal.

  • output_dir (str or pathlib.Path, optional) – Full path to the directory to which all output files will be written. If unset, the parent directory of the input data cube will be used by default.

  • output_prefix (str, optional) – File name prefix to be used as the template for all output files. For example, if output_prefix = my_data, then the output files will be named my_data_cat.xml, my_data_mom0.fits, etc. If unset, the name of the input data cube will be used as the file name template by default.

  • ncores (int, optional) – Specify the number of cores that SoFiA-2 should use. Defaults to 0, which will use the value of the environment variable OMP_NUM_THREADS.

  • **kws – Additional keyword-only arguments that will be passed to SoFiA (values should be strings). This can be used to overwrite any of the default source finding parameters for the SoFiA run.

lsmtool.filter_skymodel.sofia.get_corrected_gaussian_orientations(
image_header: Header,
catalog_table: Table,
) → ndarray

Corrects the orientation of the Gaussian sources (measured with respect to the image) to be with respect to the North Celestial Pole (NCP).

Applies a correction to the Gaussian orientations obtained from SoFiA-2 to ensure they are defined with respect to the North Celestial Pole, as required by the makesourcedb sky model format. SoFiA-2 outputs orientations relative to the vertical axis of the image, which is not necessarily aligned with the NCP.

Parameters:

  • image_header (astropy.io.fits.Header) – The FITS image header, used to extract the image center coordinates.

  • catalog_table (astropy.table.Table) – The SoFiA-2 source catalog, containing the source coordinates and orientations.

Returns:

orientations (numpy.ndarray) – The corrected Gaussian orientations in radians, defined with respect to the NCP.

lsmtool.filter_skymodel.sofia.get_source_fwhm(
image_header: Header,
catalog_table: Table,
orientations: Sequence,
) → ndarray

Compute the Full-Width Half-Maximum (FWHM) parameters for the detected sources in arcseconds.

SoFiA-2 computes the full major and minor axes of the source ellipses following approach of [Banks+]. This is equivalent to measuring twice the standard deviation along each axes of the Gaussian sources. The makesourcedb format expects the axes to be in units of the FWHM of the Gaussians. The conversion between the two parametrisations is handled by this function. In addition, converting from pixel values to arcseconds in the case of non-square pixels is also handled.

Parameters:
Returns:

fwhm (numpy.ndarray) – The ellipse semimajor and semiminor axes in arcseconds.

lsmtool.filter_skymodel.sofia.get_source_parameters(
image_header: Header,
catalog_table: Table,
) → dict

Get source parameters from SoFiA-2 catalog and image header.

Extracts source positions, axes, orientation, and fluxes from the SoFiA-2 catalog and image header. Orientations are corrected to be with respect to the North Celestial Pole (NCP).

Parameters:
Returns:

params (dict) – A dictionary containing the source parameters: RA strings, Dec strings, semimajor and semiminor axes in arcseconds, orientations wrt NCP in degrees, and fluxes in the image units.

lsmtool.filter_skymodel.sofia.rename_sources(catalog_table: Table)

Remove whitespace from source names (incompatible with makesourcedb format).

lsmtool.filter_skymodel.sofia.validate_catalog(
catalog_path: str | Path,
) → Table

Validate the source catalog generated by SoFiA-2.

lsmtool.filter_skymodel.sofia.validate_image(image_header: Header)

Check that the image units are as expected.

lsmtool.filter_skymodel.sofia.write_skymodel(
output_true_sky: str | Path,
catalog_table: Table,
source_parameters: tuple,
)

Writes the source catalog to a file, ensuring that it adheres to the makesourcedb format expected by downstream toolchain (wsclean, dp3).

Constructs an astropy Table with the necessary columns for the makesourcedb format and writes it to a CSV file. Then, it modifies the header of the CSV file to match the makesourcedb format specifications. Details of the source catalog format can be found at: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:makesourcedb

See also: https://wsclean.readthedocs.io/en/latest/component_list.html

Parameters:

  • output_true_sky (str or pathlib.Path) – Output file name for the skymodel.

  • catalog_table (astropy.table.Table) – The source catalog table.

  • source_parameters (tuple) – Tuple containing source parameters (RA, Dec, axes, orientation, flux).