A Rust library for common preprocessing tasks performed in the data pipeline of the Murchison Widefield Array (MWA), located on the land of the Wajarri Yamatji people in Murchison Shire, Western Australia.
Birl reads MWA correlator visibilities in the gpufits file format using mwalib, which supports the existing "legacy" MWA correlator, as well as the in-development "MWAX" correlator.
Birli is the Wajarri word for lightning, a common cause of outages at the MWA, and a great descriptor for the speed which this library intends to deliver.
for OS-specific instructions, check out the linux and macOS CI Scripts; the Makefile.toml; and the Dockerfile as these are tested regularly. The instructions below may be updated less frequently, but are better documented.
```bash
sudo apt install -y gcc libssl-dev pkg-config curl unzip wget
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -sSf | sh -s -- -y
cargo install --force cargo-make
export MAKEFLAGS="-j $MAKEFLAGS"
cargo make install_deps
export LDLIBRARYPATH="/usr/local/lib/:/usr/lib/x86_64-linux-gnu/" ```
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -sSf | sh -s -- -y
cargo install --force cargo-make
brew tap mwaTelescope/tap
brew cask install casacore-data casacore aoflagger erfa ```
Unfortunately most of the prerequisites aren't available on Windows. However, WSL is great, and there is a docker image! You could use VSCode remote for WSL or Docker. Your best best is Ubuntu LTS
bash
cargo --features=aoflagger install --path .
This creates a birli binary with the aoflagger feature enabled in $HOME/.cargo/bin
Having issues with Birli? run the test suite to narrow down your issue.
bash
cargo test
Experiencing segfaults? I can guarantee it's because of one of the C library dependencies. Make sure you have the right versions of all the libraries. These are specified in Prerequisites.
Get library versions on linux with:
bash
pkg-config --modversion cfitsio
pkg-config --modversion erfa
aoflagger --version
If you have something like CASA installed from apt, it's going to put an
ancient cfitsio library version in /usr/lib/x86_64-linux-gnu/, to get around
this, you must export LD_LIBRARY_PATH=/usr/local/lib/ in the shell so that Birli can find the correct library version.
You can enable additional logging on individual Rust modules by setting the RUST_LOG environment variable. For example:
bash
RUST_LOG=trace birli ... # set log level to trace for all module (including dependencies)
RUST_LOG=birli=debug birli ... # set log level to debug for birli only
RUST_LOG=birli::io=error birli ... # only show warnings for birli's io module
For more examples, see the env_logger docs
The default log level in info
Couldn't get it working on your environment? You can always run Birli in Docker
bash
docker run mwatelescope/birli:latest -h
Want to open a shell within a fully provisioned Birli development environment? Easy!
bash
docker run -it --entrypoint /bin/bash --volume $PWD:/app mwatelescope/birli:latest
Note: This mounts the current directory to /app in the Docker image, meaning both of these systems share the same
target folder. so if your host system is a different
architecture than Docker, you may need to cargo clean each time you switch between these environments. You may also want to temporarily disable any linters or language servers that use
```bash
module load singularity
singularity pull --dir . docker://mwatelescope/birli:latest
singularity exec /pawsey/mwa/singularity/birli/birlilatest.sif /app/target/release/birli ${YOURBIRLI_ARGS} ```
This will give you much more information about any problem you're having with Birli, however the debug build is not optimised, and is much slower.
```bash
salloc --partition workq --time 1:00:00 --nodes 1 -c 38 --mem=350G
module load singularity
singularity pull --dir . docker://mwatelescope/birli:debug
singularity exec /pawsey/mwa/singularity/birli/birli_debug.sif /bin/bash ```
then within this shell
```bash
export RUST_LOG=trace
gdb --args /app/target/debug/birli ${YOURBIRLIARGS}
```
birli -h
```txt
USAGE:
birli [OPTIONS] --metafits
OPTIONS:
--dry-run Just print the summary and exit
--emulate-cotter Use Cotter's array position, not MWAlib's
-h, --help Print help information
--no-cable-delay Do not perform cable length corrections
--no-geometric-delay Do not perform geometric corrections
--phase-centre
INPUT:
-m, --metafits
FLAGGING:
--flag-antennae
AVERAGING:
--avg-freq-factor
OUTPUT:
-f, --flag-template The template used to name flag files. Percents are substituted
for the zero-prefixed GPUBox ID, which can be up to 3
characters long. Example: FlagFile%%%.mwaf
-M, --ms-out
AOFLAGGER:
--aoflagger-strategy
Note: the aoflagged options are only available when the aoflagger feature is enabled.
Operations are performed in the order described by the following sections.
Cable delay correction involves adjusting visibility phases to correct for the differences in electrical length of the cable between each tile and it's receiver.
Legacy MWA correlator observations do not typically have cable delays applied, however MWAX observations can. The CABLEDEL key in the metafits describes what geometric delays have been applied.
By default, Birli will apply cable length corrections. You can use --no-cable-delay to disable this.
A baseline's cable lengths are determined by the difference between a baseline's rfInput electrical lengths, as specified the the TILEDATA HDU of the metafits. Complex visibilities are phase-shifted by an angle determined by the electrical length, and the channel's frequency.
rust
let angle = -2.0 * PI * electrical_length_m * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
By default, Birli will flag the data using the default MWA strategy in AOFlagger. You can use the
--no-rfi option to disable this, or the --aoflagger-strategy option to proived your own strategy
file.
Geometric correction involves adjusting visibility phases to correct for the differences in distance that light from the phase center has to travel to reach each tile.
Legacy MWA correlator observations are not typically phase tracked, however MWAX observations can have phase tracking applied. The GEODEL card in the metafits describes what geometric delays have been applied.
By default, Birli will apply geometric corrections at the phase center if they have not already been applied. It determines the observations phase center from the RAPHASE and DECPHASE cards in the metafits. If these are not available, the pointing center cards (RA and DEC) from the metafits are used. You can use --no-geometric-delay to disable geometric corrections, as well as the --phase-centre and --pointing-centre options to override the phase center.
A baseline's geometric length is determined by the w component of it's UVW fourier-space vector, after applying precession and nutation to it's tiles' positions and the phase center to the J2000 epoch, accounting for stellar aberration. Complex visibilities are phase-shifted by an angle determined by the w-component, and the channel's frequency.
rust
let angle = -2.0 * PI * uvw.w * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
The --emulate-cotter flag ensures that outputs match Cotter as much as possible. You should only use this flag if you need to perform a direct comparison with Cotter.
By default, Birli will use the MWA array position from MWALib in order to calculate UVWs and geometric corrections. This is more accurate than the one that Cotter uses, and is the main source of error when doing direct comparisons.
This flag is used as part of the tests in src/main.rs to validate that Birli's output matches that of Cotter to within an acceptable margin.
To average the data in time or frequency by a given whole number factor, you can provide the --avg-time-factor
or --avg-freq-factor options. This can also be achieved with the --avg-time-res and
--avg-freq-res options which take a duration [seconds] or ammount of bandwidth [kHz]
respectively. This second group of options will choose the closest whole number averaging factor
based on the resolution of the input data.
Birli can output visibility data to uvfits or measurement set with --ms-out (-M) or
--uvfits-out (-u). It can also output flags for each coarse channel in .mwaf format with
--flag-template (-f), where the % characters in the template argument are replaced with
the same zero-prefixed coarse channel identifiers that are used to identify the coarse channel
GPUBox files that the coarse channel data came from. For legacy data, use two percentage characters,
since the coarse channel identifier is the GPUBox number. However, for MWAX data, the coarse channel
identifier is the channel number, which needs three digits.
In this example, we use the aoflagger subcommand to:
-u)bash
birli \
-m tests/data/1254670392_avg/1254670392.metafits \
-f "/tmp/Flagfile.Birli.MWA.%%.mwaf" \
-u "/tmp/1254670392.birli.uvfits" \
--avg-time-res 4 --avg-freq-res 160 \
tests/data/1254670392_avg/1254670392_*gpubox*.fits
The equivalent Cotter commands would be:
```bash
cotter \ -m tests/data/1254670392avg/1254670392.metafits \ -o "tests/data/1254670392avg/1254670392.cotter.uvfits" \ -allowmissing \ -edgewidth 0 \ -endflag 0 \ -initflag 0 \ -noantennapruning \ -noflagautos \ -noflagdcchannels \ -nosbgains \ -sbpassband tests/data/subband-passband-128ch-unitary.txt \ -nostats \ -timeres 4 \ -freqres 160 \ -flag-strategy /usr/local/share/aoflagger/strategies/mwa-default.lua \ tests/data/1254670392avg/125467039220191009153257_gpubox*.fits ```
Pull requests are welcome! Please do your best to ensure that the high standards of test coverage are maintained.
Before each commit, use cargo make ci to ensure your code is formatted correctly.
This scientific work uses data obtained from the Murchison Radio-astronomy Observatory. We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.
This repo is approved by...
RELEASES.mdpackage.version in Cargo.tomlcargo make pre_commitgit tag -a $tag -m $taggit pushgit push --tags