A Rust library for common preprocessing tasks performed in the data pipeline of the Murchison Widefield Array (MWA), located on the land of the Wajarri Yamatji people in Murchison Shire, Western Australia.
Birl reads MWA correlator visibilities in the gpufits file format using mwalib, which supports the existing "legacy" MWA correlator, as well as the in-development "MWAX" correlator.
Birli is the Wajarri word for lightning, a common cause of outages at the MWA, and a great descriptor for the speed which this library intends to deliver.
for OS-specific instructions, check out the linux and macOS CI Scripts; the Makefile.toml; and the Dockerfile as these are tested regularly. The instructions below may be updated less frequently, but are better documented.
```bash
sudo apt install -y gcc libssl-dev pkg-config curl unzip wget
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -sSf | sh -s -- -y
cargo install --force cargo-make
export MAKEFLAGS="-j $MAKEFLAGS"
cargo make install_deps
export LDLIBRARYPATH="/usr/local/lib/:/usr/lib/x86_64-linux-gnu/" ```
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -sSf | sh -s -- -y
cargo install --force cargo-make
brew tap mwaTelescope/tap
brew cask install casacore-data casacore aoflagger erfa ```
Unfortunately most of the prerequisites aren't available on Windows. However, WSL is great, and there is a docker image! You could use VSCode remote for WSL or Docker. Your best best is Ubuntu LTS
bash
cargo --features=aoflagger install --path .
This creates a birli binary with the aoflagger feature enabled in $HOME/.cargo/bin
Having issues with Birli? run the test suite to narrow down your issue.
bash
cargo test
Experiencing segfaults? I can guarantee it's because of one of the C library dependencies. Make sure you have the right versions of all the libraries. These are specified in Prerequisites.
Get library versions on linux with:
bash
pkg-config --modversion cfitsio
pkg-config --modversion erfa
aoflagger --version
If you have something like CASA installed from apt, it's going to put an
ancient cfitsio library version in /usr/lib/x86_64-linux-gnu/, to get around
this, you must export LD_LIBRARY_PATH=/usr/local/lib/ in the shell so that Birli can find the correct library version.
You can enable additional logging on individual Rust modules by setting the RUST_LOG environment variable. For example:
bash
RUST_LOG=trace birli ... # set log level to trace for all module (including dependencies)
RUST_LOG=birli=debug birli ... # set log level to debug for birli only
RUST_LOG=birli::io=error birli ... # only show warnings for birli's io module
For more examples, see the env_logger docs
The default log level in info
Couldn't get it working on your environment? You can always run Birli in Docker
bash
docker run mwatelescope/birli:latest -h
Want to open a shell within a fully provisioned Birli development environment? Easy!
bash
docker run -it --entrypoint /bin/bash --volume $PWD:/app mwatelescope/birli:latest
Note: This mounts the current directory to /app in the Docker image, meaning both of these systems share the same
target folder. so if your host system is a different
architecture than Docker, you may need to cargo clean each time you switch between these environments. You may also want to temporarily disable any linters or language servers that use
```bash
module load singularity
singularity pull --dir . docker://mwatelescope/birli:latest
singularity exec /pawsey/mwa/singularity/birli/birlilatest.sif /app/target/release/birli ${YOURBIRLI_ARGS} ```
This will give you much more information about any problem you're having with Birli, however the debug build is not optimised, and is much slower.
```bash
salloc --partition workq --time 1:00:00 --nodes 1 -c 38 --mem=350G
module load singularity
singularity pull --dir . docker://mwatelescope/birli:debug
singularity exec /pawsey/mwa/singularity/birli/birli_debug.sif /bin/bash ```
then within this shell
```bash
export RUST_LOG=trace
gdb --args /app/target/debug/birli ${YOURBIRLIARGS}
```
Jones<f32>: AbsDiffEq<_> is not satisfiedif you see an error that looks like this:
txt
error[E0277]: the trait bound `Jones<f32>: AbsDiffEq<_>` is not satisfied
--> src/corrections.rs:1029:9
|
1029 | / assert_abs_diff_eq!(
1030 | | *jones_array.get((3, 3, 1)).unwrap(),
1031 | | &Jones::from([
1032 | | Complex::new(rot_1_xx_3_3_re, rot_1_xx_3_3_im),
... |
1036 | | ])
1037 | | );
| |__________^ the trait `AbsDiffEq<_>` is not implemented for `Jones<f32>`
|
= note: this error originates in the macro `abs_diff_eq` (in Nightly builds, run with -Z macro-backtrace for more info)
try
bash
cargo update
cargo update -p approx:0.5.0 --precise 0.4.0
birli -h
```txt USAGE: birli [SUBCOMMAND]
FLAGS: -h, --help Prints help information -V, --version Prints version information
SUBCOMMANDS: aoflagger flag visibilities with aoFlagger help Prints this message or the help of the given subcommand(s) ```
birli aoflagger -h
```txt flag visibilities with aoFlagger
USAGE:
birli aoflagger [FLAGS] [OPTIONS]
FLAGS: --emulate-cotter Use Cotter's value for array position instead of MWAlib for direct comparison with Cotter. -h, --help Prints help information --no-cable-delay Do not perform cable length corrections. --no-geometric-delay Do not perform geometric length corrections. --no-rfi Do not perform RFI Flagging. -V, --version Prints version information
OPTIONS:
-f
ARGS:
Cable delay correction involves adjusting visibility phases to correct for the differences in electrical length of the cable between each tile and it's receiver.
Legacy MWA correlator observations do not typically have cable delays applied, however MWAX observations can. The CABLEDEL key in the metafits describes what geometric delays have been applied.
By default, Birli will apply cable length corrections. You can use --no-cable-delay to disable this.
A baseline's cable lengths are determined by the difference between a baseline's rfInput electrical lengths, as specified the the TILEDATA HDU of the metafits. Complex visibilities are phase-shifted by an angle determined by the electrical length, and the channel's frequency.
rust
let angle = -2.0 * PI * electrical_length_m * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
Geometric correction involves adjusting visibility phases to correct for the differences in distance that light from the phase center has to travel to reach each tile.
Legacy MWA correlator observations are not typically phase tracked, however MWAX observations can have phase tracking applied. The GEODEL card in the metafits describes what geometric delays have been applied.
By default, Birli will apply geometric corrections at the phase center if they have not already been applied. It determines the observations phase center from the RAPHASE and DECPHASE cards in the metafits. If these are not available, the pointing center cards (RA and DEC) from the metafits are used. You can use --no-geometric-delay to disable this.
A baseline's geometric length is determined by the w component of it's UVW fourier-space vector, after applying precession and nutation to it's tiles' positions and the phase center to the J2000 epoch, accounting for stellar aberration. Complex visibilities are phase-shifted by an angle determined by the w-component, and the channel's frequency.
rust
let angle = -2.0 * PI * uvw.w * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
The --emulate-cotter flag ensures that outputs match Cotter as much as possible. You should only use this flag if you need to perform a direct comparison with Cotter.
By default, Birli will use the MWA array position from MWALib in order to calculate UVWs and geometric corrections. This is more accurate than the one that Cotter uses, and is the main source of error when doing direct comparisons.
This flag is used as part of the tests in src/main.rs to validate that Birli's output matches that of Cotter to within an acceptable margin.
In this example, we use the aoflagger subcommand to:
-f)-u)bash
birli aoflagger \
-m tests/data/1254670392_avg/1254670392.metafits \
-u "tests/data/1254670392_avg/Flagfile.Birli.MWA.%%.mwaf" \
-f "tests/data/1254670392_avg/1254670392.birli.uvfits" \
tests/data/1254670392_avg/1254670392_20191009153257_gpubox*.fits
Since Cotter can't output flags and uvfits at the same time, the equivalent Cotter commands would be:
```bash
cotter \ -m tests/data/1254670392avg/1254670392.metafits \ -o "tests/data/1247842824flags/FlagfileCotterMWA%%.mwaf" \ -allowmissing \ -edgewidth 0 \ -endflag 0 \ -initflag 0 \ -noantennapruning \ -noflagautos \ -noflagdcchannels \ -nosbgains \ -sbpassband tests/data/subband-passband-128ch-unitary.txt \ -nostats \ -flag-strategy /usr/local/share/aoflagger/strategies/mwa-default.lua \ tests/data/1254670392avg/125467039220191009153257_gpubox*.fits
cotter \ -m tests/data/1254670392avg/1254670392.metafits \ -o "tests/data/1254670392avg/1254670392.cotter.uvfits" \ -allowmissing \ -edgewidth 0 \ -endflag 0 \ -initflag 0 \ -noantennapruning \ -noflagautos \ -noflagdcchannels \ -nosbgains \ -sbpassband tests/data/subband-passband-128ch-unitary.txt \ -nostats \ -flag-strategy /usr/local/share/aoflagger/strategies/mwa-default.lua \ tests/data/1254670392avg/125467039220191009153257_gpubox*.fits ```
Pull requests are welcome! Please do your best to ensure that the high standards of test coverage are maintained.
Before each commit, use cargo make ci to ensure your code is formatted correctly.
This scientific work uses data obtained from the Murchison Radio-astronomy Observatory. We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.
This repo is approved by...
RELEASES.mdpackage.version in Cargo.toml git tag -a $tag -m $taggit pushgit push --tags