mwalib

MWA library to read raw visibilities and metadata into a common structure. mwalib supports the existing "legacy" MWA correlator, as well as the in-development "MWAX" correlator. This library strives to provide a single interface to work with all incarnations of MWA correlator formats and abstract away the nitty gritty details about reading MWA data.

Usage via C

In the examples directory, see build_c_examples.sh, mwalib-print-obs-context.c, and mwalib-sum-all-hdus.c for examples of usage.

Usage via Python

The primary Python interface to mwalib is pymwalib.

There are also a couple of simple examples Python scripts here that use mwalib's ffi interface instead; see examples/mwalib-print-obs-context.py, and examples/mwalib-sum-all-hdus.py.

Usage in Rust

In the examples directory, see mwalib-print-obs-context.rs and mwalib-print-obs-context.rs for examples. Also run cargo doc --open to see the rendered documentation.

Usage approach

Populate a mwalibContext struct

This struct contains only information on the metafits file and gpubox files associated with an observation. During creation, a number of checks are performed on the MWA data (metadata and gpubox files) to ensure consistency. Once created, the observation is primed for reading data.
Read raw data

The read_by_baseline function associated with mwalibContext takes in a timestep index (see: context.timesteps vector) and a coarse channel index (see: context.coarse_channels vector) and will return a vector of 32bit floats. The data is organised as [baseline][fine_channel][polarisation][r][i].

The read_by_frequency function is similar to read_by_baseline but outputs data in [fine_channel][baseline][polarisation][r][i] order.

Concepts

gpubox "batches"

gpubox batches refers to the different gpubox outputs for the same course-band channel. e.g. 1065880128_20131015134830_gpubox01_00.fits belongs to "batch 0", whereas 1065880128_20131015134930_gpubox01_01.fits belongs to "batch 1".
"scans"

A scan is the raw data from all gpubox files for a single time integration.
baselines

A baseline is the distance between any two antennas. However, raw MWA data also contains auto-correlation data, and so the number of "baselines" presented in the data is calculated as n/2 * (n+1) (where n is the number of antennas).

Building From Source

You can build mwalib from source: - Install rust

`https://www.rust-lang.org/tools/install`

Install cfitsio If using a package manager install the cfitsio-dev package. If building from source ensure you build it with with --enable-reentrant option. See: https://heasarc.gsfc.nasa.gov/fitsio/
Compile the source

cargo build --release
Statically-linking cfitsio If any of the MWALIBLINKSTATICCFITSIO, STATICCFITSIO or PKGCONFIGALL_STATIC environment variables exist and are set to a non-zero value, rustc will statically link libcfitsio. The default is to dynamically link libcfitsio. This is an attempt to give developers the choice of having a static or dynamic link of the fits library to ease the build and deploy process.
Use the dynamic-shared and/or static objects in the target/release directory

e.g. on linux, libmwalib.a or libmwalib.so

These have silly names because of how C historically links libraries. It's overall clearer to link "mwalib" than "mwa", so it is left like this.

For an example of using mwalib with C or python, look in the examples directory.
(Optional) If you are modifying the ffi functions, you'll need to regenerate mwalib.h

cbindgen -l c <path/to/mwalib/repo> > mwalib.h

Install cbindgen with your package manager or via cargo with cargo install cbindgen.

Installation

As an alternative to building from source, we produce github releases whenever features or bug fixes are completed as tarballs. In the release you will find everything you need to use mwalib from C/C++/Python or any other language that can utilise shared libraries: * lib/libmwalib.a (Statically compiled library) * lib/libmwalib.so (Dynamic library) * include/mwalib.h (C Header file) * CHANGELOG.md (Change log for this and previous relases) * LICENSE (License for using mwalib in your projects) * LICENSE-cfitsio (Since libcfitsio is statically compiled into our static and dynamic libraries, we also include it's license)

NOTE: from release 0.3.2 onwards, libcfitsio is statically linked to mwalib in order to reduce issues with conflicting/incompatible versions of cfitsio. Therefore, there is no need for you to have cfitsio installed on your machine.

To install on a regular linux x86/64 distribution, the following would be all that is needed: - Download release from mwalib github releases. wget "https://github.com/MWATelescope/mwalib/releases/download/v0.3.1/libmwalib-0.3.1-linux_x86_64.tar.gz" -O mwalib.tar.gz

Untar the tarball mkdir mwalib; tar xvf mwalib.tar.gz -C mwalib
Install sudo cp mwalib/lib/libmwalib.* /usr/local/lib sudo cp mwalib/include/libmwalib.h /usr/local/include
Register the library with ldconfig sudo ldconfig

Consistency checks

(TODO: This is non-exhaustive!)

mwalib checks input files for the presence of:

a metafits file,
gpubox files,
"GPSTIME", "NINPUTS", "CHANNELS", "NCHANS" within the metafits file,
"TIME", "MILLITIM", "NAXIS2" within the gpubox files.
Consistent number of gpubox files in each batch

i.e. if 1065880128_20131015134830_gpubox01_00.fits exists, and there are two batches, then 1065880128_20131015134930_gpubox01_01.fits also exists, and likewise for all other coarse-band channels.
Consistent "batch number format"

i.e. a gpubox file 1065880128_20131015134830_gpubox01_00.fits can not be used alongside another file like 1065880128_20131015134830_gpubox01.fits.

Example test output

``` mwalibContext ( Correlator version: v1 Legacy,

MWA latitude:             -26.703319405555554 degrees,
MWA longitude:            116.67081523611111 degrees
MWA altitude:             377.827 m,

obsid:                    1065880128,

Creator:                  DJacobs,
Project ID:               G0009,
Observation Name:         high_season1_2456581,
Receivers:                [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
Delays:                   [0, 2, 4, 6, 0, 2, 4, 6, 0, 2, 4, 6, 0, 2, 4, 6],
Global attenuation:       1 dB,

Scheduled start (UNIX)    1381844912,
Scheduled end (UNIX)      1381845024,
Scheduled start (GPS)     1065880128,
Scheduled end (GPS)       1065880240,
Scheduled start (utc)     2013-10-15 13:48:32 +00:00,
Scheduled end (utc)       2013-10-15 13:50:24 +00:00,
Scheduled start (MJD)     56580.57537037037,
Scheduled end (MJD)       56580.57666666666,
Scheduled duration        112 s,
Actual UNIX start time:   1381844910,
Actual UNIX end time:     1381845019,
Actual duration:          109 s,
Quack time:               0.5 s,
Good UNIX start time:     1381844912.5,

R.A. (tile_pointing):     3.166573685252408 degrees,
Dec. (tile_pointing):     -25.96194063106248 degrees,
R.A. (phase center):      0 degrees,
Dec. (phase center):      -27 degrees,
Azimuth:                  90 degrees,
Altitude:                 76.2838 degrees,
Sun altitude:             -42.5092395827051 degrees,
Sun distance:             141.518031617465 degrees,
Moon distance:            32.1177391815099 degrees,
Jupiter distance:         114.851619013396 degrees,
LST:                      348.061732169994 degrees,
Hour angle:               -22:58:52.56 degrees,
Grid name:                EOR1,
Grid number:              3,

num timesteps:            218,
timesteps:                [unix=1381844910.000, unix=1381844910.500, unix=1381844911.000, unix=1381844911.500, unix=1381844912.000, unix=1381844912.500, unix=1381844913.000, unix=1381844913.500, unix=1381844914.000, unix=1381844914.500, unix=1381844915.000, unix=1381844915.500, unix=1381844916.000, unix=1381844916.500, unix=1381844917.000, unix=1381844917.500, unix=1381844918.000, unix=1381844918.500, unix=1381844919.000, unix=1381844919.500, unix=1381844920.000, unix=1381844920.500, unix=1381844921.000, unix=1381844921.500, unix=1381844922.000, unix=1381844922.500, unix=1381844923.000, unix=1381844923.500, unix=1381844924.000, unix=1381844924.500, unix=1381844925.000, unix=1381844925.500, unix=1381844926.000, unix=1381844926.500, unix=1381844927.000, unix=1381844927.500, unix=1381844928.000, unix=1381844928.500, unix=1381844929.000, unix=1381844929.500, unix=1381844930.000, unix=1381844930.500, unix=1381844931.000, unix=1381844931.500, unix=1381844932.000, unix=1381844932.500, unix=1381844933.000, unix=1381844933.500, unix=1381844934.000, unix=1381844934.500, unix=1381844935.000, unix=1381844935.500, unix=1381844936.000, unix=1381844936.500, unix=1381844937.000, unix=1381844937.500, unix=1381844938.000, unix=1381844938.500, unix=1381844939.000, unix=1381844939.500, unix=1381844940.000, unix=1381844940.500, unix=1381844941.000, unix=1381844941.500, unix=1381844942.000, unix=1381844942.500, unix=1381844943.000, unix=1381844943.500, unix=1381844944.000, unix=1381844944.500, unix=1381844945.000, unix=1381844945.500, unix=1381844946.000, unix=1381844946.500, unix=1381844947.000, unix=1381844947.500, unix=1381844948.000, unix=1381844948.500, unix=1381844949.000, unix=1381844949.500, unix=1381844950.000, unix=1381844950.500, unix=1381844951.000, unix=1381844951.500, unix=1381844952.000, unix=1381844952.500, unix=1381844953.000, unix=1381844953.500, unix=1381844954.000, unix=1381844954.500, unix=1381844955.000, unix=1381844955.500, unix=1381844956.000, unix=1381844956.500, unix=1381844957.000, unix=1381844957.500, unix=1381844958.000, unix=1381844958.500, unix=1381844959.000, unix=1381844959.500, unix=1381844960.000, unix=1381844960.500, unix=1381844961.000, unix=1381844961.500, unix=1381844962.000, unix=1381844962.500, unix=1381844963.000, unix=1381844963.500, unix=1381844964.000, unix=1381844964.500, unix=1381844965.000, unix=1381844965.500, unix=1381844966.000, unix=1381844966.500, unix=1381844967.000, unix=1381844967.500, unix=1381844968.000, unix=1381844968.500, unix=1381844969.000, unix=1381844969.500, unix=1381844970.000, unix=1381844970.500, unix=1381844971.000, unix=1381844971.500, unix=1381844972.000, unix=1381844972.500, unix=1381844973.000, unix=1381844973.500, unix=1381844974.000, unix=1381844974.500, unix=1381844975.000, unix=1381844975.500, unix=1381844976.000, unix=1381844976.500, unix=1381844977.000, unix=1381844977.500, unix=1381844978.000, unix=1381844978.500, unix=1381844979.000, unix=1381844979.500, unix=1381844980.000, unix=1381844980.500, unix=1381844981.000, unix=1381844981.500, unix=1381844982.000, unix=1381844982.500, unix=1381844983.000, unix=1381844983.500, unix=1381844984.000, unix=1381844984.500, unix=1381844985.000, unix=1381844985.500, unix=1381844986.000, unix=1381844986.500, unix=1381844987.000, unix=1381844987.500, unix=1381844988.000, unix=1381844988.500, unix=1381844989.000, unix=1381844989.500, unix=1381844990.000, unix=1381844990.500, unix=1381844991.000, unix=1381844991.500, unix=1381844992.000, unix=1381844992.500, unix=1381844993.000, unix=1381844993.500, unix=1381844994.000, unix=1381844994.500, unix=1381844995.000, unix=1381844995.500, unix=1381844996.000, unix=1381844996.500, unix=1381844997.000, unix=1381844997.500, unix=1381844998.000, unix=1381844998.500, unix=1381844999.000, unix=1381844999.500, unix=1381845000.000, unix=1381845000.500, unix=1381845001.000, unix=1381845001.500, unix=1381845002.000, unix=1381845002.500, unix=1381845003.000, unix=1381845003.500, unix=1381845004.000, unix=1381845004.500, unix=1381845005.000, unix=1381845005.500, unix=1381845006.000, unix=1381845006.500, unix=1381845007.000, unix=1381845007.500, unix=1381845008.000, unix=1381845008.500, unix=1381845009.000, unix=1381845009.500, unix=1381845010.000, unix=1381845010.500, unix=1381845011.000, unix=1381845011.500, unix=1381845012.000, unix=1381845012.500, unix=1381845013.000, unix=1381845013.500, unix=1381845014.000, unix=1381845014.500, unix=1381845015.000, unix=1381845015.500, unix=1381845016.000, unix=1381845016.500, unix=1381845017.000, unix=1381845017.500, unix=1381845018.000, unix=1381845018.500],

num antennas:             128,
antennas:                 [Tile011, Tile012, Tile013, Tile014, Tile015, Tile016, Tile017, Tile018, Tile021, Tile022, Tile023, Tile024, Tile025, Tile026, Tile027, Tile028, Tile031, Tile032, Tile033, Tile034, Tile035, Tile036, Tile037, Tile038, Tile041, Tile042, Tile043, Tile044, Tile045, Tile046, Tile047, Tile048, Tile051, Tile052, Tile053, Tile054, Tile055, Tile056, Tile057, Tile058, Tile061, Tile062, Tile063, Tile064, Tile065, Tile066, Tile067, Tile068, Tile071, Tile072, Tile073, Tile074, Tile075, Tile076, Tile077, Tile078, Tile081, Tile082, Tile083, Tile084, Tile085, Tile086, Tile087, Tile088, Tile091, Tile092, Tile093, Tile094, Tile095, Tile096, Tile097, Tile098, Tile101, Tile102, Tile103, Tile104, Tile105, Tile106, Tile107, Tile108, Tile111, Tile112, Tile113, Tile114, Tile115, Tile116, Tile117, Tile118, Tile121, Tile122, Tile123, Tile124, Tile125, Tile126, Tile127, Tile128, Tile131, Tile132, Tile133, Tile134, Tile135, Tile136, Tile137, Tile138, Tile141, Tile142, Tile143, Tile144, Tile145, Tile146, Tile147, Tile148, Tile151, Tile152, Tile153, Tile154, Tile155, Tile156, Tile157, Tile158, Tile161, Tile162, Tile163, Tile164, Tile165, Tile166, Tile167, Tile168],
rf_inputs:                [Tile011X, Tile011Y, Tile012X, Tile012Y, Tile013X, Tile013Y, Tile014X, Tile014Y, Tile015X, Tile015Y, Tile016X, Tile016Y, Tile017X, Tile017Y, Tile018X, Tile018Y, Tile021X, Tile021Y, Tile022X, Tile022Y, Tile023X, Tile023Y, Tile024X, Tile024Y, Tile025X, Tile025Y, Tile026X, Tile026Y, Tile027X, Tile027Y, Tile028X, Tile028Y, Tile031X, Tile031Y, Tile032X, Tile032Y, Tile033X, Tile033Y, Tile034X, Tile034Y, Tile035X, Tile035Y, Tile036X, Tile036Y, Tile037X, Tile037Y, Tile038X, Tile038Y, Tile041X, Tile041Y, Tile042X, Tile042Y, Tile043X, Tile043Y, Tile044X, Tile044Y, Tile045X, Tile045Y, Tile046X, Tile046Y, Tile047X, Tile047Y, Tile048X, Tile048Y, Tile051X, Tile051Y, Tile052X, Tile052Y, Tile053X, Tile053Y, Tile054X, Tile054Y, Tile055X, Tile055Y, Tile056X, Tile056Y, Tile057X, Tile057Y, Tile058X, Tile058Y, Tile061X, Tile061Y, Tile062X, Tile062Y, Tile063X, Tile063Y, Tile064X, Tile064Y, Tile065X, Tile065Y, Tile066X, Tile066Y, Tile067X, Tile067Y, Tile068X, Tile068Y, Tile071X, Tile071Y, Tile072X, Tile072Y, Tile073X, Tile073Y, Tile074X, Tile074Y, Tile075X, Tile075Y, Tile076X, Tile076Y, Tile077X, Tile077Y, Tile078X, Tile078Y, Tile081X, Tile081Y, Tile082X, Tile082Y, Tile083X, Tile083Y, Tile084X, Tile084Y, Tile085X, Tile085Y, Tile086X, Tile086Y, Tile087X, Tile087Y, Tile088X, Tile088Y, Tile091X, Tile091Y, Tile092X, Tile092Y, Tile093X, Tile093Y, Tile094X, Tile094Y, Tile095X, Tile095Y, Tile096X, Tile096Y, Tile097X, Tile097Y, Tile098X, Tile098Y, Tile101X, Tile101Y, Tile102X, Tile102Y, Tile103X, Tile103Y, Tile104X, Tile104Y, Tile105X, Tile105Y, Tile106X, Tile106Y, Tile107X, Tile107Y, Tile108X, Tile108Y, Tile111X, Tile111Y, Tile112X, Tile112Y, Tile113X, Tile113Y, Tile114X, Tile114Y, Tile115X, Tile115Y, Tile116X, Tile116Y, Tile117X, Tile117Y, Tile118X, Tile118Y, Tile121X, Tile121Y, Tile122X, Tile122Y, Tile123X, Tile123Y, Tile124X, Tile124Y, Tile125X, Tile125Y, Tile126X, Tile126Y, Tile127X, Tile127Y, Tile128X, Tile128Y, Tile131X, Tile131Y, Tile132X, Tile132Y, Tile133X, Tile133Y, Tile134X, Tile134Y, Tile135X, Tile135Y, Tile136X, Tile136Y, Tile137X, Tile137Y, Tile138X, Tile138Y, Tile141X, Tile141Y, Tile142X, Tile142Y, Tile143X, Tile143Y, Tile144X, Tile144Y, Tile145X, Tile145Y, Tile146X, Tile146Y, Tile147X, Tile147Y, Tile148X, Tile148Y, Tile151X, Tile151Y, Tile152X, Tile152Y, Tile153X, Tile153Y, Tile154X, Tile154Y, Tile155X, Tile155Y, Tile156X, Tile156Y, Tile157X, Tile157Y, Tile158X, Tile158Y, Tile161X, Tile161Y, Tile162X, Tile162Y, Tile163X, Tile163Y, Tile164X, Tile164Y, Tile165X, Tile165Y, Tile166X, Tile166Y, Tile167X, Tile167Y, Tile168X, Tile168Y],

num baselines:            8256,
baselines:                0 v 0 to 127 v 127
num auto-correlations:    128,
num cross-correlations:   8128,

num antenna pols:         2,
num visibility pols:      4,
visibility pols:          XX, XY, YX, YY,

observation bandwidth:    30.72 MHz,
num coarse channels,      24,
coarse channels:          [gpu=24 corr=23 rec=131 @ 167.680 MHz, gpu=23 corr=22 rec=132 @ 168.960 MHz, gpu=22 corr=21 rec=133 @ 170.240 MHz, gpu=21 corr=20 rec=134 @ 171.520 MHz, gpu=20 corr=19 rec=135 @ 172.800 MHz, gpu=19 corr=18 rec=136 @ 174.080 MHz, gpu=18 corr=17 rec=137 @ 175.360 MHz, gpu=17 corr=16 rec=138 @ 176.640 MHz, gpu=16 corr=15 rec=139 @ 177.920 MHz, gpu=15 corr=14 rec=140 @ 179.200 MHz, gpu=14 corr=13 rec=141 @ 180.480 MHz, gpu=13 corr=12 rec=142 @ 181.760 MHz, gpu=12 corr=11 rec=143 @ 183.040 MHz, gpu=11 corr=10 rec=144 @ 184.320 MHz, gpu=10 corr=9 rec=145 @ 185.600 MHz, gpu=9 corr=8 rec=146 @ 186.880 MHz, gpu=8 corr=7 rec=147 @ 188.160 MHz, gpu=7 corr=6 rec=148 @ 189.440 MHz, gpu=6 corr=5 rec=149 @ 190.720 MHz, gpu=5 corr=4 rec=150 @ 192.000 MHz, gpu=4 corr=3 rec=151 @ 193.280 MHz, gpu=3 corr=2 rec=152 @ 194.560 MHz, gpu=2 corr=1 rec=153 @ 195.840 MHz, gpu=1 corr=0 rec=154 @ 197.120 MHz],

Correlator Mode:
Mode:                     HW_LFILES,
fine channel resolution:  40 kHz,
integration time:         0.50 s
num fine channels/coarse: 32,

gpubox HDU size:          8.0625 MiB,
Memory usage per scan:    387 MiB,

metafits filename:        1065880128.metafits,
gpubox batches:           [
batch_number=0 gpubox_files=[filename=1065880128_20131015134830_gpubox01_00.fits channelidentifier=1, filename=1065880128_20131015134830_gpubox02_00.fits channelidentifier=2, filename=1065880128_20131015134830_gpubox03_00.fits channelidentifier=3, filename=1065880128_20131015134830_gpubox04_00.fits channelidentifier=4, filename=1065880128_20131015134830_gpubox05_00.fits channelidentifier=5, filename=1065880128_20131015134830_gpubox06_00.fits channelidentifier=6, filename=1065880128_20131015134830_gpubox07_00.fits channelidentifier=7, filename=1065880128_20131015134830_gpubox08_00.fits channelidentifier=8, filename=1065880128_20131015134830_gpubox09_00.fits channelidentifier=9, filename=1065880128_20131015134830_gpubox10_00.fits channelidentifier=10, filename=1065880128_20131015134830_gpubox11_00.fits channelidentifier=11, filename=1065880128_20131015134830_gpubox12_00.fits channelidentifier=12, filename=1065880128_20131015134830_gpubox13_00.fits channelidentifier=13, filename=1065880128_20131015134830_gpubox14_00.fits channelidentifier=14, filename=1065880128_20131015134830_gpubox15_00.fits channelidentifier=15, filename=1065880128_20131015134830_gpubox16_00.fits channelidentifier=16, filename=1065880128_20131015134830_gpubox17_00.fits channelidentifier=17, filename=1065880128_20131015134830_gpubox18_00.fits channelidentifier=18, filename=1065880128_20131015134830_gpubox19_00.fits channelidentifier=19, filename=1065880128_20131015134830_gpubox20_00.fits channelidentifier=20, filename=1065880128_20131015134830_gpubox21_00.fits channelidentifier=21, filename=1065880128_20131015134830_gpubox22_00.fits channelidentifier=22, filename=1065880128_20131015134830_gpubox23_00.fits channelidentifier=23, filename=1065880128_20131015134830_gpubox24_00.fits channelidentifier=24],
batch_number=1 gpubox_files=[filename=1065880128_20131015134930_gpubox01_01.fits channelidentifier=1, filename=1065880128_20131015134930_gpubox02_01.fits channelidentifier=2, filename=1065880128_20131015134930_gpubox03_01.fits channelidentifier=3, filename=1065880128_20131015134930_gpubox04_01.fits channelidentifier=4, filename=1065880128_20131015134930_gpubox05_01.fits channelidentifier=5, filename=1065880128_20131015134930_gpubox06_01.fits channelidentifier=6, filename=1065880128_20131015134930_gpubox07_01.fits channelidentifier=7, filename=1065880128_20131015134930_gpubox08_01.fits channelidentifier=8, filename=1065880128_20131015134930_gpubox09_01.fits channelidentifier=9, filename=1065880128_20131015134930_gpubox10_01.fits channelidentifier=10, filename=1065880128_20131015134930_gpubox11_01.fits channelidentifier=11, filename=1065880128_20131015134930_gpubox12_01.fits channelidentifier=12, filename=1065880128_20131015134930_gpubox13_01.fits channelidentifier=13, filename=1065880128_20131015134930_gpubox14_01.fits channelidentifier=14, filename=1065880128_20131015134930_gpubox15_01.fits channelidentifier=15, filename=1065880128_20131015134930_gpubox16_01.fits channelidentifier=16, filename=1065880128_20131015134930_gpubox17_01.fits channelidentifier=17, filename=1065880128_20131015134930_gpubox18_01.fits channelidentifier=18, filename=1065880128_20131015134930_gpubox19_01.fits channelidentifier=19, filename=1065880128_20131015134930_gpubox20_01.fits channelidentifier=20, filename=1065880128_20131015134930_gpubox21_01.fits channelidentifier=21, filename=1065880128_20131015134930_gpubox22_01.fits channelidentifier=22, filename=1065880128_20131015134930_gpubox23_01.fits channelidentifier=23, filename=1065880128_20131015134930_gpubox24_01.fits channelidentifier=24],

], ) ```