Classification of astrophysical events with ACAI
Astronomy has been experiencing an explosive increase in the data volumes, doubling every two years or so. At the forefront of this revolution, the Zwicky Transient Facility (ZTF) – a robotic sky survey – registers millions of transient events (supernova explosions, asteroids, variable stars changing their brightness etc.) in the dynamic sky every (clear) night.
Here we present astrophysical Alert-Classifying AI (ACAI), an open-source DL framework for the phenomenological classification of ZTF events.
Created on November 5|Last edited on March 15
Comment
ACAI: Alert-Classifying AI
The Zwicky Transient Facility (ZTF) is a state-of-the-art robotic sky survey capable of visiting the entire Northern sky visible from the Palomar Mountain in Southern California every night.
The problem: too many event alerts (~1M/night), too few graduate students.
The solution: build phenomenological classifiers that could be tailored to specific scientific needs.
Presented at the Machine Learning and the Physical Sciences Workshop at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
ZTF
System architecture
Alert-Classifying Artificial Intelligence (ACAI) is an open-source deep-learning framework for the classification of the ZTF alerts. ACAI uses a set of simple independent binary classifiers to categorize events into five phenomenological classes:
- "Hosted" (acai_h) -- genuine transients in the vicinity of a "host" with detectable morphology. Objects such as supernovae (SNe), novae, or cataclysmic variables occurring near galaxy-like objects get high acai_h scores.
- "Orphan" (acai_o) -- genuine orphan transients, i.e. when there are no identifiable "hosts" in their vicinity. This category catches asteroids and host-less (or with hosts that are too faint) transients.
- "Nuclear" (acai_n) -- genuine transients occurring in the galaxy/quasar nucleus. Active galactic nuclei (AGN) and tidal disruption events (TDEs) fall into this category.
- "Variable star" (acai_v) -- genuine brightness changes of variable stars detectable in the reference image or in a reference catalog.
- "Bogus" (acai_b) -- bogus events such as cosmic ray hits, optical reflections, or data processing artifacts, get high acai_b scores.
This report specifically focuses on the acai_h classifier.
Advantages of the approach
Dataset
We have assembled a large training data set consisting of over 200,000 individual alerts (~10GB of JSON files containing, in particular, compressed raw image cutouts) and labeled them -- about 70k alerts from known variable stars of different types, 35k from nuclear transients originating from known AGN and TDEs, 30k "hosted" transients from known supernovae and novae, 50k bogus alerts, and 30k orphan events, mostly from known Solar system objects.
We employed an active-learning-like approach of alternating between data labeling and classifier training with subsequent sampling of their predictions, both confident ones and, more importantly, those near the decision boundary and with the highest loss.
"Mental hyper-parameter tuning"
The idea: use all information available about an event: features (tabular data) + images.
As a baseline, we used a simple architecture: an MLP with 25 input features (info about the event, parameters of the detection, the vicinity/surroundings) + a ConvNet on image cutouts (a stacked triplet of an image of the sky from the night of detection, a reference image of the same patch of the sky, and their difference):
Hyper-parameter optimization
We used W&B Sweeps to perform hyper-parameter optimization using a subset (~10%) of the full dataset. We tuned hyper-parameters such as the number of fully-connected layers and neurons therein, number of convolutional filters and their sizes and types (regular or separable convolution), flattening versus global average pooling, dropout rates, activation functions, and the initial learning rate for the Adam optimizer. We also experimented with the inclusion/exclusion of particular alert features in/from the input.
Sweep: w7wy00j7 1
24
We used grid search and tracked validation loss as the optimization target.
Sweep: w7wy00j7
24
Best-performing model
The best-performing model (let's call it "BPM") architecture wanted the max number of neurons in fully-connected parts of the network and the max number of convolutional filters, resulting in about 70k trainable parameters (which is about 10x larger than the "mentally-tuned" model). We trained it on the full dataset for ~9 hours using a Tesla P100 16G GPU we happen to have access to. The performance of the BPM model (run.name: acai_h-20211105_065027) evaluated on a hold-out test set (~10% of the data; see the Table below), compared to the "mentally-tuned" model (let's call it "MTM", run.name: acai_h-20211105_182833):
- Accuracy: 97.9% BPM vs 97.8% MTM
- Precision: 96.0% BPM vs 95.9% MTM
- Recall: 96.8% BPM vs 96.4% MTM
- F1 score: 96.4% BPM vs 96.1% MTM
Run set
2
Notes:
- The angular size of the images in the table below (~60 arcsec) is ~3% of that of the Moon (~0.5 deg).
- The plot below in the middle shows the predicted label of an event as a function of the sky position. You can clearly see the Galactic plane - most of the (labeled) objects there are stars (so they get low acai_h scores), because it is quite hard to detect and accurately measure supernovae and even harder to see their host galaxies looking through our Galaxy due to extinction. Thanks to Danny Goldstein for the idea to show this plot here!
- The plot below at the bottom shows misclassified events as a function of the sky position.
Run set
32
Performance on a live data stream
The best-performing models were deployed to Fritz, an open-source data platform and alert broker used by the ZTF collaboration (consisting of hundreds of astronomers worldwide). Running in production on a live (nightly) stream of event alerts (~1M/night).
Evaluated on ~110M alerts generated since Jan 2021 using a filter:
acai_h > 0.8 and max(acai_b, acai_n, acai_v, acai_o) < 0.1 and n_detections < 5 and is_positive_subtraction and not is_known_solar_system_object
12k objects filtered out, 52 are bogus (0.4% FPR). State of the art! ;)
Examples of objects detected on Nov 5, 2021:
Example extragalactic supernovae that went off a couple months ago and detected on Nov 5, 2021.
Example extragalactic supernovae that went off on Nov 5, 2021 and detected the same night.
Code
Add a comment