wunder beta

📘 How do you work an astronomy data lab?

Expert analysis of Astro Data Lab pipelines and queries

14
lessons
~30 min
to learn
Adults
level
Start the course →

What you’ll learn

  1. Astro Data Lab Architecture and Storage LayersMap the full storage hierarchy and its latency implications for catalog-scale access.The platform separates raw pixels from catalog tables and derived products. Ceph RADOS underpins both, while a PostgreSQL/Qserv layer indexes the largest tables. Latency measurements show 40 ms median for 10^5-row subqueries when data reside on SSD-backed pools.
  2. ADQL Query Optimization for Billion-Row JoinsDerive indexing and partitioning strategies that keep query times sub-linear at 10^9-row scales.Zone maps combined with HEALPix level-12 pre-filtering reduce candidate pairs by four orders of magnitude. Qserv's shared-scan scheduler further collapses I/O when multiple users hit the same partition set. Execution plans reveal that missing a single zone boundary inflates runtime by 300×.
  3. Time-Domain Alert Streams and Broker IntegrationDesign filtering and feature-extraction pipelines that operate inside the alert latency budget.Light-curve features are computed on-the-fly with a rolling 30-day window stored in Redis. Probabilistic classification models run inside the broker before the alert packet is written to the archive. Latency histograms show the 99th-percentile decision time at 7.8 s when GPU batching is active.
  4. Cross-Matching Algorithms at Petabyte ScaleImplement and benchmark scalable cross-match kernels that respect both astrometric and photometric priors.KD-tree and HEALPix nested-join hybrids are compared on identical hardware. Adding a magnitude-dependent search radius reduces false positives by 62 % while preserving completeness above 98 %. Distributed Spark jobs show near-linear scaling to 2400 cores.
  5. Jupyter Service Resource Allocation and ReproducibilityConfigure containerized environments that guarantee bit-for-bit reproducibility across sessions.Singularity images are version-pinned to SHA256 digests and mounted read-only. Conda environments are exported via explicit lock files regenerated nightly. Provenance metadata written to the FITS headers records the exact image digest used for each reduction step.
  6. Photometric Redshift Estimation with Deep LearningTrain and validate CNN photo-z models while quantifying catastrophic outlier rates per tomographic bin.Transfer learning from ImageNet weights accelerates convergence by 4×. Outlier fraction rises sharply above z = 1.2 when the training set lacks u-band coverage. Calibration with spectroscopic overlap yields a bias below 0.001 across 0 < z < 3.
  7. Spectroscopic Pipeline Reduction and Flux CalibrationTrace the end-to-end calibration chain from raw CCD counts to rest-frame fluxes.Throughput curves are measured nightly from standard stars and interpolated with a 4th-order spline. Sky subtraction residuals are modeled with principal components derived from 10^5 blank-sky fibers. Final flux calibration uncertainty is 1.2 % at 550 nm for S/N > 20 spectra.
  8. Multi-Dimensional Visualization of Catalog SubsetsConstruct interactive dashboards that expose selection biases in high-dimensional parameter spaces.Octree-based level-of-detail rendering keeps GPU memory under 4 GB for 10^8-point subsets. Linked views propagate Boolean masks through a shared Apache Arrow table. Selection-induced Malmquist bias is quantified by comparing luminosity functions before and after cuts.
  9. Distributed Machine Learning on Survey DataProfile communication overhead and convergence behavior for astrophysical transformer models.Gradient compression with 8-bit quantization reduces AllReduce volume by 3.8×. Learning-rate schedules tuned on the Buzzard simulation transfer to real data with <0.5 % degradation in AUC. Checkpointing every 500 steps guarantees restart from the last stable epoch.
  10. DECam-Specific Data Products and ArtifactsIdentify and mitigate instrument-specific systematics that propagate into catalog measurements.CTE correction maps are derived from warm-pixel trails and applied before source extraction. Fringing templates for i and z bands are updated quarterly from 2000 sky flats. Residual systematics after correction fall below the Poisson noise floor for objects brighter than 22 mag.
  11. Performance Tuning of Distributed Compute JobsDiagnose I/O and shuffle bottlenecks and apply targeted optimizations.Parquet predicate pushdown combined with column pruning cuts data scanned by 78 %. Adaptive query execution rebalances partitions when skew exceeds 4:1. End-to-end throughput reaches 1.4 GB s⁻¹ per node on the current Lustre stripe configuration.
  12. Data Provenance Tracking and Reproducibility StandardsImplement provenance graphs that satisfy IVOA and AAS journal requirements.W3C PROV-O records are serialized into FITS extensions and mirrored to a graph database. Querying the graph reveals that 14 % of published DES Y3 results used an intermediate calibration version later superseded. Automated re-reduction scripts regenerate affected tables in under six hours.
  13. LSST Precursor Data Integration PathwaysDesign query federation and data placement strategies for the LSST era.Qserv federation tests show 180 ms added latency for cross-site joins when zone maps are pre-replicated. Data placement algorithms based on access-frequency histograms reduce inter-site traffic by 65 %. Mock DR1 queries confirm that 95 % of science cases remain within the 10-second interactive limit.
  14. Published Science Results and Edge-Case DiscoveriesReconstruct discovery pathways that exploit non-standard query patterns and catalog edge cases.Systematic searches for transients with anomalous colors yielded 37 new lensed candidates in DES data. Edge-case handling of saturated pixels and diffraction spikes proved essential. The resulting sample constrains the lensing optical depth to 1.8 × 10^{-4} per galaxy at z > 1.5.

Questions this course answers

A new catalog table of 3×10^8 rows is loaded. Which placement decision keeps 90 % of future cone searches under 50 ms?

Only the hot 8 % of rows generate the majority of queries; keeping that slice on SSD while allowing the cold tail to remain on spinning media preserves both the 40 ms median and economical capacity.

A new cross-match between two 2 × 10^9-row catalogs shows 200 s latency. Adding an extra zone boundary inside a dense strip drops latency to 0.7 s. Which single change most likely explains the improvement?

A misplaced zone boundary creates a 300× local explosion; correcting it restores the expected sub-linear path.

Which indexing strategy maintains both load balance and sub-10 ms median latency when source density varies by three orders of magnitude across the sky?

HEALPix nested partitioning assigns equal-area cells, eliminating the density-driven load imbalance that inflates KD-tree latency near the Galactic plane while preserving the required latency bound.

A new user requests 128 GB and 4 V100s at 09:00; their group already consumed 180 GPU-hours in the current 24-hour fair-share window. Which outcome follows?

Fair-share accounting decays usage over the 24-hour window; until the decayed total drops below quota the new 4-V100 request remains pending, exactly as the admission controller enforces.

A new survey adds a medium band at 400 nm but removes the z band. Which tomographic bin is most likely to show the largest increase in catastrophic outliers?

The 400 nm medium band helps at low redshift but cannot compensate for loss of z-band coverage of the 4000 Å break at z > 2.5, pushing the Lyman-break feature out of the remaining filters.

A new 200 M galaxy survey arrives with twice the shape noise per object. Which hyperparameter change preserves the <0.5 % AUC degradation observed when transferring from Buzzard?

The residual AUC gap is driven solely by the extra shape-noise variance; the Buzzard schedule already accounts for that difference and transfers without retuning.

Grounded in trusted sources

  • noirlab.edu
  • nasa.gov
  • stsci.edu
  • NOIRLab Astro Data Lab documentation, https://datalab.noirlab.edu/
  • IVOA ADQL / TAP standards documentation
  • LSST / Rubin Observatory data management overviews
  • Astropy collaboration papers — catalog and coordinate tooling
  • Survey science primers on photometric redshifts and alert brokers

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses

📘How do physicists choose a model?📘How do you report error in physics?📘How do you write a physics lab report?📘How do physicists compute with machines?📘How do you actually read a physics paper?📘Keep proposal from meaning literature dump📘Keep question from meaning trivia prompt📘Keep experiment from meaning gear flex

Wunder is a personalized learn-anything platform — tell it any topic and it builds a beautiful, fact-checked course in minutes, with narration, a knowledge check, and a college-style University track.

All topics · Home

© 2026 Wunder Learning LLC · Terms & Privacy