NYC Airflow Meetup at Datadog!

Keep your plans in one place. Free account.

An Apache Airflow meetup with presentations on multi-tenant Airflow at scale and using Airflow for production-ready AI, co-hosted by Astronomer and Datadog.

AI summary · From the event listing

Join fellow Airflow enthusiasts and leaders at Datadog's offices for an evening of engaging talks, great food and drinks, and awesome swag, co-hosted by Astronomer and Datadog! PRESENTATIONS Talk 1: Multi-Tenant Airflow at Scale Speaker: *Conlan Cesar, Analytics & Data* *Platforms, Software Engineer at Datadog* Datadog is a world-class data platform ingesting more than a 100 trillion events a day, providing real-time insights. Since our internal adoption of Airflow following the release of 3.0.0, the number of teams relying on our internal Airflow platform have grown organically and quickly. This internal Airflow adoption came with a number of platform challenges, requiring novel solutions which could support multi-tenancy, scalability, and bespoke runtime environments. In this talk, we cover the changes we've implemented to allow for a secure and scalable multi-tenant Airflow. This includes plugins for the secrets backend to make different credentials available to Operators transparently, how we inject short-lived JWT tokens without needing to modify operators, methods to scale workers according to live demand with Datadog Autoscaling, and upstream contributions to the triggerer to allow for specific queue assignment. Talk 2: Apache Airflow for Production-ready AI Speaker: *George Yates, Field Engineering Manager at Astronomer* Running AI workloads in production takes more than a model call.

Garment District