{"id":1526,"date":"2024-05-18T09:44:04","date_gmt":"2024-05-18T07:44:04","guid":{"rendered":"https:\/\/easybi.lu\/?p=1526"},"modified":"2024-10-27T19:05:27","modified_gmt":"2024-10-27T18:05:27","slug":"introduction-to-apache-airflow","status":"publish","type":"post","link":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/","title":{"rendered":"Scheduling with Apache Airflow"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Airflow at a glance<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Apache Airflow is a workflow managment platform written in python<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"520\" height=\"121\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_timeline.png\" alt=\"\" class=\"wp-image-1534\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_timeline.png 520w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_timeline-300x70.png 300w\" sizes=\"auto, (max-width: 520px) 100vw, 520px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Workflows are developed also in python code, perfectly fit to send your \u00ab\u00a0configuration as code\u00a0\u00bb in a git repository.<\/li>\n\n\n\n<li>You leverage python extra libraries to get enhancement in your pipelines, without the need of clunky langage like XML.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><br>How does it work ?<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Airflow use DAGs standing for <a href=\"https:\/\/en.wikipedia.org\/wiki\/Directed_acyclic_graph\" data-type=\"link\" data-id=\"https:\/\/en.wikipedia.org\/wiki\/Directed_acyclic_graph\">Directed Acyclic Graph<\/a> for worflows orchestration and their dependencies.<\/li>\n\n\n\n<li>Worflows are <strong>read only<\/strong> rendered in a beautiful UI, pay attention that <strong>this is not a WYSIWYG tool<\/strong>.<\/li>\n\n\n\n<li>It uses out of the box core <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/operators.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/operators.html\">operators <\/a>(bash, python, email, &#8230;) and extended one for all existing <a href=\"https:\/\/airflow.apache.org\/docs\/#providers-packages-docs-apache-airflow-providers-index-html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/#providers-packages-docs-apache-airflow-providers-index-html\">providers<\/a> (http, databases, docker, all cloud vendors like AWS-GCP-Azure, &#8230;).<\/li>\n\n\n\n<li>Scalability is infinite using messaging, in this post we only introduce a <strong>local scheduling mode<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><br>Quick demo<\/h2>\n\n\n\n<h4 class=\"wp-block-heading\">Launching Airflow<\/h4>\n\n\n\n<p>Even if the <a href=\"https:\/\/github.com\/apache\/airflow\" data-type=\"link\" data-id=\"https:\/\/github.com\/apache\/airflow\">official repository <\/a>provide a docker compose file, I fine tuned the file to simplify the getting started:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>DAGs sample removing to get a brand new cleaned DAGs directory<\/li>\n\n\n\n<li>Set the orchestrator to local, no clustering<\/li>\n\n\n\n<li>As usual, add the traefik container and localtest.me domain for a ready to use DNS setup<\/li>\n<\/ul>\n\n\n\n<p>Once ready, launch these commands<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">git clone https:\/\/github.com\/jsminet\/docker-apache-airflow.git\ndocker compose up -d<\/code><\/pre>\n\n\n\n<p>Open your favorite browser and go to the Airflow login page: <a href=\"http:\/\/airflow.localtest.me\">http:\/\/airflow.localtest.me<\/a> <\/p>\n\n\n\n<p>Use <strong>airflow \/ airflow<\/strong> as credential and you should see this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"206\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01-1024x206.png\" alt=\"\" class=\"wp-image-1541\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01-1024x206.png 1024w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01-300x60.png 300w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01-768x154.png 768w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01-1536x309.png 1536w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_demo_01.png 1910w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><br>First DAG creation<\/h4>\n\n\n\n<p>As a prerequisite, create connection to postgres database by clicking on <strong>Admin <\/strong>then <strong>Connections<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"161\" height=\"139\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection.png\" alt=\"\" class=\"wp-image-1557\"\/><\/figure>\n\n\n\n<p><br>Postgres is part of this docker stack and I placed a script to automatically create a dedicated database for this demo, you can use <a href=\"https:\/\/dbeaver.io\/\" data-type=\"link\" data-id=\"https:\/\/dbeaver.io\/\">Dbeaver<\/a> for instance to test the connection, here are all the connection parameters:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>User<\/strong><\/td><td><strong>Password<\/strong><\/td><td><strong>Database<\/strong><\/td><td><strong>Host<\/strong><\/td><td><strong>Port<\/strong><\/td><\/tr><tr><td>data_sample_user<\/td><td>data_sample_password<\/td><td>data_sample<\/td><td>airflow.localtest.me<\/td><td>5432<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Follow these setps for the postgres database setup<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69d5bd4a89302&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"685\" height=\"709\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1560\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_connection_using_dbeaver.png\" alt=\"\" class=\"wp-image-1560\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_connection_using_dbeaver.png 685w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_connection_using_dbeaver-290x300.png 290w\" sizes=\"auto, (max-width: 685px) 100vw, 685px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Agrandir l\u2018image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"state.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\"><strong><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">Testing connection with Dbeaver<\/mark><\/strong><\/figcaption><\/figure>\n\n\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69d5bd4a89711&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"240\" height=\"228\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1562\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection_2.png\" alt=\"\" class=\"wp-image-1562\"\/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Agrandir l\u2018image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"state.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\"><strong><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">Add connection<\/mark><\/strong><\/figcaption><\/figure>\n\n\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69d5bd4a89a52&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"968\" height=\"603\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1569\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection_3-1.png\" alt=\"\" class=\"wp-image-1569\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection_3-1.png 968w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection_3-1-300x187.png 300w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_create_connection_3-1-768x478.png 768w\" sizes=\"auto, (max-width: 968px) 100vw, 968px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Agrandir l\u2018image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"state.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\"><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\"><strong>Airflow DB parameters<\/strong><\/mark><\/figcaption><\/figure>\n<\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>Once connection is done, open your favorite IDE and create a new python file in <strong>dags directory<\/strong> and use this code:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\"># Import datetime module\nfrom datetime import datetime, timedelta\n# Import DAG definition\nfrom airflow import DAG\n# Import bash operator\nfrom airflow.operators.bash import BashOperator\nfrom airflow.operators.python import PythonOperator\nfrom airflow.providers.postgres.operators.postgres import PostgresOperator\n\n# A simple python function\ndef greet(name):\n    print(f\"Hello {name} with python!\")\n    return name\n\n# The DAG object\nwith DAG(\n    \"tutorial_dag\",\n    description=\"A simple DAG tutorial\",\n    default_args = {\n        \"owner\":\"JS\",\n        \"retries\": 1,\n        \"retry_delay\": timedelta(minutes=5)\n        },\n    schedule=timedelta(days=1),\n    start_date=datetime(2024, 1, 1),\n) as dag:\n\n# bash task1 definition object  \n    task1 = BashOperator(\n        task_id=\"bash_greeting\",\n        bash_command=\"echo 'Hello world with bash!'\")\n    \n# python task2 definition object \n    task2 = PythonOperator(\n        task_id=\"python_greeting\",\n        python_callable=greet,\n        op_kwargs={\"name\":\"JS\"})\n    \n# postgres task3\n    task3 = PostgresOperator(\n        task_id=\"postgres_data_insertion\",\n        postgres_conn_id=\"postgres_connection\",\n        sql=\"\"\"\n        CREATE TABLE IF NOT EXISTS\n                    tutorial(\n                        id SERIAL PRIMARY KEY,\n                        current_dag_run TIMESTAMP\n                    );\n        INSERT INTO tutorial\n            (current_dag_run) VALUES ('{{ts}}')\n        \"\"\")\n    \n# Calling the tasks\n    task1 &gt;&gt; task2 &gt;&gt; task3<\/code><\/pre>\n\n\n\n<p>In this DAG, you&rsquo;ll create three sequential and dependant tasks using three different operators:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"591\" height=\"151\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/Airflow_dag_and_task_diagram.png\" alt=\"\" class=\"wp-image-1552\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/Airflow_dag_and_task_diagram.png 591w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/Airflow_dag_and_task_diagram-300x77.png 300w\" sizes=\"auto, (max-width: 591px) 100vw, 591px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Task 1 -&gt; bash script using <strong>bash operator<\/strong><\/li>\n\n\n\n<li>Task 2 -&gt; python script using <strong>python operator<\/strong><\/li>\n\n\n\n<li>Task 3 -&gt; postgres data insertion using a connection using <strong>postgres DB operator<\/strong><\/li>\n<\/ul>\n\n\n\n<p>To create data into postgres, we will use out-of-the-box default variable you can find <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/templates-ref.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/templates-ref.html\">on template reference page<\/a> and an auto increment id (keyword serial)<br><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Launching the DAG<\/h4>\n\n\n\n<p>Click on <strong>DAG <\/strong>tab then on button <strong>Trigger DAG<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"249\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_run_dag-1024x249.png\" alt=\"\" class=\"wp-image-1588\" style=\"width:840px;height:auto\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_run_dag-1024x249.png 1024w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_run_dag-300x73.png 300w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_run_dag-768x187.png 768w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_run_dag.png 1313w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><br>After succesfull runs, click on <strong>Graph <\/strong>to see all the DAG embedded tasks<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"368\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_graph_ui-1-1024x368.png\" alt=\"\" class=\"wp-image-1590\" style=\"width:840px;height:auto\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_graph_ui-1-1024x368.png 1024w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_graph_ui-1-300x108.png 300w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_graph_ui-1-768x276.png 768w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_graph_ui-1.png 1338w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Click on each task green square to activate the <strong>Logs <\/strong>tab accordingly<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"759\" height=\"96\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_task_logs.png\" alt=\"\" class=\"wp-image-1591\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_task_logs.png 759w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_dag_task_logs-300x38.png 300w\" sizes=\"auto, (max-width: 759px) 100vw, 759px\" \/><\/figure>\n\n\n\n<p><br>The <strong>Logs <\/strong>tab is now available for all different tasks<\/p>\n\n\n\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-2 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"573\" height=\"163\" data-id=\"1592\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_bash_operator_logs.png\" alt=\"\" class=\"wp-image-1592\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_bash_operator_logs.png 573w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_bash_operator_logs-300x85.png 300w\" sizes=\"auto, (max-width: 573px) 100vw, 573px\" \/><figcaption class=\"wp-element-caption\"><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">Bash operator logs<\/mark><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"671\" height=\"135\" data-id=\"1594\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_python_operator_logs.png\" alt=\"\" class=\"wp-image-1594\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_python_operator_logs.png 671w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_python_operator_logs-300x60.png 300w\" sizes=\"auto, (max-width: 671px) 100vw, 671px\" \/><figcaption class=\"wp-element-caption\"><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">Python operator logs<\/mark><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"566\" height=\"135\" data-id=\"1593\" src=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_operator_logs.png\" alt=\"\" class=\"wp-image-1593\" srcset=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_operator_logs.png 566w, https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/airflow_database_operator_logs-300x72.png 300w\" sizes=\"auto, (max-width: 566px) 100vw, 566px\" \/><figcaption class=\"wp-element-caption\"><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">DB operator logs<\/mark><\/figcaption><\/figure>\n<\/figure>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What&rsquo;s next<\/h2>\n\n\n\n<p>Only a few part of Airflow has been introduced, it&rsquo;s a powerful scheduler that need others post to explore all the features and possibilities.<\/p>\n\n\n\n<p>Here are some important topics about Apache Airflow you can find in the <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/index.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/index.html\">core concept<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cross communication using <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/xcoms.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/xcoms.html\">Xcoms<\/a><\/li>\n\n\n\n<li>Making Airflow administration using the <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/howto\/usage-cli.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/howto\/usage-cli.html\">CLI<\/a><\/li>\n\n\n\n<li>Managing taskflows using the <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/stable-rest-api-ref.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/stable-rest-api-ref.html\">Rest API<\/a><\/li>\n\n\n\n<li>Using <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/sensors.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/sensors.html\">sensors <\/a>to detect a incoming file on your file system like S3 for instance<\/li>\n\n\n\n<li>How to make <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/debug.html\" data-type=\"link\" data-id=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/debug.html\">DAG debbuging<\/a><\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Airflow at a glance How does it work ? Quick demo Launching Airflow Even if the official repository provide a docker compose file, I fine tuned the file to simplify the getting started: Once ready, launch these commands Open your favorite browser and go to the Airflow login page: http:\/\/airflow.localtest.me&#8230;<\/p>\n","protected":false},"author":3,"featured_media":1601,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40],"tags":[39,27,38],"class_list":["post-1526","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-scheduler","tag-airflow","tag-docker","tag-python"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Scheduling with Apache Airflow - EasyBI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Scheduling with Apache Airflow - EasyBI\" \/>\n<meta property=\"og:description\" content=\"Airflow at a glance How does it work ? Quick demo Launching Airflow Even if the official repository provide a docker compose file, I fine tuned the file to simplify the getting started: Once ready, launch these commands Open your favorite browser and go to the Airflow login page: http:\/\/airflow.localtest.me...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\" \/>\n<meta property=\"og:site_name\" content=\"EasyBI\" \/>\n<meta property=\"article:published_time\" content=\"2024-05-18T07:44:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-27T18:05:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"544\" \/>\n\t<meta property=\"og:image:height\" content=\"297\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"JS Minet\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@easybi_org\" \/>\n<meta name=\"twitter:site\" content=\"@easybi_org\" \/>\n<meta name=\"twitter:label1\" content=\"\u00c9crit par\" \/>\n\t<meta name=\"twitter:data1\" content=\"JS Minet\" \/>\n\t<meta name=\"twitter:label2\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\"},\"author\":{\"name\":\"JS Minet\",\"@id\":\"https:\/\/easybi.lu\/#\/schema\/person\/3c1348f644dc92c6d1f9fde393c091f2\"},\"headline\":\"Scheduling with Apache Airflow\",\"datePublished\":\"2024-05-18T07:44:04+00:00\",\"dateModified\":\"2024-10-27T18:05:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\"},\"wordCount\":534,\"publisher\":{\"@id\":\"https:\/\/easybi.lu\/#organization\"},\"image\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg\",\"keywords\":[\"Airflow\",\"Docker\",\"Python\"],\"articleSection\":[\"Scheduler\"],\"inLanguage\":\"fr-FR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\",\"url\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\",\"name\":\"Scheduling with Apache Airflow - EasyBI\",\"isPartOf\":{\"@id\":\"https:\/\/easybi.lu\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg\",\"datePublished\":\"2024-05-18T07:44:04+00:00\",\"dateModified\":\"2024-10-27T18:05:27+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#breadcrumb\"},\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage\",\"url\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg\",\"contentUrl\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg\",\"width\":544,\"height\":297},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/easybi.lu\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Scheduling with Apache Airflow\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/easybi.lu\/#website\",\"url\":\"https:\/\/easybi.lu\/\",\"name\":\"EasyBI\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/easybi.lu\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/easybi.lu\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-FR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/easybi.lu\/#organization\",\"name\":\"EasyBI\",\"url\":\"https:\/\/easybi.lu\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/easybi.lu\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2019\/12\/Couleur-avec-baseline-75px.png\",\"contentUrl\":\"https:\/\/easybi.lu\/wp-content\/uploads\/2019\/12\/Couleur-avec-baseline-75px.png\",\"width\":67,\"height\":75,\"caption\":\"EasyBI\"},\"image\":{\"@id\":\"https:\/\/easybi.lu\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/easybi_org\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/easybi.lu\/#\/schema\/person\/3c1348f644dc92c6d1f9fde393c091f2\",\"name\":\"JS Minet\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/easybi.lu\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/04444764cc721204aee8e12a7d3ccc2d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/04444764cc721204aee8e12a7d3ccc2d?s=96&d=mm&r=g\",\"caption\":\"JS Minet\"},\"url\":\"https:\/\/easybi.lu\/index.php\/author\/jsm\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Scheduling with Apache Airflow - EasyBI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/","og_locale":"fr_FR","og_type":"article","og_title":"Scheduling with Apache Airflow - EasyBI","og_description":"Airflow at a glance How does it work ? Quick demo Launching Airflow Even if the official repository provide a docker compose file, I fine tuned the file to simplify the getting started: Once ready, launch these commands Open your favorite browser and go to the Airflow login page: http:\/\/airflow.localtest.me...","og_url":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/","og_site_name":"EasyBI","article_published_time":"2024-05-18T07:44:04+00:00","article_modified_time":"2024-10-27T18:05:27+00:00","og_image":[{"width":544,"height":297,"url":"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg","type":"image\/jpeg"}],"author":"JS Minet","twitter_card":"summary_large_image","twitter_creator":"@easybi_org","twitter_site":"@easybi_org","twitter_misc":{"\u00c9crit par":"JS Minet","Dur\u00e9e de lecture estim\u00e9e":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#article","isPartOf":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/"},"author":{"name":"JS Minet","@id":"https:\/\/easybi.lu\/#\/schema\/person\/3c1348f644dc92c6d1f9fde393c091f2"},"headline":"Scheduling with Apache Airflow","datePublished":"2024-05-18T07:44:04+00:00","dateModified":"2024-10-27T18:05:27+00:00","mainEntityOfPage":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/"},"wordCount":534,"publisher":{"@id":"https:\/\/easybi.lu\/#organization"},"image":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg","keywords":["Airflow","Docker","Python"],"articleSection":["Scheduler"],"inLanguage":"fr-FR"},{"@type":"WebPage","@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/","url":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/","name":"Scheduling with Apache Airflow - EasyBI","isPartOf":{"@id":"https:\/\/easybi.lu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage"},"image":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg","datePublished":"2024-05-18T07:44:04+00:00","dateModified":"2024-10-27T18:05:27+00:00","breadcrumb":{"@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#breadcrumb"},"inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/"]}]},{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#primaryimage","url":"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg","contentUrl":"https:\/\/easybi.lu\/wp-content\/uploads\/2024\/05\/apache_airflow-1.jpg","width":544,"height":297},{"@type":"BreadcrumbList","@id":"https:\/\/easybi.lu\/index.php\/2024\/05\/18\/introduction-to-apache-airflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/easybi.lu\/"},{"@type":"ListItem","position":2,"name":"Scheduling with Apache Airflow"}]},{"@type":"WebSite","@id":"https:\/\/easybi.lu\/#website","url":"https:\/\/easybi.lu\/","name":"EasyBI","description":"","publisher":{"@id":"https:\/\/easybi.lu\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/easybi.lu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-FR"},{"@type":"Organization","@id":"https:\/\/easybi.lu\/#organization","name":"EasyBI","url":"https:\/\/easybi.lu\/","logo":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/easybi.lu\/#\/schema\/logo\/image\/","url":"https:\/\/easybi.lu\/wp-content\/uploads\/2019\/12\/Couleur-avec-baseline-75px.png","contentUrl":"https:\/\/easybi.lu\/wp-content\/uploads\/2019\/12\/Couleur-avec-baseline-75px.png","width":67,"height":75,"caption":"EasyBI"},"image":{"@id":"https:\/\/easybi.lu\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/easybi_org"]},{"@type":"Person","@id":"https:\/\/easybi.lu\/#\/schema\/person\/3c1348f644dc92c6d1f9fde393c091f2","name":"JS Minet","image":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/easybi.lu\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/04444764cc721204aee8e12a7d3ccc2d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/04444764cc721204aee8e12a7d3ccc2d?s=96&d=mm&r=g","caption":"JS Minet"},"url":"https:\/\/easybi.lu\/index.php\/author\/jsm\/"}]}},"_links":{"self":[{"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/posts\/1526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/comments?post=1526"}],"version-history":[{"count":33,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/posts\/1526\/revisions"}],"predecessor-version":[{"id":1603,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/posts\/1526\/revisions\/1603"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/media\/1601"}],"wp:attachment":[{"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/media?parent=1526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/categories?post=1526"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/easybi.lu\/index.php\/wp-json\/wp\/v2\/tags?post=1526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}