Apache Airflow is a powerful open-source tool designed for orchestrating complex workflows and data pipelines. It enables you to programmatically schedule and monitor workflows, making it perfect for automating tasks like data processing and machine learning pipelines. With dynamic pipeline generation, robust scheduling, and monitoring features, Airflow has become one of the top tools in the data engineering field. To get started, install Apache Airflow on Ubuntu 24.04 and leverage its capabilities for streamlined workflow automation.<\/p>\n\n\n\n
Prerequisites:<\/strong><\/p>\n\n\n\n An Ubuntu 24.04 instance has at least 4 GB of RAM.<\/p>\n\n\n\n A domain(skynats.example.com) with an A record pointing to the instance\u2019s IP address.<\/p>\n\n\n\n Step 1: Update Your System<\/strong><\/p>\n\n\n\n Step 2: Verify Python Installation<\/strong><\/p>\n\n\n\n If its not installed, then run:<\/p>\n\n\n\n Step 3: Install Required Packages<\/strong><\/p>\n\n\n\n Step 4: Create a Virtual Environment<\/strong><\/p>\n\n\n\n Create a new Python virtual environment called airflow_env:<\/p>\n\n\n\n Step 5: Activate the Virtual Environment<\/strong><\/p>\n\n\n\n Once activated, your prompt should change to (airflow_env).<\/p>\n\n\n\n Access the PostgreSQL<\/a> console:<\/p>\n\n\n\n Inside the PostgreSQL console, create a new user for Airflow and set the password:<\/p>\n\n\n\n — Replace ‘yourpassword’ with your desired password.Grant the new user full privileges on all tables in the public schema:<\/p>\n\n\n\n Create a new database for Airflow:<\/p>\n\n\n\n Grant the Airflow user ownership of the database:<\/p>\n\n\n\n Grant the Airflow user all privileges on the public schema and exit PostgreSQL:<\/p>\n\n\n\n If you don\u2019t see your Airflow installation directory, initialize the database and start the scheduler to generate the necessary directories:<\/p>\n\n\n\n Stop the scheduler using CTRL+C, then open the airflow.cfg file:<\/p>\n\n\n\n Find the following lines and modify them:<\/p>\n\n\n\n Replace Your StrongPassword with the password you set earlier for the PostgreSQL airflow user. Save and close the file.<\/p>\n\n\n\n Apply the configuration changes by initializing the Airflow metadata database:<\/p>\n\n\n\n Create an administrative user for accessing the Apache Airflow UI. Replace skynats with your desired username: <\/p>\n\n\n\n Start the Airflow web server on port 8080 in the background and redirect logs to a file. Start the Airflow scheduler in the background and redirect logs to another file:<\/p>\n\n\n\n Open your browser and navigate to your domain (skynats.example.com). You should be greeted by the Apache Airflow login page. Use the credentials you created earlier to log in.<\/p>\n\n\n\n Navigate to the Airflow web interface, locate your dags, enable it, and manually trigger it. You can monitor the DAG’s execution through the Graph View and Event Log.<\/p>\n\n\n\n If you need assistance or encounter any issues while following the steps to install Apache Airflow on Ubuntu 24.04, feel free to reach out to our support team. Our experts are ready to provide guidance and ensure a smooth installation process. Contact us<\/a> today for professional support and get the help you need to successfully set up Apache Airflow on your Ubuntu system!<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Apache Airflow is a powerful open-source tool designed for orchestrating complex workflows and data pipelines. It enables you to programmatically schedule and monitor workflows, making it perfect for automating tasks like data processing and machine learning pipelines. With dynamic pipeline generation, robust scheduling, and monitoring features, Airflow has become one of the top tools in […]<\/p>\n","protected":false},"author":16,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[799,960,899],"class_list":["post-14224","post","type-post","status-publish","format-standard","hentry","category-blog","tag-apache-airflow","tag-install-apache-airflow-on-ubuntu-24-04","tag-ubuntu-24-04"],"_links":{"self":[{"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/posts\/14224","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/comments?post=14224"}],"version-history":[{"count":0,"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/posts\/14224\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/media?parent=14224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/categories?post=14224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.skynats.com\/blog\/wp-json\/wp\/v2\/tags?post=14224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}sudo apt update<\/code><\/pre>\n\n\n\npython3 --version<\/code><\/pre>\n\n\n\nsudo apt install python3<\/code><\/pre>\n\n\n\napt-get install build-essential libpq-dev python3-dev<\/code><\/pre>\n\n\n\npython3 -m venv airflow_env<\/code><\/pre>\n\n\n\nsource ~\/airflow_env\/bin\/activate<\/code><\/pre>\n\n\n\nStep 6: Install Apache Airflow with PostgreSQL Support<\/h2>\n\n\n\n
pip install apache-airflow[postgres] psycopg2<\/code><\/pre>\n\n\n\nStep 7: Install PostgreSQL<\/h3>\n\n\n\n
sudo apt install postgresql postgresql-contrib \nsudo systemctl start postgresql<\/code><\/pre>\n\n\n\nStep 8: Configure the PostgreSQL for Airflow<\/h2>\n\n\n\n
sudo -u postgres psql<\/code><\/pre>\n\n\n\nCREATE USER airflow PASSWORD 'yourpassword'; <\/code><\/pre>\n\n\n\nGRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow;<\/code><\/pre>\n\n\n\nCREATE DATABASE airflowdb;<\/code><\/pre>\n\n\n\nALTER DATABASE airflowdb OWNER TO airflow;<\/code><\/pre>\n\n\n\nGRANT ALL ON SCHEMA public TO airflow; \nexit;<\/code><\/pre>\n\n\n\nStep 9: Modify the Airflow Configuration<\/h2>\n\n\n\n
airflow db init; airflow scheduler<\/code><\/pre>\n\n\n\nnano ~\/airflow\/airflow.cfg<\/code><\/pre>\n\n\n\nexecutor = LocalExecutor \nsql_alchemy_conn = postgresql+psycopg2:\/\/airflow:YourStrongPassword@localhost\/airflowdb<\/code><\/pre>\n\n\n\nStep 10: Initialize Airflow\u2019s Metadata Database<\/h2>\n\n\n\n
airflow db init<\/code><\/pre>\n\n\n\nStep 11: Create an Admin User for Airflow<\/h2>\n\n\n\n
airflow users create \\\n --username skynats \\ \n --password yourSuperSecretPassword \\ \n --firstname Skynats \\ \n --lastname User \\ \n --role Admin \\ \n --email yourmail@gmail.com<\/code><\/pre>\n\n\n\nStep 12: Start the Airflow Web Server and Scheduler<\/h4>\n\n\n\n
nohup airflow webserver -p 8080 > webserver.log 2>&1 &\nnohup airflow scheduler > scheduler.log 2>&1 &<\/code><\/pre>\n\n\n\nStep 13: Configure Nginx as a Reverse Proxy<\/h4>\n\n\n\n
Step 14: Access the Airflow UI<\/h2>\n\n\n\n
Step 15: Secure Apache Airflow with SSL Certificates<\/h2>\n\n\n\n
Step 16: Now create and run DAGs using Apache Airflow.<\/h4>\n\n\n\n