airflow – Jason R. Ralph

Recently while working on one of our EMR projects that uses lambdas and airflow, I ran into the following timeout issue:

botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/infrastructure-lambda-incoming/invocations"

1	botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/infrastructure-lambda-incoming/invocations"

We have a lambda that was invoked from boto3 in a Airflow step that would update dynamo db with values needed for our pipeline. This function worked in previous tests with no issues. We did add to the lambda function which was causing it to take longer than normal. When we tested the lambda from the console, the function worked fine, albeit it took a bit longer than the previous version. When calling from Airflow we would continually run into the timeout issue, causing the function to be executed multiple times during retries.

I thought to test this function from the awscli and it revealed the issue, the default boto3 timeout is 60 seconds, this was longer than our lambda was taking. So even though we set the lambda timeout to 4 minutes, boto was timing out at 1 minute, never getting the response back from lambda. The way we fixed this was to have boto3 setup a lambda_config that had a longer timeout.

lambda_config = config.Config(
    read_timeout=900,
    connect_timeout=900,
    retries={"max_attempts": 0}
)

lambda_config = config.Config(

read_timeout=900,

connect_timeout=900,

retries={"max_attempts": 0}

)

def call_lambda(**kwargs):
    lambda_client = boto3.client('lambda', region_name=REGION, config=lambda_config)
    payload = {--snipped--}
    # invoke the lambda and wait for it to complete
    response = lambda_client.invoke(
        FunctionName=lambda_name,
        InvocationType="RequestResponse",
        Payload=json.dumps(payload),
    )
    if response["StatusCode"] != 200:
        raise AirflowException("Preprocess lambda has failed")

def call_lambda(**kwargs):

lambda_client = boto3.client('lambda', region_name=REGION, config=lambda_config)

payload = {--snipped--}

# invoke the lambda and wait for it to complete

response = lambda_client.invoke(

FunctionName=lambda_name,

InvocationType="RequestResponse",

Payload=json.dumps(payload),

)

if response["StatusCode"] != 200:

raise AirflowException("Preprocess lambda has failed")

I came across another fun one the other day, we are in the process of migrating our on premise elastic map reduce system into the cloud. We are using AWS EMR and have AWS Managed Airflow as the executor (DAG). We came across an odd situation with a pyspark application. When using Airflow with a SparkSubmitHook, the job would bootstrap looking just fine according to the run logs, however it would fail with No module named 'requests' when the application tried to import it. This was very odd since we have this application running from spark-submit just fine when calling it from the master node command line.

I decided to investigate the differences, our bootstrap script for installing python modules via pip which we call from the EMR API RunJobFlow call looks like this:

#!/bin/bash
pip_bin=pip3
${pip_bin} install --user -U pip
${pip_bin} install --user boto3
${pip_bin} install --user boto
${pip_bin} install --user requests
${pip_bin} install --user psycopg2-binary

#!/bin/bash

pip_bin=pip3

${pip_bin} install --user -U pip

${pip_bin} install --user boto3

${pip_bin} install --user boto

${pip_bin} install --user requests

${pip_bin} install --user psycopg2-binary

This is very basic, all it does is upgrade PIP and run PIP install to install each of the modules. When checking the bootstrap log I can see that PIP upgrades and goes out to the repo and installs the packages just fine. So why were we getting the No module named 'requests' error when executing through airflow. After a ton of googling and research I have found the issue and applied a solution that worked. Turns out airflow will run as the root user when bootstrapping, so if you notice we use the --user argument in pip. This will instruct the packages to be installed in the calling users home directory, the kicker is the code is run by the hadoop user on the EMR cluster nodes after executing from airflow. So turns out, the hadoop user is unable to access the requests module since root installed it with --user. I changed the bootstrap script to the following and it all started working, by removing --user and prefixing with sudo, the packages now get installed in a globally available area for all users. I am sure there are better ways to do this, I am still learning and researching, but if you run into this, the change below with get you out of the woods.

#!/bin/bash
sudo python3 -m pip install \
                        boto3 \
	                    boto \
		                requests \
                        psycopg2-binary

#!/bin/bash

sudo python3 -m pip install \

boto3 \

boto \

requests \

psycopg2-binary

After some further research, and testing we decided to utilize a requirements.txt file to be called by the bootstrap shell script in the RunJobFlow call, first create a requirements.txt file, I like to hardcode the versions so nothing changes unexpectedly as you bootstrap a new cluster and it reaches out to PyPy to get the packages.

https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html

Add your desired packages and version numbers to a file called requirements.txt like below:

boto3==1.17.54
boto==2.49.0
requests==2.18.4
psycopg2-binary==2.8.6

boto3==1.17.54

boto==2.49.0

requests==2.18.4

psycopg2-binary==2.8.6

Then you will need to copy this file into a bucket you have access to:

aws s3 cp requirements.txt s3://YOUR_S3_BUCKET_NAME/requirements.txt

1	aws s3 cp requirements.txt s3://YOUR_S3_BUCKET_NAME/requirements.txt

Then create a shell script that has the following, call it bootstrap.sh:

#!/bin/bash

set -x 

echo '-----------RUNNING BOOTSTRAP------------------------'

echo '-----------COPYING REQUIREMENTS FILE LOCALLY--------'

aws s3 cp s3://YOUR_S3_BUCKET_NAME/requirements.txt .

echo '-----------INSTALLING REQUIREMENTS------------------'

sudo python3 -m pip install -r requirements.txt

echo '-----------DONE BOOTSTRAP---------------------------'

#!/bin/bash

set -x

echo '-----------RUNNING BOOTSTRAP------------------------'

echo '-----------COPYING REQUIREMENTS FILE LOCALLY--------'

aws s3 cp s3://YOUR_S3_BUCKET_NAME/requirements.txt .

echo '-----------INSTALLING REQUIREMENTS------------------'

sudo python3 -m pip install -r requirements.txt

echo '-----------DONE BOOTSTRAP---------------------------'

Copy that shell script to your bucket:

aws s3 cp bootstrap.sh s3://YOUR_S3_BUCKET_NAME/bootstrap.sh

1	aws s3 cp bootstrap.sh s3://YOUR_S3_BUCKET_NAME/bootstrap.sh

And execute it via the bootstrap actions in the RunJobFlow EMR API call:

"BootstrapActions": [
    {
      "Name": "string",
      "ScriptBootstrapAction": {
        "Path": "s3://YOUR_S3_BUCKET_NAME/bootstrap.sh"
      }
    }
  ],

"BootstrapActions": [

{

"Name": "string",

"ScriptBootstrapAction": {

"Path": "s3://YOUR_S3_BUCKET_NAME/bootstrap.sh"

}

As you can see the shell script will be executed which will copy the requirements.txt file locally and then run pip -r against it which will install all the packages. If you want to see the log on a running cluster, you can ssh to the master node and view the logs here to see the bootstrapping take place:

/emr/instance-controller/log/bootstrap-actions

1	/emr/instance-controller/log/bootstrap-actions

You should see the stdout log as so:

-----------RUNNING BOOTSTRAP------------------
-----------COPYING REQUIREMENTS FILE LOCALLY--------
Completed 67 Bytes/67 Bytes (629 Bytes/s) with 1 file(s) remaining
download: s3://YOUR_S3_BUCKET_NAME/requirements.txt to ./requirements.txt
-----------INSTALLING REQUIREMENTS------------------
Collecting boto==2.48.0
  Downloading boto-2.48.0-py2.py3-none-any.whl (1.4 MB)
Collecting boto3==1.6.15
  Downloading boto3-1.6.15-py2.py3-none-any.whl (128 kB)
Collecting requests==2.18.4
  Downloading requests-2.18.4-py2.py3-none-any.whl (88 kB)
Collecting psycopg2-binary==2.8.6
  Downloading psycopg2_binary-2.8.6-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)
Collecting botocore<1.10.0,>=1.9.15
  Downloading botocore-1.9.23-py2.py3-none-any.whl (4.1 MB)
Collecting s3transfer<0.2.0,>=0.1.10
  Downloading s3transfer-0.1.13-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3==1.6.15->-r jason_requirements.txt (line 2)) (0.10.0)
Collecting urllib3<1.23,>=1.21.1
  Downloading urllib3-1.22-py2.py3-none-any.whl (132 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<2.7,>=2.5
  Downloading idna-2.6-py2.py3-none-any.whl (56 kB)
Collecting chardet<3.1.0,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Requirement already satisfied: docutils>=0.10 in /usr/lib/python3.7/site-packages (from botocore<1.10.0,>=1.9.15->boto3==1.6.15->-r jason_requirements.txt (line 2)) (0.14)
Collecting python-dateutil<2.7.0,>=2.1
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<2.7.0,>=2.1->botocore<1.10.0,>=1.9.15->boto3==1.6.15->-r jason_requirements.txt (line 2)) (1.13.0)
Installing collected packages: boto, python-dateutil, botocore, s3transfer, boto3, urllib3, certifi, idna, chardet, requests, psycopg2-binary
  Attempting uninstall: boto
    Found existing installation: boto 2.49.0
    Uninstalling boto-2.49.0:
      Successfully uninstalled boto-2.49.0
Successfully installed boto-2.48.0 boto3-1.6.15 botocore-1.9.23 certifi-2021.10.8 chardet-3.0.4 idna-2.6 psycopg2-binary-2.8.6 python-dateutil-2.6.1 requests-2.18.4 s3transfer-0.1.13 urllib3-1.22
-----------DONE BOOTSTRAP---------------------

-----------RUNNING BOOTSTRAP------------------

-----------COPYING REQUIREMENTS FILE LOCALLY--------

Completed 67 Bytes/67 Bytes (629 Bytes/s) with 1 file(s) remaining

download: s3://YOUR_S3_BUCKET_NAME/requirements.txt to ./requirements.txt

-----------INSTALLING REQUIREMENTS------------------

Collecting boto==2.48.0

Downloading boto-2.48.0-py2.py3-none-any.whl (1.4 MB)

Collecting boto3==1.6.15

Downloading boto3-1.6.15-py2.py3-none-any.whl (128 kB)

Collecting requests==2.18.4

Downloading requests-2.18.4-py2.py3-none-any.whl (88 kB)

Collecting psycopg2-binary==2.8.6

Downloading psycopg2_binary-2.8.6-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)

Collecting botocore<1.10.0,>=1.9.15

Downloading botocore-1.9.23-py2.py3-none-any.whl (4.1 MB)

Collecting s3transfer<0.2.0,>=0.1.10

Downloading s3transfer-0.1.13-py2.py3-none-any.whl (59 kB)

Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3==1.6.15->-r jason_requirements.txt (line 2)) (0.10.0)

Collecting urllib3<1.23,>=1.21.1

Downloading urllib3-1.22-py2.py3-none-any.whl (132 kB)

Collecting certifi>=2017.4.17

Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)

Collecting idna<2.7,>=2.5

Downloading idna-2.6-py2.py3-none-any.whl (56 kB)

Collecting chardet<3.1.0,>=3.0.2

Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)

Requirement already satisfied: docutils>=0.10 in /usr/lib/python3.7/site-packages (from botocore<1.10.0,>=1.9.15->boto3==1.6.15->-r jason_requirements.txt (line 2)) (0.14)

Collecting python-dateutil<2.7.0,>=2.1

Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194 kB)

Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<2.7.0,>=2.1->botocore<1.10.0,>=1.9.15->boto3==1.6.15->-r jason_requirements.txt (line 2)) (1.13.0)

Installing collected packages: boto, python-dateutil, botocore, s3transfer, boto3, urllib3, certifi, idna, chardet, requests, psycopg2-binary

Attempting uninstall: boto

Found existing installation: boto 2.49.0

Uninstalling boto-2.49.0:

Successfully uninstalled boto-2.49.0

Successfully installed boto-2.48.0 boto3-1.6.15 botocore-1.9.23 certifi-2021.10.8 chardet-3.0.4 idna-2.6 psycopg2-binary-2.8.6 python-dateutil-2.6.1 requests-2.18.4 s3transfer-0.1.13 urllib3-1.22

-----------DONE BOOTSTRAP---------------------

Hope this helps.

Jason R. Ralph

Linux All Day Everyday

Tag: airflow

ImportError: cannot import name ‘is_s3express_bucket’

botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: https://lambda.us-east-1.amazonaws.com

AWS Apache Managed Airflow EMR ModuleNotFoundError: No module named ‘requests’ Bootstrap