Recently while working on one of our EMR projects that uses lambdas and airflow, I ran into the following timeout issue:
|
1 |
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/infrastructure-lambda-incoming/invocations" |
We have a lambda that was invoked from boto3 in a Airflow step that would update dynamo db with values needed for our pipeline. This function worked in previous tests with no issues. We did add to the lambda function which was causing it to take longer than normal. When we tested the lambda from the console, the function worked fine, albeit it took a bit longer than the previous version. When calling from Airflow we would continually run into the timeout issue, causing the function to be executed multiple times during retries.
I thought to test this function from the awscli and it revealed the issue, the default boto3 timeout is 60 seconds, this was longer than our lambda was taking. So even though we set the lambda timeout to 4 minutes, boto was timing out at 1 minute, never getting the response back from lambda. The way we fixed this was to have boto3 setup a lambda_config that had a longer timeout.
|
1 2 3 4 5 |
lambda_config = config.Config( read_timeout=900, connect_timeout=900, retries={"max_attempts": 0} ) |
|
1 2 3 4 5 6 7 8 9 10 11 |
def call_lambda(**kwargs): lambda_client = boto3.client('lambda', region_name=REGION, config=lambda_config) payload = {--snipped--} # invoke the lambda and wait for it to complete response = lambda_client.invoke( FunctionName=lambda_name, InvocationType="RequestResponse", Payload=json.dumps(payload), ) if response["StatusCode"] != 200: raise AirflowException("Preprocess lambda has failed") |