Recently while working on one of our EMR projects that uses lambdas and airflow, I ran into the following timeout issue:
|
1 |
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/infrastructure-lambda-incoming/invocations" |
We have a lambda that was invoked from boto3 in a Airflow step that would update dynamo db with values needed for our pipeline. This function worked in previous tests with no issues. We did add to the lambda function which was causing it to take longer than normal. When we tested the lambda from the console, the function worked fine, albeit it took a bit longer than the previous version. When calling from Airflow we would continually run into the timeout issue, causing the function to be executed multiple times during retries.
I thought to test this function from the awscli and it revealed the issue, the default boto3 timeout is 60 seconds, this was longer than our lambda was taking. So even though we set the lambda timeout to 4 minutes, boto was timing out at 1 minute, never getting the response back from lambda. The way we fixed this was to have boto3 setup a lambda_config that had a longer timeout.
|
1 2 3 4 5 |
lambda_config = config.Config( read_timeout=900, connect_timeout=900, retries={"max_attempts": 0} ) |
|
1 2 3 4 5 6 7 8 9 10 11 |
def call_lambda(**kwargs): lambda_client = boto3.client('lambda', region_name=REGION, config=lambda_config) payload = {--snipped--} # invoke the lambda and wait for it to complete response = lambda_client.invoke( FunctionName=lambda_name, InvocationType="RequestResponse", Payload=json.dumps(payload), ) if response["StatusCode"] != 200: raise AirflowException("Preprocess lambda has failed") |
Hi Jason, I tried this solution while one of my lambda invokes the other. Earlier that was also getting ReadTimeOutError just after 5 mins.
I am able to see the cloud watch success logs for the function but its giving the error in the execution results window within the lambda function like the below which is basically getting the lambda timedout now
Response
Calling the invoke API action failed with this message: Request did not complete within 960000 ms
Could you please let me know what else am I missing here ?
Thanks!!
Sounds like 960000 ms is hitting the max execution time of the lambda function. If you need longer than 15 minutes you may need to use an EC2 instance to execute your code.
https://aws.amazon.com/about-aws/whats-new/2018/10/aws-lambda-supports-functions-that-can-run-up-to-15-minutes/