botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL:

Recently while working on one of our EMR projects that uses lambdas and airflow, I ran into the following timeout issue:

We have a lambda that was invoked from boto3 in a Airflow step that would update dynamo db with values needed for our pipeline. This function worked in previous tests with no issues. We did add to the lambda function which was causing it to take longer than normal. When we tested the lambda from the console, the function worked fine, albeit it took a bit longer than the previous version. When calling from Airflow we would continually run into the timeout issue, causing the function to be executed multiple times during retries.

I thought to test this function from the awscli and it revealed the issue, the default boto3 timeout is 60 seconds, this was longer than our lambda was taking. So even though we set the lambda timeout to 4 minutes, boto was timing out at 1 minute, never getting the response back from lambda. The way we fixed this was to have boto3 setup a lambda_config that had a longer timeout.

2 thoughts on “botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL:”

  1. Hi Jason, I tried this solution while one of my lambda invokes the other. Earlier that was also getting ReadTimeOutError just after 5 mins.

    I am able to see the cloud watch success logs for the function but its giving the error in the execution results window within the lambda function like the below which is basically getting the lambda timedout now

    Calling the invoke API action failed with this message: Request did not complete within 960000 ms

    Could you please let me know what else am I missing here ?


Leave a Reply

Your email address will not be published. Required fields are marked *