AWS CLI Max Concurrent Requests Tuning

In this post I would like to go over how I tuned a test server for copying / syncing files from the local filesystem to S3 over the internet. If you ever had the task of doing this, you will notice that as the file count grows, so does the time it takes to upload the files to S3. After some web searching I found out that AWS allows you to tune the config to allow more concurrency than default.
AWS CLI S3 Config

The parameter that we will be playing with is max_concurrent_requests
This has a default value of 10, which allows only 10 requests to the AWS API for S3. Lets see if we can make some changes to that value and get some performance gains. My test setup is as follows:

I have 56 102MB files in the test directory:

For the first test I am going to run aws s3 sync with no changes, so out of the box it should have 10 max_concurrent_requests. Lets use the Linux time command to gather the time result to copy all 56 files to S3. I will delete the folder on S3 with each iteration to keep the test the same. You can also view the 443 requests via netstat and count them as well to show whats going on. In all the tests my best result was 250. So as you can see you will need to play with the settings to get the best result, these settings will change along with the server specs.

1. 1m25.919s with the default configuration:

2. Now lets set the max conqurent requests to 20 and try again, you can do this with the command below, after running we can see a little gain.

3. Bumped up to 50 shows a bit more gain:

4. Bumped up to 100, I start to notice that we lost some speed:

5. Bumped up to 250 we see the best result so far:

6. Bumped up to 500, we lose performance, most likely due to the machine resources.

So to wrap up, you can tune the amount of concurrent requests allowed from the aws cli to s3, you will need to play with this setting to get the best results for your machine.

4 thoughts on “AWS CLI Max Concurrent Requests Tuning”

  1. Hi Jason, can you comment your tweaking of max_concurrent_requests for aws cli? Have you ever correlated the optimum value (250 threads in example above) with the number of CPU cores (I think 36 cores for your “2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz”)? This implies ~8 threads per core, does this sound right? Thanks.

    1. Hi Andy,
      Thanks for you comment.
      That is a great question, I have thought about it, but never really did that math, math is not my strongest subject so keep me honest here :). But I like this type of stuff! This particular sandbox I was playing with was a vmware virtual machine. I carved out 2 cores 1 socket off that 18 core 36 thread CPU. So my understanding would be that each core has 2 threads, my test system would be 4 threads total? So not sure that correlates but open to here what you think. I do see where you are going with this in your comment, 36 cores x 8 = 288, which is very close to 250.

      I think this is the model:

      What do you think?

  2. I would think the more important factor to performance than CPU for this would be available network bandwidth. Moving data is more often I/O bound than CPU bound. So its likely that the bottleneck you experienced at 250 connections was where you saturated your available network bandwidth, either at the virtualized bus or on your internet connection. A better test would be to run this test inside an AWS VPC on an EC2 with 5Gbit to 10Gbit network with a S3 endpoint in the VPC to maximize your throughput. Also, for S3, when putting data, the object key name is used in selecting the S3 “shard”. I’m not sure if that is the terminology they use. If the names are not sufficiently entropic then you may run into a situation where the connections are not well balanced across the available S3 shards and performance could suffer from that as well.

    1. Thank you for this comment, this is valuable information. I think you are 100% correct that network bandwidth is the key factor here.

Leave a Reply

Your email address will not be published. Required fields are marked *