我可以流一个文件上传到 S3没有一个内容长度的标题？

我在一台内存有限的机器上工作，我想以流的方式上传一个动态生成的(非磁盘生成的)文件到 S3。换句话说，我不知道文件大小时，我开始上传，但我会知道它的结束。一般来说，PUT 请求有一个 Content-Llength 头，但是也许有一种方法可以绕过这个问题，比如使用 multipart 或 chunked content-type。

S3可以支持流式上传:

Http://blog.odonnell.nu/posts/streaming-uploads-s3-python-and-poster/

我的问题是，我可以完成同样的事情，而不必指定文件长度在开始上传？

77594 次浏览
小开

Refer more on HTTP multi-part enitity requests. You can send a file as chunks of data to the target.

小开
最佳答案

You have to upload your file in 5MiB+ chunks via S3's multipart API. Each of those chunks requires a Content-Length but you can avoid loading huge amounts of data (100MiB+) into memory.

  • Initiate S3 Multipart Upload.
  • Gather data into a buffer until that buffer reaches S3's lower chunk-size limit (5MiB). Generate MD5 checksum while building up the buffer.
  • Upload that buffer as a Part, store the ETag (read the docs on that one).
  • Once you reach EOF of your data, upload the last chunk (which can be smaller than 5MiB).
  • Finalize the Multipart Upload.

S3 allows up to 10,000 parts. So by choosing a part-size of 5MiB you will be able to upload dynamic files of up to 50GiB. Should be enough for most use-cases.

However: If you need more, you have to increase your part-size. Either by using a higher part-size (10MiB for example) or by increasing it during the upload.

First 25 parts:   5MiB (total:  125MiB)
Next 25 parts:   10MiB (total:  375MiB)
Next 25 parts:   25MiB (total:    1GiB)
Next 25 parts:   50MiB (total: 2.25GiB)
After that:     100MiB

This will allow you to upload files of up to 1TB (S3's limit for a single file is 5TB right now) without wasting memory unnecessarily.

A note on your link to Sean O'Donnells blog:

His problem is different from yours - he knows and uses the Content-Length before the upload. He wants to improve on this situation: Many libraries handle uploads by loading all data from a file into memory. In pseudo-code that would be something like this:

data = File.read(file_name)
request = new S3::PutFileRequest()
request.setHeader('Content-Length', data.size)
request.setBody(data)
request.send()

His solution does it by getting the Content-Length via the filesystem-API. He then streams the data from disk into the request-stream. In pseudo-code:

upload = new S3::PutFileRequestStream()
upload.writeHeader('Content-Length', File.getSize(file_name))
upload.flushHeader()


input = File.open(file_name, File::READONLY_FLAG)


while (data = input.read())
input.write(data)
end


upload.flush()
upload.close()
小开

Putting this answer here for others in case it helps:

If you don't know the length of the data you are streaming up to S3, you can use S3FileInfo and its OpenWrite() method to write arbitrary data into S3.

var fileInfo = new S3FileInfo(amazonS3Client, "MyBucket", "streamed-file.txt");


using (var outputStream = fileInfo.OpenWrite())
{
using (var streamWriter = new StreamWriter(outputStream))
{
streamWriter.WriteLine("Hello world");
// You can do as many writes as you want here
}
}
小开

If you are using Node.js you can use a plugin like s3-streaming-upload to accomplish this quite easily.

小开

You can use the gof3r command-line tool to just stream linux pipes:

$ tar -czf - <my_dir/> | gof3r put --bucket <s3_bucket> --key <s3_object>
小开

reference to :https://github.com/aws/aws-cli/pull/903

Here is a synopsis: For uploading a stream from stdin to s3, use: aws s3 cp - s3://my-bucket/stream

For downloading an s3 object as a stdout stream, use: aws s3 cp s3://my-bucket/stream -

So for example, if I had the object s3://my-bucket/stream, I could run this command: aws s3 cp s3://my-bucket/stream - | aws s3 cp - s3://my-bucket/new-stream

my cmd:

echo "ccc" | aws --endpoint-url=http://172.22.222.245:80 --no-verify-ssl s3 cp - s3://test-bucket/ccc


提交答案