AmazonS3 putObject 与 InputStream 长度示例

我正在使用 Java 将一个文件上传到 S3——这是我目前得到的:

AmazonS3 s3 = new AmazonS3Client(new BasicAWSCredentials("XX","YY"));


List<Bucket> buckets = s3.listBuckets();


s3.putObject(new PutObjectRequest(buckets.get(0).getName(), fileName, stream, new ObjectMetadata()));

文件正在上传，但是当我没有设置内容长度时会引发一个警告:

com.amazonaws.services.s3.AmazonS3Client putObject: No content length specified for stream > data.  Stream contents will be buffered in memory and could result in out of memory errors.

这是一个我正在上传的文件，stream变量是一个 InputStream，我可以从中得到如下的字节数组: IOUtils.toByteArray(stream)。

因此，当我试图像下面这样设置内容长度和 MD5(取自 给你) :

// get MD5 base64 hash
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
messageDigest.reset();
messageDigest.update(IOUtils.toByteArray(stream));
byte[] resultByte = messageDigest.digest();
String hashtext = new String(Hex.encodeHex(resultByte));


ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(IOUtils.toByteArray(stream).length);
meta.setContentMD5(hashtext);

它导致从 S3返回以下错误:

指定的 Content-MD5无效。

我做错了什么？

感谢你们的帮助！

P.S. 我使用的是 Google App Engine-我无法将文件写入磁盘或 创建临时文件，因为 AppEngine 不支持 FileOutputStream。

129963 次浏览
小开

While writing to S3, you need to specify the length of S3 object to be sure that there are no out of memory errors.

Using IOUtils.toByteArray(stream) is also prone to OOM errors because this is backed by ByteArrayOutputStream

So, the best option is to first write the inputstream to a temp file on local disk and then use that file to write to S3 by specifying the length of temp file.

小开

If all you are trying to do is solve the content length error from amazon then you could just read the bytes from the input stream to a Long and add that to the metadata.

/*
* Obtain the Content length of the Input stream for S3 header
*/
try {
InputStream is = event.getFile().getInputstream();
contentBytes = IOUtils.toByteArray(is);
} catch (IOException e) {
System.err.printf("Failed while reading bytes from %s", e.getMessage());
}


Long contentLength = Long.valueOf(contentBytes.length);


ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(contentLength);


/*
* Reobtain the tmp uploaded file as input stream
*/
InputStream inputStream = event.getFile().getInputstream();


/*
* Put the object in S3
*/
try {


s3client.putObject(new PutObjectRequest(bucketName, keyName, inputStream, metadata));


} catch (AmazonServiceException ase) {
System.out.println("Error Message:    " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code:   " + ase.getErrorCode());
System.out.println("Error Type:       " + ase.getErrorType());
System.out.println("Request ID:       " + ase.getRequestId());
} catch (AmazonClientException ace) {
System.out.println("Error Message: " + ace.getMessage());
} finally {
if (inputStream != null) {
inputStream.close();
}
}

You'll need to read the input stream twice using this exact method so if you are uploading a very large file you might need to look at reading it once into an array and then reading it from there.

小开
最佳答案

Because the original question was never answered, and I had to run into this same problem, the solution for the MD5 problem is that S3 doesn't want the Hex encoded MD5 string we normally think about.

Instead, I had to do this. 

// content is a passed in InputStream
byte[] resultByte = DigestUtils.md5(content);
String streamMD5 = new String(Base64.encodeBase64(resultByte));
metaData.setContentMD5(streamMD5);

Essentially what they want for the MD5 value is the Base64 encoded raw MD5 byte-array, not the Hex string. When I switched to this it started working great for me.

小开

i am actually doing somewhat same thing but on my AWS S3 storage:-

Code for servlet which is receiving uploaded file:-

import java.io.IOException;
import java.io.PrintWriter;
import java.util.List;


import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;


import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;


import com.src.code.s3.S3FileUploader;


public class FileUploadHandler extends HttpServlet {


protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doPost(request, response);
}


protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
PrintWriter out = response.getWriter();


try{
List<FileItem> multipartfiledata = new ServletFileUpload(new DiskFileItemFactory()).parseRequest(request);


//upload to S3
S3FileUploader s3 = new S3FileUploader();
String result = s3.fileUploader(multipartfiledata);


out.print(result);
} catch(Exception e){
System.out.println(e.getMessage());
}
}
}

Code which is uploading this data as AWS object:-

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.List;
import java.util.UUID;


import org.apache.commons.fileupload.FileItem;


import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3Object;


public class S3FileUploader {




private static String bucketName     = "***NAME OF YOUR BUCKET***";
private static String keyName        = "Object-"+UUID.randomUUID();


public String fileUploader(List<FileItem> fileData) throws IOException {
AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
String result = "Upload unsuccessfull because ";
try {


S3Object s3Object = new S3Object();


ObjectMetadata omd = new ObjectMetadata();
omd.setContentType(fileData.get(0).getContentType());
omd.setContentLength(fileData.get(0).getSize());
omd.setHeader("filename", fileData.get(0).getName());


ByteArrayInputStream bis = new ByteArrayInputStream(fileData.get(0).get());


s3Object.setObjectContent(bis);
s3.putObject(new PutObjectRequest(bucketName, keyName, bis, omd));
s3Object.close();


result = "Uploaded Successfully.";
} catch (AmazonServiceException ase) {
System.out.println("Caught an AmazonServiceException, which means your request made it to Amazon S3, but was "
+ "rejected with an error response for some reason.");


System.out.println("Error Message:    " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code:   " + ase.getErrorCode());
System.out.println("Error Type:       " + ase.getErrorType());
System.out.println("Request ID:       " + ase.getRequestId());


result = result + ase.getMessage();
} catch (AmazonClientException ace) {
System.out.println("Caught an AmazonClientException, which means the client encountered an internal error while "
+ "trying to communicate with S3, such as not being able to access the network.");


result = result + ace.getMessage();
}catch (Exception e) {
result = result + e.getMessage();
}


return result;
}
}

Note :- I am using aws properties file for credentials.

Hope this helps.

小开

For uploading, the S3 SDK has two putObject methods:

PutObjectRequest(String bucketName, String key, File file)

and

PutObjectRequest(String bucketName, String key, InputStream input, ObjectMetadata metadata)

The inputstream+ObjectMetadata method needs a minimum metadata of Content Length of your inputstream. If you don't, then it will buffer in-memory to get that information, this could cause OOM. Alternatively, you could do your own in-memory buffering to get the length, but then you need to get a second inputstream.

Not asked by the OP (limitations of his environment), but for someone else, such as me. I find it easier, and safer (if you have access to temp file), to write the inputstream to a temp file, and put the temp file. No in-memory buffer, and no requirement to create a second inputstream.

AmazonS3 s3Service = new AmazonS3Client(awsCredentials);
File scratchFile = File.createTempFile("prefix", "suffix");
try {
FileUtils.copyInputStreamToFile(inputStream, scratchFile);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, id, scratchFile);
PutObjectResult putObjectResult = s3Service.putObject(putObjectRequest);


} finally {
if(scratchFile.exists()) {
scratchFile.delete();
}
}
小开

I've created a library that uses multipart uploads in the background to avoid buffering everything in memory and also doesn't write to disk: https://github.com/alexmojaki/s3-stream-upload

小开

adding log4j-1.2.12.jar file has resolved the issue for me

小开

Just passing the file object to the putobject method worked for me. If you are getting a stream, try writing it to a temp file before passing it on to S3. 

amazonS3.putObject(bucketName, id,fileObject);

I am using Aws SDK v1.11.414

The answer at https://stackoverflow.com/a/35904801/2373449 helped me


提交答案