从 boto3检索 S3 bucket 中的子文件夹名称

首先，在 S3中没有真正的文件夹概念。 您肯定可以有一个文件@'/folder/subfolder/myfile.txt'，没有文件夹或子文件夹。

要在 S3中“模拟”文件夹，必须创建一个名称末尾带有“/”的空文件(参见 Amazon S3 boto-如何创建一个文件夹？)

对于您的问题，您可能应该使用带有两个参数的方法 get_all_keys: prefix和 delimiter

Https://github.com/boto/boto/blob/develop/boto/s3/bucket.py#l427

for key in bucket.get_all_keys(prefix='first-level/', delimiter='/'):
print(key.name)

S3是一个对象存储器，它没有真正的目录结构。 人们希望有一个目录结构的原因之一，因为他们可以维护/prune/向应用程序添加一个树。对于 S3，可以将这种结构视为索引或搜索标记的排序。

要在 S3中操作对象，您需要 boto3.client 或 boto3.resource，例如。 列出所有对象

import boto3
s3 = boto3.client("s3")
all_objects = s3.list_objects(Bucket = 'bucket-name')

Http://boto3.readthedocs.org/en/latest/reference/services/s3.html#s3.client.list_objects

实际上，如果 s3对象名称是使用“/”分隔符存储的。最新版本的 list _ Objects (list _ Objects _ v2)允许您将响应限制为以指定前缀开头的键。

将项限制为某些子文件夹下的项:

    import boto3
s3 = boto3.client("s3")
response = s3.list_objects_v2(
Bucket=BUCKET,
Prefix ='DIR1/DIR2',
MaxKeys=100 )

文件

另一种选择是使用 python os.path 函数来提取文件夹前缀。问题是这需要从不需要的目录中列出对象。

import os
s3_key = 'first-level/1456753904534/part-00014'
filename = os.path.basename(s3_key)
foldername = os.path.dirname(s3_key)


# if you are not using conventional delimiter like '#'
s3_key = 'first-level#1456753904534#part-00014'
filename = s3_key.split("#")[-1]

提醒一下 boto3: boto3.resource 是一个不错的高级 API。使用 boto3.client 比使用 boto3.resource 有利有弊。如果您开发内部共享库，那么使用 boto3.resource 将在所使用的资源之上提供一个黑盒层。

下面的代码只返回 s3 bucket 中“文件夹”中的“子文件夹”。

import boto3
bucket = 'my-bucket'
#Make sure you provide / in the end
prefix = 'prefix-name-with-slash/'


client = boto3.client('s3')
result = client.list_objects(Bucket=bucket, Prefix=prefix, Delimiter='/')
for o in result.get('CommonPrefixes'):
print 'sub folder : ', o.get('Prefix')

有关详细信息，请参阅 https://github.com/boto/boto3/issues/134

我花了很多时间才弄明白，但最后这里有一个简单的方法，可以使用 boto3列出 S3 bucket 中子文件夹的内容。希望能有帮助

prefix = "folderone/foldertwo/"
s3 = boto3.resource('s3')
bucket = s3.Bucket(name="bucket_name_here")
FilesNotFound = True
for obj in bucket.objects.filter(Prefix=prefix):
print('{0}:{1}'.format(bucket.name, obj.key))
FilesNotFound = False
if FilesNotFound:
print("ALERT", "No file in {0}/{1}".format(bucket, prefix))

AWS cli 在您运行 aws s3 ls s3://my-bucket/时执行此操作(可能不需要获取和迭代 bucket 中的所有键) ，因此我认为一定有使用 boto3的方法。

Https://github.com/aws/aws-cli/blob/0fedc4c1b6a7aee13e2ed10c3ada778c702c22c3/awscli/customizations/s3/subcommands.py#l499

看起来他们确实使用了前缀和分隔符——我能够编写一个函数，通过稍微修改一下代码，就可以得到桶的根级别的所有目录:

def list_folders_in_bucket(bucket):
paginator = boto3.client('s3').get_paginator('list_objects')
folders = []
iterator = paginator.paginate(Bucket=bucket, Prefix='', Delimiter='/', PaginationConfig={'PageSize': None})
for response_data in iterator:
prefixes = response_data.get('CommonPrefixes', [])
for prefix in prefixes:
prefix_name = prefix['Prefix']
if prefix_name.endswith('/'):
folders.append(prefix_name.rstrip('/'))
return folders

以下对我有用... S3对象:

s3://bucket/
form1/
section11/
file111
file112
section12/
file121
form2/
section21/
file211
file112
section22/
file221
file222
...
...
...

使用:

from boto3.session import Session
s3client = session.client('s3')
resp = s3client.list_objects(Bucket=bucket, Prefix='', Delimiter="/")
forms = [x['Prefix'] for x in resp['CommonPrefixes']]

我们得到:

form1/
form2/
...

配合:

resp = s3client.list_objects(Bucket=bucket, Prefix='form1/', Delimiter="/")
sections = [x['Prefix'] for x in resp['CommonPrefixes']]

我们得到:

form1/section11/
form1/section12/

我有同样的问题，但设法解决它使用 boto3.client和 list_objects_v2与 Bucket和 StartAfter参数。

s3client = boto3.client('s3')
bucket = 'my-bucket-name'
startAfter = 'firstlevelFolder/secondLevelFolder'


theobjects = s3client.list_objects_v2(Bucket=bucket, StartAfter=startAfter )
for object in theobjects['Contents']:
print object['Key']

上述代码的输出结果将显示如下:

firstlevelFolder/secondLevelFolder/item1
firstlevelFolder/secondLevelFolder/item2

Boto3 list _ Objects _ v2文档

为了只去掉 secondLevelFolder的目录名，我使用了 python 方法 split():

s3client = boto3.client('s3')
bucket = 'my-bucket-name'
startAfter = 'firstlevelFolder/secondLevelFolder'


theobjects = s3client.list_objects_v2(Bucket=bucket, StartAfter=startAfter )
for object in theobjects['Contents']:
direcoryName = object['Key'].encode("string_escape").split('/')
print direcoryName[1]

上述代码的输出结果将显示如下:

secondLevelFolder
secondLevelFolder

Python split ()文档

如果您想获取目录名和内容项目名，那么将打印行替换为以下内容:

print "{}/{}".format(fileName[1], fileName[2])

输出如下:

secondLevelFolder/item2
secondLevelFolder/item2

希望这个能帮上忙

S3最大的实现是没有文件夹/目录，只有键。明显的文件夹结构只是前面的文件名成为“关键”，所以列出 myBucket的 some/path/to/the/file/的内容，你可以尝试:

s3 = boto3.client('s3')
for obj in s3.list_objects_v2(Bucket="myBucket", Prefix="some/path/to/the/file/")['Contents']:
print(obj['Key'])

它会给你类似这样的东西:

some/path/to/the/file/yo.jpg
some/path/to/the/file/meAndYou.gif
...

简短的回答 :

• 使用 Delimiter='/'。这样可以避免对桶执行递归清单。这里的一些答案错误地建议执行完整的列表并使用一些字符串操作来检索目录名称。这可能会非常低效。请记住，S3实际上对一个 bucket 可以包含的对象数量没有限制。所以，想象一下，在 bar/和 foo/之间，有一万亿个对象: 要等很长时间才能得到 ['bar/', 'foo/']。

• 使用 Paginators。出于同样的原因(S3是一个工程师的无穷大近似值) ，您可以通过页面进行 必须的列表，并避免将所有列表存储在内存中。相反，将您的“ list”视为迭代器，并处理它生成的流。

• 使用 boto3.client，而不是 boto3.resource。resource版本似乎不能很好地处理 Delimiter选项。如果您有一个资源，比如 bucket = boto3.resource('s3').Bucket(name)，您可以使用: bucket.meta.client获得相应的客户端。

长回答 :

下面是我用于简单桶(无版本处理)的迭代器。

import os
import boto3
from collections import namedtuple
from operator import attrgetter




S3Obj = namedtuple('S3Obj', ['key', 'mtime', 'size', 'ETag'])




def s3list(bucket, path, start=None, end=None, recursive=True, list_dirs=True,
list_objs=True, limit=None):
"""
Iterator that lists a bucket's objects under path, (optionally) starting with
start and ending before end.


If recursive is False, then list only the "depth=0" items (dirs and objects).


If recursive is True, then list recursively all objects (no dirs).


Args:
bucket:
a boto3.resource('s3').Bucket().
path:
a directory in the bucket.
start:
optional: start key, inclusive (may be a relative path under path, or
absolute in the bucket)
end:
optional: stop key, exclusive (may be a relative path under path, or
absolute in the bucket)
recursive:
optional, default True. If True, lists only objects. If False, lists
only depth 0 "directories" and objects.
list_dirs:
optional, default True. Has no effect in recursive listing. On
non-recursive listing, if False, then directories are omitted.
list_objs:
optional, default True. If False, then directories are omitted.
limit:
optional. If specified, then lists at most this many items.


Returns:
an iterator of S3Obj.


Examples:
# set up
>>> s3 = boto3.resource('s3')
... bucket = s3.Bucket('bucket-name')


# iterate through all S3 objects under some dir
>>> for p in s3list(bucket, 'some/dir'):
...     print(p)


# iterate through up to 20 S3 objects under some dir, starting with foo_0010
>>> for p in s3list(bucket, 'some/dir', limit=20, start='foo_0010'):
...     print(p)


# non-recursive listing under some dir:
>>> for p in s3list(bucket, 'some/dir', recursive=False):
...     print(p)


# non-recursive listing under some dir, listing only dirs:
>>> for p in s3list(bucket, 'some/dir', recursive=False, list_objs=False):
...     print(p)
"""
kwargs = dict()
if start is not None:
if not start.startswith(path):
start = os.path.join(path, start)
# note: need to use a string just smaller than start, because
# the list_object API specifies that start is excluded (the first
# result is *after* start).
kwargs.update(Marker=__prev_str(start))
if end is not None:
if not end.startswith(path):
end = os.path.join(path, end)
if not recursive:
kwargs.update(Delimiter='/')
if not path.endswith('/'):
path += '/'
kwargs.update(Prefix=path)
if limit is not None:
kwargs.update(PaginationConfig={'MaxItems': limit})


paginator = bucket.meta.client.get_paginator('list_objects')
for resp in paginator.paginate(Bucket=bucket.name, **kwargs):
q = []
if 'CommonPrefixes' in resp and list_dirs:
q = [S3Obj(f['Prefix'], None, None, None) for f in resp['CommonPrefixes']]
if 'Contents' in resp and list_objs:
q += [S3Obj(f['Key'], f['LastModified'], f['Size'], f['ETag']) for f in resp['Contents']]
# note: even with sorted lists, it is faster to sort(a+b)
# than heapq.merge(a, b) at least up to 10K elements in each list
q = sorted(q, key=attrgetter('key'))
if limit is not None:
q = q[:limit]
limit -= len(q)
for p in q:
if end is not None and p.key >= end:
return
yield p




def __prev_str(s):
if len(s) == 0:
return s
s, c = s[:-1], ord(s[-1])
if c > 0:
s += chr(c - 1)
s += ''.join(['\u7FFF' for _ in range(10)])
return s

测试 :

import concurrent
def genkeys(top='tmp/test', n=2000):
for k in range(n):
if k % 100 == 0:
print(k)
for name in [
os.path.join(top, 'dirs', f'{k:04d}_dir', 'foo'),
os.path.join(top, 'mixed', f'{k:04d}_dir', 'foo'),
os.path.join(top, 'mixed', f'{k:04d}_foo_a'),
os.path.join(top, 'mixed', f'{k:04d}_foo_b'),
]:
yield name




with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
executor.map(lambda name: bucket.put_object(Key=name, Body='hi\n'.encode()), genkeys())

./dirs/0000_dir/foo
./dirs/0001_dir/foo
./dirs/0002_dir/foo
...
./dirs/1999_dir/foo
./mixed/0000_dir/foo
./mixed/0000_foo_a
./mixed/0000_foo_b
./mixed/0001_dir/foo
./mixed/0001_foo_a
./mixed/0001_foo_b
./mixed/0002_dir/foo
./mixed/0002_foo_a
./mixed/0002_foo_b
...
./mixed/1999_dir/foo
./mixed/1999_foo_a
./mixed/1999_foo_b