在Python中，用空格分隔字符串——保留带引号的子字符串

小开

最佳答案

你需要内置的shlex模块中的split。

>>> import shlex
>>> shlex.split('this is "a test"')
['this', 'is', 'a test']

这应该是你想要的效果。

如果你想保留引号，那么你可以传递posix=False kwarg。

>>> shlex.split('this is "a test"', posix=False)
['this', 'is', '"a test"']

小开

看一下shlex模块，特别是shlex.split。

>>> import shlex
>>> shlex.split('This is "a test"')
['This', 'is', 'a test']

小开

试试这个:

  def adamsplit(s):
result = []
inquotes = False
for substring in s.split('"'):
if not inquotes:
result.extend(substring.split())
else:
result.append(substring)
inquotes = not inquotes
return result

一些测试字符串:

'This is "a test"' -> ['This', 'is', 'a test']
'"This is \'a test\'"' -> ["This is 'a test'"]

小开

如果你不关心子字符串

>>> 'a short sized string with spaces '.split()

性能:

>>> s = " ('a short sized string with spaces '*100).split() "
>>> t = timeit.Timer(stmt=s)
>>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
171.39 usec/pass

或者字符串模块

>>> from string import split as stringsplit;
>>> stringsplit('a short sized string with spaces '*100)

性能:String模块的性能似乎比字符串方法更好

>>> s = "stringsplit('a short sized string with spaces '*100)"
>>> t = timeit.Timer(s, "from string import split as stringsplit")
>>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
154.88 usec/pass

或者你可以使用RE引擎

>>> from re import split as resplit
>>> regex = '\s+'
>>> medstring = 'a short sized string with spaces '*100
>>> resplit(regex, medstring)

性能

>>> s = "resplit(regex, medstring)"
>>> t = timeit.Timer(s, "from re import split as resplit; regex='\s+'; medstring='a short sized string with spaces '*100")
>>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
540.21 usec/pass

对于非常长的字符串，您不应该将整个字符串加载到内存中，而是将行分开或使用迭代循环

小开

由于这个问题带有正则表达式，我决定尝试正则表达式方法。我首先将引号部分中的所有空格替换为\x00，然后按空格分割，然后将\x00替换回每个部分中的空格。

这两个版本都做同样的事情，但是splitter比splitter2更具可读性。

import re


s = 'this is "a test" some text "another test"'


def splitter(s):
def replacer(m):
return m.group(0).replace(" ", "\x00")
parts = re.sub('".+?"', replacer, s).split()
parts = [p.replace("\x00", " ") for p in parts]
return parts


def splitter2(s):
return [p.replace("\x00", " ") for p in re.sub('".+?"', lambda m: m.group(0).replace(" ", "\x00"), s).split()]


print splitter2(s)

小开

我看到这里的正则表达式方法看起来很复杂和/或错误。这让我很惊讶，因为正则表达式语法可以很容易地描述“空格或引号包围的东西”，而且大多数正则表达式引擎(包括Python的)都可以在正则表达式上进行拆分。所以如果你要使用正则表达式，为什么不直接说出你的意思呢?：

test = 'this is "a test"'  # or "this is 'a test'"
# pieces = [p for p in re.split("( |[\\\"'].*[\\\"'])", test) if p.strip()]
# From comments, use this:
pieces = [p for p in re.split("( |\\\".*?\\\"|'.*?')", test) if p.strip()]

解释:

[\\\"'] = double-quote or single-quote
.* = anything
( |X) = space or X
.strip() = remove space and empty-string separators

不过，Shlex可能提供更多的特性。

小开

根据你的用例，你可能还想检查csv模块:

import csv
lines = ['this is "a string"', 'and more "stuff"']
for row in csv.reader(lines, delimiter=" "):
print(row)

输出:

['this', 'is', 'a string']
['and', 'more', 'stuff']

小开

嗯，好像找不到“回复”按钮了…不管怎样，这个答案是基于Kate的方法，但正确地分割了包含转义引号的子字符串，并删除了子字符串的开始和结束引号:

  [i.strip('"').strip("'") for i in re.split(r'(\s+|(?<!\\)".*?(?<!\\)"|(?<!\\)\'.*?(?<!\\)\')', string) if i.strip()]

这适用于'This is " a \\\"test\\\"\\\'s substring"'这样的字符串(不幸的是，疯狂的标记是必要的，以防止Python删除转义)。

如果返回列表中的字符串不需要转义，您可以使用稍微修改过的函数版本:

[i.strip('"').strip("'").decode('string_escape') for i in re.split(r'(\s+|(?<!\\)".*?(?<!\\)"|(?<!\\)\'.*?(?<!\\)\')', string) if i.strip()]

小开

要解决某些Python 2版本中的unicode问题，我建议:

from shlex import split as _split
split = lambda a: [b.decode('utf-8') for b in _split(a.encode('utf-8'))]

小开

我用shlex。拆分来处理七千万行鱿鱼日志，太慢了。所以我换成了re。

请尝试这个，如果你有性能问题与shlex。

import re


def line_split(line):
return re.findall(r'[^"\s]\S*|".+?"', line)

小开

我建议:

测试字符串:

s = 'abc "ad" \'fg\' "kk\'rdt\'" zzz"34"zzz "" \'\''

也要捕捉""和":

import re
re.findall(r'"[^"]*"|\'[^\']*\'|[^"\'\s]+',s)

结果:

['abc', '"ad"', "'fg'", '"kk\'rdt\'"', 'zzz', '"34"', 'zzz', '""', "''"]

忽略空的""和":

import re
re.findall(r'"[^"]+"|\'[^\']+\'|[^"\'\s]+',s)

结果:

['abc', '"ad"', "'fg'", '"kk\'rdt\'"', 'zzz', '"34"', 'zzz']

小开

要保留引号，使用这个函数:

def getArgs(s):
args = []
cur = ''
inQuotes = 0
for char in s.strip():
if char == ' ' and not inQuotes:
args.append(cur)
cur = ''
elif char == '"' and not inQuotes:
inQuotes = 1
cur += char
elif char == '"' and inQuotes:
inQuotes = 0
cur += char
else:
cur += char
args.append(cur)
return args

小开

不同答案的速度测试:

import re
import shlex
import csv


line = 'this is "a test"'


%timeit [p for p in re.split("( |\\\".*?\\\"|'.*?')", line) if p.strip()]
100000 loops, best of 3: 5.17 µs per loop


%timeit re.findall(r'[^"\s]\S*|".+?"', line)
100000 loops, best of 3: 2.88 µs per loop


%timeit list(csv.reader([line], delimiter=" "))
The slowest run took 9.62 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 2.4 µs per loop


%timeit shlex.split(line)
10000 loops, best of 3: 50.2 µs per loop

小开

被接受的shlex方法的主要问题是，它不会忽略带引号的子字符串外的转义字符，并且在某些极端情况下会给出略微意想不到的结果。

我有下面的用例，其中我需要一个split函数来分割输入字符串，以便保留单引号或双引号子字符串，并能够在这样的子字符串中转义引号。未加引号的字符串中的引号不应与任何其他字符区别对待。一些带有预期输出的示例测试用例:

 input string        | expected output
===============================================
'abc def'           | ['abc', 'def']
"abc \\s def"       | ['abc', '\\s', 'def']
'"abc def" ghi'     | ['abc def', 'ghi']
"'abc def' ghi"     | ['abc def', 'ghi']
'"abc \\" def" ghi' | ['abc " def', 'ghi']
"'abc \\' def' ghi" | ["abc ' def", 'ghi']
"'abc \\s def' ghi" | ['abc \\s def', 'ghi']
'"abc \\s def" ghi' | ['abc \\s def', 'ghi']
'"" test'           | ['', 'test']
"'' test"           | ['', 'test']
"abc'def"           | ["abc'def"]
"abc'def'"          | ["abc'def'"]
"abc'def' ghi"      | ["abc'def'", 'ghi']
"abc'def'ghi"       | ["abc'def'ghi"]
'abc"def'           | ['abc"def']
'abc"def"'          | ['abc"def"']
'abc"def" ghi'      | ['abc"def"', 'ghi']
'abc"def"ghi'       | ['abc"def"ghi']
"r'AA' r'.*_xyz$'"  | ["r'AA'", "r'.*_xyz$'"]
'abc"def ghi"'      | ['abc"def ghi"']
'abc"def ghi""jkl"' | ['abc"def ghi""jkl"']
'a"b c"d"e"f"g h"'  | ['a"b c"d"e"f"g h"']
'c="ls /" type key' | ['c="ls /"', 'type', 'key']
"abc'def ghi'"      | ["abc'def ghi'"]
"c='ls /' type key" | ["c='ls /'", 'type', 'key']

我最终使用以下函数来分割字符串，以便所有输入字符串的预期输出结果:

import re


def quoted_split(s):
def strip_quotes(s):
if s and (s[0] == '"' or s[0] == "'") and s[0] == s[-1]:
return s[1:-1]
return s
return [strip_quotes(p).replace('\\"', '"').replace("\\'", "'") \
for p in re.findall(r'(?:[^"\s]*"(?:\\.|[^"])*"[^"\s]*)+|(?:[^\'\s]*\'(?:\\.|[^\'])*\'[^\'\s]*)+|[^\s]+', s)]

这可不漂亮;但它确实有效。下面的测试应用程序检查其他方法(目前是shlex和csv)和自定义分割实现的结果:

#!/bin/python2.7


import csv
import re
import shlex


from timeit import timeit


def test_case(fn, s, expected):
try:
if fn(s) == expected:
print '[ OK ] %s -> %s' % (s, fn(s))
else:
print '[FAIL] %s -> %s' % (s, fn(s))
except Exception as e:
print '[FAIL] %s -> exception: %s' % (s, e)


def test_case_no_output(fn, s, expected):
try:
fn(s)
except:
pass


def test_split(fn, test_case_fn=test_case):
test_case_fn(fn, 'abc def', ['abc', 'def'])
test_case_fn(fn, "abc \\s def", ['abc', '\\s', 'def'])
test_case_fn(fn, '"abc def" ghi', ['abc def', 'ghi'])
test_case_fn(fn, "'abc def' ghi", ['abc def', 'ghi'])
test_case_fn(fn, '"abc \\" def" ghi', ['abc " def', 'ghi'])
test_case_fn(fn, "'abc \\' def' ghi", ["abc ' def", 'ghi'])
test_case_fn(fn, "'abc \\s def' ghi", ['abc \\s def', 'ghi'])
test_case_fn(fn, '"abc \\s def" ghi', ['abc \\s def', 'ghi'])
test_case_fn(fn, '"" test', ['', 'test'])
test_case_fn(fn, "'' test", ['', 'test'])
test_case_fn(fn, "abc'def", ["abc'def"])
test_case_fn(fn, "abc'def'", ["abc'def'"])
test_case_fn(fn, "abc'def' ghi", ["abc'def'", 'ghi'])
test_case_fn(fn, "abc'def'ghi", ["abc'def'ghi"])
test_case_fn(fn, 'abc"def', ['abc"def'])
test_case_fn(fn, 'abc"def"', ['abc"def"'])
test_case_fn(fn, 'abc"def" ghi', ['abc"def"', 'ghi'])
test_case_fn(fn, 'abc"def"ghi', ['abc"def"ghi'])
test_case_fn(fn, "r'AA' r'.*_xyz$'", ["r'AA'", "r'.*_xyz$'"])
test_case_fn(fn, 'abc"def ghi"', ['abc"def ghi"'])
test_case_fn(fn, 'abc"def ghi""jkl"', ['abc"def ghi""jkl"'])
test_case_fn(fn, 'a"b c"d"e"f"g h"', ['a"b c"d"e"f"g h"'])
test_case_fn(fn, 'c="ls /" type key', ['c="ls /"', 'type', 'key'])
test_case_fn(fn, "abc'def ghi'", ["abc'def ghi'"])
test_case_fn(fn, "c='ls /' type key", ["c='ls /'", 'type', 'key'])


def csv_split(s):
return list(csv.reader([s], delimiter=' '))[0]


def re_split(s):
def strip_quotes(s):
if s and (s[0] == '"' or s[0] == "'") and s[0] == s[-1]:
return s[1:-1]
return s
return [strip_quotes(p).replace('\\"', '"').replace("\\'", "'") for p in re.findall(r'(?:[^"\s]*"(?:\\.|[^"])*"[^"\s]*)+|(?:[^\'\s]*\'(?:\\.|[^\'])*\'[^\'\s]*)+|[^\s]+', s)]


if __name__ == '__main__':
print 'shlex\n'
test_split(shlex.split)
print


print 'csv\n'
test_split(csv_split)
print


print 're\n'
test_split(re_split)
print


iterations = 100
setup = 'from __main__ import test_split, test_case_no_output, csv_split, re_split\nimport shlex, re'
def benchmark(method, code):
print '%s: %.3fms per iteration' % (method, (1000 * timeit(code, setup=setup, number=iterations) / iterations))
benchmark('shlex', 'test_split(shlex.split, test_case_no_output)')
benchmark('csv', 'test_split(csv_split, test_case_no_output)')
benchmark('re', 'test_split(re_split, test_case_no_output)')

输出:

shlex


[ OK ] abc def -> ['abc', 'def']
[FAIL] abc \s def -> ['abc', 's', 'def']
[ OK ] "abc def" ghi -> ['abc def', 'ghi']
[ OK ] 'abc def' ghi -> ['abc def', 'ghi']
[ OK ] "abc \" def" ghi -> ['abc " def', 'ghi']
[FAIL] 'abc \' def' ghi -> exception: No closing quotation
[ OK ] 'abc \s def' ghi -> ['abc \\s def', 'ghi']
[ OK ] "abc \s def" ghi -> ['abc \\s def', 'ghi']
[ OK ] "" test -> ['', 'test']
[ OK ] '' test -> ['', 'test']
[FAIL] abc'def -> exception: No closing quotation
[FAIL] abc'def' -> ['abcdef']
[FAIL] abc'def' ghi -> ['abcdef', 'ghi']
[FAIL] abc'def'ghi -> ['abcdefghi']
[FAIL] abc"def -> exception: No closing quotation
[FAIL] abc"def" -> ['abcdef']
[FAIL] abc"def" ghi -> ['abcdef', 'ghi']
[FAIL] abc"def"ghi -> ['abcdefghi']
[FAIL] r'AA' r'.*_xyz$' -> ['rAA', 'r.*_xyz$']
[FAIL] abc"def ghi" -> ['abcdef ghi']
[FAIL] abc"def ghi""jkl" -> ['abcdef ghijkl']
[FAIL] a"b c"d"e"f"g h" -> ['ab cdefg h']
[FAIL] c="ls /" type key -> ['c=ls /', 'type', 'key']
[FAIL] abc'def ghi' -> ['abcdef ghi']
[FAIL] c='ls /' type key -> ['c=ls /', 'type', 'key']


csv


[ OK ] abc def -> ['abc', 'def']
[ OK ] abc \s def -> ['abc', '\\s', 'def']
[ OK ] "abc def" ghi -> ['abc def', 'ghi']
[FAIL] 'abc def' ghi -> ["'abc", "def'", 'ghi']
[FAIL] "abc \" def" ghi -> ['abc \\', 'def"', 'ghi']
[FAIL] 'abc \' def' ghi -> ["'abc", "\\'", "def'", 'ghi']
[FAIL] 'abc \s def' ghi -> ["'abc", '\\s', "def'", 'ghi']
[ OK ] "abc \s def" ghi -> ['abc \\s def', 'ghi']
[ OK ] "" test -> ['', 'test']
[FAIL] '' test -> ["''", 'test']
[ OK ] abc'def -> ["abc'def"]
[ OK ] abc'def' -> ["abc'def'"]
[ OK ] abc'def' ghi -> ["abc'def'", 'ghi']
[ OK ] abc'def'ghi -> ["abc'def'ghi"]
[ OK ] abc"def -> ['abc"def']
[ OK ] abc"def" -> ['abc"def"']
[ OK ] abc"def" ghi -> ['abc"def"', 'ghi']
[ OK ] abc"def"ghi -> ['abc"def"ghi']
[ OK ] r'AA' r'.*_xyz$' -> ["r'AA'", "r'.*_xyz$'"]
[FAIL] abc"def ghi" -> ['abc"def', 'ghi"']
[FAIL] abc"def ghi""jkl" -> ['abc"def', 'ghi""jkl"']
[FAIL] a"b c"d"e"f"g h" -> ['a"b', 'c"d"e"f"g', 'h"']
[FAIL] c="ls /" type key -> ['c="ls', '/"', 'type', 'key']
[FAIL] abc'def ghi' -> ["abc'def", "ghi'"]
[FAIL] c='ls /' type key -> ["c='ls", "/'", 'type', 'key']


re


[ OK ] abc def -> ['abc', 'def']
[ OK ] abc \s def -> ['abc', '\\s', 'def']
[ OK ] "abc def" ghi -> ['abc def', 'ghi']
[ OK ] 'abc def' ghi -> ['abc def', 'ghi']
[ OK ] "abc \" def" ghi -> ['abc " def', 'ghi']
[ OK ] 'abc \' def' ghi -> ["abc ' def", 'ghi']
[ OK ] 'abc \s def' ghi -> ['abc \\s def', 'ghi']
[ OK ] "abc \s def" ghi -> ['abc \\s def', 'ghi']
[ OK ] "" test -> ['', 'test']
[ OK ] '' test -> ['', 'test']
[ OK ] abc'def -> ["abc'def"]
[ OK ] abc'def' -> ["abc'def'"]
[ OK ] abc'def' ghi -> ["abc'def'", 'ghi']
[ OK ] abc'def'ghi -> ["abc'def'ghi"]
[ OK ] abc"def -> ['abc"def']
[ OK ] abc"def" -> ['abc"def"']
[ OK ] abc"def" ghi -> ['abc"def"', 'ghi']
[ OK ] abc"def"ghi -> ['abc"def"ghi']
[ OK ] r'AA' r'.*_xyz$' -> ["r'AA'", "r'.*_xyz$'"]
[ OK ] abc"def ghi" -> ['abc"def ghi"']
[ OK ] abc"def ghi""jkl" -> ['abc"def ghi""jkl"']
[ OK ] a"b c"d"e"f"g h" -> ['a"b c"d"e"f"g h"']
[ OK ] c="ls /" type key -> ['c="ls /"', 'type', 'key']
[ OK ] abc'def ghi' -> ["abc'def ghi'"]
[ OK ] c='ls /' type key -> ["c='ls /'", 'type', 'key']


shlex: 0.335ms per iteration
csv: 0.036ms per iteration
re: 0.068ms per iteration

因此，性能比shlex好得多，并且可以通过预编译正则表达式进一步提高，在这种情况下，它将优于csv方法。

小开

由于性能原因，re似乎更快。下面是我的解决方案，使用最小贪婪操作符，保留外部引号:

re.findall("(?:\".*?\"|\S)+", s)

结果:

['this', 'is', '"a test"']

它将像aaa"bla blub"bbb这样的构造放在一起，因为这些标记没有用空格分隔。如果字符串包含转义字符，你可以这样匹配:

>>> a = "She said \"He said, \\\"My name is Mark.\\\"\""
>>> a
'She said "He said, \\"My name is Mark.\\""'
>>> for i in re.findall("(?:\".*?[^\\\\]\"|\S)+", a): print(i)
...
She
said
"He said, \"My name is Mark.\""

请注意，这也通过模式的\S部分匹配空字符串""。

小开

作为一个选项，尝试tssplit:

In [1]: from tssplit import tssplit
In [2]: tssplit('this is "a test"', quote='"', delimiter='')
Out[2]: ['this', 'is', 'a test']