如何在另一个YAML文件中包含一个YAML文件?

我有两个YAML文件,“A”和“B”,我希望将A的内容插入到B中,要么拼接到现有的数据结构中,就像数组一样,要么作为元素的子元素,就像某个散列键的值一样。

这可能吗?怎么做?如果不是,是否有指向规范引用的指针?

341629 次浏览

据我所知,YAML中不直接支持include,你必须自己提供一种机制,然而,这通常很容易做到。

我在我的python应用程序中使用YAML作为配置语言,在这种情况下经常定义这样的约定:

>>> main.yml <<<
includes: [ wibble.yml, wobble.yml]

然后在我的(python)代码中:

import yaml
cfg = yaml.load(open("main.yml"))
for inc in cfg.get("includes", []):
cfg.update(yaml.load(open(inc)))

唯一的缺点是include中的变量将总是覆盖main中的变量,并且没有办法通过改变“includes:”语句在main中出现的位置来改变优先级。yml文件。

稍微不同的一点是,YAML不支持include,因为它并不是像基于文件的标记那样专门设计的。如果你在AJAX请求的响应中得到一个包含,它意味着什么?

你的问题不需要Python解决方案,但这里有一个使用PyYAML的解决方案。

PyYAML允许你将自定义构造函数(比如!include)附加到YAML加载器。我已经包含了一个可以设置的根目录,以便这个解决方案支持相对和绝对文件引用。

基于类的解决方案

这是一个基于类的解决方案,避免了我原始响应的全局根变量。

请参阅要点以获得类似的、更健壮的Python 3解决方案,该解决方案使用元类来注册自定义构造函数。

import yaml
import os


class Loader(yaml.SafeLoader):


def __init__(self, stream):


self._root = os.path.split(stream.name)[0]


super(Loader, self).__init__(stream)


def include(self, node):


filename = os.path.join(self._root, self.construct_scalar(node))


with open(filename, 'r') as f:
return yaml.load(f, Loader)


Loader.add_constructor('!include', Loader.include)

一个例子:

foo.yaml

a: 1
b:
- 1.43
- 543.55
c: !include bar.yaml

bar.yaml

- 3.6
- [1, 2, 3]

现在可以使用以下方法加载文件:

>>> with open('foo.yaml', 'r') as f:
>>>    data = yaml.load(f, Loader)
>>> data
{'a': 1, 'b': [1.43, 543.55], 'c': [3.6, [1, 2, 3]]}

扩展@Josh_Bode的答案,下面是我自己的PyYAML解决方案,它的优点是它是yaml.Loader的一个自包含子类。它不依赖于任何模块级的全局变量,也不依赖于修改yaml模块的全局状态。

import yaml, os


class IncludeLoader(yaml.Loader):
"""
yaml.Loader subclass handles "!include path/to/foo.yml" directives in config
files.  When constructed with a file object, the root path for includes
defaults to the directory containing the file, otherwise to the current
working directory. In either case, the root path can be overridden by the
`root` keyword argument.


When an included file F contain its own !include directive, the path is
relative to F's location.


Example:
YAML file /home/frodo/one-ring.yml:
---
Name: The One Ring
Specials:
- resize-to-wearer
Effects:
- !include path/to/invisibility.yml


YAML file /home/frodo/path/to/invisibility.yml:
---
Name: invisibility
Message: Suddenly you disappear!


Loading:
data = IncludeLoader(open('/home/frodo/one-ring.yml', 'r')).get_data()


Result:
{'Effects': [{'Message': 'Suddenly you disappear!', 'Name':
'invisibility'}], 'Name': 'The One Ring', 'Specials':
['resize-to-wearer']}
"""
def __init__(self, *args, **kwargs):
super(IncludeLoader, self).__init__(*args, **kwargs)
self.add_constructor('!include', self._include)
if 'root' in kwargs:
self.root = kwargs['root']
elif isinstance(self.stream, file):
self.root = os.path.dirname(self.stream.name)
else:
self.root = os.path.curdir


def _include(self, loader, node):
oldRoot = self.root
filename = os.path.join(self.root, loader.construct_scalar(node))
self.root = os.path.dirname(filename)
data = yaml.load(open(filename, 'r'))
self.root = oldRoot
return data

不,标准YAML不包括任何类型的“导入”;或“;include"声明。

可能在问问题时不支持,但你可以将其他YAML文件导入其中:

imports: [/your_location_to_yaml_file/Util.area.yaml]

虽然我没有任何在线参考资料,但这对我来说很有用。

不幸的是,YAML在其标准中没有提供这个功能。

但是如果你正在使用Ruby,有一个gem通过扩展Ruby YAML库来提供你所要求的功能: https://github.com/entwanderer/yaml_extend < / p >

使用Symfony,它对yaml的处理将间接地允许你嵌套yaml文件。诀窍在于使用parameters选项。例如:

common.yml

parameters:
yaml_to_repeat:
option: "value"
foo:
- "bar"
- "baz"

config.yml

imports:
- { resource: common.yml }
whatever:
thing: "%yaml_to_repeat%"
other_thing: "%yaml_to_repeat%"

其结果将与:

whatever:
thing:
option: "value"
foo:
- "bar"
- "baz"
other_thing:
option: "value"
foo:
- "bar"
- "baz"

我认为@max - b使用的解决方案看起来很棒。但是,对于嵌套的包含,它没有成功。例如,如果config_1。Yaml包含config_2。Yaml,其中包括config_3。Yaml,装弹机有问题。但是,如果您在加载时简单地将新的加载器类指向它自己,它就可以工作!具体来说,如果我们将旧的_include函数替换为稍微修改过的版本:

def _include(self, loader, node):
oldRoot = self.root
filename = os.path.join(self.root, loader.construct_scalar(node))
self.root = os.path.dirname(filename)
data = yaml.load(open(filename, 'r'), loader = IncludeLoader)
self.root = oldRoot
return data

经过反思,我同意其他评论,嵌套加载一般不适合yaml,因为输入流可能不是一个文件,但它非常有用!

也许这可以激励你,试着与jbb惯例保持一致:

https://docs.openstack.org/infra/jenkins-job-builder/definition.html#inclusion-tags

< p > <代码> -工作: 名称:test-job-include-raw-1 建造者: -壳: !包括生:include-raw001-hello-world.sh < /代码> < / p >

对于Python用户,可以尝试pyyaml-include

安装

pip install pyyaml-include

使用

import yaml
from yamlinclude import YamlIncludeConstructor


YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='/your/conf/dir')


with open('0.yaml') as f:
data = yaml.load(f, Loader=yaml.FullLoader)


print(data)

假设我们有这样的YAML文件:

├── 0.yaml
└── include.d
├── 1.yaml
└── 2.yaml
  • 1.yaml的内容:
name: "1"
  • 2.yaml的内容:
name: "2"

按名称包含文件

  • 在顶层:

    如果0.yaml是:

!include include.d/1.yaml

我们会得到:

{"name": "1"}
  • < p >映射:

    如果0.yaml是:

file1: !include include.d/1.yaml
file2: !include include.d/2.yaml

我们会得到:

  file1:
name: "1"
file2:
name: "2"
  • < p >序列:

    如果0.yaml是:

files:
- !include include.d/1.yaml
- !include include.d/2.yaml

我们会得到:

files:
- name: "1"
- name: "2"

请注意:

文件名可以是绝对的(如/usr/conf/1.5/Make.yml),也可以是相对的(如../../cfg/img.yml)。

通过通配符包含文件

文件名可以包含shell样式的通配符。从通配符找到的文件中加载的数据将按顺序设置。

如果0.yaml为:

files: !include include.d/*.yaml

我们会得到:

files:
- name: "1"
- name: "2"

请注意:

  • 对于Python>=3.5,如果!include YAML标记的recursive参数为true,则模式“**”将匹配任何文件以及零个或多个目录和子目录。
  • 在大型目录树中使用“**”模式可能会因为递归搜索而消耗过多的时间。

为了启用recursive参数,我们将以MappingSequence模式编写!include标记:

  • Sequence模式中的参数:
!include [tests/data/include.d/**/*.yaml, true]
  • Mapping模式中的参数:
!include {pathname: tests/data/include.d/**/*.yaml, recursive: true}

标准YAML 1.2本身不包括这个特性。尽管如此,许多实现提供了一些扩展来实现这一点。

我提出了一种用Java和snakeyaml:1.24(用于解析/发出YAML文件的Java库)实现它的方法,它允许创建一个自定义YAML标记来实现以下目标(你会看到我用它来加载在几个YAML文件中定义的测试套件,并且我使它作为目标test:节点的包含列表工作):

# ... yaml prev stuff


tests: !include
- '1.hello-test-suite.yaml'
- '3.foo-test-suite.yaml'
- '2.bar-test-suite.yaml'


# ... more yaml document

下面是允许处理!include标记的单类Java。文件从classpath (Maven资源目录)加载:

/**
* Custom YAML loader. It adds support to the custom !include tag which allows splitting a YAML file across several
* files for a better organization of YAML tests.
*/
@Slf4j   // <-- This is a Lombok annotation to auto-generate logger
public class MyYamlLoader {


private static final Constructor CUSTOM_CONSTRUCTOR = new MyYamlConstructor();


private MyYamlLoader() {
}


/**
* Parse the only YAML document in a stream and produce the Java Map. It provides support for the custom !include
* YAML tag to split YAML contents across several files.
*/
public static Map<String, Object> load(InputStream inputStream) {
return new Yaml(CUSTOM_CONSTRUCTOR)
.load(inputStream);
}




/**
* Custom SnakeYAML constructor that registers custom tags.
*/
private static class MyYamlConstructor extends Constructor {


private static final String TAG_INCLUDE = "!include";


MyYamlConstructor() {
// Register custom tags
yamlConstructors.put(new Tag(TAG_INCLUDE), new IncludeConstruct());
}


/**
* The actual include tag construct.
*/
private static class IncludeConstruct implements Construct {


@Override
public Object construct(Node node) {
List<Node> inclusions = castToSequenceNode(node);
return parseInclusions(inclusions);
}


@Override
public void construct2ndStep(Node node, Object object) {
// do nothing
}


private List<Node> castToSequenceNode(Node node) {
try {
return ((SequenceNode) node).getValue();


} catch (ClassCastException e) {
throw new IllegalArgumentException(String.format("The !import value must be a sequence node, but " +
"'%s' found.", node));
}
}


private Object parseInclusions(List<Node> inclusions) {


List<InputStream> inputStreams = inputStreams(inclusions);


try (final SequenceInputStream sequencedInputStream =
new SequenceInputStream(Collections.enumeration(inputStreams))) {


return new Yaml(CUSTOM_CONSTRUCTOR)
.load(sequencedInputStream);


} catch (IOException e) {
log.error("Error closing the stream.", e);
return null;
}
}


private List<InputStream> inputStreams(List<Node> scalarNodes) {
return scalarNodes.stream()
.map(this::inputStream)
.collect(toList());
}


private InputStream inputStream(Node scalarNode) {
String filePath = castToScalarNode(scalarNode).getValue();
final InputStream is = getClass().getClassLoader().getResourceAsStream(filePath);
Assert.notNull(is, String.format("Resource file %s not found.", filePath));
return is;
}


private ScalarNode castToScalarNode(Node scalarNode) {
try {
return ((ScalarNode) scalarNode);


} catch (ClassCastException e) {
throw new IllegalArgumentException(String.format("The value must be a scalar node, but '%s' found" +
".", scalarNode));
}
}
}


}


}

YML标准指定了这样做的方法。而且这个问题并不局限于YML。JSON也有同样的限制。

许多使用基于YML或JSON配置的应用程序最终都会遇到这个问题。当这种情况发生时,他们制定了自己的惯例

例如,对于swagger API定义:

$ref: 'file.yml'

例如,对于docker组合配置:

services:
app:
extends:
file: docker-compose.base.yml

或者,如果您希望将一个yml文件的内容拆分到多个文件中,就像内容树一样,您可以定义自己的文件夹结构约定并使用(现有的)合并脚本。

我举了一些例子供你参考。

import yaml


main_yaml = """
Package:
- !include _shape_yaml
- !include _path_yaml
"""


_shape_yaml = """
# Define
Rectangle: &id_Rectangle
name: Rectangle
width: &Rectangle_width 20
height: &Rectangle_height 10
area: !product [*Rectangle_width, *Rectangle_height]


Circle: &id_Circle
name: Circle
radius: &Circle_radius 5
area: !product [*Circle_radius, *Circle_radius, pi]


# Setting
Shape:
property: *id_Rectangle
color: red
"""


_path_yaml = """
# Define
Root: &BASE /path/src/


Paths:
a: &id_path_a !join [*BASE, a]
b: &id_path_b !join [*BASE, b]


# Setting
Path:
input_file: *id_path_a
"""




# define custom tag handler
def yaml_import(loader, node):
other_yaml_file = loader.construct_scalar(node)
return yaml.load(eval(other_yaml_file), Loader=yaml.SafeLoader)




def yaml_product(loader, node):
import math
list_data = loader.construct_sequence(node)
result = 1
pi = math.pi
for val in list_data:
result *= eval(val) if isinstance(val, str) else val
return result




def yaml_join(loader, node):
seq = loader.construct_sequence(node)
return ''.join([str(i) for i in seq])




def yaml_ref(loader, node):
ref = loader.construct_sequence(node)
return ref[0]




def yaml_dict_ref(loader: yaml.loader.SafeLoader, node):
dict_data, key, const_value = loader.construct_sequence(node)
return dict_data[key] + str(const_value)




def main():
# register the tag handler
yaml.SafeLoader.add_constructor(tag='!include', constructor=yaml_import)
yaml.SafeLoader.add_constructor(tag='!product', constructor=yaml_product)
yaml.SafeLoader.add_constructor(tag='!join', constructor=yaml_join)
yaml.SafeLoader.add_constructor(tag='!ref', constructor=yaml_ref)
yaml.SafeLoader.add_constructor(tag='!dict_ref', constructor=yaml_dict_ref)


config = yaml.load(main_yaml, Loader=yaml.SafeLoader)


pk_shape, pk_path = config['Package']
pk_shape, pk_path = pk_shape['Shape'], pk_path['Path']
print(f"shape name: {pk_shape['property']['name']}")
print(f"shape area: {pk_shape['property']['area']}")
print(f"shape color: {pk_shape['color']}")


print(f"input file: {pk_path['input_file']}")




if __name__ == '__main__':
main()


输出

shape name: Rectangle
shape area: 200
shape color: red
input file: /path/src/a

更新2

你可以把它们结合起来,像这样

# xxx.yaml
CREATE_FONT_PICTURE:
PROJECTS:
SUNG: &id_SUNG
name: SUNG
work_dir: SUNG
output_dir: temp
font_pixel: 24




DEFINE: &id_define !ref [*id_SUNG]  # you can use config['CREATE_FONT_PICTURE']['DEFINE'][name, work_dir, ... font_pixel]
AUTO_INIT:
basename_suffix: !dict_ref [*id_define, name, !product [5, 3, 2]]  # SUNG30


# ↓ This is not correct.
# basename_suffix: !dict_ref [*id_define, name, !product [5, 3, 2]]  # It will build by Deep-level. id_define is Deep-level: 2. So you must put it after 2. otherwise, it can't refer to the correct value.

使用Yglu,你可以像这样导入其他文件:

A.yaml

foo: !? $import('B.yaml')

B.yaml

bar: Hello
$ yglu A.yaml
foo:
bar: Hello

由于$import是一个函数,你也可以传递一个表达式作为参数:

  dep: !- b
foo: !? $import($_.dep.toUpper() + '.yaml')

这将得到与上面相同的输出。

声明:我是Yglu的作者。

加上上面@Joshbode的初始回答,我对代码片段进行了一些修改,以支持UNIX风格的通配符模式。

不过我还没有在windows中进行测试。为了便于维护,我面临着将大型yaml中的数组拆分到多个文件中的问题,并正在寻找一种解决方案,以便在基本yaml的同一个数组中引用多个文件。因此,下面的解决方案。解决方案不支持递归引用。它只支持在基本yaml中引用的给定目录级别中的通配符。

import yaml
import os
import glob




# Base code taken from below link :-
# Ref:https://stackoverflow.com/a/9577670
class Loader(yaml.SafeLoader):


def __init__(self, stream):


self._root = os.path.split(stream.name)[0]


super(Loader, self).__init__(stream)


def include(self, node):
consolidated_result = None
filename = os.path.join(self._root, self.construct_scalar(node))


# Below section is modified for supporting UNIX wildcard patterns
filenames = glob.glob(filename)
        

# Just to ensure the order of files considered are predictable
# and easy to debug in case of errors.
filenames.sort()
for file in filenames:
with open(file, 'r') as f:
result = yaml.load(f, Loader)


if isinstance(result, list):
if not isinstance(consolidated_result, list):
consolidated_result = []
consolidated_result += result
elif isinstance(result, dict):
if not isinstance(consolidated_result, dict):
consolidated_result = {}
consolidated_result.update(result)
else:
consolidated_result = result


return consolidated_result




Loader.add_constructor('!include', Loader.include)


使用

a:
!include a.yaml


b:
# All yamls included within b folder level will be consolidated
!include b/*.yaml