Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)]， flood stage disk watermark exceeded .日志含义

当尝试将文档发布到Elasticsearch时，我得到了这个错误:

cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)];

我还在Elasticsearch日志中看到了这样的消息:

flood stage disk watermark [95%] exceeded ... all indices on this node will marked read-only
238934 次浏览
小开
最佳答案

当Elasticsearch认为磁盘空间不足时，就会发生这种情况，因此它将自己设置为只读模式。

默认情况下，Elasticsearch的决定是基于空闲磁盘空间的百分比，所以在大磁盘上，即使你有许多gb的空闲空间，这种情况也会发生。

默认情况下，泛滥水位为95%，因此在1TB的驱动器上，您至少需要50GB的空闲空间，否则Elasticsearch将自己设置为只读模式。

有关洪水阶段水印的文档请参见https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html。

正确的解决方案取决于上下文——例如生产环境与开发环境。

解决方法1:释放磁盘空间

释放足够的磁盘空间，使磁盘的空闲空间超过5%，就可以解决这个问题。一旦有足够的磁盘空闲，Elasticsearch不会自动退出只读模式，你必须做一些类似这样的事情来解锁索引:

$ curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

解决方案2:修改洪水水位设置

将"cluster.routing.allocation.disk.watermark.flood_stage"设置更改为其他设置。可以将其设置为较低的百分比或绝对值。下面是一个如何从的文档更改设置的示例:

PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "100gb",
"cluster.routing.allocation.disk.watermark.high": "50gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
"cluster.info.update.interval": "1m"
}
}

同样，在完成此操作后，您必须使用上面的curl命令来解锁索引，但在此之后，它们不应该再次进入只读模式。

小开

默认情况下，当可用磁盘空间小于5%时，安装的Elasticsearch将进入只读模式。如果你看到类似这样的错误:

< p > Elasticsearch:运输:运输::错误::禁止:[403] {"错误":{“root_cause”:[{“类型”:“cluster_block_exception”,“原因”:“封锁 [FORBIDDEN/12/index read-only / allow delete . (api)];“}],”类型”:“cluster_block_exception”:“被”、“原因: [FORBIDDEN/12/index read-only / allow delete (api)];“}”状态”:403}< / p >

或者在/usr/local/var/log/elasticsearch.log中，你可以看到类似这样的日志:

泛洪阶段磁盘水位[95%]超过上 [nCxquc7PTxKvs6hLkfonvg] [nCxquc7] [/ usr /地方/ var / lib / elasticsearch /节点/ 0] Free: 15.3gb[4.1%]，该节点上的所有索引将被标记为只读

可以通过以下命令进行修复:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
小开 
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

从

https://techoverflow.net/2019/04/17/how-to-fix-elasticsearch-forbidden-12-index-read-only-allow-delete-api/

小开

此错误通常在您的机器磁盘空间不足时观察到。 以下步骤可以避免此错误消息

  1. 重置索引上的只读索引块。

    $ curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'


Response
${"acknowledged":true}

  2. 低水位至少50gb free，高水位至少20gb free，泛滥水位至少10gb free，每分钟更新一次集群信息

     Request
$curl -X PUT "http://127.0.0.1:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "50gb", "cluster.routing.allocation.disk.watermark.high": "20gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", "cluster.info.update.interval": "1m"}}'


Response
${
"acknowledged" : true,
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"disk" : {
"watermark" : {
"low" : "50gb",
"flood_stage" : "10gb",
"high" : "20gb"
}
}
}
},
"info" : {"update" : {"interval" : "1m"}}}}}

运行这两个命令后，必须再次运行第一个命令，以使索引不会再次进入只读模式

小开

仅使用以下命令更改设置在我的环境中不起作用:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

我还必须运行强制合并API命令:

curl -X POST "localhost:9200/my-index-000001/_forcemerge?pretty"

裁判:强制合并API

小开

从PostMan中删除只读设置

Check Settings, Note the read_only_settings is true .

Delete temporary the read_only_setting by PUT command .

Change cluster setting by PUT command

小开

ELK团队的一个很好的指南:

https://www.elastic.co/guide/en/elasticsearch/reference/master/disk-usage-exceeded.html

我用ELK 7.x就能做到

小开

即使计算机存储恢复到95%以上，这个问题仍然存在。

短期解决方案是将基巴纳上限提高到95%以上。此解决方案仅适用于Windows。

a. Create a json file with following parameters




{
"persistent": {
"cluster.routing.allocation.disk.watermark.low": "90%",
"cluster.routing.allocation.disk.watermark.high": "95%",
"cluster.routing.allocation.disk.watermark.flood_stage": "97%"
}
}


b.Name it anything ,e.g : json.txt


c.Type following command in command prompt


>curl -X PUT "localhost:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d @json.txt


d.Following output is received.


{
"acknowledged" : true,
"persistent" : {
"cluster" : {
"routing" : {
"allocation" : {
"disk" : {
"watermark" : {
"low" : "90%",
"flood_stage" : "97%",
"high" : "95%"
}
}
}
}
}
},
"transient" : { }
}


e.Create another json file with following parameter




{
"index.blocks.read_only_allow_delete": null
}


f.Name it anything ,e.g : json1.txt


g.Type following command in command prompt


>curl -X PUT "localhost:9200/*/_settings?expand_wildcards=all" -H "Content-Type: application/json" -d @json1.txt


h.You should get following output


{"acknowledged":true}


i.Restart ELK stack/Kibana and the issue should be resolved.

提交答案