elasticsearch 集群
集群内的原理
介绍 cluster 、 node 、 shard 等常用术语，Elastisearch 的扩容机制， 以及如何处理硬件故障的内容。
ElasticSearch 的主旨是随时可用和按需扩容。 而扩容可以通过购买性能更强大（ 垂直扩容 ，或 纵向扩容 ） 或者数量更多的服务器（ 水平扩容 ，或 横向扩容 ）来实现。
虽然 Elasticsearch 可以获益于更强大的硬件设备，但是垂直扩容是有极限的。 真正的扩容能力是来自于水平扩容–为集群添加更多的节点，并且将负载压力和稳定性分散到这些节点中。
(1) 空集群
一个运行中的 Elasticsearch 实例称为一个 节点，而集群是由一个或者多个拥有相同 cluster.name 配置的节点组成， 它们共同承担数据和负载的压力。当有节点加入集群中或者从集群中移除节点时，集群将会重新平均分布所有的数据。
(2) 集群健康
查看集群健康状态
http://localhost:9200/_cluster/health?pretty
$ curl -X GET "localhost:9200/_cluster/health?pretty"
{
"cluster_name" : "my-application",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}status 字段指示着当前集群在总体上是否工作正常。它的三种颜色含义如下：
green 所有的主分片和副本分片都正常运行。
yellow 所有的主分片都正常运行，但不是所有的副本分片都正常运行。
red 有主分片没能正常运行。
(3) 添加索引
我们往 Elasticsearch 添加数据时需要用到 索引 —— 保存相关数据的地方。 索引实际上是指向一个或者多个物理 分片 的 逻辑命名空间 。
一个 分片 是一个底层的 工作单元 ，它仅保存了全部数据中的一部分。 在分片内部机制中，我们将详细介绍分片是如何工作的，而现在我们只需知道一个分片是一个 Lucene 的实例，以及它本身就是一个完整的搜索引擎。 我们的文档被存储和索引到分片内，但是应用程序是直接与索引而不是与分片进行交互。
Elasticsearch 是利用分片将数据分发到集群内各处的。分片是数据的容器，文档保存在分片内，分片又被分配到集群内的各个节点里。 当你的集群规模扩大或者缩小时， Elasticsearch 会自动的在各节点中迁移分片，使得数据仍然均匀分布在集群里。
一个分片可以是 主 分片或者 副本 分片。 索引内任意一个文档都归属于一个主分片，所以主分片的数目决定着索引能够保存的最大数据量。
技术上来说，一个主分片最大能够存储 Integer.MAX_VALUE - 128 个文档，但是实际最大值还需要参考你的使用场景：包括你使用的硬件， 文档的大小和复杂程度，索引和查询文档的方式以及你期望的响应时长。
一个副本分片只是一个主分片的拷贝。 副本分片作为硬件故障时保护数据不丢失的冗余备份，并为搜索和返回文档等读操作提供服务。
在索引建立的时候就已经确定了主分片数，但是副本分片数可以随时修改。
索引在默认情况下会被分配5个主分片
创建一个blogs索引，设置3个主分片和一份副本(每个主分片拥有一个副本分片)
设置分片数和副本数
$ curl -X PUT "localhost:9200/blogs?pretty" -H 'Content-Type: application/json' -d'
> {
> "settings" : {
> "number_of_shards" : 3,
> "number_of_replicas" : 1
> }
> }
> '
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "blogs"
}WKQ@WKQ-PC MINGW64 /c/ProfessionalSoftWare/ElasticSearch/elasticsearch-5.6.10
$ curl -X PUT "localhost:9200/blogs" -H 'Content-Type: application/json' -d'
> {
> "settings" : {
> "number_of_shards" : 3,
> "number_of_replicas" : 1
> }
> }
> '
{"acknowledged":true,"shards_acknowledged":true,"index":"blogs"}
(4) 添加故障转移
当集群中只有一个节点在运行时，意味着会有一个单点故障问题——没有冗余。 幸运的是，我们只需再启动一个节点即可防止数据丢失。
为了测试第二个节点启动后的情况，你可以在同一个目录内，完全依照启动第一个节点的方式来启动一个新节点。多个节点可以共享同一个目录。
当你在同一台机器上启动了第二个节点时，只要它和第一个节点有同样的 cluster.name 配置，它就会自动发现集群并加入到其中。 但是在不同机器上启动节点的时候，为了加入到同一集群，你需要配置一个可连接到的单播主机列表。
(4.1) 自己搭建一个3个节点的集群
elasticsearch-5.6.10解压三份，然后把3个的配置文件里 cluster.name 全改成统一的一个名字，这儿我写的是 elasticsearch_production ，然后分别启动3个ES
λ elasticsearch
[2018-07-22T17:38:20,925][INFO ][o.e.n.Node ] [elasticsearch_node_001] initializing ...
[2018-07-22T17:38:21,034][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_001] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [81.2gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:38:21,034][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_001] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] node name [elasticsearch_node_001], node ID [gjy4N2RCQ4mLxp1lxdyZ8w]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] version[5.6.10], pid[9412], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10]
[2018-07-22T17:38:22,254][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [aggs-matrix-stats]
[2018-07-22T17:38:22,256][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [ingest-common]
[2018-07-22T17:38:22,257][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-expression]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-groovy]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-mustache]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-painless]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [parent-join]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [percolator]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [reindex]
[2018-07-22T17:38:22,260][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [transport-netty3]
[2018-07-22T17:38:22,260][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [transport-netty4]
[2018-07-22T17:38:22,261][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] no plugins loaded
[2018-07-22T17:38:24,294][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_001] using discovery type [zen]
[2018-07-22T17:38:24,844][INFO ][o.e.n.Node ] [elasticsearch_node_001] initialized
[2018-07-22T17:38:24,844][INFO ][o.e.n.Node ] [elasticsearch_node_001] starting ...
[2018-07-22T17:38:25,488][INFO ][o.e.t.TransportService ] [elasticsearch_node_001] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-07-22T17:38:28,621][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] new_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T17:38:28,887][INFO ][o.e.g.GatewayService ] [elasticsearch_node_001] recovered [2] indices into cluster_state
[2018-07-22T17:38:29,246][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[megacorp][4], [megacorp][3], [megacorp][0]] ...]).
[2018-07-22T17:38:29,360][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_001] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-07-22T17:38:29,360][INFO ][o.e.n.Node ] [elasticsearch_node_001] started
[2018-07-22T17:38:56,302][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] added {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-join
[2018-07-22T17:38:56,505][WARN ][o.e.d.z.ElectMasterService] [elasticsearch_node_001] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2])
[2018-07-22T17:38:58,989][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[megacorp][4]] ...]).
[2018-07-22T17:39:43,600][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] added {{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302},}, reason: zen-disco-node-joinC:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node2\bin
λ elasticsearch
[2018-07-22T17:38:47,851][INFO ][o.e.n.Node ] [elasticsearch_node_002] initializing ...
[2018-07-22T17:38:47,960][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_002] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [78.9gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:38:47,960][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_002] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] node name [elasticsearch_node_002], node ID [3ezLh-3jTB-c0pQXFXnr5Q]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] version[5.6.10], pid[13072], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node2]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [aggs-matrix-stats]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [ingest-common]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-expression]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-groovy]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-mustache]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-painless]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [parent-join]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [percolator]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [reindex]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [transport-netty3]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [transport-netty4]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] no plugins loaded
[2018-07-22T17:38:51,724][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_002] using discovery type [zen]
[2018-07-22T17:38:52,224][INFO ][o.e.n.Node ] [elasticsearch_node_002] initialized
[2018-07-22T17:38:52,240][INFO ][o.e.n.Node ] [elasticsearch_node_002] starting ...
[2018-07-22T17:38:52,929][INFO ][o.e.t.TransportService ] [elasticsearch_node_002] publish_address {127.0.0.1:9301}, bound_addresses {127.0.0.1:9301}, {[::1]:9301}
[2018-07-22T17:38:56,365][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] detected_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, added {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [7]])
[2018-07-22T17:38:57,005][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_002] publish_address {127.0.0.1:9201}, bound_addresses {127.0.0.1:9201}, {[::1]:9201}
[2018-07-22T17:38:57,005][INFO ][o.e.n.Node ] [elasticsearch_node_002] started
[2018-07-22T17:39:43,600][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] added {{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [16]])λ elasticsearch
[2018-07-22T17:39:34,786][INFO ][o.e.n.Node ] [elasticsearch_node_003] initializing ...
[2018-07-22T17:39:35,020][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_003] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [76.8gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:39:35,020][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_003] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] node name [elasticsearch_node_003], node ID [upVg0NksTRi8LH8mj9xqpg]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] version[5.6.10], pid[2316], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node3]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [aggs-matrix-stats]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [ingest-common]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-expression]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-groovy]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-mustache]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-painless]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [parent-join]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [percolator]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [reindex]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [transport-netty3]
[2018-07-22T17:39:36,404][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [transport-netty4]
[2018-07-22T17:39:36,404][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] no plugins loaded
[2018-07-22T17:39:38,995][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_003] using discovery type [zen]
[2018-07-22T17:39:39,605][INFO ][o.e.n.Node ] [elasticsearch_node_003] initialized
[2018-07-22T17:39:39,605][INFO ][o.e.n.Node ] [elasticsearch_node_003] starting ...
[2018-07-22T17:39:40,386][INFO ][o.e.t.TransportService ] [elasticsearch_node_003] publish_address {127.0.0.1:9302}, bound_addresses {127.0.0.1:9302}, {[::1]:9302}
[2018-07-22T17:39:43,631][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] detected_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, added {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [16]])
[2018-07-22T17:39:44,308][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_003] publish_address {127.0.0.1:9202}, bound_addresses {127.0.0.1:9202}, {[::1]:9202}
[2018-07-22T17:39:44,308][INFO ][o.e.n.Node ] [elasticsearch_node_003] startedWKQ@WKQ-PC MINGW64 /c/ProfessionalSoftWare/ElasticSearch/elasticsearch-5.6.10
$ curl -X GET "localhost:9200/_cluster/health"
{
"cluster_name":"elasticsearch_production",
"status":"green",
"timed_out":false,
"number_of_nodes":3,
"number_of_data_nodes":3,
"active_primary_shards":8,
"active_shards":16,
"relocating_shards":0,
"initializing_shards":0,
"unassigned_shards":0,
"delayed_unassigned_shards":0,
"number_of_pending_tasks":0,
"number_of_in_flight_fetch":0,
"task_max_waiting_in_queue_millis":0,
"active_shards_percent_as_number":100
}(5) 水平扩容
每个节点的硬件资源（CPU, RAM, I/O）将被更少的分片所共享，每个分片的性能将会得到提升。
分片是一个功能完整的搜索引擎，它拥有使用一个节点上的所有资源的能力。
主分片的数目在索引创建时 就已经确定了下来。实际上，这个数目定义了这个索引能够 存储 的最大数据量。（实际大小取决于你的数据、硬件和使用场景。）
但是，读操作——搜索和返回数据——可以同时被主分片 或 副本分片所处理，所以当你拥有越多的副本分片时，也将拥有越高的吞吐量。
(5.1) 设置副本数
$ curl -X PUT "localhost:9200/blogs/_settings?pretty" -H 'Content-Type: application/json' -d'
> {
> "number_of_replicas" : 2
> }
> '
{
"acknowledged" : true
}blogs 索引现在拥有9个分片：3个主分片和6个副本分片。 这意味着我们可以将集群扩容到9个节点，每个节点上一个分片。相比原来3个节点时，集群搜索性能可以提升 3 倍。
当然，如果只是在相同节点数目的集群上增加更多的副本分片并不能提高性能，因为每个分片从节点上获得的资源会变少。 你需要增加更多的硬件资源来提升吞吐量。
但是更多的副本分片数提高了数据冗余量：按照上面的节点配置，我们可以在失去2个节点的情况下不丢失任何数据。
(6) 应对故障
[2018-07-22T17:59:38,513][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [elasticsearch_node_001] updating number_of_replicas to [2] for indices [blogs]
[2018-07-22T17:59:39,286][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[blogs][2]] ...]).
[2018-07-22T18:05:07,871][INFO ][o.e.n.Node ] [elasticsearch_node_001] stopping ...
[2018-07-22T18:05:07,943][INFO ][o.e.n.Node ] [elasticsearch_node_001] stopped
[2018-07-22T18:05:08,051][INFO ][o.e.n.Node ] [elasticsearch_node_001] closing ...
[2018-07-22T18:05:08,065][INFO ][o.e.n.Node ] [elasticsearch_node_001] closed
终止批处理操作吗(Y/N)? y
[2018-07-22T18:05:07,887][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [shut_down]
[2018-07-22T18:05:07,910][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}
{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, master
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, local
[2018-07-22T18:05:07,911][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2018-07-22T18:05:07,925][WARN ][o.e.t.n.Netty4Transport ] [elasticsearch_node_002] write and flush on the network layer failed (channel: [id: 0xee4a8fab, L:0.0.0.0/0.0.0.0:9301 ! R:/127.0.0.1:53349])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-22T18:05:08,929][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_002] failed to connect to node {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_001][127.0.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:05:10,961][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] new_master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T18:05:13,003][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [elasticsearch_node_002] failed to execute on node [gjy4N2RCQ4mLxp1lxdyZ8w]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_001][127.0.0.1:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:05:13,015][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [elasticsearch_node_002] failed to execute [indices:monitor/stats] on node [gjy4N2RCQ4mLxp1lxdyZ8w]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_001][127.0.0.1:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:05:13,057][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_002] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
[2018-07-22T18:05:13,058][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] removed {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-node-failed({elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}), reason(transport disconnected)
[2018-07-22T18:05:14,096][INFO ][o.e.c.r.DelayedAllocationService] [elasticsearch_node_002] scheduling reroute for delayed shards in [58.8s] (6 delayed shards)
[2018-07-22T18:05:07,882][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [shut_down]
[2018-07-22T18:05:07,891][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}
{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, master
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, local
[2018-07-22T18:05:07,904][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2018-07-22T18:05:08,917][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_003] failed to connect to node {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_001][127.0.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:05:11,977][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] detected_master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-receive(from master [master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} committed version [27]])
[2018-07-22T18:05:13,066][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] removed {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} committed version [28]])
[2018-07-22T18:11:53,988][INFO ][o.e.n.Node ] [elasticsearch_node_002] stopping ...
[2018-07-22T18:11:54,051][INFO ][o.e.n.Node ] [elasticsearch_node_002] stopped
[2018-07-22T18:11:54,051][INFO ][o.e.n.Node ] [elasticsearch_node_002] closing ...
[2018-07-22T18:11:54,059][INFO ][o.e.n.Node ] [elasticsearch_node_002] closed
终止批处理操作吗(Y/N)? y
[2018-07-22T18:11:53,998][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}], reason [shut_down]
[2018-07-22T18:11:54,002][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, master
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, local
[2018-07-22T18:11:57,059][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] new_master {elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T18:11:58,069][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_003] failed to connect to node {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_002][127.0.0.1:9301] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:11:58,100][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [elasticsearch_node_003] failed to execute on node [3ezLh-3jTB-c0pQXFXnr5Q]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_002][127.0.0.1:9301] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:11:58,155][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [elasticsearch_node_003] failed to execute [indices:monitor/stats] on node [3ezLh-3jTB-c0pQXFXnr5Q]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_002][127.0.0.1:9301] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:11:58,170][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] removed {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-failed({elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}), reason(transport disconnected)
[2018-07-22T18:11:58,220][INFO ][o.e.c.r.DelayedAllocationService] [elasticsearch_node_003] scheduling reroute for delayed shards in [59.8s] (8 delayed shards)
[2018-07-22T18:12:02,864][INFO ][o.e.n.Node ] [elasticsearch_node_003] stopping ...
[2018-07-22T18:12:02,893][INFO ][o.e.n.Node ] [elasticsearch_node_003] stopped
[2018-07-22T18:12:02,894][INFO ][o.e.n.Node ] [elasticsearch_node_003] closing ...
[2018-07-22T18:12:02,902][INFO ][o.e.n.Node ] [elasticsearch_node_003] closed
终止批处理操作吗(Y/N)? y