为什么用ES
简单说就是ES通过分词和倒排索引,专门处理关键字搜索场景,可以参考这篇文章:https://www.cnblogs.com/luxiaoxun/p/5452502.html
入门推荐ES权威指南:https://es.xiaoleilu.com/010_Intro/00_README.html
进阶还是需要看官方文档:https://www.elastic.co/guide/en/elasticsearch/reference/master/query-filter-context.html
ElasticSearch-PHP:https://www.elastic.co/guide/cn/elasticsearch/php/current/_quickstart.html
安装运行
下载:https://www.elastic.co/cn/downloads/elasticsearch
安装:将解压后的文件夹移动到/usr/local/elasticsearch
启动:nohup ./elasticsearch -Ecluster.name=why_cluster -Enode.name=why_node
概念解析
Relational DB -> Databases -> Tables -> Rows -> Columns
Elasticsearch -> Indices -> Types -> Documents -> Fields
- node:节点,一个es实例代表一个节点
- 分布式集群:es是分布式数据库,多个node组成一个集群
- index:一个node有多个index,可以理解为index为database
- type:table
- document:row
- field:column
日志文件
大部分日志文件都有log和json两种格式:
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_deprecation.log
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_audit.json
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_deprecation.json
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_index_indexing_slowlog.log
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_index_search_slowlog.log
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_index_search_slowlog.json
-rw-r--r-- 1 why staff 0 10 5 16:45 why_cluster_index_indexing_slowlog.json
-rw-r--r-- 1 why staff 35210 10 5 16:57 why_cluster_server.json
-rw-r--r-- 1 why staff 19243 10 5 16:57 why_cluster.log
-rw-r--r-- 1 why staff 285078 10 5 17:13 gc.log
启动日志文件:
- why_cluster.log
- why_cluster_server.log
慢查询日志文件:
- why_cluster_index_search_slowlog.log
- why_cluster_index_search_slowlog.json
慢查询索引文件:
- why_cluster_index_indexing_slowlog.log
- why_cluster_index_indexing_slowlog.json
查询日志文件:
- gc.log:
每有一次客户端查询,会看到一个java偏向锁相关的Cleanup记录
[2019-10-05T09:36:16.804+0000][57933][safepoint ] Application time: 2.0066631 seconds
[2019-10-05T09:36:16.804+0000][57933][safepoint ] Entering safepoint region: Cleanup
[2019-10-05T09:36:16.804+0000][57933][safepoint ] Leaving safepoint region
[2019-10-05T09:36:16.804+0000][57933][safepoint ] Total time for which application threads were stopped: 0.0002716 seconds, Stopping threads took: 0.0000905 seconds
RESTful API
前言:末尾的pretty&pretty是为了格式化显示
创建index
curl -XPUT 'localhost:9200/why_index&pretty'
{"acknowledged":true,"shards_acknowledged":true,"index":"why_index&pretty"}
查看index
-XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open why_index&pretty CRd3kQOqRnOY_gocUde8MQ 1 1 0 0 230b 230b
根据id添加或更新document
curl -XPUT 'localhost:9200/why_index/doc/1?pretty&pretty' -H 'Content-Type:application/json' -d '{"name":"why", "age":18}'
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
直接插入document
(注意:生产环境一般都会根据mysql的主键id插入,直接插入会根据base64算法自动生成id,此时无法根据id查询)
执行两次:
-XPOST 'localhost:9200/why_index/doc?pretty&pretty' -H 'Content-Type:application/json' -d '{"name":"why", "age":18}'
-XPOST 'localhost:9200/why_index/doc?pretty&pretty' -H 'Content-Type:application/json' -d '{"name":"why", "age":18}'
查看结果:
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"desc"} }'
{
"took" : 723,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pgIwpW0Bn2hKxg6BO6zm",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pgIwpW0Bn2hKxg6BO6zm"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pQIvpW0Bn2hKxg6BQaxV",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pQIvpW0Bn2hKxg6BQaxV"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "weihaoyu",
"age" : 18
},
"sort" : [
"1"
]
}
]
}
}
根据id查询某个document
curl -XGET 'localhost:9200/why_index/doc/1?pretty&pretty'
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"name" : "why",
"age" : 18
}
}
通过url传参query查询
普通查询
curl -XGET 'localhost:9200/why_index/_search?q=*&pretty&pretty'
{
"took" : 104,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"name" : "why",
"age" : 18
}
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jzm",
"age" : 18
}
}
]
}
}
升序查询:
curl -XGET 'localhost:9200/why_index/_search?q=*&sort=_id:asc&pretty&pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
}
]
}
}
降序查询:
curl -XGET 'localhost:9200/why_index/_search?q=*&sort=_id:desc&pretty&pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
}
]
}
}
查询name包含why的记录
curl -XGET 'localhost:9200/why_index/_search?q=name:why&sort=_id:desc&pretty&pretty'
{
"took" : 8,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pgIwpW0Bn2hKxg6BO6zm",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pgIwpW0Bn2hKxg6BO6zm"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pQIvpW0Bn2hKxg6BQaxV",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pQIvpW0Bn2hKxg6BQaxV"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"3"
]
}
]
}
}
通过DSL语句查询
升序:
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
}
]
}
}
降序:
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"desc"} }'
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
}
]
}
}
查询name字段包含why的记录
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match":{"name":"why"}}}'
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 0.5389965,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pQIvpW0Bn2hKxg6BQaxV",
"_score" : 0.5389965,
"_source" : {
"name" : "why",
"age" : 18
}
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pgIwpW0Bn2hKxg6BO6zm",
"_score" : 0.5389965,
"_source" : {
"name" : "why",
"age" : 18
}
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : 0.5389965,
"_source" : {
"name" : "why",
"age" : 18
}
}
]
}
}
根据id修改数据
curl -XPOST 'localhost:9200/why_index/doc/1/_update?pretty&pretty' -H 'Content-Type:application/json' -d '{"doc":{"name":"weihaoyu"}}'
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_version" : 2,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 2,
"_primary_term" : 1
}
修改后:
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"desc"} }'
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "weihaoyu",
"age" : 18
},
"sort" : [
"1"
]
}
]
}
}
根据id删除
删除前数据
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "weihaoyu",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"3"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pQIvpW0Bn2hKxg6BQaxV",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pQIvpW0Bn2hKxg6BQaxV"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pgIwpW0Bn2hKxg6BO6zm",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pgIwpW0Bn2hKxg6BO6zm"
]
}
]
}
}
删除id为3的document
curl -XDELETE 'localhost:9200/why_index/doc/3?pretty&pretty'
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_version" : 2,
"result" : "deleted",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 8,
"_primary_term" : 1
}
查看结果
why$ curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 513,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "weihaoyu",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pQIvpW0Bn2hKxg6BQaxV",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pQIvpW0Bn2hKxg6BQaxV"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "pgIwpW0Bn2hKxg6BO6zm",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"pgIwpW0Bn2hKxg6BO6zm"
]
}
]
}
}
删除符合条件的document
查看删除前数据
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"3"
]
}
]
}
}
删除操作
curl -XPOST 'localhost:9200/why_index/_delete_by_query?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match":{"name":"why"}}}'
{
"took" : 85,
"timed_out" : false,
"total" : 2,
"deleted" : 2,
"batches" : 1,
"version_conflicts" : 0,
"noops" : 0,
"retries" : {
"bulk" : 0,
"search" : 0
},
"throttled_millis" : 0,
"requests_per_second" : -1.0,
"throttled_until_millis" : 0,
"failures" : [ ]
}
查看删除后的结果
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "jzm",
"age" : 18
},
"sort" : [
"3"
]
}
]
}
}
批量操作
操作前数据
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"1"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "jiaozimeng",
"age" : 18
},
"sort" : [
"3"
]
}
]
}
}
执行操作:更新id为3的数据、根据id=2更新插入数据、删除id=1的数据
curl -XPOST 'localhost:9200/why_index/doc/_bulk?pretty&pretty' -H 'Content-Type:application/json' -d '
{"update":{"_id":3}}
{"doc":{"name":"weihaoyu"}}
{"index":{"_id":2}}
{"name":"why","age":18}
{"delete":{"_id":1}}
> '
{
"took" : 37,
"errors" : false,
"items" : [
{
"update" : {
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_version" : 3,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 7,
"_primary_term" : 1,
"status" : 200
}
},
{
"index" : {
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 8,
"_primary_term" : 1,
"status" : 201
}
},
{
"delete" : {
"_index" : "why_index",
"_type" : "doc",
"_id" : "1",
"_version" : 2,
"result" : "deleted",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 9,
"_primary_term" : 1,
"status" : 200
}
}
]
}
查看结果
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "2",
"_score" : null,
"_source" : {
"name" : "why",
"age" : 18
},
"sort" : [
"2"
]
},
{
"_index" : "why_index",
"_type" : "doc",
"_id" : "3",
"_score" : null,
"_source" : {
"name" : "weihaoyu",
"age" : 18
},
"sort" : [
"3"
]
}
]
}
}
删除index
curl -XDELETE 'localhost:9200/why_index/?pretty&pretty'
{
"acknowledged" : true
}
再查询返回404
curl -XGET 'localhost:9200/why_index/_search?pretty&pretty' -H 'Content-Type:application/json' -d '{"query":{"match_all":{}}, "sort":{"_id":"asc"} }'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [why_index]",
"resource.type" : "index_or_alias",
"resource.id" : "why_index",
"index_uuid" : "_na_",
"index" : "why_index"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [why_index]",
"resource.type" : "index_or_alias",
"resource.id" : "why_index",
"index_uuid" : "_na_",
"index" : "why_index"
},
"status" : 404
}
持续学习补充ing...
转载:https://blog.csdn.net/why444216978/article/details/102158738