Elasticsearch項目實戰

目錄

• Elasticsearch+kibana
• 環境搭建
• windows 10環境配置
• 安裝Elasticsearch
• head安裝(非必需)
• 安裝kibana
• DSL的基本使用
• 增加
• 修改
• 查詢
• 刪除
• Elasticsearch .Net
• Low level client基本使用
• 項目實戰
• 總結
• 參考

Elasticsearch是一個基於Apache Lucene(TM)的開源搜索引擎。無論在開源還是專有領域，Lucene可以被認為是迄今為止最 先進、性能最好的、功能最全的搜索引擎庫。

一說到全文搜索,lucene久負盛名。早年間,因為項目需要,接觸過一個叫盤古分詞的開源項目,藉助其中的分詞實現了分詞搜索的功能。而盤古分詞就是lucence的.NET版本。據說這個開源項目已經恢復更新並支持. NET Core,有興趣的童鞋可以去圍觀一下(https://github.com/LonghronShen/Lucene.Net.Analysis.PanGu/tree/netcore2.0)。

我想很多童鞋都聽過ELK,ELK是Elasticsearch、Logstash、Kibana。正好公司運維同事引入了這樣一套體系,用於建立集中式日誌收集系統，將所有節點上的日誌統一收集，管理，訪問。雖然能夠從一定程度上解決基本的問題,但是原生的kibana界面和查詢方式都不夠友好,很難推向廣大的開發人員。於是我在想，我們是否可以利用這個開源的庫集成到運維自動化平臺當中,讓這把利劍發揮出更大的價值。

一、環境搭建

本文是基於windows 10操作系統的es環境的搭建。

1. java環境安裝

由於es是java語言開發的,所以這裡要安裝java環境。

jdk下載:

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

安裝完成之後就是配置環境變量:

查看是否安裝成功:

2.安裝Elasticsearch

Elasticsearch版本已經比較多,初學者可能比較懵。特別是在安裝head和Kibana的時候,如果版本不匹配,往往會導致無法使用。這裡使用的是elasticsearch-5.6.11版本。

elasticsearch-5.6.11下載:

https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-11

解壓到C:\ELk 備用。

3.head安裝(非必需)

es 4.x 版本安裝head很簡單,只需下載head插件解壓到指定目錄即可。es 5.x+需要藉助node安裝。

head下載:

https://github.com/mobz/elasticsearch-head

解壓到C:\ELk\elasticsearch-5.6.11

node下載:

https://nodejs.org/dist/v8.12.0/node-v8.12.0-win-x64.zip

安裝node

檢查node和npm是否安裝成功

path環境變量末尾 會自動增加 C:\Program Files\nodejs\

安裝 phantomjs 官網:http://phantomjs.org/下載【配置環境變量】

安裝grunt npm install -g grunt-cli

執行C:\ELk\elasticsearch-5.6.11\bin\elasticsearch.bat

執行命令啟動 head

瀏覽器訪問:http://localhost:9100/

4.安裝kibana

導致為止,其實elasticsearch自身已經安裝完成。通過Head就能很方便的操作es,但是kibana集成了head類似功能,並提供了更加友好的訪問界面。

kibana-5.6.9-windows-x86下載:

https://www.elastic.co/downloads/past-releases/kibana-5-6-9

下載之後,解壓到C:\ELk\kibana-5.6.9-windows-x86

執行C:\ELk\kibana-5.6.9-windows-x86\bin\kibana.bat

瀏覽器訪問:http://localhost:5601

二、DSL的基本使用

elasticsearch也像mysql一樣提供了專門的語法來操作數據。Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries.

• 創建文檔

PUT people/person/1?op_type=create
{
 "user" : "kimchy",
 "post_date" : "2009-11-15T14:12:12",
 "message" : "trying out Elasticsearch"
}

• 修改

POST /user/guest/20/_update
{
 "doc": {
 "RealName":"LukyHuu20"
 }
}

• 查詢

GET /user/guest/_search
{
 "query": {
 "match": {
 "Id":22
 }
 }
}

• 刪除

DELETE /user/guest/15
{
 
}

三、Elasticsearch .Net

elasticsearch是以restfulAPI方式對外提供接口,並提供客戶端給多種語言使用。Elasticsearch uses standard RESTful APIs and JSON. We also build and maintain clients in many languages such as Java, Python, .NET, SQL, and PHP. Plus, our community has contributed many more. They’re easy to work with, feel natural to use, and, just like Elasticsearch, don't limit what you might want to do with them. 參考(https://www.elastic.co/products/elasticsearch)

1.Low level client基本使用

本文是介紹ES的.NET客戶端,Elasticsearch .Net - Low level client[5.x]

通過引入對應的版本的客戶端,便可通過C#操作ES。參考(https://www.elastic.co/guide/en/elasticsearch/client/net-api/5.x/elasticsearch-net.html)

連接

var settings = new ConnectionConfiguration(new Uri("http://example.com:9200"))
 .RequestTimeout(TimeSpan.FromMinutes(2));
var lowlevelClient = new ElasticLowLevelClient(settings);

插入文檔

 var indexResponse = lowlevelClient.Index<byte>("user", "guest", user.Id.ToString(), user);
 byte[] responseBytes = indexResponse.Body;
/<byte>

更新文檔

var searchResponse = lowlevelClient.Update<string>("user", "guest", id.ToString(), new
 {
 doc = new
 {
 RealName = realname,
 Description = description
 }
 });
bool successful = searchResponse.Success;
/<string> 

查詢

var searchResponse = lowlevelClient.Search<string>("user", "guest", new
 {
 query = new
 {
 match = new
 {
 Id = id
 }
 }
 });
bool successful = searchResponse.Success;
/<string>

刪除

var searchResponse = lowlevelClient.Delete<string>("user", "guest", id.ToString());
 bool successful = searchResponse.Success;
/<string>

2.項目實戰

前面大致介紹了ES的安裝和基本使用。那麼,如何在項目中落地呢？

使用nuget安裝Elasticsearch.Net 5.6.4

Install-Package Elasticsearch.Net -Version 5.6.4

安裝完後,

基本的增刪該查在項目中的實現上面已經有所介紹,這裡重點講一下查詢:

筆者使用的.NET MVC5 Web框架,對於返回的結果筆者做了一個簡單封裝：

public class ESearchRoot
 {
 /// <summary>
 /// 
 /// /<summary> 

 public int took { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string timed_out { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public _shards _shards { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public Hits hits { get; set; }
 }
 public class _shards
 {
 /// <summary>
 /// 
 /// /<summary>
 public int total { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public int successful { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public int skipped { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public int failed { get; set; }
 }
 public class HitsItem
 {
 /// <summary>
 /// 
 /// /<summary>
 public string _index { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string _type { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string _id { get; set; }
 /// <summary> 

 /// 
 /// /<summary>
 public string _score { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public T _source { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public List sort { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public Highlight highlight { get; set; }
 }
 public class Hits
 {
 /// <summary>
 /// 
 /// /<summary>
 public int total { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string max_score { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public List<hitsitem>> hits { get; set; }
 }
 public class Highlight
 {
 /// <summary>
 /// 
 /// /<summary>
 public List<string> Description { get; set; }
 }
/<string>/<hitsitem>

因為soure返回的對象是不定的,所以使用了泛型。 本項目soure對應的類,user:

 ///<summary>
 /// 
 /// /<summary>
 public class User
 {
 /// <summary>
 /// 
 /// /<summary>
 public string Account { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string Phone { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string Email { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string RealName { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string CanReview { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string CanExcute { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string Avatar { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string IsUse { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public int Id { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public string Name { get; set; }
 /// <summary>
 /// 
 /// /<summary> 

 public string Description { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public DateTime CreateTime { get; set; }
 /// <summary>
 /// 
 /// /<summary>
 public DateTime ModifyTime { get; set; }
 }
 

項目使用了帶條件的分頁查詢:

public List<adminuser> GetBySomeWhere(string keyword, int limit, int pageSize, out int total)
 {
 List<adminuser> users = new List<adminuser>();
 total = 0;
 try
 {
 var settings = new ConnectionConfiguration(new Uri("http://localhost:9200/"))
 .RequestTimeout(TimeSpan.FromMinutes(2));
 var lowlevelClient = new ElasticLowLevelClient(settings);
 //根據不同的參數 來構建不同的查詢條件
 var request = new object();
 if (!String.IsNullOrEmpty(keyword))
 {
 request = new
 {
 from = limit,
 size = pageSize,
 query = new
 {
 match = new
 {
 Description = keyword
 }
 },
 highlight = new
 {
 fields = new
 {
 Description = new { }
 }
 },
 sort = new 

 {
 Id = new
 {
 order = "desc"
 }
 }
 };
 }
 else
 {
 request = new
 {
 from = limit,
 size = pageSize,
 query = new
 {
 match_all = new
 {
 }
 },
 highlight = new
 {
 fields = new
 {
 Description = new { }
 }
 },
 sort = new
 {
 Id = new
 {
 order = "desc"
 }
 }
 };
 }
 var searchResponse = lowlevelClient.Search<string>("user", "guest", request);
 bool successful = searchResponse.Success;
 var responseJson = searchResponse.Body;
 if (!successful)
 {
 return users;
 }
 ESearchRoot<user> root = JsonHelper.JSONStringObject<esearchroot>>(responseJson);
 if (root != null)
 {
 total = root.hits.total;
 foreach (HitsItem<user> item in root.hits.hits)
 {
 if (item._source != null) 

 {
 string highlightDescription = String.Empty;
 StringBuilder sbDs = new StringBuilder();
 if (item.highlight != null && item.highlight.Description.Count > 0)
 {
 //ighlightDescription = item.highlight.Description[0];
 foreach (var d in item.highlight.Description)
 {
 sbDs.Append(d);
 }
 highlightDescription = sbDs.ToString();
 }
 AdminUser user = new AdminUser
 {
 Id = item._source.Id,
 RealName = item._source.RealName,
 Account = item._source.Account,
 Email = item._source.Email,
 Phone = item._source.Phone,
 //IsUse=item._source.IsUse,
 Avatar = item._source.Avatar,
 Description = item._source.Description,
 HighlightDescription = highlightDescription,
 CreateTime = item._source.CreateTime,
 ModifyTime = item._source.ModifyTime
 };
 users.Add(user);
 }
 }
 }
 return users;
 }
 catch (ElasticsearchClientException ex)
 {
 //Log4Helper.Error
 }
 return users;
 }
/<user>/<esearchroot>/<user>/<string>/<adminuser>/<adminuser>/<adminuser>

項目最終的效果如下:

四、總結

elasticsearch是很強大的開源工具,在實現全文搜索上有其獨到之處,也是大數據的分析方面利器,值得大家深入去研究和實踐。

五、參考

• Elasticsearch權威指南
• Elasticsearch.Net 5.x
• Elasticsearch Reference [5.6] » Document APIs
• elasticsearch-net-example
• ElasticSearch安裝及HEAD插件配置

閱讀更多 頂級碼農 的文章

關鍵字: Lucene J 搭建