ELK

Basic Concepts

  • /etc/elasticsearch/elasticsearch.yml - main config file ( define cluster name and node name )
  • cluster - collection of one or more nodes
  • node - single server that is part of your cluster
  • incices - databases of indexes
  • index - collection of documents that have somewhat similar characteristics ( its name MUST be all lowercase)
  • type - within an index, you can define one or more types.A type is a logical category/partition of your index whose semantics is completely up to you.In general, a type is defined for documents that have a set of common fields.
  • document - basic unit of information that can be indexed.A document MUST be indexed/assigned to a type inside an index
  • shareds and replicas - when indexes are to big for one node they will be splitted in smaller pieces(shareds) which will have copies (replicas)

An Elasticsearch cluster can contain multiple Indices (databases), which in turn contain multiple Types (tables). These types hold multiple Documents (rows), and each document has Properties(columns).

http://elasticsearch-cheatsheet.jolicode.com/

REST API

get health

curl -XGET 'localhost:9200/_cat/health?v&pretty'

get nodes in the cluster

curl -XGET 'localhost:9200/_cat/nodes?v&pretty'

update existing doc

curl -XPOST 'localhost:9200/${INDEX}/${TYPE}/${ID}/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
  "doc": { "name": "Jane Doe" }
}
'

add age field

curl -XPOST 'localhost:9200/${INDEX}/${TYPE}/${ID}/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
  "doc": { "name": "Jane Doe", "age": 20 }
}
'

udpate only ‘_source.age’

curl -XPOST 'localhost:9200/${INDEX}/${TYPE}/${ID}/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
  "script" : "ctx._source.age += 5"
}
'

deleting doc

curl -XDELETE 'localhost:9200/${INDEX}/${TYPE}/${ID}?pretty&pretty'

delete index

curl -XDELETE 'localhost:9200/${INDEX}

get all recrods from index

curl -XGET 'localhost:9200/${INDEX}_count?pretty' -H 'Content-Type: application/json' -d'
{
  "query" : {
    "bool" : {
      "should": [
          { "match": { "account_number": 265 }}, 
        { "match": { "account_number": 335 }}
      ]
    }
  }
}'

delete records from index

curl -XPOST 'localhost:9200/${INDEX}/_delete_by_query?pretty' -H 'Content-Type: application/json' -d'
{
  "query" : {
    "bool" : {
      "should": [
          { "match": { "account_number": 265 }}, 
        { "match": { "account_number": 335 }}
      ]
    }
  }
}'

Various commands

DELETE logstash-2019.05.21
GET _cat/shards/logstash-2018.05.25?v

GET _cluster/state
GET /_cat/thread_pool/generic?v&h=id,name,active,rejected,completed
GET /_cat/thread_pool
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason
GET /_cat/nodeattrs?v
GET /_cat/nodes?v
GET /_cat/pending_tasks?v
GET /_cat/plugins?v&s=component&h=name,component,version,description
GET _cat/recovery?v
GET _nodes



GET _cat/allocation?v
GET _cat/shards
GET _cat/recovery?v
GET logstash-*/_mapping/
GET _template
GET _cat/indices?v
GET _cat/indices?v&s=index:desc

GET /_search?q=message:number&size=0&terminate_after=1

GET logstash-2018.05.25/_field_stats

GET .kibana/index-pattern/logstash-2018.03.*
GET .kibana/index-pattern/logstash-*
GET .kibana/index-pattern/logs*

KIBANA

LOGSTASH

old data input via tcp

#!bash

nc localhost 3333 < /var/log/milekitic/*.log

bin/logstash -e '
input { 
  stdin { }
  #file {
  #  path => "/etc/logstash/test_file"
  #  type => "java"
  #  start_position => "beginning"
  #  sincedb_path => "/dev/null"
  #  ignore_older => 0
  #}
  #stdin { }
  #tcp {
  #  type => "java"
  #  port => 3333
  #} 
} 
filter {
 grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{SYSLOG5424SD:syslog} %{WORD:method}  %{JAVACLASS:class} %{GREEDYDATA:message} %{GREEDYDATA:data}" }
  }
} 
output { 
  stdout {
    codec => rubydebug
  } 
 }
'