使用curator定期清理Elasticsearch数据

因为我的elasticsearch是用来存储各种日志数据的，需要周期的把数据清理掉，不然磁盘要炸了。

Stack里面有个 Index Lifecycle Policies 好像也可以完成这个工作，没研究明白。

这里记录下使用curator来进行index的清理。

安装curator

elastic官方给的下载地址：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html

点过去看，资源好像是404了。

在Google搜了一圈，在这里找到了安装包：https://packagecloud.io/ivan/centos/packages/el/7/elasticsearch-curator-5.8.1-1.x86_64.rpm

文件下载下来，执行命名 安装。

rpm -ivh elasticsearch-curator-5.8.1-1.x86_64.rpm

安装好了，可以执行 curator 这个命令， 他会提示你缺少参数。就安装好了。

配置任务

创建配置文件

mkdir /etc/curator
touch /etc/curator/action.yml
touch /etc/curator/config.yml

vi /etc/curator/config.yml

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
  - 192.168.1.100
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: elastic:changeme
  timeout: 120
  master_only: True
logging:
  loglevel: INFO
  logfile:
  logformat: default
  #blacklist: ['elasticsearch', 'urllib3']

这里是链接elasticsearch的配置，注意要改下你的host地址，端口号，用户名密码等信息。

vi /etc/curator/action.yml

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than ${UNIT_COUNT:1} ${UNIT:months} (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: log-stash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 15
      exclude:
  2:
    action: delete_indices
    description: >-
      Delete indices older than ${UNIT_COUNT:1} ${UNIT:months} (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: gateway-access-log-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 15
      exclude:
  3:
    action: delete_indices
    description: >-
      Delete indices older than ${UNIT_COUNT:1} ${UNIT:months} (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: True
      disable_action: False
    filters:
      - filtertype: pattern
        kind: prefix
        value: event-ack-
        exclude:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30
        exclude:

这里是脚本要执行的任务，每个任务有个序号，如果你有多个任务，按照规律增加即可。配置信息都是字面意思，可以根据需要修改。

创建定时任务清理任务

crontab -e

0 4 * * * /usr/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml

每天凌晨4点执行我们预定义的清理任务。

参考资料：

https://sysadminwork.com/cleaning-elasticsearch-with-curator/

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html

https://packagecloud.io/ivan/centos/packages/el/7/elasticsearch-curator-5.8.1-1.x86_64.rpm