Graylog is a powerful tool for log management and analysis. However, as logs accumulate over time, they can consume significant disk space and affect system performance. To address this issue, Graylog provides a feature to automatically delete old indices using a Retention Policy. This article explains how to configure Graylog to automatically delete indices and how to manually trigger the deletion process when necessary.
Graylog allows you to define policies for automatically removing indices that exceed a certain age or disk usage threshold. Follow these steps to configure this feature:
http://<server-address>:9000
).In the Retention Strategy section of the index set settings, you can choose one of the following strategies:
Once configured, save the changes. Graylog will now automatically clean up old indices according to the retention policy.
If you need to manually delete indices outside of the retention schedule, you can use the following methods:
You can also use the Elasticsearch REST API to delete indices:
curl -X GET "http://<elasticsearch-ip>:9200/_cat/indices?v"
curl -X DELETE "http://<elasticsearch-ip>:9200/<index_name>"
Example:
curl -X DELETE "http://192.168.0.104:9200/graylog_7"
If indices are locked, you must first remove the lock:
curl -X PUT "http://<elasticsearch-ip>:9200/_all/_settings" -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'
After unlocking, you can delete the indices using the above commands.
Elasticsearch automatically locks indices if disk usage exceeds its thresholds. To prevent this, adjust the disk usage settings in Elasticsearch:
/etc/elasticsearch/elasticsearch.yml
file and add the following lines: cluster.routing.allocation.disk.watermark.low: "85%" # Warning level
cluster.routing.allocation.disk.watermark.high: "90%" # Indices are locked
cluster.routing.allocation.disk.watermark.flood_stage: "95%" # Deletion starts
sudo systemctl restart elasticsearch
To automate the deletion process, you can set up a cronjob. For example, to delete old indices every night at 3:00 AM:
crontab -e
0 3 * * * curl -X DELETE "http://192.168.0.104:9200/graylog_7"
Managing indices in Graylog is essential to efficiently utilize disk space and maintain system performance. By following the steps outlined in this article, you can automate log management and optimize your system resources. If you have any questions or need further assistance, feel free to reach out through the comments section!