Tomcat Log Rotation, Compression, Encryption (GDPR Compliance) and Copy to AWS S3
In production environment the logs of applications are important, we need to store them for compliance reasons and make sure that the log files dont fill up the disk space. Since the EU GDPR Privacy law has been passed, the log files need to be encrypted.
In Linux we have a beautiful log management utility logrotate, which is run regularly to rotate the logs files of apt, dpkg, cups-daemon, rsyslog, ufw etc. configuration of these logs are found in
I am running Tomcat 9 and CATALINA_HOME=/opt/tomcat. You can install Tomcat 9 from here. Tomcat rotates the logs by itself in a default setup. which is configured in
/opt/tomcat/conf/logging.properties file. we need explicitly configure tomcat to not to rotate the log files like catalina.out itself, since we will be rotating and compressing them using system logrotate utility.
Add this lines to
I have added one line to each log entry
org.apache.juli.AsyncFileHandler.rotatable = false to stop rotation of logs by tomcat, Once this is done tomcat will not rotate the logs by itself.
Logrotate and Compression
create a new file at
compress – compress the log files
compresscmd – command used for Compression, gzip
compressoptions – options for compression command, Gzip Compression level 1-9
compressext – extention of compressed file
its just going to do
gzip -9 filename , which compresses the file.
these lines add date extention to the rotated log file in the format
nomail tells not mail the log files
copytruncate – copies the log file to new file and truncates the log file to zero bytes.
daily – rotates the log file daily
rotate 10 – keep the 10 past rotated logs
notifempty – do not rotate the log if it is empty
missingok – if the file is missing its ok
create 0644 tomcat tomcat – create the log file with these permissions, ignored if copy truncate is used.
postrotate – command to run after the log rotation.
Here we are running the a custom Bash command to encrypt and move the log files to AWS S3
Encryption and Copying to AWS S3
Create a Bash Shell Script
Give it execute permissions
chmod +x /usr/share/encrypt
This script will encrypt the file with .gz which are compressed files and accept the two arguments location of gz files and Encryption key
encrypt /opt/tomcat/log /data/key
Key is needed to encrypt and decrypt the file
cat /data/key Big3secret
Encrypt command will get list of files in given location and if the file has .gz extention the it will encrypt it with gpg (GNU Privacy Guard), move all the encrypted files which has .gpg extention to aws s3 bucket and once uploaded it will delete the encrypted file from that location. For copying the encrypted logs to S3 make sure that the ec2 instance has write to S3 IAM Access Role assigned to it.
For Decryption of files
gpg --batch --passphrase-file /data/mykey -d catalina.out-2018-09-21-1537501441.gz.gpg > catalina.out-2018-09-21-1537501441.gz