How to take Mongodb backup and store it to s3 with Mongodump
Cron to automate Mongodb backup to S3
This post focuses on how to take dump/backup of mongodb database and store it to S3 using a bash script. The script uses S3CMD tool to access S3, So if you have not installed it, get it installed and configured. It also stores dump/backup locally in backup directory and on S3 it store the backup in full-backup directory, so you can change it according to your requirement.
Following are the steps and script to automate the task.
Following are the steps and script to automate the task.
Create a file mongo_to_s3.sh
$ vim mongo_to_s3.sh
Copy and paste the below code in mongo_to_s3.sh
#!/bin/bash
# Force file syncronization and lock writes
echo "Lock writes..."
mongo admin --eval "printjson(db.fsyncLock())"
MONGODUMP_PATH="/usr/bin/mongodump"
MONGO_HOST="127.0.0.1" #replace with your server ip
MONGO_PORT="27017"
MONGO_DATABASE="my_db" #replace with your database name
TIMESTAMP=`date +%F-%H%M`
S3_BUCKET_NAME="mybucket" #replace with your bucket name on Amazon S3
S3_BUCKET_PATH="backup"
# Create backup
echo "Taking dump and storing in /home/ubuntu/backup/..."
cd /home/ubuntu/backup
mongodump -h $MONGO_HOST:$MONGO_PORT -d $MONGO_DATABASE
# Add timestamp to backup
echo "Making zip of dump created with timestamp..."
mv dump mongodb-$HOSTNAME-$TIMESTAMP
tar cf mongodb-$HOSTNAME-$TIMESTAMP.tar.gz mongodb-$HOSTNAME-$TIMESTAMP
# Upload to S3
echo "Uploading dump to S3..."
s3cmd sync /home/ubuntu/backup/mongodb*.tar.gz s3://$S3_BUCKET_NAME/full-backup
#Unlock database writes
echo "Unlock Database writes..."
mongo admin --eval "printjson(db.fsyncUnlock())"
# Delete the backup after sync to S3
sudo rm /home/ubuntu/backup/mongodb*.gz
Make the file executable
$ sudo chmod +x mongo_to_s3.sh
Run the script
./mongo_to_s3.sh
Cron to automate the backup task
Open the file /etc/crontab and add the following cron which runs at 21:30 UTC or as per your server time zone.30 21 * * * ubuntu /home/ubuntu/backup/mongodb_to_s3_backup.sh>>/home/ubuntu/backup/log.txt
NOTE: In case if you have password on mongodb than you can supply username and password in mongodump command.
Will the fsynclock() cause a problem if there are write operations being performed to the DB while the backup is in progress? Are there any other issues of fsynclock()?
ReplyDeleteIf there are pending writes in memory, then after calling syncLock(), those writes would be written to disk. But after the lock has been placed, the new writes will get rejected. Its upto application how to handle the rejected writes.
DeleteIn short, following is the procedure:
Pending writes in memory,
Call fsynclock(),
Pending writes are flushed to disk.
New writes are rejected.
Call fsyncunlock()
normal state
Although it varies depending upon the mongo version and the database engine you are using.
fsynclock() should be use only at the secondary server. its not for primary
ReplyDeleteGreat post! The only thing I struggled a bit with was the authentication (mostly because I had to add a few roles to my user). Two little remarks though;
ReplyDelete* the name of the shell script in your tutorial differs from the one you use in the cron example (mongo_to_s3.sh vs mongodb_to_s3_backup.sh)
* you don't use the S3_BUCKET_PATH variable in your s3cmd sync command