gdrive backups with service accounts
Every service needs backups, and a simple way to get these at a low cost is with Google Drive.
In this post I'll take you through a way to implement this using the Google Drive CLI Client.
The gdrive cli offers a simple way to upload, list, sync & more with a Google Drive account. By default this is done by connecting your own account which work fine but does allow access to all files in your account.
Instead I prefer to use the new service accounts feature that allows you to share access to a folder with a service account so only that folder is available.
The downside of using service accounts here is that we need to compile the gdrive package with go first but with go installed this is easy to do. Go is easy to install & you can find out more here.
Install
To compile gdrive from source you'll need to run: go get github.com/prasmussen/gdrive
This will download and compile the binary for you to $GOPATH/bin/gdrive. If you have the gopath bin folder in your $PATH then you can call gdrive straight away, but if not feel free to move this file to wherever you want, you could even scp it to another server.
Create a project in Google Developer Console.
If you don't already have one, you'll need to create a project in the Google Developer Console.
Once logged in if you don't have any projects yet it will normally prompt you to create one, otherwise if you already have projects but wish to use a new one or select a different one, you can do so by clicking the project name to the side of the Google APIs logo to the top left.
Enable Google Drive API access
On the left side menu of your project screen click library.
Create a service account
Back on the left side menu select Credentials & the click to Create credentials > Service account key.
Create a new service account, and give it a name you'll remember what it's for such as "backup".
Leave JSON selected and on create it will auto start downloading the json config file. You'll need this for gdrive later.
Create a folder for the backups in your Google Drive.
In Google Drive you'll now want to decide where to store these backups, for this demo I've created a backups folder to sync backups to.
Share access to the service account
Right click the folder and select Share.
Enter the email given to you for the new service account & untick to notify the user.
Get the backup folders file id
To test we need the file id the backup folder. If you navigate into the folder in your browser, the id will be i the url, e.g. in https://drive.google.com/drive/folders/xxxxxxxxxx the file id is xxxxxxxxxx
Test access via the gdrive
We need to check we can access the backup folder successfully.
By default gdrive expects the service account json file to be in ~/.gdrive so let's move the file here.
First make the folder: mkdir ~/.gdrive
Then run mv ~/Downloads/Demo\ Project-xxxxxx.json ~/.gdrive/sa.json
Now to run the test (replacing xxxxx with the folders file id):
gdrive --service-account sa.json info xxxxx
If everything worked, this should return some basic stats on the folder.
If an error message shows please recheck the above steps or you can contact me via the comments or raise a ticket on gdrive's GitHub page.
Create a backup folder on the server or machine you want to be backed up
For this demo I'll be using a /backup folder (mkdir /backup).
& I'll want to link in a few core log files so they will be backed up regularly, e.g. on a web server running nginx, php & a laravel app i'd probably want at least these files:
ln /var/log/auth.log /backup
ln /var/log/syslog /backup
ln /var/log/nginx/error.log /backup
ln /var/log/php7.0-fpm.log /backup
ln /var/log/redis/redis-server.log /backup
ln /app/storage/logs/laravel.log /backup
Just as simply you could setup automysqlbackup and link in the files from this so that these files are auto backed up on drive too, or have it's folder backed up to another folder on drive.
First backup run & schedule
To actually run the backup we'll run a sync upload and we'll use the --delete-extraneous to delete any files in the drive folder that were in the last backup sync but are not there now:
gdrive --service-account sa.json sync upload --delete-extraneous /backup xxxxx
You can then schedule a cron to run that above command for you daily.
e.g. crontab -e then add:
@daily /path/to/your/gdrive --service-account sa.json sync upload --delete-extraneous /backup xxxxx
This will also send you an email when it completes, though you could add > /dev/null to the end of that to ignore the output and not have it email you.
gdrive provides plenty other commands which you can find with:
~ $ gdrive help
gdrive usage:
gdrive [global] list [options] List files
gdrive [global] download [options] <fileId> Download file or directory
gdrive [global] download query [options] <query> Download all files and directories matching query
gdrive [global] upload [options] <path> Upload file or directory
gdrive [global] upload - [options] <name> Upload file from stdin
gdrive [global] update [options] <fileId> <path> Update file, this creates a new revision of the file
gdrive [global] info [options] <fileId> Show file info
gdrive [global] mkdir [options] <name> Create directory
gdrive [global] share [options] <fileId> Share file or directory
gdrive [global] share list <fileId> List files permissions
gdrive [global] share revoke <fileId> <permissionId> Revoke permission
gdrive [global] delete [options] <fileId> Delete file or directory
gdrive [global] sync list [options] List all syncable directories on drive
gdrive [global] sync content [options] <fileId> List content of syncable directory
gdrive [global] sync download [options] <fileId> <path> Sync drive directory to local directory
gdrive [global] sync upload [options] <path> <fileId> Sync local directory to drive
gdrive [global] changes [options] List file changes
gdrive [global] revision list [options] <fileId> List file revisions
gdrive [global] revision download [options] <fileId> <revId> Download revision
gdrive [global] revision delete <fileId> <revId> Delete file revision
gdrive [global] import [options] <path> Upload and convert file to a google document, see 'about import' for available conversions
gdrive [global] export [options] <fileId> Export a google document
gdrive [global] about [options] Google drive metadata, quota usage
gdrive [global] about import Show supported import formats
gdrive [global] about export Show supported export formats
gdrive version Print application version
gdrive help Print help
gdrive help <command> Print command help
gdrive help <command> <subcommand> Print subcommand help
& That's It! It's worth checking your backup files regular, and testing you can access them correctly ever so often to be safe.
Let me know in the comments if this helped or if you have any questions.