gdrive backups with service accounts

Every service needs backups, and a simple way to get these at a low cost is with Google Drive.
In this post I'll take you through a way to implement this using the Google Drive CLI Client.

The gdrive cli offers a simple way to upload, list, sync & more with a Google Drive account. By default this is done by connecting your own account which work fine but does allow access to all files in your account.
Instead I prefer to use the new service accounts feature that allows you to share access to a folder with a service account so only that folder is available.
The downside of using service accounts here is that we need to compile the gdrive package with go first but with go installed this is easy to do. Go is easy to install & you can find out more here.

Install

To compile gdrive from source you'll need to run: go get github.com/prasmussen/gdrive
This will download and compile the binary for you to $GOPATH/bin/gdrive. If you have the gopath bin folder in your $PATH then you can call gdrive straight away, but if not feel free to move this file to wherever you want, you could even scp it to another server.

Create a project in Google Developer Console.

If you don't already have one, you'll need to create a project in the Google Developer Console.
Once logged in if you don't have any projects yet it will normally prompt you to create one, otherwise if you already have projects but wish to use a new one or select a different one, you can do so by clicking the project name to the side of the Google APIs logo to the top left.

Enable Google Drive API access

On the left side menu of your project screen click library.
On here you'll need to find the Google Drive API and on it's page click to enable it. (the enable button often does not appear for a few seconds).

Create a service account

Back on the left side menu select Credentials & the click to Create credentials > Service account key.

Create a new service account, and give it a name you'll remember what it's for such as "backup".

Leave JSON selected and on create it will auto start downloading the json config file. You'll need this for gdrive later.

Create a folder for the backups in your Google Drive.

In Google Drive you'll now want to decide where to store these backups, for this demo I've created a backups folder to sync backups to.

Share access to the service account

Right click the folder and select Share.
Enter the email given to you for the new service account & untick to notify the user.

Get the backup folders file id

To test we need the file id the backup folder. If you navigate into the folder in your browser, the id will be i the url, e.g. in https://drive.google.com/drive/folders/xxxxxxxxxx the file id is xxxxxxxxxx

Test access via the gdrive

We need to check we can access the backup folder successfully.

By default gdrive expects the service account json file to be in ~/.gdrive so let's move the file here.

First make the folder: mkdir ~/.gdrive

Then run mv ~/Downloads/Demo\ Project-xxxxxx.json ~/.gdrive/sa.json

Now to run the test (replacing xxxxx with the folders file id):

gdrive --service-account sa.json info xxxxx  

If everything worked, this should return some basic stats on the folder.

If an error message shows please recheck the above steps or you can contact me via the comments or raise a ticket on gdrive's GitHub page.

Create a backup folder on the server or machine you want to be backed up

For this demo I'll be using a /backup folder (mkdir /backup).

& I'll want to link in a few core log files so they will be backed up regularly, e.g. on a web server running nginx, php & a laravel app i'd probably want at least these files:

ln /var/log/auth.log /backup  
ln /var/log/syslog /backup  
ln /var/log/nginx/error.log /backup  
ln /var/log/php7.0-fpm.log /backup  
ln /var/log/redis/redis-server.log /backup  
ln /app/storage/logs/laravel.log /backup  

Just as simply you could setup automysqlbackup and link in the files from this so that these files are auto backed up on drive too, or have it's folder backed up to another folder on drive.

First backup run & schedule

To actually run the backup we'll run a sync upload and we'll use the --delete-extraneous to delete any files in the drive folder that were in the last backup sync but are not there now:

gdrive --service-account sa.json sync upload --delete-extraneous /backup xxxxx  

You can then schedule a cron to run that above command for you daily.
e.g. crontab -e then add:

@daily /path/to/your/gdrive --service-account sa.json sync upload --delete-extraneous /backup xxxxx

This will also send you an email when it completes, though you could add > /dev/null to the end of that to ignore the output and not have it email you.

gdrive provides plenty other commands which you can find with:

~ $ gdrive help

gdrive usage:       

gdrive [global] list [options]                                 List files  
gdrive [global] download [options] <fileId>                    Download file or directory  
gdrive [global] download query [options] <query>               Download all files and directories matching query  
gdrive [global] upload [options] <path>                        Upload file or directory  
gdrive [global] upload - [options] <name>                      Upload file from stdin  
gdrive [global] update [options] <fileId> <path>               Update file, this creates a new revision of the file  
gdrive [global] info [options] <fileId>                        Show file info  
gdrive [global] mkdir [options] <name>                         Create directory  
gdrive [global] share [options] <fileId>                       Share file or directory  
gdrive [global] share list <fileId>                            List files permissions  
gdrive [global] share revoke <fileId> <permissionId>           Revoke permission  
gdrive [global] delete [options] <fileId>                      Delete file or directory  
gdrive [global] sync list [options]                            List all syncable directories on drive  
gdrive [global] sync content [options] <fileId>                List content of syncable directory  
gdrive [global] sync download [options] <fileId> <path>        Sync drive directory to local directory  
gdrive [global] sync upload [options] <path> <fileId>          Sync local directory to drive  
gdrive [global] changes [options]                              List file changes  
gdrive [global] revision list [options] <fileId>               List file revisions  
gdrive [global] revision download [options] <fileId> <revId>   Download revision  
gdrive [global] revision delete <fileId> <revId>               Delete file revision  
gdrive [global] import [options] <path>                        Upload and convert file to a google document, see 'about import' for available conversions  
gdrive [global] export [options] <fileId>                      Export a google document  
gdrive [global] about [options]                                Google drive metadata, quota usage  
gdrive [global] about import                                   Show supported import formats  
gdrive [global] about export                                   Show supported export formats  
gdrive version                                                 Print application version  
gdrive help                                                    Print help  
gdrive help <command>                                          Print command help  
gdrive help <command> <subcommand>                             Print subcommand help  

& That's It! It's worth checking your backup files regular, and testing you can access them correctly ever so often to be safe.
Let me know in the comments if this helped or if you have any questions.

comments powered by Disqus
Want to setup your own server? Digital Ocean offer $100 free for new accounts.
DigitalOcean Referral Badge