• Home
  • Cloud
    • General
    • SaaS
    • BPaaS
    • PaaS
    • IaaS
    • Other Internet Hosted Applications
      • WordPress
        • WooThemes Canvas
          • WooThemes Canvas CSS
  • About me
  • Why Badly Wired?
  • Contact

Badly Wired

Cloud solutions, technical snippets and other goodies

alan
You are here: Home / Cloud / Google Cloud / Backing up to gcloud storage from Linux/Virtualmin

Backing up to gcloud storage from Linux/Virtualmin

5th August 2016 by Alan Leave a Comment

Using Google Cloud Storage as backup for your linux server is inexpensive if you use the nearline storage option. This is cheaper that Amazon Web Services (at the time of writing)

The process I use is as follows.

  1. Backup locally
  2. Create storage bucket / folders, set lifecycle
  3. Script to move backups
  4. find way to execute script

As I use Virtualmin as my control panel, the process details they way to use it from Virtualmin, but you can tailor to your own needs.

I’m assuming you have already signed up to Google Cloud Console and set up a project / billing as required.

The process uses a local backup copy and then moves / deletes it. So if you are limited in working space, you may need to divide you backups in to workable tranches.

You will need gstutils loaded load onto your server using the Google SDK https://cloud.google.com/sdk/docs/ and properly authorised – e.g. via gcloud init

1. Using virtualmin to create scheduled backups

 

vim

e.g. creating backups into the local directory /home/backups/files

 

2.  Make a Google Storage Bucket

You can use the Google Cloud Console to do this, but as you have the gsutil command and you will need to get used  to it, here is how to do it from the CLI.

I’m assuming assuming here you want to store in EU but you can use US or ASIA as required for -l (location)

I recommend the ‘class’ to be ‘nearline’ as this is the cheaper option -c nl

(change mybucket to a unique name of course)

1
gsutil mb -c nl -l EU gs://mybucket/

set life cycle e.g. 35 days to delete  or as required, that way your storage bills won’t grow forever.

create rule file as below (35 days)  and run

1
gsutil lifecycle set lifecycle_config.json gs://mybucket/

1
2
3
4
5
6
7
8
9
{
    "rule":
    [
      {
        "action": {"type": "Delete"},
        "condition": {"age":35 }
      }
    ]
}

3. create script in /files/backups

1
2
3
4
5
6
7
#!/bin/bash
#today variable as YYYY-MM-DD
NOW=$(date +"%Y-%m-%d")
#Upload all files to our dated folder in the selected bucket
gsutil cp /home/backups/files/* gs://mybuckets/$1/$NOW
#Delete all the files after uploading them
rm /home/backups/files/*

Takes a folder argument allows different backups to use it within the same bucket  (folders get created automatically)

e.g.  /bin/sh /home/backups/upload.sh  weekly

make sure your script is executable

1
sudo chmod +x /home/backups/upload.sh

 

4. find ways to execute script

With Virualmin this is easy, in schedule set command to run after backup

schedule

 

 

Recovering backups

Of course to recover backs requires copying locally and then restoring too – but with gsutil that is fairly easy.

 

list your weekly backups

1
gsutil ls gs://mybucket/weekly

e.g.

gs://mybucket/weekly/

gs://mybucket/weekly/2016-07-31

gs://mybucket/weekly/2016-07-25/

gs://mybucket/weekly/2016-08-05/

 

get the full directory (copy recursively) for 2016-08-05  back into /home/backups/files

1
gsutil -m cp -r dir gs://mybucket/weekly/2016-08-05/ /home/backups/files

Then restore  (in my case using Virtualmin) in the normal way

 

Filed Under: Google Cloud, Linux, Tech Tips  Tagged: Backups, Google Cloud, Virtualmin

About Alan

I'm Alan from Fullworks Digital Ltd, where I develop WordPress Plugins and support and manage WordPress websites.

My day job consists of solving clients' WordPress issues and developing new code and solutions.

I started as a professional programmer in 1979 and had been involved with the IT of business technology in virtually every area that exist.

Badlywired.com is my personal blog and my aide memoire of the many interesting facts that I come across. As I spend a lot of time gathering parts of solutions from the internet and assembling them into my own solutions, and also just learning how to do things, this blog is primarily my 'note book' and a way of giving something back to the online community that has helped me extensively.

Click on a tab to select how you'd like to leave your comment
  • Badly Wired
  • Twitter
  • Facebook
  • Google
Login
Login

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Categories

  • Applications
  • Cloud
    • General
    • Google Cloud
    • IaaS
    • Other Internet Hosted Applications
      • Wordpress
        • WooThemes Canvas
        • WooThemes Canvas CSS
    • SaaS
  • Code snippets
  • Discounts
  • Genesis
  • Google Apps for Works
  • Linux
  • News
  • SEO
  • Server setup
  • Services
  • Tech Tips
  • Uncategorised
  • Useful Images
  • Useful Stuff

Tags

background jobs beadcrumbs bind brandings Cache canvas Centos chrome css fail2ban Find firefox Flash fraud genesis gocardless godaddy Google google maps hackers internet explorer javascript KashFlow Linus linux Magento mapquest maps microsoft mysql news nohup php plugin plugins queens diamond jubilee replace SED SEO skype Varnish Virtualmin Webmin woothemes Wordpress

Subscribe to my Newsletter

Sign up to my occasional newsletter

 

Affiliate and Privacy Notices

This site is free to use, but hopes to cover some costs through affiliate income, some products and links are affiliates and may earn the site advertising income.

Some affiliates use Cookies to track if you purchase from them, this allows them to apportion revenue to us you will need to refer to their specific privacy notices as to how you are tracked.

This site is a participant in the Amazon EU Associates Programme, an affiliate advertising programme designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.co.uk.

  • Privacy Policy
  • Account
  • Register
  • Memberships

Copyright © 2019 · Badly Wired

This website uses cookies to improve your experience. Also when you click on affiliate links those links may use Cookies to track you to be able to apportion revenue to us. We'll assume you're ok with this, but you can opt-out if you wish.Accept Read More
Privacy & Cookies Policy