• Home
  • Cloud
    • General
    • SaaS
    • BPaaS
    • PaaS
    • IaaS
    • Other Internet Hosted Applications
      • WordPress
        • WooThemes Canvas
          • WooThemes Canvas CSS
  • About me
  • Why Badly Wired?
  • Contact

Badly Wired

Alan's technical notebook - WordPress and other stuff

You are here: Home / Cloud / Google Compute Engine Snapshot automation

Google Compute Engine Snapshot automation

22nd December 2016 by Alan Leave a Comment

Fully Managed UK Hosting - Only £1+VAT till 1st Jan 2021 on Shared, Reseller and Dedicated Hosting! .... read more ....

Google Compute Engine comes with an incremental snapshot capability.

The product doesn’t have a specific automated method to take snapshots and to tidy them up either, but scripting using the gcloud command line is fairly straight forward, once you understand the power of the command line.

Fully Managed UK Hosting - Only £1+VAT till 1st Jan 2021 on Shared, Reseller and Dedicated Hosting! .... read more ....

I have seen many attempts at automating this using bash scripts, python and even ruby, but in the main complex scripting isn’t actually required.

For my purpose I have used bash as the scripting language, but you will see that all is required is two simple loops and could be implemented in any scripting language that can execute commands.

Before we begin, you will need to have install the Google Cloud SDK which includes the gcloud command line interface (cli) this is available for most platforms here. https://cloud.google.com/sdk/downloads

Additionally, it is well worth understanding a bit about Persistent Disk snapshots, in the way they work incrementally.

Compute Engine stores multiple copies of each snapshot redundantly across multiple locations with automatic checksums to ensure the integrity of your data. You cannot share snapshots across projects.

If you delete snapshot 1 all the data in 1 gets migrated into snapshot 2. As it is incremental you are not really duplicating a lot of data, all depending on the volatility of the data on your server.

You would have to delete ALL snapshots for a disk to effectively start again, otherwise you are always building on the previous image.  For systems that are relatively static this makes a very cost effective and simple snapshot solution.

Also read the part about consistent state of data. In a highly volatile data environment suspending updates and flushing data is important, for a more static environment (e.g. serving simple websites ) this is less so.

The use case for this script is to list all disks in a project  ( regardless of what server they are mounted on ) and to take a snapshot.

The second function, is simply to delete older snapshots where granularity is no longer required, remember deleting old snapshots doesn’t lose data it just pushed it into the more recent snapshots. Strictly speaking there is no real need to delete snapshots, but too many would be unmanageble.

Therefore we need two loops.

The elements of loop 1 are:

gcloud compute disks list --format='value(name,zone)'

Which lists all disks in a project  – the format=’value(name,zone)’  forces the command to return a tab delimited list without headings, containing name & zone. There are other format options, see https://cloud.google.com/sdk/gcloud/reference/topic/formats , so you can choose what is best for you, but this works nicely with a bash loop.

then

gcloud compute disks snapshot $DISK_NAME --snapshot-names autogcs-$DISK_NAME-$(date "+%Y-%m-%d-%s") --zone $ZONE

which uses the list to create snapshots appropriately prefixed and named ( so we can find the ones created by the script.

The elements of loop 2 are:

gcloud compute snapshots list --filter="creationTimestamp<$(date -d "-60 days" "+%Y-%m-%d")" --regexp "(autogcs.*)" --uri

Which lists all snapshots will a creation date less than, in this case, 60 days. This is driven by the –filter= expression, it seems that the filter handles dates and partial dates without any need for specific conversions. It also only lists snapshot with a prefix of “autogcs-” that we set earlier, to stop deleting other manual snapshots, this is done by the flag –regexp=”(autogcs.*)” (although I suspect that this could be incorporated into –filter).  The final flag –uri returns a full path needed in the delete.

and then for each found

gcloud compute snapshots delete $SNAPSHOT_URI

Which should be self explanatory.

If you would like the full bash script and also details on how to install and set up cron job you can find it here  https://gitlab.com/alan8/google-cloud-auto-snapshot

Please leave any comments or suggestion below or at gitlab.

[Next] Find out where to host WordPress [Read the full article…]

Filed Under: Cloud, Code snippets, Linux, Server setup, Tech Tips  

About Alan

I'm Alan from Fullworks Digital Ltd, where I develop WordPress Plugins .

My day job consists of developing new code and solutions along with support my WordPress plugin user.

I started as a professional programmer in 1979 and had been involved with the IT of business technology in virtually every area that exist.

Badlywired.com is my technical notebook, my aide memoire of the many interesting facts that I come across and 'how to' recipes of things I do infrequently. As I spend a lot of time gathering parts of solutions from the internet and assembling them into my own solutions, and also just learning how to do things, this blog is one way of giving something back to the online community that has helped me extensively.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Categories

  • Applications
  • Cloud
    • General
    • Google Cloud
    • IaaS
    • Other Internet Hosted Applications
      • Wordpress
        • WooThemes Canvas
        • WooThemes Canvas CSS
    • SaaS
  • Code snippets
  • Discounts
  • Genesis
  • Google Apps for Works
  • Linux
  • News
  • SEO
  • Server setup
  • Services
  • Tech Tips
  • Uncategorised
  • Useful Images
  • Useful Stuff
  • WordPress Hosting
  • WordPress Plugins

Tags

background jobs beadcrumbs bind brandings Cache canvas Centos chrome css fail2ban Find firefox Flash fraud genesis gocardless godaddy Google google maps hackers internet explorer javascript KashFlow Linus linux Magento mapquest maps microsoft mysql news nohup php plugin plugins queens diamond jubilee replace SED SEO skype Varnish Virtualmin Webmin woothemes Wordpress

 

Affiliate and Privacy Notices

This site is free to use, but hopes to cover some costs through affiliate income, some products and links are affiliates and may earn the site advertising income.

Some affiliates use Cookies to track if you purchase from them, this allows them to apportion revenue to us you will need to refer to their specific privacy notices as to how you are tracked.

This site is a participant in the Amazon EU Associates Programme, an affiliate advertising programme designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.co.uk.

  • Privacy Policy

Copyright © 2021 · Badly Wired