I've an AppEngine application that presently has about 15GB of information, also it appears in my experience that it's not practical to make use of the present AppEngine bulk loader tools to support datasets of the size. Therefore, I'm beginning to research different ways of copying, and would want to consider listening to practical solutions that individuals might have employed for copying their AppEngine Data.
Being an aside, I'm beginning to consider the Google Cloud Storage may well be a sensible choice. I'm curious to understand if anybody practical knowledge while using Google Cloud Storage like a backup for his or her AppEngine data, and what their experience continues to be, and when you will find any pointers or stuff that I should know before heading down this path.
Whichever solution I finish track of, I'd like a backup means to fix satisfy the following needs:
1) Reasonably fast to backup, and reasonably fast to revive (ie. if your serious error/data deletion/malicious attack hits this site, I'd rather not need to take it lower for multiple days while rebuilding the database - by fast I am talking about hrs, instead of days).
2) Another location and account from the AppEngine data - ie. I'm not going someone with admin use of my AppEngine data to always have write/remove accessibility backup data location - for instance if my AppEngine account is jeopardized with a hacker, or maybe a disgruntled worker would choose to remove my data, I must have backup copies which are outside of the AppEngine administrator accounts.
In summary, considering that obtaining the data from the cloud appears slow/painful, what I'd like is really a cloud-based backup solution that looks like the role that tape backup copies might have offered previously - basically were to possess a backup tape, nobody could customize the items in that tape - consider I can not obtain a tape, can one store a safe and secure copy of my data somewhere, that only I get access to?
You will find a couple of options here, though they are not (presently) quite what you are searching for.
Using the latest discharge of version 1.5.5 from the SDK, we currently support interfacing with Google Storage directly - you can observe how, here. With this particular you are able to write data to Google Storage, but to the very best of my understanding there is no method to write personal files the application will be not able to remove.
To really gather the information, you could utilize the App Engine mapreduce API. It's built-in support for conntacting the Application Engine blobstore conntacting Google Storage would need you to implement your personal output author, presently.
An alternative choice, as WoLpH indicates, is by using the Datastore Admin tool to support data to a different application. After some extra effort you can customize the remote_api stub to stop removes towards the target (backup) application.
One factor you need to certainly do regardless would be to enable two-factor authentication for the Google account this causes it to be a great deal tougher for anybody to obtain charge of your bank account, even when they uncover passwords.
The bulkloader is most likely among the quickest method to backup/reinstate your data.
The issue using the AppEngine is you need to fit everything in through sights. So you've the limitations that sights have... as a result a quick backup/restore still needs to make use of the same API's because the relaxation of the application. Therefore the bulkloader (possibly having a couple of modifications) is certainly the best choice here.
Possibly though... (haven't attempted it yet), you should use the brand new Datastore Admin copying the information to a different application. The one that you alone control. That method for you to copy it back in the other application if needed.