rbxi backup script - rsync support now! v-2.5
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
Now fully rewritten, tested and working in most areas.

Impatient? Install it and just read the setup directions
This script is easiest to run if you leave it in a user directory, say, ~/bin (ie, /home/username/bin ), so, assuming it's going there:

:: Code ::
cd ~/bin && wget -Nc smxi.org/rb && tar xjvf rbxi.tar.bz2

This is the bzip 2 file containing the install directory, the script, and its data subdirectory, rbxi-data, where the user configuration file + excludes + readme-rbxi.txt file all are found.

I've tried to make the directions as clear as I can, but it's likely something will be confusing since there are a lot of options.

I'd like to catch some bugs before I release the next real stable, 2.0.0, so if someone wants to download the package with this url smxi.org/rd, see rbxi rsync backup homepage for install suggestions, or see smxi.org rbxi install.

<update>
Version: 2.5.0 :: Date 15 January 2008
In rbxi, a small new option to use alternate rbxi-values files. See rbxi -h for information. To update, just do: rbxi -U
The main change however was in changing from the readme-rbxi.txt file to a more readable and easy to navigate readme-rbxi.htm file.

You can download the new file if you want it like this:
:: Code ::
# first change directory to your rbxi-data location
wget -Nc http://rbxi.googlecode.com/svn/trunk/rbxi-data/readme-rbxi.htm

then view the file in any browser.
Version: 2.4.1 :: Date: 26 December 2008
Removed broken test for destination backup directory, just do: rbxi -U
to update to newest fixed version.

<update>
Version: 2.4.0 :: Date: 23 December 2008
Large internal engine upgrade, except for better comments on rbxi-values, and a better and improved readme-rbxi.txt, which I recommend getting, no updates except
rbxi -U
required. I did some debugging on rsync ssh local network syntax and methods, and it's tested basically, and seems to be working for both local and remote backups, of local or remote machines.

<update>
Version: 2.3.1 :: Date: 21 December 2008
Fixed an issue that would only affect you if the script failed to find the values file. Just do: rbxi -U
to update to fixed version.

<update>
Version: 2.3.0 :: Date: 21 December 2008
Added color schemes. -j <0-5>
To update, in rbxi-values:
:: Code ::
Find: # miscellaneous script booleans
Under, add: ( this isn't working yet, but it will be done soon, so just add it)
B_LOGGING='false' # -L : use logging true/false

Find: B_UPDATE_NOTIFIER='false'
Under, add:
# try each one and see which works best for your terminal colors using -j <0-5>, then set it below.
# script color scheme: 0 - no colors; 1 - default simple, for dark on light or light on dark
# 2 - fancy dark on light; 3 - basic light on dark; 3 - fancy light on dark; 4 - smxi style
COLOR_SCHEME='1' # -j [0-5]

Then update rbxi engine with: rbxi -U

<update>
Version 2.2.x is a smaller update from 2.1.x

New Option: update notifier, user optional set in rbxi-values, will let you know if a newer version exists if you want. If not, it won't check.

New Argument: -r - Use --dry-run for rsync. Dry run is a very good testing tool, highly recommended to use every time you test a new backup job.

New Jobs item: add job name to job variable. This will print out the job name, which is of course easier to remember than some number, when you run the job, and it will enter it into the history as well, so you can see it when you next run rbxi, or check your history with rbxi -v.

These are the only required changes in your rbxi-data/rbxi-values file:
:: Code ::

# find:
# This is triggered by the -J <1-10> option.
# After, add:
# syntax: 'job-display-name: <job options>'
# then change all the BACKUP_JOB..='' to
BACKUP_JOB..=':'

# find:
B_TESTING_2='false' # -! 2 :testing flag
# under, add:
B_UPDATE_NOTIFIER='false' # if you want rbxi to notify you of newer versions, set true.

then update rbxi main engine by running the update command:
:: Code ::
rbxi -U

And that will bring it all into sync from 2.1.0. If you are using an earlier version than 2.1.0, just redo it, there are too many changes, but from now on, I'll always include, if required, directions on how to update your config file.

<update>
Sorry, but version 2..x is a major change again from version 2.0.x, and requires that you redo your rbxi-values file, but it's worth it, the new version is much better laid out and easy to read and understand, and has a more complete, better organized readme-rbxi.txt file.

<update>
ALERT: Version 2.x.x requires that you redo your user config file, rbxi-data/rbxi-values, the variable names have all be made more consistent, so the old one won't work. Better to break the compatibility now than later I think.

Major improvements are:
  • rsync support added.
  • Improved error handling, options, error trapping, less likelihood of error.
  • All core logic updated and improved.
  • Update history displays pre backup, or with rbxi -v
  • User configuration file so you don't have to update your configs with every release.
  • With rsync, massively improved network backup speeds. I found rdiff-backup to be far too slow over the network in general. Example: A laptop with maybe 10 gigabytes of data: back up time: 13 minutes for first backup, then just a few minutes for each subsequent backup.
  • Script svn
  • And most important, a shorter name, rbxi, that's easier to type and remember than rd-h2.sh was.

Give it a try! Post feedback. This isn't an instant on script, you need to set the rbxi-data/rbxi-values file variables manually, one time, the default excludes for both rdiff-backup and rsync should work well, and with some practice you can get them more fine tuned for your specific system, the rsync ones are heavily tested now, and should work fine for average systems running Debian. The rdiff-backup isn't quite as fine tuned, because it takes so long to run each test.
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
I did a massive update of the 1.x.x, just uploaded the zips, this is now handles 10 different backup jobs, with 10 mount options, 10 backup directories, 10 separate data backups, etc.

This was enough to warrant the 2.0.x release, so that's what it is now.

Mix and match, the setup is not super easy, but it can't be with this degree of complexity.

Basically, I made this handle anything and more that I could conceive of, or that I have needed in the past, present, or possible future.

I am probably being ambitious calling this ready for prime time, 2.0.x, but as far as i can tell, the exclude lists seem to now be fairly safe, although those may get tweaked over time, we'll see.

If anyone wants to test this, I'd appreciate feedback, but I understand that the setup might be a bit extreme, but it's actually fairly simple is you ignore all the numbered options and just use the default first values for each variable.

New readme too, probably not complete, it was hard to keep up with documentation of the changes.

Hope this benefits someone, this was made to scratch a long time personal itch, and also to work on small scale office systems etc that might have much more complex backup needs than a standard setup.

Here's to always coding in structured, coherent style, so we can actually grow and develop scripts over time, rather than having to fully rewrite or start over.
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
Version: 2.1.0

Huge upgrade, this version is NOT compatible with older versions, so you'll have to redo your rbxi-values file. But the new readme-rbxi.txt and rbxi-values files are now much more clear, the variable names are tightened and more readable, the sections are better defined, and I removed some variables that don't work as user set. This is much closer to the final version I think, it fixes a lot of little issues I didn't realize would come up if I used the Clone function.

Also added excludes for trash files in all exclude files, and completely reworked the Clone option, which required significant tightening of the core script functions to work effectively.

Added a few more items to history logging. Hopefully the user data file rbxi-values will now stay somewhat stable for a while now. I'll see how it goes, it was better to change now than to stay with some of the old formats.

I think from now on, any changes to rbxi-values can simply be posted as simple updates to the existing file, not total changes.

Ok, sorry about the large changes, but Clone required a lot of work to get working reliably as I envisioned, now it works perfectly. Jobs, Mount/Backup drive etc options are all tested and working as well.

Needless to say, Clone does NOT work with rdiff-backup (only rsync) since rdiff-backup creates its own data files in the backup directory.
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
Moved exclude question to its own rysnc exclude syntax thread.
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
Version 2.2.2

Fixed some smaller output issues, and fixed colors so they work on black on white terminals.

If you have 2.2.0 or later, you can just update: rbxi -U
Back to top
Ghstryder
Status: Interested
Joined: 08 Sep 2008
Posts: 24
Location: Michigan
Reply Quote
Works like a charm. I can see that I have some reading to do and some work with the exclude files.

The dry run feature is great!

I (re)discovered why I hadn't been religious about backups. I really didn't have a suitably formatted partition. I have a small NAS device to back up from windows, but it is FAT32. Ah well, I just stole some space from windows and created a partition, it wasn't using it anyway.

It is funny, every time I embark on one of these little adventures, I discover more things where my knowledge was tenuous at best.

Thanks again, h2!!
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
Ghstryder, tell me about it, me too. The learning curve for me with rsync wasn't trivial here, and I tried to document and script in the worst parts, and to give good rsync exclude samples and explanations.

rbxi was much harder to get to a reasonably stable state than I thought it would be, just plowing into and through the differences in rsync and rdiff-backup exclude syntax took about 2 days, filled with so many tedious test runs of backups, which are ok with rsync because it's so fast, but aren't quite as cute with rdiff-backup, which is super slow.

Glad it's working for you, let me know if after you use it a while things could be improved.

Something you'll find of exceptional use, added because it was... well, because rd-h2 didn't have this feature and it really bugged me, was the report feature: rbxi -v
That way you never have to wonder where or when the backup was done.

Actually, I just realized, I should probably add one more line to that history data, the actual job arguments used.

Glad you like it so far, if you're like me, after a while certain things might bug you as being missing etc, that's what happened to me, but i tried to pack in as many possible ideas as I could come up with into rbxi as possible to avoid having to redo it too often.
Back to top
rbxi howto
jyp
Status: New User - Welcome
Joined: 11 Jul 2008
Posts: 1
Location: montreal
Reply Quote
I am quite a newbie and very grateful to run sidux with the help of smxi, a great script.

Tried to run rbxi after reading carefully the readme-rbxi.txt file but my computer background being so thin that I had a hard time making sense of everything.

I edited the rbxi-values file with the minimum of input (no exclude or include entries) and to see what was going to happen, tried a dry-run by typing as root

:: Code ::

# ~/bin/rbxi/rbxi


but the script exited with the following message (I was not too suprised!)

Checking backup source ........INVALID
Error 20: The backup source directory: does not exist. Unable to continue.

It seems logical to setup the source directory somewhere or pass it as an argument to rbxi but I did not find any clue in the readme file; all is about the destination file.

Is the syntax for rbxi something like
rbxi -options <source>

Or could it be that one must put lines in an exclude file like
+ /home
+ /home/data
- /tmp
- /var

If possible, an example would be of great help. (I already read the rsync-sample-exclude-file)

=======
Thank you.
Back to top
techAdmin
Status: Site Admin
Joined: 26 Sep 2003
Posts: 4018
Location: East Coast, West Coast? I know it's one of them.
Reply Quote
the excludes are irrelevant to any error, those are simply things you don't want backed up.

The directory you are backing up TO must exist, and you must either mount it, or, if have it mounted already, you must NOT use the mount option.

The file rbxi/rbxi-values explains this more, and has some basic examples.

You have to set all the data in rbxi/rbxi-values to correspond to your system, and to your backup directories, and backup sub directories.
Back to top
Display posts from previous:   

All times are GMT - 8 Hours