% git-annex is magic % eins78 and contributors
git-annex is magic{data-state="start" data-background="http://upload.wikimedia.org/wikipedia/it/0/00/Matrix_Neo.jpg"}
a
shell adventure
\ {data-background="http://upload.wikimedia.org/wikipedia/commons/5/53/The.Matrix.glmatrix.1.png"}
Open your Terminal{data-background="http://upload.wikimedia.org/wikipedia/commons/5/53/The.Matrix.glmatrix.1.png"}
This is a
shell adventure
You can (safely) follow along by typing or copying
all the commands into your terminal.
You'll need a relatively recent Linux or Mac system, and maybe
an USB stick. The Stick can be simulated by any folder
on the computer, so you can also try it out on any machine.
Commands are lines start start with a '$' sign,
the '$' is not part of the command.
Try it out now:
date "+%s"1385670018
^ There are snippets of the expected output, where it is important.
For educational purposes, we define an
alias,
so it is more clear when we
are using git or git annex.
alias magic="git annex"We will be defining more variables in
this adventure,
so make sure to not close your shell
(or edit .bashrc).
Official instructions and downloads for every OS are here:
https://git-annex.branchable.com/install/
Linux 32 Bit:
cd ; wget 'http://bit.ly/tuxmagic32' -O git-annex-standalone-i386.tar.gzLinux 64 Bit:
cd ; wget 'http://bit.ly/tuxmagic64' -O git-annex-standalone-i386.tar.gzOS X 10.9 Mavericks:
curl 'http://bit.ly/macmagic9' > ~/anx.dmgOS X 10.8.2 Mountain Lion:
curl 'http://bit.ly/macmagic8' > ~/anx.dmg.bz2cd ~; tar xvzf anx.dmg.bz2OS X 10.7.5 Lion:
curl 'http://bit.ly/macmagic7' > ~/anx.dmgLinux:
tar xvzf git-annex-standalone-i386.tar.gzAPP="~/git-annex.linux"OS X:
open ~/anx.dmgDrag and Drop App to /Applications
APP="/Applications/git-annex.app\
/Contents/MacOS"-
It should work via
shellandssh! -
Linux: at beginning of '/etc/bash.bashrc'!
- user must have shell
/bin/bashin/etc/passwd
- user must have shell
-
OSX: ~/.bashrc and ??? for ssh
-
If it doesn't work:
~/.bashrcis enough for this adventure
export PATH="$PATH":"$APP"check install{#s1 data-background="hsl(150, 50%, 35%)" data-background-transition="slide"}
- in a
shell
magic versiongit-annex version: 5.20131117-gbd514dc
build flags: Assistant Webapp …
key/value backends: SHA256E WORM …
remote types: git directory…
- via
ssh(optional)
ssh localhost 'git annex version'git-annex version: 5.20131117-gbd514dc
…
Works?
Hardcore install.
Do this only if everything else failed
or
you are on a Mac and want to use it over ssh.
sudo ln -s "$APP/git-annex{,-shell} /usr/bin/."sudo ln -s "$APP/bundle /usr/bin/bundle" # on Linux, there is no $APP/bundleinitadd $FILESclone- setup &
sync use
We start with a new folder. This could be your new Dropbox.
mkdir ~/magicfoldercd ~/magicfoldergit initInitialized empty Git repository …
magic init 'My Laptop'init My Laptop ok
(Recording state in git...)
We make a sub-folder with a new text file inside.
cd ~/magicfoldermkdir foo && echo 'simsalabim' > foo/bar.txtmagic add foo/bar.txtadd foo/bar.txt (checksum...) ok
(Recording state in git...)
git commit -m 'added'[master (root-commit) 67a2114] added
1 file changed, 1 insertion(+)
create mode 120000 foo/bar.txt
To make things easier, you can
always use sync.
Should be done after every change
(or in intervals w/ cron).
It does the following:
- auto-commit
- push to all remotes
- fetch from all remotes
- merge with all remotes
cd ~/magicfolderecho '1' > one.txtmagic synccommit
ok
Now clone this repo
using any git transport (file, ssh, https,…).
Note that file works with everything
you can mount somehow,
including Dropbox, SFTP, SMB, NFS, etc..
STICK="/Volumes/USBSTICK"
Clone:
git clone file://$HOME/magicfolder "$STICK/magicfolder"Alternative Clone: SSH
git clone user@host:magicfolder "$STICK/magicfolder"
Init:
cd "$STICK/magicfolder"magic init 'My USB Stick'init My USB Stick
ok
(Recording state in git...)
magic sync(merging origin/git-annex into git-annex...)
(Recording state in git...)
commit
ok
pull origin
ok
push origin
Writing objects: 100% (8/8), 776 bytes …
* [new branch] git-annex -> synced/git-annex
ok
We seem to have our file…
ls foo/bar.txtfoo/bar.txt
(bear with me for a second)
<iframe src="//giphy.com/embed/Vaswdccu781aw?html5=true" width="480" height="480" frameBorder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe>Standard git setup.
For the Laptop:
cd ~/magicfoldergit remote add stick "$STICK/magicfolder"
For the stick:
cd "$STICK/magicfolder"The origin remote is not used
by magic, can be removed or not.
git remote remove origin # optionalgit remote add laptop ~/magicfolderWith the configured remotes, sync now
also does pull and push.
cd ~/magicfoldermagic sync(merging synced/git-annex into git-annex...)
commit
ok
pull stick
From /Volumes/USBSTICK/magicfolder
* [new branch] git-annex -> stick/git-annex
* [new branch] master -> stick/master
* [new branch] synced/master -> stick/synced/master
ok
We have just cloned the meta data.
All the files and folder are on the stick, as symlinks!
magic takes care of the symlinks.
Metadata includes info about the remotes, and what files they have available!
cd "$STICK/magicfolder"ls foo/bar.txt # foo/bar.txtWe don't have the file's content yet,
because we have not transfered it
(just its meta data).
We'll call this: "The file is unavailable".
cat foo/bar.txtcat: foo/bar.txt: No such file or directory
If we get a listing, we see where the symlinks points to
(but the object file it points to – the content — is just not there).
ls -l foo/bar.txtlrwxrwxrwx 1 mfa staff 191 Nov 28 14:02
foo/bar.txt -> ../.git/annex/objects/FP/vj/
SHA256E-s11--9062535d58….txt/
SHA256E-s11--9062535d58….txt
So, how do we get the content?
To manually transfer files, use get.
cd "$STICK/magicfolder"magic get foo/bar.txt
cat foo/bar.txtsimsalabim
To delete a file just from
a remote, use drop.
This is useful for making free space on Laptops with small hard drives, etc.
cd "$STICK/magicfolder"magic drop foo/bar.txtcat foo/bar.txt # > "No such file or directory"magic syncwait…, what?{data-background="http://upload.wikimedia.org/wikipedia/tr/0/05/Matrix_yeni_nesil_ajanlar.jpg"}
dropping?
Isn't that a fancy word to sayDELETE?
is this safe?
(unless you use the --force)
This is why we want to have a record of all files and in which remotes they are.
Let's try 'dropping' it on our Laptop, too:
cd ~/magicfolder/magic drop foo/bar.txtdrop foo/bar.txt (merging synced/git-annex into git-annex...)
(unsafe)
Could only verify the existence of 0 out of 1 necessary copies
(Use --force to override this check, or adjust annex.numcopies.)
failed
git-annex: drop: 1 failed
What if I want to push a file to the stick?
copy --toa remote- also:
move --to!
cd ~/magicfolder/magic copy foo/bar.txt --to stickcopy foo/bar.txt (to stick...)
SHA256E-s11--9062535d58….txt
11 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1)
sent 173 bytes received 42 bytes 430.00 bytes/sec
total size is 11 speedup is 0.05
ok
(Recording state in git...)
We can also just get a list of a
file availability in other remotes,
without trying to drop it:
cd ~/magicfoldermagic whereis foo/bar.txtwhereis foo/bar.txt (2 copies)
ab8aad2f-f87e-440e-baef-… -- here (My Laptop)
f7feb954-0250-4117-a368-… -- stick (My USB Stick)
ok
-
OK so far? Cool.
-
Not ok?
- remove all
cd ; \ sudo rm -rf \ ~/magicfolder \ "$STICK/magicfolder"GOTOinit
Questions?
Files are locked by default!
Also, while the content
is not tracked in git by magic,
you can still commit with git, and magic
will take of it (w/ git-hooks).
cd ~/magicfolderecho 'fail' > foo/bar.txtPermission denied
We need to unlock it first!
magic unlock foo/bar.txtecho 'booya' > foo/bar.txt # No error!
Now "commit and push".
magic add foo/bar.txtgit commit -m 'changed foo bar'magic syncNote: We didn't really have to git commit!
magic sync would have done that for us,
but committing let us put in our own message.
To get the file to the stick,
cd "$STICK/magicfolder"we sync, which fetches the meta data
with git
and then get . (everything) to actually
transfer the file.
magic sync && magic get .cat foo/bar.txtbooya
…"locking", eh?{data-background="http://upload.wikimedia.org/wikipedia/en/a/ac/TheArchitectMatrix.jpg"}
Doesn't this locking/unlocking seem complicated
in comparison to, say, Dropbox?YES. That is why we use 'direct mode'.
You may now forget the slides about locking/editing.
cd ~/magicfoldermagic direct…
direct ok
All available files are normal files;
all unavailable files are symlinks
(and we can use magic get to make them available).
Do something:
echo 'pow' > foo/bar.txtmagic syncSync it!
cd "$STICK/magicfolder"magic sync && magic get .cat foo/bar.txt #pow
Note: We could also have used copy --to stick.
Usage: If you just want to copy somewhere, for backup or transfer.
If you'll look at the directory, it will just be a bunch of object files.
Special remotes are (GPG) encrypted
by default,
so you have to explicitly turn it off
(encryption=none).
Once setup with initremote,
magic can
use them like any other remote – it just works™.
The remote will have a name and a GPG key.
BOX=dropbox # how to call the remote
BOX_PATH="~/Dropbox/BACKUP/magic/"KEY=965113EA # YOUR GPG keyid - don't take mine ;)Make a new folder in the Dropbox:
mkdir -p "$BOX_PATH"cd ~/magicfoldermagic initremote $BOX type=directory \
directory="$BOX_PATH" \
keyid=$KEYinitremote dropbox (encryption setup)
(hybrid cipher with gpg key 3771835A3BADB56D) ok
Alternative if you don't have a key handy (unecrypted):
magic initremote $BOX type=directory \
directory="$BOX_PATH" \
encryption=none
Like any other remote, just a password prompt for the key.
cd ~/magicfoldermagic copy . --to $BOXcopy foo/bar.txt (gpg)
You need a passphrase to unlock the secret key for
user: "Max F. Albrecht <[email protected]>"
4096-bit RSA key, ID 965113EA, created 2013 (main key ID FOO)
(to dropbox...)
ok
xmpp(jabber!)rsyncbupwebwebdav- (Amazon) S3
- Amazon Glacier
- More in the Docs
-
how files are matched with they meta data.
obviously very important. -
default is
SHA256E, a file hash plus the file extension -
WORM: alternative if you don't want hashing, uses just file name, date, etc. usefull if: a) you trust the disk or b) you don't care c) performance (2 TB Videos @ Raspberry Pi)
cd ~/magicfolderecho '* annex.backend=WORM' > .gitattributesUse the direct mode for normal folders,
and the indirect mode for normal git repos + magic.
Use sync to sync meta data between remotes.
Use get, drop, copy and move
to sync content between remotes.
Questions?
assistant{data-background="http://31.media.tumblr.com/48aff1d20ba83e294a987bcf601858c9/tumblr_mgtp10uvr41rywaz8o1_500.gif" data-background-transition="slide"}
- daemon
- watches folders (also: just
magic watch) - show repos, remotes, transfers
- remote settings
cd ~/magicfoldermagic assistant # or 'magic webapp' to re-open
internals
- uses direct mode by default
- uses only special remotes by default
- good bc. encrypted
- apart from that, everything done manually still works!
-
--fastdo stuff faster.
For example, rely on local data instead of updating before checking. -
magic fsck: for the data conscious -
magic describe: change the description
that was set withmagic init 'My Laptop'. -
always try to use the faster machine for hashing…
made with pandoc and reveal.js.