There is an always haunting problem of backup plans. Geographic distribution of backups. If you ever planed for a solid backup plan, you probably know the drill. The rule of three: Three copies, 2 different formats, 1 different place. There is also a adage that says only the paranoid survive. So having more of any part of the rule are always welcome with cost. Any addition to your backup structure costs you your personal time, computing power and money in different forms.

If you have relatively unlimited resources, good for you. You can muster all your resources and blast your way to your goal with it. For most people backups are so daunting they don’t even bother further than uploading their, sometimes most private information, to non-free storage services where they are screened and used against them including the society. This option is most of the time appears OK besides silently enslaving it’s users to hostile faceless companies and sometimes end up in total disaster when their account gets locked/deleted without a reason and appeal possibility.One can only discover once they do not own their own data, once they do it is most probably already too late.

Probably any slightly technical minded reader will suggest a nextcloud instance spun on a VPS at this moment. While it is a good replacement to shit-company owned, gratis file hosting, rented VPS is still someone else’s computer. You still can loose your access to your data without your involvement and also you are puting your eggs in the same basket where most other people put theirs too. Baskets are flammable. Also these VPSs are way too expensive to just backup occasional data or keep TB worth of them. Most people’s will and perceived requirement does not warrant such expense.

alt-text

OK, one might point out the possibility of just manually syncing data to a medium and sending it out to somewhere they deemed safe. While this is quite a good solution not if best in some conditions, where it fails is the labor intensive process. You may burn blu-rays and mail them or sync hard drives time to time from different locations. First you have to keep track of which data is where, second you need to manage the data delivery physically -everytime- and third keep a schedule and enthusiasm to do all that chore frequently. If your data is static, i.e long term cryptographic material, that is totally fine and robust method but besides those conditions you will run out of mental energy to keep up.

The Box⌗

While making a very, very complicated backup plan I arrived to the conclusion of a need for remote off-site backup system that will be placed strategical trusted locations where I have relatively unrestricted physical access. It needed to be able to work over regular consumer network, low powered, cheap and secure. This way, delivery and management of very security sensitive and high frequency data can be distributed in the background without even disturbing my mind or my fellow’s who hosts my box.

Most of us right away would think about RaspberryPi or any other single board computer for this task. While their plan is excellent with most parameters except cost. These little boards which started their way to be affordable computers are not very affordable anymore. They are still cheap but sometimes buying 2010 era netbook is cheaper and least they come with screens a battery. When SBC’s out of the window, there is still capable consumer devices that we all have collecting dust in some box in backroom. Routers! They are small embedded devices which are capable of connectivity and includes most GNU tools. Welcome to free world of LibreCMC

Plan is simple. To create a plug and play backup box;

  • Find the cheapest LibreCMC supported router which has enough resource
  • Flash it with LibreCMC
  • Get it connected to The Internet somewhere safe
  • Communicate with it, somehow?
  • Backup to it regularly
  • Encrypt everything

Modern Internet infrastructure sucks. Having a regular consumer connection means you are behind a NAT and whatever you are hosting will not be reachable. Without NAT obstructing the way, this project, along with many other, would be much more easier. Sadly we are not there yet and current states of IPv6 looks there is still some time. While hosting a server from home can warrant the cost of an static IP, probably it is not possible for your friend to bear that cost for your small project. So a NAT traversal is required. This capability also able your boxes to work anywere in the world with a simple Wlan connection. For this purpose I used my own self hosted server as an SSH proxy. Including that, shopping list for this project is below:

  • A LibreCMC/Openwrt router (TP-Link TL-WR842ND)
  • A USB storage (Any flash storage or external HDD)
  • Publicly available server

Plan is like this:

┌──────────────────┐                    ┌─────────────────┐
│                  │SSH Reverse Tunnel  │                 │
│ LibreCMC Router  ├──────────────────► │  Proxy Server   │
│                  │      Port 22       │  ─────┬───────  │
└──────────────────┘                    │       │         │
                                        │       ▼         │
┌──────────────────┐                    │  Localhost 2222 │
│                  │       SSH/SCP      │ ─────────────── │
│ Backup Source    ├──────────────────► │                 │
│                  │      Port 2222     │                 │
└──────────────────┘                    └─────────────────┘

Let’s Start⌗

  1. Flash your router with your choice of firmware. I chose LibreCMC for being a free software project. There is limited devices that is supported. I tried MR3020 and it was evident that hardware won’t cut it. I had a TP-Link TL-WR842ND lying around from our Freifunk Istanbul project and probably that is least can be used for this project.

  2. When your router boot up stick your storage which is pre-formated to ext4 into an USB port and follow Openwrt’s excelent guide to get it working. The only thing that I made different is the mount point of the storage device. The thing is that SCP refused to write to any other directory other than /root. That problem is circumvented by changing it’s mount point under /root. To do that, one need to edit /etc/config/fstab and change fstab.@mount[0].target='/mnt/sda1' with fstab.@mount[0].target='/root/<choice of folder name>'.

  3. After setting up storage follow Jeff Geerling’s wonderful guide for raspberry and prepare your SSH prox and setup router for SSH. Just omit autossh for persistent tunnel since another software will be used for that purpose.

Some notes:

  • Do not encrypt SSH private key of the router. There is no secure way to keep the encryption key and a reboot will lock you out.
  • Put your SSH public keys to router’s /home/<user>/.ssh/authorized_keys file to be able to SSH into the device
  • Create a separate user for SSH authentication on the server. I named my new user uncreatively sshtunnel.
  • For obvious security reasons I did not like the idea of having anyone who gets to one of my routers to be able to SSH into my server. I disabled shell access to my ssh user by usermod -s /sbin/nologin <account name>.
  • It is good practice to use a domain address for ssh relay server in case it’s IP changes. A quick DNS change can fix remote devices connection.
  • If you do keep your proxy server behind a firewall just remember to forward 2222 inbound port to servers internal 2222 port.
  1. SSH Tunnel is a persistent SSH reverse proxy software found in LibreCMC/Openwrt repo’s. Just install it through web interface or from command line with opkg update && opkg install sshtunnel after that edit /etc/config/sshtunnel file and follow Openwrt’s guide for ssh tunnel. You need an tunnelR setup and put your servers information. As an example of my config file looks like this:
config server sshrelay
	option user			sshtunnel
	option hostname			backup.alperatmaca.com.tr
	option port			22
#	option retrydelay		1	
#	option CheckHostIP		yes
#	option Compression		no
#	option CompressionLevel		6
#	option IdentityFile		~/.ssh/id_rsa
#	option LogLevel			INFO
#	option PKCS11Provider		/lib/pteidpkcs11.so
#	option ServerAliveCountMax	3
#	option ServerAliveInterval	0
#	option StrictHostKeyChecking	accept-new
#	option TCPKeepAlive		yes
#	option VerifyHostKeyDNS		yes

# tunnelR(emote) - when the connection will be initiated to the R(emote) endpoint at
# remoteaddress:remoteport and then forwarded to localaddress:localport
#
config tunnelR http
	option server		sshrelay
	option remoteaddress	*
	option remoteport	2222
	option localaddress	127.0.0.1
	option localport	22

When you reach at this point, you should have your router getting a persistent reverse SSH connection to your server and you should be able to SSH into router by SSH' to your proxy server by ssh -p 2222 <user>@my-vps-hostname-or-ip. When you achieved this you can simply send your router anywhere in the world where an internet connection available and you will be able to SSH into the router.

Welcome to the Machine⌗

At this point you have a remote machine with storage ready to SSH into. There are several methods to copy files and run backup operations on this system.

  1. SCP: The first protocol that comes to mind when file transfer operations are in question with SSH. SCP can be used manually to copy contents of directories, scripted into a cronjob or used in conjunction with a backup tool such as Duplicity.

  2. SFTP: This is another obvious option. I am not familiar with it so it is up to you to experiment with.

  3. SSHFS: Like many other remote file systems, one can mount remote points with SSH and Fuse. This will create a local mount point where any kind of file operations can be done. If a protocol agnostic system needed this method can be advantageous. The down side is fuse can be tricky some times and can cause issues as a transfer layer. Had my past with it so I skipped this option too.

Security⌗

Security is another topic. Maybe you trust your friends not to look at what’s on the drive and reasonably sure that those devices will never be stolen or get to the wrong hands. Perhaps your threat model does not warrant burden of encryption for your funny cat videos. In those cases you may simply copy your files and be with it. Transport of the files are protected by SSH’s encryption but at rest your files will be unencrypted. There is two ways to handle this with common tools. Either utilize a file base encryption such as GPG to every file either manually or automatically or mount the remote filesystem and use a file system based encryption such as Gocrypt. My prefered way is utilize GPG encryption with scheduled incremental backups with duplicity.

duplicity incr --verbosity=5 --asynchronous-upload --encrypt-key=<GPG key fingerprint> --full-if-older-than 60D <local directory to backup> scp://root@<routers address>

Besides encrypting your data physical security is also needs to be considered. While it is probably best to keep the whole router and drive where least access is, this plan does not give you much playing room since you will be crashing someone else’s living space. It is probably possible to lock everything in a lightweight safe and tamper evidently lock it. The problem with this approach is that there is not much relative security gained by this huge investment. One can argue the posibility of gaining private SSH key or alter the operating system of the device. Both has very limited uses for an attack if you plan carefully. Especially if you limit your sshtunnel user and perhaps limit IP’s. Unless you are storing unencrypted data on these devices probably altering the firmware won’t do anything. So probably glitter painting of seams and USB hub and GPG signed photographs of them would be sufficient.

Speed⌗

alt-text

This is yet another limitation of the system. Where we are going, we don’t need gbps networking. Most of residential connections are usually 35 mbps at max. While this is highest limiting factor, the low powered device and cryptographic operations it needs to do while SCP is running where everything hit bottleneck. I did not made a empirical calculation but 50MB worth of files backed up with Duplicity around 5-7 minute. Which also includes file indexing and encryption on local machine(a measly Lenova x200). Probably it is fast enough for my needs but if you try to shove GB’s worth of data at once you probably have to keep your machine running for a while and also have to explain your friend why their DRM infested streaming service does not work.

Some rough measurements are below. To mimic distance host machine is connected to a VPN in Europe:

Default settings

0de35bc9-047b-480d-be42-e3dfa709ab41.jpg	100%  374KB 137.8KB/s   00:02    
100000000000015900000159FFC173995786A674.png	100% 1293     9.0KB/s   00:00    
1000.jpg					100%  277KB 141.9KB/s   00:01    
1039609_10151746757217125_139219092_o.jpg	100%  479KB 142.2KB/s   00:03    
425205193_127566.jpg				100%  163KB 128.4KB/s   00:01    
75e17b05-aabf-470f-8dd4-e7384974c4ec.jpg	100%  203KB 139.5KB/s   00:01    
8605eb546066fd2b.jpeg				100%   53KB 112.5KB/s   00:00    
absolutely_anti-freedom.xcf			100% 1515KB 128.7KB/s   00:11    
ssss.png					100%  552KB 143.1KB/s   00:03    
yyyyyy.odt					100% 2867KB 148.7KB/s   00:19

aes128-ctr chipher

0de35bc9-047b-480d-be42-e3dfa709ab41.jpg	100%  374KB 138.2KB/s   00:02    
100000000000015900000159FFC173995786A674.png	100% 1293     9.0KB/s   00:00    
1000.jpg					100%  277KB 142.5KB/s   00:01    
1039609_10151746757217125_139219092_o.jpg	100%  479KB 142.4KB/s   00:03    
425205193_127566.jpg				100%  163KB 128.7KB/s   00:01    
75e17b05-aabf-470f-8dd4-e7384974c4ec.jpg	100%  203KB 141.0KB/s   00:01    
8605eb546066fd2b.jpeg				100%   53KB 112.4KB/s   00:00    
absolutely_anti-freedom.xcf			100% 1515KB 148.2KB/s   00:10    
ssss.png					100%  552KB 143.0KB/s   00:03    
yyyyyy.odt					100% 2867KB 150.9KB/s   00:18

With compression

0de35bc9-047b-480d-be42-e3dfa709ab41.jpg	100%  374KB 137.8KB/s   00:02    
100000000000015900000159FFC173995786A674.png	100% 1293     8.9KB/s   00:00    
1000.jpg					100%  277KB 142.5KB/s   00:01    
1039609_10151746757217125_139219092_o.jpg	100%  479KB 142.6KB/s   00:03    
425205193_127566.jpg				100%  163KB 132.1KB/s   00:01    
75e17b05-aabf-470f-8dd4-e7384974c4ec.jpg	100%  203KB 140.9KB/s   00:01    
8605eb546066fd2b.jpeg				100%   53KB 111.4KB/s   00:00    
absolutely_anti-freedom.xcf			100% 1515KB 146.9KB/s   00:10    
ssss.png					100%  552KB 148.3KB/s   00:03    
yyyyyy.odt					100% 2867KB 149.4KB/s   00:19

Direct network connection

0de35bc9-047b-480d-be42-e3dfa709ab41.jpg	100%  374KB   1.3MB/s   00:00    
100000000000015900000159FFC173995786A674.png	100% 1293   462.6KB/s   00:00    
1000.jpg					100%  277KB   1.3MB/s   00:00    
1039609_10151746757217125_139219092_o.jpg	100%  479KB   1.3MB/s   00:00    
425205193_127566.jpg				100%  163KB   1.3MB/s   00:00    
75e17b05-aabf-470f-8dd4-e7384974c4ec.jpg	100%  203KB   1.3MB/s   00:00    
8605eb546066fd2b.jpeg				100%   53KB   1.3MB/s   00:00    
absolutely_anti-freedom.xcf			100% 1515KB   1.3MB/s   00:01    
ssss.png					100%  552KB   1.3MB/s   00:00    
yyyyyy.odt					100% 2867KB   1.3MB/s   00:02

Without VPN on the same wlan 100/10mbps fiber

0de35bc9-047b-480d-be42-e3dfa709ab41.jpg	100%  374KB 760.8KB/s   00:00    
100000000000015900000159FFC173995786A674.png	100% 1293   271.2KB/s   00:00    
1000.jpg					100%  277KB 768.1KB/s   00:00    
1039609_10151746757217125_139219092_o.jpg	100%  479KB 759.1KB/s   00:00    
425205193_127566.jpg				100%  163KB 748.6KB/s   00:00    
75e17b05-aabf-470f-8dd4-e7384974c4ec.jpg	100%  203KB 747.3KB/s   00:00    
8605eb546066fd2b.jpeg				100%   53KB 749.8KB/s   00:00    
absolutely_anti-freedom.xcf			100% 1515KB 739.3KB/s   00:02    
ssss.png					100%  552KB 770.5KB/s   00:00    
yyyyyy.odt					100% 2867KB 750.3KB/s   00:03

Final Thoughts⌗

  • While everything can be encrypted an attacker still can delete your files or destroy your devices if they can get physical access. This is inevitable and it is why you have multiple backups.
  • The system reduces your anonymity and give away the structure of your backup distribution with some hint of your social circles. If someone is determined to destroy your data, a discreet location is highly advised. Hiding these devices behind a VPN or TOR is computationally not feasible without significantly upgrading them.
  • Related to signaling your social graph, a capable attacker who can monitor whole network can trace your friends and run rubber-hose cryptanalysis on them. If you will keep anything warrants such worry, probably doing something else or warn your friends about this possibility is a good choice.
  • It is still fragile duct taped system. A lot to maintain and far away from being simple. Cheap hardwares are being cheap can give up the ghost unexpectedly. You may lock your self out and most probably your friend cannot do much besides ineffectively restarting your box. A software update can ruin it all. This system looks like require some maintenance but time will tell.