Through the looking-glass, and what onefang found there, my boring Devuan mirror adventures.

At the start of August 2017 I setup an ISO mirror for Devuan on my European server https://SledjHamr.org/devuan/. Since I had been asked to report on the bandwidth used by this mirror, I thought I might just document my adventures here. But first a little background.

I live in Australia, a first world country with expensive third world Internet. I blame Telstra, an ex government monopoly ISP that was privatised, but still owns a lot of the Internet infrastructure in Australia. In particular they own the fibre in my entire suburb, and all the surrounding suburbs. They also own six of the ten WiFi access points (AP) I can see from my home, of the remaining APs two are a competing ISP, one is “NETGEAR” (likely a private one that hasn't been configured), and my own AP. There are no copper phone lines for DSL, Telstra owned all of that, but ripped it all out to replace it with fibre. Telstra at least allow other ISPs to sell bandwidth on their fibre, so that's what I'm doing. However, since it is still Telstra Fibre, it's even more expensive than usual, but I have no choice, unless I move far away. My ISP at least provides all sorts of useful stuff that is bandwidth quota free, like Netflix, plenty of Internet radio stations (including the one I like the most), and a file mirror of many popular Linux distros. So I can get all Debian stuff for free, but not Devuan. I asked them long ago to add Devuan, but so far no response. I recently reminded them of this request, including pointing to this document to make it easy for them. My Internet bundle includes a land line phone (that I don't want, don't use, don't even have a phone to plug into it, but pay extra for, coz Telstra), a mobile phone with minimal data, and 30 GB of 30/1 fibre bandwidth per month, for the grand total of AU$85.30 per month. To add insult to injury, the land line phone Telstra makes me pay for doesn't even exist, it's just an unused virtual circuit on the single bit of fibre to my home.

Many years ago some friends and I started up a virtual world server. After doing our homework we decided to go with a server hosting company in Amsterdam. Initially one of the others paid for it in her name, but it was eventually transferred to my name and me paying for it. To start with it was a VPS, but after several years I upgraded it to a real server, which was a bit more powerful than the VPS, but cheaper. I love this server company. It's a Xeon E3-1230 3.2ghz, with 16 GB RAM, 500 GB spinning rust hard drive, and 5 TB of 1000/1000 bandwidth per month. Waaay more bandwidth than we need, but I couldn't negotiate them below 5 TB. For all of this I pay 109 Euros per month.

In case you don't know, a Euro is usually worth more than an Australian dollar, but you can easily see that Aussie bandwidth is bloody expensive, and European bandwidth is bloody cheap. Mostly both end up using about the same amount of bandwidth each month, 30 GB each. Which means I have more than 4 TB of European well connected and fast Internet bandwidth per month that I'm not paying a lot for, and not using. Hmmm, what to do with it? I know, mirror Devuan. B-)

Actually, before that I tried a Bitcoin experiment. Over a year ago the hard drive on that server started reporting problems, so I asked the hosting company to replace it. The server is a proper rack mounted server with hot swappable hard drives, so they just plugged in a new one. I copied everything across to it, rebooted, tested, then told them it was all done and they could pull the old one out now. They where not in a hurry to pull it, so it stayed there for almost a year. Did I mention that I love this hosting company? So earlier this year I thought “what can I do with this excess bandwidth and extra hard drive space”, then decided to experiment with Bitcoin. I setup a full Bitcoin node on the spare disk, which involves downloading the entire blockchain, keeping it up to date, and serving bits of it to anyone that asks. Eventually the hosting company pulled that disk, and there wasn't enough room on the main disk for the blockchain, so that was the end of that experiment. I got good results from that though, so I might pay for a bigger disk sometime in the future.

I did still have plenty of space for a Devuan ISO mirror, so at the beginning of August I set that up. The instructions for doing so where simple to follow for an experienced Linux sysadmin like myself, though it didn't go precisely to the plan, with some minor changes needed, it wasn't a simple copy paste of the shell commands. As per the documentation, once I had it up and running, I reported it to the #devuan-dev IRC channel so they could add it to the Devuan mirror list. Thus begins my boring adventures. Boring coz apparently August is exactly the wrong time to get anything done in Europe. It seems most of the relevant Devuan developers and admins are European. August is European holiday season, all the European tourists visit some other part of Europe, where the locals get upset about the number of tourists, then go on holiday elsewhere in Europe to get away from all the damn tourists.

It took most of August until my “new” mirror was finally listed on the official Devuan mirror list, though EvilHam was quick to list it on his unofficial mirror list. While I was waiting EvilHam and golinux where most helpful. Evilham suggested I add an issue similar to https://git.devuan.org/devuan-editors/devuan-www/issues/113 including adding “fdo” and “mirror” labels, which a new account couldn't do so I didn't. The result was https://git.devuan.org/devuan-editors/devuan-www/issues/149 which I later pointed to another person doing the same thing as me, which resulted in https://git.devuan.org/devuan-editors/devuan-www/issues/150 .

Once my mirror was publicly listed, people started using it. I had long ago promised Lydia_K to report on my bandwidth usage, which I mentioned at the start of this page. So here are my reports so far. The graphs cover a month of the server hosting companies billing, not exactly a calendar month.

This is the reference graph of my servers bandwidth usage. A typical months use, mostly not much, with one big spike between week 23 and 24. I use my European server as a proxy for most of my web usage, and near the end of my Aussie ISPs billing period I use up what ever I have left of my 30 GBs of Aussie quota, usually on pr0n. That's what that big spike is. Otherwise a fairly quiet month, only 35.4 GB used in total.

This is the graph of my bandwidth usage for the first month of running my mirror, more or less August. At the beginning of week 31 you can see a big green spike, that is when I initially rsynced the ISOs. Nothing much happened until the middle of week 34, which was shortly after my server got listed publicly. Then people started downloading ISOs. Total bandwidth used 50.8 GB, mostly my usual 30 GB, plus the initial ISO download. You can see that the outgoing bandwidth is only slightly higher than the reference month above, incoming is way higher.

Septembers bandwidth usage. This looks like it might be typical usage for an ISO mirror. A few times a week, someone downloads an ISO or three. 67 GB of total bandwidth used, mostly the outgoing ISOs. This time the incoming bandwidth is a bit higher than the reference month, but outgoing has tripled.

For October, things got a little quieter. Also near the end of October, my server hosting company upgraded their web site, and managed to break the display of network traffic. They still collect the stats, and still generate the graphs, I just can't see them on their site. Their support people emailed the October graph to me, and promised me they are working on fixing things. It's now the end of the year, and that's still not been fixed. I have started looking at creating my own network traffic graphs, as well as other related things, but most of the solutions suck. So there's gonna be a hole in the record. On the plus side, my new solution will separate Devuan mirror stuff from the other things I do with my server, and will split HTTP, HTTPS, FTP, and RSYNC.

All the bandwidth usage for what this server was originally built for is just noise, swamped by the mirror bandwidth, and still over 4 TB of unused bandwidth left to soak up with something useful. This boring mirror adventure isn't over yet. I've been hanging out for Devuan package mirror ability to be ready for testing, something that was promised shortly after I started. I volunteered to help test that. I now have a working package mirror, and I can see the light at the end of the rabbit hole. To be continued …

onefang's adventures in Mirrorland.

This is a much shorter tale, setting up a package mirror. On the devuan-dev mailing list, KatolaZ posted a draft of https://SledjHamr.org/devuan/devuan_mirror_walkthrough.txt, and asked for volunteers to test it out before putting out a call for mirrors. Since I had been hanging out for this, I jumped to it straight away. It went smoothly, with once again only minor differences in what I actually did. What didn't go so smoothly is the above mentioned server hosting companies web site change, so I don't have traffic graphs to show yet. They promise me that will be fixed next week.

For the official package mirrors an ssh key is required. In my eagerness to get started, I had initially used the wrong account, and hence sent the wrong key. I created a new unsigned key for this job and asked KatolaZ to replace it. This helped to shake out a minor permission problem on the master package server, so not a complete waste of time.

I tested it by changing the sources.list of my various test Devuan installs on my little test box to point to http://SledjHamr.org/devuan/merged/ then using that to upgrade, including upgrading Devuan Jessie to Ascii, as well as Debian Jessie to Devuan Jessie, and then to Ascii.

There was one little extra hack that I added. As mentioned above, bandwidth in my country is ludicrously expensive, especially compared to bandwidth in the country my server is located, and my ISP runs a bandwidth quota free mirror of Debian. Devuan's package mirror instructions caters for two different ways of setting them up, either mirror both Devuan and Debian packages, or just the Devuan mirrors and redirect to Debian servers for packages shared with Debian. Since the Debian mirror is about 2 TB, the Devuan portion is only 8 GB, and I only have a 500 GB hard disk on my server, I chose the redirect option. The one little extra hack is that I test for my home's fixed IP, and redirect to my ISPs quota free mirror instead of the standard Debian servers. So for that part of Devuan, I can install and update quota free from home.

The Devuan developers are soon to announce a DNS round-robin among available package mirrors, for the ASCII release. Not much to say about this, the above walkthrough document has been updated, and below I'll update my configurations.

As promised in the previous chapter, here's an updated bandwidth graph, though still not the broken out per protocol graphs. Using collectd to collect the data, and CGraphz to plot it, the graph looks different to the previous ones. Bandwidth in this new one is in bytes per second, rather than the bits per second. The details are in different places, there is no overall total, the graph resolution is lower, and the colours are swapped.

The ISO download pattern is similar, the incoming data has more than doubled, and the outgoing data has tripled, since October.

Advice from a Caterpillar.

Here's the configuration I ended up using for both mirrors.

I host my mirror server on a Debian 8 server, running Apache 2, vsftpd, and rsync. /srv/mirrors/files.devuan.org is the directory it all lives in, though I have bits of that bind mounted to /srv/ftp for FTP security. /srv/mirrors/files.devuan.org contains the various directories rsynced from the master package server, and devuan-cd, which is separately rsynced from the master ISO server. The standard Debian /var/www/html directory contains symlinks to devuan and devuan-cd directories within /srv/mirrors/files.devuan.org. The files are owned by a newly created mirrors user, in the standard www-data group. Those bind mountings are -

mount --bind /srv/mirrors/files.devuan.org /srv/ftp/devuan
mount --bind /srv/mirrors/files.devuan.org/devuan-cd /srv/ftp/devuan-cd

The rsync commands are run by cron at six hourly (ISOs) and three hourly (packages) intervals. While the mirror instructions say “no more than once every hour”, we have been asked to do it every three hours. The rsync commands are -

rsync --chown mirrors:www-data --delete -avX files.devuan.org::devuan /srv/mirrors/files.devuan.org/devuan-cd
rsync --chown mirrors:www-data -avX vesta@pkgmaster.devuan.org:~/devuan/ /srv/mirrors/files.devuan.org

You can remove the v option from -avX to reduce the email reports. The X option is not actually needed, I just add that from habit. I should probably add –delete to the package mirror as well.

To keep the rsyncing from interfering with everything else on my server I add these commands to the beginning -

/usr/bin/ionice -c3 /usr/bin/nice -n 19

Also I do filter the output of the package mirror rsync to keep the cron emails down to a dull roar, by adding this to the end -

| grep -Ev '/$|\.gpg$|\.gz$|\.xz$|\.txt$|Release$'

On top of that, after I was happy everything was running smoothly, I no longer needed to see emails every three hours telling me nothing happened, so I also added this final bit of filtering -

| head -n -3 | tail -n +2

Note that this wiki wants to wrap lengthy code sections, which doesn't look so good for these two very long lines, so I broke them up into logical sections above.

My vsftpd configuration is mostly standard. The Apache configuration is also mostly standard, with standard Directory and Alias sections. I added rewrite rules to that section of my web server. The added Apache 2 configuration (with my home IP obscured for privacy) is -

<Directory /var/www/html/devuan-cd>
  Options Indexes FollowSymLinks SymLinksIfOwnerMatch
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>
Alias /devuan /var/www/html/devuan-cd
 
<Directory /var/www/html/devuan>
  Options Indexes FollowSymLinks SymLinksIfOwnerMatch
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>
Alias /devuan-cd /var/www/html/devuan
 
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^/devuan/merged/pool/DEVUAN/(.*) /devuan/devuan/pool/$1 [L,R]
  RewriteRule ^/devuan/merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1 [L,R]
 
  # Special rule for onefang, Internode's mirror isn't on HTTPS, but at least it's quota free.
  RewriteCond %{REMOTE_ADDR} =xxx.xxx.xxx.xxx
  RewriteRule ^/devuan/merged/pool/DEBIAN/(.*) http://mirror.internode.on.net/pub/debian/pool/$1 [L,R]
  RewriteRule ^/devuan/merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1 [L,R]
</IfModule>

The [L,R] bits added to the ends of the rules is to get around problems with later bits of my web rewrite rules. Makes them the last rule, and turns them into redirects. The HTTPS section has similar rules, redirecting to https obviously.

For the DNS round-robin, a new virtual host is needed, the configuration I use is -

<VirtualHost *:80>
        ServerName deb.devuan.org
        DocumentRoot /srv/mirrors/files.devuan.org                                                         
        <Location />
            Require all granted
        </Location>
        RewriteEngine on
        RewriteRule /merged/pool/DEVUAN/(.*) /devuan/pool/$1 [L,R]
        RewriteRule /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1
        RewriteRule /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1
</VirtualHost>

This is slightly different from the configuration mentioned in the walkthrough, I added the “Location” stanza, and the “[L,R]” bit at the end of the DEVUAN rewrite rule, which makes my server pass the automated tests. They may or may not be needed for other mirror servers.