We are using Bacula to store backups of all our servers on one big storage system. That made us sleep better for some days but then we thought it would be good to have another backup at a remote site (because a fire or another disaster could destroy servers and backup at once). Usually, one would just copy the backups over network to that remote site. Unfortunally, this was not an option for us because of some special conditions not worth to mention here. We decided to carry backup hard drives to that remote site instead.

The backup system is a 2U server with 12 hot swappable hard drive bays. 10 of them are bundled as RAID-5 to store all full (weekly) and incremental (daily) backups. The other two hard drives are mirred (RAID-1) and have the latest full backups saved on it. Once a week, we replace one hard drive from RAID-1 and carry it to that remote site. That is it: A remote backup copy with almost no extra costs.

Backups are done by Bacula automatically. All we have to do is to remove one hard drive from backup server and replace it. To make this easy, I wrote a small script that helps us doing it. It uses tw_cli, the monitoring and management software for 3ware RAID controllers. There are also Debian packages and repositories available.

#!/bin/bash

TW_CLI=/usr/sbin/tw_cli
DATE_CMD=/bin/date
WEEK=$($DATE_CMD  %V | sed s/^0//)
let "DRIVE=($WEEK%2) 18"
CONTROLLER="/c0"
SHOW_RAID_STATUS="$TW_CLI $CONTROLLER show"
RAID_CHECK_CMD="$TW_CLI /c0 show allunitstatus"
REMOVE_DRIVE="$TW_CLI $CONTROLLER/p$DRIVE remove quiet"

$SHOW_RAID_STATUS

if $RAID_CHECK_CMD | grep -q 'Not Optimal Units = [^0]'; then
    echo "RAID status is not optimal. Aborting."; echo
    exit 1
fi

echo "You are about to remove VPort p$DRIVE. This will degrade RAID status!"
read -p "Are you sure? (yes/no)"
if [ "$REPLY" == "yes" ]; then
    $REMOVE_DRIVE
fi

It checks RAID status (line 14) and if it is in an optimal state, it chooses a hard drive and removes it (line 22). Because we do not want to change the same hard drive all the time, we choice hard drive by whether calendar week is odd or even (line 6).

I have just noticed that one of our servers were connected with 100baseTx instead of 1000baseT. Both end-points support 1000baseT but it was not advertised.

# mii-tool -v eth0
eth0: negotiated 100baseTx-FD, link ok
  product info: vendor 00:50:43, model 11 rev 1
  basic mode: autonegotiation enabled
  basic status: autonegotiation complete, link ok
  capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
  advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
  link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD

First idea was to check cable. It was the same Cat 5e cable we always use. However, I replaced it with another one of the same type and suddenly Gigabit Ethernet was working again.

# mii-tool eth0
eth0: negotiated 1000baseT-FD flow-control, link ok

I saved my OpenVPN certificates in ~/.openvpn, imported the configuration in NetworkManager and expected it to working properly. But in Fedora SELinux prevents OpenVPN from reading the certificates and private keys from that directory. The policy expects the files in directories labeled as home_cert_t, which are by default ~/.cert/ or ~/.pki/. Ok, I copied the files accordingly and executed restorecon to make it work:

restorecon -R -v ~/.pki ~/.cert

Another way would be to label ~/.openvpn as home_cert_t:

semanage fcontext -a -t home_cert_t "/home/user/.openvpn(/.*)?'
restorecon -R -v /home/user/.openvpn

It is annoying when you realize that all you have done to the file that you are currently editing in vim cannot be saved because you need root permissions. Thanks to the masters of commandlinefu I know how to do it anyway.

:w !sudo tee %

Subversion stores some metadata of working copies in hidden directories. If your deployment process uses a working copy in the document root of your Apache vhost, those files are available to the public. The best way to hide those files is to respond with HTTP error 404 (file not found).

RedirectMatch 404 /\.svn(/|$)

I have just commited some graph templates to mysql-cacti-templates.

main( ) {
    printf("hello, world");
}

Hier war noch ein alter Siemens ADSL-Router (SL2-141-I) in Betrieb, der ersetzt werden sollte. Die Zugangsdaten für den DSL-Provider und DynDNS waren auf ihm konfiguriert, aber sonst unbekannt. An diese Daten zu kommen versuchte ich und zwar mit Erfolg.

Über das Webinterface konnte man ein Backup der Konfiguration anlegen. In der resultierenden XML-Datei standen auch die gesuchten Zugangsdaten, jedoch nicht im Klartext. Die Passwörter sahen aus, als wenn sie ins hexadezimale Format umgewandelt wurden. Dieses in ASCII zurückzuverwandeln sollte kein Problem sein. Das Ergebnis war jedoch nur Sonderzeichensalat. Also schaute ich mir die Daten genauer an und entdeckte eine Regelmäßigkeit. Es dauerte nicht lange und es war gelöst: Der dezimale ASCII-Wert jedes Zeichens wurde von 255 abgezogen und hexadezimal gespeichert. Mit diesem Wissen war es nun leicht die gesuchten Zugangsdaten zu ermitteln.

Die Funktion decrypt im folgenden Perl-Skript nimmt das hexadezimale Passwort entgegen und gibt es im Klartext wieder zurück. Zur Vollständigkeit gibt es auch eine Funktion zum “Verschlüsseln” (encrypt), die aber angesichts es trivialen Algorithmus besser nicht eingesetzt werden sollte.

#!/usr/bin/perl

use strict;

sub encrypt {
    my $string = shift;
    my $result;
    for ( unpack "C*", $string ) {
        $result .= sprintf uc "%x", 255 - $_;
    }
    return $result;
}

sub decrypt {
    my $hex_string = shift;
    my $result;
    for ( $hex_string =~ m/../g ) {
        $result .= chr( 255 - hex $_ );
    }
    return $result;
}

print decrypt( encrypt('mysecretpassword') ) . "n";

Simpler Patch, damit phpMyAdmin für jede Verbindung zu Datenbankservern die Zeitzone setzt. Das war hier erforderlich, weil eine Änderung der default time_zone bei AWS RDB nicht möglich ist.

--- libraries/dbi/mysqli.dbi.lib.php        2010-06-26 12:53:06.000000000  0200
    libraries/dbi/mysqli.dbi.lib.php-new    2010-06-30 16:23:54.000000000  0200
@@ -137,6  137,10 @@
         PMA_DBI_postConnect($link, $is_controluser);
     }

     // Dirty hack to set session time_zone because we cannot
     // change default time_zone in AWS RDB.
     mysqli_query($link, "SET time_zone = 'Europe/Berlin';");

     return $link;
 }

Von unseren Instanzen in EC2 werden viele Mails über einen Smarthost verschickt. Amazon limitiert die Anzahl der Mails von Instanzen auf Port 25 jedoch (aus verständlichen Gründen (Stichwort: Spam)) und empfiehlt den Einsatz von Elastic IPs und die Anmeldung als Mailversender bei ihnen. Für uns kommt das nicht in Frage, da wir ja unseren Smarthost nutzen. Also habe ich schnell Submission (Port 587/tcp) auf dem Smarthost aufgemacht und die EC2-Instanzen liefern nun dahin aus, was Amazon nicht mehr zu beanstanden hat.