Installation of a SOGo Debian Server for Groupware use
Motivation
Since the release of Microsoft Outlook 2016, it is possible to connect clients via the ActiveSync protocol, mostly known for its use on mobile devices to synchronize Calender, contacts and e-mail. So the complicated MAPI protocol (i.e. offered by OpenChange project) isn’t necessarily needed anymore.
So my goal was to set up a Linux server that offers ActiveSync protocol interfaces as well as the “classic” IMAP, Caldav and Carddav interfaces for integration with e-mail clients such as Thunderbird. For this SOGo seems to be a very good solution, because it offers you to use the standard Unix daemons in your desired configuration and is just a component that you can easily plug in to offer access to all components with above mentioned protocols which makes the server very easy to maintain without the need to use new, unfamiliar interfaces to manage it like partly with other groupware servers.
So I describe some simple setup for SOGo 2 (because I like its webinterface) on a Debian Jessie Server for handling mail on one domain in the combination Dovecot IMAP server and Postfix MTA. It is a mix of stuff taken from several tutorials available on the Internet.
Installation
1) Install SOGo 2
echo deb http://packages.inverse.ca/SOGo/nightly/2/debian/ jessie jessie >>/etc/apt/sources.list apt-key adv --keyserver keys.gnupg.net --recv-key 0x810273C4 apt-get update apt-get install sogo mysql-server sope4.9-gdl1-mysql memcached
2) Fix some packages
sed -i "s/SHOWWARNING=true/SWOWWARNING=false/" /etc/tmpreaper.conf # suppress tmpreaper warnings sed -i "s/127.0.0.1/localhost/" /etc/memcached.conf # Fix IPv6 errors /etc/init.d/mysql restart /etc/init.d/memcached restart
3) Install MySQL and create user table
Unforutnately, SOGo doesn’t support PAM authentication for whatever reason. The most common solution in the tutorials is to use a LDAP server, which is of course also possible, but for me, it is harder to maintain due to my lack of LDAP-Skills. Therefore I decided to maintain a MySQL user database which will also be used by Dovecot for authentication and which I can add new shell users to with a simple shell script presented lateron. I prefer to have my mailbox users in /etc/passwd with a disabled login shell so that they automatically have a systemwide account, get their e-mails stored and delivered to their home-directories. That’s why I chose this method:
mysql -u root -p mysql
CREATE DATABASE `sogo` CHARACTER SET='utf8';
CREATE USER 'sogo'@'localhost' IDENTIFIED BY 'sogopasswd';
GRANT ALL PRIVILEGES ON `sogo`.* TO 'sogo'@'localhost' WITH GRANT OPTION;
USE sogo;
CREATE TABLE sogo_users (c_uid VARCHAR(60) PRIMARY KEY, c_name VARCHAR(60),
c_password VARCHAR(32), c_cn VARCHAR(128), mail VARCHAR(128),
home VARCHAR(255) NOT NULL, uid INTEGER NOT NULL, gid INTEGER NOT NULL);
FLUSH PRIVILEGES;
quit
4) Install and configure dovecot IMAP-Server
apt-get install dovecot dovecot-sieve dovecot-lmtpd dovecot-managesieved dovecot-sql dovecot-mysql
You can also install dovecot-pop3d if needed.
| /etc/dovecot/conf.d/10-auth.conf |
|---|
#!include auth-system.conf.ext !include auth-sql.conf.ext |
| /etc/dovecot/conf.d/10-mail.conf |
|---|
mail_location = maildir:~/Maildir |
| /etc/dovecot/conf.d/10-master.conf |
|---|
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
user = postfix
group = postfix
}
}
|
| /etc/dovecot/conf.d/15-lda.conf |
|---|
lda_mailbox_autocreate = yes |
| /etc/dovecot/conf.d/20-lmtp.conf |
|---|
protocol lmtp {
postmaster_address = postmaster@domain.com
mail_plugins = $mail_plugins sieve
}
|
| /etc/dovecot/conf.d/20-managesieve.conf |
|---|
service managesieve-login {
inet_listener sieve {
port = 4190
address = 127.0.0.1
}
}
|
| /etc/dovecot/conf.d/auth-sql.conf.ext |
|---|
passdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb {
driver = prefetch
}
userdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
|
| /etc/dovecot/dovecot-sql.conf.ext |
|---|
driver = mysql
connect = host=localhost dbname=sogo user=sogo password=sogopasswd
default_pass_scheme = MD5
user_query = \
SELECT home, uid, gid \
FROM sogo_users WHERE c_uid = '%n'
password_query = \
SELECT c_uid AS user, c_password AS password, \
home AS userdb_home, uid AS userdb_uid, gid AS userdb_gid \
FROM sogo_users WHERE c_uid = '%u'
|
4) Install the Postfix MTA
I assume that you have 2 DNS set up pointing to your mailserver: mail.domain.com and imap.domain.com, this also facilitates automatic mail server detection in most Mail applications.
apt-get install postfix postfix-policyd-spf-python
| /etc/postfix/main.cf |
|---|
myhostname = mail.domain.com
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
home_mailbox = Maildir/
mailbox_transport = lmtp:unix:private/dovecot-lmtp
# Restrictions
policy-spf_time_limit = 3600s
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
reject_unauth_pipelining,
reject_non_fqdn_recipient,
check_policy_service unix:private/policy-spf
smtpd_sender_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unknown_hostname,
reject_unknown_recipient_domain,
reject_unknown_sender_domain
smtpd_client_restrictions = permit_mynetworks,
permit_sasl_authenticated
|
| /etc/postfix/master.cf |
|---|
# SPF
policy-spf unix - n n - - spawn
user=nobody argv=/usr/bin/policyd-spf
|
5) Install Apache 2 webserver
apt-get install apache2 sogo-activesync a2enmod proxy a2enmod proxy_http a2enmod headers a2enmod rewrite a2enconf SOGo
6) configure Microsoft ActiveSync
Please read here carefully for related documentation.
apt-get install sogo-activesync
Uncomment and modify:
| /etc/apache2/conf-available/SOGo.conf |
|---|
ProxyPass /Microsoft-Server-ActiveSync \ http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync \ retry=60 connectiontimeout=60 timeout=660 |
7) Install Let’s encrypt SSL-certificates for your site
As self-signed certificates are always needed to be added to a trust list on the client side which makes setup for the user harder and we now have the possibility to get free SSL-certificates by the Let’s encrypt project, it is a good idea to use them instead. The downside of this may be the expiration time (3 months), but as renewal process can be automated with a cron-job, it’s not such a big issue.
| /etc/apache2/sites-available/000-default.conf |
|---|
ServerAlias mail.domain.com ServerAlias imap.domain.com |
echo "deb http://ftp.debian.org/debian jessie-backports main" >>/etc/apt/sources.list apt-get update apt-get install python-certbot-apache -t jessie-backports letsencrypt --apache -d mail.domain.com -d imap.domain.com
Change TLS parameters in Postfix:
| /etc/postfix/main.cf |
|---|
smtpd_tls_cert_file=/etc/letsencrypt/live/mail.domain.com/fullchain.pem smtpd_tls_key_file=/etc/letsencrypt/live/mail.domain.com/privkey.pem |
| /etc/dovecot/conf.d/10-ssl.conf |
|---|
ssl = yes ssl_cert = </etc/letsencrypt/live/imap.domain.com/fullchain.pem ssl_key = </etc/letsencrypt/live/imap.domain.com/privkey.pem |
Now we need to create a renewal-Cronjob for the requested certificates:
| /etc/cron.weekly/letsencrypt |
|---|
#!/bin/sh letsencrypt renew result=$(find /etc/letsencrypt/live/ -type l -mtime -1 ) if [ -n "$result" ]; then /etc/init.d/apache2 restart /etc/init.d/postfix restart /etc/init.d/dovecot restart fi |
chmod a+x /etc/cron.weekly/letsencrypt
8) Install DKIM
If you want to increase your e-mail reputation, it is a good idea to set up SPF and DKIM records. For DKIM, you can use opendkim daemon. This step is optional and requires full access to your DNS server.
This was shameslessly copied from here
apt-get install opendkim opendkim-tools mkdir /etc/opendkim mkdir /etc/opendkim/keys mkdir /var/spool/postfix/var/run/opendkim
| /etc/opendkim.conf |
|---|
# OpenDKIM agiert als Mail Filter (= Milter) in den # Modi signer (s) und verifier (v) und verwendet eine # Socket-Datei zur Kommunikation (alternativ: lokaler Port) Mode sv # Socket local:/var/run/opendkim/opendkim.sock # Socket inet:12345@localhost Socket local:/var/spool/postfix/var/run/opendkim/opendkim.sock # OpenDKIM verwendet diesen Benutzer bzw. # diese Gruppe UserID postfix:postfix UMask 002 PidFile /var/run/opendkim/opendkim.pid # OpenDKIM bei Problemen neustarten, # aber max. 10 mal pro Stunde AutoRestart yes AutoRestartRate 10/1h # Logging (wenn alles funktioniert eventuell reduzieren) Syslog yes SyslogSuccess yes LogWhy yes # Verfahren, wie Header und Body durch # OpenDKIM verarbeitet werden sollen. Canonicalization relaxed/simple # interne Mails nicht mit OpenDKIM verarbeiten ExternalIgnoreList refile:/etc/opendkim/trusted InternalHosts refile:/etc/opendkim/trusted # welche Verschlüsselungs-Keys sollen für welche # Domains verwendet werden # (refile: für Dateien mit regulären Ausdrücke) SigningTable refile:/etc/opendkim/signing.table KeyTable /etc/opendkim/key.table # diesen Signatur-Algorithmus verwenden SignatureAlgorithm rsa-sha256 # Always oversign From (sign using actual From and a null From to prevent # malicious signatures header fields (From and/or others) between the signer # and the verifier. From is oversigned by default in the Debian pacakge # because it is often the identity key used by reputation systems and thus # somewhat security sensitive. OversignHeaders From |
| /etc/opendkim/trusted |
|---|
127.0.0.1 ::1 localhost hostname domain.com |
| /etc/opendkim/signing.table |
|---|
# for E-Mails from xxx@domain.com use the key 'domain' for signing
*@domain.com domain
|
| /etc/opendkim/key.table |
|---|
# The key 'domain' is located in /etc/opendkim/keys/domain.private
domain domain.com:201611:/etc/opendkim/keys/domain.private
|
cd /etc/opendkim opendkim-genkey -d domain.com -b 2048 -r -s 201611 mv 201611.private keys/domain.private mv 201611.txt keys/domain.txt chown -R postfix:postfix /etc/opendkim chmod -R go-rwx /etc/opendkim/keys
| /etc/postfix/main.cf |
|---|
milter_protocol = 6 milter_default_action = accept smtpd_milters = unix:/var/run/opendkim/opendkim.sock non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock |
Then add the record from /etc/opendkim/keys/domain.txt to your DNS server.
9) Deploy some antivirus and antispam scripts for the MTA
It is a good idea to have some virus filtering and antispam on a Mailserver obiously. Now there are a lot of filtering solutions out there which can accomplish this, but I prefer not to rely on another piece of big software, so I just use a simple shellscript to do some simple mail filtering according to my needs. I’m using Clamav as an antivirus-filter, spamassassin as a spam-filter and some custom .zip file parsing using munpack which certainly can be improved.
apt-get install clamdscan spamassassin clamav-daemon mpack zip mkdir /var/spool/spam chmod 777 /var/spool/spam mkdir /var/spool/virus chmod 777 /var/spool/virus
| /etc/spamassassin/local.cf |
|---|
rewrite_header Subject *****SPAM*****
report_contact your_email@address.com
|
| /etc/default/spamassassin |
|---|
ENABLED=1 |
/etc/init.d/spamassassin start mkdir /var/lib/dovecot/sieve chmod 777 /var/lib/dovecot/sieve/
Now we want to move spam into the Junk folder automatically on the server side:
| /var/lib/dovecot/sieve/global.sieve |
|---|
require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
fileinto "Junk";
}
|
| /etc/dovecot/conf.d/90-sieve.conf |
|---|
plugin {
sieve_before=/var/lib/dovecot/sieve/global.sieve
}
|
Now create the mail-scanner script that can be modified according to your needs:
| /etc/postfix/mail-scanner.sh |
|---|
#!/bin/sh
EX_OK=0
EX_BOUNCE=69
EX_DEFER=75
SENDMAIL="/usr/sbin/sendmail -G -i"
# prepare for scanning
INPUT=`mktemp /tmp/mail-scanner.XXXXXXXX`
OUTPUT=`mktemp /tmp/mail-scanner.XXXXXXXX`
if [ "$?" != 0 ]; then
logger -s -p mail.warning -t scanner "Unable to create temporary files, deferring"
exit $EX_DEFER
fi
trap "rm -f $INPUT $OUTPUT" EXIT TERM
cat >$INPUT
# check for viruses
/usr/bin/clamdscan --quiet - <$INPUT
return="$?"
if [ "$return" = 1 ]; then
TARGET=/var/spool/virus/`basename $INPUT`
logger -p mail.info "ClamAV found virus, quarantined as $TARGET"
cp $INPUT $TARGET || { echo Cannot copy possible-virus mail; exit $EX_DEFER; };
exit $EX_OK
elif [ "$return" != 0 ]; then
logger -s -p mail.warning -t scanner "Temporary ClamAV failure $return, deferring"
exit $EX_DEFER
fi
# check for undesired file extensions
MTMPDIR=`mktemp -d /tmp/mailattXXXXXXX`
BADEXT='[.](bat|cmd|com|cpl|exe|pif|msi|vb|vbs|js|jse|wsf)$'
munpack -C $MTMPDIR -q <$INPUT >/dev/null 2>&1
for i in $( ls $MTMPDIR | egrep -i '[.]zip' )
do
if [ $( unzip -l "${MTMPDIR}/${i}" | tail -n +4 |head -n -2 | egrep -i "${BADEXT}" | wc -l ) -gt 0 ]; then
HASBADEXT=1
break
fi
done
if [ $( ls $MTMPDIR | egrep -i "${BADEXT}" | wc -l ) -gt 0 ]; then
HASBADEXT=1
fi
rm -r $MTMPDIR
if [ "$HASBADEXT" = 1 ]; then
TARGET=/var/spool/virus/`basename $INPUT`
logger -p mail.info "Found potential virus with suspicious extension, quarantined as $TARGET"
cp $INPUT $TARGET || { echo Cannot copy possible-virus mail; exit $EX_DEFER; };
exit $EX_OK
fi
# check for spam
/usr/bin/spamc -E <$INPUT >$OUTPUT
return="$?"
if [ "$return" = 1 ]; then
# Activate this if you want to filter it out instead of just marking und filing into Junk
#TARGET=/var/spool/spam/`basename $INPUT`
logger -p mail.info "SpamAssassin found spam"
#cp $INPUT $TARGET || { echo Cannot copy possible-spam mail; exit $EX_DEFER; };
#exit $EX_OK
elif [ "$return" != 0 ]; then
logger -s -p mail.warning -t scanner "Temporary SpamAssassin failure $return, delivering"
# 1) deliver original mail
OUTPUT=$INPUT
# 2) or defer instead of delivering:
# exit $EX_DEFER
fi
# deliver
$SENDMAIL "$@" <$OUTPUT
exit $?
|
chmod 755 /etc/postfix/mail-scanner.sh
Now register mail-scanner with postfix by appending content_filter option to smtpd line and adding the filter definition:
| /etc/postfix/master.cf |
|---|
smtp inet n - n - - smtpd
-o content_filter=scanner:dummy
scanner unix - n n - 4 pipe
flags=Rq user=nobody null_sender=
argv=/etc/postfix/mail-scanner.sh -f ${sender} -- ${recipient}
|
10) Configure SOGo
| /etc/sogo/sogo.conf |
|---|
/* Database configuration (mysql:// or postgresql://) */ SOGoProfileURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_user_profile"; OCSFolderInfoURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_folder_info"; OCSSessionsFolderURL = "mysql://sogo:sogopasswd@localhost:3306/sogo/sogo_sessions_folder"; /* Mail */ SOGoIMAPServer = localhost; SOGoSieveServer = sieve://127.0.0.1:4190; SOGoSMTPServer = 127.0.0.1; SOGoMailDomain = domain.com; SOGoMailingMechanism = smtp; /* Notifications */ SOGoAppointmentSendEMailNotifications = YES; SOGoACLsSendEMailNotifications = NO; SOGoFoldersSendEMailNotifications = NO; /* Authentication */ SOGoPasswordChangeEnabled = YES; /* SQL authentication example */ /* These database columns MUST be present in the view/table: * c_uid - will be used for authentication - it's the username or username@domain.tld) * c_name - which can be identical to c_uid - will be used to uniquely identify entries * c_password - password of the user, plain-text, md5 or sha encoded for now * c_cn - the user's common name - such as "John Doe" * mail - the user's mail address * See the installation guide for more details */ SOGoUserSources = ( { type = sql; id = users; viewURL = "mysql://sogo:sogopasswd@127.0.0.1:3306/sogo/sogo_users"; canAuthenticate = YES; isAddressBook = YES; displayName = "Benutzer"; userPasswordAlgorithm = md5; } ); /* Web Interface */ SOGoVacationEnabled = YES; SOGoSieveScriptsEnabled = YES; SOGoMailAuxiliaryUserAccountsEnabled = YES; /* General */ SOGoLanguage = German; SOGoTimeZone = Europe/Vienna; SOGoSuperUsernames = (youradminacct); /* Activesync */ SOGoMaximumPingInterval = 300; SOGoMaximumSyncWindowSize = 100; SOGoMaximumSyncResponseSize = 5172; |
If you read the ACtiveSync tuning link, you know that you need to maybe increase the number of active instances for handling requests according to the number of your users by setting PREFORK= value in /etc/default/sogo
Now finally you need to add your admin user you also specified above in SOGoSuperUsernames with SOGo. For every new user, you can add the user like a normal shell user with
adduser [username]
and maybe disable login shell in /etc/passwd. Then use the following script to register your new user with the MySQL DB (reenter user’s password when prompted):
| cr_user.sh |
|---|
#!/bin/bash
if [ -z $1 ]; then
echo $0 \[Username\]
exit
fi
ID=$1
C_UID=`id -u $ID`
if [ $? -ne 0 ]; then
echo User $ID does not exist
exit
fi
C_GID=`id -g $ID`
C_CN=`getent passwd $ID | cut -d ':' -f 5 | cut -d"," -f1`
C_MAIL=$ID@`cat /etc/mailname`
C_HOME=`eval echo ~$ID`
read -sp "Password: " C_PASSWORD
echo "INSERT INTO sogo_users(c_uid,c_name,c_cn,c_password,mail,home,uid,gid) VALUES
('$ID','$ID','$C_CN',MD5('$C_PASSWORD'),'$C_MAIL','$C_HOME',$C_UID,$C_GID)
ON DUPLICATE KEY UPDATE c_uid='$ID',c_name='$ID',c_cn='$C_CN',c_password=MD5('$C_PASSWORD'),
mail='$C_MAIL',home='$C_HOME',uid=$C_UID,gid=$C_GID;" | mysql -u sogo --password=sogopasswd sogo
|
Finally you can add SOGo backup script to your daily backups (you need to adjust the file):
cp /usr/share/doc/sogo/sogo-backup.sh /usr/sbin/sogo-backup vi /usr/sbin/sogo-backup
Finally restart all the stuff
The case of the missing mouse cursor on Windows 10
Today I got a machine that was upgraded from Windows 8 to Windows 10.
The user complaine that his CDrom-Drive was not working after the
upgrade. This was eaisly solved by googling around a bit and finally
issuing:
reg.exe add "HKLM\System\CurrentControlSet\Services\atapi\Controller0" /f /v EnumDevice1 /t REG_DWORD /d 0x00000001
Ok, problem solved, but there was another strange problem.
I plugged in a PS/2 mouse device and it didn’t work.. So I plugged
in a USB mouse which worked. Then Windows 10 was doing some updates
like it always does and after reboot my USB mouse was gone too.
No chance to get PS/2 or USB mouse working.
PS/2 mouse driver always showed an exclamation mark telling me
Code 10 – “Das Gerät kann nicht gestartet werden” (the device cannot
be started).
When plugging in a USB mouse, no error was shown in the device manager
at all, but still no mousecursor appeared which made it hard to diagnose.
There just was no mouse pointer on the screen at all!
When looking at the Event log of the failed device, it told be
error 0x00000184 (which translates to STATUS_INVALID_DEVICE_STATE).
Now from https://technet.microsoft.com/en-us/library/ff542127?f=255&MSPPError=-2147217396
this oerror occurs if the Mouclass service is not connected.
So it had to me a mouclass issue.
When checking registry HKLM\System\CurrentControlSet\Services\Mouclass
I realized that there was no Enum subkey which normally contains the
attached devices indicating that the mouclass driver was not started.
When trying to start the device driver manually with “net start” command,
this succeeded so my theory about it not being started seemed to be correct. But why? So I checked the places where it is usually referenced
and found out that in
HKLM\CurrentControlSet\Control\Class\{4d36e96f-e325-11ce-bfc1-08002be10318}
(which is the Mouse device), the UpperFilters REG_MULTI_SZ value, which should contain the value
mouclass
was completely missing for some unknown reason! So I created this value,
rebooted and TADAAA, there was the mouse cursor again and mouse was working.
It would be really interesting what ruined this entry and what was causing me all these troubles, but finally it’s solved at least.
Recovering a failing SSD drive which drops off the bus on read error
This week I had to take a look at a machine with Windows 10 that randomly locked up during working with a certain application. The computer didn’t freeze, but the foreground application didn’t react. You could click around the desktop, but there also was no more reaction to user input by the operation system so that the onl choice was to press reset.
The system was equipped with a 120GB SSD drive as boot drive and after presseing RESET, the drive wasn’t detected anymore, so the BIOS showed “No boot device found”.
When powercycling the machine, the system was back to normal.
This rose suspicion that the SSD drive may be faulty, which got confirmed by doing a full drive scan with HDTune. It locked up during run and showed a bad sector of the drive. So unfortunately, it seems that the drive locked up when accessing a bad block on the media and never recovered from this fault and didn’t react to ATA Reset-commands.
So the drive had to be replaced, but as it was the system drive and a reinstall of the whole operating system would have been a tedious task (due to the lack of some applications’ installers, etc., blabla..), I tried recovering the disk to a new (bigger) SSD drive.
First try was using some Windows imaging applications like the free Macrium Reflect, but of course they failed due to the fact that the disk locked up during cloning when hitting a bad block.
So ddrescue to the rescue! This application also naturally aborts when the drive locks up, but it has a big advantage: It has the option to write a human-readable logfile to resume operations! So starting the first round was easy, just image until bad block (using -n, as scraping doesn’t work anyway if touching a bad block locks up the device. So scraping has to be done manually here):
ddrescue -n /dev/sdb sdb.dd sdb.log
Now first part of disk was written to image and on bad block, drive locked up, leaving behind a logfile that said: Good until block X, then everything bad. Now shutdown and restart machine to get SSD back working.
As ddrescue logfile format is human readable, it’s also easy to edit. The manual fortunately tells us about the logfile format:
The first non-comment line is the status line. It contains a non-negative integer and a status character. The integer is the position being tried in the input file. (The beginning of the block being tried in a forward pass or the end of the block in a backward pass).
The status character is one of these:
‘?’ copying non-tried blocks
…Every line in the list of data blocks describes a block of data. It contains 2 non-negative integers and a status character. The first integer is the starting position of the block in the input file, the second integer is the size (in bytes) of the block. The status character is one of these:
‘?’ non-tried block
‘*’ failed block non-trimmed
‘/’ failed block non-scraped
‘-‘ failed block bad-sector(s)
‘+’ finished block
…
On second run, we don’t have to do much but copy the logfile to a new file (i.e. sdb2.log) and delete all the lines in the block list that don’t contain a “+” and setting status to “?” and then run in reverse direction with this new file to image from the other end of the disk until we hit a block again that is faulty:
ddrescue -R -n /dev/sdb sdb.dd sdb2.log
Now the device falls off again and we have the sdb2.log which now contains good blocks from the beginning, then a list of failed blocks and good blocks until the end. Now there can be a huge gap in between and we have to rescue data from good blocks in there too. So shutdown, powercycle, reboot again, copy sdb2.log to sdb3.log, then modify the lines that contain ‘/’ and ‘-‘ and are beyond the first faulty block to ‘?’, modify the status header from ‘+’ to ‘?’ again and try again in forward direction like in the first sample.. If it locks up, retry the step again (after power off and on) but leaving the faulty block you tried with ‘-‘ or ‘/’ and instead just mark the next blocks until you hit one that doesn’t lockup the machine. This way you can manually do the scraping in order to finally rescue as much data as possible from the drive to the image file.
Afterwards it can be written to the new drive (dd if=sdb.dd of=/dev/sdc bs=4096), as the newer drive is bigger, partitions can be moved and resized with gparted, but I had to ensure to use a very recent version of ntfs-3g for ntfsresize to work with new Windows 10 NTFS format, with older versions this step fails in gparted and I had to do it manually with recent version. Finally, the old drive was rescued and system was properly moved to the new drive.
EPG-Änderung Österreich Mux A
Am Land, wo es oft keine Versorgung mit KabelTV gibt, bleibt DVB-T die einzige brauchbare Möglichkeit, um die terrestrisch ausgestrahlten lokalen TV-Programme empfangen zu können, wenn man sich nicht extra eine Satellitenschüssel aufs Dach stellen möchte samt Decoderbox und Entschüsselungskarte. Mit DVB-T kommt man mit einer kleinen Zimmerantenne aus und kann das TV-Programm bequem am PC ansehen.
Hierfür hat sich der TV Schduler Pro ganz gut bewährt, da er mit relativ einfachen Mitteln auskommt (Streamproducer und VLC). Damit man ein aktuelles TV-Programm in der Weboberfläche hat, muss man einmal täglich EPG scannen und das Backend damit speisen, dann reicht zum Aufnehmen einer Sendung ein einfaches Anklicken im Webinterface.Zum EPG Scannen eignet sich das kleine C++ Programm EPGScan vortrefflich. Seit etwas über einem Monat lieferte mir dieses jedoch z.B. bei ORF-Programmen keine EPG-Texte mehr. Nun bin ich endlich dazugekommen das Problem zu analysieren: Scheinbar wurde das EPG kürzlich umgestellt, um für die Beschreibungstexte die Zeichensatzcodierung mit der Kennziffer 0x0B zu verwenden. EPGScan kennt in seiner aktuellsten Version diese Zeichensatzcodierung jedoch leider nicht und somit bleibt das EPG für die Kanäle dann leer.
Ein User hat vor mehreren Jahren schon einen Patch gemacht, der u.A. auch dieses PRoblem löst, indem in der Funktion getStringFromBytes in CStreamScan.cpp pData[0] auf <0x0C abfragt und für alle Codierungen startByte=1 setzt, statt auf jede ihm bekannt separat abzufragen. Damit ist das Problem nach dem neu Kompilieren dann gelöst und EPGScan findet wieder wie gewohnt die Programminformationen. Der Patch befindert sich auf der Sourceforge-Projektseite hier. Viell. hat ja noch Jemand das gleiche Problem.
Windows 7 x64 and Office 2013 refusing to print
On a Windows 7 x64 system with Office 2013 and Adobe Acrobat 8 installed, I was unable to print in M$ office (worked fine in other Windows applications). Office always returned that there is no default printer installed and it was unable to print, regardless of the printer I chose from the list. Checking event log, I found out that on restart of the Windows printer spooler service and on first print, I got an error that the spooler was unable to load AdobePDF.dll in Event log. Therefore I had to remove the Adobe PDF printer driver from the subkey of:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Environments\Windows x64\Drivers\Version-3
And everything worked fine again (except Adobe PDF printer of course, but this printer wasn’t really necessary anway).
gpt_loader.sys revisited, file read problem
It’s been over a year since I last analyzed and fixed a bug in the Paragon GPT
loader driver which enables us Windows XP users to use GPT partitioned drives
beyond 2TB in size.
Last time, I fixed a severe bug that caused the driver to crash.
This time a user reported a strange bug with the driver in the comments section
which I also experienced once but first ignored it:
The Problem
When reading files that are located beyond the 2TB area, massive memory usage
occurs and as a result, the computer slows down to a crawl. This is especially
a problem if you are copying large files from your 2TB harddisk.
More informations about the problem can be found in the comments for the last
fix where a user reported them.
Time to take a further look at the problem:
When copying file from T: to S: where T: is the GPT-drive that gpt_loader.sys
handles, during copy the physical memory usage increases a lot and the
Lazywriter thread starts flushing data to the SOURCE file on T: as can be seen
in Filemon:
Looking at the callstack of the first write to the source file, the write
operation roots at nt!MiMappedPageWriter.
This is the lazy writer thread that periodically sweeps through the dirty
pages and flushes them to disk.
So first conclusion is that there must be some memory mapped page containing
the source file whose pages got dirty for some reason. As they are dirty, the
system needs to cache their content which in turn seems to create the huge
memory usage ignoring the disk cache limits.
Dirty pages also need to be flushed back to disk which causes additional load
for no reason and theoretically may be even dangerous as a file that is
only being read may get corrupted on power loss. As read files normally
don’t get corrupted it is assumed that pages are dirty even though they
haven’t changed their content.
The thread is processing the MmMappedPageWriterList.
Now when trying with another file not in cache and checking some statistics,
it can be seen that there are many dirty pages for the SOURCE file being copied
(which are being flushed back to disk):
lkd> !memusage ... Control Valid Standby Dirty Shared Locked PageTables name 8a0c75d8 748 77632 577680 0 0 0 mapped_file( (2013-12-28 20-10) Polt- ORF2 N.ts ) 8a074c80 208 654244 1636 0 8 0 mapped_file( (2013-12-28 20-10) Polt- ORF2 N.ts ) ...
See amount of dirty pages!
lkd> !ca 8a0c75d8
ControlArea @ 8a0c75d8
Segment e55cf488 Flink 00000000 Blink 00000000
Section Ref 1 Pfn Ref 2cc80 Mapped Views 4
User Ref 0 WaitForDel 0 Flush Count 0
File Object 8a182408 ModWriteCount 0 System Views 4
Flags (8080) File WasPurged
File: \(2013-12-28 20-10) Polt- ORF2 N.ts
Segment @ e55cf488
Type nt!_MAPPED_FILE_SEGMENT not found.
lkd> !ca 8a074c80
ControlArea @ 8a074c80
Segment e386d998 Flink 00000000 Blink 00000000
Section Ref 1 Pfn Ref 2cc50 Mapped Views 2
User Ref 0 WaitForDel 0 Flush Count 0
File Object 8a1d4628 ModWriteCount 0 System Views 2
Flags (8080) File WasPurged
File: \(2013-12-28 20-10) Polt- ORF2 N.ts
Segment @ e386d998
Type nt!_MAPPED_FILE_SEGMENT not found.
lkd> !fileobj 8a182408
\(2013-12-28 20-10) Polt- ORF2 N.ts
Device Object: 0x8b560be8 \Driver\gpt_loader
Vpb: 0x8b57a450
Event signalled
Access: Read SharedRead
Flags: 0xc0062
Synchronous IO
Sequential Only
Cache Supported
Handle Created
Fast IO Read
FsContext: 0xe5584850 FsContext2: 0xe55849a8
Private Cache Map: 0x89e93b50
CurrentByteOffset: 2cc50000
Cache Data:
Section Object Pointers: 8a1dba3c
Shared Cache Map: 89e93a78 File Offset: 2cc50000
Vacb: 8b5d87f8
Your data is at: d3ad0000
lkd> !fileobj 8a1d4628
\(2013-12-28 20-10) Polt- ORF2 N.ts
Device Object: 0x8b578e30 \Driver\Ftdisk
Vpb: 0x8b586af0
Event signalled
Access: Read Write SharedRead SharedWrite
Flags: 0x43062
Synchronous IO
Sequential Only
Cache Supported
Modified
Size Changed
Handle Created
FsContext: 0xe1584990 FsContext2: 0xe1584ae8
Private Cache Map: 0x89dd72d0
CurrentByteOffset: 2cc50000
Cache Data:
Section Object Pointers: 896adb14
Shared Cache Map: 89dd71f8 File Offset: 2cc50000
Vacb: 8b5dba68
Your data is at: c2ad0000
lkd> !object 8a182408
Object: 8a182408 Type: (8b60ee70) File
ObjectHeader: 8a1823f0 (old version)
HandleCount: 1 PointerCount: 3
Directory Object: 00000000 Name: \(2013-12-28 20-10) Polt- ORF2 N.ts {HarddiskGptVolume1}
So this is the file being READ from the GPT disk as suspected and it has dirty
pages for some unknown reason.
It is possible that the view originates from the cache manager.
Cache manager normally has some sort of write throttling so that available
cache memory cannot be exceeded, but as this occurs on a READ file, the
throttling doesn’t have any effect here leading to excessive memory usage.
So it is time to have a look at what gpt_loader is actually doing in its
processing routine for read/write. Translated to Pseudo C-Code, it’s
basically the following (largely shortened to the relevant calls):
ATA_PASS_THROUGH_DIRECT InputBuffer;
IO_STATUS_BLOCK IoStatusBlock;
NTSTATUS Status;
KEVENT Event;
PIRP AtaIRP;
union {
USHORT AtaFlags;
BOOL bRead;
} flg;
DWORD IoStatusInformation; // Returned later in Irp->IoStatus.Information as number of bytes transferred
DWORD nSectors = IoGetCurrentStackLocation(Irp)->Parameters.Read.ByteOffset.QuadPart / this->dw124;
flg.bRead = IoGetCurrentStackLocation(Irp)->MajorFunction == IRP_MJ_READ;
InputBuffer.DataBuffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, HighPagePriority);
flg.AtaFlags = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | (flg.bDoRead?ATA_FLAGS_DATA_IN:ATA_FLAGS_DATA_OUT);
for (IoStatusInformation = 0; nSectors > 0; IoStatusInformation+=InputBuffer.DataTransferLength)
{
nSectorsRead = nSectors>(31 * (4096 / this->nBytesPerSector))?(31 * (4096 / this->nBytesPerSector)):nSectors;
InputBuffer.DataTransferLength = nSectorsRead * this->nBytesPerSector;
InputBuffer.AtaFlags = AtaFlags;
// Omitted here: Fill InputBuffer with ATA-read command and data to read/write ...
KeInitializeEvent(&Event, 0, 0);
AtaIRP = IoBuildDeviceIoControlRequest(
IOCTL_ATA_PASS_THROUGH_DIRECT,
this->DeviceObject,
&InputBuffer,
sizeof(InputBuffer),
&InputBuffer,
sizeof(InputBuffer),
0,
&Event,
&IoStatusBlock);
if ((Status = IoCallDriver(this->DeviceObject, AtaIRP)) == STATUS_PENDING) {
KeWaitForSingleObject(&Event, 0, 0, 0, 0);
Status = IoStatusBlock.Status;
}
if (!NT_SUCCESS(Status)) break;
nSectors -= nSectorsRead;
InputBuffer.DataBuffer += nSectorsRead * this->nBytesPerSector;
}
When reading the documentation and what we can see here is that
IOCTL_ATA_PASS_THROUGH_DIRECT call requires not a MDL but a virtual address
where to read data to. So the driver does the obvious: It gets virtual
address from MDL via MmGetSystemAddressForMdlSafe and passes the pointer
to it to the lower level ATA driver so that the buffer gets read and filled.
Seems fine, right? And obviously works.
But from what I can see the following happens down the chain which causes the
unpleasant phenomenon mentioned above:
The lower level driver atapi.sys needs an MDL to read to, so
in IdeAtaPassThroughSetupIrp it does IoAllocateMdl for write access
with the virtual address passed in, assigns it to Irp->MdlAddress,
locks it with MmProbeAndLockPages and passes the call
through to the next driver. When the passthrough is done, it calls
its function IdeAtaPassThroughFreeIrp which does MmUnlockPages(Irp->MdlAddress).
On unlock, the page table entries of the write pages are marked as Modified
causing the unpleasant behaviour mentioned above.
Fixing it
So in order to circumvent this problem, the gpt_loader.sys driver instead
would need to allocate a buffer with size 0x1F000 bytes (maximum size supported
is 4096 * 31 for a block and it’s better to allocate the buffer once and reuse
it on every call than allocating and freeing it on every call, which looks a bit
expensive), let the lower level ATAPI driver read to that buffer
and then memcpy the read data from this buffer to the input buffer
to circumvent marking the pages dirty.
Now can this be fixed with patching? It seems to be quite hard as we must
actually add instructions to the driver without increasing its size or
overwriting vital functions.
First problem is the buffer space. This turns out to be easy. In generateLoader,
memory for the handling class is allocated with:
HandlerClass = malloc_pool(0x154u, NonPagedPool); .00010877: 57 push edi .00010878: 6854010000 push 000000154 .0001087D: E8B4650000 call .000016E36
So we just add 0x1F000 to the size of the class structure and address
HandlerClass+0x154 as the buffer. This also ensures that it gets freed properly
on exit without the need to add free-function:
.00010878: 6854200000 push 00001F154
The harder part is fixing the processIrp routine. Looking at the pseudo-code
above, we basically need to change the routine to the following:
ATA_PASS_THROUGH_DIRECT InputBuffer;
IO_STATUS_BLOCK IoStatusBlock;
NTSTATUS Status;
KEVENT Event;
PIRP AtaIRP;
union {
USHORT AtaFlags;
BOOL bRead;
} flg;
DWORD IoStatusInformation; // Returned later in Irp->IoStatus.Information as number of bytes transferred
DWORD nSectors = IoGetCurrentStackLocation(Irp)->Parameters.Read.ByteOffset.QuadPart / this->dw124;
PBYTE Buffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, HighPagePriority);
flg.bRead = IoGetCurrentStackLocation(Irp)->MajorFunction == IRP_MJ_READ;
InputBuffer.DataBuffer = flg.bRead?this->offs154:Buffer;
flg.AtaFlags = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | (flg.bDoRead?ATA_FLAGS_DATA_IN:ATA_FLAGS_DATA_OUT);
for (IoStatusInformation = 0; nSectors > 0; IoStatusInformation+=InputBuffer.DataTransferLength)
{
nSectorsRead = nSectors>(31 * (4096 / this->nBytesPerSector))?(31 * (4096 / this->nBytesPerSector)):nSectors;
InputBuffer.DataTransferLength = nSectorsRead * this->nBytesPerSector;
InputBuffer.AtaFlags = AtaFlags;
// Omitted here: Fill InputBuffer with ATA-read command and data to read/write ...
KeInitializeEvent(&Event, 0, 0);
AtaIRP = IoBuildDeviceIoControlRequest(
IOCTL_ATA_PASS_THROUGH_DIRECT,
this->DeviceObject,
&InputBuffer,
sizeof(InputBuffer),
&InputBuffer,
sizeof(InputBuffer),
0,
&Event,
&IoStatusBlock);
if ((Status = IoCallDriver(this->DeviceObject, AtaIRP)) == STATUS_PENDING) {
KeWaitForSingleObject(&Event, 0, 0, 0, 0);
Status = IoStatusBlock.Status;
}
if (!NT_SUCCESS(Status)) break;
nSectors -= nSectorsRead;
if (flg.AtaFlags & ATA_FLAGS_DATA_IN) {
RtlCopyMemory(Buffer, InputBuffer.DataBuffer, InputBuffer.DataTransferLength);
Buffer += nSectorsRead * this->nBytesPerSector;
} else InputBuffer.DataBuffer += nSectorsRead * this->nBytesPerSector;
}
First, we need more space on the stack for our pointer:
.00015DEA: 8BFF mov edi,edi .00015DEC: 55 push ebp .00015DED: 8BEC mov ebp,esp .00015DEF: 81EC8C000000 sub esp,00000008C .00015DF5: A1008C0100 mov eax,[00018C00]
So, change it to sub esp, 90h, so that [ebp-90h] is our new pointer:
.00015DEF: 81EC90000000 sub esp,000000090
As there is new code to add, we need to create a new section for the code,
because there is not enough space to stuff all that into the original function.
We can cut off 0x200 bytes of the end of the .reloc section and create a new
code section for our code there.
But due to the alignment of .reloc, we also have to change the
section table to remove the discardable flag of .reloc, otherwise our code
will vanish when .reloc gets discarded. This unfortunately adds 1,75KB of
increased memory usage to our driver, but that shouldn’t hurt you too much
I guess 😉
Next we have to ensure that our new buffer pointer gets initialized properly
with the target and the InputBuffer.DataBuffer gets setup correctly to our
new buffer. Here is the original code where buffer gets initialized:
.00015E92: 8945CC mov [ebp][-34],eax ; InputBuffer.DataBuffer .00015E95: 3BC3 cmp eax,ebx .00015E97: 7517 jne .000015EB0 .00015E99: BE170000C0 mov esi,0C0000017
We are moving this to a seperate routine in order to be able to place a
call in here:
.00015E92: E8694F0000 call .00001AE00
.00015E97: 7517 jne .000015EB0
Now there is one very important thing to consider: The routine we are patching is
a read/write routine, so we only need to do all that buffer copy magic on read,
not on write or we will be toast!
[ebp][-49] contains a flag that is set when reading and not set when writing.
We can use that.
In our new routine at 00001AE00:
0001AE00: 385DB7 cmp [ebp][-49],bl ; Check if we want to read or write 0001AE03: 8BD8 mov ebx,eax ; On write, set eax buffer directly like it used to be 0001AE05: 740C je .00001AE13 ; Jump on write, on read instead: 0001AE07: 898570FFFFFF mov [ebp][-00000090],eax ; Fill our stack variable with ptr to dest buffer 0001AE0D: 8D9E54010000 lea ebx,[esi][00000154] ; Pointer to buffer in Class that we allocated on read 0001AE13: 895DCC mov [ebp][-34],ebx ; Set InputBuffer.DataBuffer to class-buffer on read, to eax (direct MDL buffer) on write 0001AE16: 33DB xor ebx,ebx ; Restore abused ebx to 0 0001AE18: 3BC3 cmp eax,ebx ; Do comparison we had to eliminate for CALL 0001AE1A: C3 retn ; ...and back
Next comes the part IoStatusInformation+=InputBuffer.DataTransferLength at the
end of the loop that needs to be adapted so that content of temporary buffer
can be copied to input IObuffer from MDL:
.00016001: 11559C adc [ebp][-64],edx .00016004: 0FAF45A8 imul eax,[ebp][-58] .00016008: 0145CC add [ebp][-34],eax ; InputBuffer.DataBuffer+=eax .0001600B: 8B45C0 mov eax,[ebp][-40] ; eax=InputBuffer.DataTransferLength .0001600E: 014594 add [ebp][-6C],eax ; IoStatusInformation+=eax .00016011: 395DA0 cmp [ebp][-60],ebx ; nSectors==0? .00016014: 0F87C6FEFFFF ja .000015EE0 .0001601A: EB32 jmps .00001604E
ecx isn’t used for anything in this routine starting from this point, so
we can reuse it as counter for memcpy without saving.
eax also isn’t used anywhere else so we can just fill ecx instead
of eax here and use eax for incrementing src ptr later
(as DataTranferLength theoretically can be < nSectorsRead*BytesPerSector
on incomplete reads, although that shouldn’t happen).
But we also need to skip InputBuffer.DataBuffer+=eax on read operation,
as we are always reading to the same temp buffer and only incrementing
our dest pointer we copy the memory to. Therefore we move up
eax=InputBuffer.DataTransferLength to overwrite the add and then call
our new routine at AE20:
.00016008: 8B4DC0 mov ecx,[ebp][-40] ; ecx=InputBuffer.DataTransferLength .0001600B: 014D94 add [ebp][-6C],ecx ; IoStatusInformation+=ecx .0001600E: E80D4E0000 call .00001AE20 ; Call our new routine .00016013: 90 nop ; Padding
In our new routine, we can finally do the copy.
As copying up to 0x1900 bytes with repe movsb doesn’t look particularily fast,
better use memcpy which is located at 6CF2 in our driver version.
Unfortunately the bRead-Flag is a union with the USHORT AtaFlags which gets set at
5ED3, so we have to test for ATA-Flags now
Read = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | ATA_FLAGS_DATA_IN = 0x1A,
Write = ATA_FLAGS_48BIT_COMMAND | ATA_FLAGS_USE_DMA | ATA_FLAGS_DATA_OUT = 0x1C
.0001AE20: 8A5DB6 mov bl,[ebp][-4A] ; Fetch ATA Flags .0001AE23: 80FB1C cmp bl,01C ; Read or write? .0001AE26: 7505 jne .00001AE2D ; Jump on read, on write do: .0001AE28: 0145CC add [ebp][-34],eax ; InputBuffer.DataBuffer+=eax .0001AE2B: EB17 jmps .00001AE47 ; Do other missing stuff and back, on read do: .0001AE2D: 50 push eax ; Save eax to increment Dest ptr .0001AE2E: 51 push ecx ; Length = ecx .0001AE2F: FF75CC push d,[ebp][-34] ; Source = InputBuffer.DataBuffer .0001AE32: FFB570FFFFFF push d,[ebp][-00000090] ; Destination = Dest ptr to current loc in MDL .0001AE38: E8B5BEFFFF call memcpy ; ntoskrnl.exe .0001AE3D: 83C40C add esp,00C ; Fix the stack .0001AE40: 58 pop eax ; Restore eax .0001AE41: 018570FFFFFF add [ebp][-00000090],eax ; Update Dest ptr to current loc in MDL .0001AE47: 33DB xor ebx,ebx ; Restore ebx .0001AE49: 395DA0 cmp [ebp][-60],ebx ; nSectors == 0? (we deleted this with our call) .0001AE4C: C3 retn ; ...and back
Patch
Of course use that on your own risk, I do not guarantee for anything, but for me
this fixes the bug and the driver now works flawlessly 🙂
I wrote a little patcher that
patches the driver accordingly. Just run it and
if it patched successfully, reboot the system to load the fixed version of the
driver.
Feel free to try it and if you are also suffering from this problem, you can
leave a comment if this actually fixes it for you too.
If you haven’t done it already, also apply the first patch which fixes a
BSOD problem, every patch is for a certain problem only, this patch
therefore doesn’t contain the fixes from the BSOD patch.
For those who use crappy Antivirus programs like Antivir, don’t get fooled by the generic Antivirus signature-match TR/Downloader.gen (which is really stupid, as the exeutable doesn’t even call any Internet functions, so how should that download anything??), you can check with Virustotal.
If you have such an Antivirus program, use this build instead which is a larger executable but isn’t subject to false positives.
Recovery of a Panasonic DVR Recorder Harddisk
After my pervious adventures with a Pioneer DVR recorder, a user contacted me and asked me if I could also analyze the Filesystem of a Panasonic DVR recorder and sent me an image.
I saw that it uses a custom Filesystem which is called MEIHDFS V2.0. I searched for this term on the Internet and found this thread. It seems that a guy called hkmaly started researching the filesystem and found out the block size. As he obviously already found out some of the structures, I thought that I should contact him and fortunately, he immediately responded to my inquiry. He was kind enough to send me the sourcecode he wrote back in 2012 when he researched the filesystem and found out the description of an inode block. However he didn’t find the inode directory and therefore only managed to collect inode blocks on disk where it was unvertain if they were sctive or inactie as the directory and inode blocks get rewritten after every editing process resulting in multiple copies of the same inode number on the disk making it impossible to recover the current state. Using a Hex editor, I finally found the inode directory and gathered enough information to be able to restore the filesystem of the recorder in case the inode table is always at a fixed position starting from the superblock (which I’m not sure but just assuming).
The recorder basically stores the DVD-Recordings in DVD-RAM format where there unfortunately is no description available, because it’s a proprietary standard. However I found the dvd-vr program where the author managed to gather lot of information from the format using reverse engineering. Using this as a template, I found out the structure of the Recorder’s .VOB files which are slightly different than the ones on DVD-RAM (mostly they are shorter omitting some stuff that the original DVD-RAM format has incorporated). So I created a modified copy of this program that was able to parse the Pioneer DVD-RAM format used.
So I finally ended up with 2 utilities: extract for extracting the MEIHDFS Filesystem und dvd-vr for extracting movie data from the resulting .VOB files.
Technical information about the filesystem and various other informations can be found in the README file of the project.
You can download the source and binaries >> HERE <<
Hopefully this is helpful for people with failed harddisks to recover their movies. I’d love to hear from you if you managed to rescue your data with it. If you have questions or need adaptions of the program for your recorder, feel free to contact me.
Moving Acer eRecovery to a new Harddisk
I was asked if I could analyze a defective Acer Notebook that took hours to boot or to reinstall Windows and fix the problem.
As it turned out, the harddisk was defective as I was able to verify in an external enclosure. The disk had an approx. transfer-Rate from 1-2MB/s, so veeeeery slow. When the problems started (with a BSOD), the owner tried to reinstall windows after backing up his data, so there wasn’t much left to save off the drive. So I ordered a new harddisk. But a I wanted to do a clean recovery and also preserve the original license and installation, I wanted to copy the hidden Acer eRecovery Partition (the application that starts up when you press ALT+F10 in BIOS) which is named PQSERVICE and can be found at the beginning of the drive. So the obvious step (as the drive was so slow) was to clone only that Partition to the new drive. That took approx. 8 hours but worked fine.
After cloning the drive, I put it back into the original PC to do the restoration with ALT+F10 an was greeted with the following error message from the Boot loader:
"A disk read error occurred. Press Ctrl+Alt+Del to restart"
Too bad, something must have gone awfully wrong here. So my first steps were to boot up a Windows 7 PE Image, unhide the partition (set ID from 27 to 7), doing bootcfg, fixmbr, fixboot, bootsect /nt60 to it etc. But nothing helped, I was always greeted with the same error message.
Being a bit frustrated after over 2 hours of messing around with it, I thought that I should probably compare the old and new drive with a disk editor on sector level and then I found the problem:
The PQSERVICE Partition started at sector 2048 on the original drive, whereas my Partition copying software put the Partition to sector 63 onto the new drive which is quite fine and logical, as there is no need to waste space on a new drive. HOWEVER the Partition boot record (PBR) still contained the Info from the old drive. To understand the issue, let’s have a look at the first 32 bytes of my NTFS boot record:
00000000:EB 52 90 4E 54 46 53 20 -20 20 20 00 02 08 00 00 .R.NTFS ........ 00000010:00 00 00 00 00 F8 00 00 -3F 00 FF 00 00 08 00 00 ........?.......
To get a description of these bytes, look i.e. at this site.
Interesting to me is offset 1C which contains the number of “hidden sectors”.
This is simply the number of sectors preceeding the boot record. Now my value is 0x800 which is 2048 as you can see. As the boot record on the new HDD resides in sector 63, this needs to be replaced with 3F 00 00 00.
So I did this et voilà: eRecovery booted again!
Reconstruction of irremote.psiloc.com (Psiloc Irremote)
Motivation
As written in a previous blog post, I recently had to swap a defective HDD
of a Pioneer DVR recorder. After some reading in the Internet, I found out
that a user needs to have a special “Service Remote” to swap a HDD.
I finally fixed this problem by using the IrDA interface of my Notebook, as
written in the README of my project, but when thinking about how to do this,
I also thought about the IrDA interface of my Nokia N95 Symbian-based phone.
I remembered that somebody a few years ago mentioned Psiloc IrRemote for
remote control of various devices. So I searched for it and eventually found
out that the company Psiloc went out of business.
Fortunately, I found a release of the software by Googling around a
bit so that I was at least able to find a working copy of the software.
After installing the program on my phone, I was presented with a default list
of devices. Of course the remote that I was interested in, wasn’t available.
Remote definition file format
But I read that the software basically uses LIRC format for the remote
definitions and using web.archive.org revealed that there was a wizard that
was able to create IRRemote definition files online given a LIRC remote file.
So a user basically just had to map LIRC keys to preset templates for the
appropriate device type in the wizard and was then able to download a
definition file for the remote to import on his phone. These files have the
extension .ir and by executing them in the filemanager on the phone, they
are imported into the IRRemotes Remote-Database on the target phone.
But the wizard on the manufacturer’s homepage isn’t available anymore, so bad
luck for me and for most users of the application who paid for this…
But first I wanted to check the format of the remote definition files on the
phone, so I used the SISContents application to take a further look into the
directory structure of the installation package. The /private Directory is
interesting (That is a protected directory for AppData normally not accessible
to the user on Symbian phones):
./private/2000C3EC: backup_registration.xml database.dat layouts remotes ./private/2000C3EC/layouts: layout_air.xml layout_dvd.xml layout_other.xml layout_tv.xml layout_amp.xml layout_hifi.xml layout_sat.xml layout_vcr.xml layout_cd.xml layout_hometheatre.xml layout_tuner.xml ./private/2000C3EC/remotes: 1017.xml 125.xml 1570.xml 1899.xml 2088.xml 2487.xml 465.xml 730.xml 1028.xml 1261.xml 1575.xml 18.xml 2094.xml 2488.xml 466.xml 739.xml ...
So first let’s have a look at the layouts-Files:
<?xml version="1.0" encoding="utf-8"?>
<layout xmlns="http://irremote.psiloc.com/schema/layout.xsd"
author="Mariusz Ostrowski" name="Minium"
graphicsfile="\resource\apps\irRemote9\layout%i.mif" type="Amp">
<screen width="352" height="416" background="16384"
backgroundmask="16385" backgroundstretch="true">
<text x="10" y="12" width="332" height="26" align="center"
valign="center" color="000000" name="Title"/>
<button name="DEVNAME" width="338" height="36" x="7" y="7"
inactive="16676" inactivemask="16677"></button>
<button name="DUMMY" width="129" height="65" x="111" y="50"
centerx="176" centery="145" inactive="16432" inactivemask="16433">
<key>Up</key>
</button>
<button name="DUMMY" width="129" height="65" x="111" y="175"
centerx="176" centery="145" inactive="16434" inactivemask="16435">
<key>Down</key>
</button>
<button name="AmpVolumeDown" width="65" height="129" x="81" y="80"
centerx="176" centery="145" released="16486" releasedmask="16487"
pressed="16488" pressed="16488" pressedmask="16489"
inactive="16436" inactivemask="16437">
<key>Left</key>
</button>
...
What can be seen is that there is a .xsd schema definition file referred.
The original Homepage is offline, so it’s not available anymore, but Wayback
machine has it.
The layout seems to be very simple to comprehend, basicaly there are buttons
which can then be assigned to a LIRC code.
So next let’s check a remote defintion file for reference.
Here is a simple example:
<?xml version="1.0" encoding="utf-8"?> <remote xmlns="http://irremote.psiloc.com/schema/remote.xsd" name="STR-DE205" bits="12" eps="30" aeps="100" phead="2530" shead="480" pone="1300" sone="500" pzero="700" szero="500" gap="25061" minrepeat="2" type="Amp"> <flags> <flag>SPACEENC</flag> </flags> <codes> <code name="AmpSourceVCR" data="441" /> <code name="AmpSourceTV" data="561" /> <code name="AmpSourceTape" data="c41" /> <code name="AmpSourceCD" data="a41" /> <code name="AmpSourceTuner" data="841" /> <code name="AmpSourcePhono" data="41" /> <code name="AmpSourceDat" data="896" /> <code name="AmpSourceDVD" data="96" /> <code name="AmpSourceAux" data="cd6" /> <code name="AmpPower" data="8c6" /> <code name="AmpVolumeMute" data="281" /> <code name="AmpVolumeUp" data="481" /> <code name="AmpVolumeDown" data="c81" /> </codes> </remote>
There also is a schema definition file which was also archived by the Wayback
machine. The format is indeed very simple, so when looking at various .xml
files, it’s easy the correlate the given fields to the ones from the LIRC file
format. The mapping is so obvious, that I won’t go into detail about that.
So my goal was to convert a LIRC file into a remote definition for IRRemote
by reconstructing the original Wizard from the irremote homepage.
For that goal I reimplemented the LIRC file parser in PHP, as it has to work
as web application.
But for a user to get a usable file that he can use with IRremote, it is
necessary to write a valid .ir definition file to be able to import it on the
phone. Therefore this format needs to be analyzed. So the first task is to get
some .ir files. Fortunately, I found a database of .ir files on the net that
I could analyze.
When looking at them, it seems that they are the .xml IR remote definition
files with a binary header.
Let’s have a look at a short and simple one:
Sony_RM~6189.ir: 00000000 99 12 68 13 │ 01 01 00 00 │ 00 01 00 00 │ 00 01 00 00 ..h............. 00000010 00 D9 16 00 │ 00 02 00 00 │ 00 54 56 04 │ 00 00 00 53 .Ù.......TV....S 00000020 6F 6E 79 0E │ 00 00 00 52 │ 4D 38 36 32 │ 5F 4B 56 32 ony....RM862_KV2 00000030 38 57 46 33 │ 45 49 1B 00 │ 00 2D 18 00 │ 00 D9 16 00 8WF3EI...-...Ù.. 00000040 00 08 00 00 │ 00 59 6F 5F │ 6D 69 73 6D │ 6F 01 00 00 .....Yo_mismo... 00000050 00 2D 18 00 │ 00 8C 01 00 │ 00 3C 72 65 │ 6D 6F 74 65 .-.......<remote 00000060 20 78 6D 6C │ 6E 73 3D 22 │ 68 74 74 70 │ 3A 2F 2F 69 xmlns="http://i 00000070 72 72 65 6D │ 6F 74 65 2E │ 70 73 69 6C │ 6F 63 2E 63 rremote.psiloc.c 00000080 6F 6D 2F 73 │ 63 68 65 6D │ 61 2F 72 65 │ 6D 6F 74 65 om/schema/remote 00000090 2E 78 73 64 │ 22 20 6E 61 │ 6D 65 3D 22 │ 53 6F 6E 79 .xsd" name="Sony 000000A0 5F 52 4D 2D │ 38 36 32 2E │ 32 22 20 62 │ 69 74 73 3D _RM-862.2" bits= 000000B0 22 31 35 22 │ 20 65 70 73 │ 3D 22 33 30 │ 22 20 61 65 "15" eps="30" ae 000000C0 70 73 3D 22 │ 31 30 30 22 │ 20 70 68 65 │ 61 64 3D 22 ps="100" phead=" 000000D0 32 34 30 30 │ 22 20 73 68 │ 65 61 64 3D │ 22 36 30 30 2400" shead="600 000000E0 22 20 70 6F │ 6E 65 3D 22 │ 31 32 30 30 │ 22 20 73 6F " pone="1200" so 000000F0 6E 65 3D 22 │ 36 30 30 22 │ 20 70 7A 65 │ 72 6F 3D 22 ne="600" pzero=" 00000100 36 30 30 22 │ 20 73 7A 65 │ 72 6F 3D 22 │ 36 30 30 22 600" szero="600" 00000110 20 67 61 70 │ 3D 22 35 30 │ 30 30 30 22 │ 20 66 72 65 gap="50000" fre 00000120 71 75 65 6E │ 63 79 3D 22 │ 34 30 32 34 │ 34 22 20 74 quency="40244" t 00000130 79 70 65 3D │ 22 54 56 22 │ 20 3E 0D 0A │ 09 3C 66 6C ype="TV" >...<fl 00000140 61 67 73 3E │ 0D 0A 09 09 │ 3C 66 6C 61 │ 67 3E 53 50 ags>....<flag>SP 00000150 41 43 45 45 │ 4E 43 3C 2F │ 66 6C 61 67 │ 3E 0D 0A 09 ACEENC</flag>... 00000160 09 3C 66 6C │ 61 67 3E 52 │ 45 56 45 52 │ 53 45 3C 2F .<flag>REVERSE</ 00000170 66 6C 61 67 │ 3E 0D 0A 09 │ 09 3C 66 6C │ 61 67 3E 43 flag>....<flag>C 00000180 4F 4E 53 54 │ 4C 45 4E 47 │ 54 48 3C 2F │ 66 6C 61 67 ONSTLENGTH</flag 00000190 3E 0D 0A 09 │ 3C 2F 66 6C │ 61 67 73 3E │ 0D 0A 09 3C >...</flags>...< 000001A0 63 6F 64 65 │ 73 3E 0D 0A │ 09 09 3C 63 │ 6F 64 65 20 codes>....<code 000001B0 6E 61 6D 65 │ 3D 22 54 56 │ 59 65 6C 6C │ 6F 77 22 20 name="TVYellow" 000001C0 64 61 74 61 │ 3D 22 35 32 │ 33 64 22 2F │ 3E 0D 0A 09 data="523d"/>... 000001D0 3C 2F 63 6F │ 64 65 73 3E │ 0D 0A 3C 2F │ 72 65 6D 6F </codes>..</remo 000001E0 74 65 3E 0D │ 0A 00 00 00 │ 00 00 00 00 │ 00 00 00 00 te>............. 000001F0 00 00 00 00 │ 00 00 00 00 │ 00 00 00 00 │ 00 00 00 00 ................
When analyzing more files, the common header can be noticed: 99 12 63 13 01
I tried to import a file with such a header and interestingly, it was refused
by the phone (unknown file type). I did some research on the Internet and
found out that the first byte actually has to be 89, then it works, at least
with the version that I have.
It seems that the numbers in this file are organized in little endian DWORDs.
The next 3 DWORDs are all 1. I suspected that these are counters that enable
the user to import more than one definition for a certain remote, but I
didn’t find any file samples that showed how to pack multiple Remote
definitions into one file.
By reverse engineering the binaries lateron (see below), it was proven that
I was right (See the table description below for the tables):
1) Number of entries for DEVICES table
2) Number of entries for FILES table
3) Number of DEFINITION files
The meaning of the next DWORDs can be found out by comparing various files
and analyzing the structure behind them. It also helps to compare some .ir
files that refer to a device already in the database (database.dat) to find
some similarities. I noticed immediately that strings are stored by a DWORD
representing the length of the string followed by the string itself.
The numbers in the file actually represent a database consisting of 2 tables
and control files:
1) A Device table that identifies the device
2) A File table that links information about the XML definition file
and the device to control.
Why are there 2 tables?
Because there can be various devices using the same IR codes, this perfectly
makes sense.
The bytes following are representing (parts of) these 2 tables (they repeat
themselves the number of times given above):
D..DWORD S..String PK.Primary key FK.Foreign Key DEVICES ------- D ID [PK] S Device Type [Air Conditioner/Amp/...] S Device Vendor S Device Model FILES ----- D ID [PK] D File ID [FK] D Device ID [FK] S Author D Rank DEFINITIONS ----------- D File ID [PK] S The xml file defining the IR definition.
When looking at these fields, it can be noticed that the File ID is actually
the same ID as the filenames in the ./2000C3EC/remotes directory of the
application, so this is how it is matched.
Now as previously mentioned, I also had a look at the database.dat to match
some values with .ir files in order to find out their meaning, therefore here
is the format of the database.dat, which is fairly simple as well. Dumping
the DB to plaintext with a simple C program helped to see all the values that
enabled me to make conclusions about their meaning:
D Database version? Always 1 D Number of device entries DEVICES ------- D ID [PK] S Device Type [Air Conditioner/Amp/...] S Device Vendor S Device Model D Number of file entries FILES ----- D ID [PK] D File ID D Device ID [FK] S Author S Filename of xml file D Rank
I was unsure about the Rank-field but found out about it by checking the
archived copies of the irremote-site in the Wayback-machine. There was a rank
made up by some stars and when checking the range of this field, it can be
noticed that it ranges from -100 to 100 which would match best with this
field. As the UI of the application is also displaying the rank, it can be
concluded that it must be this field.
Now that I knew how to read the file formats, I also dug up some database.dat
which was filled with more remotes than the default ones on the internet
together with the corresponding .xml files.
As I also knew the format of the 2 tables making up the required information,
I wrote some scripts to import them into a MySQL DB. I added some fields which
may be interesting for the site that is presenting the data and ended up with
the following structure. As I wasn’t able to reconstruct the whole database by
this, I set the auto increment values high enough so that future additions
hopefully don’t collide with old data, should I get some from you.
If you also have databases that help me to reconstruct the original database
on irremote.psiloc.com, feel free to contact me and send me your dumps!
You can do this by dumping the /private/2000C3EC folder of your phone, zipping
its contents and send them to me.
CREATE TABLE `DEVICES` ( `ID` int(11) NOT NULL auto_increment, `DEVICE` varchar(16) NOT NULL, `VENDOR` varchar(32) NOT NULL, `MODEL` varchar(64) NOT NULL, PRIMARY KEY (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=100000 ; CREATE TABLE `FILES` ( `ID` int(11) NOT NULL auto_increment, `FILEID` int(11) NOT NULL, `DEVID` int(11) NOT NULL, `AUTHOR` varchar(16) default NULL, `FILE` varchar(16) NOT NULL, `RANK` tinyint(4) default '0', `DOWNLOADS` int(11) default '0', `NAME` varchar(32) default NULL, PRIMARY KEY (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=120000;
The DOWNLOADS-field is a download counter which was present on the original
page. Of course, this information wasn’t available in any dumps, so it’s zero.
The name of the .ir files is also quite simple, it’s just
VENDOR_MODEL_FILEID.ir
I added the NAME-field to store the name of the original LIRC file into the
DB. I did this to help identifying possible duplicates. Because when looking
at the original DB, I found many duplicates which grew the database
unnecessarily. The name can be found in the .xml definition files
in tag <remote>. Of course it is valid to do different mappings from the same
LIRC input file, but when people want to upload a remote definition, they
may not know that it already exists (especially as the naming in the current
DB is really chaotic).
For reconstrution, I didn’t only use the database.dat dump, but also a
dump of .ir files that I found.
If you are interested in the database, just contact me.
Reconstructing the Webservice
Now that I reconstructed the database, the next task was to get them into
the IRremote application.
Importing every single .ir file would have been a tedious task and by the time
I tried this I wasn’t aware how to put multiple definitions into one .ir file,
because I didn’t find any example for it out there.
How did normal users of the application get the most current database?
When looking at the application, I found that it has an online update feature.
The user just needs to press “Refresh” on a screen and the list gets updated
via some webservice running on irremote.psiloc.com.
So the first task was to find out what functions it calls. To accomplish that,
I made use of the hosts-file from Symbian and redirected the host
irremote.psiloc.com to my machine.
This isn’t so easy, as the hosts-file is also located in the inaccessible
/private folder.
Informations about it can be foundhere.
To access that folder, you first need the CapsOnOff program. There are
various descriptions on how to install this on the Internet, just google
for it. i.e. this site.
Once the hosts file redirects the application to the server, the following
Requests can be seen:
http://irremote.psiloc.com/WebService.asmx/GetVendors
http://irremote.psiloc.com/WebService.asmx/GetModels
http://irremote.psiloc.com/WebService.asmx/GetRanks
http://irremote.psiloc.com/WebService.asmx/GetRemote
http://irremote.psiloc.com/WebService.asmx/SetRank
http://irremote.psiloc.com/WebService.asmx/AddVendor
Now by putting some logging code to these scripts, it’s not really hard to
get the input data for each call, but the severe problem that arises is, that
it is unknown which answer the application expects.
So at this point, it is only possible to get the desired information by
disassembling the Application and taking a look at its inner working by
reverse engineering. Unfortunately, this isn’t that easy.
For reversing Symbian binaries, first the headers need to be restored. This
is done by the petran.exe tool available from Nokia.
But unfortunately, when loading irRemote9.exe into the disassembler, it
just has some loader stub code which passes a pointer to some packed/crypted
chunk of code to another background application called “DRM Common Solutions“.
This was one of Psiloc’s products that they not only used for their own
applications, but also tried to sell as DRM solution for other Symbian apps.
So I first had to take a look at the core DLL that the IrRemote9.exe
calls into for loading: DRMCommonSignerCore.dll
This DLL itself is crypted/packed and has its own loading routine. So it took
me quite some time to figure out, how their “protection” works. I don’t want
to bother you with alle the details, but I wrote an unpacker for their files:
The header structure of the packed/crypted data is as follows:
#define MAGIC 0x6FCB73E7
typedef struct
{
unsigned long magic;
unsigned long cbCompressed;
unsigned long cbUncompressed;
unsigned long CRC;
char key1[128];
char key2[64];
} TYP_HDR;
So it starts with a magic and contains info about the compressed and
uncompressed length of the data (because the data is also compressed), as
well as a CRC checksum for verification and 2 keys for decryption.
1) It has some static table in it. This gets “decrypted” with RC4 decode
with an also hardcoded key.
However the RC4 decode is a modified version of RC4 which does a 3x
indirection of the decoder table (look at my sources for clearification).
2) It does a public RSA decrypt of key1 with the RC4-decrypted
modulus and the well known public exponent 65537 (0x10001) (no padding).
3) It is searching the decrypted key1 for the first NULL byte.
4) Starting at the offset after the NULL-Byte, it RSA decryptes key2 with
the data after the NULL-Byte as modulus and the well known public
exponent 65537 (0x10001) (no padding).
5) Is repeats 3) for key2
6) It does its special RC4 decode on the actual data with the data from
5) after the offset 0 as a key.
7) The resulting decrypted buff is zlib uncompressed into memory and
finally executed.
Now that stuff only ran through for the DRMCommonSignerCore.dll, but when
using it on IrRemote9.exe, it doesn’t work. It turns out that for protected
executables it executes, there are other rc4-Tables and keys for step 1,
but the unpacking process is fortunately the same.
The resulting buffer gets executed, however it doesn’t have a standard
EPOC header, but a special header on its own, which contains informations for
the loader. So even though one can load up the decrypted code parts into the
disassembler, there are no valid references, sections etc. making the
resulting code quite hard to analyze.
So the header also needs to be rebuilt to be a valid EPOC header. By looking
at the binary output, some offsets can be guessed and matched to the fields
in a normal EPOC header. So I tried to do this as it can be seen in my
unpacker and finally, I got something usable (still not all fields are clear
to me, but I don’t care, as the original goal has been accomplished).
Now using the disassembled output, the request and response for the various
calls revealed (the input arrays are always the POST-Parameters sent to the
Webservice call):
GetRanks: --------- Array ( [deviceId] => 1521 [hash] => 993317623 ) --> <ArrayOfRankInfo> <RankInfo> <RankId>[rank id]</RankId> <Rank>[rank value]</Rank> <RemoteId>[remote id]</RemoteId> <Author>[author]</Author> </RankInfo> </ArrayOfRankInfo> GetModels: ---------- Array ( [deviceTypeName] => Air Conditioner [vendor] => Mitsubishi [hash] => 2634342403 ) --> <ArrayOfDeviceInfo> <DeviceInfo> <DeviceId>[id]</DeviceId> <Model>[model name]</Model> </DeviceInfo> </ArrayOfDeviceInfo> GetVendors: ----------- Array ( [deviceTypeName] => Air Conditioner [hash] => 3112346557 ) -> <ArrayOfString> <string>[Vendor]</string> ... </ArrayOfString> GetRemote --------- Array ( [remoteId] => 1933 [hash] => 862864718 ) -> The XML file for the remote definition with a prepended EFBBBF BOM in the beginning. AddVendor --------- Array ( [deviceTypeName] => Air Conditioner [vendor] => Mitsubishi [model] => RM561Vx [remoteId] => 2077 [imei] => xxxxxxxxxxxxxxx [uid] => 536882564 [method] => 0 [hash] => 3916498465 ) SetRank ------- Array ( [rankId] => 1933 [imei] => xxxxxxxxxxxxxxx [uid] => 536882564 [rankValue] => Yes [method] => 0 [hash] => 402398422 )
The Setter-Functions let the user rank a remote, if the definition file is
actually working (remember the Rank-Field in the DB), whereas the AddVendor
call adds another device name to an already existing Remote definition file,
to signify that it also works with that vendor/remote.
One interesting field here is the hash-Field given in every request. It seems
to be some security-by-obscurity feature supposedly to prevent the webservice
from being spammed.
I was able to reverse the calculation of these values for the calls that
only used DWORDs and documented them in this source file, however I wasn’t
able to find out how an INT64 is represented on the symbian phone so that
the hashes for the Setter-functions containing the IMEI can also be calculated
correctly. Maybe you have an idea?
Given the database and the knowledge about the Webservice calls, I now managed
to reimplement the webservices based on the MySQL DB that I created.
Via the host-file redirection, I was able to fetch the remote definitions
conveniently via the application.
Using the webservice on your phone
As previously mentioned, you have to put a valid hosts-file for redirecting
the calls on your phone.
So if you have a phone with platform security on, you first need to get
access to your private-Folder via CapsOff. I mirrored the tools needed for
this here.
An instruction can be found here.
Then put this hosts-file into the appropriate directory (C:\private\10000882)
or modify your existing hosts file to contain the entries from the file.
More informatios about the host-file can be found here.
Then Psiloc IrRemote should be able to access the Webservices from this
mirror site.
Reconstructing the Webpage
Next step was to reconstruct the web interface so that also other users can
gain access to the remote definition files again and – more important – to
be able to create new remote definitions based on LIRC files on their own,
so that also I can finally use it the way I intended to.
I fetched what I could get via the Wayback machine and added my own PHP
code for DB access to it. The original was ASP.NET, but I don’t like that,
PHP is more portable and easier to implement.
As for the Wizard, I wasn’t able to find out how it originally worked, as I
only found the starting page, so I made up my own interface for mapping a
layout (based on the layout XML files) to LIRC keys from a LIRC file.
I also saw that there were options to import CCF files, but that would
involve converting CCF files to pronto codes which then can be easily
converted to a LIRC input file. I actually ported pronto2lirc.py to PHP, so
you can input raw pronto codes in the composing-Wizard, but converting CCF
files has to be done via CCFTools and they are currently only available
for Windows.
I tried to contact the author, but I didn’t get any response and there is no
sourcecode publically available. So my only bet would be to reverse CCFtools
to port it to Posix, but that is quite a lot of work and I don’t know if it’s
worth it. If you think, that this would generally be a good idea, I may do
this, but it seems to be too much work for me if the only benefit from it is
to get another wizard working on my irremote-Site.
Conclusion
So finally I put up a mirror of the old irremote site with my database in
background at http://irremote.0wnz.at
Feel free to use it and use the Wizard to add your own remote definition
files. If you want to get the whole database onto your phone, you have to
patch your host file as mentioned above. Maybe someone with Symbian expertise
can write an installation package to do this, I can only give you the
instruction above on how to do this manually.
If you have an IRRemote DB on your phone, please send it to me (including the
remote definition XML files, so basically a dump of your /private/2000C3EC
directory, accessiable also via CapsOnOff) so that I can reconstruct even more
of the original remotes database.
Also contact me, if you want to have my sources or are otherwise interested in
the project to revive this application.
So I hope that this revived page is useful for the remaining IrRemote users so
that I didn’t only do this for myself.
Pioneer DVR recorder harddisk recovery
I recently was asked if I could take a look at a Pioneer DVR-633H Harddisk-Recorder as it always booted up with the “Hdd Error” message. So this was quite obvious: A dying harddisk.
When I opened up the case, I found a Seagate 250GB IDE drive in it and after finally dismounting it and connecting it to a PC, I was happy to see that it was still accessible. There were bad sectors on it, for example the first 2 sectors were bad, but I was lucky: A lot of sectors of the harddisk were still readable!
So next I searched the internet to check how to recover data from it.
I found a few scripts that people wrote to recover their harddisks, for example this one by Mike Knoop and – even better – this one by Stefan Haliner that supports multiple recorders.
However, I wasn’t really satisfied with the results. Both scripts just dump consecutive MPEG-chunks to files and leave the task of manually merging them to the user.
The first one spit out over 1700 chunks, reassembling them would have been a tedious task. Most notably the splitting was often also wrong. The second one was a bit better, there weren’t so many chunks left, but it also required me to look through all the chunks and merge/split them to reconstruct the movies.
So these solutions didn’t really satisfy me, I didn’t want to have so much work just to recover data from them. Therefore I decided to analyze the disk image myself with a Hex-editor. I found out that there are some block directories on it that looked like pointers to the blocks a movie is made of. This encouraged me to look further and after some days of analysis, I finally found out how data is stored on these drives and was able to write a litte Windows-program that parses the drive structure and dumps the movies together with their names and their recording date.
I hope that this is also useful for other users suffering from the same problem. Of course it’s written entirely in C, no scripting languages needed. If you want me to port it to Linux, just write a comment or send me an e-mail. I explained the recovery procedure and the technical deatils about the on-disk format in the README file.
You can find everything here: >> Download <<.
I’d love to hear from you if this possibly also works for other recorders or even other brands.. Who knows, maybe it’s a standardised format, it mostly looks a bit like something from the DVD-Standard.
System programmer