YAUB

Finding non-matching records in Excel files using GNU tools.

ferdi — Mon, 11 Jul 2011 16:52:00 GMT

This can be useful, for instance to determine which rows you have deleted from an Excel file, given you still have a copy of the original.
Export the Excel file (and original) to csv.
Then use some standard GNU tools:

cat file.csv original.csv | sort | uniq -u

These, and other, GNU tools are also available to Microsoft Windows users, see for instance MINGW.

Hello World in DNA

ferdi — Sun, 12 Jun 2011 04:44:00 GMT

Hello World in DNA

Monday, June 6, 2011 Stan was born.

Importing .eml files into Microsoft Outlook

ferdi — Fri, 03 Dec 2010 08:39:00 GMT

Importing .eml files into Microsoft Outlook

Importing .eml files is trivial for Mozilla Thunderbird or Outlook Express. You just drag the files and drop them in the mail folder, et voila! Life is not that simple however when using the full version of Outlook. Dropping an .eml file in Outlook will have it compose a new mail message with the .eml file attached.

The trick: Outlook Express offers a way to export messages to Outlook!

Even if you are using Outlook, Outlook Express will almost always be installed on your system as well, although you may have trouble locating it (Tip: open an explorer window and type msimn.exe). To import the files, use the following steps:

Start Outlook Express (msimn.exe)
Drag & Drop the files into Outlook Express.
In Outlook Express go to: File | Export | Messages and export the messages to Outlook.

That's all.

Running Xen on Ubuntu Hardy

ferdi — Tue, 05 May 2009 16:55:00 GMT

Don't do it!

This post was intented as praise for Xen and Ubuntu. I even created a screenshot to go with it. However, after trying to run Xen on Hardy, I am a little disappointed in Ubuntu.

The screenshot shows virtual machines (VMs) running Windows Server 2003, FreeBSD and Linux on a single cheap consumer-grade PC. Unfortunately, shortly after the screenshot was taken the machine crashed.

We have been running a Xen server for about 2 and a half years now. Initially we compiled the Xen kernel ourselves on a machine running Dapper Drake LTS and we were very happy about it. Shortly after Feisty came out we decided to try and take advantage of the packaging system provided by Ubuntu. This required a small Xen downgrade, but we were still quite happy, and the machine ran without problems for about 2 years. Unfortunately Feisty doesn't come with the 5 year support, so we needed to upgrade. We didn't have the time to upgrade to Hardy right away. Lucky us! Now we could wait until a nasty networking bug got fixed.

Using a Fresh Hardy install and some workarounds

rename Xen bridge (fair enough)
rename /lib/tls to /lib/tls.disabled (bug?)
use losetup -r /dev/loop0 /media/isos/jeos-8.04.2-jeos-i386.iso and use phy:/dev/loop0 in config (bug?)
fix grub on each linux VM
ditch the Windows VM

we finally got our virtual machines running again and I was able to take the screenshot. But shortly thereafter the machine crashed. And it kept crashing ever since. We have tried to determine the cause of the crashes for some time. We tried running only a single virtual machine, but it still crashed. Even running without virtual machines it still crashed. Every single time a single process (usually, but not always qemu-dm) grabs 100% CPU. But we couldn't get a hold on what exactly was going wrong.

In the end we switched to KVM. This turned out to be pretty easy. Since we were using hvm guests stored on LVM partitions we could use the old VMs (no modification needed). And the VMs lived happily ever after...

Deploying a Java J2EE project on Geronimo using Netbeans

ferdi — Tue, 09 Dec 2008 12:26:00 GMT

Deploying a Java J2EE project on Geronimo using Netbeans

We have been playing around with J2EE and Glassfish for a while, but wanted to try out the Apache Geronimo application server. But how to do that without also switching IDE?

I found a few projects hosted on sourceforge, apache and google code.

Plugin hosted on googlecode

This seems to be an empty project, with only the initial directory structure being put in place (on Feb 12, 2007).

Plugin hosted on Sourceforge

Last checkin for the sourceforge project was almost a year ago (2007 Dec 20). Build fails in Netbeans 6.5: The module net.sourceforge.nbgeronimo is not a friend of C:\Program Files\NetBeans 6.5\enterprise5\modules\org-netbeans-modules-j2ee-api-ejbmodule.jar

For a workaround see DevFaqImplementationDependency.

Unfortunately after hitting Install/Reload in IDE, it doesn't seem to show up in the Add Server window.

Plugin hosted on apache.org

The project on apache.org seems more active (last check in 2008 Aug 14). It also just builds. Moreover after hitting Install/Reload in IDE, a new option Apache Geronimo is listed in the Add Server window. Thus I now have an option to deploy to Apache Geronimo. Unfortunately, it seems I cannot deploy to a remote server, but a local server will do fine for now.

Conclusion

Use this netbeans plugin for Apache Geronimo.

Update: 22 December 2008

Remote (re)deployment is quite easy. Just log in to to web console of the remote geronimo server and choose Deploy New. Select your ear or war file (in the projects dist directory) and click Install.

Software Transactional Memory

ferdi — Wed, 13 Aug 2008 11:34:00 GMT

Beautiful concurrency

I had another interesting read yesterday:

Beautiful concurrency by Simon Peyton Jones of Microsoft Research

Modern computers have more than one CPU. In order to make good use of multiple CPUs, parallel programs must be written. A problem arises when a resource must not be used by multiple threads at the same time. This is conventionally solved by using locking (e.g. using synchronized in Java, or lock in C#).

The paper describes Software Transactional Memory, a new approach to programming shared-memory parallel processors in a more modular way.

Relax NG

ferdi — Wed, 13 Aug 2008 11:18:00 GMT

Relax NG (relaxing)

Why?

The following quote from the paper XML Fever by Erik Wilde and Robert J .Glushko gives a good explanation:

XSDL's complexity allows a given logical model to be encoded in a plethora of ways (this fever will mutate into an even more serious threat with the upcoming XSDL 1.1, which adds new features that overlap with existing features). A cure for schema option paralysis is to use alternative schema languages with a better separation of concerns (such as limiting itself to grammars and leaving data types and path-based constraints to other languages), most notably RELAX NG.

How?

When writing a schema I prefer the compact syntax, emacs rnc-mode being a helpful aid.

Another very useful feature is the instant validation provided by emacs nxml-mode when opening an XML file and while editing it. All you need to do is feed it the rnc schema file.

Other validation tools I use:
For Java: Jing or MSV
For .Net : Tenuto

Beware of a bug in Jing when using javax.xml.validation as in the code example below:

import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Schema;
import javax.xml.XMLConstants;
import javax.xml.transform.stream.StreamSource;
import java.io.File;

public class RELAXNGValidation {
    public static void main(String[] args) throws Exception {

        SchemaFactory sf = SchemaFactory.newInstance(XMLConstants.RELAXNG_NS_URI);
        Schema schema = sf.newSchema(new File(args[0]));
        schema.newValidator().validate(new StreamSource(args[1]));
    }
}

A useful XML schema conversion tool is Trang.

Update: 29 Oktober 2008

James Clark is working on Jing again. Jing and Trang are now hosted on Google Code

Kan geen bestanden uitvoeren vanaf een windows share.

ferdi — Wed, 11 Jun 2008 13:02:00 GMT

Cannot run files from share

Je bent Domein Administrator en je kunt geen bestanden uitvoeren vanaf een windows share?

Mogelijk ligt het aan je instellingen van Internet Explorer.

Wat zeg je Internet Explorer?
Dat is toch om web pagina's als deze te bekijken?
Inderdaad Internet Explorer kan ook bij dit soort problemen de boosdoener zijn. Je moet er maar op komen!

Het volgende loste het probleem in ieder geval op voor mijn collega:

Kies onder Extra het submenu Internet Options
Ga naar het tabblad Beveiliging en klik op Lokaal intranet
Klik op Websites en vink de optie Intranetwerk automatisch detecteren aan.

MapReduce

ferdi — Tue, 06 May 2008 12:02:00 GMT

I recently found the time to read the paper describing Google's MapReduce.

Google collects massive amounts of data. In order to process all this data in a reasonable amount of time, they use clusters of computers working in parallel. MapReduce is designed to take away the complexities of parallel computing and cluster management from the programmers. It is intimately connected with Google's own File System (GFS), making use of the same fragments of files.

The pragmatic parts are what struck me most.

Run jobs on multiple machines and get the result from the machine that finishes first.
When a job fails, if possible detect which record caused the crash, and try again on a different machine.
And especially. Skipping Bad Records. If a job fails on multiple machines just skip the records that cause crashes (Get out results first and worry about problems later!)

n.b. See Hadoop, for an open source implementation of MapReduce.

13949712720901ForOSX

ferdi — Tue, 22 Apr 2008 13:39:00 GMT

A bit late perhaps, but I did not get to work on a Mac before. Only two days and I get bitten by the fact that there is no Java 6 for Leopard. And since Leopard is running on Intel here, the developer preview is of no help either. So I have to support the petition!