Re-Enable GNOME Menu Icons

This is just a quick post to remind me and others how to make the icons in GNOME menus visible again.  There used to be an option in the Appearances dialog to turn these back on but I guess someone decided for everyone that this option should be hidden.  It’s dead-easy to turn this setting on using gconftool-2 or gconf-editor GUI.

From the command-line:

gconftool-2 --type boolean --set /desktop/gnome/interface/menus_have_icons true

Or from the gconf-editor GUI:
Go to desktop->gnome->interface and check the option menus_have_icons.

Here’s a screenshot:

UPDATE: I’ve added this as an option to my Ubuntu fixup script.

Embedding Devhelp Into Your Own Program

It’s been a while since I’ve posted here because I’ve been really busy coding and learning.  I figured I’d make a couple posts about what I’ve been learning, which might be useful to others since I found the documentation to be scarce.  I’ve been using several useful GNOME libraries lately, so I’ll just post some really basic example code/programs and describe them in the post.  I’ll assume you know how to program in C/GObject.  I don’t claim to be an expert on any of this, but the examples should be enough to get any hackers a good start.

The first thing I want to start with is an example of embedding Devhelp into your own programs.  In case you’ve never used it, DevHelp is the API documentation browser for Gtk+/GNOME which uses WebKit to display GTK-DOC documentation.  Some programs like Anjuta IDE integrate Devhelp to provide a more complete development environment.  As for myself, I’ve been writing a Devhelp plugin for Geany IDE, which is why I started looking into embedding Devhelp.

Here’s a little screenshot of the example running on my computer:

Without further ado, here’s a very simple GTK+ program with minimal functionality to depict using Devhelp within your own program:

#include 

/* These are the parts of Devhelp we're going to use */
#include 
#include 
#include 
#include 
#include 

/* We're going to use WebKitGTK to display the GTK-DOC HTML pages */
#include 

/* Prototype for the callback function to load the book page */
void on_link_clicked(GObject *not_used, DhLink *link, gpointer data);

int main(int argc, char *argv[])
{
	GtkWidget *window, *hpaned;
	GtkWidget *webview;	/* WebView to show docs in */
	GtkWidget *viewsw; 		/* To make the WebView scrollable */
	GtkWidget *treesw;		/* To make the "Contents" treeview scrollable */

	DhBase *base;			/* I guess this is the main "application" */
	DhBookManager *mgr;	/* Manages the book collection */
	GtkWidget *book_tree;	/* This is the "Contents" part of the UI */
	GtkWidget *search;		/* This is the "Search" part of the UI */

	GtkWidget *notebook;	/* Will hold the Contents and Search widgets */

	gtk_init(&argc, &argv);

	/* Create a main GtkWindow with a GtkHPaned */
	window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
	gtk_window_set_default_size(GTK_WINDOW(window), 800, 600);
	gtk_window_set_title(GTK_WINDOW(window), "Devhelp Embedding Example");
	g_signal_connect(window, "destroy", gtk_main_quit, NULL);
	hpaned = gtk_hpaned_new();
	gtk_paned_set_position(GTK_PANED(hpaned), 250);
	gtk_container_add(GTK_CONTAINER(window), hpaned);

	/* Create a WebKitGTK+ WebView to show the docs in */
	viewsw = gtk_scrolled_window_new(NULL, NULL);
	gtk_scrolled_window_set_policy(GTK_SCROLLED_WINDOW(viewsw),
		GTK_POLICY_AUTOMATIC, GTK_POLICY_AUTOMATIC);
	webview = webkit_web_view_new();
	gtk_container_add(GTK_CONTAINER(viewsw), webview);
	/* Put the webview in the right side of the HPaned */
	gtk_paned_add2(GTK_PANED(hpaned), viewsw);

	/* Get a base object for devhelp */
	base = dh_base_new();

	/* Get the book manager for this application */
	mgr = dh_base_get_book_manager(base);

	/* Get the Contents widget containing the book manager's books */
	book_tree = dh_book_tree_new(mgr);

	/* Get the Search widget containing the book manager's books */
	search = dh_search_new(mgr);

	/* hookup the callback to load the selected link in the webview
	 * when a link is clicked in the Contents widget.  We'll pass the
	 * webview along so that it can have it's uri changed. */
	g_signal_connect(book_tree, "link-selected",
		G_CALLBACK(on_link_clicked), webview);

	/* hookup the callback to load the selected link in the webview
	 * when a link is clicked in the Search widget */
	g_signal_connect(search, "link-selected",
			G_CALLBACK(on_link_clicked), webview);

	/* Create a notebook to hold the Contents and Search widgets */
	notebook = gtk_notebook_new();

	/* We'll put the Contents tab in a scrolled window to make it scrollable
	 * but we don't need it to scroll horizontally since it shortens the
	 * words in the treeview so it never scrolls horizontally. */
	treesw = gtk_scrolled_window_new(NULL, NULL);
	gtk_scrolled_window_set_policy(GTK_SCROLLED_WINDOW(treesw),
		GTK_POLICY_NEVER, GTK_POLICY_AUTOMATIC);
	gtk_container_add(GTK_CONTAINER(treesw), book_tree);
	/* We don't need to make the Search widget scrollable since it's
	 * internal widget already scrolls */

	/* Add the Contents and Search widgets to the notebook and add the
	 * notebook to the left side of the HPaned */
	gtk_notebook_append_page(GTK_NOTEBOOK(notebook), treesw,
		gtk_label_new("Contents"));
	gtk_notebook_append_page(GTK_NOTEBOOK(notebook), search,
		gtk_label_new("Search"));
	gtk_paned_add1(GTK_PANED(hpaned), notebook);

	/* Make everything visible and start the main loop */
	gtk_widget_show_all(window);
	gtk_main();

	return 0;
}

/* This callback will load the selected link into the WebView */
void on_link_clicked(GObject *not_used, DhLink *link, gpointer data)
{
	GtkWidget *webview = data;
	gchar *uri = dh_link_get_uri(link);
	webkit_web_view_open(WEBKIT_WEB_VIEW(webview), uri);
	g_free(uri);
}

Run the following command to compile this file:

gcc -g -Wall -o outputfile -c sourcefile.c `pkg-config --cflags --libs gtk+-2.0 libdevhelp-2.0`

As you can see from reading the comments, you need to have a DhBase object for the entire application, and only one or else it will complain. From there, you need to initialize the DhBookManager object which will take care of loading and managing all of the documentation books you have installed. Once you have these two items taken care of, you’re able to get one or both of the new widgets available to navigate the collection of installed books. You do this by calling dh_book_tree_new() and/or dh_search_new() passing in the DhBookManager object so they know about which books to show. The last important aspect is to make sure when a link is clicked in one of the book tree or search widgets, that it gets loaded into the WebKit web browser. Connect a callback to the link-selected signal and in the callback you get the selected URI using dh_link_get_uri which will return the URI to the local HTML file that needs to be loaded. Then it’s just a matter of displaying the HTML page in the web browser.

This example is just a barebones example but it highlights the important parts (and the only parts I know) of how to embed the Devhelp stuff in your application. Some simple things you could do from here are to add back/forward buttons which work with WebKit WebView’s history, or maybe add some zooming buttons which could call WebView’s webkit_web_view_set_zoom_level() function.

Keep in mind also, this example is for libdevhelp-2.0 which has the book manager. The earlier version of libdevhelp-1.0 won’t work exactly as above. To use the older API, you’ll need to do something like this:

DhBase *base;
GtkWidget *book_tree, *search;
GNode *books;
GList *keywords;

base = dh_base_new();

books = dh_base_get_book_tree(base);
keywords = dh_base_get_keywords(base);

book_tree = dh_book_tree_new(books);
search = dh_search_new(keywords);

I hope this helps someone out. Please leave a comment if it did, or if you have any questions. If you want to see emedding Devhelp in action check-out Anjuta’s Devhelp plugin, which also happens to be the only place I could find an example of doing this.

The next GNOME library I’ll post about will be GladeUI and I’ll show you how to embed the Glade GUI builder into your own program, I just need to find some time.

Ubuntu 10.04+ Fixup Script

I wrote a super simple script to fix some issues I have with some defaults in the latest Ubuntu release.  Nothing majour here, just annoyances really.  Below is the script, read it and comment out any lines you do not want to take effect on your system. It requires root access for a few of the commands, such as managing packages and removing system-wide files. Run it as a regular user and the sudo command will be used to prompt you for a password, run like this:

$ chmod +x ubuntu-fix.sh
$ ./ubuntu-fix.sh

And the following is the source code or download the (original) script here.

UPDATED: I’ve made the script also work for Ubuntu 10.10 as well. It will now ask which of the fixes to apply and I’ve also added a couple new fixes. Download the updated script here.

Low Level Disk IO in Managed .NET

I’ve been messing around with disk I/O in C# lately, attempting to read/modify the boot sectors and such.  I’ve used the lower-level functions for accessing disks like files, from the Win32 API.  Most of the signatures were adapted from the pinvoke.net website as well as some examples from MSDN.

What I’ve done is made one file/class to hold all the unsafe/unmanaged code and another to wrap those functions up nicely in a .NET-style class named DiskStream.  The DiskStream class inherits from the Stream object and so exposes an interface similar to a FileStream.  You can open up a DiskStream and read, write, and seek it just like a regular file.

There are a few gotcha’s associated with performing disk access like this:

  • You need to access the disk in multiples of the sector size (ie. 512 bytes).
  • I don’t believe this will work at all in any Windows below Win2K.
  • Currently has odd behaviour on disks with mounted volumes, if there’s a mounted volume, it appears only the MBR can be written.  I’m going to fix this in a future version so that the volumes get locked and/or dismounted before hand.

So here’s the code for accessing the Win32 API from .NET. I think this is pretty useful, since I had a hard time tracking down this stuff and making it work. Here’s the first class, which I call DeviceIO:

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.Win32.SafeHandles;
namespace DiskLib
{
    ///
    /// P/Invoke wrappers around Win32 functions and constants.
    ///
    internal class DeviceIO
    {
        #region Constants used in unmanaged functions
        public const uint FILE_SHARE_READ = 0x00000001;
        public const uint FILE_SHARE_WRITE = 0x00000002;
        public const uint FILE_SHARE_DELETE = 0x00000004;
        public const uint OPEN_EXISTING = 3;
        public const uint GENERIC_READ = (0x80000000);
        public const uint GENERIC_WRITE = (0x40000000);
        public const uint FILE_FLAG_NO_BUFFERING = 0x20000000;
        public const uint FILE_FLAG_WRITE_THROUGH = 0x80000000;
        public const uint FILE_READ_ATTRIBUTES = (0x0080);
        public const uint FILE_WRITE_ATTRIBUTES = 0x0100;
        public const uint ERROR_INSUFFICIENT_BUFFER = 122;
        #endregion

        #region Unamanged function declarations
        [DllImport("kernel32.dll", SetLastError = true)]
        public static unsafe extern SafeFileHandle CreateFile(
            string FileName,
            uint DesiredAccess,
            uint ShareMode,
            IntPtr SecurityAttributes,
            uint CreationDisposition,
            uint FlagsAndAttributes,
            IntPtr hTemplateFile);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool CloseHandle(SafeFileHandle hHandle);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool DeviceIoControl(
            SafeFileHandle hDevice,
            uint dwIoControlCode,
            IntPtr lpInBuffer,
            uint nInBufferSize,
            [Out] IntPtr lpOutBuffer,
            uint nOutBufferSize,
            ref uint lpBytesReturned,
            IntPtr lpOverlapped);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern unsafe bool WriteFile(
            SafeFileHandle hFile,
            void* pBuffer,
            int NumberOfBytesToWrite,
            int* pNumberOfBytesWritten,
            int Overlapped);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern unsafe bool ReadFile(
            SafeFileHandle hFile,
            void* pBuffer,
            int NumberOfBytesToRead,
            int* pNumberOfBytesRead,
            int Overlapped);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool SetFilePointerEx(
            SafeFileHandle hFile,
            long liDistanceToMove,
            out long lpNewFilePointer,
            uint dwMoveMethod);

        [DllImport("kernel32.dll")]
        public static extern bool FlushFileBuffers(
            SafeFileHandle hFile);
        #endregion

    }

}

With that out of the way, I wrote a Stream class to make accessing the disk more “,NET-ish”. Here’s the code for the stream class:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Text;
using Microsoft.Win32.SafeHandles;
using System.Runtime.InteropServices;

namespace DiskLib
{

    public class DiskStream : Stream
    {
        public const int DEFAULT_SECTOR_SIZE = 512;
        private const int BUFFER_SIZE = 4096;
        private string diskID;
        private DiskInfo diskInfo;
        private FileAccess desiredAccess;
        private SafeFileHandle fileHandle;

        public DiskInfo DiskInfo
        {
            get { return this.diskInfo; }
        }

        public int SectorSize
        {
            get { return (int)this.diskInfo.BytesPerSector; }
        }

        public DiskStream(string diskID, FileAccess desiredAccess)
        {
            this.diskID = diskID;
            this.diskInfo = new DiskInfo(diskID);
            this.desiredAccess = desiredAccess;
            // if desiredAccess is Write or Read/Write
            //   find volumes on this disk
            //   lock the volumes using FSCTL_LOCK_VOLUME
            //     unlock the volumes on Close() or in destructor
            this.fileHandle = this.openFile(diskID, desiredAccess);
        }

        private SafeFileHandle openFile(string id, FileAccess desiredAccess)
        {
            uint access;
            switch (desiredAccess)
            {
                case FileAccess.Read:
                    access = DeviceIO.GENERIC_READ;
                    break;
                case FileAccess.Write:
                    access = DeviceIO.GENERIC_WRITE;
                    break;
                case FileAccess.ReadWrite:
                    access = DeviceIO.GENERIC_READ | DeviceIO.GENERIC_WRITE;
                    break;
                default:
                    access = DeviceIO.GENERIC_READ;
                    break;
            }

            SafeFileHandle ptr = DeviceIO.CreateFile(
                id,
                access,
                DeviceIO.FILE_SHARE_READ,
                IntPtr.Zero,
                DeviceIO.OPEN_EXISTING,
                DeviceIO.FILE_FLAG_NO_BUFFERING | DeviceIO.FILE_FLAG_WRITE_THROUGH,
                IntPtr.Zero);

            if (ptr.IsInvalid)
            {
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }

            return ptr;
        }

        public override bool CanRead
        {
            get
            {
                return (this.desiredAccess == FileAccess.Read || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
            }
        }

        public override bool CanWrite
        {
            get
            {
                return (this.desiredAccess == FileAccess.Write || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
            }
        }

        public override bool CanSeek
        {
            get
            {
                return true;
            }
        }

        public override long Length
        {
            get { return this.diskInfo.Size; }
        }

        public override long Position
        {
            get
            {
                long n = 0;
                if (!DeviceIO.SetFilePointerEx(this.fileHandle, 0, out n, (uint)SeekOrigin.Current))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
                return n;
            }
            set
            {
                if (value > (this.Length - 1))
                    throw new EndOfStreamException("Cannot set position beyond the end of the disk.");
                long n = 0;
                if (!DeviceIO.SetFilePointerEx(this.fileHandle, value, out n, (uint)SeekOrigin.Begin))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
        }

        public override void Flush()
        {
            // not required, since FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING are used
            //if (!Unmanaged.FlushFileBuffers(this.fileHandle))
            //    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }

        public override void Close()
        {
            DeviceIO.CloseHandle(this.fileHandle);
            base.Close();
        }

        public override void SetLength(long value)
        {
            throw new NotSupportedException("Setting the length is not supported with DiskStream objects.");
        }

        public override unsafe int Read(byte[] buffer, int offset, int count)
        {
            int n = 0;
            fixed (byte* p = buffer)
            {
                if (!DeviceIO.ReadFile(this.fileHandle, p + offset, count, &n, 0))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
            return n;
        }

        public override unsafe void Write(byte[] buffer, int offset, int count)
        {
            int n = 0;
            fixed (byte* p = buffer)
            {
                if (!DeviceIO.WriteFile(this.fileHandle, p + offset, count, &n, 0))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            long n = 0;
            if (!DeviceIO.SetFilePointerEx(this.fileHandle, offset, out n, (uint)origin))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            return n;
        }

        public int ReadSector(DiskSector sector)
        {
            return this.Read(sector.Data, 0, sector.SectorSize);
        }

        public void WriteSector(DiskSector sector)
        {
            this.Write(sector.Data, 0, sector.SectorSize);
        }

        public void SeekSector(DiskSector sector)
        {
            this.Seek(sector.Offset, SeekOrigin.Begin);
        }
    }

    public struct DiskSector
    {
        public const int DEFAULT_SECTOR_SIZE = 512;
        private long offset;
        private byte[] data;
        private int sectorSize;
        public int SectorSize
        {
            get { return this.sectorSize; }
            set
            {
                if ((value % 2) != 0)
                    throw new ArgumentException("Sector size must be a multiple of 2.");
                this.sectorSize = value;
            }
        }

        public long Offset
        {
            get { return this.offset; }
            set
            {
                if ((value % this.SectorSize) != 0)
                    throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
                this.offset = value;
            }
        }

        public byte[] Data
        {
            get { return this.data; }
            set
            {
                if (value.Length != this.SectorSize)
                    throw new ArgumentException("Data length must be the same as SectorSize.");
                this.data = value;
            }
        }

        public DiskSector(byte[] sectorData, long sectorOffset, int sectorSize)
        {
            if ((sectorSize % 2) != 0)
                throw new ArgumentException("Sector size must be a multiple of 2.");
            this.sectorSize = sectorSize;
            if (sectorData.Length != sectorSize)
                throw new ArgumentException("Data length must be the same as SectorSize.");
            this.data = sectorData;
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }

        public DiskSector(byte[] sectorData, long sectorOffset)
        {
            int sectorSize = DEFAULT_SECTOR_SIZE;
            this.sectorSize = sectorSize;
            if (sectorData.Length != sectorSize)
                throw new ArgumentException("Data length must be the same as SectorSize.");
            this.data = sectorData;
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }

        public DiskSector(long sectorOffset)
        {
            int sectorSize = DEFAULT_SECTOR_SIZE;
            this.sectorSize = sectorSize;
            this.data = new byte[sectorSize];
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }
    }

}

You might’ve noticed the DiskInfo class, I’m not going to post the code here, but I’ll provide a download link. It’s currently just a wrapper around WMI/Win32_DiskDrive, although I’m planning on removing the WMI code and replacing it with more platform invoke calls. I’ve also added a DiskSector struct to (attempt to) make accessing the disk using sectors easier. The DiskSector struct can be used with the ReadDiskSector(), WriteDiskSector() and SeekDiskSector() methods of the DiskStream class.

And finally a WARNING to anyone intending to use this code: messing around with disks like this is EXTREMELY dangerous and you can hose your whole system very easily. I HIGHLY recommend you play with this code within a Virtual Machine (VirtualBox, Qemu, etc) or on a separate system that you don’t mind losing.

It’s also important to note that this is very early code, and it has not at all been thoroughly tested in any way. I’ve used it to toggle some bits in the MBR, read the partition table, and to wipe a disk clean, and that’s the extent of my testing. Proceed with caution and at your own risk.

I’ll update this post as I improve the code here.

Here’s a bunch of source files which could be useful in Zip format.

Python Whois

A little Whois reader I wrote while messing around with sockets in Python.  It doesn’t do any processing on the reply, it just returns the whole thing as a string.  It’s also missing any mechanism to determine which Whois servers to use for which TLDs.  But anyway, it’s a good start!

#!/usr/bin/env python
import socket
import select

PORT = 43
BUFSIZE = 1024
LINEEND = 'rn'

def whois(domain, server, port=PORT):
    '''
    Perform a WHOIS search for domain on server/port.
    '''
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((server, port))

    msg = ''
    lookup = domain + LINEEND
    readable, writable, error = select.select([],[s],[],60)

    if s in writable: s.send(lookup)

    readable, writable, error = select.select([s],[],[],60)
    while s in readable:

        data = s.recv(1024)
        msg = msg + data
        if not data:
            s.shutdown(socket.SHUT_RDWR)
            s.close()
            break
        readable, writable, error = select.select([s],[],[],60)

    else:
        s.shutdown(socket.SHUT_RDRW)
        s.close()

    return msg.strip()

if __name__ == '__main__':

    print whois('google.com', 'whois.markmonitor.com')

Python Module for Geo-Coding/Geo-Location

I’ve written a very simple python module/script to help find postal code, city, province, latitude/longitude matching the supplied postal code, city, province, or coordinate.

Check out the Google Code Project page for more information.

You can checkout the source code by issuing the following command:

svn checkout http://python-geolocate.googlecode.com/svn/trunk/ python-geolocate-read-only

Osmosis Won’t Read ‘planet-latest.osm.bz2′

I couldn’t get osmosis to read my OSM planet file, I even tried downloading another one, which of course took a long time. Then I found a post on the mailing list talking about the problem.

The solution is rather simple, just extract the file before passing it to osmosis to read, for example:

bzcat planet-latest.osm.bz2 | ./bin/osmosis --read-xml file=- --bounding-polygon file="country2pts.txt" --write-xml file="country.osm.EXT"

The error message I received was this:

kManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file planet-latest.osm.bz2.  publicId=(null), systemId=(null), lineNumber=4313, columnNumber=13.
	at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:113)
	at java.lang.Thread.run(Thread.java:636)
Caused by: org.xml.sax.SAXParseException: XML document structures must start and end within the same entity.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:391)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1390)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:878)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:581)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1369)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1740)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1469)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.seekCloseOfStartTag(XMLDocumentFragmentScannerImpl.java:1351)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1286)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2723)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:624)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:486)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:810)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:740)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:110)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:392)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
	at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:108)
	... 1 more
Sep 28, 2009 7:30:16 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30)

Installing Mapnik and Friends on Ubuntu 9.04

I’m posting this for anyone who’s had a hard time getting Mapnik installed
along with some handy utilities on Ubuntu 9.04. Here we go.

Mapnik from Source

Update the package listing and then upgrade all packages:

apt-get update
apt-get upgrade

After that install the dependencies:

sudo apt-get install g++ cpp
libboost1.35-dev libboost-filesystem1.35-dev
libboost-iostreams1.35-dev libboost-program-options1.35-dev
libboost-python1.35-dev libboost-regex1.35-dev
libboost-thread1.35-dev
libxml2 libxml2-dev
libfreetype6 libfreetype6-dev
libjpeg62 libjpeg62-dev
libltdl7 libltdl7-dev
libpng12-0 libpng12-dev
libgeotiff-dev libtiff4 libtiff4-dev
libcairo2 libcairo2-dev python-cairo python-cairo-dev
libcairomm-1.0-1 libcairomm-1.0-dev
ttf-dejavu ttf-dejavu-core ttf-dejavu-extra
libgdal1-dev python-gdal
postgresql-8.3-postgis postgresql-8.3
postgresql-server-dev-8.3 postgresql-contrib-8.3
libsqlite3-dev
subversion build-essential

Now we need to get the Mapnik source from the repository:

mkdir -v ~/src
cd ~/src
svn co http://svn.mapnik.org/trunk mapnik
cd mapnik

After that we can configure and build Mapnik:

python scons/scons.py configure INPUT_PLUGINS=all
OPTIMIZATION=3
SYSTEM_FONTS=/usr/share/fonts/truetype/ttf-dejavu/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Now we’ll do a quick test to see if Mapnik installed correctly:

python # will start the python interpreter
>>> import mapnik
>>> exit()

If the import mapnik line didn’t spit out any errors, everything is probably ok.


Setting Up the Database

First we need to tweak some configuration files. Open
/etc/postgresql/8.3/main/postgresql.conf with your favourite
text edit and modify the following lines:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

Next edit /etc/sysctl.conf and add the following at the end:

kernel.shmmax=268435456

Now to avoid a reboot, we’ll update the kernel like this:

sudo sh -c 'echo 268435456 >/proc/sys/kernel/shmmax'

Then restart the PostgreSQL server:

sudo /etc/init.d/postgresql-8.3 restart

The next thing we need to do is create the database. Substitute
your_user” below with your user name. Note: If your user name is ‘user’ you might get an error from the commands below, I did when I tried.

sudo -u postgres -i
createuser -s your_user
createdb -E UTF8 -O your_user gis
createlang plpgsql gis
exit

Now setup PostGIS on our database:

psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql

And then change some permissions on the database:

echo "ALTER TABLE geometry_columns OWNER TO your_user; ALTER TABLE spatial_ref_sys OWNER TO your_user;" | psql -d gis

Now we’ll enable the intarray module:

psql gis < /usr/share/postgresql/8.3/contrib/_int.sql

Installing Some Extra Tools

The tools we'll be installing here are useful for working with OpenStreetMap
data and also to facilitate generating maps with Mapnik.

OSM2PGSQL

This utility will import OpenStreetMap data files into the PostGIS database. First we'll install some dependencies:

sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj

And then checkout the latest version from the repository:

cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
make

Mapnik-Tools

We're going to put this right in our home directory:

cd ~
svn co http://svn.openstreetmap.org/applications/rendering/mapnik/

Setting up the World

Now we're going to grab some boundaries data for the world.

cd ~/mapnik
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2

We'll extract them each into ~/mapnik/world_boundaries:

tar xfv world_boundaries-spherical.tgz
rm -v world_boundaries-spherical.tgz
mv -v processed_p.tar.bz2 shoreline_300.tar.bz2 world_boundaries/
cd world_boundaries
tar xfv processed_p.tar.bz2
tar xfv shoreline_300.tar.bz2
rm -v processed_p.tar.bz2 shoreline_300.tar.bz2

Importing Some Data

The last thing we're going to do is put some OpenStreetMap data into our database. You can either import the whole planet:

cd ~
wget http://ftp.heanet.ie/mirrors/openstreetmap.org/planet-latest.osm.bz2

But it's a HUGE file, so if you just want to have a smaller part of the world,
you can use extracts. To create your own extracts, read this tutorial. Otherwise
grab an extract for your area of interest from somewhere (ex.CloudMade). You'll want to download the file ending in .osm.bz2. In my example I'll be using british_columbia.osm.bz2.

cd ~
wget http://downloads.cloudmade.com/north_america/canada/british_columbia/british_columbia.osm.bz2

We just need to import the data into the database now, for this we'll use osm2pgsql which we installed earlier.

cd ~/src/osm2pgsql
./osm2pgsql --slim -d gis ~/british_columbia.osm.bz2

And there you have it. If you want to test this out from the command line, you
can use the last portion of this tutorial, or to use python, try the examples here.

Resources

Most of this tutorial was ripped directly off from the following sites:

Mapnik Package for Ubuntu

I made a Debian/Ubuntu package for Mapnik v0.6.1. There’s a 32-bit and 64-bit package. I’ve only tested it on Ubuntu 9.04, but it will probably work on Debian.

To install it download the package and type:

sudo dpkg -i mapnik_0.6.1-1_i386.deb

Then when it complains about dependencies, type:

sudo apt-get install -f

To install all the dependencies and finish the install. Sorry I don’t have a repository to put this package in.

I think if you use GDebi Installer it will do everything in one step.

Note: There are lots of dependencies (~500MB on a clean Ubuntu alternate install), I used those listed here. I tried removing all the -dev packages and anything else that looked like it was just for compiling, but it didn’t work.

UPDATE: I’ve also made an osm2pgsql package from the latest subversion copy. Download it here.

Enjoy

Download the Mapnik v0.6.1 x86 Package Here

Download the Mapnik v0.6.1 amd64 Package Here