The importance of profiling

I am just working on a project where I have to parse XML files that are about 50 (+/-) megabyte big. Because I still think that PC run time is less expensive then my programing time I am writing this in python. I started coding using the xml.sax library and just filled in the gaps. But the more I added the slower the code became, in one case it took the parser about 10 min to create all the structures I need out of the XML file. After getting the functionality right with a subset of data I started profiling why it was taking sooooooooooo long. I used the excellent python package

profile.py -o statsout readRepro.py

So the first thing I found was in this method:

def characters(self, data): self.tempData +=data

This is called every time I encounter the data section of a XML element. Because it kept appending strings to stings this took ages. I replaced it with

def characters(self, data): self.tmpbuff.append(data)

and the run time went down form over 10 min to 7.2 seconds. How amazing is that. I would have never guest that would take up so much time. That is one little change. If I would have had to guess what was taking so long I would have optimized the parsing. After more work I found another method that I had just built in to do some debugging. It was basically checking I was coping with all the elements I would encounter. Useful while coding but after commenting one line I improved the performance from 55.7 to 6.6 seconds.This just proves that we as programmers have no idea where our programs spend most of their time. If you are coding and you are thinking of optimizing something, chances are you are wrong. I know a lot of people say this and a lot of people are against it, but my finding is that actually finding out where your programs spends the time is far more worth then guessing.
Lesson learned for life: "Use a tool to see where you are spending your time"
It might be in some debug method you don't really need.

A useful little bashrc addition

I really like checking if commands have executed successfully. This is something quite important if you are administering a machine. Normally I just echo $? but now I have added a new line to my .bashrc that will show me this in colour.

export PS1='\[\033[0;32m\]\u@\h \[\033[0;33m\]\w\[\033[${?/[^0]/31}m\]\$ \[\033[0;38m\]'

This will make the prompt green if everything went OK and red if the process didn't return 0. Nice little helper :)

Howto have a long itemize or enumerate split over many pages

I think it is time I start blogging about all the LaTex stuff I discover all the time. This one was a little tricky. I had a 3 page long enumeration that was bound to a table. So in latex:

\begin{longtable}{|p{\textwidth}|} \hline
\begin{enumerate}
\item ....
\repeat 500 times
...

But of course the enumeration could not be split so it just went off the bottom. What to do? After some thinking I came up with this bodge. If you know a nicer way of doing this please tell me. 

I defined a new command \breaktable

\newcommand{\breaktable}{%
 \setcounter{enumi_saved}{\value{enumi}}
 \end{enumerate}\\
 \begin{enumerate}
 \setcounter{enumi}{\value{enumi_saved}}
}

which I now enter every time I see a sort of logic end to the enumeration. This saves the current counter stops the enumeration adds a new cell and starts the enumeration with the correct value again. You have to define

\newcounter{enumi_saved}

somewhere. Works quite nicely.