The importance of profiling

I am just working on a project where I have to parse XML files that are about 50 (+/-) megabyte big. Because I still think that PC run time is less expensive then my programing time I am writing this in python. I started coding using the xml.sax library and just filled in the gaps. But the more I added the slower the code became, in one case it took the parser about 10 min to create all the structures I need out of the XML file. After getting the functionality right with a subset of data I started profiling why it was taking sooooooooooo long. I used the excellent python package

profile.py -o statsout readRepro.py

So the first thing I found was in this method:

def characters(self, data): self.tempData +=data

This is called every time I encounter the data section of a XML element. Because it kept appending strings to stings this took ages. I replaced it with

def characters(self, data): self.tmpbuff.append(data)

and the run time went down form over 10 min to 7.2 seconds. How amazing is that. I would have never guest that would take up so much time. That is one little change. If I would have had to guess what was taking so long I would have optimized the parsing. After more work I found another method that I had just built in to do some debugging. It was basically checking I was coping with all the elements I would encounter. Useful while coding but after commenting one line I improved the performance from 55.7 to 6.6 seconds.This just proves that we as programmers have no idea where our programs spend most of their time. If you are coding and you are thinking of optimizing something, chances are you are wrong. I know a lot of people say this and a lot of people are against it, but my finding is that actually finding out where your programs spends the time is far more worth then guessing.
Lesson learned for life: "Use a tool to see where you are spending your time"
It might be in some debug method you don't really need.

Media on Mac

I had huge problems playing different media files on my Mac. For example DivX files played without sound. After installing http://perian.org/ everything seams to work a little better. A little astonishing that a machine that is designed for media needs media plugins. But never mind it works fine now.

DevMob holiday for intel employees (and friends)

I am just back from DevMob 2010 that was held at the Science Museum in London. Here are some of my impressions. The first thing that you noticed when you looked the conference up on the net was this quote

London 'unconference' giving developers interested in portable devices the opportunity to discuss their issues and find solutions

Reading this now makes me chuckle as it really was an 'unconference'. But more about this later. On my free day I decided to attend this as I am currently looking into developing on mobile platforms and I have already heard many talks on the topic. There was also a competition I wanted to attend with an EeePC as a price. Arriving at the venue the first thing you noticed was the money they must have spent on the location, it was really nice. An old library room with a lot of books and all very posh. Another thing that was astonishing for a free conference was that there was coffee and little pastries up for grabs being served by a friendly well dressed young woman. After sitting around and waiting for a little wile (conference should have started at 10) we finally all gathered and started writing the schedule. You don't have to imagine this like being a Barcamp sort of style get together. No there were fixed talks that where allocated at certain times but it was made out to be a group effort. This is when I noticed that there where actually more "official" people with Intel badges then "real" guests, which seamed a little odd. Then we disbursed for the talks. I listened to the first talk about Moblin (the Intel Linux platform) which was sort of nice but the lecturer didn't seam to know what he was talking about and repeatedly asked a lady in the audience that didn't seam very interested in what was happening. At the beginning I started talking to a guy and now it was his turn. Instead of talking about something useful he used the time to describe a graph he had come up with and asked us what we would improve. He is a professional blogger so I wonder if he will cite all the ideas we gave him. He tried to save the talk by bringing loads of gadgets that you could play around with at the end, but at that time I was already considering going home (which some of my friends did). But no, I stayed. Why you might ask, bear with me there was an EeePC. Lunch was calling and again they had a really nice buffet with sandwiches and a lot of nice food. (Cost ?) After that I participated in a discussion on optimization and then on how Intel is planing to sell applications through their store. More people started arriving and I talked to some students that had also come from UCL. Now comes the main point that makes this day an absolute waste of my time and that gives me the impression that this was all a charade. On the invitation it said that

There are still plenty of opportunities to walk away with prizes and goodies, including the chance to win a netbook, courtesy of our sponsor Asus. Remember to bring along your ideas for the perfect netbook app – pitch to the panel and if they think your idea is the best, a netbook is yours to take home

I had an idea and I wanted to pitch it to the panel. So did 4 of my student friends. We spent quite some time talking about our ideas and discussing them. I repeatedly asked people from the organization about this but never got a clear response. As I had prepared something I wanted to talk about it and at least get a fair chance. This was actually the main reason we stayed. But little did I know that the Intel bribery machine was in full swing. At some point we were asked to "register" our topics for this panel. I was the first to put down my name and the area I wanted to discuss. After some time we were all asked to assemble in the main room so we could hear about who had won the competition. I didn't quite understand what was going on but went along. Asking the guy in charge I was informed that the panel had met and that they will not listen to our ideas and that they had made a decision. It became apparent that this was just a big show and that the winner was a guy that was talking to one of the speaker (who was chair of the panel) all day. Remember we where initially told to register our topic on a flip chart, this was then apparently used to evaluate our idea. This can only be described as bribery. Students where tricked into attending and putting forward ideas so that Intel could cover up inviting bloggers to London on their expenses and so get good publicity for Moblin. I am sorry Intel spent so much money for a expensive, disorganized event that this is the only conclusion I can come up with. Talking to some friends on the way home we all agreed that we should have stayed at home and spent our time doing the assignment that is due soon.

P.S. There is another contest :

Win an ASUS EeePC by blogging about DevMob!

Ahh wait. I talked to three professional mobile bloggers that where enjoying the nice expensive food from Intel. Guess who will get another EeePC. You got it, one of the bloggers that will write a friendly nice article about this event and might even mention Moblin.

Additions:

A friend that joined me said he was a little ashamed that UCL was promoting something like this. I will mention it at the next management meeting. I have further left out names on purpose but feel free to talk to me for more information.

I got an email from the organizers, so maybe my rant did change something:

Hi Didi,

Thanks for the blog and your feedback.

I'm sorry that you and your colleagues didn't enjoy the event so much but it's really good to have precise feedback so we can improve future events.

Due to time constraints, unfortunately we weren't able to run the app competition as we wanted to. I know people were disappointed about this, so we will be running a follow-up competition for anyone who wants to submit their application idea. I'll be emailing attendees about this shortly.

A couple of things to mention: the only Intel people there were Stephen Blair-Chappell (talked about optimisation and tuning), Costas and Sulamita (the two who talked about Moblin) and two in an organisational capacity. The rest of the blue badges were other event sponsors, those who had offered in advance to host a session and the event co-ordinators (non-Intel).

Also, a note on the blogging competition - pro-bloggers and journalists are ineligible, so no worries about them winning the prize.

I'll be sending around a feedback form soon and it would be great to hear any further thoughts you have on the event.

I removed the senders name and footer

Online storage

I am always looking out for online storage I can use to backup all my files. So I came along adrive.com which looked quite neat at first sight. It had all the buzz words I was looking for like ftp or ssh, quite big storage, etc ... Don't get me wrong I am happy to pay for a service so I am not just looking at the free options. But still this one got me. Look at the picture and then look again.Would you think that the free version has FTP? I though so. So I created an account and tried to run a few benchmarks on how fast the service is etc.. But I couldn't find the FTP description. Another thing that you can really notice by the design is that this page has been designed by people sitting in a dark room with green terminals. But then I had a look again and noticed that the first green tick in the free version was missing. Which I just assumed as normally all of the first items have a tick an then they are reduced going down based on the plan you choose. I felt like the people from Adrive had tried to trick me and not assuming that they have any bad intentions this is really bad UI design. Now if you look at the dropbox page, it looks really nice and they don't seam and try to trick you. So which service will I as a consumer choose where the pricing of all these services seams to be the same?

Trip to FOSDEM

I know my last post was about how I am not going to go to conferences this year. But FOSDEM is an event I just can't miss. So we are going. Here the mail I sent around

Hello

We are organizing a trip to Brussels for Fosdem in February.

FOSDEM (Free and Open Source Development European Meeting) is a
European event centered around Free and Open Source software
development. It is aimed at developers and all interested in the Free
and Open Source news in the world. Its goals are to enable developers
to meet and to promote the awareness and use of free and open source
software. It is said to be the biggest gathering of Open Source
interested people in Europe and the speaker list is quite impressive.

We will be leaving London on Friday (5. Feb) evening and returning on
Sunday (7. Feb). One of our goals is it to keep the total cost for the
weekend under 200 Pounds, which is quite cheep for a weekend in
Brussels.

People who are interested can look at the wiki [1] or drop me a mail.
Everyone is welcome.

Cheers Didi

[1] http://www.bunix.org.uk/wiki/doku.php/events:fosdem2010

I hope a lot of you can come :)

Google != Goog$e

From now on I will be calling Google Goog$e. This is to indicate that in my opinion Goog$e is just another corporate big shot, that tries to get as much money for their stock holders by any way possible. The cool hackish culutre that they are trying to promote is not different from the one Microsoft had about 20 Years ago. Funny enough every time I am talking to someone from Mirosoft I get the feeling they are coming back to it. They have this really cool research labs where you can pretty much do on what every you please. Maybe Goog$e is the new Micorsoft and vice versa :)

P.S. I am still thinking about if I should still wrinte Micro$oft. I think this will depend on how I am feeling.

Stuff I want in GMail

I have been using GMail for quite a time now and it is getting quite annoying, to be honest. The speed has just become to slow to work. Loading my mail is not a matter of minutes and searching has become pretty much impossible. Then the service is unavailable pretty much once a month which is really annoying if you really need that one address in that one email. And sometimes it just freezes and I need to close the tab and reload it. This is becoming a real pain. Where is the slick nice fast interface, that did everything I needed. I am noticing that I have been using alpine more and more lately just because it is pissing me off so hard. It seams like with the recent comments coming out of Goog$e, the increasing application error count and the impression that time has become uninteresting in favour for features Goog$e seams to be loosing it. (I know of two really good people that have already left because they are getting soo pissed of about the culture that is growing there) Another thing that I am finding quite hard to do is extend GMail. There are two features I want to implement but because Goog$e is more fascist about their source code and the Java Script they publish then Microsoft I am finding this really hard.
The first feature is a little icon next to the Compose Mail button that opens a new window with the email writing window. I have spent about 2 hours reading horrible Java script garbage to figure out how to create the correct AJAX call to trigger this but haven't really progressed so far. The second feature I really want is a TOP button at the bottom of every page that will bring me to the top of the page. It is a real hassle that every time you read a long email thread you have to scroll to the top of the page to see the list of labels again. Both these features seam impossible to integrate into the existing GMail framework. I talked to a friend who does some Windows dev work and apparently you could even script that in Outlook.

Conferences this Year

This is something I have decided a while ago but I though I might blog about it now. I will not hand in any talks to conferences this year (except T-Dose :). This has mainly one reason, TIME. While being at Bournemouth I always complained that stuff was not hard enough and that a well trained hamster could do my assignments*. So I made up a lot of stuff and made my work hard. A lot of times I decided to go the hard way ("F*** Java lets do it in C") just to learn something new. Another thing I am now noticing is that through not having to spend 24/7 doing university work I had time to do invent new stuff, which consequently got me speaking at conferences. Now at my new uni I am spending quite a lot of time doing stuff I don't know. I remember thinking "This seams like a lot of management and requirements, cool I really know nothing about that". But because I am now really learning about new topics which are totally out of my usual scope I don't get the time to invent new or funky things, hence that I cant really talk about a lot at conferences any more. I am still doing a little on objic, mostly reading, but I really need more time to properly get to the next step. I will still attend a few though, Fosdem being the obvious.

P.S. This does not mean that Bournemouth is a shit uni, even if I don't agree with everything they are doing. It just means that if you already know how to program (especially Java) some courses might become quite easy.

*Maybe that is what we need in computing. n hamsters can solve x problems. Just make n big enough and you can solve any x. Who needs monkeys

Windows 7

We got my mum a Acer ONE for Christmas and I spent one whole day trying to set it up so that it would sort of work, as it came with Windows 7. After removing all the shit that Acer seam to think you need I spent another few hours going to different web-sites and downloading flash, acrobat, etc ... and mostly waiting. As a last task I wanted to replace the background image with a picture of my family. How would you go about of doing this? I spend 30 min looking in different menu items and trying to find this one thing. After doubting my intelligence I asked Google and guess what. It is not possible, in Windows starter it is not possible to replace the background image. Hahah how about that? You spend 400 Euros on a PC and you can't even change the background image. Money well spend I would say. Now my mum will enjoy Ubuntu. What a stupid decision, I was really thinking Mic$osft was on the right way but this just blows it. Only retards would use something like that.

Do backups

I took my laptop to a very remote location to actually get some work done. As I am noticing that when I work at home the internet is to big a distraction. So while I was happily working away I wanted to save my done work. At home I normally save all files see if everything builds and then '{svn,cvs,git} commit'. As my server is in Holland and my backup server in the US I need the internet to access them. But what do I do somewhere without internet (Yes these places exist). Every normal person would just continue with his work and commit when he comes home. But let's consider this scenario for a moment. While on the car my laptop wakes up out of suspend because something has pressed the button or I didn't close the lid properly. Now the hard drive is turning but exactly now I driver over a bump => headcrash. One week of work gone. Hard drives are something so liable you should never rely on them. My friend Edd breaks about one every week. So what to do. Risk it, no way. I remembered that while at a conference last week I was given a USB drive (Thank you Microsoft Research) so I pooped it in and now save all my data on this.

$ cp -ruv uni/ /media/disk/

does the trick. Now 2 things would have to break for me to lose my work. And if everything works out OK I format the Dongle and everyone is happy. But if my laptop decides to be funny I still have all my work.
I can not stress enough how important it is to think about backups. If you want your data to be reasonably secure you should always have a backup in a different physical location and secured with a different password. Loads of my friends and lecturers (that call them selves IT specialists) do no such thing. I can only hope they don't have to learn the hard way. I think this is a really understated problem. I know loads of people that say "Ah I do regular backups. I have a external hard drive next to my pc". And how much security does that give you? A robber taking your pc will also take the drive and your data is gone. At least use a service like dropbox for your important data. Taking away all the crap (mp3, videos etc..) you can download again normally you don't have that much "really vital" data. I have about 1 GB of files I really really don't want to lose.

Professor Michael Worton thinks UCL students are stupid

As some of you may know I am a student at UCL and as you can see from my previous posts I get a lot of mail. So I get really annoyed when people think I am to stupid to see something the first time they send it to me. Especially if people just resend the same mail over and over gain. It looks like Professor Michael Worton and Claire Underwood seam to think that all students at UCL (all-postgraduates|at|ucl.ac.uk, all-undergraduates|al|ucl.ac.uk) are morons and to stupid to read their email the first time they get it. And apparently we are not able to read and respond properly (to our own judgment) the second time. Ahhh because we really haven't figured out what a email is, just send it again the third time. The same mail.

Message from Professor Michael Worton Vice Provost (Academic & International)

I agree that some student survey is really vital for my success at UCL and important but still I emailed the sender and informed here that I was already getting enough mail and that I don't want this spam. And guess what, to further insult the students at UCL they are sending out a fourth reminder. Now that makes me wonder. If we get FOUR reminders about another student survey how many do we get about something that is really going to influence my student life (Exams, etc ..). I will be expecting someone to call me personally from UCL as some people seem to think that we are to stupid to read and think.

REST vs SOAP

For the Advanced Analysis and Design course at my university I have to compare REST with SOAP. My first reaction as I heard this was WTF. How is it possible to compare a strictly defined protocol with an approach idea like REST. I don't know if my professor doesn't understand the concepts of both or if this is the realization he wants us to get. For the people that don't know the difference here a little rundown. SOAP stands for Simple Object Access Protocol and was initially designed to call functions/methods on remote machines (basically RPC shredded through the XML buzzword machine) whereas REST stands for Representational State Transfer and describes how the resources in the internet can be arranged and talk to each other. Mainly described by the example of http. Of course you can use both methods to get data from some source (Amazon for example) but the general ideas are so fundamentally different that a real thorough comparison is not possible. Sometimes I do wonder. If you would only read http://en.wikipedia.org/wiki/SOAP and http://en.wikipedia.org/wiki/Representational_State_Transfer you would understand my point. That simple.

Flamewar commence:

How much money can I make with my domain

I just had a look at http://www.thenetinfo.com and that tells me that my domain is worth:

Ribalba.de Estimated Worth $88 USD

Wtf. Who would give me that money. Really. And all the other sites I tried it just doesn't give you any info at all. Ahh wait

Stallman.org Estimated Worth $5.8 Thousand USD

and

Microsoft.org Estimated Worth $292 USD

So really who would belive such a thing?

The problem with google analytic when you run your own server

On my website www.ribalba.de I run google analytic which lets me see how many people look at my website where they come from and what they are looking for. Not that it really matters but I am just interested in why people would want to read about me. Assuming I should only be interesting for about 50 (+/-) people in the World. But I also run some analyzers on my web server log files and it turns out that the main thing my server is doing is serving files I have outside of my wiki (which I use for my site) they are files that I just have in folders and that can be accessed through directory listing turned on. But these are never accounted for in analytics. So if I direct my browser to http://www.ribalba.de it will be saved in analytics but when I goto http://www.ribalba.de/geek it does not. So in some respect analytics is giving me a wrong picture. Further image downloads from my server are not displayed. So not really useful as you are forgetting a huge junk of data and so your analysis of your site is bound to be wrong.

wrong start data => wrong conclusion

Maybe Google should offer a method by which I can upload the log-files after logrotate has run and so the file is not used anymore. So always take with a grain of salt what google is trying to tell you.

libnice for CentOS

Some guy on the CentOS mailing list needed libnice so I built it. It can be downloaded from:

http://www.ribalba.de/geek/port/

The source package is in the src folder and the builds in the respective directories.

I hope this helped some poor sole :)

Just for your inforamtion:

GLib ICE implementation

libnice is an implementation of the IETF's draft Interactive Connectivity
Establishment standard (ICE). ICE is useful for applications that want to
establish peer-to-peer UDP data streams. It automates the process of traversing
NATs and provides security against some attacks. Existing standards that use
ICE include the Session Initiation Protocol (SIP) and Jingle, XMPP extension
for audio/video calls.

Why being nice pays off

I have lived in London for little more than a month now. My daily routine includes getting the tube to Uni and on the way I pick up my morning reading normally a Financial Times, a Guardian and a bottle of water. I do this pretty much every morning. So I spend quite a lot of money for this if you take it time 365. At my tube station I can choose between two news-agents that sell papers. As I am coming from the right I naturally always went into that one. And I never though much about it. It is owned by a family (it seams) and the people that work there are not rude but they are not friendly and helpful. After going there for a month they still didn't know what I was getting. In the morning when I am quite grumpy the last thing I really want is someone to be grumpy back at me. But I just accepted it and continued buying my stuff there. Four days ago I came from the left so I thought why not try the other shop. And what a difference, I was greeted really friendly and I had a little chat with the guy behind the counter. The next day I went to the left one agin and the same guy was friendly again. Now after just four days he already knows what time I normally go to university and of course the papers I buy. It never occurred to me that in a world where prices are pretty much fixed friendliness is one of the last selling points. Location might be important too, but these two shops only differ in the staff. They have the same stock and same prices and pretty much the same location. So the reason I choose the left one is because I am treated in a friendly way and people remember me. This might sounds stupid but I bet you have all done it. If someone treated you in an unfriendly manner you would say to yourself "I am not coming back" and if someone knew your face after entering a shop three times you felt sympatric with that guy.

So what can we learn out of this for IT. In the IT ecosystem prices are quite fixed too and location is becoming even more irrelevant. So maybe friendliness is a major point. Maybe your costumer will chose you the next time because you remembered him, like I take the little hassle of walking one minute more every morning just to be treated nicely. If you are a freelancer this is really important, in my opinion, as at the end of the day you are not much more than the news agent competing at a train station. Just you are competing with 100000 other shops and not one.

CentOS Pulse #0906 - The CentOS Newsletter

Hi all,

With a little delay, issue #0906 of the CentOS Pulse newsletter has
been released. It covers topics like the CentOS 5.4 release, a very
funny featured article and a interesting event review and much much
more.

You can read the newsletter at:

http://wiki.centos.org/Newsletter/0906

More information about the newsletter and how you can contribute is
available from:

http://wiki.centos.org/Newsletter

Enjoy Reading!

Didi

And another email address

I am collecting email addresses. I have 8 different locations emails goto and then 25 different forwards to one of the 8. But one of the coolest address I have so far is:

which of course is just a forward but still quite funky

The problem in how I read mail

So I get quite a lot of mail every day, like a lot of us do. But there is no way that I can read all of them at once or even start answering them. I normally filter out the really vital stuff every morning and evening and then when I get time I read the rest. When I need to answer something and I don't have the time to do this I flag the mail with a little star. Now when I get time I start answering the mails that I have put a little star on. I further use my mail as a TODO list. So I send myself a mail with what I have to do and add a star. So I end up with a list of stuff I have to answer or do. Now there is a flaw with doing it this way. Think about it. .....

Ok the problem is that GMail orders the newest message first. So you see the latest message on the top. Now when you start working of your TODO list you start with the newest item. As you never manage to answer all messages you are always left with some at the bottom. You are answering mails like a stack. Which results in that you will have mails at the bottom that are really old. And as long as you don't work of the stack to the bottom they will age continuously. So there is really important stuff in my TODO list I just forgot about and never did. I should have really just checked the dates once :)

Howto send mail through Gmail using your UCL address

Following my previous post on how to receive UCL mail with Gmail now here is how to send mail using Gmail but with your UCL address. You have to go to settings (top right) and Accounts and Imports and then click on the button with "Send mail from another address" So here is the data you need.

Email address: yourlogin@live.ucl.ac.uk

Then you have to click on send trough SMTP server and enter the details

SMTP Server: pod51002.outlook.com
Port: 25
Username: yourlogin@live.ucl.ac.uk
Password: you should know this

And you don't click on enable SSL. I am sure you can do this with SSL somehow but I don't have time to figure it out :)

Then Google will send you a confirmation code which you have to enter. Please check in your spam folder as this is where my confirmation went. But after that you can just send mail like you are using outlook.com