/dev/dog

After reading a post on edd's blog about a cat and that it wasn't a device I though ?Why not?

14:12 <> the cat is not a device driver ffs
14:13 < #4> cat /dev/cat ;)

So her you go. Now you can have a device called dog in Linux and feed it with
echo "Food" > /dev/dog
cat /dev/dog

If you want more features just tell me :)

 1: /*
 2:  * The Dog kernel module
 3:  * Creates a /dev/dog
 4:  *
 5:  * Hoffmann Geerd-Dietger 
 6:  * Thx to:
 7:  * Valerie Henson 
 8:  *
 9:  */
10: #define BUFFER 80
11: 
12: #include <linux/fs.h>
13: #include <linux/init.h>
14: #include <linux/miscdevice.h>
15: #include <linux/module.h>
16: #include <asm/uaccess.h>
17: 
18: MODULE_LICENSE("DUAL GPL/BSD");
19: MODULE_AUTHOR("Hoffmann Geerd-Dietger ");
20: MODULE_DESCRIPTION("A little dog living in /dev");
21: MODULE_VERSION("0.1");
22: 
23: 
24: static char hello_str[BUFFER]="Hello Master, just echo to me and then cat\n";
25: 
26: static ssize_t dog_write(struct file *filp, const char __user *buff,
27:                            size_t count, loff_t *offp)
28: {
29:   /* We don't care just chop it off */
30:   if (count >= BUFFER)
31:     count = BUFFER;
32: 
33:   if (copy_from_user(hello_str,buff, count))
34:     return -EINVAL;
35: 
36:   return count;
37: }
38: 
39: 
40: static ssize_t dog_read(struct file * file, char * buf,
41:                           size_t count, loff_t *ppos)
42: {
43: 
44:   if (count < BUFFER)
45:     return -EINVAL;
46: 
47:   if (*ppos != 0)
48:     return 0;
49: 
50:   if (copy_to_user(buf, hello_str, BUFFER))
51:     return -EINVAL;
52: 
53:   *ppos =  BUFFER;
54: 
55:   return BUFFER;
56: }
57: 
58: 
59: static const struct file_operations dog_fops = {
60:   .owner    = THIS_MODULE,
61:   .read     = dog_read,
62:   .write    = dog_write,
63: };
64: 
65: static struct miscdevice dog_dev = {MISC_DYNAMIC_MINOR,"dog",&dog_fops };
66: 
67: static int __init
68: dog_init(void)
69: {
70:   int ret;
71: 
72:   ret = misc_register(&dog_dev);
73:   if (ret)
74:     printk(KERN_ERR
75:            "Unable to register \"Dog\" misc device\n");
76: 
77:   printk("dog: Woof Woof\n");
78: 
79:   return ret;
80: }
81: 
82: module_init(dog_init);
83: 
84: static void __exit
85: dog_exit(void)
86: {
87:   misc_deregister(&dog_dev);
88:   printk("dog: Grrrrrrrrrrrrrrrrr");
89: }
90: 
91: module_exit(dog_exit); 

Makefile:

obj-m := dog_dev.o

KDIR  := /lib/modules/$(shell uname -r)/build
PWD   := $(shell pwd)

default:
      $(MAKE) -C $(KDIR) M=$(PWD) modules

Of course you have to insmod the module

Cern Week 22

This week I spent most of my tim elerning how kernel modules work. To do this I write a new device node called /edv/dog that will save 80 chars, that you can pipe into and echo from. This was just to lern how the kernel works. I further optimized the ldap to file translation program. More in other posts. At Cern there is a program called phone that I started reviewing because Dan and me are thinking of rewriting it in Perl to make it platform independant on not rely on old REXX scripts. Further everyone has gone on holiday so there is not much to do

Google Ads beeing just a little bit racist

How can google post this ???

Other Ads might include :
My Son is Black?
My Son has Sex with a condom?
My Son ...

Google should start reviewing their ads. Really we are not in the Dark Ages anymore. Maybe the mothers struggle was about the boy friend not beeing able to cook good food. But I sort of doupt that.

A little bit of advice

While reviewing some scripts, I saw this quite a lot
chmod a+x *.pl
So everyone becomes execute rights. After some research I found out that this is not really needed only the user needed execute rights. So instead why didn't the author write
chmod u+x *.pl
I don't know but if everyone can execute your script you might have a security risk at hand. So never give everyone execute on your stuff.

Cern Week 21

I am still spending most of my time on quattor. Even if this was not really in my job spec. But now everything seems to work fine and the installs are working without errors. I spent most of the week writing documentation and updating man pages, for all the little changes I made. Further I wrote a perl script that queries the ldap server every x minutes. If the server fails to reply 2 times in a row the program will install the massive /etc/passwd file. This is created every time the server does reply.
On Thursday I gave my talk about OpenBSD and OpenCon, which everyone really liked. Lots of people came into my office and asked more questions. (Especially about OpenNtp). Further there was the group Christmas Lunch to attend, which was a good laugh, as there was as much vine as you could drink.

Cern Week 20

I spent this week writing a presentation about my visit to OpenCON. As we use Linux at Cern quite a few people don't really know what OpenBSD is, so I suggested that I could hold a little talk. For this I did loads of reading about latex and beamer. (Now there is a beamer rpm for cern)
Slides will be available [here]
Further I spent my time on improving userlib and my quattor modules as the first few bugfixes had to be written. Quattor went into code freeze on Friday so I had to hurry to fix the known errors, so they could be rolled out in the next stable version. There is a code freeze every six months. There were quite a few meeting to attend as the section is currently being reorganised. This was mostly boring stuff that didn't really relate to me as I am staying with my boss.

Duplicate a line in emacs

When coding you often have to duplicate a line. This is a little script for your .emacs file to do this:

;; Duplicate it
(defun duplicate()
 "Duplicate it."
 (interactive)
 (let (
       (beg (line-beginning-position))
       (end (line-end-position)))
   (copy-region-as-kill beg end)
   (beginning-of-line)
   (forward-line 1)
   (yank)
   (newline)
   (forward-line -1)))

I love it to bits

I hate text boxes. You know these boxes you get on Gmail, twiki, etc .. where you are supposed to write your text. Then they only have limited editor functions. No delete line, No goto beginning, but this has an end now with
https://addons.mozilla.org/en-US/firefox/addon/4125
you can edit these boxes with the editor of your choice. And it is amazing. Just click the little edit button and your defined editor fires up and you can start editing in real time.

If you are stuck with twiki you should have a look at http://www.neilvandyke.org/erin-twiki-emacs/
this combination makes wiki pages actually usable

This really made me think for the whole weekend

What row of numbers comes next in this series? 1
11
21
1211
111221
312211
13112221

Cern Week 18

The first few days where spent migrating lxserv (lx = linux, serv = server) from slc3 to slc 4. Basically it was to install a new machine to do the job of the old 32bit slc3 machine. This is quite difficult as lots of people have installed little workarounds on the old machine. But now all the servers are running slc4 on 64 bit machines. Further I started writing a patch set for libuser. This is really something that has to be done to use it in production as there are lots of small errors that need fixing. On Friday I went to OpenCon the OpenBsd conference in Venice. This was really cool. Even if I didn't succeed in pushing my patch into stable.

Cern Week 17

The first few days of the week were spent trying to figure out why getpwent() kept failing. I had to do quite a lot of debugging and reading the source code of Perl to find out how the wrapper worked. At the end it turned out to be something insanely easy and stupid. What getpwent does when it realizes that you are going to loop through the whole list is it gets all the user names and then queries each one of them every time you call the function. But it kept the connection open, for the specific user query. But in the config file I had specified that no connection should be open for over 30 seconds. (As getting the whole list takes:

real 0m4.536s

So now the connection was closed but the program still tried to read data from it. So basically there was a dead lock. After setting up the limit the problem disappeared. Further Marco and Me looked into using the Coda file system for our Laptops. We have now requested a server and hopefully we can start installing next week. This should be really cool as this is a networked file system that will sync when it reconnects. So you can take your laptop home work offline and when you come back to work you can keep on working on your big work pc. I further did some research into shadow-utils and userlib. Without going into to much detail userlib is really nice. I don't really understand why so many people still use shadow-utils. I am currently lobbing for userlib to become the standard at Cern. I started thinking about disaster recovery and disaster management. I wrote a script that will run on a server and query the Ldap server every 15 minutes about it's entries then it creates the /etc/passwd, /etc/groups and /etc/shadow. So in the unlikely event that Ldap goes down and Kerberos is still up. The files can just be copied to all the machines and users can still use them.

I started to have a look at the quattor sendmail component that automatically configures the sendmail program. The syntax is really horrible of the sendmail config file. But more to come about this. While writing this I am waiting for my sendmail patches to be commit to the test cluster. Through some minor changes I reduced the run time from about 1 1/2 minutes (real 1m12.017s ) to half a second. (real 0m0.875s).

Further I attended quite a few meetings. And a talk about the new castor scheduler.

I was quite happy to hear that the average uptime is 99.73 % for the machines my department maneges.

Added some licenses

I added a GPLv3 and a BSD compatible version of the PACIFIST PUBLIC LICENSE. Please think about using it :
http://www.ribalba.de/licenses

Cern Week 16

This week my Perl script or quattor module went into production. I build the rpm and then did loads of testing on my cluster. I had to pretty much try all the different scenarios under which my program could be run (Dan: installed a machine... lots of times). There were some minor problems but now it is on swrep (the Cern software repository). Because it was such a huge change I got my own version number. Normally the counting is done 2.0.x or something like that, but because I changed so much I now started the 3.0.0 tree. I did some benchmarking on my code and found that it behaved quite linear.

http://www.ribalba.de/cern2

After some research I found out that the library function, I was calling to execute a shell command, had a sleep(1) in it. The idea behind this is to be sure that all buffers have been flushed before it returns. But this of course has a linear execution time as a result. As I don’t rely on the output I simply created a work around, but for the long run I am thinking of changing my program to be multithreaded.

Further I helped Dan to design his new Aims replacement. A system to distribute pxe images, and get the right machine to boot the right OS.

I had a look at the new Google Android mobile development SDK. It seems to be really cool. I wrote a little “hello world” program in it. Currently I am thinking about something I could write for it. Ideas will come.

Dinosaur Brains: Dealing with All Those Impossible People at Work

So another book review. After Ruth told me that my social skills needed improving I bought this book. It is really good and worth reading. It is not one of those boring, this is how your brain works type of books. It takes a different approach. More like this is the situation, this is how Mr. X responds WHY.

I can notice the difference at Work. How I see the people I work with and how I interact with them. For 3 Euro this was one of the best books I have read. Another really good point is the size. It is so small that you can easily fit it in a pocket and big enough to nicely read it.

Cern Week 15

So it is a little late, to write my weekly review, on a Sunday afternoon. I spent most of the week rewriting the Quattor module, I was already working on last week. But now it should be finished. I spent quite some time finding errors in the core libraries. The way Quattor works is actually quite simple. There is the core program that, depending on the configurations passed to it (xml), calls so called Components. These are little Perl scripts that then do the appropriate things according to the XML config file. So the error I found was that if there was an exception in one of these Components the error would propagate down the call tree and then get the core to exit. This is not a good thing as it will leave the system in an inconsistent state. I can't believe this hasn't happened before. Further I went on some further training. This time it was advanced Perl. The really nitty gritty of that horrible language (Don't get me started again). I have now been to a week worth of Perl training. What is quite nice and will look nicely on my CV, I guess. So the summary of the week would be programming Perl and learning Perl. I hope next week I can do some more research into why getpwent() keeps failing in Perl.

Buy the way my Week count is still correct. So i haven't missed writing a entry :)

Dan about tonight

> So how are we going to get drunk tonight

Well, one usually enters a drinking establishment, with money in ones
pocket. One then approaches the bar, inquires to what selection of
drinks are available and based on the reply, selects the cheapest drink
with the high percentage of alcohol. Upon delivery of the drink, one
then adjusts the angle of the glass to allow the liquid to fall down
ones throat. The above is then repeated until one can long complete any
of the above actions, either due to intoxication, lack of money or gets
lucky and takes the barmaid home.

Hey

Marco and me have been solving riddles for fun this afternoon. Here is one:
http://www.freeworldgroup.com/games/gameindex/goat.htm

And another one:

This is a classic. I really love it. It has a long description, but stick with me because it's worth it.

There are 4 people who need to cross a narrow bridge at night with only one torch.

The four people each have different travelling times:

Person 1 crosses in 1 minute
Person 2 crosses in 2 minutes
Person 1 crosses in 5 minutes
Person 1 crosses in 10 minutes

Only two people can cross at a time and one person has to come back over the bridge to give the torch to the others still waiting to cross.

While crossing, you have to use the slowest time of the two people because they have to walk the same pace.

You have to add the person who comes back to your total time.

They need to cross in 17 minutes. How?

Here's an example:

5 and 10 go over (that's 10 minutes) and then 5 comes back with the torch (that's a total of 15 minutes). Then 5 and 2 go over (that's 20 minutes---and you're already over the time limit . . . )

How can you get all four people to the other side in 17 minutes?

Note that there is a logical answer. It won't be anything like "they can throw the torch to the other side instead of walking it over", or "they can all wait till sunrise", or "why don't they jog over and increase their times", etc!


Try them, it is really good fun. Don't Google it. We didn't

Some web development tools

Hey some FireFox plugins Jay and Filipe told me about, to make web development easier:
https://addons.mozilla.org/de/firefox/addon/60
https://addons.mozilla.org/de/firefox/addon/1843

Bournemouth University email security

At Cern we are can use Microsoft Outlook Web Access©, so I thought is this as secure as at my Uni (a little sarcasm here). Yes, it is far more secure, everything is done over https (you have to use it). But because of the horrible state of security at Bournemouth I want to enlighten some people. If you access the uni mail through a web browser you get a pop up dialog similar to this

So now you think ahhh brilliant this will be nice and secure. So you enter your User Name and your password. Lets assume my name is 'r2d2' and my password is 'security'. Lets have a look at the package that is sent over the so trustworthy Internet.

Cookie: sessionid=1245b528-ae7e-4022-9300-0f580a07f33e:0x409; ASPSESSIONIDCC DRTSCS=NKDGCHNAEPBLGGFDAOHGPAHM\r\n
Authorization: Basic cjJkMjpzZWN1cml0eQ==\r\n
    Credentials: r2d2:security
\r\n

Can you spot the password. This is in plain text, I just caught the packet, with wireshark. So lets create a little scenario here. I am sitting in the Library with my laptop over wireless, I want to read my email so I log into the email server. Now someone in the reach of my wireless can sniff the package and get my password. Because this password is used all around uni he can now see everything I see, so my results (mybu), my assignments(h drive), ....
If you want you can use https but it is not enforced.
You can view the whole package here

Lol

Date: Tue, 6 Nov 2007 15:31:22 +0100
From: Luis Fernando Mu�oz Mej�as
To: project-quattor@cernNOSPAM.ch
Subject: Re: AFS usage

On Tuesday 06 November 2007 15:25, Marco Emilio Poleggi wrote:
> , such as Kerberos (I don't think, f.i., that NFS supports it, though
> I might be wrong).

NFS v4 supports Kerberos, although, you know, NFS=Not For Security. ;)

Cheers.

Cern Week 14

I came back from my Girlfriends place in Berlin on Wednesday. So I literally only worked two days. On Thursday I started fixing some Quator modules. These are basically Perl scripts that get called from the underlying framework. In particular I had a look at accounts.pm and kerberos5.pm. I created a little patch set that will hopefully be committed into production by next week. I added some error catching and some extended options. Further there were some serious bugs, like that usermod didn't check if there was a home directory and if it should be created through the 'creatHome' flag. Then I did some research in the carpool project I want to start at Cern. I am now contributing to an open source project (https://launchpad.net/carpool/) to create a nice little app, which can be used at Cern. I spent most of Friday listing to talks about projects at Cern. They were really interesting. The first one was on how to create and distribute virtual machine images around the network. The talker had invented a method where the images are hashed and then only little parts are transferred, through this he could save up to 90% bandwidth. Really intelligent stuff. Further I listened to a talk about IPMI and 'The dark side of vitalisation'. Then I started to debug a Perl script that kept crashing when using Ldap. After a few hours of debugging I found that the getpwent() method in Perl is broken. I am currently trying to fix the bug and then create a Patch.

Perl and OpenBsd

I am currently patching Perl. Here is something I found in reentr.inc
Waiting to go to OpenCon :)

/*
* As of OpenBSD 3.7, reentrant functions are now working, they just are
* incompatible with everyone else. To make OpenBSD happy, we have to
* memzero out certain structures before calling the functions.
*/
#if defined(__OpenBSD__)
# define REENTR_MEMZERO(a,b) memzero(a,b)
#else
# define REENTR_MEMZERO(a,b) 0
#endif

# Must make OpenBSD happy
my $memzero = '';

Google Apps

I still had a URL I wasn't really using (http://www.sexmitmoebeln.de) So I thought I might as well give Google Apps a try. For now it looks really sexy. You can have emails with [something]@sexmitmoebeln.de, web pages, a start page, docs, chat and so on. Really everything a small company could want. And it is for free. I think this is really useful as otherwise this would all be scattered throughout the whole firm. Now one admin can do everything over one web interface. Further, you get all the storage and backup for free and if you want to pay without adds.

Cern Week 13

This week I attended a python training that went over three days. After learning Pearl, I though it would be very useful to learn the other big scripting language. This course was extremely good. Through the interactive learning style I could grasp the concepts far quicker than, if I would have just heard about them. I really like python. I like the concept that everything is an object. And the interpreter just calls predefined methods on these objects __add__ for example. This is a nice thing between Java and C++. Through this you can easily program objects than can be added through the (+) operator. I can equally understand that some hard core object oriented programmers can’t cope with the way it handles encapsulation. My personal opinion is that it is useful for little scripts where the algorithm is the priority. But I don’t think it can be used for a very big program except when there are only exceptionally good people at work.
The rest of the week I took holidays to go to Germany and pick up my snowboard.

>>> import this

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>

Cern Week 12

This week I spent some time reinstalling a test server with different configurations. For this I did some reading on quator (quattor is a system administration toolkit providing a powerful, portable and modular tool suite for the automated installation, configuration and management of clusters and farms running UNIX derivates like Linux and Solaris). I further wrote an RPM for OpenJDK for Scientific Linux. As people are really annoyed with the Java license. But the problem is to bootstrap the OpenJDK you need proprietary software again. So we can't publish the rpm externally. So after reading I did some research about Licensing, as my course doesn't cover this in great depth. I had quite a lot to do, with getting an idea of what there is out there and what it really means. What is FSF compatible and so on. After this I decided to commit to one Licence the PPL ( THE PACIFIST PUBLIC LICENSE) my self. This seams to be the License that I agree with the most. (I suppose I could have used the BSD + some ideas, but do want my software to be published again if someone modifies it)
I did some C++ programing on a Project and some reasearch in my dash program. Further I still have my french lessons which are going really well.

ArrayList in Java

I have just ported OpenJDK to Scientific Linux and so I had a look around in how Java is implemented. You should do this. Just to see how ArrayList works or Math.Random does teach you quite a lot about Java and how compilers are designed.
Because OpenJDK still relies on proprietary Modules I cannot publish my rpm here. Sorry.

CVS rant

I have been using CVS for many years now, sometimes Subversion. And it works. It is not very fast but it works. I had a look at git a while ago and found it to be very kernel specific. Now I had a look at
mercurial
And it seams to be really sweet. One feature I really don't like in CVS is the branching. When I have an idea I want to try it in an unstable tree or an featurexyz tree. This is very hard to do in cvs because if you bug fix some stuff in the main tree you get into merge hell. So developing on two trees is a pain and I end up not doing it. So I taught git might be good because it was designed to be able to handle this very nicely. And again it does. But mercurial does it too and is more non kernel orientated.
Further I don't like the server approach CVS has. As I move around the world quite a lot having a central CVS server is not always easy. So a server less approach seams to be more suitable for my case.
I will try developing my next project in it and will report, but for now have a look at it. Many big project use it

A nice little flash app

Everyone is talking about Google generating their own OS and so on. I came across this
http://www.yourminis.com/start
It is really quite nice. And I think this is somewhat how the Google Operating System would look like if they produce one. Have a look at it. Maybe don't use it for everyday life :) Instead try IceWm.

Picasa

Lots has been said about Picasa and Flickr, and I don't want to get involved. But I use Picasa and it works.
On my windows box I have the Picasa program and it uploads quite nicely to my web page. It works with videos and it can remove red eyes. But what has been annoying me for a while is that my friends can really see when I upload new pictures to a already existing album. So I have to create a new album every time I have 3 pictures from a night out. But after some research I found that Picasa Web has a program interface. So now if you use this rss feed you should be able to see all the new pictures I upload, quite like flickr does it with the photo stream.

With Preview
http://picasaweb.google.com/data/feed/base/user/ribalba/?kind=photo

Just the name
http://picasaweb.google.com/data/feed/api/user/ribalba?kind=photo

(But please don't rely on this there is nothing like a good look at http://photos.ribalba.de/)

The Design of Everyday Things

I just finished reading this book. It sounded really good. And the first 100 (+/-) Pages are quite interresting but then the Author just slipps off in a repeating rant about the same things. If you have the frame of mind like Linda Hole and you really are interrested in what makes things useable this is a really good book and you can learn a lot. But if you just want some easy tipps it is definatelly to much. The Author really know what he is talking about. But as lost of authors tend to do, he only talks about the most impressive projects he has worked on (like Eric Steven Raymond with fetchmail, or Joel with Exel) It's a light read, and quite interresting but not really a must read like 'The Pragmatic Programmer".

New blog

I am fed up with having all my ideas scatted around the place now I created a blog for this:

http://computingideas.blogspot.com/

Rss feed it and please give me feedback.

Applications used at work

I am just reading the paper my university gave me for my placement. And I have to state on what environment I work on so here we go:
My Desktop Pc

$ uname -a
Linux pcitfio88 2.6.9-55.0.9.EL.cernsmp #1 SMP Tue Oct 2 05:33:14 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/cpuinfo | grep "model name" | head -n1
AMD Athlon(tm) 64 X2 Dual Core Processor 4600+

My development Cluster

# of hosts (down): 60 (1)
operating system(s): 2.4.21-51.EL.cernsmp, 2.6.9-55.0.6.EL.cernsmp, 2.6.9-55.0.6.EL.cern1
# of CPUs (down): 204 (4)
average up time: 10 days, 20h:38m

I use gcc languages are c, c++, perl

Cern Week 11

Monday and Tuesday where spent learning Pearl. As the my University course does some Perl but not really in depth I decided to do some training. It was 8 hours a day and quite complete. Not much to say about it really.
The rest of the week was spent in bed ill, not touching a pc.
Further my French courses started. At Cern every employee can learn French for free. As I am living in a french environment, I have to speak it all the time, I have to be able to communicate in this language. So now I have classes 6 hours a week. We are starting right at the beginning (Je suis ..). But apparentelly it gets quite hard.

cern week 10

I spent most of Monday checking some stuff in the requirements document. This was due as some people had given me more input and I had to sync the online version with my notes. Further I spent some time on dbook. I spent Tuesday with fixing the sound (alsa) in the new kernel version. We had to publish a new kernel as an exploit was found with which normal users can get root rights. Further through this we had to reinstall all Linux machines. Fun. Lxplus was down while lunch what a few people didn't really appreciate. The main machines are being drained and installed separately as the main cluster lxbatch can't just stop. Further I was trying to fix the -Waffel++ errors. Wednesday I spent converting dbook to expat. Expat is far better suited than Xerces-c for my app. Xerces is a nice xml parser but quite big and it seams heavy with lots of features I didn't use. I think I was using about 0.1% of it. Expat seams to be more what I was looking for from the beginning. I learnt quite a lot through this as Xerces and Expat have two totally different approaches to parsing. Expat is is a stream-oriented parser so parsing whereas Xerces is DOM, SAX and SAX2. On Thursday and Friday Intel came in and did a talk and training on multi- thread/core programming. This was really cool. These managers from Intel talking about the future and what Intel is going to do in the next 10(+/-) Years. Further we got to play around with the Intel tools Intel® Thread Checker / Intel® Thread Profiler and Intel® Threading Building Blocks.
On the first day they did the some theoretical stuff and some aspects of multi threaded programming. Further they did some stuff on OpenMp. Which seams to be really nice. I wrote a little program in it and it is so much easier than pthreads. Just add a comment and let the compiler do the work.

-Weffc++ (C++ only)

OK I spent the whole day today trying to find an error in my new project. I keep getting an error out of the guts of the vector library. I was convinced that I was doing somthing wrong. So after lunch and and some attemts to fix my code I decided to rewrite my Makefile because it was an horrible hack and I wanted to upload it to sourceforge. While doing this I noticed that I had added the -Weffc++ g++ flag after I had read Effective C++: 55 Specific Ways to Improve Your Programs and Designs
So for a test I just removed it and I had a clean build. So after some searching I found a nice little post that tells me that this is an known error. So that's it, one day of debugging for something they can't put in the deocumentation.

So if someone gets this error, you know now :
$man g++
When selecting this option, be aware that the standard library
headers do not obey all of these guidelines; use grep -v to filter
out those warnings.

bash-3.00$ g++ -c -g -Weffc++ `curl-config --cflags` Ui.cpp -o Ui.o
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/allocator.h: In instantiation of `std::allocator':
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:79: instantiated from `std::_Vector_base >::_Vector_impl'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:110: instantiated from `std::_Vector_base >'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:142: instantiated from `std::vector >'
Ui.cpp:45: instantiated from here
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/allocator.h:81: warning: base class `class __gnu_cxx::new_allocator' has a non-virtual destructor
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h: In instantiation of `std::_Vector_base >::_Vector_impl':
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:110: instantiated from `std::_Vector_base >'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:142: instantiated from `std::vector >'
Ui.cpp:45: instantiated from here
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:79: warning: base class `class std::allocator' has a non-virtual destructor
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:79: warning: `struct std::_Vector_base >::_Vector_impl' has pointer data members
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:79: warning: but does not override `std::_Vector_base >::_Vector_impl(const std::_Vector_base >::_Vector_impl&)'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:79: warning: or `operator=(const std::_Vector_base >::_Vector_impl&)'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h: In instantiation of `std::vector >':
Ui.cpp:45: instantiated from here
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:142: warning: base class `struct std::_Vector_base >' has a non-virtual destructor
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h: In instantiation of `__gnu_cxx::__normal_iterator > >':
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:564: instantiated from `void std::vector<_tp,>::push_back(const _Tp&) [with _Tp = std::string, _Alloc = std::allocator]'
Ui.cpp:95: instantiated from here
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: `class __gnu_cxx::__normal_iterator > >' has pointer data members
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: but does not override `__gnu_cxx::__normal_iterator > >(const __gnu_cxx::__normal_iterator > >&)'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: or `operator=(const __gnu_cxx::__normal_iterator > >&)'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h: In instantiation of `__gnu_cxx::__normal_iterator > >':
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:375: instantiated from `size_t std::vector<_tp,>::size() const [with _Tp = std::string, _Alloc = std::allocator]'
Ui.cpp:130: instantiated from here
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: `class __gnu_cxx::__normal_iterator > >' has pointer data members
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: but does not override `__gnu_cxx::__normal_iterator > >(const __gnu_cxx::__normal_iterator > >&)'
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:587: warning: or `operator=(const __gnu_cxx::__normal_iterator > >&)'
bash-3.00$

The Pragmatic Programmer. From Journeyman to Master

So after reading another book I though I might write a little review again.
This was one of the best book I have read in a long time. I am sort of sad that no one at Uni told me about this one (Frank???). It would have made my second year a whole lot easier. It is amazing how I can study for one year and then by reading a 300 page book everything becomes clear. The two authors realty now what they talk about. Through a funny writing style you don't get board and it sometimes has parallels with joel on software. Really if I would have read this before the Integrated assignment I would have been able to sleep. Everything they talk about is summarised in 70 short tips that really make sense. Further there are loads of interesting and good hints for a software developer. Buy this book and read it, NOW, Otherwise you can't say you know computing.

Cern Week 9

So Monday was spent converting LeTex to wiki and doing some final stress test and such on my server. Further I checked the failover mechanism. Which seams to work fine. Quite simple in OpenLdap to sync servers. The next day I started benchmarking client performance. Which seams to be totaly ok. This is a little test where lxb5477 authentikates over Ldap and lxb5478 uses /etc/passwd.

bash-3.00$ time ssh lxb5477 "/bin/true"

real    0m0.388s
user    0m0.013s
sys     0m0.010s
bash-3.00$ time ssh lxb5478 "/bin/true"

real    0m0.373s
user    0m0.009s
sys     0m0.014s

This is quite a nice result. Further I investigated in using a caching mechanism on the client so that not every

ls

has to do an ldap query. I tested some tools and sort of stuck with nscd (name service cache daemon) which improved performace hugely.

[root@lxb5477 ~]# /etc/init.d/nscd stop
Stopping nscd:                                            [  OK ]
$ time for i in `seq 100`; do ls ; done
real    0m1.684s
user    0m0.253s
sys     0m0.453s

[root@lxb5477 ~]# /etc/init.d/nscd start
Starting nscd:                                            [  OK ]
$ time for i in `seq 100`; do ls ; done
real    0m0.393s
user    0m0.099s
sys     0m0.254s

So I did some research on this an configured it to fit with owr cluster system.

I registred for an intel training on multi core programming which Cern is paying for me. And the good thing is that I get a fancy Certificate everyone wants nowerdays.
From : http://openlab-mu-internal.web.cern.ch
A Multi-threading and Parallelism Workshop will be held on the 4^th and 5^th of October at CERN. Experts from Intel will lead the two day event and help you improve your knowledge by explaining the key intricacies of parallel programming and presenting the most efficient solutions to popular multi-threading problems. A hands-on lab is also planned where participants will be able to improve their understanding of multi-threaded programming methodologies and learn to use tools for debugging and profiling multi-threaded applications.
So to prepare for this I started to hack up some pthreaded programs. What was quite fun. I wrote a little clone of GNU seq to use both cores of my 2 core machine. This was basically a tradeoff from memory to speed. Bacause I cach the results of one core in memory. But now I hink I can keep up with the geeks that work here.
Then I startd a little car share app, which I will talk about some other time. Then I decided to develop dbooks a little further so I can use it in my final year.

Cern Week 8

Another week is over, so what did I do. I finally decided that OpenLdap is quite fast. After one whole night of benchmarking and a network load of 100%, the server was still rock solid. Not to mention the clients (See other post). So after that I started thinking about security and did some testes with TLS, SASL and certificates. This turned out to be incredibly slow. The initial setup cost of the encryption stands in no relation to the security it offers. I further implemented a server certificate so that the server can not be changed without the clients knowing. Which seams to only work with TLS. (I have to spend a little more time here) Further I finished the migration of my development cluster to ldap. So now all machines authenticate over ldap and it should work for all people at cern.
After asking a few people about my requirements document, I was happy to realise that Bournemouth Uni does actually teach you something. Really Ruth, Frank and Peter teach you something you will really need in a production environment. The guy here at Cern that writes quite a few requirements was quite impressed to see what I had come up with and quote "It is nice to see such a clean and good requirements document from a student" One thing they don't teach you at bmth is security, what is quite important here and should be everywhere. So I have to do some private study about that.
I spent some time having to migrate my loved tex and cvs combo to a shitty little twiki interface that doesn't allow comments in your twiki make up language and crashes on you all the time. Now I am spending hours counting spaces to get the bullet points right. And it is very slow. Just klicking on save takes 5 seconds in one case and in the other it crashes. Further it has no spell checker.
I did a little work on my project (ldap bench) but not even a cvs commit worth.

/var full

Yesterday I was benchmarking my OpenLdap server. For this I used the production cluster with 210 machines. So I sshed to all machines and started my little ldap bench program in a `while [ true ]` loop and whent home. This morning I came back stopped all the jobs and had a look at my ldap server. It was still fine and I was quite happy with the outcome. This afternoon then I got and Mail from one of the cluster admins that I had shredded quite a few machines through filling up /var. This happend because every network connection is logged in /var/log/messages. So when the log wanted to roll over gzip failed because it didn't have any space, further sendmail had gone into zomby mode as it couldn't log anymore. So my first real fuck up. But at least then we knew that any user can take down the cluster through syslog. This has been fixed :)

coreutils-6.9

Marco and me had a look at the GNU coreutlis because we where fed up with some functions they did or didn't provide us. So I had a quite brows through them, now I want to comment on them :
While looking at the code I found a huge amount of FIXME comments. To be precise 119.
Just for the people that don't know coreutils are programs like ln, echo, rm, cp, mv ....
Let me list some examples:

copy.c :

/* FIXME: this should (at the very least) be moved into the following
if-block. More likely, it should be removed, because it inhibits
making backups. But removing it will result in a change in behavior
that will probably have to be documented -- and tests will have to
be updated. */

/* FIXME: shouldn't this be testing whether we're making symlinks? */

/* FIXME: this is a little fragile in that it relies on rename(2)
failing with a specific errno value. Expect problems
on non-POSIX systems. */

/* See if the destination is already the desired symlink.
FIXME: This behavior isn't documented, and seems wrong
in some cases, e.g., if the destination symlink has the
wrong ownership, permissions, or time stamps. */

/* Can't preserve ownership of symlinks.
FIXME: maybe give a warning or even error for symlinks
in directories with the sticky bit set -- there, not
preserving owner/group is a potential security problem. */


Ok this is just copy. I can't belive it. Every linux distro has this installed and then I read comments like this

head.c
/* FIXME: is this dead code?
Consider the test, pos == start_pos, above. */

test.c
/* FIXME: is this dead code? */


Now I am board.

P.S. Edd is going to be very happy. How many fixme's are there in OpenBSD?

Cern Week 7

So my main task this week was to benchmark the OpenLdap Server I set up. For this I wrote a little test suit. (It can be found under http://sourceforge.net/) I built in loads of features to do a proper stress test (Cache prevention, simultaneous thread execution(for the 4 core machines), a little data base system, etc..). I further migrated all the current user files to the Ldap server (20874 active user entries) There are actually more users but quite a few are mapped to one user name. Especially for batch jobs and groups. Like MIT has one user but many people log on on as this one user. After doing this I did some analysis on the passwd file. And had some ideas of optimising this (mainly ordering and caching). I will have to wait till next week to submit the patch set because my boss wasn't there this week. Further I wrote a little bash script that can send out an email without having anything configured through interacting with smtp over a file descriptor that is actually a tcp socket (here). I further attended a meeting and some other management stuff(Not really of any interest).
So in conclusion I programmed a hell of a lot of C, what was nice as a little refresh.

How to send a mail in bash

This is something very simple. How do you send a email in bash without having sendmail configured. Ok so I spent the whole afternoon writing this. It basically creates a file descriptor over a tcp socket and then sends data over it. Not really error robust, but it should do. I have to sleep all the time as I don't know when the server will respond. And I can't cat,tail or head the filedescriptor as I don't have a EOF. Anyway quite horrible hack, but maybe you can learn something from it. For example how smtp works :)

 1 #!/bin/bash
 2 #
 3 # A little script that sends out a mail purly in Bash :)
 4 # Very very slow act
 5 # {Jan.Michael,Geerd-Dietger.Hoffmann}@cern.ch
 6 #
 7 
 8 #set up some vars we need
 9 nameofme=`uname -n`
10 ipofmail=`host cernmxlb.cern.ch | cut -d ' ' -f 4` #Have fun if dhcp goes down
11 
12 #TODO : Something like if [ `ping -c 1 $ipofmail | wc -l` -eq 5];then echo "host down"; fi
13 
14 #Create file pointer
15 exec 4<>/dev/tcp/$ipofmail/25
16 sleep 2
17 echo -en "EHLO $nameofme.cern.ch\r\n" >&4
18 sleep 5
19 echo -en "MAIL From:<you@root.ch> SIZE=770\r\n" >&4
20 sleep 2
21 echo -en "RCPT To:<you@root>\r\n" >&4
22 sleep 2
23 #And off we go
24 echo -en "DATA\r\n" >&4
25 sleep 1
26 echo -en "Nobody said computers were going to be polite.\r\n" >&4
27 sleep 1
28 echo -en ".\r\n" >&4
29 sleep 1
30 echo -en "QUIT\r\n" >&4
31 
32 # Disable this if you don't wan't output
33 # Could be used for error checking something like this would be ok
34 # if [! `wc -l <&4` -eq 22];then echo "error in line count"; fi
35 sleep 1
36 cat <&4
37

Note: This of course will only work if you can reverse DNS and own the world

C rand function

Just for completeness here the c rand function
#include stdlib.h
as I compared the Java one in August:

A nother window manager

If you have followed my Blog you know that Edd and me made a deal in switching to another window manager, So as I started my placement I thought OK next one. (After using enlightenment)
I decided to try JWM (Joe's Window Manager) this is a nice little wm. It works really well has nearly no dependencies and looks really good. It has all the nice little features a wm should have and is really nicely configurable. After using it for a while I encountered a few problems though. Trough grabbing the modifier of a key right away shortcuts in programs tend not to work. (So if I press CTRL-S for save) this will not work as I used CTRL for my special functions.
nextstacked
Remapping this didn't work either. I tried to rewrite the code but then noticed that this is a little too much to do while at work. Further I found out what the hyper key is :)
So now I installed ice wm to try the next one.

Have a look at this

Ldap benchmark

A little program I wrote to test the ldap servers.
http://sourceforge.net/projects/ldaptest/
Please have a look at it.

Cern Week 6

So this week was a little bit stupid as I managed to pull a muscle and not beeing able to go to work. Otherwise I went to Berlin on the Weekend to see Juanan which was really nice. But workwise I summarized some emails and put them up on twiki

sendmail automation script

Here a little script that will send a email to some people out of a bash script. Not very hard.

 1 #!/bin/bash
 2 from="root@microsoft.com"
 3 emailtarget="root@sun.com, root@ibm.com"
 4 subject="What do you want"
 5 sendmail=/usr/sbin/sendmail
 6 
 7 
 8 CONTENT="
 9 Really what you want to put here
10 ----
11 "`fortune`
12 
13 msgdate=`date +"%a, %e %Y %T %z"`
14 daemail=$(cat <<! 
15 Date: $msgdate
16 From: $from
17 To: $emailtarget
18 Subject: $subject
19 Mime-Version: 1.0
20 X-Mailer: Didi Mail
21 
22 $CONTENT
23 
24 !)
25 
26 echo "$daemail" | $sendmail -t

Cern week 5

So tomorrow I am moving into my new flat so blog entry today. I spent the week benchmarking OpenLdap and writing a tool for this. I am planning to publish it under sf when I resolved the memory leek. I wrote it in c as I felt like I should take this up again. Now I remember why I stopped using it for every day programming. I spent one whole afternoon in ddd and valgrind to find that the connect_to_host ldap system call was pointing to an empty struct. This is of course not documented when you encounter the error. But when you have found it and you know what to search for you can find it everywhere. And it can be resoled through just letting it point to NULL and not to the struct.
Further I moved office on Monday. Now I am in a nicer bigger office with a coffee machine right next to me (Photo).
I registered for some Perl lessons with certificate so at the end of my placement I am a certified Python and Perl programmer what can't hurt. After winter I want to do the same with C++ and maybe some Linux sysadmin. I have to talk to Jan about this once time has come. That's about it. The flat I am moving in tomorrow is really nice. Quite big (40m^2), in the middle of town and fully furnished what is something very hard to find.

Some notes on slapd.conf:
If you want to run a Ldap server some hints.
Set this to the max as otherwise the values are far to low
conn_max_pending_auth 65435
conn_max_pending 65435

By default this is off but to be on the safe side
reverse-lookup off

This is not really required but it can't hurt
sizelimit unlimited

If you are benchmarking this will stop giving you the timeout errors. In production I would take this out.
timelimit unlimited

Normally there are 16 threads. For a shared server this might be ok but not if I have the server only for ldap.
threads 50

Just adding thees lines made my Ldap server respond 500 times quicker.
Standard config about 20 queries a second.
New config 1000 queries a second.