Https not supported
I am currently trying to map all http pages to https pages. As the network providers are now investigating into giving advertisement based on your traffic. So I starting to encrypt all my traffic. While this is find quite a few pages that don't have https or map to http. The Google https start page for example just redirects to the http page. I don't really like this approach. Why don't the web page providers just enable https. I am aware that this will create a little bit more load on the servers but today there are many hardware encryption cards that will take away the load from the CPU. So I will still use Tor. When I get time I will start writing a crawler that checks a few websites if they enable https.
atime
Quite a few times now I had to time a program and take the average execution time. I normally did this with a little bash loop and then used a rough estimate or gnuplot to tell me the average runtime. Now I wrote a little python script that does exactly that. So you give it a number of loops and the program to execute.
for now you have to put the commands in quotes so it will pass in the whole command. Further be carefull not to loop to many times. Error catching works on a very rudenmental level.
So if for your next assignment you need to take the average value of a program run time try atime.
[DOWNLOAD]
$ ./atime.py 4 "sleep 0.5"
real 0.5
user 0.0
sys 0.0
for now you have to put the commands in quotes so it will pass in the whole command. Further be carefull not to loop to many times. Error catching works on a very rudenmental level.
$ ./atime.py 4 "xyz"
A error in the command has happend
/bin/sh: xyz: command not found
So if for your next assignment you need to take the average value of a program run time try atime.
1 #!/usr/bin/python
2 # A little tool that takes in a count and a command to
3 # run and averages the execution time
4 # ribalba@gmail.com
5
6 import sys;
7 import string;
8 import os;
9
10 av_real = av_user = av_sys= 0
11
12 looptimes = string.atoi(sys.argv[1])
13
14 for i in xrange(looptimes):
15 commandout = os.popen3("time -p " + sys.argv[2])
16 sys.stdout.write ( commandout[1].read(),);
17 steddout = commandout[2].readlines()
18 if (len (steddout) > 3):
19 sys.stderr.write("A error in the command has happend\n")
20 sys.stderr.write(steddout[0] + "\n")
21 sys.exit()
22 av_real += (string.atof(steddout[0].split()[1]) / looptimes)
23 av_user += (string.atof(steddout[1].split()[1]) / looptimes)
24 av_sys += (string.atof(steddout[2].split()[1]) / looptimes)
25 else:
26 sys.stderr.write("real " + str(av_real) + "\n")
27 sys.stderr.write("user " + str(av_user) + "\n")
28 sys.stderr.write("sys " + str(av_sys ) + "\n")
[DOWNLOAD]
Cern Week 42
Most of the week was spent with the porting effort to SLC5. I tried fixing a bug in nscd that created an error when called two times after each other
This was due to a programming error in nscd that sent a SIGTERM to all its child processes but didn't wait for the return val. After quite a lot of research I just replaced the restart with a reload and it works. Further work went into the creation of SELinux rules so I don't get the horrible acv: denied errors in dmesg. Then I had to go to my Univeristy for the placement day. Which I will not comment on.
$ /etc/init.d/nscd restart; /etc/init.d/nscd restart
Stopping nscd: [ OK ]
Starting nscd: [ OK ]
Stopping nscd: [FAILED]
Starting nscd: /usr/sbin/nscd: already running
[FAILED]
This was due to a programming error in nscd that sent a SIGTERM to all its child processes but didn't wait for the return val. After quite a lot of research I just replaced the restart with a reload and it works. Further work went into the creation of SELinux rules so I don't get the horrible acv: denied errors in dmesg. Then I had to go to my Univeristy for the placement day. Which I will not comment on.
Another pain with google
I come to work and normally I start reading my emails and my rss feeds. As I already commented gmail is becomming soooo slow, so I started reading my mail in pine again. Now Google Reader stopped working. And my friends tell me that my photos on Picasa Web take about one hour to load. So I will be migrateing all my rss feeds to somewhere else. Goolge is stedely loosing me. Here a little break down what I used to use from google.
Mail => moved to pine : gmail to slow
Blog = > moving to wordpress : more functionality, better service, spellchecking works
Pictures => looking for something new : picasa to slow
Webpage => wordpress : Editor stopped working in Firefox
Rss Reader => looking for something new : Can't read my feeds
Calendar
iGoogle => switched off because of security conzerns
Code => back to sf : sf gives me ssh and a far better service
And quite a few more. So I used to have my whole life in google. And now I am noticing that I am moving away. A shame really google seamed such a nice company.
Looking for a replacement for gmail
I have been using gmail for quite some time now. And it is becoming worse by the month. Now they have a loading screen with a funky little loading bar. Besides it becoming so slow that a search for a word in all my mails takes about 10 seconds now I have to wait for it to load. I remember that's when I stopped using Thunderbird. So Google get your act together and remember 'every second counts' 
Why open source does not work (sometimes)
Two friends and Me are currently starting a company. I can't say too much about the idea but it will be a traffic management online system. So I am off coding lots of nights. So the question was if we should open source the code. I normally publish everything I do under the BSD or GPLv3 license. I wouldn't say I am a Open Source activist but I try o keep my computer blob free. So now I am trying to sell a product. How would this work if it would be Open Source. Let's have a look at a successful open Source company that is not in the service business. So a company where 20 people sit in a room and code and someone gives them money for something they can have for free. Honestly I can not think of one company that sells a product and is open source. If you have a look at Google. They would never open source theire algorithm, if they would Microsoft would come along and build exactly the same search engine and give you 1 Cent if you search with it. Making money and open sourcing your code seam to be impossible if you are selling a product. It is brilliant if you are a service provider and you get your money from selling manuals or such, but if you are a little start up like us the code base is the only asset you have. Giving this away for free would ruin the whole project. So I feel quite sad to say that the next big thing I am coding will be closed source. Not because I believe in the closed source idea but because I will have to maybe live of this. And I can't see anyone giving us money for something that they could have for free.
While thinking about this I asked my self why is facebook, youtube and gmail the whole web 2 not Open Source. They all build on free software (mainly Linux I suppose) but still I would love to see the source code of gmail and find out why its sooooo slow. (See next post) The Google idea is that gmail will pay through adverts. But if there would be an open source version I would host it at home and not put up with that slow advert full version. So even a service has to be closed source sometimes to close in there customers to a product.
While thinking about this I asked my self why is facebook, youtube and gmail the whole web 2 not Open Source. They all build on free software (mainly Linux I suppose) but still I would love to see the source code of gmail and find out why its sooooo slow. (See next post) The Google idea is that gmail will pay through adverts. But if there would be an open source version I would host it at home and not put up with that slow advert full version. So even a service has to be closed source sometimes to close in there customers to a product.
Cern Week 41
This week I spent most of the time with the migration effort to SLC5. This can be described in 3 main tasks.
- componenet porting
- certificate porting
- template porting
While porting the certificates I found a bug in openssl. If you did not specify any CApath it would not look into the standard location but if you set CApath to /dev/null if would find nothing in /dev/null and then look into the sandard certs direectory. Jan Iven sent a mail to Red Hat and hopefully this will be fixed upstream. I created a patch / workaround for the time beeing. Further we inforce template checking now. So I found loads of templates that did not conform with the appropriate schema. Further programs in SLC5 behave differntlelly as in SLC4. For example the autconfig tool does not set host and port anymore it uses the uri command. Stuff like this is quite annoying because you spend loads of time debugging something stupid like this. But finally we can run quattor and all the components I am responsible for on SLC5. Further zuul is now the default for all SLC5 so it has moved from "userspace" (the user includes pro_service_zuul) to the base tpl. So every quattor machine now uses zuul.
- componenet porting
- certificate porting
- template porting
While porting the certificates I found a bug in openssl. If you did not specify any CApath it would not look into the standard location but if you set CApath to /dev/null if would find nothing in /dev/null and then look into the sandard certs direectory. Jan Iven sent a mail to Red Hat and hopefully this will be fixed upstream. I created a patch / workaround for the time beeing. Further we inforce template checking now. So I found loads of templates that did not conform with the appropriate schema. Further programs in SLC5 behave differntlelly as in SLC4. For example the autconfig tool does not set host and port anymore it uses the uri command. Stuff like this is quite annoying because you spend loads of time debugging something stupid like this. But finally we can run quattor and all the components I am responsible for on SLC5. Further zuul is now the default for all SLC5 so it has moved from "userspace" (the user includes pro_service_zuul) to the base tpl. So every quattor machine now uses zuul.
Cern Week 40
This week was spent partly at hepix
From hepix.org
Or I was preparing my presentation. Slides can be found here
Then on Thursday I gave my talk on 'The problem of managing 236 million user accounts'. This was a 30 min talk and I think it went quite well some people asked some interesting question so it was a success.
I listened to quite a few presentations but I didn't really learn as much as would have hoped. No further comments otherwise insults would be unavoidable.
The other main task for the week was to get good benchmark results of my implantation. I started looking for a time utility that would run a command n times and then give me the average run time but I could not find something like that. I had always done this with a horrible bash script, but this was now to inflexible as I could not pipe the output into a file that I then could parse with g
nuplot and so on. So I started the atime project. The basic idea is that you call it like time just with a number as first parameter. It will then run the command n times and output exactly like the standard *nix time utility would. Next post will have a full description. I wrote the tool in python because I didn't want to loose my python skills in favor of horrible Perl. I still don't fully understand why people use Perl for stuff like this. It's like hammering in a nail with the end of a screwdriver.
From hepix.org
The HEPiX forum unifies IT system support engineers from the High Energy Physics (HEP) laboratories and institutes, such as BNL, CERN, DESY, FNAL, IN2P3, INFN, JLAB, NIKHEF, RAL, SLAC, TRIUMF and others. The HEPiX meetings have been held regularly since 1991, and are an excellent source of information for IT specialists. That's why they enjoy large participation also from the non-HEP organizations.
Or I was preparing my presentation. Slides can be found here
Then on Thursday I gave my talk on 'The problem of managing 236 million user accounts'. This was a 30 min talk and I think it went quite well some people asked some interesting question so it was a success.
I listened to quite a few presentations but I didn't really learn as much as would have hoped. No further comments otherwise insults would be unavoidable.
The other main task for the week was to get good benchmark results of my implantation. I started looking for a time utility that would run a command n times and then give me the average run time but I could not find something like that. I had always done this with a horrible bash script, but this was now to inflexible as I could not pipe the output into a file that I then could parse with g
nuplot and so on. So I started the atime project. The basic idea is that you call it like time just with a number as first parameter. It will then run the command n times and output exactly like the standard *nix time utility would. Next post will have a full description. I wrote the tool in python because I didn't want to loose my python skills in favor of horrible Perl. I still don't fully understand why people use Perl for stuff like this. It's like hammering in a nail with the end of a screwdriver.
Word 2007 cool feature
If you press Alt in Word these little info boxes will appear to help you navigate to the correct menu. This is really cool . As I could never remember the shortcuts anyway. Now it tells me them. Well done Microsoft.
Change return values in nss_ldap
After some time searching I found the nss_override_attribute_value config file parameter for /etc/ldap.conf. This has the effect that you can overwrite the ldap return value through a standard value. So as an example:
NOTE: the nss_override_attribute_value has to come before the nss_map_attribute in the config file.
Further there is the nss_default_attribute_value entry which can be quite usefull for the shell for example.
nss_override_attribute_value unixHomeDirectory /varThis will change the home directories of all users returned through ldap to /var.
NOTE: the nss_override_attribute_value has to come before the nss_map_attribute in the config file.
Further there is the nss_default_attribute_value entry which can be quite usefull for the shell for example.
Cern Week 36
I spent most of the week writing a paper for a meeting I am organizing. This will be the 'Cern ldap meeting'. The idea behind this is to get all the people who will be involved with the migration, maintenance and security to give talks and brainstorm about different aspects. I am writing the paper and the slides in docutils, about which I will rant in another blog entry, but for now, I really like the concepts behind it. Further there is a flew going around and so I was at home for two days.
Cern Week 35
I spent the beginning of the Week trying to get the Cern CA to work on all machines. This was not as easy as I thought it would be, but now I have a rpm package that can be install and that will verify the certificates I get from the Active Directory Server. This will avoid man in the middle attacks. Further I extended ncm-authconfig to automatically modify /etc/ldap.conf to what I need it to be. While doing this I did some general module cleanup and added 13 new tpl options. After some testing I can now happily say that the Linux machines will securely, essentially authenticate against an AD server and use Quattor to configure themselves. Further I attended another group meeting.
One other big problem was that the AD server only returns 1024 entries, but this is of course not the total user number. So when I call getpwent it will only return the first 1024 entries and forget about 21086 users. I solved this through trying to get '--enable-paged-results' added to the spec file as compile option. But this will have to be done by red hat. I suppose this is going to take some time, till this goes upstream. But getpwent should not be used anyway :). Further I updated my passwd and group create script to acquire for this and enable paging in ldapseatch '-E pr=2000/noprompt'. Maybe a little off topic I am going to LinuxTag, Europe's leading exhibition on Linux and Open Source from May 28 - 31, 2008 at Berlin's Messezentrum unter dem Funkturm. So if you want to join should be good fun.
One other big problem was that the AD server only returns 1024 entries, but this is of course not the total user number. So when I call getpwent it will only return the first 1024 entries and forget about 21086 users. I solved this through trying to get '--enable-paged-results' added to the spec file as compile option. But this will have to be done by red hat. I suppose this is going to take some time, till this goes upstream. But getpwent should not be used anyway :). Further I updated my passwd and group create script to acquire for this and enable paging in ldapseatch '-E pr=2000/noprompt'. Maybe a little off topic I am going to LinuxTag, Europe's leading exhibition on Linux and Open Source from May 28 - 31, 2008 at Berlin's Messezentrum unter dem Funkturm. So if you want to join should be good fun.
Cern Week 34
So I have been slacking my blog. I haven't posted anything decent in the last few weeks, mainly due to the fact that I have not been doing much interesting stuff. This week was spent trying to get Linux to authenticate against active directory. There were loads of problems. The biggest was a problem with the certificates. The AD server was load balanced through a round robin dns switch. So the host was dc.cern.ch but the actual dns resolve lead to dc3.cern.ch so the certificate didn't match. This took quite a while till I had made all the changes to have a second host line in the certificates. Further I had some problems with getting the cern ca certificate on all my machines. After some work I found out that the rpm was broken. Brilliant. Then I found out that AD can only return 1024 entries at max what caused another error as getpwent failed. This is due to getpwent requesting all users at once (26000 in AD) returned only 1024. So lots of problems with Linux windows communication, but now it seams to work.
ldap_open and ldap_init with Active Directory
All my ldap programs have till now used the OpenLdap server. As we are now migrating to the Ldap fronted of Active Directory I had to migrate quite a few c programs. But here is the problem:
If I use ldap_init it will fail and shout at me
If I use ldap_init it will fail and shout at me
ldap_simple_bind_s: Can't contact LDAP serverbut if I use ldap_open it all works. No idea why this is happening. But ldap_open is deprecated so lots of fun coming up. That's what you get for writing c programs.
Makes Java a little nicer
If you program Java you have to know this http://findbugs.sourceforge.net/
Subscribe to:
Posts (Atom)
Posts
