Upgrading the cluster recently has brought nothing but trouble, with seeming conflicts between python-devel and python. Hacking and slashing through this on the UI I have now realised why. The conflict comes not from the x86_64 python RPM, but from the extra i386 python RPM we install to provide a 32bt python (needed by LCG modules like the python lfc plugin).
To avoid having to present the whole i386 repo to YUM, we'd picked out a few choice i386 packages and dropped them into the local cluster repo. However, in the meantime the i386 python had been updated and our version remained old and stale - causing the unsatisfiable dependency.
I have now updated and rebuilt the repo and things seem to upgrade smoothly at last.
However, roll on ye native 64bit middleware. This is too much of a pain at times...
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts
Friday, February 08, 2008
Tuesday, September 25, 2007
SL4 x86_64 UI now Availiable
I reinstalled the site's svr020 UI on Saturday. This involved an incredible amount of pain related to the bizarre inability of SL4 to properly install GRUB on a linux software RAID partition. Although the machine would install absolutely fine, on reboot it would just halt after the GRUB prompt.
In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an
That finally gave me a base SL4 install I could work with.
After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).
Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)
Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).
In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an
That finally gave me a base SL4 install I could work with.
After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).
Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)
Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).
Wednesday, September 19, 2007
Yarr! Thats not line noise, it's Perl (me hearties)
Avast Ye Salty Sea-dogs - It be the glasgow upgrade day today (as well as international talk like a pirate day). Graeme gave me a minor task to do - come up with a new userid/groupid/passwd/shadow/yaim config generator script for the pool accounts. Simple enough, should only take half a day or so....
Much swearing at Python later, I gave up at 1AM this morning and resorted to trusty Perl. Done in 48 lines (including comments) and 2 hours. I think Python and I are going to take a looooong time to get aquainted properly.
I'm sure G will blog in more detail - Worker nodes went fairly smothly, few niggles - seems that about 24 simultaneous installs cause tftp timeouts. Discovered the Sandbox dirs on the RB needed their ownerships changing - again another Perl script to the rescue with the trick I learned from Steve Andrews - just get your script to print the command line you'd like to stdout, then once you've checked it looks reasonable, run it again piped through | /bin/sh
Much swearing at Python later, I gave up at 1AM this morning and resorted to trusty Perl. Done in 48 lines (including comments) and 2 hours. I think Python and I are going to take a looooong time to get aquainted properly.
I'm sure G will blog in more detail - Worker nodes went fairly smothly, few niggles - seems that about 24 simultaneous installs cause tftp timeouts. Discovered the Sandbox dirs on the RB needed their ownerships changing - again another Perl script to the rescue with the trick I learned from Steve Andrews - just get your script to print the command line you'd like to stdout, then once you've checked it looks reasonable, run it again piped through | /bin/sh
Thursday, May 24, 2007
Users: The Good, The Bad and The Ugly...
Last Thursday I went to see a few potential ScotGrid users. There are a couple of users in Civil Engineering who'd like to use the cluster. One of them has monte carlo code and this should be no problem (the good!) Another has a real need for MPI code (the bad!), which is challenging, but we obviously hope to build on the good work done here in GridIreland (see their wiki).
Finally, I spoke to a postgrad student in IBLS (Institute of Biomedical and Life Sciences) who has a lot of code to run on protein data (don't really understand the problem, actually). Now, she's got to run code she hasn't written herself, and neither does she have the source code and the damn thing asks questions in an interactive mode before it runs. (The ugly!)
On the computing service cluster she has to run the code in PBS interactive mode, in order to answer the questions. Then she has to hope that the network stays up between her machine and the cluster, because if it goes down (and often it does) it kills her job.
The first thing I did was show her how to use screen, so that at least she can run in a detached terminal. (I found a good tutorial.)
Last night, I started to write a python wrapper for the program which will take a defined set of default options, which can then be overridden on the command line, and pass them to the program. This turned out to be quite troublesome - using the python Popen3 class (pipe open) I just could not get the read() or readline() methods to behave properly with the select() call (even in nonblocking mode). select() would return a filehandle with data only once; and neither did the write() call seem to push the output properly into the program. However, when I switched to using the os.read() and os.write() calls, instead of the class methods, things started to work exactly as I expected. Hopefully this will run the program properly in a non-interactive mode and open up the possibility of running on our cluster.
Of course, there's a different track to this as well - we urgently need to get VOs setup for these users so that they can use edg-job-* commands or ganga to run their jobs. Watch this space...
Finally, I spoke to a postgrad student in IBLS (Institute of Biomedical and Life Sciences) who has a lot of code to run on protein data (don't really understand the problem, actually). Now, she's got to run code she hasn't written herself, and neither does she have the source code and the damn thing asks questions in an interactive mode before it runs. (The ugly!)
On the computing service cluster she has to run the code in PBS interactive mode, in order to answer the questions. Then she has to hope that the network stays up between her machine and the cluster, because if it goes down (and often it does) it kills her job.
The first thing I did was show her how to use screen, so that at least she can run in a detached terminal. (I found a good tutorial.)
Last night, I started to write a python wrapper for the program which will take a defined set of default options, which can then be overridden on the command line, and pass them to the program. This turned out to be quite troublesome - using the python Popen3 class (pipe open) I just could not get the read() or readline() methods to behave properly with the select() call (even in nonblocking mode). select() would return a filehandle with data only once; and neither did the write() call seem to push the output properly into the program. However, when I switched to using the os.read() and os.write() calls, instead of the class methods, things started to work exactly as I expected. Hopefully this will run the program properly in a non-interactive mode and open up the possibility of running on our cluster.
Of course, there's a different track to this as well - we urgently need to get VOs setup for these users so that they can use edg-job-* commands or ganga to run their jobs. Watch this space...
Subscribe to:
Posts (Atom)