Wed, 10 Jun 2009

Upgrading Python with Portmaster

If you are upgrading python on FreeBSD to 2.6 the steps as they stand now require portupgrade to be installed. This is based purely on the usage of pkg_which when walking /usr/local/lib/python$ver to determine what ports to rebuild.

Last night I came up with a patch that allows one to use portmaster instead of portupgrade. It's slower because it uses pkg_info. My guess is that pkg_which is using the pkgdb used by the portupgrade suite, which makes it significantly faster then how pkg_info works. We try to parallelize the processing but it's still slow - IMO we could greatly increase the amount of parallelism going on but I'm not going to push it too far.

The patch is available at my FreeBSD space and will apply cleanly to the latest lang/python port. I intend to commit it tomorrow morning while I'm eating breakfast but if you want to test it out now just do the following: 

cd /usr/ports/lang/python
fetch - http://people.freebsd.org/~wxs/python26-portmaster.diff | patch
portmaster -o lang/python26 lang/python25
make upgrade-site-packages -DUSE_PORTMASTER

posted at: 12:45 | tags: , , | path: /entries/freebsd | permanent link to this entry

Tue, 02 Sep 2008

Bug Found and Fixed in fastest_sites.

There was a bug in fastest_sites which I just committed a fix for. The idea for the fix came from Ryan Steinmetz, though the patch has a couple smaller additions by me to make it slightly more friendly.

The bug is because there are things like this in Mk/bsd.sites.mk: 

.if !defined(IGNORE_MASTER_SITE_DEBIAN_POOL)
MASTER_SITE_DEBIAN_POOL+=       \
	${MASTER_SITE_DEBIAN:C|(/%SUBDIR%/)|/pool/main/${PORTNAME:C/^(.).*$/\1/}/${PORTNAME}/|}
.endif

...

.if !defined(IGNORE_MASTER_SITE_GOOGLE_CODE)
.if defined(PROJECTHOST)
MASTER_SITE_GOOGLE_CODE+= \
	http://${PROJECTHOST}.googlecode.com/files/
.else
MASTER_SITE_GOOGLE_CODE+= \
	http://${PORTNAME}.googlecode.com/files/
.endif
.endif

The way fastest_sites works is by doing 'make -V MASTER_SITE_FOO -f $PORTSDIR/Mk/bsd.sites.mk' for every MASTER_SITE definition in the file. It then splits the result and makes the connections to everything (not technically true, it only does the first 10 which respond) in the list in order to get the RTT estimate for a TCP handshake. The problem is that ${PORTNAME} is not defined resulting in: 

wxs@ack ~ % make -V MASTER_SITE_GOOGLE_CODE -V MASTER_SITE_DEBIAN_POOL -f /usr/ports/Mk/bsd.sites.mk
http://.googlecode.com/files/
http://www.gtlib.cc.gatech.edu/pub/debian/pool/main/// ftp://ftp.us.debian.org/debian/pool/main/// ftp://ftp.au.debian.org/debian/pool/main/// ftp://ftp.bg.debian.org/debian/pool/main/// ftp://ftp.cl.debian.org/debian/pool/main/// ftp://ftp.cz.debian.org/debian/pool/main/// ftp://ftp.de.debian.org/debian/pool/main/// ftp://ftp.ee.debian.org/debian/pool/main/// ftp://ftp.es.debian.org/debian/pool/main/// ftp://ftp.fi.debian.org/debian/pool/main/// ftp://ftp.fr.debian.org/debian/pool/main/// ftp://ftp.hk.debian.org/debian/pool/main/// ftp://ftp.hr.debian.org/debian/pool/main/// ftp://ftp.hu.debian.org/debian/pool/main/// ftp://ftp.ie.debian.org/debian/pool/main/// ftp://ftp.is.debian.org/debian/pool/main/// ftp://ftp.it.debian.org/debian/pool/main/// ftp://ftp.jp.debian.org/debian/pool/main/// http://ring.nict.go.jp/archives/linux/debian/debian/pool/main/// http://ring.sakura.ad.jp/archives/linux/debian/debian/pool/main/// http://ring.riken.jp/archives/linux/debian/debian/pool/main/// ftp://ftp.nl.debian.org/debian/pool/main/// ftp://ftp.no.debian.org/debian/pool/main/// ftp://ftp.pl.debian.org/debian/pool/main/// ftp://ftp.ru.debian.org/debian/pool/main/// ftp://ftp.se.debian.org/debian/pool/main/// ftp://ftp.si.debian.org/debian/pool/main/// ftp://ftp.sk.debian.org/debian/pool/main/// ftp://ftp.uk.debian.org/debian/pool/main/// ftp://ftp.wa.au.debian.org/debian/pool/main/// ftp://ftp2.de.debian.org/debian/pool/main///
wxs@ack ~ %

The results are obviously bad: http://.googlecode.com is not legal, and while the seecond result is a legal URL it is not correct in the context of ports - it's missing the piece about the PORTNAME.

The patch looks for results that contain '//.', '[a-zA-Z]//' or '..' as those are the common cases. Technically the '..' case doesn't exist yet but it's not a far stretch to see a MASTER_SITE_FOO definition that uses http://www.${PORTNAME}.mirror-network.com in the future. We might as well catch that case now.

I committed the patch tonight, thanks again Ryan, and notified Jordan (he's the author of fastest_sites) about some other minor issues with it. I'd fix them myself but my python-fu is very weak and I just don't have the time since I'm moving in a week. Hopefully Jordan will get around to it once he's back from his honeymoon.

posted at: 22:46 | tags: , , | path: /entries/freebsd | permanent link to this entry

Mon, 24 Mar 2008

Fastest Sites is Committed.

Coming to a portsnap (or c[v]sup) mirror near you is something I just committed tonight. It's actually really small but quite useful and written by Jordan Sissel called fastest_sites.py. Basically, for every MASTER_SITE_foo in bsd.sites.mk it will run through the list and sort it based upon RTT for a TCP handshake (which is the best guess for least effort). The sorted list is output'ed in such a fashion that it can be included into your make.conf (.include "/path/to/output" is best).

It's quite useful for people who want to control the MASTER_SITE_foo closely. It's available at ports-mgmt/fastest_sites.

posted at: 21:42 | tags: , , | path: /entries/freebsd | permanent link to this entry

Wed, 26 Apr 2006

Tiddlywiki With a Python Backend

I use TiddlyWiki for all my note taking/TODO list management at work and at home. Once I got my head wrapped around the concept I found it to be very suitable for what I need. While poking around the various TiddlyWiki community websites I stumbled across a TiddlyWiki adaptation to a Python backend (PYTW). This allows you to put the TiddlyWiki on a server and use it as a collaboration type of software. It's a neat idea, so I ported it. The port works, the TiddlyWiki itself works, but the problem is that it's impossible to get the pkg-plist right. The inital port is correct, but PYTW writes things to %%PYTWDIR%%/store/ with user-defined names (I wonder how it handles a title of ../../../../../etc/passwd, but that's a different story), which means I can never possibly predict what the files will be. To this end I won't be submitting the port, but it's still available at the link posted earlier if you want to use it.

Another thing wrong with it is that .pyc files are generated, so I'd have to add those to the pkg-plist. I suppose one solution would be to patch the code to put things in a more sane place (/var/db/pytw for the tiddlers) but I don't really care to do that. The stand-alone TiddlyWiki works very well for my note taking needs, I'm not going to use PYTW for anything even though it is very cool.

posted at: 18:24 | tags: , | path: /entries/geek | permanent link to this entry