Thu, 24 Dec 2009
CFT: Sudo Update
If you use Sudo on FreeBSD and want to test out an update for it please apply the patch from here and rebuild/reinstall the port. I'm especially interested in environments using ldap, kerberos and newer releases of FreeBSD (specifically 8.0). I'll take reports of success or failure for anything though. Since this is such a heavily used port I want to make sure I get it correct and cause as little headache for myself and others as possible.
posted at: 14:02 | tags: sudo, freebsd, ports | path: /entries/freebsd | permanent link to this entry
Fri, 17 Jul 2009
My, How They Grow...
I just made the commit to note that my mentee, Steven Kreuzer, is flying solo. He's been a pleasure to work with and I know will go forth and do great things. Keep up the good work Steven!
posted at: 09:30 | tags: freebsd | path: /entries/freebsd | permanent link to this entry
Sat, 20 Jun 2009
What is really meant by "STABLE" branches in FreeBSD.
A question I am routinely asked by people is what is the latest stable FreeBSD release. People are often tripped up by the fact that there are usually two releases listed on the front page of FreeBSD.org. The best way to answer that question is to explain how branches are done in FreeBSD. The question almost always comes from people who don't understand the details so I'll do my best to explain them here.
Perhaps it's best explained in picture form (somewhat simplified for explanation purposes):
---V-------------------------------V----------------------> CURRENT
\-V------V----V-----------------\---> RELENG_6
\------\----\--> RELENG_6_0 \
\----\--> RELENG_6_1 \
\----> RELENG_6_2 \---V---V-------> RELENG_7
\---\---> RELENG_7_0
\------> RELENG_7_1
Tag: | Meaning:
. | CURRENT
RELENG_6 | 6-STABLE (at the time of fetching)
RELENG_6_0 | 6.0-RELEASE-WITH-PATCHES-FOR-SECURITY-AND-MAJOR-ERRATA
RELENG_6_1 | 6.1-RELEASE-WITH-PATCHES-FOR-SECURITY-AND-MAJOR-ERRATA
RELENG_6_2 | 6.2-RELEASE-WITH-PATCHES-FOR-SECURITY-AND-MAJOR-ERRATA
RELENG_7 | 7-STABLE (at the time of fetching)
RELENG_7_0 | 7.0-RELEASE-WITH-PATCHES-FOR-SECURITY-AND-MAJOR-ERRATA
RELENG_7_1 | 7.1-RELEASE-WITH-PATCHES-FOR-SECURITY-AND-MAJOR-ERRATA
There is a lot going on here so we'll start with the terminology. In FreeBSD, "CURRENT" is the very tip of the tree, "STABLE" is a branch off of that tree, and "RELEASE" are branches off of the STABLE line. In FreeBSD there are multiple STABLE branches and each one has multiple RELEASE branches off of it. With that picture and terminology in mind it's best to next explain what gets committed into each branch.
All new commits go into head (CURRENT) first - there are very few exceptions to this. Simply leaving things in head would make for a pretty hectic process for consumers of the FreeBSD source code as people/companies often use FreeBSD as a base for a product. Only having CURRENT to develop a product with is never a good idea. After some settling time in head a commit can possibly be merged to a STABLE branch.
STABLE branches provide a point in the tree where the team has decided that data structures look good and are not going to change. I'll provide more details on what this means later. The most important factor in determining what can be merged from CURRENT (MFC, sometimes called merge from head (MFH)) to a STABLE branch is something called ABI stability. I'll explain what this means later but for now the commit is generally not allowed if it breaks the ABI. It's important to note that not all commits in head get merged to a STABLE branch. It really depends upon what it is and if it is deemed important enough to make it into the next release off that particular STABLE branch. Something that often trips up users of FreeBSD is that a STABLE branch is still a development branch. Due to the way commits are done in CURRENT first and merged into a STABLE branch later it's usually the case that a STABLE branch is more stable (in the "least likely to crash" sense) but it's important to note that it is still an active development effort.
RELEASE branches are branched off of a STABLE line and are the most tightly controlled of them all. These only get security and major errata updates. In the "least likely to crash" sense, a RELEASE branch is the right choice.
If all that makes sense then it's time to explain what ABI stability is and why it is so important to the FreeBSD development model. Imagine a situation where a company is building a something with FreeBSD. If they were using CURRENT they would be subject to any of the data structure changes going on. If any of these changes were to affect their code they would have to adjust their code. By picking STABLE branch they are not subject to any data structure changes (which is probably the biggest worry as other parts of the ABI are so rarely changed), nor any of the other things which could possibly break ABI. Beyond providing a slightly more tested area for users and consumers a STABLE branch allows for code to be built on FreeBSD that is guaranteed to work across the lifetime of the STABLE branch. This allows for resources to be better utilized. Compare that to other development efforts where not only is the ABI constantly broken but entire subsystems are ripped out and replaced between releases. Imagine having to keep track of what changed and what that means to your product. By picking a STABLE branch that worry is lessened (you should still pay attention to the commits to see if they provide new features or bug fixes that interest you) because you know the underlying binary interface you are using will not change.
So what does this mean to you? If you're an average user who just wants things to work I would recommend a RELEASE branch and upgrading between them when a new one is made. If you're a company building something using FreeBSD I would recommend following a STABLE branch. If you have the resources and are looking to the future I would also recommend that as a new STABLE branch nears you focus on moving your code to CURRENT for testing. This gives you the ability to maintain releases based upon code in a STABLE branch while making sure that you have an upgrade path to the next STABLE branch in the future, because the last thing you want to do is get stuck maintaining a EoL (End of Life) STABLE branch on your own (it's probably a waste of your resources). If you're an enthusiast, developer or just plain looking to get the most cutting edge features you can run CURRENT.
Hopefully this made sense to people not very familiar with how the development in FreeBSD works at a high level. If you're looking for more reading I would recommend at least this page in the handbook.
posted at: 15:18 | tags: freebsd | path: /entries/freebsd | permanent link to this entry
Tue, 16 Jun 2009
Package List (plist) Details
For the most part plists (in FreeBSD ports) are simply a listing of files installed by the port relative to ${PREFIX}. The contents of the plist get put into the +CONTENTS file in the package. Failure to get the plist correct in the port causes the package and the port to not properly clean up after itself, which leads to stray files on your machine. Not only is it a tidy problem it can potentially cause problems later on down the road. It's not hard to imagine a scenario where a binary is left laying around with a vulnerability in it. It is important to get plists correct to ensure a happy user experience.
Probably the most common thing you will see done in plists besides listing files is handling configuration files during (de)installation. This is well documented in the handbook (and don't forget to put the corresponding part in the Makefile as the plist is not processed when using the port). You can actually do a lot more with the plist if you need to, though it is not often the case.
I've had a few people ask me recently how plists work so I thought I would document what I consider to be a good resource regarding plists. When in doubt I always take a look at pkg_create(1) - specifically the PACKING LIST DETAILS section. If you're working on a port and get yourself confused when dealing with the plist I would look through that and it will most likely clear up any issue you have.
posted at: 17:10 | tags: freebsd, ports | path: /entries/freebsd | permanent link to this entry
Wed, 10 Jun 2009
Upgrading Python with Portmaster
If you are upgrading python on FreeBSD to 2.6 the steps as they stand now require portupgrade to be installed. This is based purely on the usage of pkg_which when walking /usr/local/lib/python$ver to determine what ports to rebuild.
Last night I came up with a patch that allows one to use portmaster instead of portupgrade. It's slower because it uses pkg_info. My guess is that pkg_which is using the pkgdb used by the portupgrade suite, which makes it significantly faster then how pkg_info works. We try to parallelize the processing but it's still slow - IMO we could greatly increase the amount of parallelism going on but I'm not going to push it too far.
The patch is available at my FreeBSD space and will apply cleanly to the latest lang/python port. I intend to commit it tomorrow morning while I'm eating breakfast but if you want to test it out now just do the following:
cd /usr/ports/lang/python fetch - http://people.freebsd.org/~wxs/python26-portmaster.diff | patch portmaster -o lang/python26 lang/python25 make upgrade-site-packages -DUSE_PORTMASTER
posted at: 12:45 | tags: freebsd, portmaster, python | path: /entries/freebsd | permanent link to this entry
Mon, 08 Jun 2009
ZFS + NFS = Crash :(
I started to experience a crash recently that was triggered when building something in my tinderbox setup. This particular tinderbox is running on ZFS and uses NFS mounts on localhost. The panic and backtrace look like this:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x6dc
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80572e7f
stack pointer = 0x28:0xffffff803e722530
frame pointer = 0x28:0xffffff803e722550
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1030 (nfsd: service)
[thread pid 1030 tid 100140 ]
Stopped at prison_priv_check+0xff: movl 0x6dc(%rsi),%eax
db> bt
Tracing pid 1030 tid 100140 td 0xffffff00029ea000
prison_priv_check() at prison_priv_check+0xff
priv_check_cred() at priv_check_cred+0x4c
secpolicy_vnode_access() at secpolicy_vnode_access+0x28
zfs_zaccess() at zfs_zaccess+0x1d5
zfs_freebsd_access() at zfs_freebsd_access+0xd0
VOP_ACCESS_APV() at VOP_ACCESS_APV+0x44
nfsrv_access() at nfsrv_access+0xf3
nfsrv3_access() at nfsrv3_access+0x386
nfssvc_program() at nfssvc_program+0x1fb
svc_run_internal() at svc_run_internal+0x6d2
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a0e1c, rsp = 0x7fffffffe6d8, rbp = 0x5 ---
db>
Here's the relevant pieces of prison_priv_check():
/*
* Check with permission for a specific privilege is granted within jail. We
* have a specific list of accepted privileges; the rest are denied.
*/
int
prison_priv_check(struct ucred *cred, int priv)
{
if (!jailed(cred))
return (0);
switch (priv) {
...
/*
* Depending on the global setting, allow privilege of
* mounting/unmounting file systems.
*/
case PRIV_VFS_MOUNT:
case PRIV_VFS_UNMOUNT:
case PRIV_VFS_MOUNT_NONUSER:
case PRIV_VFS_MOUNT_OWNER:
if (cred->cr_prison->pr_allow & PR_ALLOW_MOUNT)
return (0);
else
return (EPERM);
...
Loading up the core in kgdb for analysis it becomes very clear what is going on.
(kgdb) frame 12
#12 0xffffffff80572e7f in prison_priv_check (cred=0xffffff00168f5900, priv=334) at /usr/home/wxs/freebsd/src/head/sys/kern/kern_jail.c:3315
3315 switch (priv) {
(kgdb) p/x *cred
$7 = {cr_ref = 0x1, cr_uid = 0x0, cr_ruid = 0x0, cr_svuid = 0x0,
cr_ngroups = 0x1, cr_groups = {0x0 }, cr_rgid = 0x0,
cr_svgid = 0x0, cr_uidinfo = 0x0, cr_ruidinfo = 0x0, cr_prison = 0x0,
cr_vimage = 0x0, cr_flags = 0x0, cr_pspare = {0x0, 0x0}, cr_label = 0x0,
cr_audit = {ai_auid = 0x0, ai_mask = {am_success = 0x0, am_failure = 0x0},
ai_termid = {at_port = 0x0, at_type = 0x0, at_addr = {0x0, 0x0, 0x0,
0x0}}, ai_asid = 0x0, ai_flags = 0x0}}
(kgdb)
It's clear that cred->cr_prison is bad. But this isn't the real meat of the problem. The first check in prison_priv_check() is to see if we are jailed, and that looks something like this:
/*
* Return 1 if the passed credential is in a jail, otherwise 0.
*/
int
jailed(struct ucred *cred)
{
return (cred->cr_prison != &prison0);
}
Up until fairly recently this function used to contain:
return (cred->cr_prison != NULL);
So, because cred->cr_prison is NULL in our case the check in prison_check_cred() is evaluating to false, when it should be evaluating to true. So now we know why we are crashing (NULL ptr dereference) we still don't know what is causing cr_cred to be NULL.
Credentials like this are derived from a very small number of places. The reason these are wrong is that the RPC code in the kernel doesn't know which credentials to assign when it handles the request. Luckily a workaround has been put in place while a more proper solution is being worked on.
posted at: 12:48 | tags: freebsd, zfs, nfs | path: /entries/geek | permanent link to this entry
Tue, 14 Apr 2009
Gnome requires Apache?
wxs@syn wxs % make -C /usr/ports/x11/gnome2 all-depends-list | grep apache /usr/ports/www/apache22 wxs@syn wxs %
WTF? Who thought that including an entire webserver just to run a desktop system is a good idea?
posted at: 16:16 | tags: freebsd, rant | path: /entries/rant | permanent link to this entry
Mon, 06 Apr 2009
getrusage() and another example of Linux failure.
So I'm writing some code for work and I need to know the current stack usage. One would think getrusage() would be nice to use. Think again. To quote the manual page on Ubuntu:
struct rusage {
struct timeval ru_utime; /* user time used */
struct timeval ru_stime; /* system time used */
long ru_maxrss; /* maximum resident set size */
long ru_ixrss; /* integral shared memory size */
long ru_idrss; /* integral unshared data size */
long ru_isrss; /* integral unshared stack size */
long ru_minflt; /* page reclaims */
long ru_majflt; /* page faults */
long ru_nswap; /* swaps */
long ru_inblock; /* block input operations */
long ru_oublock; /* block output operations */
long ru_msgsnd; /* messages sent */
long ru_msgrcv; /* messages received */
long ru_nsignals; /* signals received */
long ru_nvcsw; /* voluntary context switches */
long ru_nivcsw; /* involuntary context switches */
};
[...]
The above struct was taken from 4.3BSD Reno. Not all fields are mean-
ingful under Linux. In linux 2.4 only the fields ru_utime, ru_stime,
ru_minflt, and ru_majflt are maintained. Since Linux 2.6, ru_nvcsw and
ru_nivcsw are also maintained.
I can understand having the same struct for portability reasons. It's nice that code can be portable between the two at compile time but this is opening itself up to run-time problems if code assumes all the members of the struct will be populated.
I'm not sure why it's only barely implemented but it's quite annoying. The same function call on FreeBSD gives all the information, and the manual page under FreeBSD has useful information on what each of the members of the struct means. Linux fails, again.
gdb output on Linux:
(gdb) p/x *usage
$1 = {ru_utime = {tv_sec = 0x0, tv_usec = 0xfa0}, ru_stime = {tv_sec = 0x0,
tv_usec = 0xbb83}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0,
ru_isrss = 0x0, ru_minflt = 0x2fe, ru_majflt = 0x0, ru_nswap = 0x0,
ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0,
ru_nsignals = 0x0, ru_nvcsw = 0x7b, ru_nivcsw = 0x1}
(gdb)
gdb output on FreeBSD:
(gdb) p/x *usage
$3 = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0,
tv_usec = 0x3ac7}, ru_maxrss = 0x8e8, ru_ixrss = 0x3f8, ru_idrss = 0xec,
ru_isrss = 0x100, ru_minflt = 0xeb, ru_majflt = 0x0, ru_nswap = 0x0,
ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0,
ru_nsignals = 0x0, ru_nvcsw = 0xc, ru_nivcsw = 0x1}
(gdb)
posted at: 16:46 | tags: linux, freebsd, code | path: /entries/geek | permanent link to this entry
Fri, 13 Mar 2009
I've Got a Minion!
The account has not been created yet but my proposal to get Steven Kreuzer a commit bit has been approved. I got the good news a few days ago while I was traveling for work and I have yet to get the necessary information from Steven to get his account created, but he will be my first minion mentee. It certainly brings back memories of when I was going through the process. I hope to be as helpful to Steven as Renato (my mentor) was to me. Congratulations Steven on the formal recognition of your hard work over the years.
posted at: 10:20 | tags: freebsd | path: /entries/freebsd | permanent link to this entry
Fri, 06 Mar 2009
Spring Cleaning!
I put a couple of ports in the Attic today. These were ports which expired sometime in 2007. Since we are now in 2009 I figured it was safe to 'cvs rm' them. There are two more that expired in 2007 that I intend to clean up shortly but those will require a bit of work as they are depended upon by other things. Once I'm done with these I will move on to stuff that expired in 2008. After that I intend on making pretty graphs related to PR statistics and not handle as many incoming PRs until I do so.
posted at: 16:50 | tags: freebsd | path: /entries/freebsd | permanent link to this entry
