2015/11/08

This is how I like opensource

While OpenSource Operating System are different, have different views on how things should be implemented. That does not prevent collaboration, collaboration can happen in various areas:

Here I want to talk about the second point. While it is "easy" when we speak about third party tools: simple binaries etc, it is a bit more complicated when we speak about components that are low level or has an inpact on the whole system like libc.

The story I want to talk about is how FreeBSD, DragonflyBSD and Illumos has collaborated to get a good support for String Collation and improved support for locales.

It all started on Illumos. Garrett d'Amore implemented unicode string collation on Illumos while working for Nexenta and made it BSD license (the choice of this license allowed us to be able to grab the code more easily).

Few years later, John Marino (from Dragonfly) imported the work done on Illumos into Dragonfly, while he was doing that he decided, it was probably a good idea to rework how locales are handled.

He discovered that Edwin Groothuis (from FreeBSD) started long ago a project to simplify locales handling on FreeBSD: (edwin's project)

He extended the tools written by Edwin and has been able to update Dragonfly to the latest (v27 so far) unicode definition from cldr.

John and I already work together on multiple projects so he came to me to talk about his project, I jumped on it and decided to merge that work on FreeBSD (unicode string collation was a long overdue project in FreeBSD, and beside several attempts via Summer of Codes, none of it managed to get far enough to be mergeable).

What I did was merging the code from Dragonfly, I spotted a couple of bugs and worked with John on fixing them: issues with eucJP encoding, issues with Russian encoding (John did most of the work on tracking down and fixing the bugs), I also converted localedef (the tool to generate the locales) into using BSD license only code (original version used the CDDL libavl library which I modified to use tree(3)), fixed issues. I also took the locale generation from Edwin (extended by John).

This work resulted in a nice flow of patches going from Dragonfly to FreeBSD and from FreeBSD to Dragonfly.

And now Garrett is interested in grabbing back our patches into Illumos!

The result of this collaboration is that now 3 OS share the same implementation for collation support! This is very good because when one discovers a bug the 3 of them benefit the fix!

Concerning the locales we have ended up with different definitions (different policies or views) but the tools and code are the same so we can benefit from each other improvements.

This collaboration also allowed us to quickly get a working implementation for a tough subject: "string collation", and we helped improving Dragonfly and Illumos implementation.

Pedro Giffuni wrote on 2015/11/12 :

Great post: kudos to everyone involved.
For the record, in FreeBSD we have been able to collaborate quite nicely with Illumos on many fronts beyond DTrace and ZFS, which are of course huge Jewels.
With Garrett in particular, much of the crosspolination came from their adoption of some of our utilities like sed, tr, printf and the internationalization parts of libc. Illumos has strong quality policies and code has gone back and forth. Another company, known to be in the fruit business, also took the same code but collaboration never reached this level.
A real lesson of professionalism for other communities that want to make the best out of open source development.


Grego wrote on 2016/01/07 :

It's all about the people! Congratulations and thank you to everyone who made this happen.


Finn Espen Gundersen wrote on 2016/02/12 :

Did this land in FreeBSD, in 10.3 or 11.0? It has great implications for Postgresql. I need multiple collation support there and have to migrate the database server. But to which version of FreeBSD?


Name :
URL (optional) :
Write here: "ETOOMUCHSPAM"
Comments :