2014/06/25

fossil in prison

Since my last post about how I do host fossil I have been asked write about the new setup I do have

The jail content

I have created a minimal jail:

$ find /usr/local/jails/fossil -print
/usr/local/jails/fossil/var
/usr/local/jails/fossil/var/tmp
/usr/local/jails/fossil/libexec
/usr/local/jails/fossil/libexec/ld-elf.so.1
/usr/local/jails/fossil/bin
/usr/local/jails/fossil/bin/sh
/usr/local/jails/fossil/bin/fossil
/usr/local/jails/fossil/lib
/usr/local/jails/fossil/lib/libc.so.7
/usr/local/jails/fossil/lib/libssl.so.7
/usr/local/jails/fossil/lib/libreadline.so.8
/usr/local/jails/fossil/lib/libz.so.6
/usr/local/jails/fossil/lib/libcrypto.so.7
/usr/local/jails/fossil/lib/libncurses.so.8
/usr/local/jails/fossil/lib/libedit.so.7
/usr/local/jails/fossil/data
/usr/local/jails/fossil/dev

/bin/sh is necessary to get the exec.start jail argument to work /var/tmp is necessary to get fossil to open his temporary files (I created it with 1777 credential) /data is a empty directory where the fossil files will be stored

Jail configuration

The configuration file is the following:

fossil {
	path = "/usr/local/jails/fossil";
	host.hostname = "fossil.etoilebsd.net";
	mount.devfs;
	ip4.addr="127.0.0.1";
	exec.start = "/bin/fossil server -P 8084 --localhost --files *.json,*.html,*.js,*.css,*.txt --notfound /index.html /data &";
	exec.system_jail_user = "true";
	exec.jail_user = "www";
	exec.consolelog = "/var/log/jails/fossil.log" ;
}

More about fossil itself

In /data I created an index.html which is an almost empty html with a bit of Javascript.

When loading the javascript will request a list.txt file.

This file contain the list of repositories I want to show publically (one per line).

For each of them the javascript will use the json interface of fossil (meaning your fossil has to be built with json) and gather the name and the description of the repo to print them on the index.

Starting/Stopping the service

2 simple command are necessary to manage the service:

Starting up:

# jail -c fossil

Stopping:

# jail -r fossil

The service is only listening on the localhost, it is up to you to create your reverse proxy, in my case I do use nginx with the following config:

server {
	server_name fossil.etoilebsd.net;
	listen       [::]:443 ssl;
	listen       443 ssl;
	ssl_certificate     ssl/fossil.crt;
	ssl_certificate_key ssl/fossil.key;

	location / {
		client_max_body_size 10M;
		proxy_buffering off;
		proxy_pass http://127.0.0.1:8084/;
		proxy_set_header HTTPS on;
		proxy_set_header   Host             $host;
		proxy_set_header   X-Real-IP        $remote_addr;
		proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
	}
}

2013/06/24

pkg 1.1

After almost a year of development pkg 1.1 has reached the ports tree, actually pkg 1.1.1 has 1.1 was too buggy :(

What happened in 1 year of development (I'll focus on use visible features)

multi-repository

The multi-repository support was experimental in pkg 1.0 and to be honest it was not really usable. With pkg 1.1 the support has been greatly improved and it is now the default behaviour (you can't deactivate).

To define repository you just have to create a simple configuration file in /usr/local/etc/pkg/repos/myrepo.conf

myrepo:
  url: http://myurl
  pubkey: /usr/local/etc/pkg/repos/myrepo.key
  mirror_type: SRV

Meaning you can provide a package to autosetup a repository creating a package containing like this one:

$ tar tf myrepo-1.0.txz
+MANIFEST
/usr/local/etc/pkg/repos/myrepo.conf
/usr/local/etc/pkg/repo/myrepo.key

Host this file somewhere and say to the use to do the following

$ pkg add http://yourhost/myrepo-1.0.txz

Now you can see that the repository is configured properly pkg -vv should show you in the last lines:

Repositories:
  packagesite:
                    url: http://private.etoilebsd.net/91-default-server
                    key:
                enabled: yes
            mirror_type:
  myrepo:
                    url: http://myurl
                    key: /usr/local/etc/pkg/repos/myrepo.key
                enabled: yes
            mirror_type: SRV

The user can also choose to make sure a given package will always be updated from 'myrepo'

$ pkg install -r myrepo mypackage
$ pkg annotate -A mypackage reposiroty myrepo

Now the package 'mypackage' will only be updated from 'myrepo'

pkg lock/unlock

If a use want to prevent a package from being updated anyway he can just lock it:

$ pkg lock mypackage

To unlock it just update use the following command:

$ pkg unlock mypackage

ssh transport

If your server has pkg 1.1+ installed then you do not need so set up a HTTP server or a FTP server, pkg can now use ssh to share the packages

packagesite: ssh://user@host:/path

Or in the repository configuration:

url: ssh://user@host:/path

Do not forget to restrict on the server the directory where files can be retrieved by adding the following line on the server pkg.conf:

SSH_RESTRICT_DIR: /path

annotate

This allows to add any key/value annotation to a given package once installed, if you recreate the package after that, the annotation will be added to the manifest and then a new reinstallation will keep the annotation.

plugins

pkg now supports 2 kind of plugins: commands (to add new subcommand to pkg) and hooks (which will be executed in the middle of any process of pkg).

I'll write another post dedicated to plugins later.

explained reinstallation

As pkg is able to determine that a package needs to be reinstalled because the remote one has been compiled with new options or the required shared libraries for the package has changed, pkg now explains why a package will be reinstalled.

misc

We have stabilized the public API, so now bindings, and program using libpkg are more than welcome :) Lots of cleanup has occurred in the code, and lots of code optimisation. New pkg_printf(3) function to help printing a preparing strings with pkg informations. We are more and more adding some regressions test using the ATF framework. The catalog has changed and is now a simple yaml files which gives us more flexibility and allow simples incremental update. pkg audit can now directly read the vuxml native format.

Way more things but I'll let you discover :)

Thanks to all people that has been involved in the new release (coders, testers, doc writers, etc.)

2013/06/21

Stash for svn

When hacking on the ports tree or on the sources, you often have unfinished patches you are testing step by step.

I'm also hacking on something unfinished and then some other area needs some fixes with a higher priority and in the same time some people are asking for some testing/review on their own patches. So I need to quickly interrupt what I was working on get back to a clean tree, and switch from patches to patches.

While doing this is easy with git, fossil or mercurial it is more complicated with svn. The feature in particular I use on fossil/git for that is the stash feature.

So I wrote my own stash for svn, and because of mobility I was willing to be able to share my patches across boxes, so I have made stash able to be under a vcs itself, with support for git, hg, fossil and svn as a vcs for the stash.

How does it works: The stash command will discover that .svn of the working copy you are working on and will create a patches subdirectory.

Now imagine that directory itself contains a .hg, .fslckout, a .git or a .svn then stash will know it is being under vcs control.

$ stash save <name> [-u] [files...]

This will create a .svn/patches/.patch file using svn diff --git on the specified files (if none is provided it will diff the current directory the stash command is being run on).

Once the diff created it will rollback the tree (on specified files or current directory) to the clean state before any modification.)

By default it will not overwrite a patch with the same name except is -u is provided on the command line.

If the stash directory is under a vcs control then a add/commit (or just commit in case of update) will be performed in the stash directory.

$ stash ls

List all the available patches.

$ stash show <name>

Print on stdout the content of a given patch

$ stash apply <name>

Apply a specified patch on the working copy using svn patch --strip 1 from the root of the working copy.

$ stash rm <name>

Remove/destroy a patch from the stash directory. In case the stash directory is under vcs control then the proper rm command followed by the needed commit will be performed.

$ stash pop <name>

It is the equivalent of stash apply followed by stash rm this is useful when your patch is finished and you want to commit it directly.

$ stash push <name>

Push (scp) the patch on a remote site (currently freefall is hard coded :)

$ stash sync <name>

This command is only useful if the stash directory is under vcs control, it performs the necessary pull/push mechanism depending on the VCS used.

I use fossil to maintain the stash script And here is for example my repository of patches for the ports tree

No git svn won't have worked in my case for multiple reason: 1/ want something flexible which can also only work with svn 2/ the ports tree can work properly with git svn (properties setting adding new files etc will not work as one expect) 3/ I want to use fossil for the stash, other might prefer svn or hg.

Disclaimer: hg and git support hasn't been tested yet, patches welcome to fix them if needed. if you want to add support for your own favorite vcs just them me the patches I'll integrate them.

2013/06/06

Conversion to new options: done

It took a year but at least it is done! Huge thanks to all people involved in the new Option Framework.

So now what we have:

What still needs to be done:

With the previous done, my 2 main priorities for FreeBSD now are:

2012/06/05

New options framework is in, what next?

The new options framework has been committed one week ago, I would have expecting things to have been smoother but lots of fixes has been needed to have it correctly working.

In one week everything at least every major issues reported has been fixed, and now everything is working again as expected (if not please report it to me asap so I can fix it)

So from a maintainer point of view what does it brings:

From a port developer point of view:

From a user perspective:

How to enable/disable an options? simple in make.conf:

OPTIONS_SET= BLA # enable BLA for the whole ports tree
OPTIONS_UNSET=	BLA # disable BLA for the whole ports tree
foo_SET= BLA # enable BLA just for foo
foo_UNSET= BLA # diable BLA just for foo

the user settings have an order or priority:

Note that you can disable this automatic make config by adding NO_DIALOG to your /etc/make.conf.

All of this means that ports-mgmt/portconf is not needed anymore for a centalisation of your configurations.

Short term goal: to avoid having the ports tree infrastructure keeping for ever old code like it was the case in the past for other manjor improvements, please update your ports to the newoption framework as soon as possible.

Long term goal:

The options framework was an important step in the cleanup of the ports infrastructure, next step in my concern will be to provide stage directory support aka: a ports will install everything in a stage directory and sync the files following the plist to the system or directly create the package out of this stage directory. Every sane package building system are using that kind of thing but the FreeBSD ports this is necessary as it also open the way to tons of new possibilities. Stay tune, this is coming soon!

2012/05/25

SQLite backend for CBlog

The trunk of cblog is slowly moving its backend from cdb to sqlite.

This blog is now running on top of sqlite.

Why this change? just for simplicity, now everything is contained in a single and simple sqlite file, this simplify a lot some part of the code (which wasn't very complicated anyway).

The migration of the code was done the quick and dirty way, I'll try to cleanup and improve when I'll have more time.

Note that in the trunk there is a new tool cblogconvert that can convert from 0.1 tinycdb + files backend to sqlite.

At the same time I also remove fcgi support and now uses libevent2 instead, it help cleaning up the code.

Maybe in the future I'll also write my alternative to clearsilver, maybe something using haml format, still undecided now.

2012/05/01

Fresh blood needed for LibreOffice

For some time now, I have been maintaining pretty much alone the LibreOffice port for FreeBSD, trying to provide all the features LibrOffice has to our users.

I managed to port the 3.3.x series of LibreOffice to FreeBSD some time ago and tried to keep updating it as fast as possible as soon as new releases were out.

At the beginning I was thinking about maintaining 2 concurrent of LibreOffice at the same time: legacy and "normal" to reflect the upstream support. But this take too much time, I'll kill libreoffice-legacy in the next weeks.

The problem is that it is really time consuming, not that it is really complex, the LibreOffice upstream is really nice and really responsive, other maintainers from the BSD community, has also been really helpful.

Some time ago, I decided to create office@ to help maintaining every office related ports maintained on FreeBSD, it worked quite well, sunpoet@ taking good care for examples of the different hunspell/hyphen/mythes dictionnaries/thesaurus, pfg and maho taking care of Apache OpenOffice and all of us working on the third party libraries/fonts/etc needed by all those big office project.

But I'm still missing help on LibreOffice itself, I just committed 3.5.2 some days ago, and 3.5.3 is almost there.

We also still have known issues on 3.5.2:

All the libreoffice work is done on redports svn

if you have an account and are willing to help tell me so that you can have access to the office svn on redports.

2012/05/01

From git to fossil

I recently killed git.etoilebsd.net, while I still appreciate git, and will keep it for pkgng I was looking for a new solution to be able to share my other projects.

The main problem I have with hosting git, is that I need lot of third party tools, to have some kind of project management.

To display in a web browser my git projects, I was using cgit which does its job very well, and is simple to maintain: simple and clean configuration files, just a simple C cgi, all what I do like. But for those projects I was also needing 2 others things, a simple web page, if possible maintainable within the git repository itself which cgit can do and a ticket system (some users complained not being able to report bug/feature request).

I first tried to setup and install roundup, it is quite simple and does its job quite well, it can use sqlite, to avoid me running a useless database, it doesn't require too much dependencies so it was great but badly integrated with git (I don't want to spend time tunning too much the software I use)

I also had a look to all those forge available, like chiliproject or redmine, or the old but still good trac. While I did like trac, it requires too much dependencies for my small hostings needs. The two first are even worst in that area plus I find their url completely illogical to me.

The others available were using php or java and were most of them needing a mysql/postgresql database, I rejected them because I don't want any php or java software running on my small server, and I don't want any database constantly running on it either.

For a while now I am following the development of an alternative scm, named fossil, it is a small all-in-one project: scm, wiki, events, ticket contained in a single binary. It is really easy to use, requires nearly no administration, have all the modern features you can expect from a DVCS. And not that important for me but still good, it is BSD licensed.

To migrate from git to fossil it was really easy:

$ cd poudriere
$ git fast-export --all | fossil import --git poudriere.fossil

That is all, I know have a fully working poudriere.fossil.

To serve the fossil repositories on my server here is what I did:

Add an entry to /etc/services:

$ echo "fossil	8080/tcp" >> /etc/services
$ services_mkdb /etc/services

Add an entry to inetd:

$ echo fossil stream tcp nowait.1000 www /usr/local/bin/fossil /usr/local/bin/fossil http /data/fossil" >> /etc/inetd.conf
$ echo "inetd_enable=YES >> /etc/rc.conf
$ service inetd start

Add a simple virtual host to nginx:

server {
	server_name fossil.etoilebsd.net;
	listen 80;
	listen 443 ssl;
	location / {
		access_log /var/log/nginx/fossil.access.log main;
		proxy_pass http://127.0.0.1:8080/;
		proxy_redirect off;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

That's all now, your fossil repositories are served properly, you can now access them from the fossil cli using both ssh or http (I only serve http for now in my case)

My fossil repositories are:

2012/03/03

Home made pkgng repositories

By popular demand I have been requested a tutorial about how to maintain your own pkgng repositories.

Currently the best solution for that is to use poudriere (ports-mgmt/poudriere).

Let say you want to maintain 2 repositories: 8.2-RELEASE i386 and 9.0-RELEASE amd64.

For that you will need an amd64 (to support both amd64 and i386 box with 9.0 binary support and 8.2 binary support, a default 9.0-RELEASE amd64 should be enough :)

poudriere only depends on zfs and sh, so you won't need much, just a zfs pool available.

First: install poudriere:

$ make -C /usr/ports/ports-mgmt/poudriere install clean

To be able to build the repository the host would also need to have pkgng installed (no need to convert it to pkgng anyway.)

$ make -C /usr/ports/ports-mgmt/pkg install clean

Now configure your poudriere, defining some configuration to /usr/local/etc/poudriere.conf:

BASEFS=/poudriere
ZPOOL=myzpool
FTPHOST=ftp.freebsd.org
POUDRIERE_DATA=/poudriere_data
RESOLV_CONF=/etc/resolv.conf
DISTFILES_CACHE=/usr/ports/distfiles

This should be enough (see poudriere.conf.sample for more informations)

Create the default ports tree:

$ poudriere ports -c

Create the two jails (in fact it will be chroot :))

$ poudriere jails -c -j 82i386 -v 8.2-RELEASE -a i386
$ poudriere jails -c -j 90amd64 -v 9.0-RELEASE -a amd64

Make them pkgng aware

$ mkdir /usr/local/etc/poudriere.d
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/82i386-make.conf
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/90amd64-make.conf

Add the list of packages you want to build:

$ cat ~/mylist1
editors/vim
www/nginx
$ cat ~/mylist2
www/firefox
editors/libreoffice

If you want special options just add them to you different make.conf in poudriere.d

to share options between all the jails managed by poudriere, add them to /usr/local/etc/poudriere.d/make.conf

You can now create your packages:

$ poudriere bulk -f ~/mylist1 -j 82i386
$ poudriere bulk -f ~/mylist2 -j 90amd64

When finished you will get 2 pkgng repositories in /poudrieredata/packages/82i386-default and /poudrieredata/packages/90amd64-default

You can now provide them through your webserver.

On you client boxes just:

$ echo "packagesite: http://yoururl/82i386-default" >> /usr/local/etc/pkg.conf

or

$ echo "packagesite: http://yoururl/90amd64-default" >> /usr/local/etc/pkg.conf

If your client box already has packages from an old installation, first convert it to pkgng

$ fetch http://yoururl/90amd64-default/Latest/pkg.txz
$ tar xf ./pkg.txz -s ",/.*/,,g" "*/pkg-static"
$ ./pkg-static add ./pkg.txz
$ pkg2ng

Now you can forget about pkg_install and pkg2ng

Just use pkgng (for example):

$ pkg update
$ pkg upgrade
$ pkg install firefox

To update your repository:

$ poudriere ports -u # this update your default ports tree
$ poudriere bulk -f ~/mylist1 -j 82i386 -k
$ poudriere bulk -f ~/mylist2 -j 90amd64 -k

bonus poudriere will rebuild only what has changed and what is impacted by this change.

Once it is done simply upgrade your client box:

$ pkg update
$ pkg upgrade

More informations on poudriere here

2012/02/22

Full binary upgrade with pkgng 1.0-beta7

My laptop (a small eeepc 1000H) is running FreeBSD 8.2-RELEASE for a while now, I didn't upgraded it to 9 because it would have taken eons to rebuild all the packages I use on this laptop.

Of course this laptop has been converted to pkgng long time ago, and I'm happily using it in a full binary way: building packages and preparing pkgng repositories on a build box using poudriere (ports-mgmt/poudriere).

In the git version of pkgng (what would become soon 1.0-beta7) one of the new feature is the ability to reinstall the whole set of packages. This allow us to now perform a full binary upgrade of the system.

Here is how I performed an upgrade from 8.2-RELEASE to 9.0-RELEASE without building anything on that small laptop:

Create a 9.0-RELEASE repository

First you need to prepare a set of packages for 9.0-RELEASE on poudriere:

# create the new jail:
$ poudriere jail -c -j eeepc9 -a i386 -v 9.0-RELEASE
# copy the make.conf from the old to the new jail:
$ cp /usr/local/etc/poudriere.d/eeepc-make.conf /usr/local/etc/poudriere.d/eeepc9-make.conf
# build the set of packages:
$ poudriere bulk -f ~/bulkeeepc -j eeepc9 -k

Now that your repository of packages is ready just expose it in your http server and you are done with that part.

Upgrading the laptop

Update to the lastest 8.2-RELEASE

You need to upgrade your 8.2-RELEASE to the latest version available if not already done (it contains a fix for freebsd-update which is necessary to perform the upgrade to 9.0)

$ freebsd-update fetch
$ freebsd-update install

Upgrade to 9.0-RELEASE

You can now upgrade to 9.0:

$ freebsd-update -r 9.0-RELEASE upgrade
$ freebsd-update install
$ reboot
$ freebsd-update install

Once you are here your system now upgraded to 9.0-RELEASE but still have some old shared object files, it is the moment when you should perform the upgrade of the packages.

Edit your /usr/local/etc/pkg.conf file and adjust the url of the PACKAGESITE to the new eeepc9 repository

$ pkg update
$ pkg upgrade -fy

you can now remove the old shared object:

$ freebsd-update install

In case you would have forgotten to perform the package upgrade, pkg-static is there to save your life, you can still do it now:

$ pkg-static update
$ pkg-static upgrade -fy

Pages : 1 2 3 4 5 6 7