Filed under: FreeBSD — ivoras @ 11:12 am

The day has finally come – FreeBSD is using Subversion instead of CVS for the base source tree! Congratulations to everyone involved, especially Peter Wemm :)

FreeBSD’s source CVS is one of the oldest and biggest in existence; it’s approximately 12 years old and has apparently had something like 180,000 commits over the years, or on average slightly more than 41 commits daily. A checkout of RELENG_7 branch holds more than 42,000 files (in 482 MB as du sees it).

This move was discussed extensively during the DevSummit at BSDCan 2008; there have been many issues with CVS over the years, most of which are minor enough to be overlooked, but some of which are just nasty (the inability of CVS to move/rename files, bad handling of branching in the event of constant new development and additions to the directory tree, non-atomic commits) and have frequently required manual interventions in the CVS repository (“repocopy” is one of the relatively frequently requested operations to the CVS admins).

Old infrastructure, of which cvsup / csup is probably the most important part, will continue to work as code will continually be mirrored from SVN to CVS, until suitable replacements or upgrades to the above tools are created. This is also the reason the name “CVS” will be present for some time in the system infrastructure, until all of it is updated. Ports will continue to use CVS for the foreseeable future.

To make a source base this large work efficiently on SVN, version 1.5 had to be used, since it creates its database files in a hash-tree of directories on the file system instead of one huge directory with all the files in it.
http://blogs.freebsdish.org/ivoras/2...on-subversion/

On Mon, 2 Jun 2008 17:14:59 -0600 (MDT),
Borked Pseudo Mailed <nobody> wrote:
>
> Filed under: FreeBSD — ivoras @ 11:12 am
>
> The day has finally come – FreeBSD is using Subversion instead of CVS
> for the base source tree! Congratulations to everyone involved,
> especially Peter Wemm :)
> [...]
> [..]

The conversion is still an ongoing process, so it's not something
'done'. Having said that, it's certainly true that Peter deserves
everyone's congratulations for his persistence in making this happen,
and all his efforts :)

On Tue, 03 Jun 2008 05:32:02 +0300,
Giorgos Keramidas <keramida> wrote:
> On Mon, 2 Jun 2008 17:14:59 -0600 (MDT),
> Borked Pseudo Mailed <nobody> wrote:
>
> The conversion is still an ongoing process, so it's not something
> 'done'. Having said that, it's certainly true that Peter deserves
> everyone's congratulations for his persistence in making this happen,
> and all his efforts :)

Does he? I'm still not sure that despite the hype subversion is actually
substantially better. The code base is new where CVS's is pretty stable,
things they could've improved they just ``forgot'' to add to the design,
and I hear they recently swapped backends because of performance and
data integrity problems.

So, I'm not sure it doesn't suffer from second system effect and that
this isn't a lot of effort for the sake of hype after all. subversion
has seen more than its fair share of hype, often from people who didn't
know how to correctly use CVS in the first place.

Without having read that blog, I do wonder about perforce.


Oh, and to the OP: do something about your line lengths please, even if
you insist on using an anonymous remailer. As to that: Those tend to be
killfiled due to troll abuse -- there are better ways to keep your name
out of the limelight on usenet.

On 3 Jun 2008 06:41:42 GMT, jpd <read_the_sig> wrote:
> On Tue, 03 Jun 2008 05:32:02 +0300,
> Giorgos Keramidas <keramida> wrote:
>
> Does he? I'm still not sure that despite the hype subversion is
> actually substantially better. The code base is new where CVS's is
> pretty stable, things they could've improved they just ``forgot'' to
> add to the design, and I hear they recently swapped backends because
> of performance and data integrity problems.

Yes, after putting the hype aside, subversion *is* an improvement over
CVS in a few areas which are important to FreeBSD developers:

* It uses changesets, not per-file records that have to be aggregated
later into meaningful `changesets' by a lot of guesswork.

* It's slightly better than CVS at pulling one of those changesets out
of a branch and merging it in another branch. There are far better
systems out there, but at least subversion doesn't require one to go
through all the hoops of `cvs update -j OLDREV -j OLDREV2' and the
manual tracking of each file-revision for OLDREV/OLDREV2.

* Much lighter 'server-side' processes. Running a Subversion server
doesn't need the same amount of resources as a CVS server.

There are probably a few others I'm forgetting right now. Coffee hasn't
kicked in yet, and I just woke up a couple of minutes ago. If I've
missed something and I remember it later I'll follow up later.

Please also note that I am not even _interested_ at hearing why SVN is
better than CVS in the general case. That falls very hear 'hype' that
you already referenced below. I'm only interested specifically in what
SVN or ${NEWSCM} has to offer to FreeBSD. If there's nothing we see as
substantial, or there are insurmountable differences in the way things
work, that's a deal-breaker.

As I wrote `The conversion is still an ongoing process, so it's not
something "done"'. So let's wait and see how well things go :-)

> So, I'm not sure it doesn't suffer from second system effect and that
> this isn't a lot of effort for the sake of hype after all. subversion
> has seen more than its fair share of hype, often from people who didn't
> know how to correctly use CVS in the first place.

It's not perfect. I am actually *not* a fan at all of Subversion. Its
`merge tracking' features are a joke when compared to Git or Mercurial,
and it needs `support' systems like SVK to be usable in offline mode.

My own preference would be something like Git or Mercurial. But those
systems come with their own set of issues when a large tree like the
FreeBSD `src/' is the matter.

> Without having read that blog, I do wonder about perforce.

Perforce isn't free software. It's not even merely open source. So we
can't _require_ something like Perforce for anyone who wants to get the
source of FreeBSD :-/

- Giorgos

jpd <read_the_sig> wrote:
> Giorgos Keramidas <keramida> wrote:
>
> Does he? I'm still not sure that despite the hype subversion is actually
> substantially better.

It is...

> The code base is new where CVS's is pretty stable, things they could've
> improved they just ``forgot'' to add to the design, and I hear they recently
> swapped backends because of performance and data integrity problems.

I am not entirely familiar with the internals of Subversion. I remember that
it originally used a Berkeley DB backend which had "issues" interoperating
with things like NFS. They fixed this with a file-based system with lots and
lots of files in it.

As far as performance goes: the killer is the fact that branches are constant
time. Getting anything involving branches done with CVS is hopelessly painful
and excruciatingly slow. And it only gets slower and more painful as the
amount of code grows.

> Without having read that blog, I do wonder about perforce.

Perforce is still there and won't be going away anytime soon. Some people
prefer it for developing features. Having said that, CVS isn't completely
"gone" either. It will need to stay supported at least until all the
currently supported releases are no longer supported.

The conversion to Subversion is still on-going but it's starting to work well.
I merged some files to releng_6 and releng_7 this morning and I didn't damage
my teeth!

- Philip

In article <878wxmly0j.fsf>,
Giorgos Keramidas <keramida> writes:
> On 3 Jun 2008 06:41:42 GMT, jpd <read_the_sig> wrote:
>> Without having read that blog, I do wonder about perforce.
>
> Perforce isn't free software. It's not even merely open source. So we
> can't _require_ something like Perforce for anyone who wants to get the
> source of FreeBSD :-/

Which leads to the question of whether anonymous SVN server
is available for non-committer access to src/?

Giorgos Keramidas <keramida> writes:
> There are probably a few others I'm forgetting right now. Coffee hasn't
> kicked in yet, and I just woke up a couple of minutes ago. If I've
> missed something and I remember it later I'll follow up later.

Atomic commits. This is one of the biggest reasons to move away from
CVS IMHO, especially for a project as big as FreeBSD where many people
are pulling the latest commits from a branch.

STABLE isn't very stable if I'm pulling at the exact second someone is
doing a big commit and I get half of their changes.

Steven G. Kargl <kargl> wrote:
> In article <878wxmly0j.fsf>,
> Giorgos Keramidas <keramida> writes:
>> On 3 Jun 2008 06:41:42 GMT, jpd <read_the_sig> wrote:
>>> Without having read that blog, I do wonder about perforce.
>>
>> Perforce isn't free software. It's not even merely open source. So we
>> can't _require_ something like Perforce for anyone who wants to get the
>> source of FreeBSD :-/
>
> Which leads to the question of whether anonymous SVN server
> is available for non-committer access to src/?

http://svn.freebsd.org/
svn://svn.freebsd.org/

But for the next five-six years (at least!) cvs will still exist in the usual
way, I think. I don't see CVS 'leaving' completely any day soon!

- Philip

In article <slrng4b03k.r8.philip+usenet>,
Philip Paeps <philip+usenet> writes:
> Steven G. Kargl <kargl> wrote:
>
> [..]
> svn://svn.freebsd.org/
>
> But for the next five-six years (at least!) cvs will still exist in the usual
> way, I think. I don't see CVS 'leaving' completely any day soon!

Philip,

Thanks for the info. Yes, I understand that cvs is going to be around
for a few more years, but the atomic changesets are a very nice feature
to have.

On Tue, 03 Jun 2008 11:32:12 +0300,
Giorgos Keramidas <keramida> wrote:
>
> Yes, after putting the hype aside, subversion *is* an improvement over
> CVS in a few areas which are important to FreeBSD developers:
[snip!]

Alright, thanks for clearing that up. :-)


> [...] I'm only interested specifically in what SVN or ${NEWSCM} has
> to offer to FreeBSD. If there's nothing we see as substantial, or
> there are insurmountable differences in the way things work, that's a
> deal-breaker.
>
> As I wrote `The conversion is still an ongoing process, so it's not
> something "done"'. So let's wait and see how well things go :-)

It'd be interesting to know how well the conversion goes, especially
WRT preservation of prior history. More to the point, though as you
note perforce isn't open source in any sense, be interesting to see how
that conversion compares.


> My own preference would be something like Git or Mercurial. But those
> systems come with their own set of issues when a large tree like the
> FreeBSD `src/' is the matter.

Personally I'd be leery of git for reasons of hype and the observation
that the author managed to hold off using any scm for the longest time,
ignoring for the moment the bitkeeper happenings. Also, that code is
pretty young too.

I've toyed with the idea of investigating darcs for the ``theory of
patches'' bit, but failed to get around to it. I also heard rumours
about performance problems. My SCM needs are still covered by cvs or
even plain rcs, but then I don't do FreeBSD's release engineering.

Still, it'd be nice to hear about the differences, especially about
trying to make various systems work with something as large as FreeBSD's
tree, and especially especially the RELENG side of things.

How does one preserve commit history across wildly different SCMs
anyway? :-)


One thing I'd like to add, though, is that if $NEWSCM is going to replace
CVS in base, it'd be nice to spend some thought on its applicability to
systems administration tasks, as it'd be the natural choice by virtue of
alreadying being there. Despite programmers' misgivings, RCS is still
useful for tracking a collection of individual unrelated files like, oh,
those found in /etc or /usr/local/etc.


>> Without having read that blog, I do wonder about perforce.

(Now that I have read it: It proved to be wonderfully uninformative.)


> Perforce isn't free software. It's not even merely open source. So we
> can't _require_ something like Perforce for anyone who wants to get the
> source of FreeBSD :-/

That's a given. But it's still in use, isn't it?


There are quite a few SCMs out there, the most visible being the free
ones that sprouted up like mushrooms since it became the hip thing to
create a better successor to CVS[1]. Any choice will be controversial,
so it doesn't surprise me that people sit down and write their own,
resulting in even more options. But that doesn't say anything about
whether such results will be suitable for anyone else.


[1] Guess what, despite all its (many) faults, it's been around for ages,
works, and misteaks it can't fix itself are much more fixable than,
say, trying to spit through binary databases. Automation is a good
thing, but in cases like this, continuity and fixability even more so.
I also get the feeling many are written for programmers, not release
engineers. Arguably the latter are the more important for SCM use.

On 4 Jun 2008 14:02:25 GMT, jpd <read_the_sig> wrote:
> On Tue, 03 Jun 2008 11:32:12 +0300,
> Giorgos Keramidas <keramida> wrote:
>> My own preference would be something like Git or Mercurial. But those
>> systems come with their own set of issues when a large tree like the
>> FreeBSD `src/' is the matter.
>
> Personally I'd be leery of git for reasons of hype and the observation
> that the author managed to hold off using any scm for the longest time,
> ignoring for the moment the bitkeeper happenings. Also, that code is
> pretty young too.

True. It's also a fast moving target.

> One thing I'd like to add, though, is that if $NEWSCM is going to
> replace CVS in base, it'd be nice to spend some thought on its
> applicability to systems administration tasks, as it'd be the natural
> choice by virtue of alreadying being there. Despite programmers'
> misgivings, RCS is still useful for tracking a collection of
> individual unrelated files like, oh, those found in /etc or /usr/local/etc.

RCS isn't going away, so I don't see a reason to worry about this :)

>> Perforce isn't free software. It's not even merely open source. So we
>> can't _require_ something like Perforce for anyone who wants to get the
>> source of FreeBSD :-/
>
> That's a given. But it's still in use, isn't it?

It is, but it would be nice if we could support the lightweight
branching that Perforce has in ${official-scm}. The main reason why
branching isn't used in CVS for the Perforce projects is that it's so
difficult to use branches in CVS and to keep track of stuff done in HEAD
or RELENG_XXX branches.

On Thu, 05 Jun 2008 12:58:22 +0300,
Giorgos Keramidas <keramida> wrote:
>> One thing I'd like to add, though, is that if $NEWSCM is going to
>> replace CVS in base, it'd be nice to spend some thought on its
>> applicability to systems administration tasks, as it'd be the natural
>> choice by virtue of alreadying being there. Despite programmers'
>> misgivings, RCS is still useful for tracking a collection of
>> individual unrelated files like, oh, those found in /etc or /usr/local/etc.
>
> RCS isn't going away, so I don't see a reason to worry about this :)

You may know that, but I don't. 'Sides, it's easy to forget details like
this so it's worth mentioning them even if that superficially would seem
to be superfluous.

On 5 Jun 2008 11:51:36 GMT, jpd <read_the_sig> wrote:
> On Thu, 05 Jun 2008 12:58:22 +0300,
> Giorgos Keramidas <keramida> wrote:
>>> One thing I'd like to add, though, is that if $NEWSCM is going to
>>> replace CVS in base, it'd be nice to spend some thought on its
>>> applicability to systems administration tasks, as it'd be the natural
>>> choice by virtue of alreadying being there. Despite programmers'
>>> misgivings, RCS is still useful for tracking a collection of
>>> individual unrelated files like, oh, those found in /etc or /usr/local/etc.
>>
>> RCS isn't going away, so I don't see a reason to worry about this :)
>
> You may know that, but I don't. 'Sides, it's easy to forget details like
> this so it's worth mentioning them even if that superficially would seem
> to be superfluous.

True, true :)