
Subject:
Merging Subversion repos
From:
Paul Kimpel <paul.kimpel@digm.com>
Date:
2013-03-07 7:32 PM
To:
Nigel Williams <nw@retrocomputingtasmania.com>

As we previously discussed, I have replaced the Subversion repository in our Google Code project with the merged product of the GC repo as of 5 March and the emulator, web UI, tools, and DCMCP portions of my the private repo where I've been doing more or less daily snapshot check-ins. The primary purpose of this exercise is to make the emulator work public, and to preserve the emulator and web UI development history as part of the project's repo.

This seems like a useful technique for other cases, so I've made notes of how I've gone about it, as follows.

First, download the Windows version of svndumptool-0.6.1. This is a python package, there is a version that is compiled into a Windows executable, so you don't even need Python on your system to use it.

http://svn.borg.ch/svndumptool/0.6.1/svndumptool-0.6.1.zip

Current versions of TortoiseSVN include the SVN command-line tools, so I just used svnrdump and the other SVN commands from that package.

Dump the Google Code repo (this generated an 11MB file):

svnrdump dump https://retro-b5500.googlecode.com/svn > retro-b5500.20130305.dump

Dump my private repo (this generated a 425MB file):

svnrdump dump svn://S1/B5500-WIP > WIP-S1.20130305.dump

These are version-3 SVN dumps (as with the --deltas option), but svndumptool can only handle version-2 dumps, so I simply created temporary repos, loaded the svnrdump files into them, and used svnadmin to create v2 dumps from the temporary repos -- retro.dump and wip.dump.

svnadmin create RETRO
svnadmin load RETRO < retro.b5500.20130305.dump
svnadmin dump RETRO > retro.dump

svnadmin create WIP
svnadmin load WIP < WIP-S1.20130305.dump
svnadmin dump WIP > wip.dump

Next, I used svndumptool to verify each of those dumps, and in particular that they had monotonically ascending revision timestamps, e.g.,

svndumptool-0.6.1\svndumptool check -v -A retro.dump

Even when requesting verbose output, it simply outputs "OK".

The S1-WIP dump contains more nodes that I want to migrate over to GC, so I ran svndumpfilter against that dump to extract just the stuff I wanted (line below folded for clarity):

svndumpfilter --drop-empty-revs --renumber-revs include 
    trunk/emulator trunk/webUI trunk/tools trunk/DCMCP.esp_m
    < wip.dump 
    > wip-filtered.dump 2> filter.log

The log output of the filtering process is in the attached filter.log file.

Now for the interesting part: svndumptool can merge a series of dumps by revision timestamp (line again folded for clarity):

svndumptool-0.6.1\svndumptool merge 
    -i retro.dump 
        -r trunk/tools/B5500LibMaintExtract.html trunk/tools/B5500LibMaintExtract-1.html
    -i wip-filtered.dump -x trunk -x tags -x branches -x trunk/tools
        -r trunk/DCMCP.esp_m trunk/SYMBOL/DCMCP.esp_m
    -o merged.dump > merge.log

The log output of the merging process is in the attached merge.log file.

Because both repos had mkdirs for the root directories, the merged dump will have multiple mkdir transactions for them, which will cause a conflict when the merged dump is loaded into another repo. The "-x" option excludes the mkdirs for the named directories from the preceding input file.

The "-r" option renames a path as it is extracted from one of the input dumps. In the case of B5500LibMaintExtract.html, I had that file in both dumps. The better revision history was in wip.dump, so I renamed the one from retro.dump, and will eventually delete it. In the case of DCMCP.esp_m, I renamed it to match the folder where it should be located in the GC repo. There are also options to do regex replacements on file paths, but I didn't need those.

Next, I created a new temporary repo for the merged data so I can verify it before sending it up to GC:

svnadmin create MERGED
svnadmin load MERGED < merged.dump > merge-load.log

The log output from that process in in merge-load.log. After checking out this temp repo into a WC and inspecting it with TortoiseSVN, it looks as if the merge was successful, and everything I wanted from both repos is now included on the merged.dump file in the proper chronological sequence. Also, the UUID of the merged repo is the same as that for the original GC repo, so anyone who has previously connected to the repo will still be looking at the "same" repo, although many of the revision numbers will have changed.

Now for the scary part. On the Admin>Source tab of the GC web interface is a link to reset the repo. I think this effective does a svnadmin create on the repo. The site really tries to discourage you from doing this, and I had to lie to it about why I was doing it. But it worked.

I tried uploading the dump of the new merged repo using svnrsync, but that didn't work due to a missing or non-permissive pre-revprop-change hook. Instead I used svnsync, as documented in the GC Subversion FAQ page (lines again folded for clarity):

svnsync init --username paul.kimpel@digm.com 
    https://retro-b5500.googlecode.com/svn
    file:///C:/Users/Paul/Desktop/svndumptool-0.6.1/MERGED
svnsync sync --username paul.kimpel@digm.com
    https://retro-b5500.googlecode.com/svn > sync.log

I've temporarily uploaded a zipped copy of the original GC repo dump as a deprecated download to the GC project Downloads page. We can get rid of that in a few weeks after we are sure the new merged repo is stable. I'll also commit these notes and the log files to the new repo so we have them in the archive. At some point, this might make a good Wiki page. I also have all of the dumps and intermediate files from the merge process on file here.

The new repo looks good, but you will need to blow away any existing WC and check out a fresh one from the repo, as the UUID has changed, as well as the revision numbering.

Attachments:
MergingSVNReposDocs.zip	12.9 KB