How do you merge two git repositories?

I’ve had the issue of needing to merge multiple git repositories into one repository with preserving the history, branches and tags. This isn’t a trivial feat, as the repositories have a large history, many branches and tags and of course names of those branches and tags overlap.

While trying to solve this issue i searched the net for solutions, often landing on a stackoverflow.com page of some kind. Gathering this information i have managed to put together my own script, which seems to solves the issue rather well.

#!/bin/bash
#
################################################################################
## Script to merge multiple git repositories into a new repository
## - The new repository will contain a folder for every merged repository
## - The script adds remotes for every project and then merges in every branch
##  and tag. These are renamed to have the origin project name as a prefix
##
## Usage: mergeGitRepositories.sh <new_project> <my_repo_urls.lst>
## - where <new_project> is the name of the new project to create
## - and <my_repo_urls.lst> is a file contaning the URLs to the respositories
## which are to be merged on separate lines.
##
## Author: Robert von Burg
## eitch@eitchnet.ch
##
## Version: 0.2.0
## Created: 2015-06-17
##
################################################################################
#

# disallow using undefined variables
shopt -s -o nounset

# Script variables
declare SCRIPT_NAME="${0##*/}"
declare SCRIPT_DIR="$(cd ${0%/*} ; pwd)"
declare ROOT_DIR="$PWD"


# Detect proper usage
if [ "$#" -ne "2" ] ; then
 echo -e "ERROR: Usage: $0 <new_project> <my_repo_urls.lst>"
 exit 1
fi


# Script functions
function failed() {
 echo -e "ERROR: Merging of projects failed:"
 echo -e "$1"
 exit 1
}

function commit_merge() {
 current_branch="$(git symbolic-ref HEAD 2>/dev/null)"
 CHANGES=$(git status | grep "working directory clean")
 MERGING=$(git status | grep "merging")
 if [[ "$CHANGES" != "" ]] && [[ "$MERGING" == "" ]] ; then
 echo -e "INFO: No commit required."
 else
 echo -e "INFO: Committing ${sub_project}..."
 if ! git commit --quiet -m "[Project] Merged branch '$1' of ${sub_project}" ; then
 failed "Failed to commit merge of branch '$1' of ${sub_project} into ${current_branch}"
 fi
 fi
}


## Script variables
PROJECT_NAME="${1}"
PROJECT_PATH="${ROOT_DIR}/${PROJECT_NAME}"
REPO_FILE="${2}"
REPO_URL_FILE="${ROOT_DIR}/${REPO_FILE}"


# Make sure the REPO_URL_FILE exists
if [ ! -e "${REPO_URL_FILE}" ] ; then
 echo -e "ERROR: Repo file ${REPO_URL_FILE} does not exist!"
 exit 1
fi


# Make sure the required directories don't exist
if [ -e "${PROJECT_PATH}" ] ; then
 echo -e "ERROR: Project ${PROJECT_NAME} already exists!"
 exit 1
fi


# create the new project
echo -e "INFO: Creating new git repository ${PROJECT_NAME}..."
echo -e "===================================================="
cd ${ROOT_DIR}
mkdir ${PROJECT_NAME}
cd ${PROJECT_NAME}
git init
echo "Initial Commit" > initial_commit
# Since this is a new repository we need to have at least one commit
# thus were we create temporary file, but we delete it again.
# Deleting it guarantees we don't have conflicts later when merging
git add initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
git rm --quiet initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
echo


# Merge all projects into th branches of this project
echo -e "INFO: Merging projects into new repository..."
echo -e "===================================================="
for url in $(cat ${REPO_URL_FILE}) ; do

 # extract the name of this project
 export sub_project=${url##*/}
 sub_project=${sub_project%*.git}

 echo -e "INFO: Project ${sub_project}"
 echo -e "----------------------------------------------------"

 # Fetch the project
 echo -e "INFO: Fetching ${sub_project}..."
 git remote add "${sub_project}" "${url}"
 if ! git fetch --no-tags --quiet ${sub_project} 2>/dev/null ; then
 failed "Failed to fetch project ${sub_project}"
 fi

 # add remote branches
 echo -e "INFO: Creating local branches for ${sub_project}..."
 while read branch ; do 
 branch_ref=$(echo $branch | tr " " "\t" | cut -f 1)
 branch_name=$(echo $branch | tr " " "\t" | cut -f 2 | cut -d / -f 3-)

 echo -e "INFO: Creating branch ${branch_name}..."

 # create and checkout new merge branch off of master
 git checkout --quiet -b "${sub_project}/${branch_name}" master
 git reset --hard --quiet
 git clean -d --force --quiet

 # Merge the project
 echo -e "INFO: Merging ${sub_project}..."
 if ! git merge --quiet --no-commit "remotes/${sub_project}/${branch_name}" 2>/dev/null ; then
 failed "Failed to merge branch 'remotes/${sub_project}/${branch_name}' from ${sub_project}"
 fi

 # And now see if we need to commit (maybe there was a merge)
 commit_merge "${sub_project}/${branch_name}"

 # relocate projects files into own directory
 if [ "$(ls)" == "${sub_project}" ] ; then
 echo -e "WARN: Not moving files in branch ${branch_name} of ${sub_project} as already only one root level."
 else
 echo -e "INFO: Moving files in branch ${branch_name} of ${sub_project} so we have a single directory..."
 mkdir ${sub_project}
 for f in $(ls -a) ; do
 if [[ "$f" == "${sub_project}" ]] || 
 [[ "$f" == "." ]] || 
 [[ "$f" == ".." ]] ; then 
 continue
 fi
 git mv -k "$f" "${sub_project}/"
 done

 # commit the moving
 if ! git commit --quiet -m "[Project] Move ${sub_project} files into sub directory" ; then
 failed "Failed to commit moving of ${sub_project} files into sub directory"
 fi
 fi
 echo
 done < <(git ls-remote --heads ${sub_project})


 # checkout master of sub probject
 if ! git checkout "${sub_project}/master" 2>/dev/null ; then
 failed "sub_project ${sub_project} is missing master branch!"
 fi

 # copy remote tags
 echo -e "INFO: Copying tags for ${sub_project}..."
 while read tag ; do 
 tag_ref=$(echo $tag | tr " " "\t" | cut -f 1)
 tag_name=$(echo $tag | tr " " "\t" | cut -f 2 | cut -d / -f 3)

 # hack for broken tag names where they are like 1.2.0^{} instead of just 1.2.0
 tag_name="${tag_name%%^*}"

 tag_new_name="${sub_project}/${tag_name}"
 echo -e "INFO: Copying tag ${tag_name} to ${tag_new_name} for ref ${tag_ref}..."
 if ! git tag "${tag_new_name}" "${tag_ref}" 2>/dev/null ; then
 echo -e "WARN: Could not copy tag ${tag_name} to ${tag_new_name} for ref ${tag_ref}"
 fi
 done < <(git ls-remote --tags ${sub_project})

 # Remove the remote to the old project
 echo -e "INFO: Removing remote ${sub_project}..."
 git remote rm ${sub_project}

 echo
done


# Now merge all project master branches into new master
git checkout --quiet master
echo -e "INFO: Merging projects master branches into new repository..."
echo -e "===================================================="
for url in $(cat ${REPO_URL_FILE}) ; do

 # extract the name of this project
 export sub_project=${url##*/}
 sub_project=${sub_project%*.git}

 echo -e "INFO: Merging ${sub_project}..."
 if ! git merge --quiet --no-commit "${sub_project}/master" 2>/dev/null ; then
 failed "Failed to merge branch ${sub_project}/master into master"
 fi

 # And now see if we need to commit (maybe there was a merge)
 commit_merge "${sub_project}/master"

 echo
done


# Done
cd ${ROOT_DIR}
echo -e "INFO: Done."
echo

exit 0

Java Equals and HashCode by Reflection

Writing proper ”equals()” and ”hashCode()” methods
A class which is to be used in Maps and Sets must always have their equals() and hashCode() method overridden so that they reflect the data structure of the actual object.

This means that the fields of the object which are required for equality must be taken into account when overriding these methods. The JavaDoc of Object.equals() and Object.hashCode() go into further detail, but it remains to be mentioned that if a field has a unique identifier then this will mostly suffice for the equality implementation.

When using the Eclipse IDE, implementing the equals() and hashCode() method are a matter of:

  • Opening the context menu in the class’s source editor
  • Opening the Source menu
  • Selecting Generate hashCode() and equals()…
  • Select on the fields required for uniqueness
  • Further configure the dialog and then selecting ok

Note: For further information, see Joshua Bloch’s book Effective Java which describes this issue further.

Performance issues
Apache Commons has two classes HashCodeBuilder and EqualsBuilder as an easy way of implementing these methods. These classes use reflection to dynamically build the hashCode or check equality of two instances.

They should be avoided as they can have a serious impact on performance as the following test will show.

Test Class:

  public static class TestClass {
    public static final int GENERATED = 0;
    public static final int REFLECTION = 1;
    public static final int COMMONS_BUILDER = 2;
    public static int hashCodeBuilding = GENERATED;
    private int a;
    private int b;
    private String c;
    private String d;

    public TestClass(final int a, final int b, final String c, final String d) {
      this.a = a;
      this.b = b;
      this.c = c;
      this.d = d;
    }

    @Override
    public int hashCode() {
      if (hashCodeBuilding == REFLECTION) {
        return HashCodeBuilder.reflectionHashCode(this);
      }
      if (hashCodeBuilding == COMMONS_BUILDER) {
        return new HashCodeBuilder()
                      .append(this.a)
                      .append(this.b)
                      .append(this.c)
                      .append(this.d)
                      .toHashCode();
      }

      final int prime = 31;
      int result = 1;
      result = prime * result + this.a;
      result = prime * result + this.b;
      result = prime * result + ((this.c == null) ? 0 : this.c.hashCode());
      result = prime * result + ((this.d == null) ? 0 : this.d.hashCode());
      return result;
    }

    @Override
    public boolean equals(final Object obj) {

      if (hashCodeBuilding == REFLECTION) {
        return EqualsBuilder.reflectionEquals(this, obj);
      }

      if (this == obj) {
        return true;
      }
      if (obj == null) {
        return false;
      }
      if (getClass() != obj.getClass()) {
        return false;
      }

      TestClass other = (TestClass) obj;
      if (hashCodeBuilding == COMMONS_BUILDER) {
        return new EqualsBuilder().append(this.a, other.a)
                     .append(this.b, other.b)
                     .append(this.c, other.c)
                     .append(this.d, other.d)
                     .isEquals();
      }

      if (this.a != other.a) {
        return false;
      }

      if (this.b != other.b) {
        return false;
      }

      if (this.c == null) {
        if (other.c != null) {
          return false;
        }
      }
      else if (!this.c.equals(other.c)) {
        return false;
      }

      if (this.d == null) {
        if (other.d != null) {
          return false;
        }
      }
      else if (!this.d.equals(other.d)) {
        return false;
      }

      return true;
    }
  }

JUnit Test method:

  @Test
  public void hashCodePerformanceTest() {
    TestClass a = new TestClass(22, 11, "bla bla", "no no");
    TestClass b = new TestClass(52, 63, "dfgdfgdfg", "xv3dfg1");

    System.out.println("First pass...");
    runTest(a, b, TestClass.REFLECTION);
    runTest(a, b, TestClass.COMMONS_BUILDER);
    runTest(a, b, TestClass.GENERATED);

    System.out.println("Second pass...");
    runTest(a, b, TestClass.REFLECTION);
    runTest(a, b, TestClass.COMMONS_BUILDER);
    runTest(a, b, TestClass.GENERATED);

    System.out.println("Third pass...");
    runTest(a, b, TestClass.REFLECTION);
    runTest(a, b, TestClass.COMMONS_BUILDER);
    runTest(a, b, TestClass.GENERATED);

    System.out.println("Fourth pass...");
    runTest(a, b, TestClass.REFLECTION);
    runTest(a, b, TestClass.COMMONS_BUILDER);
    runTest(a, b, TestClass.GENERATED);

    System.out.println("Fifth pass...");
    runTest(a, b, TestClass.REFLECTION);
    runTest(a, b, TestClass.COMMONS_BUILDER);
    runTest(a, b, TestClass.GENERATED);
  }

  private void runTest(final TestClass a, final TestClass b, final int hashCodeBuilding) {
    TestClass.hashCodeBuilding = hashCodeBuilding;
    long start = System.nanoTime();
    for (int i = 0; i < 10000; i++) {
      a.equals(b);
      a.hashCode();
    }

    long time = System.nanoTime() - start;
    if (hashCodeBuilding == TestClass.REFLECTION) {
      String msg = String.format("With reflection took %.3f ms", (time / 1000000.0D));
      System.out.println(msg);
    }
    else if (hashCodeBuilding == TestClass.COMMONS_BUILDER) {
      String msg = String.format("With apache commons builder took %.3f ms", (time / 1000000.0D));
      System.out.println(msg);
    }
    else {
      String msg = String.format("With eclipse generated took %.3f ms", (time / 1000000.0D));
      System.out.println(msg);
    }
  }

Test result:

First pass...
With reflection took 72.796 ms
With apache commons builder took 1.689 ms
With eclipse generated took 0.607 ms
Second pass...
With reflection took 48.158 ms
With apache commons builder took 1.199 ms
With eclipse generated took 0.151 ms
Third pass...
With reflection took 48.240 ms
With apache commons builder took 1.095 ms
With eclipse generated took 0.151 ms
Fourth pass...
With reflection took 47.274 ms
With apache commons builder took 2.264 ms
With eclipse generated took 0.150 ms
Fifth pass...
With reflection took 46.475 ms
With apache commons builder took 1.199 ms
With eclipse generated took 0.160 ms

Postfix Satellite configuration: Remote SMTP with auth

If you have ever run into the problem that your computer will not send e-mails directly, then this might help.

Install Postfix and configure it as a satellite server. This was done on a Ubuntu desktop installation, so I don’t know what Fedora and co. will say.

In the /etc/postfix/main.cf file add or modify the following lines:

relayhost = smtp.gateway.ch
smtp_sasl_auth_enable = yes
smtpd_tls_auth_only = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous

This configures postfix to relay all e-mails over the defined host. Now the sasl_passwd file must be created with the credentials for the SMTP gateway:

smtp.gateway.ch user@gateway.ch:password

Now create the map of the password file and reload postfix:

sudo postmap /etc/postfix/sasl_passwd
sudo /etc/init.d/postfix reload

Test the configuration by sending a test e-mail:

echo -e "From: someone@somewhere.ch\nTo: some@otherplace.com\nSubject: Hello World\n\Body: \nHello beautiful world" | sendmail -t

And see what postfix said:

tail /var/log/mail.info

Have fun!

Makefile to compile C code

During my phases where I learn to write in C, I don’t want to keep creating new Makefiles. Thus after searching all over the net I managed to put together this simple Makefile. It expects your source code to be in ./src and it will put the compiled executables in ./bin

#
# Makefile to compile arbitrary C sources to an executable with the same name
# The source files are in the src/ sub directory
# The compiled executables are in the bin/ sub directory
#

# $@ is automatic variable that holds name of target
# $< is automatic variable that holds the name of the prerequisite

# defines the sub directory in which the sources are found
SRC_PATH=./src/
# defines the sub directory in which the executables are compiled to
BIN_PATH=./bin/

# tells make to search in $(SRC_PATH) for the .c files
VPATH = src:$(SRC_PATH)

# defines the name of the executables which are to be compiled
EXECUTABLES= $(patsubst $(SRC_PATH)%.c,%,$(shell ls $(SRC_PATH)*.c))

# default target which prepares the bin/ sub directory, and initiates the compilation
all: prepare $(EXECUTABLES)

# the target which is expanded to compile each found C source file
$(EXECUTABLES): %: %.c
	gcc $< -lm -o $(BIN_PATH)$@

# declare phony targets
.PHONY: prepare clean

# create the $(BIN_PATH) sub directory
prepare:
	mkdir -p $(BIN_PATH)
# remote the $(BIN_PATH) subdirectory
clean:
	rm -rf $(BIN_PATH)

Adding new LaTeX packages and styles

This is a quick explanation on how to install new LaTeX styles so that you can use them, but without having to change your system.

  • First download a style from CTAN
  • Then unpack the zip somewhere in your home folder
  • Copy the styles/packages to ~/texmf/tex
  • Run the command mktexlsr or texhash
  • Use the package and it should work

This helped me: StackOverflow

Ubuntu Netboot

Notes on getting Ubuntu netboot to work so that I don’t have to burn any CD’s

This document expects a DHCP server to be available in the network

Install tftp

sudo aptitude install tftpd-hpa tftp-hpa

Get netboot files:

cd /var/lib/tftpboot
sudo wget -r -nH -np --cut-dirs=8 http://archive.ubuntu.com/\
ubuntu/dists/maverick/main/installer-amd64/\
current/images/netboot
find ./ -name "index.*" | sudo xargs rm

Add the following to the DHCP subnet configuration:

next-server 10.0.0.68;
filename "pxelinux.0";

Now normal netboot should work.

If one wants to have liveCD, then carry on by installing NFS-Server:

sudo aptitude install nfs-kernel-server

Add export for the livecd in /etc/exports:

/var/lib/tftpboot/maverick 10.0.0.0/255.255.255.0(async,no_root_squash,no_subtree_check,ro)

Mount and copy liveCD contents to exported NFS share:

sudo mount -o loop /tmp/maverick-desktop-amd64.iso /mnt
sudo cp -ar /mnt/* /var/lib/tftpboot/maverick/

Add a new menu entry in ubuntu-installer/amd64/boot-screens/rqtxt.cfg

label maverickLiveCD
menu label ^Live CD
kernel maverick/casper/vmlinuz
append vga=788 boot=casper netboot=nfs \
nfsroot=10.0.0.68/var/lib/tftpboot/maverick initrd=maverick/casper/initrd.lz --

Restart NFS:

sudo /etc/init.d/nfs-kernel-server reload

If problems arise, try using the following boot append line:

append file=/var/lib/tftpboot/maverick/preseed/ubuntu.seed vga=788 \
boot=casper root=/dev/nfs netboot=nfs \
nfsroot=10.0.0.68:/var/lib/tftpboot/maverick initrd=maverick/casper/initrd.lz --

VirtualBox resolution problems in a headless environment

In certain cases running a VirtualBox VM in a headless linux environment can lead to a Windows XP having an undesired resolution.

I fixed this using the following two commands:

Global command:
VBoxManage setextradata global GUI/MaxGuestResolution 1920,1200

For a running VM:
VBoxManage controlvm "remote" setvideomodehint 1920 1200 24

I found the resolution in this forum:
VirtualBox Forums

aptitude search vs apt-cache search

Over the years I have grown used to the fact that when I search for a package on a debian based distro, then I use:
apt-cache search foo bar
The result was a list of all the packages which had the text ‘foo’ and ‘bar’ in the description.

Lately it has been discouraged to use apt-get and apt-cache, but one should rather use aptitude as this tool is more powerful.

It is more powerful, but it does not behave the same. So the following command:
aptitude search foo bar
would return any result where either ‘foo’ or ‘bar’ was in the package name, not the description.

Since I have gotten used to apt-cache way this is how I do it with aptitude:
aptitude search "~dfoo bar"

But! This command does not check package names. Package names are with ~n. Maybe someone out there can explain how I can combine the two correctly?

Debugging And Logging Java RMI Calls

If you ever need to debug Java RMI, then add these properties to the VM configuration:

-Djava.rmi.server.logCalls=true
-Dsun.rmi.server.logLevel=VERBOSE
-Dsun.rmi.client.logCalls=true
-Dsun.rmi.transport.tcp.logLevel=VERBOSE

Determining the base directory of a bash script

While writing bash scripts, I find it a good idea to put reusable functions into their own scripts and then source them into other scripts.

This means that one must use the source /file/to/source call to load these files. To be able to do this, the bash script must know where the scripts to source are. To not have to hardwire this into the script, I just have the requirement, that the script must be in the same directory as the script itself.

The following code shows how one can find the base directory, with out using any external commands and it should work with out regards to how the script is called:

SCRIPT_NAME="${0##*/}"
SCRIPT_DIR="${0%/*}"

# if the script was started from the base directory, then the 
# expansion returns a period
if test "$SCRIPT_DIR" == "." ; then
  SCRIPT_DIR="$PWD"
# if the script was not called with an absolute path, then we need to add the 
# current working directory to the relative path of the script
elif test "${SCRIPT_DIR:0:1}" != "/" ; then
  SCRIPT_DIR="$PWD/$SCRIPT_DIR"
fi

# now we can source our other scripts
source $SCRIPT_DIR/logger.sh