blog.mhartl | Michael Hartl's tech blog

2008-10-14

Setting up your Git repositories for open source projects at GitHub

Filed under: Git, Insoshi — long @ 12:08

[This is a guest post from Long Nguyen. —mhartl]

Like a lot of projects in the Ruby on Rails world, the Insoshi social networking platform uses Git and GitHub to manage its open source development and contributions. In setting up the repositories for Insoshi, I’ve applied the version control experience I gained at Discover, where I was technical lead for the software configuration management (SCM) team. Since some aspects of our setup aren’t obvious if you haven’t managed large projects before, we at Insoshi decided to share the details so that other GitHub projects might benefit as well.

We’ll start by reviewing the typical Git workflow based on pull requests, then discuss some problems you might run into with a “typical” repository setup, and finally explain the details of preparing the Insoshi Git repository for collaboration.

Why Pull Requests?

Git was originally developed by Linus Torvalds to host the Linux kernel, and pull requests are the de-facto standard for submitting contributions in Git because that’s what Linus does. (He talked about this in his Google Tech Talk on Git.) The concept of the pull request is straightforward: You notify someone that you’ve made an update via email, messaging on GitHub, etc. and let them know where to find it. They can then pull in your changes and merge it with their work.

Except for that interaction, everyone works within their own repository and on their own schedule. There’s no process waiting to be completed that blocks you from moving on to whatever you need/want to do next. And you’re not forcing anyone to drop what they’re doing to right now to handle your request.

It’s all very polite. And it works well in the context of distributed development since you avoid all kinds of coordination issues.

What’s needed?

If you want to contribute to an open source project, here’s really all that you need:

  1. A publicly accessible repository where your changes can be found
  2. A local repository for your development

Even if you’re new to Git, these both seem like pretty straightforward things to do—especially if you’re using GitHub for the public repository: your repository is just a fork of the main project repository.

Let’s set up our repository by going to the official Insoshi repository and clicking on the fork button:

I’ll need make note of the public clone URL for the official repository and my private clone URL for my newly created fork:

  • Official Insoshi public clone URL
    git://github.com/insoshi/insoshi.git
  • My fork’s private clone URL
    git@github.com:long/insoshi.git

Your local repository: The “obvious” thing to do

At this point, I’ll be tempted to go ahead and make a local clone of my fork:

$ git clone git@github.com:long/insoshi.git

and immediately get to work.

Technically, there’s nothing wrong with that. And as an individual developer starting a new project, it’s what you do, but there are several disadvantages to this seemingly straightforward approach. One of the major benefits of a distributed version control system like Git is that each repository is on an equal footing; in particular, we would like every fork to have the same master branch, so that if the “official” Insoshi repository should ever be lost there would be plenty of redundant backups. We also want it to be easy for each developer to pull in changes from the official repository; the “obvious” approach isn’t set up for that. Finally, it’s a bad idea in general to work on the master branch; experienced Git users typically work on separate development branches and then merge those branches into master when they’re done.

What we’d like is a way to connect up the local repository in a way that will

  • Keep the repositories in sync so that each contains the full “official” repository
  • Allow developers to pull in official updates
  • Encourage working on branches other than master

In the “obvious” configuration, I’m not set up to do any of that:

  • There’s no local connection to the official repository for updates
  • There’s no mechanism in place to push official updates to my fork on GitHub
  • We’re working directly on the master branch

Your local repository: The “right” way

Keeping the big picture in mind, here are the commands I’ve run to set up my local repository (using the GitHub id long):

$ git clone git://github.com/insoshi/insoshi.git
$ cd insoshi
$ git branch --track edge origin/edge
$ git branch long edge
$ git checkout long
$ git remote add long git@github.com:long/insoshi.git
$ git fetch long
$ git push long long:refs/heads/long
$ git config branch.long.remote long
$ git config branch.long.merge refs/heads/long

Let’s take a detailed look at what these steps accomplish.

So what does it all mean?

Step one

Create a local clone of the Insoshi repository:

$ git clone git://github.com/insoshi/insoshi.git

You should note that the Git URL for the clone references the official Insoshi repository and not the URL of my own fork (i.e., the clone URL is git://github.com/insoshi/insoshi.git instead of git@github.com:long/insoshi.git). This way, the official repository is the default remote (aka ‘origin’), and the local master branch tracks the official master.

Step two

I have to change into the repository to perform additional git setup:

$ cd insoshi
Step three

Insoshi also has an ‘edge’ branch for changes that we want to make public but may require a bit more polishing before we’d consider them production-ready (in the past this has included migrating to Rails 2.1 and Sphinx/Ultrasphinx).  Our typical development lifecycle looks something like

development -> edge -> master

I want to create a local tracking branch for it:

$ git branch --track edge origin/edge
Steps four and five

As I mentioned before, I’m resisting the temptation to immediately start working on the local ‘master’ and ‘edge’ branches. I want to keep those in sync with the official Insoshi repository.

I’ll keep my changes separate by creating a new branch ‘long’ that’s based off edge and checking it out:

$ git branch long edge
$ git checkout long

By the way, you can actually combine the two commands if you like, using just the ‘git checkout’ command with the -b flag:

$ git checkout -b long edge

You can name this branch anything that you want, but I’ve chosen my GitHub id so that it’s easy to identify.

I’m starting my changes off of ‘edge’ since that contains all the latest updates and any contribution I submit a pull request for will be merged first into the official Insoshi ‘edge’ branch to allow for public testing before it’s merged into the ‘master’.

Steps six and seven

I’m finally adding the remote reference to my fork on GitHub:

$ git remote add long git@github.com:long/insoshi.git

I’ve used my GitHub id once again, this time as the remote nickname.

We should run a fetch immediately in order to sync up the local repository with the fork:

$ git fetch long
Step eight

I’m pushing up my new local branch up to my fork. Since it’ll be a new branch on the remote end, I need to fully specify the remote refspec:

$ git push long long:refs/heads/long
Steps nine and ten

Now that the new branch is up on my fork, I want to set the branch configuration to track it:

$ git config branch.long.remote long
$ git config branch.long.merge refs/heads/long

Setting the remote lets me just simply use

$ git push

to push changes on my development branch up to my fork

Setting the merge configuration is mainly for completeness at this point. But if you end up working on more than one machine (work/home, desktop/laptop, etc.), it’ll allow you to just use

$ git pull

to grab the changes you’ve pushed up to your fork.

Isn’t that a lot of extra work to do?

This may seem like a lot work up front, but it’s all configuration work that you’d eventually do anyway. If you’re really that concerned about the extra typing, I’ve got a shell script for you.

The extra work is worth the effort, because with this configuration

  • My changes will be easily identifiable in my named branch
  • I can easily get updates from the main Insoshi repository
  • Any updates I’ve pulled into master and edge are automatically pushed up to my fork on GitHub

The last one is a bonus because the default refspec for remotes is refs/heads/*:refs/heads/*. This means that the simple ‘git push’ command will push up changes for all local branches that have a matching branch on the remote. And if I make it a point to pull in updates to my local master and edge but not work directly on them, my fork will match up with the official repository.

So what is the benefit of all this to open source projects like Insoshi?

  • The easier it is for the contributor to pull in updates, the more likely it will be that the pull request will be for code that merges easily with the latest releases (with few conflicts)
  • You can tell if someone is pulling updates by looking at their master and edge branches and seeing if they match up with the latest branches on the main repository
  • By getting contributors in the habit of working on branches, you’re going to get better organized code contributions

Basically, the less effort that’s required to bring in code via a pull request, the sooner it can be added to the project release. And at the end of the day, that’s really what it’s all about.

Putting (pushing and pulling) it all together

Now that we’ve covered all the details, let’s go through the full set of steps needed to make a contribution to a project like Insoshi:

  1. Fork the Insoshi repository on GitHub:

  2. Follow the Git steps above or use the shell script to set up your local repository
  3. Checkout the local branch, just to be sure:
    $ git checkout long
  4. Make some changes (and remember your development branch is against ‘edge’) and commit them:
    [make changes in a text editor]
    $ git commit -m "My great contribution"
    $ git push
  5. Go to your fork and branch at GitHub (I’m at long/insoshi @ long) and click on the pull request button:

  6. Tell us about what you just did and make sure “insoshi” is a recipient:
  7. Bask in the glory of being an open-source contributor!
About these ads

32 Comments

  1. […] disso ai, então fui as pesquisas, demorei bastante até entrar em um gist que me levou até um post do blog do Nathaniel Felsen, e ai consegui resolver meus problemas. ok vamos […]

    Pingback by Git: Participando de um projeto OpenSource no GitHub - Comandos « .xX [_ MaxOnRails _] Xx. — 2008-10-18 @ 05:41

  2. Thank you very much for this insightful tutorial. I will remember the next time I fork a project.

    Greetings

    Comment by Patrick — 2008-10-23 @ 02:27

  3. […] Setting up your Git repositories for open source projects at GitHub « Insoshi Ruby on Rails bl… – "In setting up the repositories for Insoshi, I’ve applied the version control experience I gained at Discover, where I was technical lead for the software configuration management (SCM) team. Since some aspects of our setup aren’t obvious if you haven’t managed large projects before, we at Insoshi decided to share the details so that other GitHub projects might benefit as well." […]

    Pingback by Bookmarks for October 22nd through October 23rd | Bieber Labs — 2008-10-23 @ 06:04

  4. Thanks for this great post!

    We’ve found that GitHub-gem http://github.com/defunkt/github-gem/tree/master makes it less necessary about leaving master and edge to track main branches.

    The gem’s a command-line tool that automates keeping contributors’ masters (elstudio/master, say), separate from yours. The readme at that URL has details.

    Comment by Eric — 2008-10-30 @ 10:47

  5. This makes a lot of sense.

    Can you tell us more about your release process? On which branches would you run continuous integration, or a staging deployment? Before or after merging?

    How would you suggest working with a project that is not maintaining an edge branch?

    Comment by Vince — 2008-11-1 @ 03:38

  6. How can push over http? I am behind http proxy!

    I get
    Error: no DAV locking support on remote repo http://github.com/dilipm/gitscm.git

    Comment by dilipm — 2008-11-6 @ 07:45

  7. @Eric: I’ve just started taking a look at the GitHub gem. Part of our setup isn’t just local management of the master and edge branches but how they get pushed up to your fork. I need to see how what the gem does fits into that process and how to incorporate what we’re doing into it.

    @Vince: We don’t have any specific rule about what gets released out to where. A lot of it depends on how significant the changes that we’re working on are. A general rule is that if we need to check to see if something will play nice with an existing site (clean migrations, etc.), we release to staging (where we’ve copied either the demo or dogfood site). Edge changes get released to demo first (since we can restore or refresh the database at any time). Once we’ve let the edge changes run in demo for a while, we merge it to master and release it to dogfood.

    @dilipm: GitHub doesn’t allow pushes via http (http://github.com/guides/dealing-with-errors-when-pushing). I can understand why: it’s a completely separate security model from what they have set up (either under the git protocol or ssh git@github:…). You might be able to set up a ssh tunnel via an http proxy. I haven’t tried it before so I can’t tell exactly what to do. But this looks like a good resource to start from: http://www.mtu.net/~engstrom/ssh-proxy.php.

    Comment by long — 2008-11-6 @ 15:24

  8. Thank you for this. This post is excellent except for the code example, which is a bit too obfuscated for my taste. Do you name everything in your life after yourself? ;)

    Here’s a version in which GitHub user “mechfish” forks Insoshi to a GitHub-hosted fork known as “fork”, creating a “devel” branch on the local repository and pushing that up to the fork:

    $ git clone git://github.com/insoshi/insoshi.git

    $ cd insoshi

    $ git branch devel master

    $ git checkout devel

    $ git remote add fork git@github.com:mechfish/insoshi.git

    $ git fetch fork

    $ git push fork devel:refs/heads/devel

    $ git config branch.devel.remote fork

    $ git config branch.devel.merge refs/heads/devel

    I also created a paraphrased version of this post on my blog at http://mechanicalrobotfish.com/posts/127-checking-out-open-source-git-project-right-way

    Comment by mechfish — 2008-11-7 @ 11:08

  9. Great stuff! Thanks a lot! You just wrapped exactly what I needed to know 1hr after starting to work with Git/GitHub.

    I took your script and modified it to be able to work with any GitHub project. It’s available at http://github.com/colinsurprenant/localforkedclone/

    Thanks,
    Colin.

    Comment by Colin Surprenant — 2008-11-21 @ 17:45

  10. […] Setting up your Git repositories for open source projects at GitHub […]

    Pingback by Useful Git Links — 2009-02-12 @ 18:25

  11. […] I still don’t consider myself an expert – especially when it comes to truly distributed collaboration – I did manage to pass on some basic knowledge to those around me just getting […]

    Pingback by » Git Basics: Getting Started FND’s Blag: Just Another Personal Wobsite — 2009-03-28 @ 03:48

  12. Sweet blog. I never know what I am going to come across next. I think you should do more posting as you have some pretty intelligent stuff to say.

    I’ll be watching you . :)

    Comment by effegioxy — 2009-06-3 @ 22:38

  13. […] Setting up your Git repositories for open source projects at GitHub […]

    Pingback by Mindstab.net » Blog Archive » Setting up a remote git repository with just git — 2009-07-25 @ 11:57

  14. […] used method. I have chosen to use a different approach that I originally saw on an excellent blog post by Long Nguyen. This version builds on Long’s work by making it a bit more generic by using the […]

    Pingback by Fork It! – Contributing to FLOSS Projects using Git and GitHub « kevin g fourie’s personal provoker — 2009-08-19 @ 04:18

  15. Hi Long
    Thanks for the post. It was a really helpful guide for me getting up and running as a contributor.
    It would be great to see a follow up post that outlines the flow from the project maintainer side of things – receiving pull requests and merging them back into the main project.
    Best,
    Joseph

    Comment by Joseph Sofaer — 2009-10-19 @ 14:34

  16. Hi swaroop here.I want to know how to write Git proxy settings.The code look like this (what i have written)
    #!/bin/sh
    # Put your own values
    PROXY_IP=10.114.1.17
    PROXY_PORT=3128

    nc -x${PROXY_IP}:${PROXY_PORT} -X5 $*.
    when i clone git there is an error saying
    git clone git://gitorious.org/gumstix-oe/mainline.git org.openembedded.dev
    fatal: destination directory ‘org.openembedded.dev’ already exists.

    i think the error may be in above script

    Comment by swaroop — 2009-10-23 @ 22:28

  17. Awesome post. I’m going to be setting up a project on github for a school class next semester and I was curious how I could manage this exact thing simply and effectively. (My git experience thus far has been solo usage.)

    Comment by Jach — 2009-11-28 @ 06:11

  18. […] Setting up your Git repositories for open source projects at GitHub […]

    Pingback by Destillat KW49-2009 | duetsch.info - GNU/Linux, Open Source, Softwareentwicklung, Selbstmanagement, Vim ... — 2009-12-4 @ 01:44

  19. What is the license on that shell script? (I want to copy it, edit, and make the copy public.)

    Comment by Julian Morrison — 2009-12-6 @ 11:19

  20. this is wonderful tutorial .. i read it 3 times and get a fantastic results and sure i put a
    copy of this lesson on my site here

    Comment by saher — 2009-12-15 @ 10:55

  21. thank you this is my 1 time that i approuve my sucess

    Comment by g0g0 — 2010-01-1 @ 14:22

  22. very nice lesson .. I will change my logo for http://www.saudicol.com or forums.saudicol.com as the logo in this tutorial

    Comment by saudicol — 2010-01-2 @ 22:53

  23. very nice lesson >> iwill ghange in my for http://www.almnaheel.com/vb thanks

    Comment by almnaheel.com — 2010-01-8 @ 10:26

  24. […] Setting up your Git repositories for open source projects at GitHub (41): Mostly so pulling and pushing works correctly; and pull requests are ok […]

    Pingback by git top links: 2010-1 « git blog — 2010-01-13 @ 16:44

  25. ولا انا فاهم اى حاجة خالص

    Comment by توب عرب — 2010-01-28 @ 06:41

  26. this is wonderful tutorial .. i read it 3 times and get a fantastic results and sure i put a copy of this lesson on my site here

    http://www.azhar2day.com/vb

    Comment by azhar2day — 2010-02-11 @ 07:16

  27. good ….

    Comment by كول — 2010-12-11 @ 14:41

  28. Thank you on the subject …

    And to the best

    Comment by منتدى كول — 2010-12-11 @ 14:42

  29. شكرا
    iwill ghange in my for

    http://www.xcoolx.com or forums.saudicol.com as the logo in this tutorial

    Comment by منتديات كول — 2010-12-13 @ 06:10

  30. very nice lesson

    Comment by دردشة عراقية — 2011-06-28 @ 22:57

  31. […] Caso você queira aprender melhor como trabalhar com o GitHub, além de você ter estudado tudo que foi dito dito acima, indicamos esta leitura:http://blog.mhartl.com/2008/10/14/setting-up-your-git-repositories-for-open-source-projects-at-githu… […]

    Pingback by Iniciando com GIT | Edivan Camargo - SEO - jQuery - Plugins jQuery - PHP - Ruby on Rails | Edivan Camargo - SEO - jQuery - Plugins jQuery - PHP - Ruby on Rails — 2012-10-9 @ 17:40


RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

The Shocking Blue Green Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 30 other followers

%d bloggers like this: