There are many blog posts on the topic of effective Git workflows, SO questions and answers, BitBucket tutorials and GitHub guides and an article in the BBC. So why another post on git workflow? None of these workflows seemed right for us, but recently it's just clicked, and I feel like we've finally found the process that works for us. The key was finding the simplest workflow that included the most valuable best practices. In particular, we found that complicated multi-branch strategies were unnecessary, but test driven development (TDD) and continuous integration (CI) were a must.
Setting up Remotes
We start with the assumption that all of collaborators fork the upstream repository to their personal profile. Then each person clones their profile to their laptop as origin and adds another remote pointing to the upstream repository. For convenience, they may also create remotes to the forks of their most frequent collaborators.
[myusername@mycomputer ~/Projects] $ git clone firstname.lastname@example.org:myusername/myrepo.git [myusername@mycomputer ~/Projects] $ cd myrepo [myusername@mycomputer ~/Projects/myrepo] $ git remote add upstream email@example.com:mycompany/myrepo.git [myusername@mycomputer ~/Projects/myrepo] $ git remote add mycollaborator firstname.lastname@example.org:mycollaborator/myrepo.git [myusername@mycomputer ~/Projects/myrepo] $ git remote show origin upstream mycollaborator
The next assumption is that we all keep our version of master synchronized with upstream master. And we never work out of our own master branch! Basically this means at the start of any new work we do the following:
- I like to do
git fetch --allto get the lay of the land. This combined with
git log --all --graph --date=short --pretty=format:"%ad %h %s%d [%an]"
let's me know what everyone is working on, assuming that I've made remotes to their forks.
- Then I pull from upstream master to get the latest nightly or release,
- and push to origin master to keep my fork current.
Recommended Project Layout
I'm also going to assume that everyone is following the recommended project layout. This means that their project has all dependencies listed in
requirements.txt, is developed and deployed in its own virtual environment, includes testing and documentation that aims for >80% coverage, has a boilerplate design that allows testing, documentation and package data to be bundled into a distribution and enables use with a test runner with self discovery, and is written with docstrings for autodocumentation. Nothing is ever perfect, so being diligent of path clashes, aware of the arcana of Mac OS X1 or Windows2 and able to use Stack Overflow to find answers is still important.
Branching, Testing, Pull Requests and Collaboration
- Now I switch to a new feature branch with a meaningful name - I'll delete this branch everywhere later so it can be verbose.
- The very first code I write is a test or two that demonstrates more or less exactly what we want the feature or bug fix to do. This is one of the most valuable steps because it clearly defines the acceptance criteria. Although it's also important to be thoughtful and flexible - just because your tests pass doesn't necessarily mean the feature is implemented as intended. Some new tests or adjustments may be needed along the way.
- Now, before I write any more code, is when I submit a pull request (PR) from my fork's feature branch to upstream/master. So many people are surprised by this. Many collaborators have told me they thought that PR's should be submitted after their work is complete and passing all tests. But in my opinion that defeats the entire point of collaborating on a short iteration cycle.
- If you wait until the end to submit your work you risk diverging from the feature's intended goals especially if the feature's requirements shift or you've misinterpreted the goals even slightly.
- Waiting also means you're missing out on collaborating with your teammates and soliciting their feedback mid-project.
- Every push to your fork will trigger a build that runs your tests.
- Your teammates will get continuous updates so they can monitor your progress in real-time but also on their time so you won't have to hold a formal review, since collaborators can review your work anytime as the commits will all be queued in the PR.
- Now comes the really fun part. We hack and collaborate until the tests all pass. But we don't have too much fun - there should be at most 10 commits before we realize we've embarked on an epic that needs to be re-organized, otherwise the PR will become difficult to merge. That will sap moral and waste time. So keep it simple.
- The final bit of collaboration is the code review and merging the PR into upstream master. This is fairly easy, since there are
- already tests that demonstrate what the code should do,
- only a few commits,
- and all of the collaborators have been following the commits as they've been queuing in the PR.
- Whoever the repository owner or maintainer is should add the tag and push it to upstream. This triggers the CI to test, build and deploy a new release.
This is key. Set up Travis, Circle, AppVeyor or Jenkins on upstream master to test and build every commit, every commit to an open PR and to deploy on every tag. Easy!
There are some features of this style that stand out:
- There is only one master branch. Using CI to deploy only on tags eliminates our need for a dev or staging branch because any commits on master not tagged are the equivalent of the bleeding edge.
- This method depends heavily on an online hosted Git repo like GitHub or BitBucket, use of TDD, strong collaboration and a CI server like Travis.
- On Mac OS X matplotlib will not work in a virtual environment unless a framework interpreter is used. The easiest way to do this is to run python as
PYTHONHOME=/home/you/path/to/project/venv/ pythoninstead of using
- On Windows pip often creates an executable for scripts that is bound to the Python interpreter it was installed with. If the virtual environments was created with system site packages or if the package is not installed in the virtual environment then you may get a confusing path clash. For example running the
nosetestsscript will use your system Python and therefore the Python path will not include your virtual environment. The solution is to never use system site packages and install all dependencies directly in your virtual environment.