TortoiseGit works great with remotes. If you are used to TortoiseSVN, then after a very short transition to DVCS style SCM you'll be cloning, branching, pushing and pulling in no time.
But what's that funny GitBash thing for? Well if you're like me and you started using Git from a terminal then you'll need to use GitBash (unless you opted out of the default mysysgit installation) to use commands. To get help, use git help instead of man git.
So the point of this post is how to use GitBash with remote repositories like GitHub or BitBucket with SSH. Of course both sites support HTTPS, but SSH is so much more James Bond you know you want it. If you installed TortoiseGit then you're probably using
PuTTY/Plink and Pageant which works pretty seamlessly with both sites. But I could not get the GitBash to work with PuTTY/Plink even with Pageant running. Luckily mysysgit (and Cygwin) come with SSH, so just generate a new key and give your public one to your remote. You also have to make a config file in your .ssh folder with host, user and private key file.
BitBucket has a good help page on how to set up mysysgit on windows here. That's where they show you step by step how to make a key, set up your SSH config file and also how to set up ssh-agent, so you don't have to keep typing in your SSH passphrase. I highly recommend this last step; that snippet of code is from a very old Cygwin post, so it's been tested literally millions of times. There's also some general information on using SSH with BitBucket here. Of course GitHub also has help on SSH issues here too. It has been revised since I originally wrote this post (over a year ago) and now has excellent setup instructions for all platforms, including Windows.
BTW: TortoiseHg will really only work with PuTTY/Plink and Pageant.