I have a Git repository which contains a number of subdirectories. Now I have found that one of the subdirectories is unrelated to the other and should be detached to a separate repository.
How can I do this while keeping the history of the files within the subdirectory?
I guess I could make a clone and remove the unwanted parts of each clone, but I suppose this would give me the complete tree when checking out an older revision etc. This might be acceptable, but I would prefer to be able to pretend that the two repositories doesn't have a shared history.
Just to make it clear, I have the following structure:
XYZ/
.git/
XY1/
ABC/
XY2/
But I would like this instead:
XYZ/
.git/
XY1/
XY2/
ABC/
.git/
ABC/
The Easy Way™
It turns out that this is such a common and useful practice that the overlords of Git made it really easy, but you have to have a newer version of Git (>= 1.7.11 May 2012). See the appendix for how to install the latest Git. Also, there's a real-world example in the walkthrough below.
Prepare the old repo
cd <big-repo>
git subtree split -P <name-of-folder> -b <name-of-new-branch>
Note: <name-of-folder> must NOT contain leading or trailing characters. For instance, the folder named subproject MUST be passed as subproject, NOT ./subproject/
Note for Windows users: When your folder depth is > 1, <name-of-folder> must have *nix style folder separator (/). For instance, the folder named path1\path2\subproject MUST be passed as path1/path2/subproject
Create the new repo
mkdir ~/<new-repo> && cd ~/<new-repo>
git init
git pull </path/to/big-repo> <name-of-new-branch>
Link the new repo to GitHub or wherever
git remote add origin <git#github.com:user/new-repo.git>
git push -u origin master
Cleanup inside <big-repo>, if desired
git rm -rf <name-of-folder>
Note: This leaves all the historical references in the repository. See the Appendix below if you're actually concerned about having committed a password or you need to decreasing the file size of your .git folder.
Walkthrough
These are the same steps as above, but following my exact steps for my repository instead of using <meta-named-things>.
Here's a project I have for implementing JavaScript browser modules in node:
tree ~/node-browser-compat
node-browser-compat
├── ArrayBuffer
├── Audio
├── Blob
├── FormData
├── atob
├── btoa
├── location
└── navigator
I want to split out a single folder, btoa, into a separate Git repository
cd ~/node-browser-compat/
git subtree split -P btoa -b btoa-only
I now have a new branch, btoa-only, that only has commits for btoa and I want to create a new repository.
mkdir ~/btoa/ && cd ~/btoa/
git init
git pull ~/node-browser-compat btoa-only
Next, I create a new repo on GitHub or Bitbucket, or whatever and add it as the origin
git remote add origin git#github.com:node-browser-compat/btoa.git
git push -u origin master
Happy day!
Note: If you created a repo with a README.md, .gitignore and LICENSE, you will need to pull first:
git pull origin master
git push origin master
Lastly, I'll want to remove the folder from the bigger repo
git rm -rf btoa
Appendix
Latest Git on macOS
To get the latest version of Git using Homebrew:
brew install git
Latest Git on Ubuntu
sudo apt-get update
sudo apt-get install git
git --version
If that doesn't work (you have a very old version of Ubuntu), try
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git
If that still doesn't work, try
sudo chmod +x /usr/share/doc/git/contrib/subtree/git-subtree.sh
sudo ln -s \
/usr/share/doc/git/contrib/subtree/git-subtree.sh \
/usr/lib/git-core/git-subtree
Thanks to rui.araujo from the comments.
Clearing your history
By default removing files from Git doesn't actually remove them, it just commits that they aren't there anymore. If you want to actually remove the historical references (i.e. you committed a password), you need to do this:
git filter-branch --prune-empty --tree-filter 'rm -rf <name-of-folder>' HEAD
After that, you can check that your file or folder no longer shows up in the Git history at all
git log -- <name-of-folder> # should show nothing
However, you can't "push" deletes to GitHub and the like. If you try, you'll get an error and you'll have to git pull before you can git push - and then you're back to having everything in your history.
So if you want to delete history from the "origin" - meaning to delete it from GitHub, Bitbucket, etc - you'll need to delete the repo and re-push a pruned copy of the repo. But wait - there's more! - if you're really concerned about getting rid of a password or something like that you'll need to prune the backup (see below).
Making .git smaller
The aforementioned delete history command still leaves behind a bunch of backup files - because Git is all too kind in helping you to not ruin your repo by accident. It will eventually delete orphaned files over the days and months, but it leaves them there for a while in case you realize that you accidentally deleted something you didn't want to.
So if you really want to empty the trash to reduce the clone size of a repo immediately you have to do all of this really weird stuff:
rm -rf .git/refs/original/ && \
git reflog expire --all && \
git gc --aggressive --prune=now
git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune
That said, I'd recommend not performing these steps unless you know that you need to - just in case you did prune the wrong subdirectory, y'know? The backup files shouldn't get cloned when you push the repo, they'll just be in your local copy.
Credit
http://psionides.eu/2010/02/04/sharing-code-between-projects-with-git-subtree/
Remove a directory permanently from git
http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/
How to remove unreferenced blobs from my git repo
Update: This process is so common, that the git team made it much simpler with a new tool, git subtree. See here: Detach (move) subdirectory into separate Git repository
You want to clone your repository and then use git filter-branch to mark everything but the subdirectory you want in your new repo to be garbage-collected.
To clone your local repository:
git clone /XYZ /ABC
(Note: the repository will be cloned using hard-links, but that is not a problem since the hard-linked files will not be modified in themselves - new ones will be created.)
Now, let us preserve the interesting branches which we want to rewrite as well, and then remove the origin to avoid pushing there and to make sure that old commits will not be referenced by the origin:
cd /ABC
for i in branch1 br2 br3; do git branch -t $i origin/$i; done
git remote rm origin
or for all remote branches:
cd /ABC
for i in $(git branch -r | sed "s/.*origin\///"); do git branch -t $i origin/$i; done
git remote rm origin
Now you might want to also remove tags which have no relation with the subproject; you can also do that later, but you might need to prune your repo again. I did not do so and got a WARNING: Ref 'refs/tags/v0.1' is unchanged for all tags (since they were all unrelated to the subproject); additionally, after removing such tags more space will be reclaimed. Apparently git filter-branch should be able to rewrite other tags, but I could not verify this. If you want to remove all tags, use git tag -l | xargs git tag -d.
Then use filter-branch and reset to exclude the other files, so they can be pruned. Let's also add --tag-name-filter cat --prune-empty to remove empty commits and to rewrite tags (note that this will have to strip their signature):
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC -- --all
or alternatively, to only rewrite the HEAD branch and ignore tags and other branches:
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC HEAD
Then delete the backup reflogs so the space can be truly reclaimed (although now the operation is destructive)
git reset --hard
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now
and now you have a local git repository of the ABC sub-directory with all its history preserved.
Note: For most uses, git filter-branch should indeed have the added parameter -- --all. Yes that's really --space-- all. This needs to be the last parameters for the command. As Matli discovered, this keeps the project branches and tags included in the new repo.
Edit: various suggestions from comments below were incorporated to make sure, for instance, that the repository is actually shrunk (which was not always the case before).
Paul's answer creates a new repository containing /ABC, but does not remove /ABC from within /XYZ. The following command will remove /ABC from within /XYZ:
git filter-branch --tree-filter "rm -rf ABC" --prune-empty HEAD
Of course, test it in a 'clone --no-hardlinks' repository first, and follow it with the reset, gc and prune commands Paul lists.
I’ve found that in order to properly delete the old history from the new repository, you have to do a little more work after the filter-branch step.
Do the clone and the filter:
git clone --no-hardlinks foo bar; cd bar
git filter-branch --subdirectory-filter subdir/you/want
Remove every reference to the old history. “origin” was keeping track of your clone, and “original” is where filter-branch saves the old stuff:
git remote rm origin
git update-ref -d refs/original/refs/heads/master
git reflog expire --expire=now --all
Even now, your history might be stuck in a packfile that fsck won’t touch. Tear it to shreds, creating a new packfile and deleting the unused objects:
git repack -ad
There is an explanation of this in the manual for filter-branch.
When running git filter-branch using a newer version of git (2.22+ maybe?), it says to use this new tool git-filter-repo. This tool certainly simplified things for me.
Filtering with filter-repo
Commands to create the XYZ repo from the original question:
# create local clone of original repo in directory XYZ
tmp $ git clone git#github.com:user/original.git XYZ
# switch to working in XYZ
tmp $ cd XYZ
# keep subdirectories XY1 and XY2 (dropping ABC)
XYZ $ git filter-repo --path XY1 --path XY2
# note: original remote origin was dropped
# (protecting against accidental pushes overwriting original repo data)
# XYZ $ ls -1
# XY1
# XY2
# XYZ $ git log --oneline
# last commit modifying ./XY1 or ./XY2
# first commit modifying ./XY1 or ./XY2
# point at new hosted, dedicated repo
XYZ $ git remote add origin git#github.com:user/XYZ.git
# push (and track) remote master
XYZ $ git push -u origin master
assumptions:
* remote XYZ repo was new and empty before the push
Filtering and moving
In my case, I also wanted to move a couple of directories for a more consistent structure. Initially, I ran that simple filter-repo command followed by git mv dir-to-rename, but I found I could get a slightly "better" history using the --path-rename option. Instead of seeing last modified 5 hours ago on moved files in the new repo I now see last year (in the GitHub UI), which matches the modified times in the original repo.
Instead of...
git filter-repo --path XY1 --path XY2 --path inconsistent
git mv inconsistent XY3 # which updates last modification time
I ultimately ran...
git filter-repo --path XY1 --path XY2 --path inconsistent --path-rename inconsistent:XY3
Notes:
I thought the Git Rev News blog post explained well the reasoning behind creating yet another repo-filtering tool.
I initially tried the path of creating a sub-directory matching the target repo name in the original repository and then filtering (using git filter-repo --subdirectory-filter dir-matching-new-repo-name). That command correctly converted that subdirectory to the root of the copied local repo, but it also resulted in a history of only the three commits it took to create the subdirectory. (I hadn't realized that --path could be specified multiple times; thereby, obviating the need to create a subdirectory in the source repo.) Since someone had committed to the source repo by the time I noticed that I'd failed to carry forward the history, I just used git reset commit-before-subdir-move --hard after the clone command, and added --force to the filter-repo command to get it to operate on the slightly modified local clone.
git clone ...
git reset HEAD~7 --hard # roll back before mistake
git filter-repo ... --force # tell filter-repo the alterations are expected
I was stumped on the install since I was unaware of the extension pattern with git, but ultimately I cloned git-filter-repo and symlinked it to $(git --exec-path):
ln -s ~/github/newren/git-filter-repo/git-filter-repo $(git --exec-path)
Edit: Bash script added.
The answers given here worked just partially for me; Lots of big files remained in the cache. What finally worked (after hours in #git on freenode):
git clone --no-hardlinks file:///SOURCE /tmp/blubb
cd blubb
git filter-branch --subdirectory-filter ./PATH_TO_EXTRACT --prune-empty --tag-name-filter cat -- --all
git clone file:///tmp/blubb/ /tmp/blooh
cd /tmp/blooh
git reflog expire --expire=now --all
git repack -ad
git gc --prune=now
With the previous solutions, the repository size was around 100 MB. This one brought it down to 1.7 MB. Maybe it helps somebody :)
The following bash script automates the task:
!/bin/bash
if (( $# < 3 ))
then
echo "Usage: $0 </path/to/repo/> <directory/to/extract/> <newName>"
echo
echo "Example: $0 /Projects/42.git first/answer/ firstAnswer"
exit 1
fi
clone=/tmp/${3}Clone
newN=/tmp/${3}
git clone --no-hardlinks file://$1 ${clone}
cd ${clone}
git filter-branch --subdirectory-filter $2 --prune-empty --tag-name-filter cat -- --all
git clone file://${clone} ${newN}
cd ${newN}
git reflog expire --expire=now --all
git repack -ad
git gc --prune=now
This is no longer so complex you can just use the git filter-branch command on a clone of you repo to cull the subdirectories you don't want and then push to the new remote.
git filter-branch --prune-empty --subdirectory-filter <YOUR_SUBDIR_TO_KEEP> master
git push <MY_NEW_REMOTE_URL> -f .
Update: The git-subtree module was so useful that the git team pulled it into core and made it git subtree. See here: Detach (move) subdirectory into separate Git repository
git-subtree may be useful for this
http://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt (deprecated)
http://psionides.jogger.pl/2010/02/04/sharing-code-between-projects-with-git-subtree/
Here is a small modification to CoolAJ86's "The Easy Way™" answer in order to split multiple sub folders (let's say sub1and sub2) into a new git repository.
The Easy Way™ (multiple sub folders)
Prepare the old repo
pushd <big-repo>
git filter-branch --tree-filter "mkdir <name-of-folder>; mv <sub1> <sub2> <name-of-folder>/" HEAD
git subtree split -P <name-of-folder> -b <name-of-new-branch>
popd
Note: <name-of-folder> must NOT contain leading or trailing characters. For instance, the folder named subproject MUST be passed as subproject, NOT ./subproject/
Note for windows users: when your folder depth is > 1, <name-of-folder> must have *nix style folder separator (/). For instance, the folder named path1\path2\subproject MUST be passed as path1/path2/subproject. Moreover don't use mvcommand but move.
Final note: the unique and big difference with the base answer is the second line of the script "git filter-branch..."
Create the new repo
mkdir <new-repo>
pushd <new-repo>
git init
git pull </path/to/big-repo> <name-of-new-branch>
Link the new repo to Github or wherever
git remote add origin <git#github.com:my-user/new-repo.git>
git push origin -u master
Cleanup, if desired
popd # get out of <new-repo>
pushd <big-repo>
git rm -rf <name-of-folder>
Note: This leaves all the historical references in the repository.See the Appendix in the original answer if you're actually concerned about having committed a password or you need to decreasing the file size of your .git folder.
The original question wants XYZ/ABC/(*files) to become ABC/ABC/(*files). After implementing the accepted answer for my own code, I noticed that it actually changes XYZ/ABC/(*files) into ABC/(*files). The filter-branch man page even says,
The result will contain that directory (and only that) as its project root."
In other words, it promotes the top-level folder "up" one level. That's an important distinction because, for example, in my history I had renamed a top-level folder. By promoting folders "up" one level, git loses continuity at the commit where I did the rename.
My answer to the question then is to make 2 copies of the repository and manually delete the folder(s) you want to keep in each. The man page backs me up with this:
[...] avoid using [this command] if a simple single commit would suffice to fix your problem
To add to Paul's answer, I found that to ultimately recover space, I have to push HEAD to a clean repository and that trims down the size of the .git/objects/pack directory.
i.e.
$ mkdir ...ABC.git
$ cd ...ABC.git
$ git init --bare
After the gc prune, also do:
$ git push ...ABC.git HEAD
Then you can do
$ git clone ...ABC.git
and the size of ABC/.git is reduced
Actually, some of the time consuming steps (e.g. git gc) aren't needed with the push to clean repository, i.e.:
$ git clone --no-hardlinks /XYZ /ABC
$ git filter-branch --subdirectory-filter ABC HEAD
$ git reset --hard
$ git push ...ABC.git HEAD
Proper way now is the following:
git filter-branch --prune-empty --subdirectory-filter FOLDER_NAME [first_branch] [another_branch]
GitHub now even have small article about such cases.
But be sure to clone your original repo to separate directory first (as it would delete all the files and other directories and you probable need to work with them).
So your algorithm should be:
clone your remote repo to another directory
using git filter-branch left only files under some subdirectory, push to new remote
create commit to remove this subdirectory from your original remote repo
It appears that most (all?) of the answers here rely on some form of git filter-branch --subdirectory-filter and its ilk. This may work "most times" however for some cases, for instance the case of when you renamed the folder, ex:
ABC/
/move_this_dir # did some work here, then renamed it to
ABC/
/move_this_dir_renamed
If you do a normal git filter style to extract "move_this_dir_renamed" you will lose file change history that occurred from back when it was initially "move_this_dir" (ref).
It thus appears that the only way to really keep all change history (if yours is a case like this), is, in essence, to copy the repository (create a new repo, set that to be the origin), then nuke everything else and rename the subdirectory to the parent like this:
Clone the multi-module project locally
Branches - check what's there: git branch -a
Do a checkout to each branch to be included in the split to get a local copy on your workstation: git checkout --track origin/branchABC
Make a copy in a new directory: cp -r oldmultimod simple
Go into the new project copy: cd simple
Get rid of the other modules that aren't needed in this project:
git rm otherModule1 other2 other3
Now only the subdir of the target module remains
Get rid of the module subdir so that the module root becomes the new project root
git mv moduleSubdir1/* .
Delete the relic subdir: rmdir moduleSubdir1
Check changes at any point: git status
Create the new git repo and copy its URL to point this project into it:
git remote set-url origin http://mygithost:8080/git/our-splitted-module-repo
Verify this is good: git remote -v
Push the changes up to the remote repo: git push
Go to the remote repo and check it's all there
Repeat it for any other branch needed: git checkout branch2
This follows the github doc "Splitting a subfolder out into a new repository" steps 6-11 to push the module to a new repo.
This will not save you any space in your .git folder, but it will preserve all your change history for those files even across renames. And this may not be worth it if there isn't "a lot" of history lost, etc. But at least you are guaranteed not to lose older commits!
I recommend GitHub's guide to splitting subfolders into a new repository. The steps are similar to Paul's answer, but I found their instructions easier to understand.
I have modified the instructions so that they apply for a local repository, rather than one hosted on GitHub.
Splitting a subfolder out into a new repository
Open Git Bash.
Change the current working directory to the location where you want to create your new repository.
Clone the repository that contains the subfolder.
git clone OLD-REPOSITORY-FOLDER NEW-REPOSITORY-FOLDER
Change the current working directory to your cloned repository.
cd REPOSITORY-NAME
To filter out the subfolder from the rest of the files in the repository, run git filter-branch, supplying this information:
FOLDER-NAME: The folder within your project that you'd like to create a separate repository from.
Tip: Windows users should use / to delimit folders.
BRANCH-NAME: The default branch for your current project, for example, master or gh-pages.
git filter-branch --prune-empty --subdirectory-filter FOLDER-NAME BRANCH-NAME
# Filter the specified branch in your directory and remove empty commits
Rewrite 48dc599c80e20527ed902928085e7861e6b3cbe6 (89/89)
Ref 'refs/heads/BRANCH-NAME' was rewritten
I had exactly this problem but all the standard solutions based on git filter-branch were extremely slow. If you have a small repository then this may not be a problem, it was for me. I wrote another git filtering program based on libgit2 which as a first step creates branches for each filtering of the primary repository and then pushes these to clean repositories as the next step. On my repository (500Mb 100000 commits) the standard git filter-branch methods took days. My program takes minutes to do the same filtering.
It has the fabulous name of git_filter and lives here:
https://github.com/slobobaby/git_filter
on GitHub.
I hope it is useful to someone.
Use this filter command to remove a subdirectory, while preserving your tags and branches:
git filter-branch --index-filter \
"git rm -r -f --cached --ignore-unmatch DIR" --prune-empty \
--tag-name-filter cat -- --all
For what it's worth, here is how using GitHub on a Windows machine. Let's say you have a cloned repo in residing in C:\dir1. The directory structure looks like this: C:\dir1\dir2\dir3. The dir3 directory is the one I want to be a new separate repo.
Github:
Create your new repository: MyTeam/mynewrepo
Bash Prompt:
$ cd c:/Dir1
$ git filter-branch --prune-empty --subdirectory-filter dir2/dir3 HEAD
Returned: Ref 'refs/heads/master' was rewritten (fyi: dir2/dir3 is case sensitive.)
$ git remote add some_name git#github.com:MyTeam/mynewrepo.git
git remote add origin etc. did not work, returned "remote origin already exists"
$ git push --progress some_name master
As I mentioned above, I had to use the reverse solution (deleting all commits not touching my dir/subdir/targetdir) which seemed to work pretty well removing about 95% of the commits (as desired). There are, however, two small issues remaining.
FIRST, filter-branch did a bang up job of removing commits which introduce or modify code but apparently, merge commits are beneath its station in the Gitiverse.
Screenshot: Merge Madness!
This is a cosmetic issue which I can probably live with (he says...backing away slowly with eyes averted).
SECOND the few commits that remain are pretty much ALL duplicated! I seem to have acquired a second, redundant timeline that spans just about the entire history of the project. The interesting thing (which you can see from the picture below), is that my three local branches are not all on the same timeline (which is, certainly why it exists and isn't just garbage collected).
Screnshot: Double-double, Git filter-branch style
The only thing I can imagine is that one of the deleted commits was, perhaps, the single merge commit that filter-branch actually did delete, and that created the parallel timeline as each now-unmerged strand took its own copy of the commits. (shrug Where's my TARDiS?) I'm pretty sure I can fix this issue, though I'd really love to understand how it happened.
In the case of crazy mergefest-O-RAMA, I'll likely be leaving that one alone since it has so firmly entrenched itself in my commit history—menacing at me whenever I come near—, it doesn't seem to be actually causing any non-cosmetic problems and because it is quite pretty in Tower.app.
The Easier Way
install git splits. I created it as a git extension, based on jkeating's solution.
Split the directories into a local branch
#change into your repo's directory
cd /path/to/repo
#checkout the branch
git checkout XYZ
#split multiple directories into new branch XYZ
git splits -b XYZ XY1 XY2
Create an empty repo somewhere. We'll assume we've created an empty repo called xyz on GitHub that has path : git#github.com:simpliwp/xyz.git
Push to the new repo.
#add a new remote origin for the empty repo so we can push to the empty repo on GitHub
git remote add origin_xyz git#github.com:simpliwp/xyz.git
#push the branch to the empty repo's master branch
git push origin_xyz XYZ:master
Clone the newly created remote repo into a new local directory
#change current directory out of the old repo
cd /path/to/where/you/want/the/new/local/repo
#clone the remote repo you just pushed to
git clone git#github.com:simpliwp/xyz.git
You might need something like "git reflog expire --expire=now --all" before the garbage collection to actually clean the files out. git filter-branch just removes references in the history, but doesn't remove the reflog entries that hold the data. Of course, test this first.
My disk usage dropped dramatically in doing this, though my initial conditions were somewhat different. Perhaps --subdirectory-filter negates this need, but I doubt it.
Check out git_split project at https://github.com/vangorra/git_split
Turn git directories into their very own repositories in their own location. No subtree funny business. This script will take an existing directory in your git repository and turn that directory into an independent repository of its own. Along the way, it will copy over the entire change history for the directory you provided.
./git_split.sh <src_repo> <src_branch> <relative_dir_path> <dest_repo>
src_repo - The source repo to pull from.
src_branch - The branch of the source repo to pull from. (usually master)
relative_dir_path - Relative path of the directory in the source repo to split.
dest_repo - The repo to push to.
Put this into your gitconfig:
reduce-to-subfolder = !sh -c 'git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter cookbooks/unicorn HEAD && git reset --hard && git for-each-ref refs/original/ | cut -f 2 | xargs -n 1 git update-ref -d && git reflog expire --expire=now --all && git gc --aggressive --prune=now && git remote rm origin'
I'm sure git subtree is all fine and wonderful, but my subdirectories of git managed code that I wanted to move was all in eclipse.
So if you're using egit, it's painfully easy.
Take the project you want to move and team->disconnect it, and then team->share it to the new location. It will default to trying to use the old repo location, but you can uncheck the use-existing selection and pick the new place to move it.
All hail egit.
You can easily try the https://help.github.com/enterprise/2.15/user/articles/splitting-a-subfolder-out-into-a-new-repository/
This worked for me. The issues i faced in the steps given above are
in this command git filter-branch --prune-empty --subdirectory-filter FOLDER-NAME BRANCH-NAME
The BRANCH-NAME is master
if the last step fails when committing due to protection issue follow - https://docs.gitlab.com/ee/user/project/protected_branches.html
I've found quite straight forward solution,
The idea is to copy repository and then just remove unnecessary part.
This is how it works:
1) Clone a repository you'd like to split
git clone git#git.thehost.io:testrepo/test.git
2) Move to git folder
cd test/
2) Remove unnecessary folders and commit it
rm -r ABC/
git add .
enter code here
git commit -m 'Remove ABC'
3) Remove unnecessary folder(s) form history with BFG
cd ..
java -jar bfg.jar --delete-folders "{ABC}" test
cd test/
git reflog expire --expire=now --all && git gc --prune=now --aggressive
for multiply folders you can use comma
java -jar bfg.jar --delete-folders "{ABC1,ABC2}" metric.git
4) Check that history doesn't contains the files/folders you just deleted
git log --diff-filter=D --summary | grep delete
5) Now you have clean repository without ABC,
so just push it into new origin
remote add origin git#github.com:username/new_repo
git push -u origin master
That's it. You can repeat the steps to get another repository,
just remove XY1,XY2 and rename XYZ -> ABC on step 3
Found this wonderful article Original reference easy to follow. Documenting it here in case if it get's inaccessible.
1. Preparing the current repository
$ cd path/to/repository
$ git subtree split -P my-folder -b my-folder
Created branch 'my-folder'
aecbdc3c8fe2932529658f5ed40d95c135352eff
The name of the folder must be a relative path, starting from the root of the repository.
2. Creating the new repository
$ cd my-folder
$ git init
Initialized empty Git repository in /Users/adamwest/Projects/learngit/shop/my-folder/.git/
$ git add .
$ git commit -m "initial commit"
[master (root-commit) 192c10b] initial commit
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 file
Here we just need to cd to the new folder, initialise the new repository, and commit any contents.
3.Add new remote repository and push
$ git remote add origin git#github.com:robertlyall/my-folder.git
$ git push origin -u master
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 199 bytes | 199.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To github.com:robertlyall/my-folder.git
* [new branch] master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
We add the new repository remote from GitHub here, then push our first commit to it.
4. Remove folder from main repository and push
$ cd ../
$ git rm -rf my-folder
rm 'my-folder/file'
$ git commit -m "Remove old folder"
[master 56aedbe] remove old folder
1 file changed, 0 insertions(+), 0 deletions(-)
delete mode 100644 my-folder/file
$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 217 bytes | 217.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To github.com:robertlyall/shop.git
74dd8b3..56aedbe master -> master
Finally, we cd back to the rooot directory, remove the folder from our main repository, then commit and push the change.
Now, we have the folder in our main repository but linked to a completely separate repository that can be reused across multiple projects.
Related
Let's say I have one project in PhpStorm IDE, and within it I have many Git repositories (1 framework module = 1 repo). I just want to ask how to pull changes from all those repositories at once?
I know, that I can commit or push changes on repo massively, but when pulling, it asks me for single Git root.
You could either:
use submodule (which can be tricky), to pull all subrepos as submodules
git submodule update --rebase --remote
or script it as in this gist
find . -name .git -type d \
| xargs -n1 -P4 -I% git --git-dir=% --work-tree=%/.. remote update -p
In both instances, you would need to declare an external tool on the External Tool page of PhpStorm.
I would like to clone a Laravel 4 from Git HERE and after using a Composer to install all dependencies, I wish to create a new Git on my HDD where I have my Dropbox synchronized.
Is it even possible ?
After getting Laravel 4 being cloned into C:/www/laravel-project/, I would like to commit the whole project into my localhost REPO seating in D:/Dropbox/REPOS/Git/ so it will become D:/Dropbox/REPOS/Git/laravel-project/
Thanks
Two possible fixes:
Edit .git/config so that your origin is your repo (replace https://github.com/laravel/laravel.git with file:///D:/Dropbox/REPOS/Git/laravel-project/).
Delete the .git directory and then run git init and git remote add file:///D:/Dropbox/REPOS/Git/laravel-project/ origin
The problem I've seen with local directory repos is that the D:/Dropbox/REPOS/Git/laravel-project/ repo will need to be on a branch that will not be used (i.e. A DoNotUse branch).
There are two paths you can take to do this:
Change the origin to your "remote" and remove or rename the Laravel origin.
Preserve the Laravel origin and use a second "remote" for your files.
Option 1 is probably the easiest.
Go to D:/Dropbox/REPOS/Git/laravel-project/ and do git init to make an empty git repo.
Go to your C:/www/laraval-project/ folder and do git remote -v and save the URL for the current origin if you want to keep it.
Run the command git remote origin set-url file:///D:/Dropbox/REPOS/Git/laravel-project
Optional: run git remote add github url-you-saved-from-step-2 so you can do git pull github if you want to update from the Laravel github
Run git push -u origin master and it should push into your Dropbox's git repo.
For Option 2 you just skip steps 2 through 4 and - run git remote add dropbox file:///whatever and change step 5 to git push -u dropbox master or whatever branch you want. Once you use -u once you can just do git push and it should push to whatever you set as your upstream with -u.
I have created my project repository on git. Its a web development project. I want to upload a file to it.
For other kind of files (.txt, .php, .html, etc.) I know I can create a new file by clicking on + button and then copying my file's content there.
How can I upload an image to the specific directory (./img in my case). I have searched a lot but didn't got any suitable answer.
You can use this method if You are a beginner.
You can add files using git add, example git add README, git add /*, or even git add *
Then use git commit -m "" to commit files
Finally git push -u origin master to push files.
When you make modifications run git status which gives you the list of files modified, add them using git add * for everything or you can specify each file individually, then git commit -m and finally, git push -u origin master
Example - say you created a file README, running git status gives you
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# README
Source :: How do I add files and folders into github repos?
I'm in the process of trying to set up automatic deployments when I make a git push to my bitbucket repository. I have a php deploy script that I leveraged from this blog but when the script runs it is logging that it's only updating from a previous commit.
Here is an example. Let say I log into my server and type git pull. The server will update with the latest changes and lets say the hash for that commit was 001. However if I make several commits lets call them 002, 003, and 004 my script should run every time assuming I pushed those changes to bitbucket after every commit. The script runs but every time it will keep the changes from 001. Only when I log into my server and type git pull, will the server update to 004. Do you know what would cause this?
// Make sure we're in the right directory
exec('cd '.$this->_directory, $output);
$this->log('Changing working directory... '.implode(' ', $output));
// Discard any changes to tracked files since our last deploy
exec('git reset --hard HEAD', $output);
$this->log('Reseting repository... '.implode(' ', $output));
// Update the local repository
exec('git pull '.$this->_remote.' '.$this->_branch, $output);
$this->log('Pulling in changes... '.implode(' ', $output));
// Secure the .git directory
exec('chmod -R og-rx .git');
$this->log('Securing .git directory... ');
if (is_callable($this->post_deploy))
{
call_user_func($this->post_deploy, $this->_data);
}
$this->log('Deployment successful.');
What I would recommend is to release not based on latest version in your master, but a latest tag.
/home/my-user/my-application/1.0.12/www
/home/my-user/my-application/1.0.13/www
etc. This provides rollback functionality. You could make a PHP script that connects to your server over SSH and makes a new clone based on that tag. If you use Composer, you can use this to execute commands. If not, you can do it with a makefile.
Edit: I have forgot to mention how you actually link it.
You have a symlink
/home/my-user/my-application/www -> /home/my-user/my-application/1.0.12/www
When your entire deployment script is finished without errors, you switch the symlink to:
/home/my-user/my-application/www -> /home/my-user/my-application/1.0.13/www
Your application is now live without downtime.
A very simple and efficient approach is to use bitbucket pipelines, you can create a YAML script to compile your dependences and push the code to a server on each commit, automatimagicaly or manually.
Here is an example of a Bitbucket Pipelines YAML script:
image: php:7.1.1
pipelines:
default:
- step:
caches:
- composer
script:
- apt-get update && apt-get install -y unzip
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install
- step:
deployment: staging
name: Deploy to Staging Server
script:
- apt-get update
- apt-get -qq install rsync
- echo "Upload Project files..."
- rsync -avH * -e "ssh" root#$IP_SERVER:/var/www/html/
This script installs the Composer dependencies inside a Docker instance and pushes the projects files to server.
The problem is file permissions.
I was following the same link to that blogpost. I found that the 'www-data' user that is used by php and nginx processes, does not have write permissions to your repository code. It cannot even use git.
For verifying this try doing a 'git pull' on the server as 'www-user'. You can switch to it by 'sudo su www-data'. You will find that it does not even recognize that as a valid git repo and is unable to run your 'deploy.php' script without errors.
You need to set proper permissions to your repository so that www-data can access it.
Or you change the whole approach. Follow this post http://toroid.org/ams/git-website-howto which I feel is a much better method than the above. I ended up using this method.
You want to set a post-update hook in the remote you push to.
In the default case git will not checkout any data you push to a remote. That's why the default git configuration refuses to push to the checked out branch as the checked out files aren't up-to-date with HEAD anymore, when you push to the checkout-out branch.
When the remote receives a push to a branch you can react on that in the post-update hook, though. To see what happens you should first start with some logging:
echo "post update `date`: $*" >> /home/ingo/test/githooks.out
When I push to the branch new, for example I get the line
post update Mi 24. Jun 13:01:14 CEST 2015: refs/heads/new
, where $* contains the branches I pushed to.
With this you can write simply write a script to checkout that branch. I prefer checking out to detached heads, as its much simpler to combine work from several module repositories (no submodules) into a deployed version. See my answer to the source code deployment on how to checkout code outside of the git repository and work with that.
For example you could do
for b in $*
do B=`basename $b`
if [ "$B" = "publish" ] # react on changes to the publish branch
then git --work-tree "PATH TO THE PUBLISHED WORK TREE" checkout publish
do_some_extra_work # to change permissions or inform your team leader
fi done
Of course you can do the checkout step too, that is done on a pull
if [ "`cat HEAD`" = "ref: refs/heads/master" ]
# only do that checkout if the repository is in a sane state
then git merge new # I assume you push to new
fi
The cool thing is that you will see the output of your hook on the remote in the terminal where you executed the push:
# git push remote master:new
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 296 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Updating f81ba5b..a99a710
remote: Fast-forward
remote: a.txt | 2 ++
remote: 1 file changed, 2 insertions(+)
To ../remote
db48da1..a99a710 master -> new
I am trying to setup git for automatic deployment. Here is what I am doing..
I have created one empty bare repository on my repos server which hosted on xyz.com.
Then in eclipse using EGit I have clone newly created bare repository.
On my local machine I have one php project.
Now I want to create 3 branches development, staging, production and want to add my project to development branch.
Maybe I am doing something wrong or missed something to setup. Can anyone please guide me for git setup?
All help would be appreciated.
To create the 3 branches, you can do the following (using whatever names you prefer) from the git command line:
git checkout -b prod
git checkout -b stage
git checkout -b dev
To add your project to the dev branch, copy your project files into the git working directory on your filesystem - this will be the directory that contains the '.git' directory (i.e. not the '.git' directory itself).
Then from the command line run:
git status
which should give you a list of the files that you copied - they will be under the heading 'Untracked files:'.
Now run the following commands to add them to the dev branch (replacing "MyCommitMessage" with an appropriate commit message):
git add .
git commit -m "MyCommitMessage"
At this point you can use
git push --all
to synchronise the remote bare repository with what you have locally.
Since you have cloned from the hosted repository, you should have a git 'remote' configured called 'origin', which refers to your hosted repository.
When you work and commit on a local branch (e.g. master) you can push that to a branch on the other repository with `git push origin :'
git push origin master:development
This assuming you have a git command line client available. You should be able to do the same thing with your graphical client, the main concept is that you push a branch with a certain name to a branch with a certain name on the other side.
I'm not sure how comfortable you are with git command line, but since it's how I use it, here it goes:
instead of cloning your remote repo, go to your code folder and create a new local repo
git init
commit something to your local master branch to proper init it
echo "first commit" > README.txt
git add README.txt
git commit -m "first commit"
now you can create your three branchs
git branch development
git branch staging
git branch production
change to the development branch, and add all your code
git checkout development
git add .
git commit -m "code commit"
for last, add your remote repository and push your branchs
git remote add origin <url_to_your_remote_repo>
git push -u origin --all