Let's say I have 3 git repositories, each with a lib and tests folder in the root. All 3 repositories are part of what I want to be a single package, however it is important to me to keep the repositories separate.
I am new to git coming from svn, so I have been reading up on submodules and how they differ from svn:externals. In SVN I could have a single
lib/vendor/package
directory, and inside package I could setup 3 externals pointing to each of my 3 repositories lib directory, renaming it appropriately like
lib/vendor/package/a -> repo1/lib
lib/vendor/package/b -> repo2/lib
lib/vendor/package/c -> repo3/lib
but from my understanding this is not possible with git. Am I missing something?
Really I'm hoping this can be solved in one of two ways.
Someone will point out how to create a 4th git repository which has the other 3 as submodules organized as I have mentioned above (where I can have an a, b, and c folder inside the root)
Someone will point out how to set this up using svn:externals in combination with githubs svn support, referencing the lib directory within each git repository (from my understanding this is impossible)
Update:
I had actually tried to follow the submodules tutorial you linked to, but I run into the following problem.
Doing things as shown above, instead of a mapping like
lib/vendor/package/a -> repo1/lib
lib/vendor/package/b -> repo2/lib
lib/vendor/package/c -> repo3/lib
I am left with
lib/vendor/package/a -> repo1
lib/vendor/package/b -> repo2
lib/vendor/package/c -> repo3
this is not ideal since now to access ClassA inside repo1's lib folder, the path is
lib/vendor/package/a/lib/ClassA
when I'm really trying to get (and this is possible with svn:externals)
lib/vendor/package/a/ClassA
since a above is actually repo1/lib, and not the root directory of repo1.
Something like this is important since, with PHP5.3 for example, using the SplClassLoader ( http://gist.github.com/221634 ), it requires a namespace-to-directory mapping like
\Package\a\ClassA -> lib/vendor/package/a/ClassA
this is where my conceptual misunderstanding is, how to setup that 4th git repository to allow my directory mappings like above.
You are right, Git submodules can not directly do exactly what you want. It works in SVN because the root of a repository, branches, and any subdirectory thereof are the same kind of object. In Git, a repository, a branch, and a directory are all distinct kinds of objects (you can not use a directory as a full repository or as a branch).
There are a couple of indirect ways to accomplish what you want though.
Using Submodules and Symlinks
The core of a Git submodule is a clone of another repository in the work tree of the “superproject”*.
Git only clones full repositories. It is not possible to clone just a single subdirectory out of an existing repository†.
* Normal submodules also require a special reference in the superproject's commits/index and (normally) an entry in the superproject's .gitmodules file.
It is possible to have non-tracked clones of other repositories in an unrelated working tree, but such usage does not create a submodule.
† Git 1.7.0 and later has a “sparse checkout” feature, but it would not help to relocate the lib directory the top level of each submodule clone.
You might, however be able to use Git's support for symbolic links to do something that is fairly close:
#
# Make the lib directory of each submodule appear in the superproject as
# lib/vendor/packages/$submod_name
#
# With this structure in each of the submodules (a, b, c):
#
# lib/
# tests/
#
# We end up with this structure in the superproject:
#
# lib/
# vendor/
# packages/
# a (a symlink to ../../../_submodules/a/lib)
# b (a symlink to ../../../_submodules/b/lib)
# c (a symlink to ../../../_submodules/c/lib)
# _submodules
# a/ (a Git submodule)
# lib/
# tests/
# b/ (a Git submodule)
# lib/
# tests/
# c/ (a Git submodule)
# lib/
# tests/
#
add_one() {
dir=lib/vendor/package
dest="$dir/$1"
# use fewer ".."s to put the _submodules closer to the symlinks
s=../../../_submodules/"$1"
git submodule add "$2" "$dir/$s"
ln -s "$s"/lib "$dest"
git add "$dest"
}
cd "$main_repo_toplevel"
mkdir -p lib/vendor/package
add_one a git#githost.example.com:user/package-a.git
add_one b git://public.example.com/work/package-b-dev.git
add_one c ssh://special.example.com/foo.git
Using git subtree
apenwarr's git subtree can split off and merge parts of repositories (i.e. individual subdirectories; it is a wrapper around “subtree merging” with other nice features). The first step would be to extract the history of lib in each of your sub-projects. Then, either directly use the extracted history as a submodule, or use git subtree to do a subtree merge into your main repository. Either way, this would introduce an extra step (re-extracting the lib history) before you could integrate changes from a sub-project into your main repository.
You can have a forth 'lib' main Git repo, with:
lib/vendor/package content
3 submodules (see "true nature of submodules")
But this reference is totally independent from any 'svn:external' property you may have setup in a mirror SVN repo.
So if you have 3 SVN repos already, you can git-svn them, publish them on GitHub, and then create a fourth repo on GitHub in which you will add those 3 Git repos as submodules, following the submodules tutorial (supposing here you have all 4 repos already on GitHub)
$ mkdir -p lib/vendor/package
$ cd lib/vendor/package
$ for package in a b c d; do
$ git submodule add git://github.com/path/to/$package.git $package
$ done
$ cd ..
$ git commit -m "lib with submodules"
$ git push
Related
In my PHP project I use composer.
For some reasons (stupid, but it's not depending on me) I must store all project files in a git repository, including "vendor/" files fetched by composer. I've removed "vendor/" from the .gitignore file, but some modules (like https://github.com/RWOverdijk/AssetManager) are treated as submodules -
and they are not being added to the repository (git add ignores them).
What do I need to do, to force addition?
You can use the following commands:
# remove `.git` folders recursively
find vendor -type d -name '.git' -exec rm -rf {} \;
# Add the vendor folder
git add vendor
Probably it would be better to persuade your boss that composer isn't meant to work like that.
I've created my own site on my local computer and it works wonderfully. I'm having tons of fun developing with it. But when it came time to move it to my server and test I ran into issue.
I'm using the FOSUserBundle. Obviously being a Git of it's own, when I did a git commit and push, git ignored everything in vendor/friendsofsymfony/user-bundle. So when I cloned to my server, the folder came empty, and of course now my site doesn't work.
I want to do this the right way. So if it means destroying the git repo and doing it a different way, I'm ok with that.
So far I've tried.
Following the original install instructions for FOSUserBundle
Running the following:
composer update friendsofsymfony/user-bundle
composer install friendsofsymfony/user-bundle
Both return "nothing to update/install"
I've also tried clearing composer's cache between each command attempt.
rm -rf ~/.composer/cache
Answer is in the comments. Ignore vendor/ use composer to install the appropriate files.
also this Symfony project cloned with git vendors not installed
What it sounds like your trying to do is create a mywebsite parent repo within which you have a third party child repo that has the FOSUserBundle code. This uses git submodules and requires related submodule commands like git submodule init.
The ... directory is there, but empty. You must run two commands: git submodule init to initialize your local configuration file, and git submodule update to fetch all the data from that project and check out the appropriate commit listed in your superproject...
I don't know anything about composer. Here is how you could do it with git.
cd mywebsite
git init
git submodule add <url-to-FOSUserBundle-repo>
You should now have a directory structure like this:
mywebsite
.git
.gitmodules
FOSUserBunder
.git
See also http://git-scm.com/book/en/v2/Git-Tools-Submodules
i have a git repository where i have the following directory structure:
src/
vendor/
app/
public/
vendor/ contains as you'd expect, 3rd party libraries (symfony, etc) which i do not want included as part of the repository. as a result of this, in my jenkins template which i have set up as 'Poll SCM' so builds can be automated from commits, it does not copy over the vendor/ directory into the workspace build, and so test fail because the library is not included in part of the build.
i came across git submodules whilst doing some troubleshooting into this, but im not sure they are the answer. as i'm sure im not the only one to use jenkins with the git plugin to create builds, and with ignored files/directories how can i go about getting the vendor/ directory and files across without putting them into the git repository?
managed to find a workaround after digging in the Jenkins Job Configure screen - under the 'Build' section, aswell as 'Invoke Ant' there was an option to 'execute shell' ... i put the line 'composer install' in there and that successfully installs everything needed from the vendor/ folder before running ant and then the unit tests.
As a PHP developer I find myself working with Composer a lot. In the past it was on personal projects and such so I didn't have much problems with it, but now with Laravel 4 it's on project that require deploying and I'm in kind of a struggle to adapt my workflow.
All my projects are git repositories, thus per convention and because it's still quite buggy, like most developers I put the vendor directory in my .gitignore.
Now the problem is : I also use Git to deploy to the server, and by all logic the vendor directory is not uploaded as it's not tracked by the repository.
So my question is towards people that have worked with Composer and Git for longer than me : what is the best workflow to keep the server in sync ? How to track the vendor folder without really tracking it ?
I tried uploading it every time I update with Composer but some of my vendor folders are quite big and I can't manually upload 30Mb of files every time something updates.
I don't really know, how do you guys work it out ? I tried not ignoring the vendor folder but Git just messes it up, half are recognized as cloned repos and are just ignored anyway, etc.
UPDATE : Note that I'm on a shared host so I don't have access to the server's terminal.
The best way is to run composer install on the server after updating to the latest code. You should also make sure you commit your composer.lock file, which the server will then use to install (you should not run composer update on the server).
Capistrano (or Capifony if you are using Symfony2) is very useful for deployments with composer. You can trigger a deployment remotely and it will run a composer install in isolation so the site remains online until it has been deployed successfully. There are many other benefits such as retaining previous deployments and rolling back, copying old vendors before deployments, compiling assets etc. etc.
I'm working on something like this in the git post-receive hook on the server. This isn't tested and may be buggy, but you should get the idea.
#!/bin/bash
# get the updated composer.json
git checkout master -- composer.json
# only do this stuff if composer.json is different
# you could check this manually, or with git or cmp
cp composer.json tmp/composer.json
# may take a minute, but won't take the site down
(cd tmp; composer install --prefer-dist)
# this doesn't seem to be atomic
git checkout -f
# switch vendors over
# this isn't quite an atomic operation, but is very close
# you could probably do it with symlinks and "mv -Tf" to make it atomic
mv vendor vendor.old
mv tmp/vendor vendor
rm -r tmp vendor.old
Ideally all of the deploy (i.e. in this case the git checkout and the composer install) except one single mv would happen in isolation, outside of www. This doesn't work if you have untracked files (eg CMS uploads) in your working tree and rely on PHP's __FILE__ not resolving symlinks (due to this PHP bug).
This is an old question but in case anybody is looking a solution:
I slightly modify the #dave1010 answer to use git pull instead of git checkout--force
#!/bin/bash
# get only composer files
git fetch
git checkout origin/master -- composer.json
git checkout origin/master -- composer.lock
# make sure tmp is empty
rm -rf tmp
mkdir tmp
# copy the composer files to tmp
cp -r vendor tmp/vendor
cp composer.json tmp/composer.json
cp composer.lock tmp/composer.lock
# may take a minute, but won't take the site down
(cd tmp; composer install --no-scripts --verbose; cd ..)
# switch vendors over
rm -rf vendor_old
mv vendor vendor_old
mv tmp/vendor vendor
# update your code
git pull
# run again composer install. This time will print 'Nothing to install or update'
# but will execute pre/post scripts & generate autoload files
composer install --optimize-autoloader
There is maybe a better solution using capistrano/composer. But I like mine better.
You can use something like jenkins to ftp your files over this way you can direct jenkins to run composer install on the jenkins server and then ftp the files over.
This also allows you to ignore the vendor folder.
It does require a build server to be made and you would need to be able to execute commands vs the build server
The key is your composer.lock file. The composer.lock keeps track of exactly what packages (and versions) you have installed. When you deploy, send your composer.lock file up to the production server as well, and simply do a composer update. All the exact same package versions will be installed. With deployment software like Capistrano or Flightplan you can make the composer update step part of the process so it happens automatically.
I use git as my primary version control system, and have recently started using git on my CakePHP projects. This is my current .gitignore file:
app/tmp
vendors/
As used in the cakephp git repo, but this causes a bit more work for me when deploying the project to a server, because I have to go in and create all the app/tmp/ sub-directories by hand before they will work correctly. Is there a way to set it up to ignore the contents on these folders, but to still have them under git control so they appear when I clone the repo into the hoted directory?
I also have been having an issue with my git index being reset while I am working on it, which is causing me to have to do a lot more commits than should be necessary, any ideas on that also?
Git stores only files, not directories, so you can for example add a hidden file into that directory and commit it.
Remove app/tmp/ from .gitignore
touch app/tmp/.keep
git add app/tmp/.keep
git commit
Add app/tmp/ to .gitignore
As mentioned git only stores files, not directories. By default cake's .gitignore file ignores all contents in the tmp folder to prevent tmp files being added to your repository.
You can (and should) however do this after you create a project:
cd /my/app
git add -f tmp
which will do this:
$ git status
# On branch master
#
# Changes to be committed:
# (use "git rm --cached <file>..." to unstage)
#
# new file: tmp/cache/models/empty
# new file: tmp/cache/persistent/empty
# new file: tmp/cache/views/empty
# new file: tmp/logs/empty
# new file: tmp/sessions/empty
# new file: tmp/tests/empty
As such your tmp folder structure is ready to be committed, but all other files in your tmp dir will (continue to) be ignored.
My .gitignore file.
tmp/*
[Cc]onfig/core.php
[Cc]onfig/database.php
webroot/files/
webroot/img/photos/
!empty
.DS_Store
If you'll notice I have !empty which saves me from creating .keep files all over which is so SVN ago. Lastly you'll also see that I use this config for both cakePHP 1.x and 2.x projects noted by the [Cc]. I have some folders setup that I store user files in so I always ignore them as well. Finally the .DS_Store ignores my MAC created thumbnail views for my project.