I'm attempting to setup a continuous deployment script for Hugo on a LAMP stack server, and the last snag I'm running into is getting Hugo to run.
The Background
I have Hugo and PHP installed on a VPS, and for my app, two directories:
public, where my actual site lives.
hugo, where my entire Hugo setup gets pulled into from GitHub.
I'm using a GitHub webhook to ping my deploy.php script when I merge into the master branch. Here's the relevant part of that script:
// 1. Pull the latest build from GitHub
exec('cd /path/to/hugo && git fetch --all && git reset --hard origin/master');
// 2. Run Hugo to compile any new posts and pages into HTML
exec('cd /path/to/hugo && hugo');
// 3. Copy the files in the /public directory from Hugo into the /public app directory
exec('cd /path/to/hugo && cp -r /path/to/hugo/public/. public');
What's happening
Items 1 and 3 run. Item 2 does not.
More clearly stated: the latest build pulls from GitHub, and any files I've built ahead of time are copied from the Hugo build into the /public directory.
I cannot get hugo to run via exec(), though, and I'm not entirely sure why.
If I SSH into the server in a terminal window, cd into the /path/to/hugo directory, and run hugo there, everything compiles as expected, which eliminates things like, "Maybe Hugo is installed wrong" or "Maybe Hugo isn't accessible in that directory".
So... what am I missing? Thanks in advance!
Check if this is a path issue, meaning by default, a bash session won't execute a program just because you are in the right folder: you have to specify its path (in your case: the current folder)
That means: replace && hugo with && ./hugo
Or, as the OP Chris Ferdinandi did (comments), you can use an absolute path:
I ssh-ed into the server and ran which hugo to get the appropriate path, and replaced && hugo with that.
I have changed a few files name by de-capitalize the first letter, as in Name.jpg to name.jpg. Git does not recognize this changes and I had to delete the files and upload them again. Is there a way that Git can be case-sensitive when checking for changes in file names? I have not made any changes to the file itself.
As long as you're just renaming a file, and not a folder, you can just use git mv:
git mv -f yOuRfIlEnAmE yourfilename
(As of a change in Git 2.0.1, the -f flag in the incantation above is superfluous, but it was needed in older Git versions.)
Git has a configuration setting that tells it whether to expect a case-sensitive or insensitive file system: core.ignorecase. To tell Git to be case-senstive, simply set this setting to false. (Be careful if you have already pushed the files, then you should first move them given the other answers).
git config core.ignorecase false
Note that setting this option to false on a case-insensitive file system is generally a bad idea. Doing so will lead to weird errors. For example, renaming a file in a way that only changes letter case will cause git to report spurious conflicts or create duplicate files(from Mark Amery's comment).
Documentation
From the git config documentation:
core.ignorecase
If true, this option enables various workarounds to enable git to work better on filesystems that are not case sensitive, like FAT. For example, if a directory listing finds makefile when git expects Makefile, git will assume it is really the same file, and continue to remember it as Makefile.
The default is false, except git-clone(1) or git-init(1) will probe and set core.ignorecase true if appropriate when the repository is created.
Case-insensitive file-systems
The two most popular operating systems that have case-insensitive file systems that I know of are
Windows
OS X
Using SourceTree I was able to do this all from the UI
Rename FILE.ext to whatever.ext
Stage that file
Now rename whatever.ext to file.ext
Stage that file again
It's a bit tedious, but if you only need to do it to a few files it's pretty quick
Fix git filename case on whole repo:
git rm -r --cached .
git add --all .
git status ##Review that **only** changes staged are renames
## Commit your changes after reviewing:
git commit -a -m "Fixing file name casing"
git push origin master
Explanation from #Uriahs Victor comment:
What this command actually does is deletes the cached version of the
file/folder names that git thought still existed. So it will clear its
cache but leave everything in the current folder (where you've made
your changes locally) but it will see that those other wrong case
folders/files do not exist anymore so will show them as deleted in git
status. Then you can push up to github and it will remove the
folders/files with wrong cases. This answer has a graphic depicting
what the command means.
This is what I did on OS X:
git mv File file.tmp
git mv file.tmp file
Two steps because otherwise I got a “file exists” error. Perhaps it can be done in one step by adding --cached or such.
Sometimes it is useful to temporarily change Git's case sensitivity.
Method #1 - Change case sensitivity for a single command:
git -c core.ignorecase=true checkout mybranch to turn off case-sensitivity for a single checkout command. Or more generally: git -c core.ignorecase= <<true or false>> <<command>>. (Credit to VonC for suggesting this in the comments.)
Method #2 - Change case sensitivity for multiple commands:
To change the setting for longer (e.g. if multiple commands need to be run before changing it back):
git config core.ignorecase (this returns the current setting, e.g. false).
git config core.ignorecase <<true or false>> - set the desired new setting.
...Run multiple other commands...
git config core.ignorecase <<false or true>> - set config value back to its previous setting.
We can use git mv command. Example below , if we renamed file abcDEF.js to abcdef.js then we can run the following command from terminal
git mv -f .\abcDEF.js .\abcdef.js
Under OSX, to avoid this issue and avoid other problems with developing on a case-insensitive filesystem, you can use Disk Utility to create a case sensitive virtual drive / disk image.
Run disk utility, create new disk image, and use the following settings (or change as you like, but keep it case sensitive):
Make sure to tell git it is now on a case sensitive FS:
git config core.ignorecase false
Similar to #Sijmen's answer, this is what worked for me on OSX when renaming a directory (inspired by this answer from another post):
git mv CSS CSS2
git mv CSS2 css
Simply doing git mv CSS css gave the invalid argument error: fatal: renaming '/static/CSS' failed: Invalid argument perhaps because OSX's file system is case insensitive
p.s BTW if you are using Django, collectstatic also wouldn't recognize the case difference and you'd have to do the above, manually, in the static root directory as well
rename file Name.jpg to name1.jpg
commit removed file Name.jpg
rename file name1.jpg to name.jpg
amend added file name.jpg to previous commit
git add name.jpg
git commit --amend
I tried the following solutions from the other answers and they didn't work:
git mv filename
git rm -f filename
If your repository is hosted remotely (GitHub, GitLab, BitBucket), you can rename the file on origin (GitHub.com) and force the file rename in a top-down manner.
The instructions below pertain to GitHub, however the general idea behind them should apply to any remote repository-hosting platform. Keep in mind the type of file you're attempting to rename matters, that is, whether it's a file type that GitHub deems as editable (code, text, etc) or uneditable (image, binary, etc) within the browser.
Visit GitHub.com
Navigate to your repository on GitHub.com and select the branch you're working in
Using the site's file navigation tool, navigate to the file you intend to rename
Does GitHub allow you to edit the file within the browser?
a.) Editable
Click the "Edit this file" icon (it looks like a pencil)
Change the filename in the filename text input
b.) Uneditable
Open the "Download" button in a new tab and save the file to your computer
Rename the downloaded file
In the previous tab on GitHub.com, click the "Delete this file" icon (it looks like a trashcan)
Ensure the "Commit directly to the branchname branch" radio button is selected and click the "Commit changes" button
Within the same directory on GitHub.com, click the "Upload files" button
Upload the renamed file from your computer
Ensure the "Commit directly to the branchname branch" radio button is selected and click the "Commit changes" button
Locally, checkout/fetch/pull the branch
Done
With the following command:
git config --global core.ignorecase false
You can globally config your git system to be case sensitive for file and folder names.
Mac OSX High Sierra 10.13 fixes this somewhat. Just make a virtual APFS partition for your git projects, by default it has no size limit and takes no space.
In Disk Utility, click the + button while the Container disk is selected
Select APFS (Case-Sensitive) under format
Name it Sensitive
Profit
Optional: Make a folder in Sensitive called git and ln -s /Volumes/Sensitive/git /Users/johndoe/git
Your drive will be in /Volumes/Sensitive/
How do I commit case-sensitive only filename changes in Git?
so there are many solutions to this case sensitivity deployment problem with how GitHub handles it.
In my case, I had changed the filename casing convention from uppercase to lowercase.
I do believe that git can track the change but this command
git config core.ignorecase false dictates how git operates behind the scenes
In my case, I ran the command and git suddenly had lots of files to track labeled untracked.
I then hit git add. , then git committed and ran my build on netlify one more time.
Then all errors now displayed could be traced e.g
Module not found: Can't resolve './Components/ProductRightSide' in '/opt/build/repo/components/products and fixed such that git was able to track and implement the changes successfully.
It's quite a workaround and a fingernail away from frustration but trust me this will surely work.
PS: after fixing your issue you may want to run the command
git config core.ignorecase true to restore how git works with case sensitivity.
Also, note git config core.ignorecase false has issues with other filename extensions so you may want to watch out, do it if you know what you’re doing and are sure of it.
Here's a thread on netlify that can help out, possibly
When you've done a lot of file renaming and some of it are just a change of casing, it's hard to remember which is which. manually "git moving" the file can be quite some work. So what I would do during my filename change tasks are:
remove all non-git files and folder to a different folder/repository.
commit current empty git folder (this will show as all files deleted.)
add all the files back into the original git folder/repository.
commit current non-empty git folder.
This will fix all the case issues without trying to figure out which files or folders you renamed.
I've faced this issue several times on MacOS. Git is case sensitive but Mac is only case preserving.
Someone commit a file: Foobar.java and after a few days decides to rename it to FooBar.java. When you pull the latest code it fails with The following untracked working tree files would be overwritten by checkout...
The only reliable way that I've seen that fixes this is:
git rm Foobar.java
Commit it with a message that you cannot miss git commit -m 'TEMP COMMIT!!'
Pull
This will pop up a conflict forcing you to merge the conflict - because your change deleted it, but the other change renamed (hence the problem) it
Accept your change which is the 'deletion'
git rebase --continue
Now drop your workaround git rebase -i HEAD~2 and drop the TEMP COMMIT!!
Confirm that the file is now called FooBar.java
I took #CBarr answer and wrote a Python 3 Script to do it with a list of files:
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import os
import shlex
import subprocess
def run_command(absolute_path, command_name):
print( "Running", command_name, absolute_path )
command = shlex.split( command_name )
command_line_interface = subprocess.Popen(
command, stdout=subprocess.PIPE, cwd=absolute_path )
output = command_line_interface.communicate()[0]
print( output )
if command_line_interface.returncode != 0:
raise RuntimeError( "A process exited with the error '%s'..." % (
command_line_interface.returncode ) )
def main():
FILENAMES_MAPPING = \
[
(r"F:\\SublimeText\\Data", r"README.MD", r"README.md"),
(r"F:\\SublimeText\\Data\\Packages\\Alignment", r"readme.md", r"README.md"),
(r"F:\\SublimeText\\Data\\Packages\\AmxxEditor", r"README.MD", r"README.md"),
]
for absolute_path, oldname, newname in FILENAMES_MAPPING:
run_command( absolute_path, "git mv '%s' '%s1'" % ( oldname, newname ) )
run_command( absolute_path, "git add '%s1'" % ( newname ) )
run_command( absolute_path,
"git commit -m 'Normalized the \'%s\' with case-sensitive name'" % (
newname ) )
run_command( absolute_path, "git mv '%s1' '%s'" % ( newname, newname ) )
run_command( absolute_path, "git add '%s'" % ( newname ) )
run_command( absolute_path, "git commit --amend --no-edit" )
if __name__ == "__main__":
main()
If nothing worked use git rm filename to delete file from disk and add it back.
I made a bash script to lowercase repository file names for me:
function git-lowercase-file {
tmp="tmp-$RANDOM-$1"
git mv -f $1 $tmp
git mv -f $tmp ${1,,}
}
then you can use it like this:
git-lowercase-file Name.jpg
Or simply rename the required file over git repository web UI interface and commit :)
If you're doing more complex change, like directory name casing change,
you can make that change from a Linux machine, because Linux itself (as well as git on Linux) treats files/directories with same names but different casing as completely different files/directories.
So if you're on Windows, you can install Ubuntu using WSL, clone your repo there, open the cloned repo directory using VSCode (use WSL remote extension to access WSL Ubuntu from Windows), then you will be able to make your renames through VSCode and commit/push them using VSCode git integration.
I have an www-data running with php which is controlling a git server.
The www-data user creates Unix users (having given it sudo adduser), and those users are supposed to control their own private git directory, where each user can house his/hers repositories.
I've followed this guide at least ten times, in addition to which, I'm also following this guide in order to create the git server.
Apache adds a unix user bar, with a home in /var/www/git/bar and the user has no password (--disable-password)
The user bar is part of group gitusers which allows +rwx to the group members, and has his shell set to /usr/bin/git-shell.
This is done so that www-data can access his home directory and populate it with repositories and ssh keys.
The skeleton home directory is also populated with git-shell-commands and the user www-data creates an /var/www/git/bar/.ssh/authorized_keys where it appends my test user's foo public key.
When www-data adds a new user and then a new repository it does:
sudo adduser --disabled password\
--home /var/www/git/bar\
--conf /var/www/conf/adduser.conf\
--ingroup gitusers\
bar
The above ^^ is done via php.
The .ssh and authorized_keys are owned by www-data.
Then www-data proceeds to create a new directory and initialize it:
mkdir /var/www/git/bar/test.git
cd /var/www/git/bar/test
git --bare init
My test user foo can read it from ssh (it just clones an empty repository).
Once I try to push an initial commit:
git clone ssh://foo#localhost:/var/www/git/bar/test.git
cd test
touch readme
vim readme
git add .
git commit -m "init"
git push origin master
foo#127.0.0.1's password:
Counting objects: 6, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), 411 bytes | 0 bytes/s, done.
Total 6 (delta 0), reused 0 (delta 0)
remote: error: insufficient permission for adding an object to repository database ./objects
remote: fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To ssh://foo#localhost:/var/www/git/bar/random.git
! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'ssh://foo#localhost:/var/www/git/bar/random.git'
I am asked for foo user's password (which is the user with the public key).
This is NOT the Unix user who owns the home directory, that is user bar who has a disabled password.
why am I being asked for an ssh password? shouldn't they ssh key take care of that?
If I create a bar with a password, then I can use that git repository, replacing foo#localhost:/var/www/git/bar with bar#localhost:/var/www/git/bar
When I don't use ssh:// at all, I am still able to clone but get the same error when pushing asfoo but not as bar provided I enable the password.
What am I doing wrong?
Is it because the permissions of .ssh and authorized_keys are too open or not owned by bar?
Even when I go (as sudo) into the bar homedir and make everything owned by him, I still get the same error.
Finally, I have set my .ssh/config for test user foo so that:
Host localhost
Hostname 127.0.0.1
IdentityFile ~/.ssh/foo
User foo
It turns out it was indeed the permissions.
tailing the /var/log/auth.log provided the insight:
Authentication refused: bad ownership or modes for directory /var/www/git/bar
Googling this I figured out the problem was that the entire home directory was accessible by the group.
So back to square one, allowing www-data to be part of the group with +rwx is not a possibility, since it breaks ssh.
EDIT:
The comment from iveqy was very wise, using unix users for this sort of operation is an overkill and opens potential security holes, since it requires that you escalate user www-data to a superuser.
I've ended up using gitolite-admin the following way (I am adding it for future reference on how www-data the apache user under Debian/Ubuntu can control gitolite).
Instructions
The webserver apache2 runs as www-data on the system.
All php scripts are executed as www-data.
This requires that we enable this user to administrate our git server automatically.
For gitolite to work, it requires that the administrator provides an ssh key.
This in turn requires that user www-data (apache2) has a pair of ssh keys.
The key pair must be protected else ssh won't work:
create a new user git with home directory in: /var/www/git
sudo adduser git --home /var/www/git
cd into /var/dir/git and remove all skeleton files (.bashrc, .profile, .bash_logout)
log locally as user git: su - git and use the password you created earlier
make sure permissions of git user home dir are set to 755 (g+rx)
create the .ssh dir: mkdir .ssh and then make it private: chmod 700 -R .ssh
Now as www-data create your ssh key (exit from git user):
go into /var/www/ and create an ./ssh owned by www-data and with 700 mask.
sudo -u www-data ssh-keygen -t rsa
If everything went accordingly, copy your /var/www/.ssh/id_rsa.pub into /var/www/git/.ssh/
give ownership of key to user git: sudo chown git.git .ssh/id_rsa.pub
log back in as git: su - git
make the pub key private: chmod 600 .ssh/id_rsa.pub
Time to install gitolite:
git clone git://github.com/sitaramc/gitolite
mkdir -p $HOME/bin
gitolite/install -to $HOME/bin
Setup the www-data public rsa key to be used as the administrator of gitolite:
$HOME/bin/gitolite setup -pk .ssh/id_rsa.pub
You should get:
Initialised empty Git repository in /var/www/git/repositories/gitolite-admin.git/
Initialised empty Git repository in /var/www/git/repositories/testing.git/
WARNING: /var/www/git/.ssh/authorized_keys missing; creating a new one
(this is normal on a brand new install)
the user git is now setup.
logout: exit
DO NOT TOUCH THE DIRECTORY /var/www/git FROM NOW ON
We will clone it locally, and control it from there.
Clone it as user www-data:
first create the local copy owned by www-data: sudo mkdir gitolite-admin && chown -R www-data.www-data gitolite-admin/
then execute as user www-data the command: sudo -u www-data git clone git#localhost:gitolite-admin gitolite-admin/
This clones the gitolite-admin in /var/www/gitolite-admin from which you control the gitolite server.
For instructions on how to control the gitolite-server, see: https://github.com/sitaramc/gitolite
From now on, any gitolite command you execute, execute it as user www-data.
Any command run as root or with sudo will break the server!
PS: I accept no responsibility if this breaks your apache configuration. I hope it helps anyone coming here for the same reason I did: creating a git server. I'm open to suggestions and better ideas or improvements.
I have a Gitlab server (Ubuntu 14.04) where I am trying use it as both a host for my repositories as well as a testing server for my PHP projects. Ideally, I would like to have Gitlab/Git export the "release" branch to /var/www/git/<project-name> when that branch is updated.
My Question: How can I export a specific branch in Gitlab, to a specific directory on the localhost, when the branch is updated?
I am aware that there are webhooks available in Gitlab, but it seems unnecessary and wasteful to have the server POST to itself for a local operation.
I suppose you are running the community edition of gitlab.
Then, only the server administrator can configure hook scripts by copying the required scripts into the affected repositories.
gitlab itself is using the $GIT_DIR/hooks directory for its own scripts already. Fortunately they forward control to any hook script in the gitlab specific $GIT_DIR/custom_hooks directory. See also this question about how to run multiple hooks with the same type on gitlab.
The script itself could look like this:
#!/bin/bash
#
# Hook script to export current state of repo to a release area
#
# Always hardcode release area - if configured in the repo this might incur data loss
# or security issues
echo "Git hook: $0 running"
. $(dirname $0)/functions
git=git
release_root=/gitlab/release
# The above release directory must be accessible from the gitlab server
# and any client machines that want to access the exports. Please configure.
if [ $(git rev-parse --is-bare-repository) = true ]; then
group_name=$(basename $(dirname "$PWD"))
repo_name=$(basename "$PWD")
else
cd $(git rev-parse --show-toplevel)
group_name=$(basename $(readlink -nf "$PWD"/../..))
repo_name=$(basename $(readlink -nf "$PWD"/..))
fi
function do_release {
ref=$1
branch=$2
# Decide on name for release
release_date=$(git show -s --format=format:%ci $ref -- | cut -d' ' -f1-2 | tr -d -- -: | tr ' ' -)
if [[ ! "$release_date" =~ [0-9]{8}-[0-9]{6} ]]; then
echo "Could not determine release date for ref '$ref': '$release_date'"
exit 1
fi
dest_root="$release_root/$group_name/$repo_name"
dated_dir="dated/$release_date"
export_dir="$dest_root/$dated_dir"
# Protect against multiple releases in the same second
if [[ -e "$export_dir" ]]; then
export_dir="$export_dir-02"
dated_dir="$dated_dir-02"
while [[ -e "$export_dir" ]]; do
export_dir=$(echo $export_dir | perl -pe 'chomp; print ++$_')
dated_dir=$(echo $dated_dir | perl -pe 'chomp; print ++$_')
done
fi
# Create release area
if ! mkdir -pv "$export_dir"; then
echo 'Failed to create export directory: ' "$export_dir"
exit 1
fi
# Release
if ! git archive $branch | tar -x -C "$export_dir"; then
echo 'Failed to export!'
exit 1
fi
chmod a-w -R "$export_dir" # Not even me should change this dir after release
echo "Exported $branch to $export_dir"
( cd "$dest_root" && rm -f latest && ln -s "$dated_dir" latest )
echo "Adjusted $dest_root/latest pointer"
}
process_ref() {
oldrev=$(git rev-parse $1)
newrev=$(git rev-parse $2)
refname="$3"
set_change_type
set_rev_types
set_describe_tags
echo " Ref: $refname","$rev_type"
case "$refname","$rev_type" in
refs/heads/*,commit)
# branch
refname_type="branch"
function="branch"
short_refname=${refname##refs/heads/}
if [[ $short_refname == release ]]; then
echo " Push accepted. Releasing export for $group_name/$repo_name $short_refname"
do_release "$refname" "$short_refname"
else
echo " Push accepted. No releases done for $group_name/$repo_name $short_refname"
fi
;;
refs/tags/*,tag)
# annotated tag
refname_type="annotated tag"
function="atag"
short_refname=${refname##refs/tags/}
;;
esac
}
while read REF; do process_ref $REF; done
exit 0
The script was started based on this post-receive.send_email script which is already quoted on SO multiple times.
Configure a release area in the variable hardcoded in the script, or e.g. add a mechanism to read a config file in the repo. Maybe you want to give users control over this area. Depends on your security circumstances.
The release area must be accessible by the git#gitlab user, and of course by any client expecting the export.
The branch to export is hardcoded in the script.
The release area will be populated like this:
$release_root/$group_name/$repo_name/dated/$release_date
Plus a symbolic link latest pointing to the latest $release_date. The idea is that this is extensible to later be able to also export tags. If you expect to export different branches, a $branch should be included as a path component, too.
Access control of the gitlab server is not passed down to the directory structure. Currently I do this manually, and that is why I do not auto-populate all new repositories with this hook. I'd rather configure manually, and then adjust unix group permissions (and/or ACLs) on the $release_root/$groupname paths accordingly. This needs to be done only once per group and works because no one else is allowed to create new groups on my gitlab instance. This is very different from the default.
Anything else we can do for you? ;-)
I have very annoying problem here that I am completely lost on.
Am just trying to run a bash script from a php page.
The bash script is a long one.... so I created a caller.sh which calls the ./mainScript.sh to run in the background in the following:
nohup /bin/bash /home/test/customcoincode/CoinCreationBashFile.sh $coinName $coinNameAbreviation $blockReward $blockSpacing $targetTimespan $totalCoins $seedNode $nameSeedNode $headline >> /tmp/BASH2log.txt 2>&1 &
in reading my log file it seems some variables are not being passed in...
and at the following lines of code:
echo "Creating New Coin - Downloading code base repo"
echo "$localFolder/$coinName"
mkdir -p "$localFolder/$coinName";
cd "$localFolder/$coinName"
git clone "$baseRepository" "$localFolder/$coinName"
echo "Made it here 1"
i get outputs of:
Creating New Coin - Downloading code base repo
/home/test/Foocoin
cloning into '/home/test/Foocoin'
could not create directory '/var/www/.ssh'
host key verification failed
blah blah ....
Why is it looking in the /var/www/ directory?? works fine if I run the script from terminal?
many thanks
So to pack up my comments in an answer:
The shell script is now run as apache, as git uses ssh, corresponding config files are needed. Which were created in /var/www; apaches home directory. Apache did not have write permissions in /var/www thus could not create these files.
To resolve, create the /var/www/.ssh directory yourself and give www-data (or whatever user apache runs under in your system) write access to that folder.
Next, github requires you to authorize ssh keys. It is safer to create a new one for apache in the newly created /var/www/.ssh directory and add this key to your github keychain.