I have some data in file which contains Day in one of the lines. Based on the day, I want to copy the data into specific directory. How can i do this in PHP or Shell Script?
file1.txt:
101
RJ
Printer
Monday
file2.txt:
101
RJ
Switch
Wednesday
I would like to copy first data into a directory "Monday" and other one "Wednesday". Any suggestions in either PHP or Shell script?
The main point here is how you detect "the day".
Presuming that "day"-line contains "day" at the end of the string:
for i in file*.txt
do
day=$(cat $i| grep day$ | head -1)
[ -z "$day" ] && day=UNKNOWN
mkdir -p "$day"
cp "$1" "$day"
done
In the case:
when there are two ore more "days" in file, will be used the first one
when there are no "day" in file, file will be copied to UNKNOWN directory
Shell (if always 4th line):
for i in file*.txt; do cp $i "`sed -n 4p $i`/$i";done;
Related
I have this shell script running every 1 minute via cronjob, and the output will store in result.txt:
CODE 1 - shell script check.sh
#!/bin/sh
netstat -anp |grep 'SYN' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | awk '{$1=$1;print}' > result.txt
Here is the output of result.txt, where the number 1, 2 and 4 on the left side are from the total how many time an IP make SYN_FLOOD request:
1 73.207.15.237
2 73.229.249.226
4 137.74.155.171
So, to display the output on my website, I have the following PHP script file called index.php that reads this result.txt and display it when someone goes to url/index.php
CODE 2 - PHP script index.php
$file = 'result.txt';
$count = 0;
$handle = fopen($file, "r");
if ($handle) {
while (($ip_raw = fgets($handle)) !== false) {
$count++;
$syn_count = explode(' ',trim($ip_raw))[0];
$ip = substr($ip_raw, 2);
echo "<div>{$count}) <font color='orange'>[ $syn_count ]{$ip}</font> [ <a
href='https://www.abuseipdb.com/check/{$ip}' target='_blank'>info</a> ]</div>";
}
if($count==0)
echo "<div> <font color='orange'>No IP with SYN_RECV status at this moment</font></div>";
fclose($handle);
} else {
echo "<div>[maxidos | info]: <font color='red'>Error opening ip data</font></div>";
}
The output for this index.php is:
1) [ 1 ] 73.207.15.237 [ info ]
2) [ 2 ] 73.229.249.226 [ info ]
3) [ 4 ] 137.74.155.171 [ info ]
When the shell script run again from CODE 1, I might get different output in result.txt based on how many time the IP make SYN request. Example the new result.txt I will get like this:
4 73.207.15.237
1 77.129.349.226
2 133.74.155.171
Based on the new result.txt output above, you can see that only the first IP 73.207.15.237 has a previous history of doing syn request which was 1 time and the other 2 IPs are new. My question is, how do I sum up the total SYN request made by IP 73.207.15.237 which has a previous history of making SYN request of 1 time + 4 (current). The output that I want on the website including the new 2 IPs should be:
1) [ 5 ] 73.207.15.237 [ info ] -- has history of SYN count=1 and current SYN=4. So total SYN count, 1+4=5
2) [ 2 ] 73.229.249.226 [ info ] -- has history of SYN count but no current SYN count found in result.txt, so no sum is done here
3) [ 4 ] 137.74.155.171 [ info ] -- has history of SYN count but no current SYN count found in result.txt, so no sum is done here
4) [ 1 ] 77.129.349.226 [ info ] -- new IP
5) [ 2 ] 133.74.155.171 [ info ] -- new IP
So when the shell script keep running and if found the same IP making the same request, it should update the total SYN request on the left side. I'm not sure if I need to implement this via shell or PHP.
Maybe my brief question for this long post would be:
How do I SUM up the first word from a string with multi-line in result.txt based on distinct occurrence?
Edit 1 (My idea):
My idea, the output from result.txt will be stored in a new txt file called result_final.txt, then when the new result came from result.txt, we compare both result.txt and result_final.txt. So the website will only read result_final.txt. But how do I make this comparison? I started to think that I need some kind of algorithm to make this comparison, or is there a simpler way of implementing this in shell script or bash?
Edit 2 (about the given answer):
Based on the solution given by #ChrisBrownie55,
I made changes to CODE 1 that output the result.txt:
#!/bin/sh
netstat -anpt | awk '/SYN/{split($5,a,":");seen[a[1]]++}END{for(i in seen){print seen[i],i}}' | sed -e "s/ /:/g" > result.txt
and it will produce output for result.txt like below:
1:174.137.58.23
2:71.14.74.120
1:51.36.113.39
So, in order not to disturb my CODE 1, I created another script that will produce result_final.txt. This file is used to store the count:
So the code for producing the result_final.txt is based on answer from #ChrisBrownie55 with a little bit change on the output:
#!/bin/sh
file="result.txt"
file2="result_final.txt"
if [[ ! -f "$file2" || ! -s "$file2" ]]; then
echo "file2 not exist or empty. init"
cat $file > $file2
fi
# get all results in "<count>:<ip>" format
entries=`cat $file $file2 | tr ' ' ':'`
results=""
for entry in $entries; do
# separate the count and ip address
count=${entry/:*/}
ip=${entry/*:/}
# check for an existing result
result=`echo $results | grep -F "$ip"`
if [[ $result ]]; then
# if one was found, extract its count
currentCount=${result/:*/}
totalCount=$(( currentCount + count )) # add em up
# replace the old result entry with the new one
results=${results/$currentCount:$ip/$totalCount:$ip}
else
# if no result exists, create one
results="$results$count:$ip\n"
fi
done
# print results in "<count> <ip>" format
echo -e "$results" | tr ':' ' ' | sed -e "s/ /:/g" | grep "\S" | sort -nr > $file2
The above script will produce the following output for result_final.txt:
1:174.137.58.23
2:71.14.74.120
1:51.36.113.39
Edit 3: SOLVED Thank you so much #ChrisBrownie55
I thought I was having problem with the output, but I notice it was my mistake because I had this running on cronjob:
0 * * * * root cat /dev/null > /root/result_final.txt
I changed it to clear the result every 12AM:
0 0 * * * root cat /dev/null > /root/result_final.txt
The solution given by #ChrisBrownie55 works.
EDIT: I changed the answer because the old way of finding and replacing was overly complex, associative arrays make the code much simpler.
Associative Arrays
To add up all the counts, we will iterate over all of the lines inside of each file with a read-while loop. With each line, you can extract both the count and the ip.
Once we have our values for that line, we can check our associative array, results, for an existing entry.
if one is found: add the count to the existing count
otherwise: create an entry with count
Once that's all said and done, we can now print out the results associative array. To do this, we will iterate over the keys inside results with the ${!var[*]} syntax.
#!/bin/bash
declare -A results
while read entry; do
# separate the count and ip address
count=${entry/:*/}
ip=${entry/*:/}
# check for an existing result
if [[ ${results[$ip]} ]]; then
# add the count to existing entry
results[$ip]=$(( ${results[$ip]} + count ))
else
# if no result exists, create one
results[$ip]=$count
fi
done < <(cat result.txt result2.txt)
# print results in "<count>:<ip>" format
for ip in ${!results[*]}; do
echo ${results[$ip]}:$ip
done
What the heck is done < <(cat result.txt result2.txt)?
It is the combination of a read-while loop taking input from a file and process substitution which allows us to use commands as inputs where we can't use the pipe operator |.
See also associative arrays.
Shorter Option: awk + associative arrays
The awk command also supports associative arrays in its code. We can leverage this to quickly handle the two fields (count and ip) and convert them to an associative array and print them out easily.
#!/usr/bin/awk -f
BEGIN {
FS=":"
OFS=":"
}
{
ips[$2]+=$1
}
END {
for (ip in ips)
print ips[ip], ip
}
BEGIN Block
In this code, we start by defining the FS (field separator) and the OFS (output field separator). These allow us to configure awk to look split each line to fields by the : and, when we print them out, to put them back together with :.
Main Code Block
In awk, we don't need to declare our associative array or any initial values. We are able to access them right away and add to them as we did above.
With this, we are saying take the second field $2 (our IP address) and use it as the key for ips. Then, we'll add the value held in the first field $1 (the count) to it.
END Block
Here we're iterating over the keys in the associative array. With each key, we will print out the value at that key in the associative array and then the key itself. Each field is separated by the OFS.
Pipeline version
If you wish to avoid making new files, you can also use the pipeline version of this (aka one-liner).
<command> | awk 'BEGIN {FS=":"; OFS=":"} {ips[$2]+=$1} END {for (ip in ips) print ips[ip], ip}'
I'm setting up a script which takes some user data with the read command. Using this data I need to search the file range and then do some filtering.
Here's how it is,
Enter fromtime
read fromtime
Enter totime
read totime
Enter the ID
read id
Initially I SSH into a server and then there I have a directory, Records with path cd home/report/records here, I have:
REC_201901020345.gz (yyyymmddhhmm)
REC_201901120405.gz
REC_201903142543.gz
and so on.
These files have data along with the $id.
When the user inputs $fromtime and $totime it will be of format yyyymmddhh . Here, I need to go to that range of files and then grep for the $id and display. For example:
If $fromtime is 2019010103 and $totime is 2019031425. I need to go to only those specific range of files that is REC_201901020345.gz, REC_201901120405.gz, REC_201903142543.gz and perform the grep to find the id entered by the user.
I have tried this using an if condition but it doesn't seem to work. I am new to writing scripts like these. There might be mistakes when I have described everything here. Sorry for the same.
source config.sh
Enter fromtime
read fromtime
Enter totime
read totime
Enter the ID
read id
ssh $user#$ip
cd /home/report/records
# <-- need to know what to add here as described here, to navigate to the
# <-- specific range $fromtime-$totime. Then the command to find id will be
zfgrep $id *.gz
The result should be only the the data with the id's in the specified range of .gz files.
Try below command.
echo -e "$(ls -1 REC_????????????.gz 2>/dev/null)\nREC_${fromtime}##\nREC_${totime}##" | sort | sed -n "/##/,/##/p" | sed '1d;$d' | xargs zfgrep -a "$id"
Explanation:
'fromdate' and 'todate' along with a ## (say marker) is appended to the output of ls.
Sorted the input, resulting in desired file names enclosed with marker.
Both sed, prints only lines between marker.
Last one is the command, supposed to be executed for each file name.
You can omit pipes and all next commands, starting from end, and see how output is building.
To get the list of files within the given range (fromtime, totime), the following shell script may be used:
declare -i ta
for file in REC*.gz
do
ta=$(echo "${file}" | grep -oP 'REC_\K(.*)(?=[[:digit:]]{2}.gz)')
if [ "${ta}" ] ; then
if [ ${ta} -le ${totime} -a ${ta} -ge ${fromtime} ] ; then
echo -e "${file}"
fi
fi
done
I am looking for a command that will print all the contents of an archive (including sub-folders and it's files) without extracting the actual archive on the disk, but only on screen.
I achieve something using some other questions and answers from this site, and here is my command:
unzip -l test.zip | awk '/-----/ {p = ++p % 2; next} p {print $NF}'
The output:
0 04-11-2009 13:43 jodconverter-webapp-2.2.2/ 1815 04-11-2009 13:43 jodconverter-webapp-2.2.2/README.txt 0 04-11-2009 13:43 jodconverter-webapp-2.2.2/src/ 5349 04-11-2009 13:42 jodconverter-webapp-2.2.2/src/jodconverter-webapp-2.2.2-sources.jar 26436 04-11-2009 13:43 jodconverter-webapp-2.2.2/LICENSE.txt 3819 04-11-2009 13:43 jodconverter-webapp-2.2.2/ChangeLog.txt 3314202 04-11-2009 13:42 jodconverter-webapp-2.2.2/jodconverter-webapp-2.2.2.war
As you can see the output is one line, and includes some extra information that I don't really need.
I want an output of this kind:
jodconverter-webapp-2.2.2/
jodconverter-webapp-2.2.2/README.txt
jodconverter-webapp-2.2.2/src/
jodconverter-webapp-2.2.2/src/jodconverter-webapp-2.2.2-sources.jar
.
.
.
So not only I want to output the file names only (and their full path) and avoid any extra other information like time permissions and so on, but also I want to use something like break-line to distinguish different files.
Keep in mind that this command will run on a PHP file to get the contents of the file, so I don't know if this can help us to use the <br> to do the break lines.
Is that possible with a single command?
Well, I can't think a command that will return you back the output you need, but may someone else can know something more.
What I would do in your case is to split the line into array and play with it until you get what you want.
You actually have to find a pattern that will work for all cases.
With a brief look on the command's output I came with the following decisions:
You splitting the line using as separator the space (i.e explode() )
The pattern I can see is that there exist a time of the form xx:xx exactly before the actual file/directory ! So you can check if the current line is a valid 24-based time using a regular expression then that means if that is the case and you are in the $i position of the array the $i+1 is what you are looking for, and therefore you can copy that to a new array.
Repeat
I think that is a bit pain, but at least is a solution.
I wrote this in Linux BASH shell, but if there's a better solution in PHP that would be fine.
I need to produce a random selection from an array of 12 elements. This is what I've been doing so far:
# Display/return my_array that's been randomly selected:
# Random 0 to 11:
r=$(( $RANDOM % 12 ))
echo ${my_array[$r]}
Each time the call is made, it randomly selects an element. However, too often, it "randomly" selects the same element in a row sometimes several times. How can this be accomplish in BASH shell or PHP so make a random selection which is also not a repeat of the last one selected? Thanks!
r=$last
while [ "$last" = "$r" ]
do
r=$(($RANDOM % 12))
done
export last=$r
If you are calling the script again and again, then suppose the script name is test.sh you need to call it like . test.sh instead of ./test.sh, it will make the script run in current shell. Else even the export is not needed. Otherwise creating a temp file approach is another robust way of getting the last value.
You can create a permutation and then pop values from it
perm=`echo $perm | sed 's/[0-9]\+//'` #remove the first number from $perm
if [ -z "$perm" ];then #if perm == ""
perm=`shuf -e {0..12}` #create new permutation
#now perm="11 7 0 4 8 12 9 5 10 6 2 1 3" for example
fi
echo $perm | cut -d' ' -f1 #show the first number from $perm
Note that this script is stateful. It need to store the generated permutation between executions. Is does it by storing them in a shell variable $perm. Because shell scripts cannot modify the calling shell environment, you need to execute it itside your current shell:
source ./next-index.sh
having saved the script is to next-index.sh file.
You could alternatively save $perm to file between executions.
I have a small issue here, I need to be able to read a file of unknown size it could be a a few hundred lines or many more the log files change all the time and depending on when i check. I would like to have a method that is in php or in linux that i can read a range of lines from a file. I dont want to have to read the entire file in to php memory then remove the lines because the file may be larger then the allowed memory of php.
I also want it to be using default php modules or default linux tools dont want to need to install anything because it needs to be portable.
Edit:
For the linux based options I would like to be able to supply more then one range, i may need to get a few different ranges of lines I know how to do it in php by not in linux and to avoid reading past lines i have already read?
With awk:
awk 'NR>=10 && NR<=15' FILE
With awk (two ranges):
awk 'NR>=10 && NR<=15 || NR>=26 && NR<=28' FILE
With ed:
echo 2,5p | ed -s FILE
With ed and two ranges :
echo -e "2,5p\n7,8p" | ed -s FILE
Last but not least, a sed solution with two ranges (fastest solution, tested with time):
sed -n '2,5p;7,8p' FILE
What about something like
head -100 | tail -15
gives you lines 86-100
$ cat input.txt
q
w
e
r
t
y
u
i
$ sed -n 2,5p input.txt
w
e
r
t
while ($lines_read < last_line_desired) {
while ($line = fgets($filehandle, $buffersize) !== false)
if (line >= first_desired_line) {
push($interesting_lines, $line)
}
}
$lines_read++
}
Opening the file handle, selecting the appropriately large enough buffer size to cover any expected line lengths, etc. is up to you.
If you're reading files that are regularly appended to, you should look into the ftell and fseek functions to note where you are in your data and skip past all the old stuff before reading more.