I'm setting up a script which takes some user data with the read command. Using this data I need to search the file range and then do some filtering.
Here's how it is,
Enter fromtime
read fromtime
Enter totime
read totime
Enter the ID
read id
Initially I SSH into a server and then there I have a directory, Records with path cd home/report/records here, I have:
REC_201901020345.gz (yyyymmddhhmm)
REC_201901120405.gz
REC_201903142543.gz
and so on.
These files have data along with the $id.
When the user inputs $fromtime and $totime it will be of format yyyymmddhh . Here, I need to go to that range of files and then grep for the $id and display. For example:
If $fromtime is 2019010103 and $totime is 2019031425. I need to go to only those specific range of files that is REC_201901020345.gz, REC_201901120405.gz, REC_201903142543.gz and perform the grep to find the id entered by the user.
I have tried this using an if condition but it doesn't seem to work. I am new to writing scripts like these. There might be mistakes when I have described everything here. Sorry for the same.
source config.sh
Enter fromtime
read fromtime
Enter totime
read totime
Enter the ID
read id
ssh $user#$ip
cd /home/report/records
# <-- need to know what to add here as described here, to navigate to the
# <-- specific range $fromtime-$totime. Then the command to find id will be
zfgrep $id *.gz
The result should be only the the data with the id's in the specified range of .gz files.
Try below command.
echo -e "$(ls -1 REC_????????????.gz 2>/dev/null)\nREC_${fromtime}##\nREC_${totime}##" | sort | sed -n "/##/,/##/p" | sed '1d;$d' | xargs zfgrep -a "$id"
Explanation:
'fromdate' and 'todate' along with a ## (say marker) is appended to the output of ls.
Sorted the input, resulting in desired file names enclosed with marker.
Both sed, prints only lines between marker.
Last one is the command, supposed to be executed for each file name.
You can omit pipes and all next commands, starting from end, and see how output is building.
To get the list of files within the given range (fromtime, totime), the following shell script may be used:
declare -i ta
for file in REC*.gz
do
ta=$(echo "${file}" | grep -oP 'REC_\K(.*)(?=[[:digit:]]{2}.gz)')
if [ "${ta}" ] ; then
if [ ${ta} -le ${totime} -a ${ta} -ge ${fromtime} ] ; then
echo -e "${file}"
fi
fi
done
Related
I have this shell script running every 1 minute via cronjob, and the output will store in result.txt:
CODE 1 - shell script check.sh
#!/bin/sh
netstat -anp |grep 'SYN' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | awk '{$1=$1;print}' > result.txt
Here is the output of result.txt, where the number 1, 2 and 4 on the left side are from the total how many time an IP make SYN_FLOOD request:
1 73.207.15.237
2 73.229.249.226
4 137.74.155.171
So, to display the output on my website, I have the following PHP script file called index.php that reads this result.txt and display it when someone goes to url/index.php
CODE 2 - PHP script index.php
$file = 'result.txt';
$count = 0;
$handle = fopen($file, "r");
if ($handle) {
while (($ip_raw = fgets($handle)) !== false) {
$count++;
$syn_count = explode(' ',trim($ip_raw))[0];
$ip = substr($ip_raw, 2);
echo "<div>{$count}) <font color='orange'>[ $syn_count ]{$ip}</font> [ <a
href='https://www.abuseipdb.com/check/{$ip}' target='_blank'>info</a> ]</div>";
}
if($count==0)
echo "<div> <font color='orange'>No IP with SYN_RECV status at this moment</font></div>";
fclose($handle);
} else {
echo "<div>[maxidos | info]: <font color='red'>Error opening ip data</font></div>";
}
The output for this index.php is:
1) [ 1 ] 73.207.15.237 [ info ]
2) [ 2 ] 73.229.249.226 [ info ]
3) [ 4 ] 137.74.155.171 [ info ]
When the shell script run again from CODE 1, I might get different output in result.txt based on how many time the IP make SYN request. Example the new result.txt I will get like this:
4 73.207.15.237
1 77.129.349.226
2 133.74.155.171
Based on the new result.txt output above, you can see that only the first IP 73.207.15.237 has a previous history of doing syn request which was 1 time and the other 2 IPs are new. My question is, how do I sum up the total SYN request made by IP 73.207.15.237 which has a previous history of making SYN request of 1 time + 4 (current). The output that I want on the website including the new 2 IPs should be:
1) [ 5 ] 73.207.15.237 [ info ] -- has history of SYN count=1 and current SYN=4. So total SYN count, 1+4=5
2) [ 2 ] 73.229.249.226 [ info ] -- has history of SYN count but no current SYN count found in result.txt, so no sum is done here
3) [ 4 ] 137.74.155.171 [ info ] -- has history of SYN count but no current SYN count found in result.txt, so no sum is done here
4) [ 1 ] 77.129.349.226 [ info ] -- new IP
5) [ 2 ] 133.74.155.171 [ info ] -- new IP
So when the shell script keep running and if found the same IP making the same request, it should update the total SYN request on the left side. I'm not sure if I need to implement this via shell or PHP.
Maybe my brief question for this long post would be:
How do I SUM up the first word from a string with multi-line in result.txt based on distinct occurrence?
Edit 1 (My idea):
My idea, the output from result.txt will be stored in a new txt file called result_final.txt, then when the new result came from result.txt, we compare both result.txt and result_final.txt. So the website will only read result_final.txt. But how do I make this comparison? I started to think that I need some kind of algorithm to make this comparison, or is there a simpler way of implementing this in shell script or bash?
Edit 2 (about the given answer):
Based on the solution given by #ChrisBrownie55,
I made changes to CODE 1 that output the result.txt:
#!/bin/sh
netstat -anpt | awk '/SYN/{split($5,a,":");seen[a[1]]++}END{for(i in seen){print seen[i],i}}' | sed -e "s/ /:/g" > result.txt
and it will produce output for result.txt like below:
1:174.137.58.23
2:71.14.74.120
1:51.36.113.39
So, in order not to disturb my CODE 1, I created another script that will produce result_final.txt. This file is used to store the count:
So the code for producing the result_final.txt is based on answer from #ChrisBrownie55 with a little bit change on the output:
#!/bin/sh
file="result.txt"
file2="result_final.txt"
if [[ ! -f "$file2" || ! -s "$file2" ]]; then
echo "file2 not exist or empty. init"
cat $file > $file2
fi
# get all results in "<count>:<ip>" format
entries=`cat $file $file2 | tr ' ' ':'`
results=""
for entry in $entries; do
# separate the count and ip address
count=${entry/:*/}
ip=${entry/*:/}
# check for an existing result
result=`echo $results | grep -F "$ip"`
if [[ $result ]]; then
# if one was found, extract its count
currentCount=${result/:*/}
totalCount=$(( currentCount + count )) # add em up
# replace the old result entry with the new one
results=${results/$currentCount:$ip/$totalCount:$ip}
else
# if no result exists, create one
results="$results$count:$ip\n"
fi
done
# print results in "<count> <ip>" format
echo -e "$results" | tr ':' ' ' | sed -e "s/ /:/g" | grep "\S" | sort -nr > $file2
The above script will produce the following output for result_final.txt:
1:174.137.58.23
2:71.14.74.120
1:51.36.113.39
Edit 3: SOLVED Thank you so much #ChrisBrownie55
I thought I was having problem with the output, but I notice it was my mistake because I had this running on cronjob:
0 * * * * root cat /dev/null > /root/result_final.txt
I changed it to clear the result every 12AM:
0 0 * * * root cat /dev/null > /root/result_final.txt
The solution given by #ChrisBrownie55 works.
EDIT: I changed the answer because the old way of finding and replacing was overly complex, associative arrays make the code much simpler.
Associative Arrays
To add up all the counts, we will iterate over all of the lines inside of each file with a read-while loop. With each line, you can extract both the count and the ip.
Once we have our values for that line, we can check our associative array, results, for an existing entry.
if one is found: add the count to the existing count
otherwise: create an entry with count
Once that's all said and done, we can now print out the results associative array. To do this, we will iterate over the keys inside results with the ${!var[*]} syntax.
#!/bin/bash
declare -A results
while read entry; do
# separate the count and ip address
count=${entry/:*/}
ip=${entry/*:/}
# check for an existing result
if [[ ${results[$ip]} ]]; then
# add the count to existing entry
results[$ip]=$(( ${results[$ip]} + count ))
else
# if no result exists, create one
results[$ip]=$count
fi
done < <(cat result.txt result2.txt)
# print results in "<count>:<ip>" format
for ip in ${!results[*]}; do
echo ${results[$ip]}:$ip
done
What the heck is done < <(cat result.txt result2.txt)?
It is the combination of a read-while loop taking input from a file and process substitution which allows us to use commands as inputs where we can't use the pipe operator |.
See also associative arrays.
Shorter Option: awk + associative arrays
The awk command also supports associative arrays in its code. We can leverage this to quickly handle the two fields (count and ip) and convert them to an associative array and print them out easily.
#!/usr/bin/awk -f
BEGIN {
FS=":"
OFS=":"
}
{
ips[$2]+=$1
}
END {
for (ip in ips)
print ips[ip], ip
}
BEGIN Block
In this code, we start by defining the FS (field separator) and the OFS (output field separator). These allow us to configure awk to look split each line to fields by the : and, when we print them out, to put them back together with :.
Main Code Block
In awk, we don't need to declare our associative array or any initial values. We are able to access them right away and add to them as we did above.
With this, we are saying take the second field $2 (our IP address) and use it as the key for ips. Then, we'll add the value held in the first field $1 (the count) to it.
END Block
Here we're iterating over the keys in the associative array. With each key, we will print out the value at that key in the associative array and then the key itself. Each field is separated by the OFS.
Pipeline version
If you wish to avoid making new files, you can also use the pipeline version of this (aka one-liner).
<command> | awk 'BEGIN {FS=":"; OFS=":"} {ips[$2]+=$1} END {for (ip in ips) print ips[ip], ip}'
I am facing a problem while writing a shell script. i have table in a database where there is a text field.. i need to write that field into a file.
database fields are id, file_name, text
here is my script
#!/bin/bash
while read -a row
do
cat <<EOF >/root/${row[1]}
${row[2]}
EOF
sleep 1
done < <(echo "SELECT * FROM installation WHERE status = 1" | mysql -h 127.0.0.1 -u root -padmin -D data -s)
I have tried with
echo "${row[2]}" > /root/${row[1]}
But its writing the 1st line only, but the
text field is multiline text and i need write full text in that file
please help
By default read -r row will do word splitting by space/tab/new-line. Because the text field from table contains new-lines, word splitting would cut your text result. In this situation it's hard to avoid it.
I recommend you should replace the new-line to another mark, e.g. ### or just new-line, using REPLACE(text, "\n", 'new-line')(I am not familiar with sql REPlACE function) in SQL, then get result from shell, then use ${row[2]//new-line/$'\n'} to convert the new-line back.
I wrote this in Linux BASH shell, but if there's a better solution in PHP that would be fine.
I need to produce a random selection from an array of 12 elements. This is what I've been doing so far:
# Display/return my_array that's been randomly selected:
# Random 0 to 11:
r=$(( $RANDOM % 12 ))
echo ${my_array[$r]}
Each time the call is made, it randomly selects an element. However, too often, it "randomly" selects the same element in a row sometimes several times. How can this be accomplish in BASH shell or PHP so make a random selection which is also not a repeat of the last one selected? Thanks!
r=$last
while [ "$last" = "$r" ]
do
r=$(($RANDOM % 12))
done
export last=$r
If you are calling the script again and again, then suppose the script name is test.sh you need to call it like . test.sh instead of ./test.sh, it will make the script run in current shell. Else even the export is not needed. Otherwise creating a temp file approach is another robust way of getting the last value.
You can create a permutation and then pop values from it
perm=`echo $perm | sed 's/[0-9]\+//'` #remove the first number from $perm
if [ -z "$perm" ];then #if perm == ""
perm=`shuf -e {0..12}` #create new permutation
#now perm="11 7 0 4 8 12 9 5 10 6 2 1 3" for example
fi
echo $perm | cut -d' ' -f1 #show the first number from $perm
Note that this script is stateful. It need to store the generated permutation between executions. Is does it by storing them in a shell variable $perm. Because shell scripts cannot modify the calling shell environment, you need to execute it itside your current shell:
source ./next-index.sh
having saved the script is to next-index.sh file.
You could alternatively save $perm to file between executions.
I am total noob in scripting, and try to find out where is the problem in my code. I have a text files with customer details, first line of the file is username, second is real name. This details i need to pass to php. First, I need to find oldest file in directory. I created bash script which returns customer username:
oldest=`ls -tr1 temp/data/*.details|head -1`
head -n1 $oldest
This finds odest file (temp/data/xxx.details) and returns username, let's say xxx.
Now, to the php:
$username = shell_exec('./oldest_order.sh');
$name = shell_exec('head -2 temp/data/'.$username.'.details | sed -n 2p');
I can get username in my php, with the first line: it's xxx, but second line does not work, However, if I change '.$username.' to xxx, it works, so problem is somewhere around this variable.
Please advice.
Perhaps there is a new-line or space at the end or the beginning.
You can check that by doing a var_dump($username);. That will give you the actual length of the string so that you can compare it with the length you think it should have.
If it is white-space at the beginning or the end, you can use:
$name = shell_exec('head -2 temp/data/'.trim($username).'.details | sed -n 2p');
I have some data in file which contains Day in one of the lines. Based on the day, I want to copy the data into specific directory. How can i do this in PHP or Shell Script?
file1.txt:
101
RJ
Printer
Monday
file2.txt:
101
RJ
Switch
Wednesday
I would like to copy first data into a directory "Monday" and other one "Wednesday". Any suggestions in either PHP or Shell script?
The main point here is how you detect "the day".
Presuming that "day"-line contains "day" at the end of the string:
for i in file*.txt
do
day=$(cat $i| grep day$ | head -1)
[ -z "$day" ] && day=UNKNOWN
mkdir -p "$day"
cp "$1" "$day"
done
In the case:
when there are two ore more "days" in file, will be used the first one
when there are no "day" in file, file will be copied to UNKNOWN directory
Shell (if always 4th line):
for i in file*.txt; do cp $i "`sed -n 4p $i`/$i";done;