Background: Non developer with bash shell scripting knowledge needs to query a READ ONLY DB instance and generate data for given variables.
Data in existing file
A1 B1
A2 B2
......
...
....
An Bn
I would like to connect to the mysql database within the bash script and loop through a select statement "n" number of times
select x, y,z from table tablet_vendors where a=A1 and b=B1
except that A1 and B1 need to be passed as parameters first time, A2 and B2 next time and to n in a for loop. I know the shell side of things, but don't know how to integrate both.
Tried something that can define all possible values for "a" at the beginning of the shell script, but it's way too tedious. Any help is appreciated.
I am open to suggestions on using other scripting languages if they are easy to work with mysql and if you can provide some guidance. My searches are pointing to php and python??
Here's a simpler version:
import MySQLdb as mysql
import csv
con = mysql.connect('localhost','user','password','dbname')
cur = con.cursor()
q = "select x, y,z from tablet_vendors where a=%s and b=%s"
with open('somefile.txt') as f:
rows = csv.reader(f,delimiter='\t')
for row in rows:
cur.execute(q,row)
result = cur.fetchone()
while result:
print result
result = cur.fetchone()
Any errors thrown by MySQL will stop execution. You can wrap the cur. lines in a try/except block to be more elegant.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import _mysql, sys
argumentfile = sys.argv[1]
with open(argumentfile) as f:
sqlarguments = f.readlines()
con = None
username, password = 'testuser', 'testpass'
database = 'testdb'
try:
con = _mysql.connect('localhost', username, password, database)
for line in sqlarguments:
con.query("SELECT x, y,z FROM TABLE tablet_vendors WHERE a='%s' and b='%s;'" % line.split('\t'))
result = con.use_result()
print "MySQL data returned: %s" % \
result.fetch_row()[0]
except _mysql.Error, e:
print "Error %d: %s" % (e.args[0], e.args[1])
sys.exit(1)
finally:
if con:
con.close()
More info on Python and MySQL: http://zetcode.com/db/mysqlpython/
(Skip to "First example" if you've got the MySQL and everything setup)
Note the retrieval of sqlarguments which is taken from a textfile which is passed as a parameter to the script as the first and only parameter like: python2 script.py sqlfile.txt.
Also note that i assumed that the SQL parameters are delimited by \t (tab) and not spaces, which i then pass in as a='...' and b='...' encapsulated in ' in case of spaces.
Related
I am interested to import this 5000 line .txt file into a mysql database.
P|1|A PRODUCT|1|0002000|204|123|
P|2|ANOTHER PRODUCT|10000371|0001990|055|031|
B|055|A BRAND NAME|
B|204|ANOTHER BRAND NAME|
G|123|GROUP NAME|
G|031|ANOTHER GROUP NAME|
Where P means that line refers to a "Product" table, B means "Brand" table and G means "Group". I need them separatedly in the database.
If it's impossible to do it only with sql, there is some way i'll get the result I want using PHP?
the php explode will do the job for you, here is how
$data = file_get_contents('data.txt')
// make an array for each line
$lines = explode('/n', $data)
foreach ($lines as $line) {
// Make array for each part of the line
$eline = explode('|', $line) ;
// $eline[0] is first part that have b or g or p
if ($eline[0] == 'B') {
// the sql code for brands
}
}
if you want i can complete the full code for you just tell me what the sql querys
It would be easiest if the file was separated by type. I'd use grep because its there already and saves writing the same functionality in php:
grep ^P file.txt > P.txt
grep ^G file.txt > G.txt
grep ^B file.txt > B.txt
LOAD DATA LOCAL INFILE can be used to import each individual file into its own table. Ensure mysql client is started with --local-infile to allow it to pass the file.
I have a bunch of R scripts that do some calculations and return a result. I am planning of building a PHP website that the user can actually submit a form where the data gets passed to my R script, processed and then return the result to the PHP and update the interface.
The plan is to have a database so when a user submits a form, the data gets stored in the database so R can read, process the input and then insert the result in the database so PHP can grab it. However, there are 2 problems:
How do my R script knows that certain values have been stored in the database so it can grab those values and do the processing?
When my R script finishes processing the data and insert it to mysql db, how do I get PHP to understand that at this moment PHP needs to query the database and grab the value?
Let's say my R script is like the following:
range<-1:20
m<-mean(range)
s<-sum(range)
print(m)
print(s)
As you can see the input at this case would be 1 and 20 to define the range, and the output is to show the values of m and s on my webpage.
Any idea how to accomplish that?
thanks!
shell_exec() or exec() are likely your best choices in PHP. This answer explains the difference.
echo shell_exec("Rscript my_script.R {$_GET['range']}");
I'm no r expert, but it's been done :
/ poorman.php
echo "
";
echo "Number values to generate:
";
echo "Submit";
echo ""
;
if(isset($_GET['N']))
{
$N = $_GET['N'];
// execute R script from shell
// this will save a plot at temp.png to the filesystem
exec("Rscript my_rscript.R $N");
// return image tag
$nocache = rand();
echo("");
}
?>
and the R script…
my_rscript.R
args <- commandArgs(TRUE)
N <- args[1]
x <- rnorm(N,0,1)
png(filename="temp.png", width=500, height=500)
hist(x, col="lightblue")
dev.off()
source
I'm struggling to understand this.
I have the following piece of code:
$result=$db->query(" select `job_id`,`jobs`.`client`
from `stores`
left join `jobs`
on `jobs`.`client`=`stores`.`id`
where `stores`.`postcode`='BD5 8HP'
and `stores`.`company`='MATALAN'
and `jobs`.`description`='MONTHLY external - Clean Both sides of External Glazing, Frames, Ledges & Entrance Doors (up to 1.8m Height) - Including Black out windows - Removing cobwebs and dust from all areas' ")->fetch();
echo "info:";
print_r($result);
die();
The query returns absolutely nothing, aka $result is an empty array.
If I run exactly the same query through PhpMyAdmin or from the MySQL binary client (on the server) I have reults.
The PHP code was a prepare -> execute construction, but since I was getting an empty result I tried to replace it (for test purpose only) with a direct query.
I had a look at a very similar topic on SO here which suggested that the problem was related to the binding, now I have no binding at all in above query, yet the result is an empty string.
Running other queries like, "select * from table" do return results, so I have NO clue on what can be wrong here.
Any idea/advice please?
As a footnote I'm running the query against a remote server. Can it be something with the charset?
EDIT:
While I received suggestion to change the value of DESCRIPTION, I cannot do that. That value is read out from a PDF and one location can have similar values, so using a LIKE and a % won't really work for me. I need to get the exact value matched.
EDIT 2
Ok the problem seems to be a bit more complex and the root cause of it might be somewhere deeper. If I manually write the string into the PHP code I do get the correct result, however when I pass it through as a variable it doesn't seem to work any more. Here is a bigger chunk of my script. The script is checking email with IMAP if an email matches some rules and it has attachment it opens the attachment which is a PDF, converts it to text, extract some images and updates a database accordingly.
$invoicedata=$a['attachment'];
$descriptorspec = array(
0 => array("pipe", "r"), // stdin is a pipe that the child will read from
1 => array("pipe", "w"), // stdout is a pipe that the child will write to
2 => array("file", "/tmp/error-output.txt", "a") // stderr is a file to write to
);
$cwd = '/tmp';
$process = proc_open('pdftotext - -', $descriptorspec, $pipes, $cwd);
if (is_resource($process)) {
fwrite($pipes[0], $invoicedata);
fclose($pipes[0]);
$all_text=stream_get_contents($pipes[1]);
fclose($pipes[1]);
proc_close($process);
preg_match("/ARL SUPPORT SERVICES (.*)/",$all_text,$extracted);
$company=$extracted[1];
preg_match("/[A-Z]{1,2}[0-9][0-9A-Z]?\s?[0-9][A-Z]{2}/",$all_text,$extracted);
$postcode=$extracted[0];
preg_match("/Date\n*(.*-)/",$all_text,$extracted);
$date=trim(str_replace("/","-",str_replace("-","",$extracted[1])));
$date=date("Y-m-d H:i:s",strtotime($date));
preg_match("/SPECIFICATION:((.|\n)*)EQUIPMENT/",$all_text,$extracted);
$job_desc=trim($extracted[1]));
preg_match("/Resource\n*(.*)/",$all_text,$extracted);
$who_signed=trim(str_replace("-","",$extracted[1]));
$db = new PDO('mysql:host=XXX.XXX.XXX.XXX;dbname=ZZZZZZ;charset=utf8', 'USER','PASSWORD');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$result=$db->query("select `job_id`,`jobs`.`client` from `stores`
left join `jobs` on `jobs`.`client`=`stores`.`id`
where `stores`.`postcode`='$postcode' and `stores`.`company`='$company'
and `jobs`.`description`='$job_desc'")->fetch();
echo "info: ";
print_r($result);
die();
In this case the $result is empty!
If I do:
echo "select `job_id`,`jobs`.`client` from `stores`
left join `jobs` on `jobs`.`client`=`stores`.`id`
where `stores`.`postcode`='$postcode' and `stores`.`company`='$company' and `jobs`.`description`='$job_desc'"
die();
It will just print:
select `job_id`,`jobs`.`client` from `stores` left join `jobs` on `jobs`.`client`=`stores`.`id` where `stores`.`postcode`='BD5 8HP' and `stores`.`company`='MATALAN' and `jobs`.`description`='MONTHLY external - Clean Both sides of External Glazing, Frames, Ledges & Entrance Doors (up to 1.8m Height) - Including Black out windows - Removing cobwebs and dust from all areas'
If I do copy back this into my PHP code then the code executes correctly, so the problem might be somewhere with converting the characters. I thought that it might be due to endash/emdash but checking the $job_desc with ord() returned char 45 for the - which is correct. also the & seem to be correct.
I'm running out of ideas, is there something so obvious which just doesn't jump out to me?
The problem was related with a wrong ASCII Char being fed from the PDFTOTEXT output.
implementing a str_replace function did solve the issue.
Thanks for all the support/suggestions.
Today I thought I would write a small bash script to create DB tables on a SQL database. What I thought would be quick turned out differently. Anyway I'm at a mental block and need advice. What I was want to do in the end is have the user enter the amount of columns in the table, then specify the column label and data type for the amount they entered to begin with. Then I'll take the array elements to assemble my SQL server command. It seemed rather obvious to me at first but not so much anymore. Any help is appreciated!
I was going to do something like...
#!/bin/bash
echo "Enter database name: "
read dbname
echo "Enter table name: "
read tablename
echo "Enter column count: "
read columns
echo "Enter database username: "
read dbuser
echo "Enter database user password: "
read dbpw
for i in ${columns[#]} do
echo "Column "$i" Name: "
read array_cname[$i]
echo "Column "$i" Data Type: "
read array_ctype[$i]
done
table="USE $dbname CREATE TABLE $tablename(array_cname[] array_ctype[] etc etc etc)"
mysql -u$dbuser -p$dbpw -e "$table"
The above is obviously a broken script, I just whipped it up after having cannibalized what I originally had going.
I'm just going to look at the part that you're struggling with - looping over the columns - and you should be able to take these lessons back to your script.
This is what I wrote, omitting the easy bits you've done, and setting columns to 2, to save typing whilst I tested.
#!/bin/bash
columns=2
declare -a cols
for ((i=1; i<=$columns; i++))
do
read -p "Column $i Name: " n
read -p "Column $i Data Type: " t
cols[$i]="$n $t"
done
echo "CREATE TABLE t (${cols[*]} etc etc etc)"
Explanation
I use declare to ensure that our array variable is of the correct type. Bash will infer it's an array when we first assign to an element, but I like to be clear.
To enumerate the integers from 1 to $columns, I use a Bash for loop.
You were missing the semicolon or newline between for and do.
Instead of trying to read directly into array elements, I read into two temporaries: n for name and t for type. I also use -p to make read emit the prompt for us.
I don't use two separate arrays for name and type, because Bash doesn't provide a simple way to interleave the values. Instead, I put name and type into a single array element. I could put them into separate elements if needed, with cols+=("$n" "$t").
To expand all elements of the array in a substitution, I use ${cols[*]}.
I haven't done any validation on the input; I'm assuming this script is a convenience for your own use, rather than a robust tool to share.
Test result
$ ./36337995.sh
Column 1 Name: a
Column 1 Data Type: int
Column 2 Name: b
Column 2 Data Type: int
CREATE TABLE t (a int b int etc etc etc)
I'm using a Serial to WiFi module which are able to send HTTP. I have made a sketch to update a Table in MySQL with a PHP script.
My problem is, I'm not able to update the Table when using variables. It works fine with static values.
I want to know if there is a problem in the Arduino sketch or the way to use the HTTP command.
Look at lines below from loop() and PHP script also:
float tm = 21.8;
Serial.write("AT+HTTPPH=/update1x.php?tmp=tm\n"); // Parse values to PHP script
If I insert the value 21.8 instead of variable tm, it works.
<?php
$aa = (isset($_GET['tmp']) ? $_GET['tmp'] : null);
mysql_connect('my.dk.mysql','my_dk','my_pw') or die("Can't connect that way!");
#mysql_select_db('my_dk') or die("Unable to select a database called 'My'");
date_default_timezone_set("Europe/Copenhagen");
$dat = date("Y-m-d");
$tim = date("H:i:s");
$qry = "INSERT INTO temp1(temp, date, time) VALUES('$aa','$dat','$tim')";
mysql_query($qry);
mysql_close();
exit('OK');
?>
because C does not "scan" the string and sobstitute "tm" with the value (think about what should happen all the time you used a variable called "i" or like...)
C is a lot "lower" level, you have to concatenate by hand. actually a easy one may be:
Serial.print("AT+HTTPPH=/update1x.php?tmp=");
Serial.print(tm);
Serial.print("\n");
using String class is possible, but you will use RAM dinamically, witch is not advised on a microcontroller:
String s = "AT+HTTPPH=/update1x.php?tmp=";
s += tm;
s += "\n";
Serial.print(s);
note that i concatenate one string for line; that is because on the right we don't have object String, but C string: array of char. That is not valid for tm
also real C concatenation (what is appening inside String class can be found here:
How do I concatenate const/literal strings in C?