PHP: large amount of requests with file_get_contents - php

I currently have a db table filled with organisation numbers [1,002 million rows].
Now, what im trying to do is fetch the phone number of the organisation from a remote website API. I've got this working, but just after 30-50 or so requests to the API, i dont see any new changes to the table im inserting the phone numbers into. I've still got 1 million++ rows to fetch the phone number from, but i cant seem to get further than a small amount of rows.
Thanks in advance for the help.
I dont know if this is gonna help, but here is the code im using to do this.
// Remove timeout limit
// This is going to take alot of time!
set_time_limit(0);
// Initialize...
include $_SERVER['DOCUMENT_ROOT'] . '/core/Init.php';
// Profiles
$url = 'http://finnrett.no/API/business/get/quickresults?q=';
$profile_url = 'http://finnrett.no/API/business/get/profile?id=';
// Select names
$sql = 'SELECT organisasjonsnummer FROM brreg ORDER BY id LIMIT 10000';
$result = $Dbh -> query($sql, []);
// Each name
foreach ($result as $orgnr) {
// Pre for output explananation
echo'<pre>';
// Grab json from quick results url
$bedrift = json_decode(file_get_contents($url.$orgnr['organisasjonsnummer']));
$ID = get_object_vars($bedrift[0])['ID'];
$profile = json_decode(file_get_contents($profile_url.$ID));
$CONTACT = get_object_vars($profile);
$number = $CONTACT['contact'] ? $CONTACT['contact']: '0';
$sql = 'INSERT INTO profiles (orgnr, telefon) VALUES (:orgnr, :telefon)';
$args = ['orgnr' => $orgnr['organisasjonsnummer'], 'telefon' => $number];
if (!$Dbh -> query($sql, $args)) {
echo 'Sjekk opp org: ' . $orgnr['organisasjonsnummer'] . ' fordi her skjedde det noe galt.';
}
}
Looks like i solved it by choosing curl instead of file_get_contents.

Solved it by switching to curl instead of file_get_contents()

Related

Fetching Data From MySQL with $this->db

So, I am trying to get the IMDb id for the from the table called videos. I tried it with these these are giving me error.
Code 1:
$imdb_id = $this->db->get_where('videos', array('imdbid'))->result_array();
Code 2:
$imdb_id = $this->db->get('videos','imdbid');
code 3:
$query = $this->db->query("SELECT * FROM videos;");
$row = $query->row(0, 'videos');
$imdb_id = $row['imdbid'];
Here is a screenshot of database table. Thank you for your contribution in advance.
Please explain the error and output expectations.
you want only one return data or multiple data?
This Codeigniter 3 right?
the current code you are using is incorrect :
Code 1 :
// will get only one data with where condition
// videos_id = 1
$video_id = 1;
$data = $this->db->get_where('videos', array('videos_id'=>$video_id))->result_array();
$imdb_id = $data[0]['imdbid'];
Code 2 :
// will get all data without where condition
// must use loop foreach
$data = $this->db->get('videos')->result_array();

How can I get a value from a table into an XML without bringing in the column title?

I'm using the following function to bring a PaymentAmount from my database into an XML.
function GetPaymentTotal($orderID) {
Global $cnx;
$ptotal = null;
$sql = "SELECT value FROM orders_total WHERE class='ot_total' AND orders_id = '" . $orderID . "'";
$result = mysql_query($sql, $cnx) or die('Couldn\'t get order ot_shipping class from ' . SHOPPING_CART . ' orders_total table: ' . mysql_error());
if (mysql_num_rows($result) > 0)
{
$ptotal = mysql_fetch_assoc($result);
}
return $ptotal;
}
Using this to output:
$a_order['PaymentAmount'] = GetPaymentTotal($order->OrderNumber);
It's working, but for some reason it's bringing in the 'value' tags either side of my value... which I don't want. Here's how it looks at the minute:
<PaymentAmount>
<value>74.8501</value>
</PaymentAmount>
Here's how I want it to look.
<PaymentAmount>74.8501</PaymentAmount>
Any help would be greatly appreciated.
From the code you've presented, the XML you dislike:
<PaymentAmount>
<value>74.8501</value>
</PaymentAmount>
is fetched from the database. So your question suggests, that you have stored unwanted data into the database. However you have not shown how you store the data in the database with your question, so not a concrete programming example can be given how to fix storing the data.
So only the general answer seems applicable to say, that you need to change the way you store the data in the database.
From the other side you can try to fix the data when you fetch it from the database. However that seems really cumbersome, as you need to parse the data from the database first and then create a new XML Document in the format you want:
$dbData
= '<PaymentAmount>
<value>74.8501</value>
</PaymentAmount>';
$dbDoc = new SimpleXMLElement($dbData); # XML doc from database
$tag = $dbDoc->getName(); # Get tag name
$newDoc = new SimpleXMLElement("<$tag/>"); # XML doc for application
$newDoc->{0} = (string) $dbDoc->value[0]; # import value
$newData = explode("\n", $newDoc->asXML(), 2)[1]; # remove XML declaration
echo $newData; # <PaymentAmount>74.8501</PaymentAmount>

Run 2 sql query with 1 query for save query result as .csv format

In my web I've 2 page. 1) admin.php 2) csv.php. In admin.php page following query is showing data from db. In csv.php page I used same query to save data to .csv format but Can't save it.
I decided to run this same query in ONE QUERY. So that I can get the query result and can save it to csv format.
Questions:
1) How do i run this query to ONE query ?
2) Following query is showing data successfully. So how do i save it as .csv file ?
I search google for that and found many result which is showing how do I save data as .csv format with only one query. But you see that I've 2 while statement in my query then how do i save it as .csv format ? NO idea :(
Thanks and Looking for your help. :)
Note: I'm new learner about php and mysql.
$sqlagentdetails = "select * from users WHERE company_name != ''";
$rowresult = mysql_query($sqlagentdetails);
while($row = mysql_fetch_array($rowresult, MYSQL_ASSOC))
{
$pc1 = $row['pc1'];
$pc2 = $row['pc2'];
$pc3 = $row['pc3'];
$pc4 = $row['pc4'];
$emailAgent = $row['user_email'];
$user_id = $row['id'];
$myQuery = mysql_query("
SELECT *
FROM user_property upr
WHERE (postcode = '$pc1' OR
postcode = '$pc2' OR
postcode = '$pc3' OR
postcode = '$pc4') AND
datediff(CURDATE(), upr.creation_date) <= 7 AND
NOT EXISTS(SELECT ofr.property_id
FROM offers ofr
WHERE ofr.property_id = upr.property_id AND
ofr.agent_id IN(SELECT id
FROM users
WHERE company_name !=''
)
)
ORDER BY property_id DESC");
while($row = mysql_fetch_array($myQuery)){
// more data are goes to here...
}
}
1) How do i run this query to ONE query ?
You don't want it to run as one query. It's usually better to have lots of small simple queries instead of one complicated query. In fact I would suggest you update your code to have even more queries, for example the contents of the "not exists()" should not be done as a subquery, it should be a completely separate query to improve performance.
2) Following query is showing data successfully. So how do i save it as .csv file ?
There are two parts, first you need to send the correct HTTP headers to trigger a CSV download:
header('Content-type: application/vnd.ms-excel');
header('Content-Disposition: attachment; filename="export.csv";' );
Then just print out the data in CSV format:
while ($row = mysql_fetch_array($myQuery)) {
$first = true;
foreach ($row as $cell) {
if ($first)
$first = false;
else
print ',';
print '"' . addslashes($cell) . '"';
}
print "\n";
}
Note: CSV is a bad format, and this will only work in some editions of Microsoft Excel. Depending where the user lives (eg: Europe) it might not work properly. For most editions of Excel the above will work however. There is no good solution except to avoid using CSV.

MySQL: multiple search/select queries at the same time?

I have a question on how to go about the next phase of a project I am working on.
Phase I:
create a php script that scraped directory for all .txt file..
Open/parse each line, explode into array...
Loop through array picking out pieces of data that were needed and INSERTING everything into the database (120+ .txt files & 100k records inserted)..
this leads me to my next step,
Phase II:
I need to take a 'list' of several 10's of thousand of numbers..
loop through each one, using that piece of data (number) as the search term to QUERY the database.. if a match is found I need to grab a piece of data in a different column of the same record/row..
General thoughts/steps I plan to take
scrape directory to find 'source' text file.
open/parse 'source file'.... line by line...
explode each line by its delimiting character.. and grab the 'target search number'
dump each number into a 'master list' array...
loop through my 'master list' array.. using each number in my search (SELECT) statement..
if a match is found, grab a piece of data in another column in the matching/returned row (record)...
output this data.. either to screen or .txt file (havent decided on that step yet,..most likely text file through each returned number on a new line)
Specifics:
I am not sure how to go about doing a 'multiple' search/select statement like this?
How can I do multiple SELECT statements each with a unique search term? and also collect the returned column data?
is the DB fast enough to return the matching value/data in a loop like this? Do I need to wait/pause/delay somehow for the return data before iterating through the loop again?
thanks!
current function I am using/trying:
this is where I am currently:
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
$query = "SELECT guar_nu FROM placements WHERE har_id IN ($harNumArray2)";
echo $query;
$match = mysql_query($query);
//$match = mysql_query('"' . $query . '"');
$results = $match;
echo("<BR><BR>");
print_r($results);
I get these outputs respectively:
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN ("108383442","106620416","109570835","109700427","100022236")
&
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN (108383442,106620416,109570835,109700427,100022236)
Where do I stick this to actually execute it now?
thanks!
update:
this code seems to be working 'ok'.. but I dont understand on how to handle the retirned data correctly.. I seem to only be outputting (printing) the last variable/rows data..instead of the entire list..
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
//$query = "'SELECT guar_num FROM placements WHERE har_id IN ($harNumArray2)'";
$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (" . $harNumArray2 . ")")
//$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (0108383442,0106620416)")
or die(mysql_error());
// store the record of the "example" table into $row
$row = mysql_fetch_array($result);
$numRows = mysql_num_rows($result);
/*
while($row = #mysql_fetch_assoc($result) ){
// do something
echo("something <BR>");
}
*/
// Print out the contents of the entry
echo("TOTAL ROWS RETURNED : " . $numRows . "<BR>");
echo "HAR ID: ".$row['har_id'];
echo " GUAR ID: ".$row['guar_num'];
How do I handle this returned data properly?
thanks!
I don't know if this answers your question but I think you're asking about sub-queries. They're pretty straightforward and just look something like this
SELECT * FROM tbl1 WHERE id = (SELECT num FROM tbl2 WHERE id = 1);
That will only work if there is one unique value to that second subquery. If it returns multiple rows it will return a parse error. If you have to select multiple rows research JOIN statements. This can get you started
http://www.w3schools.com/sql/sql_join.asp
I am not sure how to go about doing a 'multiple' search/select statement like this?
With regards to a multiple select, (and I'll assume that you're using MySQL) you can perform that simply with the "IN" keyword:
for example:
SELECT *
FROM YOUR_TABLE
WHERE COLUMN_NAME IN (LIST, OF, SEARCH, VALUES, SEPARATED, BY COMMAS)
EDIT: following your updated code in the question.
just a point before we go on... you should try to avoid the mysql_ functions in PHP for new code, as they are about to be deprecated. Think about using the generic PHP DB handler PDO or the newer mysqli_ functions. More help on choosing the "right" API for you is here.
How do I handle this returned data properly?
For handling more than one row of data (which you are), you should use a loop. Something like the following should do it (and my example will use the mysqli_ functions - which are probably a little more similar to the API you've been using):
$mysqli = mysqli_connect("localhost", "user", "pass");
mysqli_select_db($mysqli, "YOUR_DB");
// make a comma separated list of the $ids.
$ids = join(", ", $id_list);
// note: you need to pass the db connection to many of these methods with the mysqli_ API
$results = mysqli_query($mysqli, "SELECT har_id, guar_num FROM placements WHERE har_id IN ($ids)");
$num_rows = mysqli_num_rows($results);
while ($row = mysqli_fetch_assoc($results)) {
echo "HAR_ID: ". $row["har_id"]. "\tGUAR_NUM: " . $row["guar_num"] . "\n";
}
Please be aware that this is very basic (and untested!) code, just to show the bare minimum of the steps. :)

error grabbing ad code from database using mysql

I am trying to grab ad code from my database and echo it on to the page, but for some reason it is not showing up?
$getad = ("SELECT * FROM ads WHERE place='non-mobile' AND who='adbrite' ");
while($rows = mysql_fetch_array($getad))
{
$code = $rows['code'];
}
$ad1 = $code;
later down the page i print it like this.
<?php print $ad1 ?>
I think your problem is that you don't actually execute the query, you just have saved it in a variable ($getad) and then try to do a fetch af an array containing a string as I see it. If I remeber correctly you have to save you query in a variable, as you did, and then type
$getad = "SELECT * FROM ads WHERE place='non-mobile' AND who='adbrite' ";
$q = $db->query($getad);
// generate results:
while ($q->fetchInto($row)) {
//display or store
}
You should also include checks, for example that this code has extracted at least one row, or that database connection is working, etcetera.

Categories