Currently have a file that is set to read a CSV file. The CSV file contains 1600 api queries. Then each api query then returns more queries that need to be run. I am using Xampp v3.2.2 on windows 10 and running php v5.6.15. When running the file through my browser it ran fine for the first 800+ records in the CSV before timing out. When I rerun the file now I get an error "Site can't be reach ERR_CONNECTION_RESET". Not sure what could be causing this. abbreviated version of the code is included below
<?php
error_reporting(E_ERROR | E_PARSE);
set_time_limit (28800);
$csv = array_map('str_getcsv', file('file.csv'));
for($i = 0; $i < count($csv); $i++){
if($csv[$i][2] == 1){ //this item in csv is flag to check if this item has been run yet
if($csv[$i][3] != 'NULL' && trim($csv[$i][3]) != ''){ // check to make sure there is a URL
$return = file_get_contents(trim($csv[$i][3])); //Get the contents that links to new api calls
if($return){
$isCall = array(); // array to store all new calls
$data = array(); // array to store all data to put in csv
$doc = new DOMDocument('1.0'); // create new DOM object
$doc->loadHTML($return); // load page string into DOM object
$links = $doc->getElementsByTagName('a'); // get all <a> tags on the page
if($links->length > 0){ // if there is at least one <a> tag on page
for($j = 0; $j < $links->length; $j++){ // loop through <a> tags
$isCall[]= $links->item($j)->getAttribute('href'); // get href attribute from <a> tag and push into array
}
for($x = 0; $x < count($isCall); $x++){ // loop through all the calls and search for data
$string = file_get_contents($isCall[$x]);
if($string) {
$thispage = new DOMDocument('1.0');
$thispage->loadHTML($string);
$pagedata = $thispage->getElementsByTagName('div');
if ($pagedata->length > 0) {
for($j = 0; $j < $pagedata->length; $j++) {
$data[] = $pagedata->item($j)->C14N();
}
}
}
if(count($data) >= 5) break; // limiting to 5 data points to be added to csv
}
}
if(!empty($data)) $csv[$i] = array_merge($csv[$i], $data); // if we have data points lets add them to the row in the csv
}
}
$csv[$i][2] = 2; // set the flag to 2
$fp = fopen('file.csv', 'w'); // write the contents to the csv each time through loops so if it fails we start at last completed record
foreach ($csv as $f) {
fputcsv($fp, $f);
}
fclose($fp);
}
}
?>
Usually ERR_CONNECTION_RESET is an error that occurs when the site you are trying to connect to is unable to establish that connection. This usually happens due to reasons like firewall blocking on issues with ISP cache etc.
However in your case, I feel that the site you are connecting to is voluntarily closing connection attempts because what you are trying to do is loop over and hit that API site 1600 times continuously.
The API site is allowing the first 800-odd attempts but after that it gets worried that you are perhaps a malicious script trying to harm it. Like a classical example of DOS (Denial Of Service) attempt.
You should check if there is any restriction to the number of attempts a client can make to the API site with a fixed time (like say 500 hits every 24 hours) or you should try to sleep N seconds after each hit to the site or after each X number of hits to the site.
Related
I have my code already working, it will parse the files and insert the records, the issue that is stumping me as I have never had to do this is, how can I tell my code to parse 1-300 files then wait then parse the next "batch" 301-500 and so on until it's finished parsing all the files. I'm needed to parse over 50 thousand files, so obviously I'm reaching php's memory limit and execution time which has already been increased but I don't think I could set it extremely high to process 50 thousand.
I need help with how do I tell my code to run 1-x then rerun and run x-y?
My code is (Note, I am gathering more information that what's in my snip below)
$xml_files = glob(storage_path('path/to/*.xml'));
foreach ($xml_files as $file) {
$data = simplexml_load_file($file);
... Parse XML and get certain nodes ...
$name = $data->record->memberRole->member->name;
... SQL to insert record into DB ...
Members::firstOrCreate(
['name' => $name]
);
}
Simplest, if inelegant solution, is calling the script multiple times with an offset and using a for loop instead of forach.
$xml_files = glob(storage_path('path/to/*.xml'));
$offset = $_GET['offset'];
// Or if calling the script via command line:
// $offset = $argv[1];
$limit = $offset + 300;
for ($i = $offset; $i < $limit; $i++) {
$data = simplexml_load_file($xml_files[$i]);
// process and whatever
}
If you're calling the script as a web page, just add a query param like my-xml-parser.php?offset=300 and get offset like this: $offset = $_GET['offset'].
If you're calling this as a command line script, call it like this: php my-xml-parser.php 300, and get the offset from argv: $offset = $argv[1]
EDIT
If it's a web script, you can try and add a curl call that would call itself with the next offset without waiting for an answer.
I'm very new to PHP, making errors and learning as I go. Please be gentle! :)
I want to access some data from Blizzard.com's API. For this particular data set, it's not a block of data in JSON, rather each object has it's own URL to access. I estimate that there are approx 150000 objects, however I don't know the start or end points of the number range. So I'm having to assume 1 and work past the highest number I know (269065)
To get the data, I need to access each object's data via a JSON file, which I read, get the contents of & drop in to a text file (this could be written as an insert in to a SQL db too, as I'm able to do this if it's the text file that's the issue). But to be honest, I would love to get to the bottom of why this is happening as much as anything!
I wasn't going to try and run ~250000 iterations in a for loop, I thought I'd try something I considered small, 2000.
The for loop starts with $a as 1, uses $a as part of the URL, loads & decodes the JSON, checks to see if the first field (ID) in the object is set, if it is, it writes a few fields to data.txt & if the first field (ID) isn't set it just writes $a to data.txt (so I know it's a null for other purposes not outlined here).
Simple! Or so I thought, after approx after 183 iterations, the data written to the text file goes awry as seen by the quote below. It is out of sequence and starts at 1 again, then back to 184 ad nauseam. The loop then seems to be locked in some kind of infinite loop of running, outputting in a random order until I close the page 10-20 minutes later.
I have obviously made a big mistake! But I have no idea what I have done wrong to have caused this. During my attempts I have rewritten the code with new variable names, so a new text does not conflict with code that could be running in memory.
I've tried resetting variables to blank at the end of the loop in case it something was being reused that was causing a problem.
If anyone could point out any errors in my code, or suggest something for me to look in to, to handle bigger loops that would be brilliant. I am assuming my issue may be a time out or memory problem. But I don't know where to start & was hoping I'd find some suggestions here.
If it's relevant, I am using 000webhostapp.com as my host provider for now, until I get some paid for hosting.
1 ... 182 183 1 184 2 3 185 4 186 5 187 6 188 7 189 190 8 191
for ($a = 1; $a <= 2000; $a++) {
$json = "https://eu.api.battle.net/wow/recipe/".$a."?locale=en_GB&<MYPRIVATEAPIKEY>";
$contents = file_get_contents($json);
$data = json_decode($contents,true);
if (isset($data['id'])) {
$file = fopen("data.txt","a");
fwrite($file,$data['id'].",'".$data['name']."'\n");
fclose($file);
} else {
$file = fopen("data.txt","a");
fwrite($file,$a."\n");
fclose($file);
}
}
The content of the file I'm trying to access is
{"id":33994,"name":"Precise Strikes","profession":"Enchanting","icon":"spell_holy_greaterheal"}
I scrapped the original plan and wrote this instead. Thank you again who took the time out of their day to help and offer suggestions!
$b = $mysqli->query("SELECT id FROM `static_recipes` order by id desc LIMIT 1;")->fetch_object()->id;
if (empty($b)) {$b=1;};
$count = $b+101;
$write = [];
for ($a = $b+1; $a < $count; $a++) {
$json = "https://eu.api.battle.net/wow/recipe/".$a."?locale=en_GB&apikey=";
$contents = #file_get_contents($json);
$data = json_decode($contents,true);
if (isset($data['id'])) {
$write [] = "(".$data['id'].",'".addslashes($data['name'])."','".addslashes($data['profession'])."','".addslashes($data['icon'])."')";
} else {
$write [] = "(".$a.",'a','a','a'".")";
}
}
$SQL = ('INSERT INTO `static_recipes` (id, name, profession, icon) VALUES '.implode(',', $write));
$mysqli->query($SQL);
$mysqli->close();
$write = [];
for ($a = 1; $a <= 2000; $a++) {
$json = "https://eu.api.battle.net/wow/".$a."?locale=en_GB&<MYPRIVATEAPIKEY>";
$contents = file_get_contents($json);
$data = json_decode($contents,true);
if (isset($data['id'])) {
$write [] = $data['id'].",'".$data['name']."'\n";
} else {
$write [] = $a."\n";
}
}
$file = fopen("data.txt","a");
fwrite($file, implode('', $write));
fclose($file);
Also, why you are think what some IDS isn't duplicated at several "https://eu.api.battle.net/wow/[N]" urls data?
Also if you are I wasn't going to try and run ~250000 think about curl_multi_init(): http://php.net/manual/en/function.curl-multi-init.php
I can't really see anything obviously wrong with your code, can't run it though as I don't have the JSON
It could be possible that there is some kind of race condition since you're opening and closing the same file hundreds of times very quickly.
File operations might seem atomic but not necessarily so - here's an interesting SO thread:
Does PHP wait for filesystem operations (like file_put_contents) to complete before moving on?
Like some others' suggested - maybe just open the file before you enter the loop then close the file when the loop breaks.
I'd try it first and see if it helps.
There's nothing in your original code that would cause that sort of behaviour. PHP will not arbitrarily change the value of a variable. You are opening this file in append mode, are you certain that you're not looking at old data? Maybe output some debug messages as you process the data. It's likely you'd run up against some rate limiting on the API server, so putting a pause in there somewhere may improve reliability.
The only substantive change I'd suggest to your code is opening the file once and closing it when you're done.
$file = fopen("data_1_2000.txt", "w");
for ($a = 1; $a <= 2000; $a++) {
$json = "https://eu.api.battle.net/wow/recipe/$a?locale=en_GB&<MYPRIVATEAPIKEY>";
$contents = file_get_contents($json);
$data = json_decode($contents, true);
if (!empty($data['id'])) {
$data["name"] = str_replace("'", "\\'", $data["name"]);
$record = "$data[id],'$data[name]'";
} else {
$record = $a;
}
fwrite($file, "$record\n");
sleep(1);
echo "$a "; if ($a % 50 === 0) echo "\n";
}
fclose($file);
I would like an advice on best approach on this task.
I have a text log file holding data from a gps, this is the format:
time,lat,lon,elevation,accuracy,bearing,speed
2014-07-08T12:56:52Z,56.187344,10.192660,116.400024,5.000000,285.000000,1.063350
2014-07-08T12:56:58Z,56.187299,10.192754,113.799988,5.000000,161.000000,3.753000
2014-07-08T12:57:07Z,56.186922,10.193048,129.200012,5.000000,159.000000,5.254200
2014-07-08T12:57:13Z,56.186694,10.193133,109.799988,5.000000,152.000000,3.878100
2014-07-08T12:57:16Z,56.186745,10.193304,142.900024,5.000000,149.000000,3.940650
2014-07-08T12:57:20Z,56.186448,10.193417,118.700012,5.000000,154.000000,2.376900
2014-07-08T12:57:27Z,56.186492,10.193820,131.299988,5.000000,65.000000,5.379300
I need to find the line where the speed exceeds a certain value, then get the time from that line, then scroll trough the lines and find the line where the speed is below this value, get the time and write these 2 time values into my database.
This has to be an automated task, so I assume that a cron PHP script could do the job.
Best regards Thomas
Despite the fact that there is no special advice needed but wanting someone to code your problem - I will try to put you in the right direction. I've written easy to understand code where you can build on (untested)...
<?php
// Setup.
$pathGpsFile = 'gps.log';
$speedThreshold = 5;
//
// Execution.
//
if(!file_exists($pathGpsFile)) {
die('File "'. $pathGpsFile .'" does not exist.');
}
// Read entries into array.
$gpsEntries = file($pathGpsFile, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
// Loop through entries.
$lineCount = 0;
$currentDifferences = array();
$currentDifference = array();
foreach($gpsEntries as $gpsEntry) {
// Skip head.
if($lineCount == 0) {
$lineCount++;
continue;
}
// Extract values from gps entry.
list($time, $lat, $lon, $elevation, $accuracy, $bearing, $speed) = explode(',', $gpsEntry);
// Check if there is currently a difference monitored.
if(count($currentDifference) == 1) {
if($speed < $speedThreshold) {
$currentDifference[] = $gpsEntry;
}
// Add to differences list.
$currentDifferences[] = $currentDifference;
// Reset current difference.
$currentDifference = array();
} else {
if($speed > $speedThreshold) {
$currentDifference[] = $gpsEntry;
}
}
// Increase line count.
$lineCount++;
}
// Check output.
var_dump($currentDifferences);
?>
still struggling with PHP and CSV file manipulation. I will try to ask this properly so I can get some help.
I have a CSV file with about 4000 lines/rows, and I wrote this code to create an array of the entire CSV, and pull the the LAST line of the CSV file out to use in my script. The code works to to all this wit success.
// CREATE ASSOCIATIVE ARRAY FROM LAST ROW OF CSV FILE
$csv = array();
if (FALSE !== $handle = fopen("Alabama-TEST.csv", "r"))
{
while (FALSE !== $row = fgetcsv($handle))
{
$csv[] = $row;
}
}
$new_csv = array();
foreach ($csv as $row)
{
$new_row = array();
for ($i = 0, $n = count($csv[0]); $i < $n; ++$i)
{
$new_row[$csv[0][$i]] = $row[$i];
}
$new_csv[] = $new_row;
}
The variable $new_row is the last row in the CSV and I am able to use the data fine. But when the script is finished running I want to delete this last row called $new_row.
Here is an example of my CSV file at the top view;
CITY ADDRESS STATE NAME
Birmingham 123 Park St. Alabama Franky
So I just want to remove the last row and keep the header at the top, for the next time the script runs. I've been trying for 3 days solid trying to figure this out, so I'm hoping some kind person who KNOWS WHAT THEY ARE DOING can help.
Here is the code I have tried to remove the last row;
$inp = file('Alabama-TEST.csv');
$out = fopen('Alabama-TEST.csv','w');
or ($i=0;$i<count($inp)-1;$i++)
fwrite($out,$inp[$i]);
fclose($out);
Since I'm using a large file at around 4000 rows, is there a way to do this without using too much memory?
You need to use array_pop on your array ($new_csv) and fputcsv to save the file:
array_pop($new_csv); // Removes the last element in the array
$out = fopen('Alabama-TEST-new.csv','w');
foreach ($new_csv as $row) { // sorry for writing it badly, try now
fputcsv($out, $row);
}
fclose($out);
I don't have an environment to test this, sorry.
So I have a problem with a script I'm working on. I have a folder full of JSON files called roster0.json, roster1, etc. etc.
$dir = "responses/";
$files = glob($dir . "roster*");
$failed = array();
$failcnt = 0;
if (isset($files)) {
$data = null;
for ($i = 0; $i < count($files); $i++) {
$data = json_decode(utf8_decode(file_get_contents($files[$i])));
if(isset($data)){
// Process stuff
When I var_dump($files) I get an array with over 100 paths "responses/roster0.json".
When I test $data I get a proper array of data.
However, once the loop goes to the next file, it never loads it, and never processes it.
Here's the crazy part. If I change the start of the for loop, e.g. $i = 20. It will load the 21st file in the directory and parse it and insert it into the db properly!
Ignoring the failcnt stuff at the bottom, here's the current version of the script in it's entirety. http://pastebin.com/yqyKi5Ag
PS - I have full WARNING/ERROR reporting on in PHP and not getting any error messages...HELP! Thanks!
When I was writing the insert string the ID was being duplicated and thus was invalid. Switched to auto-inc and tada. It works. Thanks for the assistance. –