Efficient handling of IO in PHP - php

I'm using php, and handling file operation. My server need to response for every client request(concurrently minimum 5000 clients), My server open a xml file and convert xml to php array and will do some calculations and response as a json file, For this i'm using below code
$xmlstring = file_get_contents("../api/rate.xml");
$xml = simplexml_load_string($xmlstring);
$rate_json = json_encode($xml);
$rate_array = json_decode($rate_json, true);
$return_rates = array();
$return_rates['Baserates'] = $rate_array['Baserates'];
/* Here i will do some process and create $return_rates array */
$return_rates = json_encode($return_rates);
echo $return_rates;
This code producre resulat as i need, but i'm getting some times 500 internal server error, because of issue of handling IO(My server people saying this issue for internal server error). When many concurrent access happens in reading the file this issue happening, Please help any once to solve this issue.
XML File will be produce by third party application. For every second i'm receiving this XML file from one of my third party application.

You need to execute this code only when rate.xml changed. For first time, check if rate.xml content has changed by using hash value. If hash value is not same with last checked hash value then execute following code and store json file in local storage and new hash value.
$xmlstring = file_get_contents("../api/rate.xml");
$xml = simplexml_load_string($xmlstring);
$rate_json = json_encode($xml);
Otherwise you just need to read local json file and decode it. Or you can simply read from $xml without call to json_encode. Or you can request third party app developer to add JSON output instead of XML.
Also instead of file_get_contents(), you can use cUrl and sent HTTP HEAD first to find if rate.xml size changed instead of directly read rate.xml contents. When they do changed then you can call HTTP GET to retrieve their content.
The goal is to minimize network I/O and file I/O. Try cache rate.xml as long as possible in local storage or RAM (try Redis or Memcached)

Related

Sanitize an external xml file with php

Here's my problem:
function parse_xml_to_json($url) {
$fileContents = file_get_contents($url);
$simpleXml = simplexml_load_string($fileContents, null
, LIBXML_NOCDATA);
$json = json_encode($simpleXml);
return $json;
}
$jsonto = parse_xml_to_json('myxmlfile.html');
echo $jsonto;
Essentially I need to use an XML file from an external source and loop it through to display nicely some data.
I created a function that gets content from the external URL (file_get_contents), then I turn the string of XML into an object (I use LIBXML_NOCDATA as a parameter because it contains ), right after I turn the object into a JSON file and for the very last step, I echo the result.
So far so good, it worked but I'm wondering if I can do anything if the XML file contains a malicious script or else.
Is the function simplexml_load_string and then the JSON encode enough to prevent a malicious script or an invalid XML?
You code is prone to a Denial of Service (DOS) attack.
$fileContents = file_get_contents($url);
This can blow your memory limit. Or come close to, while taking a long time (the server you request the data from stales in the middle after providing a lot of content - and then only some little bytes each couple of seconds). So your script will "hang" while consuming the memory.
If the script can then be triggered with another HTTP request multiple times, this can consume your servers resources (the echo statement suggests this is entirely possible).

How do I cache and reuse a copy of GuzzleHttp\Psr7\Response for debugging purposes?

I'm processing a large API response that comes as GuzzleHttp\Psr7\Response. While debugging and profiling my script, I'd like to eliminate the varying network and API server response times.
My idea was to simply save a copy of the JSON that the server responds with into a text file, and keep reading it from the local filesystem until I'm done. Since I can't neither inject the JSON into the curl handler, nor serialize() it, I'd have to save the whole GuzzleHttp\Psr7\Response. But it contains some streams, which it later tries to read and fails.
I have tried serializing and immediately unserializing just to see if it works (it doesn't):
src/Handler/CurlFactory.php:106
if (!$easy->response || $easy->errno) {
return self::finishError($handler, $easy, $factory);
}
$temporaryCopy = serialize($easy->response);
$easy->response = unserialize($temporaryCopy);
src/Handler/CurlHandler.php:40
curl_exec($easy->handle);
$temporaryCopy = serialize($easy->handle);
$easy->handle = unserialize($temporaryCopy);
The above doesn't work. Before I continue fighting the framework, is there an out-of-the box way to get it done?

Traverse array structure with string

Can I store a pre-made array traversal?
I want to store several API calls, and also how I get to the relevant information from their response.
For example:
$url = 'http://maps.googleapis.com/maps/api/elevation/json?locations='.$location->$latitude.','.$location->$longitude.'&sensor=true';
$response = json_decode(file_get_contents($url), true);
$result = $response['results'][0]['elevation'];
Can I save this part as a string, for storage in my DB or a variable:
$elevation = "['results'][0]['elevation']";
Then later somehow use it to parse the response, ie.
$result = $response[$elevation];
The answer is no, sorry ! you will need to store your $response as it is and call it later on using the correct format $response['results'][0]['elevation']
You may however want to use serialize() if the problem is about how to persist the array into your database:
$db->insert(serialize($reponse));
then when you retrieve the response from your db use unserialize:
$response=unserialize($db->fetchReponse());
$elevation=$response['results'][0]['elevation'];
EDIT
Based on your comment below it seems what you need is a Cache. Whereby prior to sending the request to the web service API, your application checks in a cache to see if you already have the data available locally. As above example you would most likely want to serialize the PHP array or simply cache the raw response, given that it is in JSON format (PHP serialization will create something very similar anyway).
You would create the Cache key from the query params : location, etc.
Your cached object can be stored in a DB if you choose or on the file system, or even in Memory.
Check out ZF2 Cache component :
http://framework.zend.com/manual/2.0/en/modules/zend.cache.storage.adapter.html

Status report during form process

I created a little script that imports wordpress posts from an xml file:
if(isset($_POST['wiki_import_posted'])) {
// Get uploaded file
$file = file_get_contents($_FILES['xml']['tmp_name']);
$file = str_replace('&', '&', $file);
// Get and parse XML
$data = new SimpleXMLElement( $file , LIBXML_NOCDATA);
foreach($data->RECORD as $key => $item) {
// Build post array
$post = array(
'post_title' => $item->title,
........
);
// Insert new post
$id = wp_insert_post( $post );
}
}
The problem is that my xml file is really big, and when i submit the form, the browser just hangs for a couple of minutes.
Is it possible to display some messages during the import, like displaying a dot after every item is imported?
Unfortunately, no, not easily. Especially if you're building this on top of the WP framework you'll find it not worth your while at all. When you're interacting with a PHP script you are sending a request and awaiting a response. However long it takes that PHP script to finish processing and start sending output is how long it usually takes the client to start seeing a response.
There are a few things to consider if what you want is for output to start showing as soon as possible (i.e. as soon as the first echo or output statement is reached).
Turn off output buffering so that output begins sending immediately.
Output whatever you want inside the loop that would indicate to you the progress you wish to be know about.
Note that if you're doing this with an AJAX request content may not be ready immediately to transport to the DOM via your XMLHttpRequest object. Also note that some browsers do their own buffering before content can be ready for the user to display (like IE for example).
Some suggestions you may want to look into to speed up your script, however:
Why are you doing str_replace('&','&',$file) on a large file? You realize that has cost with no benefit, right? You've acomplished nothing and if you meant you want to replace the HTML entity & then you probably have some of your logic very wrong. Encoding is something you want to let the XML parser handle.
You can use curl_multi instead of file_get_contents to do multiple HTTP requests concurrently to save time if you are transferring a lot of files. It will be much faster since it's none-blocking I/O.
You should use DOMDocument instead of SimpleXML and a DOMXPath query can get you your array much faster than what you're currently doing. It's a much nicer interface than SimpleXML and I always recommend it above SimpleXML since in most cases SimpleXML makes things incredibly difficult to do and for no good reason. Don't let the name fool you.

json to php (server) possible and how?

Basically i want this json output to be transferred to my server/php.
i try this
$url="http://maps.google.com/maps/nav?q=from:9500 wil to:3000 bern";
$conte = file_get_contents($url);
echo $conte;
the json is not echo, how can i save the output to my server?
You need to urlencode the GET parameters:
echo file_get_contents('http://maps.google.com/maps/nav?q=from:9500%20wil%20to:3000%20bern');
# Returns
# {"name":"from:9500 wil to:3000 bern","Status":{"code":200,"request":"directions"},"Placemark":[{"id":"","address":"Wil, Switzerland","AddressDetails":{"Country":{"CountryNameCode":"CH","CountryName":"Schweiz","AdministrativeArea":{"AdministrativeAreaName":"St. Gallen","SubAdministrativeArea":{"SubAdministrativeAreaName":"Wil","Locality":{"LocalityName":"Wil"}}}},"Accuracy": 4},"Point":{"coordinates":[9.048081,47.463817,0]}},{"id":"","address":"Frohbergweg 7, 3012 Bern District, Switzerland","AddressDetails":{"Country":{"CountryNameCode":"CH","AdministrativeArea":{"AdministrativeAreaName":"BE","SubAdministrativeArea":{"SubAdministrativeAreaName":"Bern","Locality":{"LocalityName":"Bern District","DependentLocality":{"DependentLocalityName":"Länggasse-Felsenau","Thoroughfare":{"ThoroughfareName":"Frohbergweg 7"},"PostalCode":{"PostalCodeNumber":"3012"}}}}}},"Accuracy": 0},"Point":{"coordinates":[7.436386,46.954897,0]}}],"Directions":{"copyrightsHtml":"Map data \u0026#169;2010 Google, Tele Atlas ","summaryHtml":"178\u0026nbsp;km (about 2 hours 2 mins)","Distance":{"meters":177791,"html":"178\u0026nbsp;km"},"Duration":{"seconds":7343,"html":"2 hours 2 mins"},"Routes":[{"Distance":{"meters":177791,"html":"178\u0026nbsp;km"},"Duration":{"seconds":7343,"html":"2 hours 2 mins"},"summaryHtml":"178\u0026nbsp;km (about 2 hours 2 mins)","Steps":[{"descriptionHtml":"Head \u003Cb\u003Esouth\u003C\/b\u003E on \u003Cb\u003EToggenburgerstrasse\u003C\/b\u003E toward \u003Cb\u003ELerchenfeldstrasse\/\u003Cwbr\/\u003ERoute 16\/\u003Cwbr\/\u003ERoute 7\u003C\/b\u003E","Distance":{"meters":29,"html":"29\u0026nbsp;m"},"Duration":{"seconds":2,"html":"2 secs"},"Point":{"coordinates":[9.048030,47.463830,0]}},{"descriptionHtml":"Take the 1st left onto \u003Cb\u003ERoute 7\u003C\/b\u003E","Distance":{"meters":625,"html":"650\u0026nbsp;m"},"Duration":{"seconds":109,"html":"2 mins"},"Point":{"coordinates":[9.047930,47.463570,0]}},{"descriptionHtml":"At the traffic circle, take the \u003Cb\u003E1st\u003C\/b\u003E exit onto \u003Cb\u003EGeorg Rennerstrasse\u003C\/b\u003E","Distance":{"meters":871,"html":"850\u0026nbsp;m"},"Duration":{"seconds":77,"html":"1 min"},"Point":{"coordinates":[9.056170,47.463110,0]}},{"descriptionHtml":"Take the ramp to \u003Cb\u003EZürich\/\u003Cwbr\/\u003EFrauenfeld\u003C\/b\u003E","Distance":{"meters":330,"html":"350\u0026nbsp;m"},"Duration":{"seconds":22,"html":"22 secs"},"Point":{"coordinates":[9.053350,47.455800,0]}},{"descriptionHtml":"Merge onto \u003Cb\u003EA1\u003C\/b\u003E\u003Cdiv class=\"google_impnote\"\u003EToll road\u003C\/div\u003E","Distance":{"meters":173696,"html":"174\u0026nbsp;km"},"Duration":{"seconds":6790,"html":"1 hour 53 mins"},"Point":{"coordinates":[9.050270,47.453900,0]}},{"descriptionHtml":"Take exit \u003Cb\u003E36-Bern-Neufeld\u003C\/b\u003E toward \u003Cb\u003EBremgarten\u003C\/b\u003E","Distance":{"meters":579,"html":"600\u0026nbsp;m"},"Duration":{"seconds":33,"html":"33 secs"},"Point":{"coordinates":[7.436980,46.966570,0]}},{"descriptionHtml":"At the traffic circle, take the \u003Cb\u003E2nd\u003C\/b\u003E exit onto \u003Cb\u003ENeubrückstrasse\u003C\/b\u003E","Distance":{"meters":1357,"html":"1.4\u0026nbsp;km"},"Duration":{"seconds":243,"html":"4 mins"},"Point":{"coordinates":[7.429580,46.966790,0]}},{"descriptionHtml":"Turn right at \u003Cb\u003EMittelstrasse\u003C\/b\u003E","Distance":{"meters":146,"html":"150\u0026nbsp;m"},"Duration":{"seconds":24,"html":"24 secs"},"Point":{"coordinates":[7.437750,46.956720,0]}},{"descriptionHtml":"Take the 1st left onto \u003Cb\u003EBrückfeldstrasse\u003C\/b\u003E","Distance":{"meters":104,"html":"100\u0026nbsp;m"},"Duration":{"seconds":33,"html":"33 secs"},"Point":{"coordinates":[7.436060,46.956100,0]}},{"descriptionHtml":"Take the 1st right onto \u003Cb\u003EFrohbergweg\u003C\/b\u003E\u003Cdiv class=\"google_note\"\u003EDestination will be on the left\u003C\/div\u003E","Distance":{"meters":54,"html":"54\u0026nbsp;m"},"Duration":{"seconds":10,"html":"10 secs"},"Point":{"coordinates":[7.436830,46.955320,0]}}],"End":{"coordinates":[7.436234,46.955057,0]}}]}}
How do you want to save it? As a file?
If you can open via file_get_contents(), then URL opening for fopen() wrappers is on.
Then you can just do...
$url = 'http://maps.google.com/maps/nav?q=from:9500 wil to:3000 bern';
$content = file_get_contents($url);
file_put_contents('google-map-json.txt', $content);
You can get that into a usable object in PHP with json_decode().
You may want to do that if you want to save it to your database.
If you don't want to overwrite the file each time, you could generate a random hash of the response for the filename, or something similar.
Update
sorry my bad. i know how to save file. but the json is not even echoed through file_get_contents.
You may not have URL fopen() wrappers enabled.
You can find out by running this...
var_dump(ini_get('allow_url_fopen'));
If it is disabled, and you can't or don't want to turn it on, you can use cURL (if you have that library installed).
Update
When I tried to access the page via file_get_contents(), I got...
HTTP/1.0 400 Bad Request
You may need to use cURL, and mimic a browser (user agent etc).
Or you can set
ini_set('user_agent', 'Mozilla or something');
And then use file_get_contents().
Update
I tried with cURL too, and it didn't work :(
I think the next step is to examine all the headers your browser sends (when it works), and then send the equivalent via cURL.
Update
I noticed the Markdown editor wasn't liking the URL (see my OP's edit), and it dawned me - urlencode() the GET params!
You can write the data to a file using PHP's IO functions
$fp = fopen('data.txt', 'a');
fwrite($fp, $conte);
fclose($fp);
You can use json_decode to parse the data from the file_get_contents() variable. (I would personally use cURL instead of file_get_contents()).

Categories