I have a simpleXML file that is being generated from a rest api call and will be fed into a database on a scheduled basis via a cronjob.
The results returned from the API calls are limited to 10 per page and the problem I am having is updating the page number parameter for the call to get the next set of results
$page_number = 1;
$page = $page_number++;
$api_url = "example.com/rest/api?products=new&pageid=$page";
I'm not sure how to get the variable $page to increment on each cron, any help is much appreciated
You could store in:
File => See other answers. You need access to file-system for this(But I think you should have this or move along).
Redis(or Memcached) => http://redistogo.com offers free instance of Redis and predis very popular PHP library to connect to redis. This solution will (almost' always work and will be very fast. You only need socket for this.
SQL => You should use PDO to connect to SQL if your hosting provider offers this. Also I think it should offer this or you should move along.
make new empty file in /tmp/file.txt
$page_number = file_get_contents("/tmp/file.txt");
if (!$page_number) $page_number = 1;
$page = $page_number + 1;
$f = fopen("/tmp/file.txt", "w+");
fwrite($f, $page);
fclose($f);
$api_url = "example.com/rest/api?products=new&pageid=$page";
Saving the number in a DB Or as a file with only the number as content would be the safest way.
$fname = '/tmp/pager_counter'; // don't use tempnam function here!!!
$fp = fopen($fname, 'w+');
$page = (int)fread($fp, filesize($fname));
if (!$page) {
$pgae=0;
}
$page++;
fwrite($fp, $page, strlen((string)$page));
$api_url = "example.com/rest/api?products=new&pageid=$page";
You could as easily use any another database or cache (MySQL, memcache, etc.)
Related
I am making a Covid-19 statistics website - https://e-server24.eu/ . Every time somebody is entering the website, the PHP script is decoding JSON from 3 urls and storing data into some variables.
I want to make my website more optimized so my question is: Is there any script that can update the variables data one time per day, not every time someone accesses the website?
Thanks,
I suggest looking into memory object caching.
Many high-performance PHP web apps use caching extensions (e.g. Memcached, APCu, WinCache), accelerators (e.g. APC, varnish) and caching DBs like Redis. The setup can be a bit involved but you can get started with a simple role-your-own solution (inspired by this):
<?php
function cache_set($key, $val) {
$val = var_export($val, true);
// HHVM fails at __set_state, so just use object cast for now
$val = str_replace('stdClass::__set_state', '(object)', $val);
// Write to temp file first to ensure atomicity
$tmp = sys_get_temp_dir()."/$key." . uniqid('', true) . '.tmp';
file_put_contents($tmp, '<?php $val = ' . $val . ';', LOCK_EX);
rename($tmp, sys_get_temp_dir()."/$key");
}
function cache_get($key) {
//echo sys_get_temp_dir()."/$key";
#include sys_get_temp_dir()."/$key";
return isset($val) ? $val : false;
}
$ttl_hours = 24;
$now = new DateTime();
// Get results from cache if possible. Otherwise, retrieve it.
$data = cache_get('my_key');
$last_change = cache_get('my_key_last_mod');
if ($data === false || $last_change === false || $now->diff($last_change)->h >= $ttl_hours ) { // cached? h: Number of hours.
// expensive call to get the actual data; we simple create an object to demonstrate the concept
$myObj = new stdClass();
$myObj->name = "John";
$myObj->age = 30;
$myObj->city = "New York";
$data = json_encode($myObj);
// Add to user cache
cache_set('my_key', $data);
$last_change = new DateTime(); //now
// Add timestamp to user cache
cache_set('my_key_last_mod', $last_change);
}
echo $data;
Voila.
Furthermore; you could look into client-side caching and many other things. But this should give you an idea.
PS: Most memory cache systems allow to define a time-to-live (TTL) which makes this more concise. But I wanted to keep this example dependency-free. Cache cleaning was omitted here. Simply delete the temp file.
Simple way to do that
Create a script which will fetch , decode JSON data and store it to your database.
Then set a Cron jobs with time laps of 24 hours .
And when user visit your site fetch the data from your database instead of your api provider.
It works fine but later sometime the count just goes down to random
number. My guess is my code cannot process multiple visits at a time.
Where increment heppens
Where it displays the count
<?php
$args_loveteam = array('child_of' => 474);
$loveteam_children = get_categories($args_loveteam);
if(in_category('loveteams', $post->ID)){
foreach ($loveteam_children as $loveteam_child) {
$post_slug = $loveteam_child->slug;
echo "<script>console.log('".$post_slug."');</script>";
if(in_category($loveteam_child->name)){
/* counter */
// opens file to read saved hit number
if($loveteam_child->slug == "loveteam-mayward"){
$datei = fopen($_SERVER['DOCUMENT_ROOT']."/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-".$post_slug."-2.txt","r");
}else{
$datei = fopen($_SERVER['DOCUMENT_ROOT']."/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-".$post_slug.".txt","r");
}
$count = fgets($datei,1000);
fclose($datei);
$count=$count + 1 ;
// opens file to change new hit number
if($loveteam_child->slug == "loveteam-mayward"){
$datei = fopen($_SERVER['DOCUMENT_ROOT']."/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-".$post_slug."-2.txt","w");
}else{
$datei = fopen($_SERVER['DOCUMENT_ROOT']."/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-".$post_slug.".txt","w");
}
fwrite($datei, $count);
fclose($datei);
}
}
}
?>
I would at least change your code to this
foreach ($loveteam_children as $loveteam_child) {
$post_slug = $loveteam_child->slug;
echo "<script>console.log('".$post_slug."');</script>";
if($loveteam_child->slug == "loveteam-mayward"){
$filename = "{$_SERVER['DOCUMENT_ROOT']}/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-{$post_slug}.txt";
}else{
$filename = "{$_SERVER['DOCUMENT_ROOT']}/wp-content/themes/inside-showbiz-Vfeb13.ph-updated/countlog-{$post_slug}-2.txt";
}
$count = file_get_contents($filename);
file_get_contents($filename, ++$count, LOCK_EX);
}
You could also try flock on the file to get a lock before modifying it. That way if another process comes along it has to wait on the first one. But file_put_contents works great for things like logging where you may have many processes competing for the same file.
Database should be ok, but even that may not be fast enough. It shouldn't mess up your data though.
Anyway hope it helps. This is kind of an odd question, concurrency can be a real pain if you have a high chance of process collisions and race conditions etc etc.
However as I mentioned (in the comments) using the filesystem is probably not going to provide the consistency you need. Probably the best for this may be some kind of in memory storage such as Redis. But that is hard to say without full knowing what you use it for. For example if it should persist on server reboot.
Hope it helps, good luck.
I am writing a PHP script which, upon a request, will make a call to a SOAP service with various parameters, some of which are taken from the request.
However, the particular SOAP service I am using requires that each request includes a unique ID, which in this case needs to increment for each request. It must not be based on time, and must be unique for each request, however it does not matter if values are skipped.
Using a MySQL data base to store a single value seems massively overkill. I have thought about storing and loading it into a file, but the issue of race conditions springs to mind.
I do have complete access to the server, which will be some kind of Linux flavour dedicated to this task.
Is there a simple way this can be achieved?
Before any new request get incremental value using PHP's time() function, since time will be unique for each request.
$increment_id = time();
If your application is single server you can try to store incremental ID in APC using:
$key = 'soap_service_name';
if (!apc_exists($key)) {
apc_store($key, 0);
}
$id = apc_inc($key);
You need to check if a key exists in APC cache and set 0, otherwise apc_inc fails and returns false
If you have multiserver application you can store incremental id in Memcache/Redis (that needs to run additional service):
$key = 'soap_service_name';
$memcache = memcache_connect('memcache_host', 11211);
if (!empty(memcache_exists($memcache, $key))) {
memcache_set($memcache, 0);
}
$id = memcache_increment($memcache, $key);
Same situation as APC if you call memcache_increment it will fail if key doesn't exists yet.
If that incremental ID should be stored persistently Redis would be more usefull because it has disk write of all data. It's kind of Memcache with disk write.
This is how I achieved this in the end. After considering the various options, databases and the various caching options seemed a bit overkill. In addition, caching, cookies and sessions seem to be designed to be relatively temporary, whereas I was really looking for a non-volatile solution.
This is what I came up with - a simple file locking solution. I hadn't realised PHP could deal with file locks but on discovering this, it seems the best way to go.
This example acquires an exclusive lock on the file, before reading and updating the value. If it hits max int, it resets. Then it waits for 5 seconds. If the script is called a few times in quick succession, observe that each request will wait for the lock to be release from the previous before continuing.
What's nice is, as this is PHP, non-existent file, invalid contents etc, will just cause the value to default to zero.
<?php
$f = fopen('sequence_num.txt', 'r+');
echo "Acquiring lock<br />\n";
flock($f, LOCK_EX);
echo "Lock acquired, updating value<br />\n";
$num = intval(fread($f, strlen(PHP_INT_MAX)));
echo "Old val = " . $num;
if ($num >= PHP_INT_MAX) {
$num = 0;
} else {
$num++;
}
echo " New val = " . $num;
echo "<br />Waiting 5 seconds<br />\n";
rewind($f);
ftruncate($f, 0);
fwrite($f, $num);
sleep(5);
echo "Releasing lock<br />\n";
flock($f, LOCK_UN);
fclose($f);
If you're happy to use a float as a unique value use:
$unique_id = microtime(true);
If you wish to simply increment, you may do so using a session var:
/**
* Get session increment.
*
* #param string $id
* #param int $default
* #return int
*/
function get_increment($id, $default = 0)
{
if (array_key_exists($id, $_SESSION)) $_SESSION[$id] += 1;
else $_SESSION[$id] = $default;
return $_SESSION[$id];
}
var_dump(get_increment('unique_id'));
I've got this variable which I want to save which is:
$counter = $counter['counter']++;
to make it so that it increments when I refresh the page.
Therefore, I decided to use the fopen, override the $counter variable and then save it.
I am still a beginner and I do not know how the fopen, fgets, fclose works.
So I wrote something like this.
$fp = fopen("file.php","r");
fwrite($fp, $counter);
fclose($fp);
So I wanted to open the file(file.php)(the file which is I am writing this code FYI), and then override the variable after it had been incremented and then save it by closing.
But the code doesn't seem to want to work and the variable does not seem to want to increment.
What am I doing wrong?
Do I want to have this $counter in a different file and pull the variable from there.
FYI: I don't want to use session_start() and $_SESSION because I am using cron job and it will not work.
EDIT
$result = mysql_query('SELECT MIN(ID) AS min, MAX(ID) AS max FROM ytable') or exit(mysql_error());
$row = mysql_fetch_assoc($result);
if($counter['counter'] < $row['max']){
if (isset($counter['counter'])){
$counter = $counter['counter']++;
}else{
$counter = $counter['counter'] = 0;
}
}
This is more of the code for those who are confused
You could read file like this. Open the file with previous counter & fetch the counter, increment it like this
$counter = readfile("file.php");
$counter++;
To write to the file, open the file in write mode like this
$fp = fopen("file.php","w");
fwrite($fp, $counter);
fclose($fp);
Is that result you are looking for ?
As chris85 suggested, database is used for this kind of cases. Better to use database instead of file handling.
So say you have file.txt and that contains 5. In that same directory you have script.php; this file would have
$counter = file_get_contents('file.txt'); // this takes in whatever value is in the file e.g in this case '5'
$counter++; // increment the value by 1 so we are now at 6
file_put_contents('file.txt', $counter); //write the value '6' back to the file
file.txt would then have 6, after the first load.
I would like to create a cache for my php pages on my site. I did find too many solutions but what I want is a script which can generate an HTML page from my database ex:
I have a page for categories which grabs all the categories from the DB, so the script should be able to generate an HTML page of the sort: my-categories.html. then if I choose a category I should get a my-x-category.html page and so on and so forth for other categories and sub categories.
I can see that some web sites have got URLs like: wwww.the-web-site.com/the-page-ex.html
even though they are dynamic.
thanks a lot for help
check ob_start() function
ob_start();
echo 'some_output';
$content = ob_get_contents();
ob_end_clean();
echo 'Content generated :'.$content;
You can get URLs like that using URL rewriting. Eg: for apache, see mod_rewrite
http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html
You don't actually need to be creating the files. You could create the files, but its more complicated as you need to decide when to update them if the data changes.
In my opinion this is the best solution. I use this for cache JSON file for my Android App. It can be simply use in other PHP files.
It's optimize file size from ~1mb to ~163kb (gzip).
Create cache folder in your directory
Then Create cache_start.php file and paste this code
<?php
header("HTTP/1.1 200 OK");
//header("Content-Type: application/json");
header("Content-Encoding: gzip");
$cache_filename = basename($_SERVER['PHP_SELF']) . "?" . $_SERVER['QUERY_STRING'];
$cache_filename = "./cache/".md5($cache_filename);
$cache_limit_in_mins = 60 * 60; // It's one hour
if (file_exists($cache_filename))
{
$secs_in_min = 60;
$diff_in_secs = (time() - ($secs_in_min * $cache_limit_in_mins)) - filemtime($cache_filename);
if ( $diff_in_secs < 0 )
{
print file_get_contents($cache_filename);
exit();
}
}
ob_start("ob_gzhandler");
?>
Create cache_end.php and paste this code
<?php
$content = ob_get_contents();
ob_end_clean();
$file = fopen ( $cache_filename, 'w' );
fwrite ( $file, $content );
fclose ( $file );
echo gzencode($content);
?>
Then create for example index.php (file which you want to cache)
<?php
include "cache_start.php";
echo "Hello Compress Cache World!";
include "cache_end.php";
?>
Manual caching (creating the HTML and saving it to a file) may not be the most efficient way, but if you want to go down that path I recommend the following (ripped from a simple test app I wrote to do this):
$cache_filename = basename($_SERVER['PHP_SELF']) . "?" . $_SERVER['QUERY_STRING'];
$cache_limit_in_mins = 60 * 32; // this forms 32hrs
// check if we have a cached file already
if ( file_exists($cache_filename) )
{
$secs_in_min = 60;
$diff_in_secs = (time() - ($secs_in_min * $cache_limit_in_mins)) - filemtime($cache_filename);
// check if the cached file is older than our limit
if ( $diff_in_secs < 0 )
{
// it isn't, so display it to the user and stop
print file_get_contents($cache_filename);
exit();
}
}
// create an array to hold your HTML output, this is where you generate your HTML
$output = array();
$output[] = '<table>';
$output[] = '<tr>';
// etc
// Save the output as manual cache
$file = fopen ( $cache_filename, 'w' );
fwrite ( $file, implode($output,'') );
fclose ( $file );
print implode($output,'');
I use APC for all my PHP caching (on an Apache server)
If you're not opposed to frameworks, try using the Zend Frameworks's Zend_Cache. It's pretty flexible, and (unlike some of the framework modules) easy to implement.
Can use Cache_lite from PEAR:
Details here
http://mahtonu.wordpress.com/2009/09/25/cache-php-output-for-high-traffic-websites-pear-cache_lite/
I was thinking from the point of load on the database, and charges for data bandwidth and speed of loading. I have some pages which are unlikely to change in years, (I know it is easy to use a CMS system based on a database ). Unlike in US, here the cost of bandwidth can be high. Anybody has any views on that, whether to create htmal pages or dynamic (php, asp.net)
Links to the pages would be stored on a database anyway.