I have made a small script which uses the Twitch API. The API only allows a maximum of 100 results per query. I would like to have this query carry on until there are no more results.
My theory behind this, is to run a foreach or while loop and increment the offset by 1 each time.
My problem however, is that I cannot change the foreach parameters within itself.
Is there anyway of executing this efficiently without causing an infinite loop?
Here is my current code:
<?php
$newcurrentFollower = 0;
$offset=0;
$i = 100;
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
foreach ($json->follows as $follow)
{
echo $follow->user->name . ' (' . $newcurrentFollower . ')' . "<br>";
$newcurrentFollower++;
$offset++;
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
}
?>
Using a While loop:
while($i < $total)
{
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
echo $json->follows->user->name . ' (' . $newcurrentFollower . ')' . "<br>";
$newcurrentFollower++;
$offset++;
$i++;
}
Ends up echoing this (No names are successfully being grabbed):
Here is the API part for $json->follows:
https://github.com/justintv/Twitch-API/blob/master/v2_resources/channels.md#get-channelschannelfollows
You can use this:
$offset = 0;
$count = 1;
do {
$response = json_decode(file_get_contents(
'https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=100&offset=' . $offset
));
foreach($response->follows as $follow) {
echo $follow->user->name . ' (' . ($count++) . ')' . "</br>";
}
$offset+=25;
} while (!empty($response->follows));
You want to use a while loop here, not just a foreach. Basically:
while (the HTTP request returns results)
{
foreach ($json->follows as $follow)
{
do stuff
}
increment offset so the next request returns the next one not already processed
}
The trickiest part is going to be getting the while condition right so that it returns false when the request gets no more results, and will depend on what the API actually returns if there are no more results.
Also important, the cleanest way would be to have the HTTP request occur as part of the while condition, but if you need to do some complicated computation of the JSON return to check the condition, you can put an initial HTTP request before the loop, and then do another request at the end of each while loop iteration.
The problem is you're only capturing the key not the value. Place it into a datastructure to access the information.
Honestly I find a recursive function much more effective than a iterative/loop approach then just update a datatable or list before the next call. It's simple, uses cursors, lightweight and does the job. Reusable if you use generics on it too.
This code will be in c#, however I know with minor changes you'll be able to get it working in php with ease.
query = //follower object get request//
private void doProcessFollowers(string query)
{
HTTPParse followerData = new HTTPParse(); //custom json wrapper. using the basic is fine. Careful with your cons though
var newRoot = followerData.createFollowersRoot(query); // generates a class populated by json
if (newRoot[0]._cursor != null)
{
populateUserDataTable(newRoot); //update dataset
doProcessFollowers(newRoot[0]._links.next); //recurse
}
}
Anyway - This just allows you to roll through the cursors without needing to worry about indexes - unless you specifically want them for whatever reason. If you're working with generics you can just reuse this code without issue. Find a generic example below. All you need to do to make it reuseable is pass the correct class within the <> of the method call. Can work for any custom class that you use to parse json data with. Which is basically what the 'createfollowerroot()' is in the above code, except that's hard typed.
Also I know it's in c# and the topic is php, with a few minor changes to syntax you'll get it working easily.
Anyway Hope this helped somebody
Generic example:
public static List<T> rootSerialize<T>(JsonTextReader reader)
{
List<T> outputData = new List<T>();
while (reader.Read())
{
JsonSerializer serializer = new JsonSerializer();
var tempData = serializer.Deserialize<T>(reader);
outputData.Add(tempData);
}
return outputData;
}
Related
for a long time I dreamed to be able to have a function that executes an algorithm only the first time it is called and saves the algorithm result.
Every other time the function is called it would just return the saved result.
I have found a way to do it, and it goes like this:
$myFunction = function() use ( &$myFunction ){
// execute an algorithm and get a result
$result = 10 * 5;
// re-write $myFunction to return the result
$myFunction = function() use ( $result ) { return $result; };
// return the result (for the first function call)
return $result;
};
It seems to work fine.
for example, if I execute this code:
$myFunction = function() use ( &$myFunction ){
$result = (function() {
echo 'this is calculated only once';
return 10 * 5;
})();
$myFunction = function() use ( $result ) {
echo "\n" . 'this is a cached result';
return $result;
};
return $result;
};
echo "\n" . $myFunction();
echo "\n" . $myFunction();
echo "\n" . $myFunction();
I get the following:
this is calculated only once
50
this is a cached result
50
this is a cached result
50
My questions are:
Do you know of any problems using this method?
Do you know of another way to achieve the same result?
A fellow programmer gave me another, more elegant way to do the same thing, by using static variable inside the function.
It can even work with arguments and cache multiple results
function myFunction( $int ) {
static $cache;
if( !isset( $cache[ $int ] ) ) {
$cache[ $int ] = $int * $int;
}
return $cache[ $int ];
}
Your example, as someone mentioned in the comments, is basically assigning a new function to the variable that was holding your original function. It is indeed a convoluted way to achieve your goal, and has several drawbacks which make it a not very suitable solution.
One immediate issue is expiring the cache, so that the real value can be calculated again. It isn't very straightforward with this approach. Another issue is that your cache implementation is tightly coupled to your code.
Consider readability, isolation and separation of concerns. Your function is doing at least 2 things: calculating a value and managing a cache, and it doesn't run in isolation: it creates side effects by reassigning an external (global?) variable. It's also not very readable: it isn't clear what parts are calculating the values and which are for managing the cache, and that's evident because you had to add comments to explain your code.
I would suggest a more Object Oriented Approach using a Design Pattern such as Chain of Responsibility or Decorator. You would have to define one class to calculate the value, let's say ValueCalculator. Then you would define another class called ValueCalculatorFromCacheDecorator, which would handle the cache and rely on ValueCalculator if the cache doesn't exist/has expired/is unavailable for some reason. In your code you would call it as:
$service = new ValueCalculatorFromCacheDecorator(new ValueCalculator());
$value = $service->execute();
Note that each class a single responsibility now. Your cache concerns have been isolated in ValueCalculatorFromCacheDecorator, whereas the main algorithm for calculating a value is in ValueCalculator. These two classes can also be easily unit tested now.
Some users mentioned using Redis/Memcached/etc. You don't really have to do that. You can use an in-memory data structure to implement a rudimentary cache if importing a whole Cache system like Redis is an overkill for your project. With the approach I suggested, you can even go further and create a specific Decorator for each Cache implementation, which would allow you to encapsulate the underlying cache system in a single class.
You can read more about these ideas here: https://refactoring.guru/design-patterns/decorator and here https://refactoring.guru/design-patterns/chain-of-responsibility.
You should be using a cache layer such as redis or memcached
Here's an example how to implement this:
function myCoolAlgo($var1, $var2) {
# Set the cache key which is built from these two vars
$cacheKey = $var1 . $var2;
# Check if we have this result in our cache
# the myCache function is a helper that reads/sets data in the cache
$result = myCache($cacheKey, 'get');
// Note: if $result can be equal to 0, check it is not NULL
if ( ! empty( $result ) ) {
# We have it in the cache, return it
return $result;
}
# It's not in the cache, calculate it
// Algorithem code comes here
$result = 1234;
# Store in the cache
myCache($cacheKey, 'set', $result);
return $result;
}
I am sure this has been asked 100 times but cannot find the form of words to get at the answer either here or on Google.
I have a variable number of messages. They arrive as
$_GET['message1']; $_GET['message2']; $_GET['messageX']; etc
where X can be 1 to 100.
I need to test if they exist and then push them out to a DB. I tried
$i=1;
while (isset(parse_str("message$i")))
{
echo parse_str("output=message$i");
echo "<h1>This is test $output </h1>";
$i++;
}
which does not work. I thought the middle part worked but just re-tested and that is wrong too.
I am new to parse_str(). I thought I understood it and I understand the problem (it is a void function so cannot be used as a test) but cannot work out a solution for getting through the variables.
parse_str parses a string. What do you expect in a string "message$i"?
If you're sure that all your messages come from $_GET, use $_GET:
$i = 1;
while (isset($_GET['message' . $i])) {
echo $_GET['message' . $i];
$i++;
}
But obviously for storing such data, arrays are move convenient.
I am using PHP to get the contents of an API. The problem is, sometimes that API just sends back a 502 Bad Gateway error and the PHP code can’t parse the JSON and set the variables correctly. Is there some way I can keep trying until it works?
This is not an easy question because PHP is a synchronous language by default.
You could do this:
$a = false;
$i = 0;
while($a == false && $i < 10)
{
$a = file_get_contents($path);
$i++;
usleep(10);
}
$result = json_decode($a);
Adding usleep(10) allows your server not to get on his knees each time the API will be unavailable. And your function will give up after 10 attempts, which prevents it to freeze completely in case of long unavailability.
Since you didn't provide any code it's kind of hard to help you. But here is one way to do it.
$data = null;
while(!$data) {
$json = file_get_contents($url);
$data = json_decode($json); // Will return false if not valid JSON
}
// While loop won't stop until JSON was valid and $data contains an object
var_dump($data);
I suggest you throw some sort of increment variable in there to stop attempting after X scripts.
Based on your comment, here is what I would do:
You have a PHP script that makes the API call and, if successful, records the price and when that price was acquired
You put that script in a cronjob/scheduled task that runs every 10 minutes.
Your PHP view pulls the most recent price from the database and uses that for whatever display/calculations it needs. If pertinent, also show the date/time that price was captured
The other answers suggest doing a loop. A combo approach probably works best here: in your script, put in a few loops just in case the interface is down for a short blip. If it's not up after say a minute, use the old value until your next try.
A loop can solve this problem, but so can a recursive function like this one:
function file_get_contents_retry($url, $attemptsRemaining=3) {
$content = file_get_contents($url);
$attemptsRemaining--;
if( empty($content) && $attemptsRemaining > 0 ) {
return file_get_contents_retry($url, $attemptsRemaining);
}
return $content;
}
// Usage:
$retryAttempts = 6; // Default is 3.
echo file_get_contents_retry("http://google.com", $retryAttempts);
I am using the "elastica" php client for ElasticSearch.
I'm a bit new to OO-programming, especially in php.
However, I have managed to search my elasticsearch server using the elastica php client and store the response in an "Elastica_ResultSet" object. I have had no luck accessing the contents of that object whatsoever.
I would like to be able to list the total number of results, find an elasticsearch record id of a result and get the full content of an elasticsearch record for that result.
The Elastica class reference can be found here http://ruflin.github.com/Elastica/api/index.html , although I don't know what to do with it.
Here is the php code I have been using to get this far:
<?php
function __autoload_elastica ($class) {
$path = str_replace('_', '/', $class);
if (file_exists('extentions/' . $path . '.php')) {
require_once('extentions/' . $path . '.php');
//echo "$path EXISTS!!!";
}
}
spl_autoload_register('__autoload_elastica');
// New ES Client
$client = new Elastica_Client();
// Set Index
$index = $client->getIndex('test1');
// Set Document Type
$type = $index->getType('user');
// Perform Search
$resultSet = $index->search('halo');
?>
So basicaly you can use var_export to output your resultset
But in general the elastica search returns a Elastica_ResultSet object which has several attributes you can use like count, totalHits facets and so on.
and also holds an array of Elastica_Result objects these can be accessed either by calling the Elastica_ResultSet getResults() method or by using the current() and next() methods or by simply using the php foreach function
The Elastica_Result the data of the results and also has several methods you can use.
getId(), getVersion(), getData() and so on.
// Set Document Type
$type = $index->getType('user');
// Perform Search
$resultSet = $index->search('halo');
// Get IDs
$resultIDs = array();
foreach($resultSet as $result){
$resultIDs[] = $result->getId();
}
I would like to let you know something that was a bit hard for me to get.
The query and the sorting of results
// Set the query terms for your search
$queryTerm = new Elastica_Query_Terms();
$queryTerm->setTerms('user', array("test", "test1"));
// Create the sorting array
$sort = array("user" => array("order" => "desc"));
// Create the query
$query = Elastica_Query::create($queryTerm);
// Set the sorting to the query
$query->setSort($sort);
// Perform the search
$resultSet = $index->search($query);
Hope this helps
After a couple of months OO practise, it seemed performing a simple var_dump($resultSet) would have provided me with the structure and contents of the returned object... can't believe that nobody made any suggestions for such a basic question ;)
I'm building a script that takes the contents of several (~13) news feeds and parses the XML data and inserts the records into a database. Since I don't have any control over the structure of the feeds, I need to tailor an object operator for each one to drill down into the structure in order to get the information I need.
The script works just fine if the target node is one step below the root, but if my string contains a second step, it fails ( 'foo' works, but 'foo->bar' fails). I've tried escaping characters and eval(), but I feel like I'm missing something glaringly obvious. Any help would be greatly appreciated.
// Roadmaps for xml navigation
$roadmap[1] = "deal"; // works
$roadmap[2] = "channel->item"; // fails
$roadmap[3] = "deals->deal";
$roadmap[4] = "resource";
$roadmap[5] = "object";
$roadmap[6] = "product";
$roadmap[8] = "channel->deal";
$roadmap[13] = "channel->item";
$roadmap[20] = "product";
$xmlSource = $xmlURL[$fID];
$xml=simplexml_load_file($xmlSource) or die(mysql_error());
if (!(empty($xml))) {
foreach($xml->$roadmap[$fID] as $div) {
include('./_'.$incName.'/feedVars.php');
include('./_includes/masterCategory.php.inc');
$test = sqlVendors($vendorName);
} // end foreach
echo $vUpdated." records updated.<br>";
echo $vInserted." records Inserted.<br><br>";
} else {
echo $xmlSource." returned an empty set!";
} // END IF empty $xml result
While Fosco's solution will work, it is indeed very dirty.
How about using xpath instead of object properties?
$xml->xpath('deals/deal');
PHP isn't going to magically turn your string which includes -> into a second level search.
Quick and dirty hack...
eval("\$node = \"\$xml->" . $roadmap[$fID] . "\";");
foreach($node as $div) {