So in keeping with my last question, I'm working on scraping the friends feed from Twitter. I followed a tutorial to get this script written, pretty much step by step, so I'm not really sure what is wrong with it, and I'm not seeing any error messages. I've never really used cURL before save from the shell, and I'm extremely new to PHP so please bear with me.
<html>
<head>
<title>Twitcap</title>
</head>
<body>
<?php
function twitcap()
{
// Set your username and password
$user = 'osoleve';
$pass = '****';
// Set site in handler for cURL to download
$ch = curl_init("https://twitter.com/statuses/friends_timeline.xml");
// Set cURL's option
curl_setopt($ch,CURLOPT_HEADER,1); // We want to see the header
curl_setopt($ch,CURLOPT_TIMEOUT,30); // Set timeout to 30s
curl_setopt($ch,CURLOPT_USERPWD,$user.':'.$pass); // Set uname/pass
curl_setopt($ch,CURLOPT_RETURNTRANSER,1); // Do not send to screen
// For debugging purposes, comment when finished
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,0);
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST,0);
// Execute the cURL command
$result = curl_exec($ch);
// Remove the header
// We only want everything after <?
$data = strstr($result, '<?');
// Return the data
$xml = new SimpleXMLElement($data);
return $xml;
}
$xml = twitcap();
echo $xml->status[0]->text;
?>
</body>
</html>
Wouldn't you actually need everything after "?>" ?
$data = strstr($result,'?>');
Also, are you using a free web host? I once had an issue where my hosting provider blocked access to Twitter due to people spamming it.
note that if you use strstr the returend string will actually include the needle-string. so you have to strip of the first 2 chars from the string
i would rather recommend a combination of the function substr and strpos!
anways, i think simplexml should be able to handle this header meaning i think this step is not necessary!
furthermore if i open the url i don't see the like header! and if strstr doesnt find the string it returns false, so you dont have any data in your current script
instead of $data = strstr($result, '<?'); try this:
if(strpos('?>',$data) !== false) {
$data = strstr($result, '?>');
} else {
$data = $result;
}
Related
Hey guys I'm working with PHP, cURL and JSON data.
Architecture:
FRONT <-> MIDDLE <-> BACK
Problem location:
MIDDLE <-> BACK
Description:
In my MIDDLE file (PHP server) I call a function which make a cURL to my BACK file (PHP server).
The issue I'm having is that it will execute one of the calls (HTTP cURL requests) but not the other, overthought the one not executing is the call first.
Can you guys guide me on the issue, how can I solve this?
What I've try:
I research (articles, videos) info on curl thinking that maybe I had to handle it mysql-like, in which I had to create multiple cURL connection if I needed to request different items from the same server (BACK).
I've implemented dummy print outs to see whats executed and came about the conclusion I describe in the description part.
Note:
Following you will find the code for MIDDLE, BACK and two images:
Image 1 is the code with dummy print out showing how one cURL executes and the other does not. Please noticed the text "what back gets".
Image 2 is how cURL is not executed.
MIDDLE FILE:
Note:
Look how under case "EXAMSOLVED" I call two functions. That is related to Image 1.
If I comment the second function that is related to Image 2.
<?php
/*------------------------------------------------------------------*/
include "exam_grading_system.php";
/*------------------------------------------------------------------*/
function http_post_back_server($url, $data)
{
$obj = curl_init();
curl_setopt($obj, CURLOPT_URL, $url);
curl_setopt($obj, CURLOPT_POST, strlen($data));
curl_setopt($obj, CURLOPT_POSTFIELDS, $data);
curl_setopt($obj, CURLOPT_RETURNTRANSFER, true);
$ans = curl_exec($obj);
curl_close($obj);
return $ans;
}
//--------------------------------------------------------------------
function get_question_info($ulr, $id_question)
{
//Set UP Request Packet
$askDB->case = "GradingInfo";
$askDB->id_question = $id_question;
//Convert Packet to JSON format
$askDB = json_encode($askDB);
//echo $askDB;
//Ask Back Server
$BK_Srv_Ans = http_post_back_server($url,$askDB);
return $BK_Srv_Ans;
}
//--------------------------------------------------------------------
/*URL TO BACK SERVER*/
$url_myserver = "https: MY URL ";
/*GLOBAL VARS*/
$back_ans ="";
/*RECEIVE DATA FROM POST REQUEST*/
$indata = file_get_contents("php://input");
$data = json_decode($indata,true);
/*MAKE REQUEST TO SERVERS*/
switch($data["case"]){
case "_EXAMSOLVED_":
$questions_Info = get_question_info($url_myserver, $data["id_question"]);
//$indata = Grading($data, $questions_Info);
//$back_ans = http_post_back_server($url_myserver,$indata);
break;
default:
$back_ans = http_post_back_server($url_myserver,$indata);
break;
}
/*ANSWER BACK TO FRON END*/
echo $back_ans;
?>
BACK FILE:
Note: Look the dummy echo statement I have at the begging of the file
<?php
/*------------------------------------------------------------------*/
include "Prof_backend_tools.php";
include "Student_backend_tools.php";
/*------------------------------------------------------------------*/
echo "what back gets: ";
/*RECEIVING DATA FROM POST REQUEST */
$indata = file_get_contents("php://input");
/*DATA TO JSON OBJ*/
$indata = json_decode($indata, true);
/*CHECKING DATABASE CONNECTIVITY */
if(mysqli_connect_error())
{ echo "Connection Error: ".mysqli_connect_error; }
switch($indata["case"])
{
case "_EXAMSOLVED_":
echo store_exam_slutions($indata,DB_s());
break;
case "GradingInfo":
echo "--IN BACK--";
//echo Needed_data_for_grading($indata);
break;
default:
echo "NADA";
break;
}
?>
Image 1:
Image 2:
Any guide would be great guys.
Thank you.
OK,I've found what was happening!!!
Debugging:
Check MIDDLE file.
Check the parameters in function definition get_question_info
Now check arguments given to function call http_post_back_server
There lies the problem.
My script is working most of the times, but in every 8th try or so I get an error. I'll try and explain this. This is the error I get (or similar):
{"gameName":"F1 2011","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/81163/movie_max.webm?t=1447354814","gameId":"44360","finalPrice":1499,"genres":"Racing"}
{"gameName":"Starscape","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/900679/movie_max.webm?t=1447351523","gameId":"20700","finalPrice":999,"genres":"Action"}
Warning: file_get_contents(http://store.steampowered.com/api/appdetails?appids=400160): failed to open stream: HTTP request failed! in C:\xampp\htdocs\GoStrap\game.php on line 19
{"gameName":"DRAGON: A Game About a Dragon","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/2038811/movie_max.webm?t=1447373449","gameId":"351150","finalPrice":599,"genres":"Adventure"}
{"gameName":"Monster Mash","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/900919/movie_max.webm?t=1447352342","gameId":"36210","finalPrice":449,"genres":"Casual"}
I'm making an application that fetches information on a random Steam game from the Steam store. It's quite simple.
The script takes a (somewhat) random ID from a text file (working for sure)
The ID is added to the ending of an URL for the API, and uses file_get_contents to fetch the file. It then decodes json. (might be the problem somehow)
Search for my specified data. Final price & movie webm is not always there, hence the if(!isset())
Decide final price and ship back to ajax on index.php
The error code above suggests that I get the data I need in 4 cases, and an error once. I only wanna receive ONE json string and return it, and only in-case $game['gameTrailer'] and $game['final_price'] is set.
This is the php (it's not great, be kind):
<?php
//Run the script on ajax call
if(isset($_POST)) {
fetchGame();
}
function fetchGame() {
$gameFound = false;
while(!$gameFound) {
////////// ID-picker //////////
$f_contents = file("steam.txt");
$url = $f_contents[mt_rand(0, count($f_contents) - 1)];
$answer = explode('/',$url);
$gameID = $answer[4];
$trimmed = trim($gameID);
////////// Fetch game //////////
$json = file_get_contents('http://store.steampowered.com/api/appdetails?appids='.$trimmed);
$game_json = json_decode($json, true);
if(!isset($game_json[$trimmed]['data']['movies'][0]['webm']['max']) || !isset($game_json[$trimmed]['data']['price_overview']['final'])) {
continue;
}
$gameFound = true;
////////// Store variables //////////
$game['gameName'] = $game_json[$trimmed]['data']['name'];
$game['gameTrailer'] = $game_json[$trimmed]['data']['movies'][0]['webm']['max'];
$game['gameId'] = $trimmed;
$game['free'] = $game_json[$trimmed]['data']['is_free'];
$game['price'] = $game_json[$trimmed]['data']['price_overview']['final'];
$game['genres'] = $game_json[$trimmed]['data']['genres'][0]['description'];
if ($game['free'] == TRUE) {
$game['final_price'] = "Free";
} elseif($game['free'] == FALSE || $game['final_price'] != NULL) {
$game['final_price'] = $game['price'];
} else {
$game['final_price'] = "-";
}
}
////////// Return to AJAX (index.php) //////////
echo
json_encode(array(
'gameName' => $game['gameName'],
'gameTrailer' => $game['gameTrailer'],
'gameId' => $game['gameId'],
'finalPrice' => $game['final_price'],
'genres' => $game['genres'],
))
;
}
?>
Any help will be appreciated. Like, are there obvious reason as to why this is happening? Is there a significantly better way? Why is it re-iterating itself at least 4 times when it seems to have fetched that data I need? Sorry if this post is long, just trying to be detailed with a lacking php/json-vocabulary.
Kind regards, John
EDIT:
Sometimes it returns no error, just multiple objects:
{"gameName":"Prime World: Defenders","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/2028642/movie_max.webm?t=1447357836","gameId":"235360","finalPrice":899,"genres":"Casual"}
{"gameName":"Grand Ages: Rome","gameTrailer":"http://cdn.akamai.steamstatic.com/steam/apps/5190/movie_max.webm?t=1447351683","gameId":"23450","finalPrice":999,"genres":"Simulation"}
I've been using this over 2 months and worked fine until some days ago, when an error message appeared.
I use the steam api to get some info of the players.
$url = "http://steamcommunity.com/id/CGaKeepoN/?xml=1";
The page is not blank, it has an xml document. So my first thinking was that my host had turned allow_url_fopen off, but they don't (I asked them).
I also tried using error_reporting(E_ALL); ini_set('display_errors', 1);
And that's what I get:
Warning: simplexml_load_file() [function.simplexml-load-file]: I/O warning : failed to load external entity "" on line 6
Notice: Trying to get property of non-object on line 7
Now I'm using this: $xml = simplexml_load_file(file_get_contents($url));
And I would love to continue using it because installing cURL it's not an option right now. Do you know of a better (or a working) way to get this done? Or how to fix this error?
My full code:
error_reporting(E_ALL);
ini_set('display_errors', 1);
//$url = "http://steamcommunity.com/id/CGaKeepoN/?xml=1";
$url = "xml.txt";
ini_set('allow_url_fopen ','ON');
$xml = file_get_contents($url) or die ("file_get_contents failed");
$xml = simplexml_load_string($xml) or die ("simplexml_load_string failed");
$profilepic = $xml->avatarIcon;
$pic = $xml->avatarFull;
$steamID = $xml->steamID;
$lastonline = $xml->stateMessage;
echo $xml;
echo $profilepic;
echo $pic;
echo $steamID;
echo $lastonline;
EDIT:
If I use the internal url it loads the data, but when I try to use any url that uses http protocol just launches the file_get_contents failed error, even if the url is my website's one. I'm willing to use cURL if there's no other solution. I also thought about making a php script that loads the data and saves it in a file in the server (and then run a cronjob every 10 min), but it would use the file_get_contents anyway...
file_get_content returns a string so use simplexml_load_string instead.
This code works for me, tested.
$url = "http://steamcommunity.com/id/CGaKeepoN/?xml=1";
$xml = simplexml_load_string(file_get_contents($url));
$profilepic = $xml->avatarIcon;
$pic = $xml->avatarFull;
$steamID = $xml->steamID;
$lastonline = $xml->stateMessage;
var_dump($url);
var_dump($xml); //-> string(45) "http://steamcommunity.com/id/CGaKeepoN/?xml=1" bool(false)
echo $xml;
echo $profilepic;
echo $pic;
echo $steamID;
echo $lastonline;
I can retrieve certain information with a rest command, the data it shows (in the browser) is already an XML. How do I save it to an XML file on the server after retrieving the information.
I have already tried it with the $dom-save command but I seem to do something wrong. Any help would be appreciated. See below for code (I want to save the $response to XML.)
<?php
require_once 'includes/rest_connector.php';
require_once 'includes/session.php';
// check to see if we start a new session or maintain the current one
checksession();
$rest = new RESTConnector();
$url = "/api/tax_codes/0/";
$rest->createRequest($url,"GET", null, $_SESSION['cookies'][0]);
$rest->sendRequest();
$response = $rest->getResponse();
$error = $rest->getException();
// save our session cookies
if ($_SESSION['cookies']==null)
$_SESSION['cookies'] = $rest->getCookies();
// display any error message
if ($error!=null)
echo $error;
// display the response
if ($response!=null)
echo $response;
else
echo "There was no response.";
?>
The RESTConneconnector is a specific class Lightspeed. I solved it by using this:
ibxml_use_internal_errors(true);//load if improperly formatted
file_put_contents("exportProduct.xml", $responseProduct);
So it was very easy in the end :)
I dont know any about RESTConnector class. But I suppose, you can try something like it:
$dom = new DOMDocument('1.0','utf-8');
$dom->preserveWhiteSpace = false;
$dom->formatOutput = true;
$dom->loadXML($response->asXML());
$dom->save($this->fileexport);
I'm using PHP cURL to fetch information from another website and insert it into my page. I was wondering if it was possible to have the fetched information cached on my server? For example, when a visitor requests a page, the information is fetched and cached on my server for 24 hours. The page is then entirely served locally for 24 hours. When the 24 hours expire, the information is again fetched and cached when another visitor requests it, in the same way.
The code I am currently using to fetch the information is as follows:
$url = $fullURL;
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
Is this possible? Thanks.
You need to write or download a php caching library (like extensible php caching library or such) and adjust your current code to first take a look at cache.
Let's say your cache library has 2 functions called:
save_cache($result, $cache_key, $timestamp)
and
get_cache($cache_key, $timestamp)
With save_cache() you will save the $result into the cache and with get_cache() you will retrieve the data.
$cache_key would be md5($fullURL), a unique identifier for the caching library to know what you want to retrieve.
$timestamp is the amount of minutes/hours you want the cache to be valid, depending on what your caching library accepts.
Now on your code you can have a logic like:
$cache_key = md5($fullURL);
$timestamp = 24 // assuming your caching library accept hours as timestamp
$result = get_cache($cache_key, $timestamp);
if(!$result){
echo "This url is NOT cached, let's get it and cache it";
// do the curl and get $result
// save the cache:
save_cache($result, $cache_key, $timestamp);
}
else {
echo "This url is cached";
}
echo $result;
You can cache it using memcache ( a session ) you can cache it using files on your server and you can cache it using a database, like mySQL.
file_put_contents("cache/cachedata.txt",$data);
You will need to set the permissions of the folder you want to write the files to, otherwise you might get some errors.
Then if you want to read from the cache:
if( file_exists("cache/cachedata.txt") )
{ $data = file_get_contents("cache/cachedate.txt"); }
else
{ // curl here, we have no cache
}
Honza's suggestion to use Nette cache worked great for me, and here's the code I wrote to use it. My function returns the HTTP result if it worked, false if not. You'll have to change some path strings.
use Nette\Caching\Cache;
use Nette\Caching\Storages\FileStorage;
Require("/Nette/loader.php");
function cached_httpGet($url) {
$storage = new FileStorage("/nette-cache");
$cache = new Cache($storage);
$result = $cache->load($url);
if ($result) {
echo "Cached: $url";
}
else {
echo "Fetching: $url";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo "ERROR " . curl_error($ch) . " loading: $url";
return false;
} else
$cache->save($url, $result, array(Cache::EXPIRE => '1 day'));
curl_close($ch);
}
return $result;
}
Use Nette Cache. All you need solution, simple to use and of course - thread-safe.
If you've got nothing against file system access, you could just store it in a file. Then maybe use a script on the server that checks the file's timestamp against the current time and deletes it if it's too old.
If you don't have access to all aspects of the server you could just use the above idea and store a timestamp with the info. Every time the page is requested check against the timestamp.
And if you're having problems with the fs bottlenecking, you could use a MySQL database stored entirely in RAM.
I made a pretty cool simple function to store data gotten from your curl for 1 hour or 1 day off Antwan van Houdt's comment (shout out to him) .. firstly create a folder with name "zcache" in public_html and make sure the permission is at "755"
1 hour:
if( file_exists('./zcache/zcache-'.date("Y-m-d-H").'.html') )
{ $result = file_get_contents('./zcache/zcache-'.date("Y-m-d-H").'.html'); }
else
{
// put your curl here
file_put_contents('./zcache/zcache-'.date("Y-m-d-H").'.html',$result);
}
1 day:
if( file_exists('./zcache/zcache-'.date("Y-m-d").'.html') )
{ $result = file_get_contents('./zcache/zcache-'.date("Y-m-d").'.html'); }
else
{
// put your curl here
file_put_contents('./zcache/zcache-'.date("Y-m-d").'.html',$result);
}
you are welcome
The best way to avoid caching is applying the time or any other random element to the url, like this:
$url .= '?ts=' . time();
so for example instead of having
http://example.com/content.php
you would have
http://example.com/content.php?ts=1212434353