I want to get with (curl) guzzle html content of a other page inside my laravel app.
The classic way would be:
$client = new Client();
$client = $client->request('GET', route('print.page'))->getBody();
The problem is, all this routes are auth protected and I get there only html from my login page.
I tried to send login trough guzzle again but I think this is not a good idea with double login.
Is there any better way to get html from this protected route?
In case you calling this inside a controller and you have a current authenticated user, you have to get the session name and the real session id:
public function FooController()
{
$name = Session::getName();
$sessionId = $_COOKIE[$name];
$cookieJar = CookieJar::fromArray([
$name => $sessionId,
], 'example.com');
$client = new Client();
$body = $client->request( // changed the variable from $client to $body here
'GET',
route('print.page'),
['cookies' => $cookieJar]
)->getBody();
}
Related
I have created a REST API using the Yii2 documentation. It seems to be working fine as I can use curl like this:
curl -i "https://example.com/api/v3/user" \
-H "Accept:application/json" \
-H "Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
I would now like to be able to consume this data from another Yii2 site. I am trying to use the Yii2 REST API client. I won't post the whole code as it's basically a copy of the Facebook client in yiisoft/yii2-authclient.
Does anyone know of a guide to help me amend this to comsume my API? In the first instance, I'm struggling with what to put for $authUrl and $tokenUrl.
I am not sure if you need to extend outh2 class as I believe you don't have the authentication logic completed in the first Yii2 webapp, like authenticating using first webapp url then redirect to the second webapp to extract the token from url.
It could be simpler just create a component that have those methods
class YourRestClient {
const BASE_URL = 'https://example.com/api/v3';
private $_token = null;
public function authenticate($username,$password){
$client = new Client();
$response = $client->createRequest()
->setMethod('POST')
->setUrl(BASE_URL.'/user/login')
->setData(['username' => $username, 'password' => $password])
->send();
if ($response->isOk) {
$this->_token = $response->data['token'];
}
}
public function logout(){
//your logut logic
}
public function refreshToken(){
//your refresh logic
}
public function userList(){
$client = new Client();
$response = $client->createRequest()
->setMethod('GET')
->setUrl(BASE_URL.'/user/users')
->addHeaders([
'content-type' => 'application/json',
'Authorization' => 'Bearer '.$_token,
])
->send();
if ($response->isOk) {
return $response->getData();
}
}
}
for more info httpclient
If I am not wrong what you will need for this, is to use yiisoft/yii2-httpclient
Ref: https://github.com/yiisoft/yii2-httpclient
Add it: php composer.phar require --prefer-dist yiisoft/yii2-httpclient
Then make the call «I would probably build a model to handle this»
use yii\httpclient\Client;
$client = new Client();
$response = $client->createRequest()
->setMethod('GET')
->setUrl('https://example.com/api/v3/user')
->addHeaders(['Authorization' => 'Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'])
->send();
if ($response->isOk) {
// use your data
}
I want to setup callback url for blockchain.info receive api.
Below is a php example but i don't know to to setup this in laravel
$my_callback_url = 'https://mystore.com?invoice_id=058921123&secret='.$secret;
my route is following
Route::get('btc_ipn/{invoice_id}/{secret}',['as'=>'btc_ipn','uses'=>'HomeController#btcIPN']);
I tried but ipn doesn't work.
If you want to build URL for the route you've shown, do something like this:
$url = route('btc_ipn', ['invoice_id' => $invoice->id, 'secret' => $secret]);
If you need to create a route for this URL:
https://mystore.com?invoice_id=058921123&secret=' . $secret;
Create a get or post route:
Route::get('/', 'HomeController#btcIPN');
Then in the controller:
public function btcIPN()
{
$invoiceId = request('invoice_id');
$secret = request('secret');
}
To meet routes you created as
Route::get('btc_ipn/{invoice_id}/{secret}',['as'=>'btc_ipn','uses'=>'HomeController#btcIPN']);
Your url should be like
$invoice_id = $request->invoice_id;
$secret = $request->secret;
$my_callback_url = "/btc_ipn/$invoice_id/$secret";
If you want to return with some data like error use
return redirect()->back()->withErrors(['msg', 'The Message']);
I'm trying to access to the Guzzle Response object from Goutte. Because that object has nice methods that i want to use. getEffectiveUrl for example.
As far as i can see there is no way doing it without hacking the code.
Or without accessing the response object, is there a way to get the last redirected url froum goutte?
A little late, but:
If you are only interested in getting the URL you were last redirected to, you could simply do
$client = new Goutte\Client();
$crawler = $client->request('GET', 'http://www.example.com');
$url = $client->getHistory()->current()->getUri();
EDIT:
But, extending Goutte to serve your needs is fairly easy. All you need is to override the createResponse() method and store the GuzzleResponse
namespace Your\Name\Space;
class Client extends \Goutte\Client
{
protected $guzzleResponse;
protected function createResponse(\Guzzle\Http\Message\Response $response)
{
$this->guzzleResponse = $response;
return parent::createResponse($response);
}
/**
* #return \Guzzle\Http\Message\Response
*/
public function getGuzzleResponse()
{
return $this->guzzleResponse;
}
}
Then you can access the response object as desired
$client = new Your\Name\Space\Client();
$crawler = $client->request('GET', 'http://localhost/redirect');
$response = $client->getGuzzleResponse();
echo $response->getEffectiveUrl();
I'm working on crawling web sites and there is no problem for parsing HTML with Goutte so far. But I need to retrieve JSON from a web site and because of the cookie management, I don't want to do this with file_get_contents() - that doesn't work.
I can do with pure cURL but in this case I just want to use Goutte and don't want to use any other library.
So is there any method that I can parse only text via Goutte or do I really have to do this with good old methods?
/* Sample Code */
$client = new Client();
$crawler = $client->request('foo');
$crawler = $crawler->filter('bar'); // of course not working
Thank you.
After very deep search inside Goutte libraries I found a way and I wanted to share. Because Goutte is really powerful library but there are so complicated documentation.
Parsing JSON via (Goutte > Guzzle)
Just get needed output page and store json into an array.
$client = new Client(); // Goutte Client
$request = $client->getClient()->createRequest('GET', 'http://***.json');
/* getClient() for taking Guzzle Client */
$response = $request->send(); // Send created request to server
$data = $response->json(); // Returns PHP Array
Parsing JSON with Cookies via (Goutte + Guzzle) - For authentication
Send request one of the page of the site (main page looks better) to get cookies and then use these cookies for authentication.
$client = new Client(); // Goutte Client
$crawler = $client->request("GET", "http://foo.bar");
/* Send request directly and get whole data. It includes cookies from server and
it automatically stored in Goutte Client object */
$request = $client->getClient()->createRequest('GET', 'http://foo.bar/baz.json');
/* getClient() for taking Guzzle Client */
$cookies = $client->getRequest()->getCookies();
foreach ($cookies as $key => $value) {
$request->addCookie($key, $value);
}
/* Get cookies from Goutte Client and add to cookies in Guzzle request */
$response = $request->send(); // Send created request to server
$data = $response->json(); // Returns PHP Array
I hope it helps. Because I almost spend 3 days to understand Gouttle and it's components.
I figured this out after several hours of search , simply do this :
$client = new Client(); // Goutte Client
$crawler = $client->request("GET", "http://foo.bar");
$jsonData = $crawler->text();
mithataydogmus' solution didn't work for me. I created a new class "BetterClient":
use Goutte\Client as GoutteClient;
class BetterClient extends GoutteClient
{
private $guzzleResponse;
public function getGuzzleResponse() {
return $this->guzzleResponse;
}
protected function createResponse($response)
{
$this->guzzleResponse = $response;
return parent::createResponse($response);
}
}
Usage:
$client = new BetterClient();
$request = $client->request('GET', $url);
$data = $client->getGuzzleResponse()->json();
I also could get JSON with:
$client->getResponse()->getContent()->getContents()
I am using the php guzzle Client to grab the website, and then process it with the symfony 2.1 crawler
I am trying to access a form....for example this test form here
http://de.selfhtml.org/javascript/objekte/anzeige/forms_method.htm
$url = 'http://de.selfhtml.org/javascript/objekte/anzeige/forms_method.htm';
$client = new Client($url);
$request = $client->get();
$request->getCurlOptions()->set(CURLOPT_SSL_VERIFYHOST, false);
$request->getCurlOptions()->set(CURLOPT_SSL_VERIFYPEER, false);
$response = $request->send();
$body = $response->getBody(true);
$crawler = new Crawler($body);
$filter = $crawler->selectButton('submit')->form();
var_dump($filter);die();
But i get the exception:
The current node list is empty.
So i am kind of lost, on how to access the form
Try using Goutte, It is a screen scraping and web crawling library build on top of the tools that you are already using (Guzzle, Symfony2 Crawler). See the GitHub repo for more info.
Your code would look like this using Goutte
<?php
use Goutte\Client;
$url = 'http://de.selfhtml.org/javascript/objekte/anzeige/forms_method.htm';
$client = new Client();
$crawler = $client->request('GET', $url);
$form = $crawler->selectButton('submit')->form();
$crawler = $client->submit($form, array(
'username' => 'myuser', // assuming you are submitting a login form
'password' => 'P#S5'
));
var_dump($crawler->count());
echo $crawler->html();
echo $crawler->text();
If you really need to setup the CURL options you can do it this way:
<?php
$url = 'http://de.selfhtml.org/javascript/objekte/anzeige/forms_method.htm';
$client = new Client();
$guzzle = $client->getClient();
$guzzle->setConfig(
array(
'curl.CURLOPT_SSL_VERIFYHOST' => false,
'curl.CURLOPT_SSL_VERIFYPEER' => false,
));
$client->setClient($guzzle);
// ...
UPDATE:
When using the DomCrawler I often times get that same error. Most of the time is because I'm not selecting the correct element in the page, or because it doesn't exist. Try instead of using:
$crawler->selectButton('submit')->form();
do the following:
$form = $crawler->filter('#signin_button')->form();
Where you are using the filter method to get the element by id if it has one '#signin_button' or you could also get it by class '.signin_button'.
The filter method requires The CssSelector Component.
Also debug your form by printing out the HTML (echo $crawler->html();) and ensuring that you are actually on the right page.