My task is to get product list with prices from the link:
http://www.sanalmarket.com.tr/kweb/sclist/30011-tum-meyveler
Namely, I want to get this URL (or a better alternative):
http://www.sanalmarket.com.tr/kweb/getProductList.do?shopCategoryId=30011
(By the way, I cannot reach this URL on my own hook using PHP).
To get the cookie from original URL (first one) was simple:
<?php
$ch = curl_init('http://www.sanalmarket.com.tr/kweb/sclist/30011-tum-meyveler/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$result = curl_exec($ch);
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $result, $m);
parse_str($m[1], $cookies);
var_dump($cookies);
$cookie1 = reset($cookies);
echo $cookie1;
?>
Nevertheless, I cannot find a way to use this info to reach any JSON file or the URL which contains product info. I want to scrape the AJAX URL the website triggers to show the product list. Is there a way to do that?
Thank you in advance.
No need for cookies, use your browser's devTools to monitor Network activity.
CTRL-J to display the console, then click on the Network tab and reload your page, you'll see this :
And there you go, use a simple file_get_contents(url) function and you're good.
Related
Hi guys so i'm using a third party iframe and I would like to load that iframe "internally" or at backend? , once it is loaded a link would be fetch at the iframe and it's at the network, fetching a link from CDN and I would like to get that link and display it to the frontend. But i'm not quite sure if i'm using curl the right away or is there a better way.
(just in case)
Network = when you inspect your google tab you will see a tab called network
$filtering = "https://streamwebsite.com/movie/batman";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $filtering);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_exec($curl);
I have previously made a code that allows me to make an inquiry to an external web page where I sent by GET the id and it shows me the name and surname of a person. That code works very well, is this:
function nome($id)
{
$url = 'https://www.dateas.com/es/consulta_cuit_cuil?name=&cuit='.$id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER,array("Accept-Lenguage: es-es,es"));
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$html = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);
preg_match_all('/<tr class="odd"><td><a href="(.*?)">/',$html ,$matches);
if (count($matches[1]) == 0)
{
$result = "Error in the server";
}
else if(count($matches[1]) == 1)
{
$result = $matches[1][0];
$result = str_replace('/es/persona/','', $result);
$result = substr($result, 0,-12);
$result = str_replace('-', ' ', $result);
$result = ucwords($result);
}
return $result;
}
$Name = nome($_GET['id']);
echo "<br><br>The name of the person is : ". $Name;
This code works well but now I need to extract this information from another website, but to do so I must log in first on that web page and then go to the query section and enter the id, click on the search button and then the page returns the information of the person. The website does not make requests by GET but by POST, to search and display the information of a person.
My problem is that I do not know how to extract the information in this new page because I must first login and then look for the person entering their id but the requests are handled by POST.
How could I get the information of the person making the query through CURL or any other PHP method?
this is the page where is the login form to enter the web where the query is made: http://buscardatos.com/Socios/ingresso.php
After logging in, we should go to: http://www.buscardatos.com/Personas/DNI
there is a text field where the id is entered and by clicking on the search button the information of the person is displayed.
the website makes the request by POST. I have tried to use the same code that I have shown here but it throws me an error of "page not found".
Could you please help me with some code or information about how to solve this issue?
Maybe you can try to capture the interaction between your local port and the server with a capture tool (such as Fiddler. In fact, the browser itself can see this information).
First, you need to use a browser to complete the login and query process, and then you can clearly see the http/https data information that the browser interacts with the server.
Next, you need to use PHP code to simulate the browser to submit the login information like the server, and then get the data returned by the server (this data may be a session value, generally stored locally in the browser's cookie.)
Next you can use the CURL above to get the data. Remember to bring the data needed for server verification. If it is a cookie, maybe you can add it like this:
curl_setopt($ch,CURLOPT_COOKIE,'cookie= XXXX ; cookie_type= YYYY;');
As long as you provide the correct data and URL, you can get the data you want.
From everything I've read, it seems that this is an impossible. But here is my scenario:
I need to scrape a table's content containing for sale housing information. The page is not password protected or anything, but you first have to click an "I Agree" link on the previous page so that a cookie gets set saying you agree that the content may not be 100% accurate. You are only then shown the data. Is there any way at all to accomplish this using php/jquery/javascript? I know you cannot create an iframe because of the fact that it is cross-domain. I also do not have access to this other website.
Thanks for any answers, as I'm not really expecting anything positive. :) And many thanks if you can tell me how to do this. :D
Use server side script (PHP using cURL) to crawl the website and return the information you need. Make sure you set the appropriate HTTP header with your request that represents the "I agree" cookie.
Sample:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/');
curl_setopt($ch, CURLOPT_COOKIE, 'I_Agree=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$responseBody = curl_exec($ch);
curl_close($ch);
// Read the information you need from $responseBody and return it as response body
?>
Now you can access the information from your website by calling your server side script above. For details about how to use cURL take a look at the documentation.
CURL can store or recall cookies from a file depending on the options you set. Here is the "cookiejar" example:
http://curl.haxx.se/libcurl/php/examples/cookiejar.html
Check out the CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options
So I have a contact form on one website, but I want to use the mailing function on another.
I can make forms and get the text and send them in the PHP mail function just fine, I'm just not sure how to do this for another website without opening another tab.
Also, I'm not even sure what to call this.
For example:
I have text fields and a submit button on one website, and I want to send that data to another URL like so:
http://myurl.com?act=phptools&email=example#test.com&subject=Hello&message=How are you?
How would I do this without opening another tab in the browser?
Just set the action attribute of your form to the URL of the processing script on the other domain.
But make sure this is secure, I mean, if you control the second domain, then its okay. Else, you may have to be concerned about sharing data with another domain. But then, I don't know what your requirement is :)
Set the action attribute of the form to the other website url. The user on submit will be taken there, and on success/failure you can redirect the user to the old website url or wherever you want.
Even better use XHR/Ajax, so that the request is made behind the scenes and all you need to do is show the error msg/success msg based on the json response you sent from the other website url. However with this method there are limitations as browsers do not allow cross origin requests. So check if you can enable CORs!
Another thing you can do is send request to current website url only and that would interact with the other website url server-size, hope it makes sense.
As far as i understand your question You can use CURL php functions for this
<?php
$urltopost = "http://somewebsite.com/script.php";
$datatopost = array (
"firstname" => "ex_name",
"lastname" => "ex_lastname",
"email" => "my#email.com",
);`enter code here`
<span id="more-40"></span>`enter code here
$ch = curl_init ($urltopost);
curl_setopt ($ch, CURLOPT_POST, true);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $datatopost);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$returndata = curl_exec ($ch);
?>
I would like to be able to send variables to another website without actually going to the website using php.
I am building an ecommerce website where the shipping warehouse is being outsourced. After the person checks out with their products, I would like to send some variables over to the shipper's website using $_GET['vars']. This is a completely different URL. The problem is, I don't want the person actually going to the shipper's webpage. I just want to ping that info over there.
Is is possible to send Variables via URL to another site without leaving yours?
Yes, you can. the simplest way:
$contents = file_get_contents("http://example.com/some/page.php?var=abcd");
For more advanced features see Curl.
You should be storing all the relevant order info within your database then using cron to trigger a script that will process the unprocessed, this way systematic checks can be made on orders before any request to your outsource site. Dont rely on your users browser to hit a curtain point in the order process to trigger the API call or trust them not to triple click or inject values before submitting.
And I advise to use curl todo your actual request as its faster. Something as simple as:
<?php
function curl_do_api($url){
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
?>
actually there is a more simpler way of solving this...using ajax
include the jquery header first
refer here
http://api.jquery.com/jQuery.ajax/
Both are right and you should make your choice based on security you're looking for.
cURL will be more secure and you should use it if you do not want to pass some argument in query string. At the same time when you pass data using file_get_cotents("URL?data=value"); you will have limit of about 2k for data being passed.
On the other side cURL will be secure if you use it with https it's much more secure. With cURL you will also be able to post files and emulate form post.