I try to get a booking.com page from a hotel to fetch the prices afterwards with regex. The problem is the following:
I call file_get_contents with parameter like checkin and checkout (file_get_contents("/hotel/at/myhotel.html?checkin=2017-10-12&checkout=2017-10-13")) dates so that the prices are shown to the visitor. If I watch the source code in the browser I see the entry:
b_this_url : '/hotel/at/myhotel.html?label=gen173nr-1FCAsoDkIcbmV1ZS1wb3N0LWhvbHpnYXUtaW0tbGVjaHRhbEgHYgVub3JlZmgOiAEBmAEHuAEHyAEM2AEB6AEB-AEDkgIBeagCAw;sid=58ccf750fc4acb908e20f0f28544c903;checkin=2017-10-12;checkout=2017-10-13;dist=0;sb_price_type=total;type=total&',
If I echo the string from file_get_contents the string looks like:
b_this_url : '/hotel/at/myhotel.html',
So all parameters that I passed to the url with file_get_contents are gone and therefore I couldn't find any prices with my regex on the page ...
Does anyone have a solution for this problem?
The webpage is not completely generated server-side, but it relies heavily on JavaScript after the HTML part loads. If you are looking for rendering the page as it looks in browser, I think you should use php curl instead of file_get_contents() for this kind of web scraping thing. I generated an automatic code for you from Postman (a google chrome extension / standalone desktop app) for your given url. The response contains the full url with params. See the image and I posted the code for you also.
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.booking.com/hotel/at/hilton-innsbruck.de.html?checkin=2017-10-10%3Bcheckout%3D2017-10-11",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"postman-token: 581a75a7-6600-6ed6-75fd-5fb09c25d927"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Related
I am creating a local website that makes use of the OpenAI API using XAMPP and HTML/CSS and PHP. I have implemented a page to enter a question and display the answer from the API, but the result has no relation to the question asked. I am looking for help to solve this problem. Here is my php code (it works with another file calling it).
Thanks in advance for your help.
PS: I voluntarily removed the api key when I posted the request on stack overflow
Translated with www.DeepL.com/Translator (free version)
<?php
$apiKey = 'sk-';
$modelId = 'text-davinci-002';
$prompt = "";
$temperature = 0.5;
$maxTokens = 256;
$topP = 1;
$frequencyPenalty = 0;
$presencePenalty = 0;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.openai.com/v1/completions",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS =>"{\"model\": \"".$modelId."\",
\"prompt\": \"".$prompt."\",
\"temperature\": ".$temperature.",
\"max_tokens\": ".$maxTokens.",
\"top_p\": ".$topP.",
\"frequency_penalty\": ".$frequencyPenalty.",
\"presence_penalty\": ".$presencePenalty."}",
CURLOPT_HTTPHEADER => array(
"Authorization: Bearer ".$apiKey,
"Content-Type: application/json"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
$result = json_decode($response);
echo $result->choices[0]->text;
}
?>
I tried several solutions such as changing the API model and rewriting the code, but I could not solve the problem. I am looking for help to solve this problem.
I am working on integrating my application with the API of another application. Previously I have done such integrations using JSON and had no problem, unfortunately, the application I want to connect to uses XML which I have not interacted with before.
The task is to load an .xml file into a PHP script, then perform gzip on that XML, and then encode the byte array to base64.
I have the XML file ready, I can use base64_encode() too, but I sit on this gzip for several hours and nothing works. (I load the XML file into the script using simplexml_load_file)
I tried using gzencode, gzcompress but it doesn't work.
Authentication involves combining HTTP parameters and encoding them with urlencode(). The req_sig (signature) parameter contains the encoded parameters (mentioned above) + API code. If everything matches then a response is returned, if not then an error is displayed: ERROR4000Signature does not match, even though the signature matches, because it is generated the same way in other queries using GET.
My code:
$xml=simplexml_load_file("request.xml") or die("Failed to load");
$gzip = gzencode($xml);
$base64 = base64_encode($gzip);
$username = 'test-app-user';
$req_id = rand(10000000000, 99999999999);
$api_token = "za5rth23ku776fdvbn2";
$string_bazowy = "command=$base64"."req_id=$req_id"."username=$username";
$encoded = urlencode($string_bazowy);
$string_zmieniony = "$encoded"."$api_token";
$md5 = md5($string_zmieniony);
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://saldeo-test.brainshare.pl/api/xml/1.0/company/synchronize?command=$base64&req_id=$req_id&username=$username&req_sig=$md5",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'POST',
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
Good day! My client gave us an API endpoint where the response is XML when you pass in some commands, I would like to know if there's a way where we can run the endpoint in the PHP and get the response back, I have tried using PHP curl but the response shows 400 or bad request. I can access the endpoint with the commands/parameters in the URL browser so it works. But when I try it using curl, there is no response in it. Is this possible, or am I doing something wrong with my code?
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.winquote.net/cgi-bin/compete.pl?dc=-cv1.5 -ccca -qt0 -pccaXXXXXXXXX -rt0 -dob11061992 -gen1 -rR -fa500000 -pg0 -pi4 -lc1 -pm0 -rc0 -rop0 -langen -fmt -ceilp -faEXACT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST"
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Have a look at White spaces in postFields in PHP Curl which may answer your question.
The only issue here is the query param are not proper and '400' response is saying you so. In browser you you look the spaces are converted to '%20'.The browser has done that work for you. You have to do the similar thing i.e. encode your query parameter because of the spaces before doing the post request.
From https://curl.haxx.se/docs/manpage.html#--data-urlencode
--data-urlencode
(HTTP) This posts data, similar to the other -d, --data options with
the exception that this performs URL-encoding.
To be CGI-compliant, the part should begin with a name followed
by a separator and a content specification. The part can be
passed to curl using one of the following syntaxes:
I used the function file_get_contents to get content from a website. but just see messege "sorry! something went wrong."
My code here:
<?php
$kkk = 'https://batdongsan.com.vn/phan-tich-nhan-dinh/thi-truong-can-ho-cao-cap-can-mot-su-sang-loc-khat-khe-ar97716';
$ddd = file_get_contents($kkk);
echo $ddd;
?>
Can you help me explain this error or any idea
thank you so much!
Yes, file_get_contents() returns that msg "sorry! something went wrong." for me also. Make API call using PHP CURL. Let's try like this way-
Note:
URL which is not retrieved by file_get_contents(), because their
server checks whether the request come from browser or any script?. If
they found request from script they simply disable page contents.
So that you have to make a request similar as browser request. PHP
Curl is suitable choice for this kind of job. See here
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://batdongsan.com.vn/phan-tich-nhan-dinh/thi-truong-can-ho-cao-cap-can-mot-su-sang-loc-khat-khe-ar97716",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POSTFIELDS => "",
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
My Controller.
public function showMonthlyReport($site_id, $report_id){
$reports = Report::where('report_id', $report_id)->firstOrFail();
$uptime = ???
return view('records', compact('site_id', 'report_id', 'reports', 'uptime'));
}
And my UptimeRobot.php reference https://uptimerobot.com/api getMonitors()method
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.uptimerobot.com/v2/getMonitors",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "Your Api Key",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"content-type: application/x-www-form-urlencoded"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
$data = json_decode($response);
$custom_uptime = ($data->monitors[0]->custom_uptime_ratio);
$uptime = explode("-",$custom_uptime);
}
?>
ApiCommand.php
public function handle()
{
include(app_path() . '/Includes/DeepCrawl.php');
include(app_path() . '/Includes/Uptime.php');
include(app_path() . '/Includes/HelloAnalytics.php');
$stringData = ApiCommand::drawGraph($uptime, $dates, $users, $otherResultsRows, $array_issues, $array_pages_breakdown, $array_uncrawled_url, $array_non_200_pages, $array_orphaned_pages, $array_non_indexable_pages, $array_crawl_source_gap, $array_https_http);
Storage::disk('local')->put('hello.txt', $stringData);
}
Currently building a laravel web application.
I am just wondering how can i able to gather data from uptimerobot. I'm going to use my controller so I can pass it to my view but I don't know how. I have code below with the curl php type above. Really confused what am I doing new programmer here. Can someone explain if I'm at the right path or is it possible to do in controller. Thanks in advance.
I can suggest slightly different solution:
Extract your curl code in a separate console command and run this command each minute (for example, as a cron job).
The result of the command save to database/file/memory.
In your showMonthlyReport() refer to existing result.
Benefits:
In this way you would not have to wait for your curl result on each showMonthlyReport(). All code will run asynchronously
All errors processing will be in one place
Command is testable