I have PHP generating JSON data but I am having a problem making a example.json file from it. What am I doing wrong? I just get a blank screen.
<?php
require('wp-load.php');
$EM_Events = EM_Events::get( array(
'scope'=>'future',
'orderby'=>'event_start_date,event_start_time',
'limit'=>1,
'owner'=>false,
'category'=>'6,7,8,10,11,12,13,14')
);
foreach ($EM_Events as $event) {
//Event Detail
$event_id = esc_html($event->event_id);
$event_name = esc_html($event->name);
$events[] = array(
'event_id'=> $event_id,
'event_name'=> $event_name,
);
}
$response['events'] = $events;
$fp = fopen('example.json', 'w');
fwrite($fp, json_encode($response));
fclose($fp);
?>
Try using a response sniffing tool like Fiddler http://www.fiddler2.com/ or use the network tab of the browser's developer tools like Firebug http://getfirebug.com/ Maybe the content came down just fine, it's just that your browser doesn't know how to render it.
Related
and thanks in advance. I try to build a webscraper with PHP and I use Visual Studio Code.
When I run the following code, the following problem shows up:
Use of unknown class: 'Goutte\Client'
Does anyone know how to solve that issue?
I have googled all over the place, looked at SO and asked the forbidden one, but still after three days I have not achieved any progress. (I am also a noob, so maybe it is not as difficult to solve as I think).
Looking forward to your feedback and tips.
<?php
require 'vendor/autoload.php';
use Goutte\Client;
// Initialize the Goutte client
$client = new Client();
// Create a new array to store the scraped data
$data = array();
// Loop through the pages
for ($i = 0; $i < 3; $i++) {
// Make a request to the website
$crawler = $client->request('GET', 'https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives_de?page=' . $i);
// Find all the initiatives on the page
$crawler->filter('.initiative')->each(function ($node) use (&$data) {
// Extract the information for each initiative
$title = $node->filter('h3')->text();
$link = $node->filter('a')->attr('href');
$description = $node->filter('p')->text();
$deadline = $node->filter('time')->attr('datetime');
// Append the data for the initiative to the data array
$data[] = array($title, $link, $description, $deadline);
});
// Sleep for a random amount of time between 5 and 10 seconds
$sleep = rand(5,10);
sleep($sleep);
}
// Open the output file
$fp = fopen('initiatives.csv', 'w');
// Write the header row
fputcsv($fp, array('Title', 'Link', 'Description', 'Deadline'));
I have a SPA (react) that is uses PHP calls to connect to a MongoDB.
Works great.
However, due to 'rules' I need to serve the javascript files from a different server -- a server that supports neither MongoDB nor MongoDB php client. Let's call this server, SERVER_A.
Let's call the server hosting the MongoDB PHP client and the MongoDB, SERVER_B. ('B' for 'backend'... :) )
The solution, I believe, was to build a php 'middleman' on SERVER_A that simply passes data on to SERVER_B. Research shows me that file_get_contents is the tool for this.
So I take my original known-working PHP file, I put it on SERVER_B and rename it to "mongoPatch_backend.php".
<?php
$user = "xxxx";
$pwd = 'xxx';
$filter = [];
if (isset($_REQUEST['needleKey'])) {
$needleKey = $_REQUEST['needleKey'];
$needle = $_REQUEST['needle'];
$filter = [$needleKey => $needle];
}
$newData = $_REQUEST['newData'];
$filter = ['x' => ['$gt' => 1]];
$options = [
'projection' => ['_id' => 0],
'sort' => ['x' => -1],
];
$bson = MongoDB\BSON\fromJSON($newData);
$value = MongoDB\BSON\toPHP($bson);
$manager = new MongoDB\Driver\Manager("mongodb://${user}:${pwd}#SERVER_B:27017");
$bulk = new MongoDB\Driver\BulkWrite;
$bulk->update(
[$needleKey => $needle],
['$set' => $value],
['multi' => false, 'upsert' => true]
);
$results = $manager->executeBulkWrite('sedwe.users', $bulk);
var_dump($results);
?>
On SERVER_A I then make a new file, dbPatch.php, like so:
<?php
$API = "https://SERVER_B/php/mongoPatch_backend.php?";
if (isset($_REQUEST['needleKey'])) {
$needleKey = $_REQUEST['needleKey'];
$needle = $_REQUEST['needle'];
$filter = [$needleKey => $needle];
}
$newData = $_REQUEST['newData'];
$postData = "needleKey=".urlencode($needleKey);
$postData .= "&needle=".urlencode($needle);
$postData .= "&newData=".urlencode($newData);
// echo $API.$postData;
$data = file_get_contents($API.$postData);
echo $data;
?>
But when I call it, I get nothing echo'd back out of $data.
Any idea why? Remember, the exact same call directly to mongoPatch_backend.php works just fine (but it's a CORS call, so it's not a valid option).
So here is what I've tried:
First, to ensure my AJAX call was still working, I spit out the response to the console. I get nothing.
So I changed the last line of the middleman (dbPatch.php) to echo "Hello World" instead of echo $data and received "Hello World" on the console as expected.
Next, I then changed the last line of my middleman (dbPatch.php) back to echo $data and tried reducing the 'backend' to a simple <?php echo "Hello Back World"; ?> and got nothing on the console.
Next, I go to https://SERVER_B/php/mongoPatch_backend.php in a browser ... and I'm greeted with "Hello Back World" as expected in the browser.
... which leads me to believe something is up with the transferring of info from server to server. Logical call, eh?
Unfortunately, when I try the same thing with just a 'fetch' command, it works perfectly fine:
Here is my 'middleman' (dbFetch.php) script on SERVER_A:
<?php
$API = "https://SERVER_B/php/mongoFetch_backend.php?";
$collection = $_REQUEST['collection'];
$postData = "collection=".urlencode($collection);
$needleID = $_REQUEST['needleID'];
$postData .= "&needleID=".urlencode($needleID);
$data = file_get_contents($API.$postData);
echo $data;
?>
And this is the file on the backend, SERVER_B:
<?php
$user = "xxxx";
$pwd = 'xxxx';
$filter = [];
if (isset($_REQUEST['needleID'])) {
$needleID = $_REQUEST['needleID'];
$filter = ['id'=> $needleID];
}
if (isset($_REQUEST['collection'])) {
$collection = $_REQUEST['collection'];
}
//Manager Class
$connection = new MongoDB\Driver\Manager("mongodb://${user}:${pwd}#SERVER_B:27017");
// Query Class
$query = new MongoDB\Driver\Query($filter);
$rows = $connection->executeQuery("db_name.$collection", $query);
// Convert rows to Array and send result back to client
$rowsArr = $rows->toArray();
echo json_encode($rowsArr);
?>
Huzzah, this works! ... and it's also proof there doesn't appear to be a problem with the server-to-server communication.
So back to ground zero.
I then go with a very simple newData value -- shorter, but same general layout - stringified json. It works! The data ends up in the database.
So I'm forced to think something is wrong with the data in newData (which is only a few hundred lines of stringified JSON made like this: encodeURIComponent(JSON.stringify(newData)). But, this bears repeating: this works if I skip the middleman.
... and that puts me at a loss. PHP isn't something I understand well... can you help?
EDIT to answer comment:
When I call mongoPatch_backend.php directly on SERVER_B it works (as stated above). I had it echo the $newData and it spits out a stringified JSON version of the variable (not URLencoded).
When I call dbPatch.php, it does not give me anything that was passed back from mongoPatch_backend.
As I said in the OP, if I modify mongoPatch_backend.php to do nothing other than echo "hellow world" I am still unable to log it to console when calling it via the above dbPatch.php file.
EDIT: I also tried putting the two PHP files on the same server and am getting the same results. (ie: both the dbPatch.php and mongoPatch.php files are in the same directory on the same server)
EDIT2: I have done a var_dump from the middleman and I get standard-looking human-readable stringified text back.
I do the same var_dump($_REQUEST['newData']); in the backend file and I get nothing back.
When I call webservices and response convert into jsonp format, at that time response come correctly but it is download as a file. File format is not displayed. Just download file. I try below code.
$result = array(
'Result'=>'Error',
'ErrorMessage'=>'Please enter valid Data
);
header('Content-type: application/jsonp');
return json_encode($result);
I also try with echo and print replace of return keyword.
$result = array(
'Result'=>'Error',
'ErrorMessage'=>'Please enter valid Data
);
$this->output->set_content_type('application/json')->set_output(json_encode($result ));
I am trying to download a rapidshare file using its "download" subroutine as a free user. The following is the code that I use to get response from the subroutine.
function rs_download($params)
{
$url = "http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=download&fileid=".$params['fileid']."&filename=".$params['filename'];
$reply = #file_get_contents($url);
if(!$reply)
{
return false;
}
$result_arr = array();
$result_keys = array(0=> 'hostname', 1=>'dlauth', 2=>'countdown_time', 3=>'md5hex');
if( preg_match("/DL:(.*)/", $reply, $reply_matches) )
{
$reply_altered = $reply_matches[1];
}
else
{
return false;
}
foreach( explode(',', $reply_altered) as $index => $value )
{
$result_arr[ $result_keys[$index] ] = $value;
}
return $result_arr;
}
For instance; trying to download this...
http://rapidshare.com/files/440817141/AutoRun__live-down.com_Champ.rar
I pass the fileid(440817141) and filename(AutoRun__live-down.com_Champ.rar) to rs_download(...) and I get a response just as rapidshare's api doc says.
The rapidshare api doc (see "sub=download") says call the server hostname with the download authentication string but I couldn't figure out what form the url should take.
Any suggestions?, I tried
$download_url = "http://$the-hostname/$the-dlauth-string/files/$fileid/$filename"
and a couple other variations of the above, nothing worked.
I use curl to download the file, like the following;
$cr = curl_init();
$fp = fopen ("d:/downloaded_files/file1.rar", "w");
// set curl options
$curl_options = array(
CURLOPT_URL => $download_url
,CURLOPT_FILE => $fp
,CURLOPT_HEADER => false
,CURLOPT_CONNECTTIMEOUT => 0
,CURLOPT_FOLLOWLOCATION => true
);
curl_setopt_array($cr, $curl_options);
curl_exec($cr);
curl_close($cr);
fclose($fp);
The above curl code doesn't seem to work, nothing gets downloaded. Probably its the download url that is incorrect.
Also tried this format for the download url:
"http://rs$serverid$shorthost.rapidshare.com/files/$fileid/$filename"
With this curl writes a file entry but that is all it does(writes a 0/1 kb file).
Here is the code that I use to get the serverid, shorthost, among a few other values from rapidshare.
function rs_checkfile($params)
{
$url = "http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=checkfiles_v1&files=".$params['fileids']."&filenames=".$params['filenames'];
// the response from rapishare would a string something like:
// 440817141,AutoRun__live-down.com_Champ.rar,47768,20,1,l3,0
$reply = #file_get_contents($url);
if(!$reply)
{
return false;
}
$result_arr = array();
$result_keys = array(0=> 'file_id', 1=>'file_name', 2=>'file_size', 3=>'server_id', 4=>'file_status', 5=>'short_host'
, 6=>'md5');
foreach( explode(',', $reply) as $index => $value )
{
$result_arr[ $result_keys[$index] ] = $value;
}
return $result_arr;
}
rs_checkfile(...) takes comma seperated fileids and filenames(no commas if calling for a single file)
Thanks in advance for any suggestions.
You start by requesting ?sub=download&fileid=X&filename=Y, and it returns $hostname,$dlauth,$countdown,$md5hex.. since you're a free user you have to delay for $countdown seconds, and then call ?sub=download&fileid=X&filename=Y&dlauth=Z to perform the download.
There's a working implementation in python here that would probably answer any of your other questions.
I had an interesting task today and couldn't find much on the subject. I wanted to share this, and ask for any suggestions on how this could have been done more elegantly. I consider myself a mediocre programmer who really wants to improve so any feedback is highly appreciated. There is also a strange bug I can't figure out. So here goes..and hopefully this helps someone who ever has to do something similar.
A client was redoing a site, moving content around, and had a couple thousand redirects that needed to be made. Marketing sent me an XLS with old URLs in one column, new URLs in the next. These were the actions I took:
Saved the XLS as CSV
Wrote a script which:
Formatted the list as valid 301 redirects
Exported the list to a text file
I then copy / pasted all the new directives into my .htaccess file.
Then, I wrote another script that checked to make sure each of the new links was valid (no 404s). The first script worked exactly as expected. For some reason, I can get the second script to print out all the 404 errors (there were several), but the script doesn't die when it finishes traversing the loop, and it doesn't write to the file, it just hangs in command line. No errors get reported. Any idea what's going on? Here is the code for both scripts:
Formatting 301s:
<?php
$source = "301.csv";
$output = "301.txt";
//grab the contents of the source file as an array, prepare the output file for writing
$sourceArray = file($source);
$handleOutput = fopen($output, "w");
//Set the strings we want to replace in an array. The first array are the original lines and the second are the strings to be replaced
$originalLines = array(
'http://hipaasecurityassessment.com',
','
);
$replacementStrings = array(
'',
' '
);
//Split each item from the array into two strings, one which occurs before the comma and the other which occurs after
function setContent($sourceArray, $originalLines = array(), $replacementStrings = array()){
$outputArray = array();
$text = 'redirect 301 ';
foreach ($sourceArray as $number => $item){
$pattern = '/[,]/';
$item = preg_split($pattern, $item);
$item = array(
$item[0],
preg_replace('#"#', '', $item[1])
);
$item = implode(' ', $item);
$item = str_replace($originalLines, $replacementStrings, $item);
array_push($outputArray,$text,$item);
}
$outputString = implode('', $outputArray);
return $outputString;
}
//Invoke the set content function
$outputString = setContent($sourceArray, $originalLines, $replacementStrings);
//Finally, write to the text file!
fwrite($handleOutput, $outputString);
Checking for 404s:
<?php
$source = "301.txt";
$output = "print404.txt";
//grab the contents of the source file as an array, prepare the output file for writing
$sourceArray = file($source);
$handleOutput = fopen($output, "w");
//Split each item from the array into two strings, one which occurs before the space and the other which occurs after
function getUrls($sourceArray = array()){
$outputArray = array();
foreach ($sourceArray as $number => $item){
$item = str_replace('redirect 301', '', $item);
$pattern = '#[ ]+#';
$item = preg_split($pattern, $item);
$item = array(
$item[0],
$item[1],
$item[2]
);
array_push($outputArray, $item[2]);
}
return $outputArray;
}
//Check each URL for a 404 error via a curl request
function check404($url = array(), $handleOutput){
$handle = curl_init($url);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE);
$content = curl_exec( $handle );
$response = curl_getinfo( $handle );
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
if($httpCode == 404) {
//fwrite($handleOutput, $url);
print $url;
}
};
$outputArray = getUrls($sourceArray);
foreach ($outputArray as $url)
{
$errors = check404($url, $handleOutput);
}
You should have used fgetcsv() for generating the original URL list. This splits up CSV files into an array, simplifying the transformation.
Can't say anything about the 404s or the error cause. But using the wacky curl functions is almost always a bad indicator. For testing purposes I would have used a commandline tool like wget instead so the results can be proof-checked manually.
But maybe you could try PHPs own get_headers() instead. It's supposed to show the raw result headers; shouldn't not follow redirects itself.