I'm currently doing a PHP script on a web app to scrap my drive :
By opening a PHP in my browser, I have a script based on examples provided by Google (https://github.com/google/google-api-php-client).
This script simply proceeds to a Files List and copy the files on the server.
It's working fine, but I want my Drive files to be synchronized with my server files. I have to run my PHP in my browser every time I want to copy again my files. Here are my questions about that :
Could I make a PHP script which would be executed programmatically by CRON everyday ?
At the moment the PHP opened in my browser needs an authentication (If no cookie of Google account already signed in is detected, I first need to login). I'd like to authorize my Google App once on my account to be able to scrap the drive every day.
Can I apply this script on multiple Google Accounts (Like having a list of G adresses or tokens, and the script runs for each one). The web app I'm doing is for a dizain of persons, and I'd like to synchronize their drives files with my server.
Thank you in advance for any answer. I don't know if this is very useful, but here is my code :
function retrieveAllFiles($service, $parameters) {
$results = array();
$pageToken = NULL;
do{
try {
if ($pageToken) {
$parameters['pageToken'] = $pageToken;
}
$files = $service->files->listFiles($parameters);
$results = array_merge($results, $files->getFiles());
$pageToken = $files->getNextPageToken();
} catch (Exception $e) {
print "Une erreur s'est produite : " . $e->getMessage();
$pageToken = NULL;
}
}while($pageToken);
return $results;
}
// Function to get a list of files id
function scrapDrive($service, $excluded) {
$final_files = array();
$query = "";
if(sizeof($excluded) == 0){
$query = "name = '--------'";
}else{
$query = "name = '".$excluded[0]."'";
}
// I just do a custom query if the user has chosen to exclude personal folders from his drive
foreach ($excluded as $folder_name) {
$query .= " or name ='".$folder_name."'";
}
// Get folders ID
$excluded_folders = retrieveAllFiles($service, array('q' => $query));
$excluded_folders_id = array();
foreach ($excluded_folders as $folder) {
array_push($excluded_folders_id, $folder['id']);
}
$usermail = getUserMail($service);
// Getting only files from last month
$artworks_query = "modifiedTime > '".date("Y-m-d\TH:i:sP",strtotime("-1 month"))."'";
// Only getting images
$artworks_query .= " and mimeType contains 'image/' and '".$usermail."' in owners";
foreach ($excluded_folders_id as $id) {
$artworks_query .= " and not '".$id."' in parents";
}
$artworks = retrieveAllFiles($service, array('q' => $artworks_query));
foreach ($artworks as $artwork) {
$final_artworks[$artwork['id']] = $artwork['name'];
}
return $final_artworks;
}
$excluded = getExcludedFolders($_SESSION['id']);
$artworks = scrapDrive($service, $excluded);
$nb_of_artworks = sizeof($artworks);
$i = 1;
// Copy each file
foreach ($artworks as $artwork_id => $artwork_name) {
$response = $service->files->get($artwork_id, array(
'alt' => 'media' ));
$content = $response->getBody()->getContents();
file_put_contents("folder/".$artwork_id.'_'.$artwork_name, $content);
}
$i++;
}
Thank you in advance for any help ! :-)
Could I make a PHP script which would be executed programmatically by CRON everyday?
Yes you can run a php script in cron. how to run a php script daily with the cron job on Ubuntu os
At the moment the PHP opened in my browser needs an authentication (If no cookie of Google account already signed in is detected, I first need to login).
That is because you are using Oauth2. You could just as easily save your refresh token in a file and then have your script read the refresh token from the file rather than reading it from the cookie. check this
I'd like to authorize my Google App once on my account to be able to scrap the drive every day.
If this is an account that you personally have control of then you should consider using service account authentication rather than Oauth2. check this
Can I apply this script on multiple Google Accounts (Like having a list of G adresses or tokens, and the script runs for each one). The web app I'm doing is for a dizain of persons, and I'd like to synchronize their drives files with my server.
Using Oauth2 you can have a refresh token for as many accounts as you wish. The owner of the account must simply authenticate your application and you will have a refresh token to access it.
I am trying to get all my MLS listing via PHP using PHRETS which I downloaded from here:
https://github.com/dangodev/PHRETS-Example/blob/master/lib/phrets.php
and I used this example to download my listings into a csv format:
https://github.com/troydavisson/PHRETS/wiki/Connect,%20download%20listing%20data%20in%20CSV%20format,%20disconnect
I got the RETS URL, username and password from my MLS board, but I still can’t connect.
My code returns false when call the PHRETS library here:
require_once("phrets.php");
// start rets connection
$rets = new phRETS;
echo "+ Connecting to {$rets_login_url} as {$rets_username}<br>\n";
$connect = $rets->Connect($rets_login_url, $rets_username, $rets_password);
if ($connect) {
echo " + Connected<br>\n";
}
else {
echo " + Not connected:<br>\n";
print_r($rets->Error());
exit;
}
When I goto to library and look at the method Connect, that code returns false here:
// make request to Login transaction
$result = $this->RETSRequest($this->capability_url['Login']);
if (!$result) {
return false;
}
And when I look at my RETSRequest Method, it returns false here because the response code is 0 and not 200
if ($response_code != 200) {
$this->set_error_info("http", $response_code, $response_body);
return false;
}
and here is where its trying to connect:
if ($this->ua_auth == true) {
$session_id_to_calculate_with = "";
// calculate RETS-UA-Authorization header
$ua_a1 = md5($this->static_headers['User-Agent'] .':'. $this->ua_pwd);
$session_id_to_calculate_with = ($this->use_interealty_ua_auth == true) ? "" : $this->session_id;
$ua_dig_resp = md5(trim($ua_a1) .':'. trim($this->request_id) .':'. trim($session_id_to_calculate_with) .':'. trim($this->static_headers['RETS-Version']));
$request_headers .= "RETS-UA-Authorization: Digest {$ua_dig_resp}\r\n";
}
$this->last_request_url = $request_url;
curl_setopt($this->ch, CURLOPT_URL, $request_url);
curl_setopt($this->ch, CURLOPT_HTTPHEADER, array(trim($request_headers)));
// do it
$response_body = curl_exec($this->ch);
$response_code = curl_getinfo($this->ch, CURLINFO_HTTP_CODE);
Why can’t I connect?
I was able to login via http://retsmd.com with the url, username and password. I really need to get my listings in the format of CSV.
PLEASE HELP
I do have curl installed on my server, I checked with this method:
Check to see if cURL is installed locally?
A response code of 0 usually indicates that your server failed to open a connection with the server, likely due to firewall issues between you and them. Some RETS servers still run on a port 6103, so if your server (hosting company, etc.) prevent outbound connections from being opened on that port, that could be the cause for what you're seeing.
I'd recommend trying your code example from a different server or computer that doesn't have any connection restrictions. You could also try https://retsmd.com/auth as a way to verify that the credentials you've been given will work (assuming your local environment has what it needs for PHRETS to run).
Adding to troydavisson's answer,
You can include the following extra line of code to get the log of your connection for checking the issue.
// start rets connection
$rets = new phRETS;
$rets->SetParam('debug_mode', true);//extra line of code to get log
You will get full log in the rets_debug.txt file.
I use the Lists-WS from Sharepoint to retrieve information about DocumentLibraries and the files in these Libraries. Now I want to upload new files. How do I implement uploads with PHP? Till now I use Thybag SharePointAPI to get information (Link).
(the SharepointServer uses NTLM-Authentication)
THX in advance!!
UPDATE:
I want to call the Copy.asmx WS from sharepoint. To do so, I use the following lines:
$sourceurl = 'http://null';
$params = '
<CopyIntoItems xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<SourceUrl>'.$sourceurl.'</SourceUrl>
<DestinationUrls>' . $destinationURLs . '</DestinationUrls>
<Stream>' . $stream . '</Stream>
</CopyIntoItems>
';
$xmlvar = new \SoapVar($params, XSD_ANYXML);
// Attempt to run operation
try {
$result = $this->soapClient->CopyIntoItems($xmlvar)->CopyIntoItemsResponse->CopyIntoItemsResult;
} catch (\SoapFault $fault) {
$this->onError($fault);
}
But I dont even get any response ($result == NULL).....
You can use SPServices' CopyIntoItems method. You can find a detailed conversation here containing the details on using the CopyIntoItems service to upload a document.
I want to post an image on twitter but I can't. I am able to post only text message with twitter API.
The error is: Error: Server error: couldn't open file "Capture.png"
I use wamp server. The picture is in the same folder with the script.
Where is the problem?
require_once '../src/twitter.class.php';
// ENTER HERE YOUR CREDENTIALS (see readme.txt)
$twitter = new Twitter. ('raQBedybFLb', 'HWgN6Qt11jg0LY', '862846056-ndeGfMA83r9ldh', 'VMBvHDahXnGp');
$p='./Capture.png';
try {
$tweet = $twitter->send('testing twitter api', $p); // you can add $imagePath as second argument
} catch (TwitterException $e) {
echo 'Error: ' . $e->getMessage();
}
You may use $p = realpath('Capture.png');
I'm developing an API to let my users access to files stored on another server.
Let's call my two servers, server 1 and server 2!
server 1 is the server im hosting my web site, and
server 2 is the server im storing my files!
My site is basically Javascript based one, so I will be using Javascript to post data to API when user needs to access files which are stored on server 2.
when users requests to access files, the data will be posted to API URL via Javascript! API is made of PHP. Using that PHP script(API) on server 1, I will made another request to server 2 asking for files so there will be another PHP script(API) on server 2.
I need to know how should I do this authentication between two servers as server 2 has no access to user details on server 1?
I hope to do that like this, I can use the method which is used by most payment gateways.
When API on server 2 received a request with some unique data of the user , post back those unique data through SSL to server 1 API and match them with user data in the database, then post back result through SSL to server 2 so then server 2 knows file request is a genuine request.
In this case what kind of user data/credentials server 1 API should post to server 2 and server 2 API should post back to server 1? and which user data should be matched with the data in the database? like user ID, session, cookies, ip, time stamp, ect!
Any clear and described answer would be nice! Thanks.
I would go with this:
user initiates action, javascript asks Server 1 (ajax) for request for file on Server 2
Server 1 creates URL using hash_hmac with data: file, user ID, user secret
when clicking that URL (server2.com/?file=FILE&user_id=ID&hash=SHA_1_HASH) server 2 asks server 1 for validation (sends file, user_id and hash)
server 1 does the validation, sends response to server 2
server 2 pushes file or sends 403 HTTP response
This way, server 2 only needs to consume API of server 1, server 1 has all the logic.
Pseudocode for hash and url creation:
// getHash($userId, $file) method
$user = getUser($userId);
$hash = hash_hmac('sha1', $userId . $file, $user->getSecret());
// getUrl($userId, $file) method
return sprintf('http://server2.com/get-file?file=%1&user_id=%2&hash=%3',
$userId,
$file,
$security->getHash($userId, $file)
);
Pseudocode for validation:
$hash = $security->getHash($_GET['id'], $_GET['file']);
if ($hash === $_GET['hash']) {
// All is good
}
Edit: getHash() method accepts user ID and file (ID or string, what ever suits your needs). With that data, it produces a hash, using hash_hmac method. For the secret parameter of hash_hmac function, users "secret key" is used. That key would be stored together with users data in the db table. It would be generated with mt_rand or even something stronger as reading /dev/random or using something like https://stackoverflow.com/a/16478556/691850.
A word of advice, use mod_xsendfile on server 2 (if it is Apache) to push files.
Introduction
You can use 2 simple method
Authentication Token
Signed Request
You can also combine both of them by using Token for authentication and using signature to verify integrity of the message sent
Authentication Token
If you are going to consider matching any identification in the database perhaps you can consider creating authentication token rather than user ID, session, cookies, ip, time stamp, etc! as suggested.
Create a random token and save to Database
$token = bin2hex(mcrypt_create_iv(64, MCRYPT_DEV_URANDOM));
This can be easily generated
You can guaranteed it more difficult to guess unlike password
It can easily be deleted if compromised and re generate another key
Signed Request
The concept is simple, For each file uploaded must meat a specific signature crated using a random generated key just like the token for each specific user
This can easily be implemented with HMAC with hash_hmac_file function
Combine Both Authentication & Signed Request
Here is a simple Prof of concept
Server 1
/**
* This should be stored securly
* Only known to User
* Unique to each User
* Eg : mcrypt_create_iv(32, MCRYPT_DEV_URANDOM);
*/
$key = "d767d183315656d90cce5c8a316c596c971246fbc48d70f06f94177f6b5d7174";
$token = "3380cb5229d4737ebe8e92c1c2a90542e46ce288901da80fe8d8c456bace2a9e";
$url = "http://server 2/run.php";
// Start File Upload Manager
$request = new FileManager($key, $token);
// Send Multiple Files
$responce = $request->send($url, [
"file1" => __DIR__ . "/a.png",
"file2" => __DIR__ . "/b.css"
]);
// Decode Responce
$json = json_decode($responce->data, true);
// Output Information
foreach($json as $file) {
printf("%s - %s \n", $file['name'], $file['msg']);
}
Output
temp\14-a.png - OK
temp\14-b.css - OK
Server 2
// Where to store the files
$tmpDir = __DIR__ . "/temp";
try {
$file = new FileManager($key, $token);
echo json_encode($file->recive($tmpDir), 128);
} catch (Exception $e) {
echo json_encode([
[
"name" => "Execption",
"msg" => $e->getMessage(),
"status" => 0
]
], 128);
}
Class Used
class FileManager {
private $key;
function __construct($key, $token) {
$this->key = $key;
$this->token = $token;
}
function send($url, $files) {
$post = [];
// Convert to array fromat
$files = is_array($files) ? $files : [
$files
];
// Build Post Request
foreach($files as $name => $file) {
$file = realpath($file);
if (! (is_file($file) || is_readable($file))) {
throw new InvalidArgumentException("Invalid File");
}
// Add File
$post[$name] = "#" . $file;
// Sign File
$post[$name . "-sign"] = $this->sign($file);
}
// Start Curl ;
$ch = curl_init($url);
$options = [
CURLOPT_HTTPHEADER => [
"X-TOKEN:" . $this->token
],
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_POST => count($post),
CURLOPT_POSTFIELDS => $post
];
curl_setopt_array($ch, $options);
// Get Responce
$responce = [
"data" => curl_exec($ch),
"error" => curl_error($ch),
"error" => curl_errno($ch),
"info" => curl_getinfo($ch)
];
curl_close($ch);
return (object) $responce;
}
function recive($dir) {
if (! isset($_SERVER['HTTP_X_TOKEN'])) {
throw new ErrorException("Missing Security Token");
}
if ($_SERVER['HTTP_X_TOKEN'] !== $this->token) {
throw new ErrorException("Invalid Security Token");
}
if (! isset($_FILES)) {
throw new ErrorException("File was not uploaded");
}
$responce = [];
foreach($_FILES as $name => $file) {
$responce[$name]['status'] = 0;
// check if file is uploaded
if ($file['error'] == UPLOAD_ERR_OK) {
// Check for signatire
if (isset($_POST[$name . '-sign']) && $_POST[$name . '-sign'] === $this->sign($file['tmp_name'])) {
$path = $dir . DIRECTORY_SEPARATOR . $file['name'];
$x = 0;
while(file_exists($path)) {
$x ++;
$path = $dir . DIRECTORY_SEPARATOR . $x . "-" . $file['name'];
}
// Move File to temp folder
move_uploaded_file($file['tmp_name'], $path);
$responce[$name]['name'] = $path;
$responce[$name]['sign'] = $_POST[$name . '-sign'];
$responce[$name]['status'] = 1;
$responce[$name]['msg'] = "OK";
} else {
$responce[$name]['msg'] = sprintf("Invalid File Signature");
}
} else {
$responce[$name]['msg'] = sprintf("Upload Error : %s" . $file['error']);
}
}
return $responce;
}
private function sign($file) {
return hash_hmac_file("sha256", $file, $this->key);
}
}
Other things to consider
For better security you can consider the follow
IP Lock down
File Size Limit
File Type Validation
Public-Key Cryptography
Changing Date Based token generation
Conclusion
The sample class can be extended in so many ways and rather than use URL you can consider a proper json RCP solution
A long enough, single-use, short-lived, random generated key should suffice in this case.
Client requests for a file to Server 1
Server 1 confirms login information and generates a long single-use key and sends it to the user. Server 1 keeps track of this key and matches it with an actual file on Server 2.
Client sends a request to Server 2 along with the key
Server 2 contacts Server 1 and submits the key
Server 1 returns a file path if the key is valid. The key is invalidated (destroyed).
Server 2 sends the file to the client
Server 1 invalidates the key after say 30 seconds, even if it didn't receive a confirmation request from Server 2. Your front-end should account for this case and retry the process a couple of times before returning an error.
I do not think there is a point in sending cookie/session information along, this information can be brute-forced just like the random key.
A 1024-bit long key sounds more than reasonable. This entropy can be obtained with a string of less than 200 alphanumeric characters.
For the absolute best security you would need some communication from server 2 to server 1, to double check if the request is valid. Although this communication could be minimal, its still communication and thus slows down the proces.
If you could live with a marginally less secure solution, I would suggest the following.
Server 1 requestfile.php:
<?php
//check login
if (!$loggedon) {
die('You need to be logged on');
}
$dataKey = array();
$uniqueKey = 'fgsdjk%^347JH$#^%&5ghjksc'; //choose whatever you want.
//check file
$file = isset($_GET['file']) ? $_GET['file'] : '';
if (empty($file)) {
die('Invalid request');
}
//add user data to create a reasonably unique fingerprint.
//It will mostlikely be the same for people in the same office with the same browser, thats mainly where the security drop comes from.
//I double check if all variables are set just to be sure. Most of these will never be missing.
if (isset($_SERVER['HTTP_USER_AGENT'])) {
$dataKey[] = $_SERVER['HTTP_USER_AGENT'];
}
if (isset($_SERVER['REMOTE_ADDR'])) {
$dataKey[] = $_SERVER['REMOTE_ADDR'];
}
if (isset($_SERVER['HTTP_ACCEPT_LANGUAGE'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT_LANGUAGE'];
}
if (isset($_SERVER['HTTP_ACCEPT_ENCODING'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT_ENCODING'];
}
if (isset($_SERVER['HTTP_ACCEPT'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT'];
}
//also add the unique key
$dataKey[] = $uniqueKey;
//add the file
$dataKey[] = $file;
//add a timestamp. Since the request will be a different times, dont use the exact second
//make sure its added last
$dataKey[] = date('YmdHi');
//create a hash
$hash = md5(implode('-', $dataKey));
//send to server 2
header('Location: https://server2.com/download.php?file='.urlencode($file).'&key='.$hash);
?>
On server 2 you will do almost the same.
<?php
$valid = false;
$dataKey = array();
$uniqueKey = 'fgsdjk%^347JH$#^%&5ghjksc'; //same as on server one
//check file
$file = isset($_GET['file']) ? $_GET['file'] : '';
if (empty($file)) {
die('Invalid request');
}
//check key
$key = isset($_GET['key']) ? $_GET['key'] : '';
if (empty($key)) {
die('Invalid request');
}
//add user data to create a reasonably unique fingerprint.
if (isset($_SERVER['HTTP_USER_AGENT'])) {
$dataKey[] = $_SERVER['HTTP_USER_AGENT'];
}
if (isset($_SERVER['REMOTE_ADDR'])) {
$dataKey[] = $_SERVER['REMOTE_ADDR'];
}
if (isset($_SERVER['HTTP_ACCEPT_LANGUAGE'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT_LANGUAGE'];
}
if (isset($_SERVER['HTTP_ACCEPT_ENCODING'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT_ENCODING'];
}
if (isset($_SERVER['HTTP_ACCEPT'])) {
$dataKey[] = $_SERVER['HTTP_ACCEPT'];
}
//also add the unique key
$dataKey[] = $uniqueKey;
//add the file
$dataKey[] = $file;
//add a timestamp. Since the request will be a different times, dont use the exact second
//keep the request time in a variable
$time = time();
$dataKey[] = date('YmdHi', $time);
//create a hash
$hash = md5(implode('-', $dataKey));
if ($hash == $key) {
$valid = true;
} else {
//perhaps the request to server one was made at 2013-06-26 14:59 and the request to server 2 come in at 2013-06-26 15:00
//It would still fail when the request to server 1 and 2 are more then one minute apart, but I think thats an acceptable margin. You could always adjust for more margin though.
//drop the current time
$requesttime = array_pop($dataKey);
//go back one minute
$time -= 60;
//add the time again
$dataKey[] = date('YmdHi', $time);
//create a hash
$hash = md5(implode('-', $dataKey));
if ($hash == $key) {
$valid = true;
}
}
if ($valid!==true) {
die('Invalid request');
}
//all is ok. Put the code to download the file here
?>
You can restrict access to server2. Only server1 will be able to send request to server2. You can do this by whitelisting ip of server1 on server side or using .htaccess file. In php you can do by checking request generated ip and validate it with server1 ip.
Also you can write a algorithm which generates a unique number. Using that algorithm generate a number on server1 and send it to server2 in request. On server2 check if that number is generated by algorithm and if yes then request is valid.
I'd go with a simple symetric encryption, where server 1 encodes the date and the authenticated user using a key known only by server 1 and server 2, sending it to the client who cant read it, but can send it to server 2 as a sort of ticket to authenticate himself. The date is important to not let any client use the same "ticket" over the time. But at least one of the servers must know which user have access to which files, so unless you use dedicated folders or access groups you must keep the user and file infos together.