At the moment I am able to write to the datastore once I deploy my code, but I can't write to the datastore emulator with code running locally since it throws a ca-bundle error. The local datastore is visible at localhost:8000
use google\appengine\api\users\User;
use google\appengine\api\users\UserService;
use google\appengine\api\app_identity\AppIdentityService;
echo AppIdentityService::getApplicationId()."<br>";
echo AppIdentityService::getDefaultVersionHostname()."<br>";
# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\ServiceBuilder;
$cloud = new ServiceBuilder([
'projectId' => AppIdentityService::getApplicationId(),
'keyFilePath'=>'review-9504000716d8.json'
]);
$datastore = $cloud->datastore();
# The kind for the new entity
$kind = 'Task';
# The name/ID for the new entity
$name = 'sampletask1';
# The Cloud Datastore key for the new entity
$taskKey = $datastore->key($kind, $name);
# Prepares the new entity
$task = $datastore->entity($taskKey, ['description' => 'Buy milk']);
# Saves the entity
$datastore->upsert($task);
This code runs without any issues when deployed. But locally throws:
Fatal error: Uncaught exception 'Google\Cloud\Exception\ServiceException' with message 'No system CA bundle could be found in any of the the common system locations. PHP versions earlier than 5.6 are not properly configured to use the system's CA bundle by default. In order to verify peer certificates, you will need to supply the path on disk to a certificate bundle to the 'verify' request option: http://docs.guzzlephp.org/en/latest/clients.html#verify. If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here (provided by the maintainer of cURL): https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option. See http://curl.haxx.se/docs/sslcerts.html for more information.' in D:\Google\php\appengine-php-guestbook-phase0-helloworld\appengine-php-guestbook-phase0-hellowo in D:\Google\php\appengine-php-guestbook-phase0-helloworld\appengine-php-guestbook-phase0-helloworld\vendor\google\cloud\src\RequestWrapper.php on line 219
I didn't manage to make the local server even consider the php.ini file nor did I manage to upgrade the bundled php55 to at least php56.
Thus I actually have 2 questions:
how to properly connect from the local instance (dev_appserver.py) on windows to Google's remote datastore?
how to properly connect from the local instant to the local emulated datastore so I can view the data on localhost:8000?
The APIs are using CA certificate files for authentication more specifically they are using curl.cainfo.
Now you server might already have this file configured in php.ini. You can check in server file. Remember there could be different ini files for different environments like apache, cli.
Now you can either copy that file or Create your own authority file
Option 1:
Set absolute path in php.ini
Option 2:
Use ini_set to set this config.
Option 3:
Try with some other mode of authentication, i am sure google will have that.
Option 4:
As given in your question itself.
If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here
https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option
Related
I tried to connect to AmazonMQ with amqps protocol in AmazonMQ and get this error:
NOTICE: PHP message: [error] Symfony\Component\Messenger\Exception\InvalidArgumentException
cfi-lms-api_1 | No CA certificate has been provided. Set "amqp.cacert" in your php.ini or pass the "cacert" parameter in the DSN to use SSL. Alternatively, you can use amqp:// to use without SSL.
Where I can get this certificate and how to properly work with amqps in case AmazonMQ? What path for cacert I should specify?
I tried use amqp protocol but Amazon MQ does not support it.
In the docs:
If you want to use TLS/SSL encrypted AMQP, you must also provide a CA certificate. Define the certificate path in the amqp.cacert PHP.ini setting (e.g. amqp.cacert = /etc/ssl/certs) or in the cacert parameter of the DSN (e.g amqps://localhost?cacert=/etc/ssl/certs/).
(at https://symfony.com/doc/current/messenger.html)
It says cacert is "Path to the CA cert file in PEM format."
You can download the file through your web browser, clicking the "locker" icon next to "https" in the url bar. For example, the ca cert for StackOverflow can be downloaded like so :
Prefer the one from the "root" tab, which is usually valid way longer (+ 10 years).
Source:
https://github.com/symfony/amqp-messenger/blob/5.4/Transport/Connection.php#L232
https://github.com/symfony/amqp-messenger/blob/4175a0a98507e7ec575dca9b36e6c0a5a072d3fd/Transport/Connection.php#L285
For the noobs o news with AmazonMQ/rabbitmq
Get the PEM from this url : https://www.amazontrust.com/repository/AmazonRootCA1.pem
and pass = ?cacert=/path/to/pem/Amazon_Root_CA_1.pem
I hope it helps.
#Musa thanks for the answer! I will provide more details for somebody like me who never setup this certificate:
If you need work with Amazon MQ you should download the root certificate for mq.eu-central-1.amazonaws.com in pem format (see the answer from #Musa on how to do it).
Then you should change MESSENGER_TRANSPORT_DSN in .env file to add the new parameter cacert (see Symfony doc for more details https://symfony.com/doc/current/messenger.html#amqp-transport):
MESSENGER_TRANSPORT_DSN=amqps://username:password#your-secure-url.mq.eu-central-1.amazonaws.com:5671/%2f/_messages?cacert=/path-to-root-cert/mq-eu-central-1-amazonaws-com.pem
Another possible solution is to specify Amazon Root CA 1. Every Linux distribution has this certificate (see package ca-certificates). In this case, MESSENGER_TRANSPORT_DSN must be like this:
MESSENGER_TRANSPORT_DSN=amqps://username:password#your-secure-url.mq.eu-central-1.amazonaws.com:5671/%2f/_messages?cacert=/etc/ssl/certs/Amazon_Root_CA_1.pem
I want to connect to a WebSphere MQ Queue using SSL authentification from a PHP application (script).
Queue Manager Version is 7.0 or 7.5
MQ Client Version is 8.0
PHP Version is 7.0 (docker)
Using PHP mqseries pecl extension v0.15 (with custom fixes)
So far I was able to connect without SSL connection.
Here are the steps I have to done so far:
I have installed the WebSphere MQ Client v8 for Linux (CentOS in my case)
I have downloaded the PECL 0.15 version of the mqseries php extension. (There as a small bug in the extension, I had to recompile it to make it work properly. I used to get a segmentation fault on MQGET).
I linked the the mqseries.so to php and enabled the extension.
I succesfully (without SSL)
Connect to the queue manager
Open the queue for reading
Get messages on the queue
Close the connection
When I set the USE_SSL in my script to true, I get the error code 2393 that means "An MQCONN or MQCONNX call was issued with SSL configuration options specified, but an error occurred during the initialization of the SSL environment." This error message is very generic and does not help me pin point where is the problem.
The MQ_KEYSTORE is set as /path/to/my/key and my filename is key.kdb and has at the same level key.sth as suggested by this documentation
The MQ_SSL_CIPHER_SPEC is the same in the script than specified by on the queue manager for the specified MQ_CHANNEL_SSL. Checked multiple times. These are NOT the Cipher Suite used with JMS connections
The security cache has been refreshed on the queue manager.
On the server side, I checked the error logs for the queue manager and didn't seem to see my channel name. I say "seem" because there was a lot of noise and there were a few ??? channel name in the lot. So I feel like it did not reach the queue manager for some reason.
I also used the "amqssslc" command found in the MQ Client installation bin folder to test my ssl config. I get the same error than using the PHP script.
I also used WireShark to sniff packets on the corresponding MQ_PORT. The content of the packages contained certificate information. So there is something that looks like a SSL hand shake going on.
I am now out of ideas as of how to debug the case. Does anyone has an idea of what to check next? Is there connection logs on my MQ Client installation that I should check?
Here is an example of a connection using SSL in PHP
Here is a simplified version of my MQ script (I removed the outputs). Some of the constants are not disclosed for security purposes.
All MQSERIES_* constants are defined in the extension
All MQ_* are hardcoded parameters to test my script but their definition does not appear in the script excerpt.
<?php
// Constants defined here
$options = [
'Version' => MQSERIES_MQCNO_VERSION_4,
'Options' => MQSERIES_MQCNO_STANDARD_BINDING,
'MQCD' => [
'Version' => 7,
'ChannelName' => MQ_CHANNEL,
'ConnectionName' => sprintf('%s(%s)', MQ_HOST, MQ_PORT),
'TransportType' => MQSERIES_MQXPT_TCP,
]
];
if (USE_SSL) {
$options['MQSCO'] = [
'KeyRepository' => MQ_KEYSTORE
];
$options['MQCD']['ChannelName'] = MQ_CHANNEL_SSL;
$options['MQCD']['SSLCipherSpec'] = MQ_SSL_CIPHER_SPEC;
}
mqseries_connx(MQ_QUEUE_MANAGER, $options, $conn, $comp_code, $reason );
$mqods2 = [
'ObjectType' => MQSERIES_MQOT_Q,
'ObjectName' => MQ_QUEUE
];
mqseries_open($conn, $mqods2, MQSERIES_MQOO_INPUT_AS_Q_DEF | MQSERIES_MQOO_FAIL_IF_QUIESCING, $obj, $comp_code, $reason);
$gmd = [];
$gmo = [
'Options' => MQSERIES_MQGMO_FAIL_IF_QUIESCING | MQSERIES_MQGMO_WAIT, 'WaitInterval' => 3000
];
$msg = "";
$data_length = "";
for ($i = 0; $i < 1000; $i++) {
mqseries_get($conn, $obj, $gmd, $gmo, 10000, $msg, $data_length, $comp_code, $reason);
if ($reason === 2033) {
printf("No more messages to process\n");
break;
}
// Business logic
}
mqseries_disc($conn, $comp_code, $reason);
?>
UPDATE
Using the client side logs that I didn't know existed. We finally found out that our server was not using a proper certificate. The server's certificate was actually self-signed, thus not granting us access because our .kdb file didn't have it's public key. We added the server's own public key to the .kdb file making that step work out.
As suggested in the comments from JoshMC, for our next problem, we suspect the label is wrong in the keystore file. We did not specify a specific label from the .kdb file. From the docs "For queue managers and clients respectively, the following sources are searched in sequence for a non-empty value. The first non-empty value determines the certificate label. The certificate label must exist in the key repository. If no matching certificate in the correct case and format is found that matches a label, an error occurs and the SSL or TLS handshake fails."
And it also states that "The certificate label cannot contain spaces.".
I will try again tomorrow with proper label naming and sending a specific label name. I will try to see if the name convention ibmwebspheremq<user_that_runs_the_php_process> actually impacts the validity.
UPDATE 2
It finally worked out. As mentioned by JoshMC, the private key needs to have the specific label ibmwebspheremq. I didn't try the CertificateLabel yet. I might dig into that next week.
At first we were using the GUI (ikeyman) from ibm to generate the .kdb file. But we found out it had quite a few bugs in it. For instance, it imports twice the private key with the same label (auto-generated label from the certificate). Editing the label name was not possible (hence the connection problems). To solve that part we used the command line tool ikeycmd that basically offers the same features on the command line (minus the bugs). To run the command, you need the IBM jre (Not verified, needs to be checked out) and run it as java com.ibm.gsk.ikeyman.ikeycmd. From here, there is a whole world of documentation on IBM web site about how to create a certificate, rename labels, check details, etc. Lots and lots of fun!
With a full IBM MQ client install, client side errors will be logged to the directory: /var/mqm/errors
The errors if any will be logged to the file AMQERR01.LOG (this gets rotated with two other files ending in 02 and 03. If the error is something that MQ is not expecting it may also create a file that ended in .FDC with additional details.
The Queue Manager will log a channel name of ??? if the connection has failed during the channel negotiating prior to the client the sending the channel name. If the IBM MQ the queue manager is using v7.5 or lower the channel name is not exchanged until after the TLS handshake is complete.
With v8 and later clients connecting to v8 and later queue managers, MQ will use TLS SNI to exchange the channel name during TLS negotiation, however I am unsure if MQ has also been enhanced to log this channel name where in prior versions it logged ???. Note that IBM MQ Classes for Java and IBM MQ Classes for JMS client even at v8 and later do not support the SNI feature and do not send the channel name until after the TLS handshake is complete.
Check that your client key.kdb has a private key. There are three ways which MQ uses to identify which private key to use.
With MQ Client v7.5 and earlier the label of the private key must be:
ibmwebspheremq<user_that_runs_the_php_process>
With MQ client v7.5 and earlier an alternative is to set the private key as the default certificate and make sure the following environment variable is set in the environment which executes the program. Note this also works with 8.0.0.7/9.0.0.1 but I prefer the next method over this one.
AMQ_SSL_ALLOW_DEFAULT_CERT=1
With MQ Client v8 and later you can use CertificateLabel=anylabelvalue in the SSL stanza of the mqclient.ini. Example below:
SSL:
CertificateLabel=anylabelvalue
The CertificateLabel setting is documented in the IBM MQ v8 KC page "SSL
stanza of the client configuration file".
A template for mqclient.ini can be copied from /var/mqm/mqclient.ini and placed in the same directory where the application executes. The location can also be specified via a environment variable and MQ will look in a few other locations for the file if it the variable is not set and it does not find it in the same directory where the application executes. The various ways in which MQ will look for this file is documented in the IBM MQ v8 KC page "Location of the client configuration file"
I use ZF3 and code in the development mode. I configured it like the tutorial suggests:
composer development-enable
So everything works fine if this mode is enabled. If I disable it I get a database connection error, like this one:
Connect Error: SQLSTATE[HY000] [1044] Access denied for user
''#'localhost' to database 'xyz'
I still work on the same computer.
So what error it might be?
The main topic would be, how is the right way to change between development and production, does the composer statement also make clear to use the production configfiles?
If I have changed the mode via composer, what do I have to do additional? I really blueeyed thought, it would be enough to just disable:
composer development-disable
Do I have to rename the development config files also? Of which files do we talk about? Is it just application-config.php and development-config.php?
Where and how should I place the different database connections? I now use the files you see above.
And last, how to change the mode on the production server? I now just disabled the mode on my developmentsystem and then uploaded the hole project. Afterwards I only upload the changed files.
EDIT1: Here additional a screensot, which configuration files I use in which folders:
In my application.config.php the configuration links to:
'config_glob_paths' => [
realpath(__DIR__) . '/autoload/{{,*.}global,{,*.}local}.php',
],
and in my development.config.php the configuration links to
'module_listener_options' => [
'config_glob_paths' => [realpath(__DIR__) . '/autoload/{,*.}{global,local}-development.php'],
'config_cache_enabled' => false,
'module_map_cache_enabled' => false,
],
for me it looks correct. My database connection is in local.php (for the production) and in local-development.php (for the development mode).
Enabling/Disabling the mode is just the same as having/not having the config/development.config.php file.
If you look closely, you'll see that the development mode disables the cache.
Your problem is that the cache files have been created (non dev mode) while the configuration wasn't fine for the environment. Remove data/cache/application.config.cache and application.module.cache as configured in config/application.config.php.
If you use development-mode enable (Development) it mean config_cache_enabled set to false. So your new configuration like module, services, controllers, etc will load by ZF3, because ZF3 will not read the configuration from cache (in data/cache/*).
If development-mode disable (Production) configuration will be cached, so when you deploy your code with new configuration like I mention above, will not read by ZF3. Because ZF3 still read the configuration in cache.
I usually remove the cache when deploying to Production. Here the sample shell script I used for deploying
#/bin/bash
rsync --exclude data --exclude .git -av temp_example.com/. /var/www/example.com/.
echo -e "Removing cache..."
rm -f /var/www/example.com/data/cache/*.php
So, the main key, if you used development-mode disable, just remove the cache after deploying the code.
I am just trying to do a very simple file_get_contents('gs://[bucket][file]');
I am getting this error when I run locally.
"\google\appengine\ext\cloud_storage_streams\CloudStorageStreamWrapper::stream_open" call failed.
However, when I run it on my production env it works fine.
I know I am missing something with the setup, and I am just not sure what.
I have set the project gcloud config set project PROJECT_ID, set the account gcloud config set account ACCOUNT, and run gcloud auth login. Any ideas on what I am missing in my setup here?
I ran composer install , composer require google/cloud, I don't get any errors when including use Google\Cloud\Storage\StorageClient; I created the service-account-keyfile as an owner of the bucket. Set owners to full permissions. Ran this code...
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => __DIR__ .'/../keys/appdocs-com.json',
]);
echo file_get_contents('gs://[myBucket][myFile]');
and I am still getting ...streamWrapper::stream_open" call failed
I even tried the service builder approach.
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\ServiceBuilder;
use Google\Cloud\Storage\StorageClient;
$gcloud = new ServiceBuilder([
'keyFilePath' => __DIR__ .'/../keys/appdocs-com.json',
'projectId' => 'appdocs-com',
]);
$storage = $gcloud->storage();
$bucket = $storage->bucket('test-appdocs-sendgrid-inbound');
$object = $bucket->object('4/envelope.json');
$stream = $object->downloadAsStream();
echo $stream->getContents();
this returns
Fatal error: Uncaught exception 'Google\Cloud\Core\Exception\ServiceException' with message 'No system CA bundle could be found in any of the the common system locations. PHP versions earlier than 5.6 are not properly configured to use the system's CA bundle by default. In order to verify peer certificates, you will need to supply the path on disk to a certificate bundle to the 'verify' request option: http://docs.guzzlephp.org/en/latest/clients.html#verify. If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here (provided by the maintainer of cURL): https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option. See http://curl.haxx.se/docs/sslcerts.html for more information.' in /Users/thom/Engine/appdocs-com/vendor/google/cloud-core/RequestWrapper.php:241 Stack trac in /Users/thom/Engine/appdocs-com/vendor/google/cloud-core/RequestWrapper.php on line 241
I have my runtime at php55. Could that be preventing me from using StorageClient or serviceBuilder
When outside the appengine environment, you will need to manually register the cloud storage stream wrapper to access files using the gs:// protocol:
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => '/path/to/service-account-keyfile.json'
]);
$storage->registerStreamWrapper();
This example assumes you have Google Cloud PHP installed.
More information is available in the Google Cloud PHP docs.
To resolve the guzzle ssh error, refer to this answer. NOTE that setting verify to false is not safe, and should only be used to test, never in production.
As #jdp mentioned in his answer, the error I was getting was related to my installation of curl.
So I looked at Google's php_ini docs to see if I could set the openssl.cafile. https://cloud.google.com/appengine/docs/standard/php/config/php_ini
If you want to use Google Storage Client you need to change from standard to flexible env.
https://cloud.google.com/appengine/docs/flexible/php/quickstart
Then read through cloud storage docs to see how to download dependencies for the API usage: https://cloud.google.com/storage/docs/reference/libraries
Once you have set up your flex env and gathered dependencies, you can then follow these directions to set up the connection.
Set up a service account key In Google Cloud Console:
Select "your-project"->IAM & ADMIN-> Service Accounts.
Click the "Create Service Account" button.
Assign role (in my case I assigned the role "owner" since this service account was specific to my local development), mark the checkbox "Furnish New Private Key".
This just downloaded your "service-account-keyfile.json"
Then in the code initiate Cloud Storage like so:
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => '/path/to/service-account-keyfile.json'
]);
$storage->registerStreamWrapper();
You can now retrieve bucket data like you would from the production env.
echo file_get_contents('gs://[Bucket][File]');
Make sure you have set the bucket permissions to allow the previously selected role to access the bucket.
However, I would still love to find a way to set up a local environment with with php standard that can interact with gs://[bucket]/[file].
I have set up satis private composer packet manager.
Satis is running on "packages.asc.company", I protected the site by apache2 http basic authentication and can open it in browser by entering http basic auth credentials.
Now my question: How can I pass composer the credentials to access the satis site in the best and most secure manner when running e.g. "composer update"?
Currently I registered only one user with a password in apache .htpasswd file and need to pass its credentials somewhere to be able to connect from composer to satis.
There are two cases where I need to connect from:
1) From the project during development
2) From jenkins during continuous integration process.
3) Edit: SSL
I am trying to use openssl now to secure the credentials when logging in. On my linux, where the apache runs, I created a private key and a .crt file (see: Apache SSL . On my linux, I can now open the satis packages page with https, and I even redirect http to https, all working (Im using my own certificate generating with openssl because its an internal application and I don't need a trusted ca).
Now, when I switch to my Windows from my Linux vm (here I am coding), and I try to run composer update, I get the following error message:
(hosts file is configured correct)
[Composer\Downloader\TransportException]
The "https://packages.asc.company/packages.json" file could not be downloaded: SSL operation failed with code 1. OpenSSL Error messages:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Failed to enable crypto
failed to open stream: operation failed
What did I miss? I'm pretty new to ssl, but read the whole day now information about it and can't get it to work.
From getcomposer satis site, I have this information but don't know how to use it.
{
"repositories": [
{
"type": "composer",
"url": "https://example.org",
"options": {
"ssl": {
"local_cert": "/home/composer/.ssl/composer.pem"
}
}
}
]
}
Regards.
There is a documentation page for this.
Composer will work with adding user names into the Satis URL. Works for me, I just wanted to get around the useless default passworded server in the local network. There's a read only account, and I used it.
Additionally: Every developer in the company has an account on the repository server, and there isn't much use in protecting downloaded ZIP files with even more security. Composer itself currently doesn't support any code signing methods or hash comparison, so there is no way to know if a package has been tampered with either where stored or during transmission.
According to the docs, not giving credentials in the URL will make Composer ask for them, or you can add them to auth.json. On the other hand: Saving clear text passwords in a dedicated file doesn't sound like the best idea, and transmitting them without using HTTPS is even worse.
You have to define what kind of security you want to have. What is the goal or the threat scenario you want to protect against?