I was wondering does anyone know if it is possible in Zend to run one query which will connect to two different databases on two different servers and combine the results in one result set?
You can federate one database in the other one, via a wrapper. You reference table via nicknames, and then you execute a query normally, as both tables were in the same database (joining, sorting, etc.)
Federation is free between DB2 databases or with Informix (because is from IBM). If you want to federate another data source (Oracle, Excel, flat files), you have to buy that separately.
With federation, you do not need to do the join at application level, but at database level.
Related
I have a database that contains a large number of tables that can be divided into multiple databases. The connection between the tables is, for example:
DB1: users (contains the field 'client_id')
DB2: customers (contains all the tables and relationships)
The two DBs are therefore connected via the 'client_id' field in the users table in DB1, and the 'id' field in the customers table in DB2.
Additionally, I also have a third DB that is connected in a similar way to the second DB.
Is this good practice? I have read that it can create performance problems, but keeping everything in a single DB doesn't seem ideal either.
Do you have any ideas or suggestions? Can this approach work?
In MySQL, databases (aka schemas) are just subdirectories under the datadir. Tables in different schemas on the same MySQL Server instance share the same resources (storage, RAM, and CPU). You can make relationships between tables in different schemas.
Assuming they are on the same MySQL Server instance, there is no performance implication with keeping tables together in one schema versus separating them into multiple schemas.
Using schemas is mostly personal preference. It can make certain tasks more convenient, such as granting privileges, backing up and restoring, or using replication filters. These have no direct effect on query performance.
So far we have a server with 2 databases and a mysql user that accesses any of them. For example
`select * from magento.maintable, erp.maintable`
now our erp is very slow and we want to separate our database on another server, but we have hundreds (almost a thousand) sql queries that have access in the same query to the two databases, for example
`insert into magento.table
select * from erp.maintable`
or
select * from erp.maintable inner join magento.table...
and more and more
How can I make everything work the same without changing these queries? but with the databases on different servers
To access the databases I have created a class for each database and through an object I make the queries, insertions, updates and deletions, like this
` public function exec($query, $result_array = true)
{
$this->data->connect();
$result = $this->data->query($query, $result_array);
$this->data->disconnect();
return $result;
}`
all help is welcome, the point is to find an optimal way to do this and not have to manually change 1000 sql queries made by another programmer
To access more than one database server in one query, you either have to use FEDERATED database engine or use replication to replicate the ERP-data from another server to the original one.
The use of FEDERATED engine is likely to cause additional performance problems and the replication requires some work to set up.
If the sole reason for the new server is the performance in ERP, you might want to see why the ERP is slow and try to solve that (optimize, move both databases to a new server, etc). When you have both databases on the same server, the query optimizer is able to combine and make efficient use of indexes.
How can I connect two tables from different databases using PostgreSQL? My first database is called "payments_details" and my second one is called "insurance". Also I want to display and highlight the id's they don't have in common using php, is that possible?
Databases are isolated from each other, you cannot access data from different databases with one SQL statement. That is not a bug, but a design feature.
There are three ways to achieve what you want:
Don't put the data in different databases, but in different schemas in one database. It is a common mistake for people who are more experienced with MySQL to split up data that belong to one application in multiple databases and then try to join them. This is because the term database in MySQL is roughly equivalent to what in (standard) SQL is called a schema.
If you cannot do the above, e.g. because the data really belong to different applications, you can use the PostgreSQL foreign data wrapper. This enables you to access tables from a different database (or even on a different machine) as if they were local tables. You'll have to write your statements more carefully, because complicated queries can sometimes be inefficient if large amounts of data have to me transferred between the databases.
You can use dblink, which is an older and less comfortable interface than foreign data wrappers, but can allow you to do things that you could not do otherwise, like call a remote function.
I currently have 2 ODBC connections set up on my web server. One that connects to the our enterprise QAD database and another that connects to our custom database used to extend our database. In this paticular example I have my employee records in the QAD database, and then an employee number in another table in the custom database.
Is there any way for me to set up a cross join between the two odbc connections in php so that I don't have to loop through the results of the first query and send several queries based on the returned results to tie my records together in a php array?
The best i've been able to come up with is to build an IN clause from my first query from our custom database, send the second query to the QAD database, and then do an array merge in php. However, this is an extremely slow process compared to a normal SQL join.
Not sure if you've already found a solution to this but there is a Progress article on how to do this.
Quick Guide To Setting Up MultiDatabase ODBC Connectivity
I had a similar requirement - I wanted to create a join between a table in the primary QAD database and a custom table in our custom database. I have tested this and it works well although my setup is slightly different. I needed to connect to QAD from Microsoft SSRS to create reports against the QAD data - I needed to create some reports that the standard QAD report designer could not handle.
I have tested this on Progress 10.1c (this method is only supported in 10.1b+).
So the steps I took were:
Create the oesql.properties config file as per the article relevant to the primary and custom databases.
Create the ODBC System DSN on the client machine (in my case a Windows Server 2008 R2 machine running SQL Server 2008 R2 with SSRS) with the additional database references as per the article.
Create a Linked Server in SQL Server via the ODBC DSN
Create a view which uses the OpenQuery syntax to extract data from QAD (in my case this was created in the ReportServer database) via the linked server.
Create standard T-SQL query using the view in point 3 as the data source. This was ultimately the datasource for my SSRS report.
I believe it is important that the bit versions of the OS/Database and the ODBC drivers match but haven't confirmed this yet.
Whilst my requirement is different to your's ultimately it's the QAD server config and ODBC setup that's key. As long as your PHP client can perform a similar capability in terms of the OpenQuery command then you may get this working. I don't have any experience with PHP so can't help you there.
It seems a bit convoluted but actually works very well and in a lot of cases actually outperforms querying data using QAD browses!
Hope this helps.
Edit:
Here's a sample of an OpenQuery command - you can see that the table joins work in the normal way but just require and additional piece in the table reference.
CREATE VIEW [dbo].[vQADData] AS SELECT * FROM OPENQUERY(LinkedServerName,
'
SELECT custTable.item_date AS DESP_DATE, so_mstr.so_site AS SITE, so_mstr.so_po AS PO_NO, so_mstr.so_inv_nbr AS INV_NO,
ad_mstr.ad_name AS ADNAME, ad_mstr.ad_city AS ADCITY, ad_mstr.ad_state AS ADSTATE
FROM customdbname.pub.customtable custTable
INNER JOIN pub.so_mstr ON so_mstr.so_nbr = custTable.so_nbr
INNER JOIN pub.ad_mstr ON ad_mstr.ad_addr = so_mstr.so_ship
INNER JOIN pub.sod_det ON sod_det.sod_nbr = custTable.so_nbr
WHERE so_mstr.so_site = ''SiteName'' AND so_mstr.so_shipvia = ''SHIPPER'' AND custTable.item_date IS NULL
')
Then just access the view using normal SQL syntax.
SELECT * FROM vQADData
Thanks to Tiran for his suggested solution. For those people that are trying to reference multiple tables via SQL server as Tiran was doing, I have additional input.
I'm trying to pull data from multiple sources (Progress), same table structure at the same time and insert it into our data warehouse (SQL Server). So I'm just trying to do a union of multiple same structured tables in different databases. Tiran's solution started me down that same path but the linking of Progress databases was a cumbersome process that required me to find a Progress DBA with 2-3 days free time (his quote) to put this together. When I spoke with people at Progress directly, they also pointed out that if I created a view with a union on the Progress side, it would sequentially extract the data from each source in the view, not simultaneously. However, this led me to another discovery that looks like it's going to solve our needs and totally skips dealing with linking tables on the Progress side.
Here's an example with three sources, same tables (this should work for cross source joined different tables as well). All names here are provided just for clarity in the examples.
Source 1 - Table_A
Source 2 - Table_A
Source 3 - Table_A
Create an ODBC connection to Source 1 named source1.
Create an ODBC connection to Source 2 named source2.
Create an ODBC connection to Source 3 named source3.
(Note, you typically want to be sure to set the connection setting to Read Uncommitted).
In SQL Server, create linked Server connections to each Source.
ls_source1
ls_source2
ls_source3
In your SQL Server database that you need to reference the Progress databases in, create a view joining the three different linked server connections together using a union. The linked server references will each need to use openquery. This example using select * from each linked server source presumes that all columns are named and structured the same from each source.
CREATE VIEW table_name_v as
SELECT *
FROM
(SELECT *
FROM OPENQUERY(ls_source1,
'select *
from source1.dbo.Table_A
')
union
SELECT *
FROM OPENQUERY(ls_source2,
'select *
from source2.dbo.Table_A
'
union
SELECT *
FROM OPENQUERY(ls_source3,
'select *
from source3.dbo.Table_A
'
)
) x
With the view created, you can now query all three tables in different Progress sources at the same time. No extra set up on the Progress side is necessary.
There is an important caveat that I'm currently working on a work-around for. If you are on a 64bit machine using 64bit SQL Server, you need to use a 64bit driver to connect to the Progress database with the linked server option. My needs require I have both the 32bit and 64bit drivers on the same machine and have run into issues with that as apparently they don't play nice together when on the same machine. I have been able to install both 64bit and 32bit drivers on the same machine (there was a glitch in Progress' website that was supposed to send me a link for that driver but I was able to get someone there to direct me to the correct place to retrieve the 64bit odbc driver. The average person should not need both drivers and can just use the 64bit. As an alternate work around, if I'm not able to get both drivers co-existing on the same machine, I've found and confirmed that the company Connx provides a driver that provides a 64bit/32bit bridge that resolves that issue for me. Ideally though, no third party software will be necessary.
A new issue has cropped up now unfortunately, as I the linked servers that I set up and were using are no longer functioning properly. Two steps forward, one step back....
Just thought I'd share my findings as I'm sure there are others looking.
Short Answer: You can't JOIN tables between two connections.
Scenarios: (all of them in one single connection)
By default, in most databases, you can Join tables in different schemas by prefixing the schema name before the table, like this:
(...) FROM defaultDB.TableA INNER JOIN extensionDB.TableA ON ({Condition}) (...)
Depending on your Database (I don't know about Progress DB deeply), you may not be able to join Tables that belongs to schemas in different servers.
Joining tables in different databases (Eg: Progress x MySQL) it's even more complicated. I've heard about Oracle Gateway, a proprietary solution that (not really sure) could achieve this last scenario.
In summary:
If your situation does not fit the first scenario (which points to the most obvious approach), I guess the shortest solution would be profiling your code and optimize possible performance bottlenecks. Adapting your code for parallel processing could be a bolder improvement.
In MySQL, I have two different databases -- let's call them A and B.
Database A resides on server server1, while database B resides on server server2.
Both servers {A, B} are physically close to each other, but are on different machines and have different connection parameters (different username, different password etc).
In such a case, is it possible to perform a connection between a table that is in database A, to a table that is in database B of different servers?
If so, how do I go about it, programatically, in php? (I am using php, MySQLDB to separately interact with each one of the databases).
The only way I can think of, is by opening 2 separate connections (i.e. instantiate 2 PDO objects) with all the different parameters, use 2 queries to query all the data you need into PHP, and then work with that on PHP.
You can make two separate MySQL connections in PHP, do two queries to the two tables, and then work with the results in PHP.
Another option, since the servers are physically close, is to setup the one or both servers to replicate the needed database/tables to each other. You can look here for more on MySQL replication:
http://dev.mysql.com/doc/refman/5.6/en/replication.html