I've been working on a project using the LAMP server that requires frequent read of the MS-ACCESS database, stored in *.mdb file. The database has one table and about 40.000 rows in it.
I've installed mdb-tools and configured unixODBC to use it. First impression was positive:
SELECT * FROM [Table]
Worked fine, however returned a huge structure, that overflew my buffer and the tail of data was truncated.
SELECT [col1], [col2], [col3] FROM [Table]
Worked fine as well, but the amount of data was still on the edge of usability (barely didn't exceed the buffer). So I've tried to limit result with the WHERE statement, but neither
SELECT * FROM [Table] WHERE [col1]=X
nor
SELECT * FROM [Table] WHERE [col2] LIKE 'xxx'
nor
SELECT [col1], [col2], [col3] FROM [Table] WHERE [col1]=X
nor
SELECT [col1], [col2], [col3] FROM [Table] WHERE [col2] LIKE 'xxx'
isn't working. They just return empty structure.
But if the condition is all-matching - it works:
SELECT * FROM [Table] WHERE [col2] LIKE '%'
returned that huge bulk of data
I know that mdb-tools is quite an ancient solution, that provides read-only access - but that's ok for me. I just want it to work.
I've tried both versions - from sourceforge and from github.
If col1 is a text column then use ... WHERE [col1] = 'some text' (note the use of quotes) for an exact match.
... WHERE [col1] LIKE 'xxx' is equivalent to ... WHERE [col1] = 'xxx'.
When using LIKE the wildcard character % matches zero or more characters, and _ matches one character, so
... WHERE [col1] LIKE 'To%'
should match "Toronto", "Tokyo", "Toledo", etc. ...
Edit
The man page for the mdb-sql command of MDB Tools does claim to support LIKE, so I put together a test .mdb file with a table named [Clients] that contained
ID LastName FirstName Email
-- ---------- -------------- ------------------
1 Thompson Gord gord#example.com
2 Loblaw Bob bob#example.com
3 Kingsley Hank hank#example.com
4 Thompson Hunter S. hunter#example.com
I did sudo apt-get install mdbtools on my test server (Ubuntu 12.04.02), uploaded the .mdb file and did the following
gord#pingu:~$ mdb-sql -p ~/ubuTest2003.mdb
1 => SELECT ID, LastName FROM Clients
2 => go
ID LastName
1 Thompson
2 Loblaw
3 Kingsley
4 Thompson
4 Rows retrieved
1 => SELECT ID, LastName FROM Clients WHERE LastName LIKE 'Thomp%'
2 => go
ID LastName
1 Thompson
4 Thompson
2 Rows retrieved
If you are having difficulty with WHERE clauses I would suggest running a similar test (with a small sample dataset) on your system using mdb-sql to see if WHERE clauses work in that context. If they don't work, then your MDB Tools is broken. If they do work, then we'll need to investigate further.
Edit
I spent some time trying to recreate this issue in PHP on my Ubuntu 12.04 test server. I was unable to do so, only because I was unable to get the odbc_ functions in PHP to work with mdbtools at all. I could establish a "valid" connection (no errors) and I could "execute" a query (again, no errors), but I could not get any query to actually return results, not even SELECT * FROM Clients.
In searching for assistance with my problem I happened upon the Stack Overflow article here, which further discouraged me from pursuing the matter further.
Instead, since this is apparently an active Access database hosted on a Windows machine, I personally would opt for using ODBTP. It is a free (GPL) TCP/IP protocol that allows you to pass queries to a Windows machine, which then submits the query via its ODBC driver and passes the results back to you.
It does require that a service be installed and run on the Windows machine, and the PHP client component must be compiled on Linux boxes, but if those requirements aren't too daunting then it is a pretty robust solution to this type of problem. (I have used it a few times in the past.)
And while it is true that ODBTP hasn't been updated for several years, it still works: I tested it just now with my LAMP server running the following PHP script to query an .mdb file residing on my older Vista notebook:
<?php
echo '<html><body><pre>';
echo "ODBTP test:\n";
$con = odbtp_connect('192.168.1.254',
'DRIVER={Microsoft Access Driver (*.mdb)};DBQ=c:\\__tmp\\ubuTest.mdb;UID=admin;PWD=;');
if (!$con) die('_connect error: ' . odbtp_get_error($con));
$rst = odbtp_query("SELECT * FROM Clients WHERE LastName LIKE 'Thomp%'");
if (!$rst) die('_query error: ' . odbtp_get_error($con));
while ($row = odbtp_fetch_assoc($rst)) {
echo $row['ID'] . ': ' . $row['LastName'] . "\n";
}
echo '</pre></body></html>';
The results as displayed in my browser window are
ODBTP test:
1: Thompson
4: Thompson
It's been a while...
mdb-tools is CRAP.
We have ended up by writing out own Windows service listening to SQL queries on a port given and proxing them to Access by an ADO connection.
So if only you have such opportunity, I recommend you not to use dead mdb-tools.
I have a sneaky suspicion that your WHERE col data contains some special characters like single quotes or double quotes OR there is some issues with PrimaryKey.
I'm using MDBTools and I successfully select cols from tables with wildcards, BUT when i try to select from a certain col that has some double quotes in the data, i get ZERO results. When I run php blah.php on that particular col, I get the error
CRITICAL: can't find column with internal id ### in index PrimaryKey
Found that in the code on line 309, but still not past it...
https://github.com/brianb/mdbtools/blob/master/src/libmdb/index.c
Related
first time posting here.
I am facing a problem with unpredicted behavior on my PROD server and my local environment.
Here is some background on the situation:
In my application (backend Laravel 7, frontend regular html/javascript) I need to search for entries in a particular table based on JSON data stored in one of the columns:
Table: flights
columns: id, date, passengers, ... pilot_id, second_pilot_id, flight_data, updated_at, created_at
There are flights, that are directly linked to either a pilot or a second pilot via pilot_id or second_pilot_id. That is fine so far, because I can easily query them. However there are also flight entries, where no registered user is doing the entry and they are only represented by a name that is entered. This works only if the name doesn't contain special characters, in particular the german Umlaute (ö, ä, ü), also doesn't work for other specials like â or ß or é, è etc. But ONLY ON PROD, on Local everything works even with special characters.
flight_data has the data type "JSON" in my migration files.
$table->json('flight_data') ...
Now the problem:
On my local environment I can run the following and will get results returned:
... ->where(function($q) use ($r) {
$q->whereRaw("IF(payee = 2, JSON_CONTAINS(flight_data, '{\"second_pilotname\":\"$r\"}'), JSON_CONTAINS(flight_data, '{\"pilotname\":\"$r\"}'))");
})->...
This will get me my example results without issues, as expected
($r is filled a particular name of a pilot, in my example he is called "Jöhn Düe")
If I run this on my PROD system I will get no retuns. I tracked it down to the JSON_CONTAINS() function, that prevents the results. I also tried playing around with "Joehn Duee", which would be found correctly, so it basically comes down to the german Umlaute (ö, ä, ü) not being handled correctly somehow.
I also tried some SQL statements in phpmyadmin and these are the results:
LOCAL
select id, flight_data, comments, updated_at from logbook where JSON_CONTAINS(flight_data, '{"pilotname": "Juehn Duee"}')
1 result found
select id, flight_data, comments, updated_at from logbook where JSON_CONTAINS(flight_data, '{"pilotname": "Jühn Düe"}')
1 result found
PROD
select id, flight_data, comments, updated_at from logbook where JSON_CONTAINS(flight_data, '{"pilotname": "Juehn Duee"}')
1 result found
select id, flight_data, comments, updated_at from logbook where JSON_CONTAINS(flight_data, '{"pilotname": "Jühn Düe"}')
0 result found
I also checked the raw data that is stored:
PROD:
column
data
flight_data
{"pilotname":"J\u00fchn D\u00fce"}
LOCAL:
column
data
flight_data
{"pilotname":"J\u00fchn D\u00fce"}
So logically the data is transformed. Which is ok, because the data is then shown according to UTF-8 and then correctly displayed ("Jühn Düe")
The problem is, that in the backend I need to compare this data.
The differences are that on my local environment I am using MYSQL 8.0 (it's a homestead server, so select ##version; => 8.0.23-0ubuntu0.20.04.1) and on PROD (the hosted server) I am seeing "10.3.28-MariaDB-log-cll-lve"
Therefore the difference is clear, MariaDB vs. MYSQL and the handling of german Umlaute.
I tried various things around changing the conversion / charset of the entries, of the database, that all didn't solve the problem. I searched for quite a while for various similar problems, but most of them resulted in having the data stored not in UTF-8 - which I checked and is the case for me here.
Even querying for the raw data doesn't work somehow:
The following doesn't work neither on PROD nor on LOCAL:
select id, flight_data, comments, updated_at from logbook where JSON_CONTAINS(flight_data, '{"pilotname": "J\u00fchn D\u00fce"}')
0 results found
Can you help me figuring out what I am missing here?
Obviously it has to do something with the database, what else can I check or do I need to change?
Thanks a lot everybody for your help!
You should use the same software in development that you use in production. The same brand and the same version. Otherwise you risk encountering these incompatible features.
MariaDB started as a fork of the MySQL project in 2010, and both have been diverging gradually since then. MySQL implements new features, and MariaDB may or may not implement similar features, either by cherry-picking code from the MySQL project or by implementing their own original code. So over time these two projects grow more and more incompatible. At this point, over 10 years after the initial fork, you should consider MariaDB to be a different software product. Don't count on any part of it remaining compatible with MySQL.
In particular, the implementation of JSON in MariaDB versus MySQL is not entirely compatible. MariaDB creating their own original code for the JSON data type as an alias for LONGTEXT. So the internal implementation is quite different.
You asked if there's something you need to change.
Since you use MariaDB in production, not MySQL, you should use MariaDB 10.3.28 in your development environment, to ensure compatibility with the database brand and version you use in production.
I think the problem is a collation issue. Some unicode collations implement character expansions, so ue = ü would be true in the German collation.
Here's a test using MySQL 5.7 which is what I have handy (I don't use MariaDB):
mysql> select 'Juehn Duee' collate utf8mb4_unicode_520_ci = 'Jühn Düe' as same;
+------+
| same |
+------+
| 0 |
+------+
mysql> select 'Juehn Duee' collate utf8mb4_german2_ci = 'Jühn Düe' as same;
+------+
| same |
+------+
| 1 |
+------+
As you can see, this has nothing to do with JSON, but it's just related to string comparisons and which collation is used.
See the explanation in https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html in the section "_general_ci Versus _unicode_ci Collations"
Thank you all for your inputs and response!
I figured out a different solution for the problem. Maybe it helps someone..
I went a step back and checked how I am storing the data. I was using json_encode() for that, which created the table contents as shown above. By just using a raw array to save it, it was working then
$insert->pilotname = ['pilotname' => $request->pilotname];
Somehow the storing of data before was already the issue.
I want to make a fulltext search with metaphone.
Everythings works fine. I have 4 fields ie.
ID |Category | Type |Title |Meta
1 |Vehicle |4 Wheelers |Farrari Car for Sale |FHKL WLRS FRR KR FR SL
2 |Real Estate |Residential Apt|3BHK for sale |RL ESTT RSTN APT BK FR SL
3 |Music |Instruments |Piano for sale |MSK INST PN FR SL
4 |Stationary |College |Bag for $50 |STXN KLJ BK FR
5 |Services |Job |Vacancy for Jr.Web Developer |SRFS JB FKNS FR JRWB TFLP
The above is the sample data. Here I want to use metaphone and fulltext search using match()against().
Everything works fine. However Some words like Bag, Job and Car are ignored as the default minimum character 4. The issue is now that I use shared hosting and the hosting provider has told me that he cannot provide me a mysql config file nor can they change this so doing this in config file
ft_min_word_len = 2
is not an option.
//Code for generating metaphone
<?php
$string = "Vacancy for Jr.Web Developer";
$a = explode(" ", $string);
foreach ($a as $value) {
echo metaphone($value,4)."<br>";
}
?>
I am using normal
SELECT * FROM tbl_sc WHERE MATCH(META) AGAINST('$USER_SEARCH');
All the information in the database are user generated so I cannot supervise. Since I use mysql, PHP and on a shared hosting. I cannot use any elastic search library or solr like things. I have searched google and stack overflow however I am not able to get anything
One options is using LIKE operator but I want to use MATCH() AGAINST() if possible.
Kindly help me out with some work around or alternate route.
first there are three types of fulltext searches
Natural language full text search
Boolean fulltext searches
Query expansion searches
what suits your question here is the natural language full-text search, since your queries are mostly in free language and uses no special characters.the syntax goes like this
SELECT * FROM table_name WHERE MATCH(col1, col2)
AGAINST('search terms' IN NATURAL LANGUAGE MODE)
in your case first, add the fulltext functionality to your table
$stmt_txt_search = $conn->prepare("ALTER TABLE tbl_sc ADD FULLTEXT (Category, Type, Title, Meta)");
$stmt_txt_search->execute();
your query should be something like this
$stmt_match = $conn->prepare("SELECT * FROM tbl_sc WHERE MATCH (Meta) AGAINST(? IN NATURAL LANGUAGE MODE)");
$stmt_match->bind_param("s",$USER_SEARCH);
$stmt_match->execute();
to alter the ft_min_word_len you have to go the the my.cnf file, change it to the desired value, restart the server and rebuild your indexes like so
[mysqld]
set-variable = ft_min_word_len=3
then
mysql> ALTER TABLE tbl_sc DROP INDEX Title, Category...;
mysql> ALTER TABLE tbl_sc ADD FULLTEXT Title, Category...;
but since you are in shared hosting account, you cannot access the my.cnf file. However, you using SHOW VARIABLES and INFORMATION SCHEMA you can see all set variables and even change them using SET in your session such that all db connections will be based on the newly set values
for instance to SHOW VARIABLES in sql you can use
SELECT * FROM information_schema.global_variables;
this shows all existing variables in your current session, for a variable like flush time it can be set to 1 using SET flush_time = 1; so now the database will have a flushtime of 1 onwards, in your case i suppose the variables ft_max_word_len and ft_min_word_len are dynamically changeable and i would therefore suggest trying
SET ft_min_word_len = 2; within your current session, for more information see server system variables
I have looked through a TON of the mysql view examples so I am guessing what I am trying to do may not be possible.
I have multiple databases : db1, db2, db3 .......
Each have a table: quotes
Columns in identical tables: DateSubmitted DATETIME, TimeFinished DATETIME, Status VARCHAR(64)
I am trying to get a view that I would create or re-create as new databases are added that can do some calculations between the DateSubmitted and TimeFinished based on Status (Error, TimeOut, Success)
The result I am looking for would be something like this:
Database|AvgTimeLast24Hours|AvgTimeLastWeek|AvgTimeLastMonth
|db1|60|48 | 40 |
| db2 | 185 | 125 | 105
|db3 | 14 | 18 | 23 |
The average columns would be in minutes, I have the queries to get the calculations but when I try to put it into a view I know I am doing it wrong.
Does anyone have any examples?
You have a query to generate the result set you want, presumably from one of your multiple databases. Let's say it's SELECT SUM(foo) foo, baz FROM quotes ORDER BY baz . (I know it isn't, but you didn't show your query.)
Then, in that same database you can create this as a view, easily.
CREATE VIEW summary
AS
SELECT SUM(foo) foo, baz FROM quotes ORDER BY baz;
Get that view working properly.
Then you can create another view taking the union of the tables in your various databases.
CREATE VIEW all_quotes AS
SELECT * FROM db1.quotes
UNION ALL SELECT * FROM db2.quotes
UNION ALL SELECT * FROM db3.quotes
/* etc etc ad nauseam */
Then change your first view to reference all_quotes instead of quotes. Of course, this only works if all the databases are on the same server (or if you do something with remote table references, which you can look up).
You can't use variables in queries for database or table names. Those must be constant text. Raymond is correct when he says you won't be able to write a query that successfuly refers to the union of all the databases in your system. You'll have to bootstrap that operation.
write a query (using information_schema.TABLES, probably) to write your CREATE VIEW AS... query, or some other query referencing all your databases.
run that query you made.
You can create and run it either in a php program, or using MySQL's server-side flavor of prepared statements (a different flavor from the prepared statements in mysqli or PDO).
The trick is to get things working stage by stage.
Pro tip: A separate database for each customer is notoriously hard to scale up. The more successful you become, the harder it gets. That's not good.
I'm super aware of all the questions with pretty much this exact same name on here, but none of their solutions seemed to be the answer to my problems.
The query I'm using isn't very big, and I definitely have my packet size settings all configured correctly (never had a problem like this and some of my queries are properly large, much larger than the query in question).
I'm using prepared statements to pass some data to a fulltext search, and only when I seem to use it this way, I get this error. If I take the text out and paste it in as part of the query instead of preparing it, it works fine.
Also, in the MySQL log I get a huge error that starts like this
21:31:08 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
The query is this:
insert into`unpairedbillsuggestions`(`UnpairedBillSuggestionID`,
`ShippingBillID`,`InvoiceID`,`Score`,`DateTimeAdded`)
select`buuid`(),?,`InvoiceID`,`Score`,now()
from(select if(`invoices`.`InvoiceNumber`in(?,?,?),
5,0)`InvoiceNumberScore`,
if(`states`.`APO`=0,match(`shippingoptions`.`Company`,`shippingoptions`.`FullName`,`shippingoptions`.`AddressLine1`,
`shippingoptions`.`AddressLine2`)against(?),
match(`invoices`.`_APOCustomerAddress`)against(?))/
20`MatchScore`,
if(`invoices`.`_ShippingAccountNumber`=?,0.3,0)`ShippingAccountNumberScore`,-abs(datediff(date(`invoices`.`DateTimeShipped`),
ifnull(?,date(`invoices`.`DateTimeShipped`)-interval 14 day)))/7`DateScore`,
if(`invoices`.`_InvoiceTrackingNumberCount`=0,2,0)`InvoiceTrackingNumberCountScore`,`invoices`.`InvoiceID`,
`invoices`.`InvoiceNumber`,`invoices`.`DateTimeShipped`,`shippingoptions`.`FullName`,
`shippingoptions`.`Company`,`shippingoptions`.`AddressLine1`,`shippingoptions`.`AddressLine2`,
`shippingoptions`.`City`,`states`.`StateCode`,`countries`.`CountryCode3`,`shippingoptions`.`Zip`,
`invoices`.`_Total`,`invoices`.`_ShippingAccountNumber`,`companies`.`Name` `CompanyName`,`factories`.`Name` `Factory`,
`networks`.`Icon`,(select`InvoiceNumberScore`+`InvoiceTrackingNumberCountScore`+
`ShippingAccountNumberScore`+`MatchScore`+`DateScore`)`Score`
from`invoices`
left join`invoiceshippingoptions`using(`InvoiceID`)
left join`shippingoptions`
on`shippingoptions`.`ShippingOptionID`=`invoiceshippingoptions`.`ShippingOptionID`
left join`countries`on`countries`.`CountryID`=`shippingoptions`.`CountryID`
left join`states`on`states`.`StateID`=`shippingoptions`.`StateID`
join`companies`on`companies`.`CompanyID`=`invoices`.`CompanyID`
join`networks`on`networks`.`NetworkID`=`companies`.`NetworkID`
join`factories`on`factories`.`FactoryID`=`invoices`.`FactoryID`
where`invoices`.`InvoiceStatusID`<>'c9be156b-ffca-11e4-888d-3417ebdfde80'
having`Score`>0
order by`Score`desc
limit 5)`a`;
With the parameters being (all passed as strings):
'0a1c6452-4ec2-11e6-b570-12c139c58877'
'123456'
'789456123456'
''
'SOME COMPANY - SOME DUDE 117 W MASTER CHIEF LN, ORLANDO, FL 32816 USA'
'SOME COMPANY - SOME DUDE 117 W MASTER CHIEF LN, ORLANDO, FL 32816 USA'
'456789123'
'2016-04-27'
I see myself as a competent MySql-user, but now I have a basic problem with executing queries like SELECT * FROM User; and other trivial statements on my new MacBook Pro 15" with Retina Display and OS X 10.9.2.
I am trying to develop a web application using MySql 5.6 and PHP 5.5 on the server side, but almost every time I write and execute a statement I got the 1064 error code. I had attempted to do this in Coda 2 where I make my PHP scripts and MySql Workbench 6.0, but it did not work.
When I right-click on a table in Workbench and select Select Rows - Limit 1000 or let Wb generate an SQL-statement for me, everything is ok. Even if I copy the statement and modify some parts of the statement, the query gets executed and MySql does not give any error message.
I have checked my PHP document and that the database uses utf8. I did look for viable solutions for this problem in the web, but in vain.
Here is the code I tried to execute:
1 SELECT * FROM bo14g19.User LIMIT 0, 1000;
2 SELECT * FROM bo14g19.User;
3 SELECT * FROM bo14g19.User;
4 USE bo14g19
5 SELECT * FROM User
6 SELECT * FROM User
Line 1, 2, 4, and 6 is ok. Line 1 and 6 is generated by Workbench and line 2 is a direct copy from line 1 without the LIMIT. Line 3, 4, and 5 is manually typed in by hand.
Here you can find a screenshot from Workbench:
https://www.dropbox.com/s/4w6v9v3c0vainpx/mysql-workbench-execute-problems.jpg
Is this a known issue?
I found the problem! :-) My Keyboard language was set to "Sami", not "Norsk, bokmål"(Norwegian). So stupid and embarrassing, but the symbols and letters are just the same as in Norwegian, so I cant understand why it didmt work... The Norwegian and the Sami keyboard look exactly the same.