problem
I have two data tables SEQUENCES and ORGANISMS whose many-to-many-relationship is mappend in the table SOURCES. There is also a 1-m relationshipt between SOURCES and ENTRIES. I will append a detailed structure.
What i want to achieve, is the display of all sequences with all associated organisms and entries, where a condition within the sequences table is met. I have some ideas on how to achieve this, but i need the solution with the best performance, as each of these contains 50k+ entries.
idea one
Select all organisms that belong to the same sequence as a concatenated string in sql, and split it in PHP. I have no idea though, how to do the concatenation in SQL.
idea two
select same sequences with different organisms as distinct records, order by organism, and join them later in php. though this somehow feels just wrong.
idea three
use views. ANY idea on this one appreciated
structure
SEQUENCES
SEQUENCE_ID
DESCRIPTION
ORGANISMS
ORGANISM_ID
NAME
SOURCES
SOURCE_ID
SEQUENCE_ID FK to SEQUENCES.SEQUENCE_ID
ORGANISM_ID FK to ORGANISMS.ORGANISM_ID
ENTRIES
SOURCE_ID FK to SOURCES.SOURCE_ID
ENTRY_VALUE
desired outcome
array(
array(
"SEQUENCE_ID" => 4,
"DESCRIPTION" => "Some sequence",
"SOURCES" => array(
array(
"ORGANISM_ID" => 562,
"ORGANISM_NAME" => "Escherichia coli",
"ENTRIES" => array(
"some entry",
"some other entry"
),
array(
"ORGANISM_ID" => 402764,
"ORGANISM_NAME" => "Aranicola sp. EP18",
"ENTRIES" => array()
)
)
),
array(
"SEQUENCE_ID" => 5,
.....
)
)
PHP5 and FIREBIRD2.5.1
You can't fetch a nested array like that directly from a flat table structure. But if I get you right, what you want to do is not that hard to achieve.
I don't understand why you would concatenate things and then split them again, that's hard to maintain and probably slow.
I see two approaches here:
Fetch everything at once as flat table using JOIN and loop through it in PHP. This approach creates a lot of duplication but it's fast because you can fetch all data in one query and then process it with PHP.
Fetch every entity separately, loop and fetch the next hierarchy level as you go. This approach will be slower. It takes complexity away from the SQL query and doesn't fetch redunant data. It also gives you more freedom as to how you loop through your data and what you do with it.
Alternatively you might want to actually store hierarchical data in a no-sql way, where you could already store the array structure you mentioned.
Related
TL;DR:
I want to use data from a 1-dimensional array of arbitrary size, created by userinput, and fill its values into the appropriate fields of a 2-dimensional array of arbitrary size, created via query from the Database.
I have a webapplication where the user can access the DBs data both in read-mode and write-mode.
The DB records accessible to him are determined by the departments he belongs to.
The records are organized in a DB structure where a coretable contains data visible to ALL departments, while extensiontables referencing the coretable via FK contain data which are only visible to users who belong to the department associated with this extensiontable.
I'm using Lumen/Laravel and its eloquent model to interact with the DB from my Backend, some examplecode for accessing the DB looks like this:
$join = coretable::with($permittedTables)->find(1);
Here, $permittedTables is an array of tablenames, referencing the extensiontables accessible to the user.
The above code then fetches the following result:
{
"id": 1,
"Internal_key": "TESTKEY_1",
"extensiontable_itc": {
"description": "EXTENSION_iTC_1"
},
"extensiontable_sysops": {
"description": "EXTENSION_SYSOPS_1"
}
}
Now, the user will have a list-like view in the front-end where all this data has been merged into a big table. In this list, the user can click a field and change its value, then send an http-request to persist these changes to the DB.
This http-request will be an array in JSON format, which i will json_decode() in my backend, and use the transmitted id to fetch the model as seen above.
Now, at this point, two sets of data, organized in the structure of associative arrays, will face each other. The input from the http-request will likely be a 1-dimensional array, while the model from the DB will almost certainly be the multidimensional array youve seen above.
Furthermore, there is a huge amount of possible combinations of datasets. It can be the combination seen above, but it also can be a combination of data from other tables not listed here, and it can be both a bigger or smaller set of tables aggregated into the model and userinput.
Therefore, the process setting the incoming input to the DB must be able to determine dynamically which field of the input must be put into which field of the DB.
I'm doing this for the first time and I don't have a lot of ideas how to do this. The only thing that came to my mind was mirroring the DB column names to the indexes of the input's array and then loop through the model and compare its indexes to the index of the currently selected element of the input. If they match, the value from the respective field of the input will be set to the respective field of the DB.
I am aware that in order for this to work, each column of the tables affected by this process MUST have a unique name across the DB. This is doable though.
Still, this is currently the only idea I could come up with.
However, I wanted to ask you two things about this:
1) Are there other, less "hacky" approaches to solve the problem outlined above?
2) Can someone give me a custom function/a set of custom functions capable of iterating over two arrays, of which at least one will be multidimensional, while comparing their indexes, then setting the value from Array A to Array B when the indexes match?
Below I have a little codeexample, creating two arrays which reproduce the described situation, with which you can fiddle around:
First, the inputside:
$inputArray = array(
"id" => 1,
"internal_key" => "TESTKEY_1",
"CPU" => "intelTest1",
"GPU" => "nvidiaTest1",
"Soundcard" => "AsusTest1"
"MAC" => "macTest1",
"IP" => "ipTest1",
"VLAN" => "vlanTest1"
);
Then, the DB-side:
$modelArray = array(
"id" => 1,
"internal_key" => "TESTKEY_2",
"extensiontable_itc" => array (
"CPU" => "intelTest1",
"GPU" => "nvidiaTest2",
"Soundcard" => "AsusTest1"
),
"extensiontable_sysops" => array (
"MAC" => "macTest2",
"IP" => "ipTest1",
"VLAN" => "vlanTest1"
)
);
I need to make an import method that takes the CSV file and imports everything in the database.
I've done the parsing with one of Laravel's CSV addons and it works perfectly giving me a big array of values set as:
[
'col1_name' => 'col1 value',
'col2_name' => 'col2 value',
'col3_name' => 'col3 value,
'...' => '...'
]
This is also perfect since all the column names fit my model which makes the database inserts a breeze.
However - a lot of column values are strings that i'd like to set as separate tables/relations. For example, one column contains the name of the item manufacturer, and i have the manufacturer table set in my database.
My question is - what's the easy way to go through the imported CSV and swap the strings with the corresponding ID from the relationship table, making it compatible with my database design?
Something that would make the imported line:
[
'manufacturer' => 'Dell',
]
into:
[
'manufacturer' => '32',
]
I know i could just do a foreach loop comparing the needed values with values from the relationship models but I'm sure there's an easier and more clean way of doing it.
I don't think theres any "nice" way to do this - you'll need to look up each value for "manufacturer" - the question is, how many queries will you run to do so?
A consideration you need to make here is how many rows you will be importing from your CSV file.
You have a couple of options.
1) Querying 1 by 1
I'm assuming you're going to be looping through every line of the CSV file anyway, and then making a new model? In which case, you can add an extra database call in here;
$model->manufacturer_id = Manufacturer::whereName($colXValue)->first()->id;
(You'd obviously need to put in your own checks etc. here to make sure manufacturers exist)
This method is fine relatively small datsets, however, if you're importing lots and lots of rows, it might end up sluggish with alot of arguably unnecessary database calls.
2) Mapping ALL your Manufacturers
Another option would be to create a local map of all your Manufacturers before you loop through your CSV lines;
$mappedManufacturers = Manufacturer::all()->pluck('id', 'name');
This will make $mappedManufacturers an array of manufacturers that has name as a key, id as a value. This way, when you're building your model, you can do;
$model->manufacturer_id = $mappedManufacturers[$colXValue];
This method is also fine, unless you have tens of thousands of Manufacturers!
3) Where in - then re-looping
Another option would be to build up a list of manufacturer names when looping through your CSV lines, going to the database with 1 whereIn query and then re-looping through your models to populate the manufacturer ID.
So in your initial loop through your CSV, you can temporarily set a property to store the name of the manufacturer, whilst adding it to another array;
$models = collect();
$model->..... = ....;
$model->manufacturer = $colXValue;
$models->push($colXValue);
Then you'll end up with a collection of models. You then query the database for ONLY manufacturers which have appeared:
$manufacturers = Manufacturer::whereIn('name', $models->lists('manufacturer'))->get()->keyBy('name')->toArray();
This will give you array of manufacturers, keyed by their name.
You then loop through your $models collection again, assigning the correct manufacturer id using the map;
$model->manufacturer_id = $manufacturers[$model->manufacturer];
Hopefully this will give you some ideas of how you can achieve this. I'd say the solution mostly depends on your use case - if this was going to be a heavy duty ask - I'd definitely Queue it and be tempted to use Option 1! :P
I have follower and subscriber project and when a user follows other user, I should inc 1 to follower and inc 1 to subscriber's count
I use these codes to update,
this one is k which is follower's count,
$m->obarax->user->update(array("_id" => $_SESSION["u"]["_id"]),array('$inc' => array("k" => (int)1)));
this one is t which is subscriber's count,
$m->obarax->uye->update(array("_id" => new MongoId($_GET["idi"])),array('$inc' => array("t" => (int)1)));
the thing that bothers me that, I searched a lot, but I could not find a way to merge those two queries into one basic query, Is there a way so I can merge those queries ? thank you :)
You can't merge them since you're updating two different keys on two different documents. Besides that, your collections also differ, but it still not possible even if you were using the same collection because your data will be de-normalized in this case.
My question is quite simple but I can't manage to find an answer.
When I execute a query like:
$query->select('t2.name as t2_name, t1.name as t1_name')
->from('table1 t1')
->leftJoin('t1.table2 t2')
->execute(array(), Doctrine_Core::HYDRATE_ARRAY);
Doctrine returns me an array like:
array(
[0] => array(
't1_name' => 'foo',
't2_name' => 'bar'
)
)
Where i expected to get field t2_name to be set in the array before t1_name.
Is there anyway to keep the order of these selected fields in Doctrine ?
Doctrine will automatically include the primary (root) table's key field and automatically make it the first column in any query, in almost all hydration types.
Since table1 is the root table in your query, it moves that to the beginning for its own internal processing benefits.
I find this behavior annoying and somewhat unpredictable at times also, but have found great relief by creating custom hydrators.
There's a good example of creating a key/value hydrator which I have used beneficially many times in our code.
You could do something similar to rearrange the fields in the order you want.
Also I have posted an explanation to a very similar question here which may be beneficial.
Ok to make it more clear:
I am Using doctrine
I have a table Brands and Products
Brand
id
name
Product
id
name
brand_id
I have a lot of brands and Products of those brands in the database.
I would like to retrieve List of brands(+ count of its products) Grouped by Brand.name's first latter.
ex:
array(
n => array(
0 => array('Nike', 4 ),
1 => array('North Pole', 18)
.....
)
.....
)
So my question was can this be done with one query in a efficient way.
I really don't wan't to run separate queries for each brand.name's first latter.
Doctrines "Hierarchical Data" cross my mind but I believe it for different thing?.
thanks
If you are going to use this form of result more than once, it might be worthwhile to make the formatting into a Hydrator, as described here.
In your case, you can create a query that select 3 columns
first letter of brand.name
brand.name
count(product.id)
Then hydrate the result
$results = $q->execute(array(), 'group_by_first_column');
You cannot take it from database in that way, but you can fetch data as objects or arrays and then transform it to described form. Use foreach loops.
When using Doctrine you can also use raw SQL querys and hydrate arrays instead of objects. So my Solution would be to use a native SQL Query:
SELECT
brand.name,
count(product.id)
FROM
brand
JOIN
product ON
brand.id=product.brand_id
GROUP BY
brand.id ORDER BY brand.name;
And then iterate in PHP over the result to build the desired array. Because the Result is ordered by Brand Name this is quite easy. If you wasn't to keep database abstraction I think it should also be possible to express this query in DQL, just hydrate an array instead of objects.