I received post data as string as follow:
data01,data02
data11,
data21,data22
...
dataxx,dataxx
The data can be as long as 10.000 rows, column 1 is mandatory and unique, column 2 is optional.
so have sanitize the string (ensure column 1 unique (only within the input data - not through DB), format, etc.) and have convert it to array:
[0:
[col1: data01,
col2: data02],
1:
[col1: data11,
col2: ""],
2:
[col1: val11,
col2: val12],
.....
9999:
[col1: dataxx,
col2: dataxx],
]
so that array now sits at my Controller...
now, on my own opinion, it's best to send that array to mysql through Stored Proc and let SP do the bulk insert job (let me know if you have a better way, and why)
questions:
how to pass that array to Mysql?
and should I use insert select on duplicate key ignore?
I need to get a report to be presented in view, to tell user which data column 1 is duplicated (ignored)
And just a side question,
if i do:
$validator = Validator::make($request->all(), [
'array.*.column1' => 'unique:items'
]);
will it actually query to DB as many as my data quantity? (eg. 10000 times?)
I just affraid the code performance is too expensive
What my solution for this is
First define rules of validation it always run on request not on DB.
--
$rules = array("column 1" => "unique:KEY_NAME|required");
$validator = Validator::make($request->all(), $rules);
Make an array
--
$data = array(
array('k1'=> 'v1', 'k2'=> 'v2'), all data
);
Now use Eloquent::insert(); and pass data into this.
Well don't query on database. Just take array difference on final data and request data. Also add one more column which state whether is duplicate or not. So it's gonna save a good time.
Related
I have a column named - points in a table
I need to insert/store multiple values inside a points column field and display them as list items
EX: points (column):
[100% Wool, Width approx 72cm / 28 inches, Can be used as an area carpet]
output required:
100% Wool
Width approx 72cm / 28 inches
Can be used as an area carpet
My way was to do it with an array, but how?
May I know any other ways to handle it using php & MySQL.
Yes, you can store an array in a single table column. But first you need to convert the array into a string. For examle:
$dataarray = [
'title' => '100% Wool',
'properties' => [
'Width approx 72cm / 28 inches',
'Can be used as an area carpet'
]
]
$datastring = serialize($dataarray); // returns string
// save to to your database, using $datastring as a value for the 'points' column
[...]
After you read the data from your database again, simply un-serialize the stored data and you get the array back:
// load data from database
[...]
$datastring = $row['points'];
$dataarray = unserialize($datastring);
print_r($dataarray);
be aware that there are limits to the amount of data a single mysql-column can store, see https://dev.mysql.com/doc/refman/8.0/en/char.html
as an alternative to serialize() / unserialize(), you may use json_encode() / json_dencode(), which is a bit safer and uses a more universal storage format, see https://www.php.net/manual/en/function.unserialize.php
I am encoding a php array into json format which have data from a table.
My json_encode produces result with real column name of that table.I want to use the real column name in php side and after it encode to json format I will like to use some other custom name so, that if some user checks in .js file it won't be any problem for me.Below code is the result of json_encode.
What is now :-
{"result":[{"pals_id":"20","from_user":"hancy061","to_user":"hari061","username":"hancy061"}
What I want :-
{"result":[{"pid":"20","fu":"hancy061","tu":"hari061","un":"hancy061"}
Ya, there isn't any need to show user column name and it seems unsecure too.You guys can see what i want have the json_encode format which I want it to be.Is it possible from php side?I mean in php side before encoding the array into json format can we first make custom name of those columns?
You cannot safely replace these columns on the client-side, because it will be available to a user somehow. If you want a user to never learn how your columns are actually named, you should do this at the server-side.
The most common way is to use SQL aliases.
In your PHP change your SQL query to the following:
SELECT pals_id AS pid, from_user AS fu, to_user AS tu, username AS un FROM YourTable ...
However, that's a security through obscurity and doesn't provide any safety.
If you have an SQL-injection vulnerability, then a hacker will be able to query your data structure from system tables or simply SELECT *.
You could also manually set the array keys in the format you want before encoding, like:
foreach ($result as $ind => $r) {
$result[$ind] = [ // For PHP Versions < 5.4 use 'array('
"pid" => $r['pals_id'],
"fu" => $r['from_user'],
"tu" => $r['to_user'],
"un" => $r['username'],
]; // For PHP Version < 5.4 use ');'
}
However you would then have to reverse this if data were to be sent back to the server from the client for updates or something.
If that is needed, then you could set up a map to switch between the two.
I'm working with a table in which information is stored in a table in JSON format. The JSON value field looks like:
select * from k2_extra_fields where id = 2 and published = 1;
id | value
2,[{"name":"Apples","value":1,"target":null,"alias":"","required":0,"showNull":1},{"name":"Pears","value":2,"target":null,"alias":"","required":0,"showNull":1},{"name":"Mangos","value":3,"target":null,"alias":"","required":0,"showNull":1},{"name":"Guava","value":4,"target":null,"alias":"Fruit","required":0,"showNull":1},{"name":"Pineapple","value":5,"target":null,"alias":"Fruit","required":0,"showNull":1}]
Or values in a simple line by line view (minus the ID):
[
{"name":"Apples","value":1,"target":null,"alias":"","required":0,"showNull":1},
{"name":"Pears","value":2,"target":null,"alias":"","required":0,"showNull":1},
{"name":"Mangos","value":3,"target":null,"alias":"","required":0,"showNull":1},
{"name":"Guava","value":4,"target":null,"alias":"Fruit","required":0,"showNull":1},
{"name":"Pineapple","value":5,"target":null,"alias":"Fruit","required":0,"showNull":1}
]
The query that leads me here returns the value of 3. 3 = Mangos. How do I take the '3' value and match it up with the stored names/values so that I end up with the output, Mangos?
It should be possible with build in mysql functionality, but very hard and 'not clever' idea to do. If you really need to compute this problem within mysql, you would need to actually add new funtionality to your mysql. Look up on UDF plugins: http://dev.mysql.com/doc/refman/5.1/en/udf-compiling.html
I have following table fields in the database :
id, user_id, permission_id, value, created.
I have one result of an array which may contain more than 20 different values at a time. That whole contains the permission_id values and rest of the fields are similar. like user_id will be only one which will be inserted same with each permission_id and value will be always 1 and created is same as it would contain current date and time.
Now I am able to insert into database with following code:
$user_perms=$this->input->post('permissions');
foreach($user_perms as $perms) {
$userPerms['permission_id']=$perms;
$userPerms['user_id']=$uid;
$userPerms['value']=1;
$userPerms['created']=date("Y-m-d H:i:s");
$this->admins->insertPerms($userPerms);
}
Now it runs very well. But i want to make it more efficient and fast. As you may have noticed that i run that insert query in the foreach loop . So, when the user will click the submit at the back end the query may run more than 30 times at a time. which is not a good idea.
Therefore, how can i insert data without loop at once ?
You can use $this->db->insert_batch() to insert multiple rows at once:
$this->db->insert_batch(
'table_name',
array(
// first row:
array('permission_id' => 1, 'user_id' => 1, 'value' => 1, 'created' => '...'),
// second row:
array('permission_id' => 2, 'user_id' => 1, 'value' => 1, 'created' => '...')
)
);
(read more here)
However, you obviously don't avoid the foreach loop that way because you'd need to somehow create the array of data that you're passing to it ...
So, another way to optimize it is to run those inserts inside a transaction. That would (as far as SQL is concerned at least) be the equivalent of inserting them all at once in a single query, because it's COMMIT that's the most expensive operation and therefore 1 commit is faster than 20 commits.
I'm using Laravel 4, and I need to insert some rows into a MySQL table, and I need to get their inserted IDs back.
For a single row, I can use ->insertGetId(), however it has no support for multiple rows. If I could at least retrieve the ID of the first row, as plain MySQL does, would be enough to figure out the other ones.
It's mysql behavior of
last-insert-id
Important
If you insert multiple rows using a single INSERT statement, LAST_INSERT_ID() returns the value generated for the first inserted row only. The reason for this is to make it possible to reproduce easily the same INSERT statement against some other server.
u can try use many insert and take it ids or after save, try use $data->id should be the last id inserted.
If you are using INNODB, which supports transaction, then you can easily solve this problem.
There are multiple ways that you can solve this problem.
Let's say that there's a table called Users which have 2 columns id, name and table references to User model.
Solution 1
Your data looks like
$data = [['name' => 'John'], ['name' => 'Sam'], ['name' => 'Robert']]; // this will insert 3 rows
Let's say that the last id on the table was 600. You can insert multiple rows into the table like this
DB::begintransaction();
User::insert($data); // remember: $data is array of associative array. Not just a single assoc array.
$startID = DB::select('select last_insert_id() as id'); // returns an array that has only one item in it
$startID = $startID[0]->id; // This will return 601
$lastID = $startID + count($data) - 1; // this will return 603
DB::commit();
Now, you know the rows are between the range of 601 and 603
Make sure to import the DB facade at the top using this
use Illuminate\Support\Facades\DB;
Solution 2
This solution requires that you've a varchar or some sort of text field
$randomstring = Str::random(8);
$data = [['name' => "John$randomstring"], ['name' => "Sam$randomstring"]];
You get the idea here. You add that random string to a varchar or text field.
Now insert the rows like this
DB::beginTransaction();
User::insert($data);
// this will return the last inserted ids
$lastInsertedIds = User::where('name', 'like', '%' . $randomstring)
->select('id')
->get()
->pluck('id')
->toArray();
// now you can update that row to the original value that you actually wanted
User::whereIn('id', $lastInsertedIds)
->update(['name' => DB::raw("replace(name, '$randomstring', '')")]);
DB::commit();
Now you know what are the rows that were inserted.
As user Xrymz suggested, DB::raw('LAST_INSERT_ID();') returns the first.
According to Schema api insertGetId() accepts array
public int insertGetId(array $values, string $sequence = null)
So you have to be able to do
DB::table('table')->insertGetId($arrayValues);
Thats speaking, if using MySQL, you could retrive the first id by this and calculate the rest. There is also a DB::getPdo()->lastInsertId(); function, that could help.
Or if it returened the last id with some of this methods, you can calculate it back to the first inserted too.
EDIT
According to comments, my suggestions may be wrong.
Regarding the question of 'what if row is inserted by another user inbetween', it depends on the store engine. If engine with table level locking (MyISAM, MEMORY, and MERGE) is used, then the question is irrevelant, since thete cannot be two simultaneous writes to the table.
If row-level locking engine is used (InnoDB), then, another possibility might be to just insert the data, and then retrieve all the rows by some known field with whereIn() method, or figure out the table level locking.
$result = Invoice::create($data);
if ($result) {
$id = $result->id;
it worked for me
Note: Laravel version 9