I have a series of csv i need to grab from a url, they have different namings but the same extention .csv and the same exact format.
Then convert it to json.
The following is what i use for one but how to loop for any .csv?
Also once downloaded them how to merge them? So let's say we have different dates, we should expect a merge like:
{
"date": 2/4/20,
"state": "AK",...
},
{
"date": 3/4/20,
"state": "AK"...
This is the php I am using for a single csv
header('Content-Type: application/json');
if (($handle = fopen("example.com/NAME.csv", "r")) !== FALSE) {
$csvs = [];
while(! feof($handle)) {
$csvs[] = fgetcsv($handle);
}
$datas = [];
$column_names = [];
foreach ($csvs[0] as $single_csv) {
$column_names[] = $single_csv;
}
foreach ($csvs as $key => $csv) {
if ($key === 0) {
continue;
}
foreach ($column_names as $column_key => $column_name) {
$datas[$key-1][$column_name] = $csv[$column_key];
}
}
$json = json_encode($datas, JSON_PRETTY_PRINT);
fclose($handle);
print_r($json);
}
What if we have
example.com/NAME.csv
example.com/NAME2.csv
example.com/NAME3.csv
UPDATE
This is the series of csv and its data
Since there is not a way (as I know) to just scrape a site and find all files with the same extension (as per comments said and as I could not find how to using php), I ended up creating an array of names (in my case are dates) and then loop it, then the output will be a merged json:
$arr = array("04-12-2020",
"04-13-2020",
"04-14-2020",
"04-15-2020",
"04-16-2020",
"04-17-2020",
"04-18-2020",
"04-19-2020",
"04-20-2020",
"04-21-2020",
"04-22-2020",
"04-23-2020",
"04-24-2020",
"04-25-2020",
"04-26-2020",
"04-27-2020",
"04-28-2020",
"04-29-2020",
"04-30-2020",
"05-01-2020",
"05-02-2020",
"05-03-2020",
"05-04-2020",
"05-05-2020"
);
foreach($arr as $date) {
$url = $url.$date.".csv";..
// here the rest of the code as per the question
Not sure if this is the best way but works.
The result is a mega json with all dates and merged data as a single obj
Related
I have a report table. A report can have many documents (which are csv file paths). A document has many mappings.
So I initially loop my documents for a report and load the csv data using ParseCsv
foreach ($this->report->uploadedDocuments->documents as $document) {
$csv = new ParseCsv\Csv();
$csv->limit = 1;
$csv->parse(storage_path("app/" . $document->fullName));
$this->csvData = $csv->data;
}
If I output this, I see something like the following for each document (only showing one CSV row here from one document)
[
{
"id":"1",
"field1":"Some data",
"something":"Some data",
"something":"Some data",
"something":"Some data",
"something":"Some data",
}
]
Now each document has mappings so I can also do something like this
foreach ($this->report->uploadedDocuments->documents as $document) {
//Load CSV data
foreach ($document->mappings as $column) {
Log::debug(json_encode($column));
}
}
The output for the above is something like this
{
"document_id":"6434fc74-18f7-43ff-be33-3a8b0cf1cadc",
"field1":1,
"field2":null,
"field3":null,
"created_at":"2020-10-26T10:19:36.000000Z",
"updated_at":"2020-10-26T10:19:36.000000Z"
}
So from the above, I can see that field1 is located in column (index) 1 in the csv data. I can also see that this CSV does not have a field2 or field3. So my idea is to turn the original CSV data into this
[
{
"field1":"Some data",
}
]
So this is my thinking. I first need to loop the documents for the reports, load the CSV files, and read the data
foreach ($this->analysis->uploadedDocuments->documents as $document) {
$csv = new ParseCsv\Csv();
$csv->limit = 5;
$csv->parse(storage_path("app/" . $document->fullName));
if (empty($csv->data)) {
throw new Exception('Not data found in file.');
}
foreach ($csv->data as $dataKey => &$item) {
foreach ($item as $key => $value) {
//Now I have access to the keys and values from csv
}
}
}
But then how can I cross check this against my document mappings? I obviously have to loop the mappings but where should I do this? I have something which I think could work but it is very messy, and relies on my knowing the column names e.g.
foreach ($this->report->uploadedDocuments->documents as $document) {
$csv = new ParseCsv\Csv();
$csv->limit = 5;
$csv->parse(storage_path("app/" . $document->fullName));
if (empty($csv->data)) {
throw new Exception('Not data found in file.');
}
foreach ($document->mappings as $column) {
foreach ($csv->data as $dataKey => &$item) {
foreach ($item as $key => $value) {
//match the mapping column to the csv column
if ($column->fieldOne !== NULL && $key === $column->fieldOne) {
//now I have access to the values for that column
}
}
}
}
}
Would there be a better way of achieving something like this?
Thanks
I suggest the following as I understand from your question
<?PHP
$csvData = $csv->data;
$csvData = json_encode($csvData);
$arr = json_decode($csvData, true);
$field1 = array_column($arr, "field1");
print_r($field1);
?>
This line convert object into json string
$csvData = json_encode($csvData);
This line convert json string to pure array
$arr = json_decode($csvData, true);
This line extract column value from multidimensional array
$field1 = array_column($arr, "field1");
I hope this will work.
I am using PHP and converting the JSON data into the CSV format and later on read the same CSV file for further processing.
Below is my code that converts the JSON data in to the CSV format.
function LoadDataFromFile($file)
{
$json = file_get_contents($file);
$array = json_decode($json, true);
$f = fopen('out.csv', 'w');
$firstLineKeys = false;
$keys = array();
//first collect keys
foreach($array as $line){
foreach($line as $key => $value){
if(!in_array($key, $keys))
$keys[] = $key;
}
}
$keyString = implode(",", $keys);
fwrite($f, "$keyString\n");
foreach ($array as $line)
{
$outputArray = array();
foreach($keys as $key){
if(array_key_exists($key, $line)){
$outputArray[] = str_replace(',', '', $line[$key]);;
} else {
$outputArray[] = "";
}
}
$outputString = implode(",", $outputArray);
fwrite($f, "$outputString\n");
}
fclose($f);
}
As we can see that, i am writing data into the "out.csv" file, and later on i am reading same CSV file and assign the value/ full contents of the same file line by line to $array variable, as shown below.
$array = explode("\n", file_get_contents('out.csv'));
Now, I want to directly assign the value of csv contents into the $array variable with out using the intermediate "out.csv" file.
Wondering what data structure should i use while converting JSON data to CSV format, and possible code changes required for "LoadDataFromFile" method?
If you can already convert the json into csv, then just append the output strings together assigned to a string variable instead of writing it to a file. Or am I misunderstanding what you want?
Instead of:
fwrite($f, "$outputString\n");
you can put:
$csv .= $outputString;
And finish it with:
$array = explode("\n", $csv);
I am parsing thorugh a eBay API response. I want to deliver this back to a website cleaner and easier to parse with JavaScript. I successfuly Parsed through the XML... but now turning that into JSON to resend back to the client is giving me some headaches.
NOTE: $resp is the response from eBay. It's their full length XML that is successfully parsed with the code below.
For example... $valueName could be Grade. And then I go into the next foreach loop and get the values for this. These values may be 10, 9.5, 9 etc.
Here is my PHP code.
$arrayName = array();
$arrayValue = array();
foreach($resp->aspectHistogramContainer->aspect as $name) {
$nameAspect = $name['name'];
//$arrayName["aspectName"] = $nameAspect;
foreach($name->valueHistogram as $value) {
$valueAspect = $value['valueName'];
//$arrayValue["aspectValue"] = $valueAspect;
}
//array_push($arrayName, $arrayValue);
}
echo json_encode($arrayName);
So, without me trying to create my own JSON, I am getting that I need. I echos results and it was similar to this...
NAME
----- Value
----- Value
----- Value
NAME
----- Value
NAME
etc etc
For a JSON response... Im looking for something like...
[
{
"name": "NAME",
"value": ["value", "value"]
}, {
"name": "name",
"value": ["value", "value"]
}
]
Any help and guidance would be greatly appreciated.
eBay's response is like this (there are A LOT more <aspect> and <valueHistogram>)
<getHistogramsResponse xmlns="http://www.ebay.com/marketplace/search/v1/services">
<ack>Success</ack>
<version>1.13.0</version>
<timestamp>2018-11-07T15:32:20.380Z</timestamp>
<aspectHistogramContainer>
<domainDisplayName>Baseball Cards</domainDisplayName>
<aspect name="Card Manufacturer">
<valueHistogram valueName="Ace Authentic">
<count>19</count>
</valueHistogram>
<valueHistogram valueName="American Caramel">
<count>2024</count>
</valueHistogram>
<valueHistogram valueName="APBA">
<count>10554</count>
</valueHistogram>
<valueHistogram valueName="Bazooka">
<count>8826</count>
</valueHistogram>
<valueHistogram valueName="Be A Player">
<count>17</count>
</valueHistogram>
<valueHistogram valueName="Bell Brand Dodgers">
<count>334</count>
To encode it (and assuming SimpleXML), then it's just a case of building each internal $aspect data array and then adding the values to it. I use (string) to ensure the data is not stored as a SimpleXMLElement, which can cause side effects...
$arrayName = array();
foreach($resp->aspectHistogramContainer->aspect as $name) {
$aspect = [ "name" => (string)$name['name']];
foreach($name->valueHistogram as $value) {
$aspect["value"][] = (string)$value['valueName'];
}
$arrayName[] = $aspect;
}
echo json_encode($arrayName);
with the sample XML, this gives...
[{"name":"Card Manufacturer","value":["Ace Authentic","American Caramel","APBA","Bazooka","Be A Player","Bell Brand Dodgers"]}]
Create one single array $resultArray and store values in it. By maintaining your current code structure with minimal changes, here is the updated code snippet,
$resultArray = array();
$i = 0; // Maintain Array Index value
foreach($resp->aspectHistogramContainer->aspect as $name) {
$resultArray[$i]["aspectName"] = (string)$name['name'];;
foreach($name->valueHistogram as $value) {
$resultArray[$i]["aspectValue"][] = (string)$value['valueName'];
}
$i++; // Increment array index to store next value
}
echo json_encode($resultArray);
$results = array();
// Parse the XML into a keyed array
foreach($resp->aspectHistogramContainer->aspect as $name) {
$nameAspect = (string) $name['name'];
$values = array();
foreach($name->valueHistogram as $value) {
$values[] = (string) $value['valueName'];
}
$results[$nameAspect] = $values;
}
// This keeps things simple - rewrite to the required JSON format
$outputForJSON = array();
foreach ($results as $name => $values) {
$outputForJSON[] = array(
"name" => $name,
"values" => $values
);
}
echo json_encode($outputForJSON);
I have a json file with the following syntax:
[
{
"fields": {
"service_number": "service_number",
"physical_address": "physical_address",
"account_id": "account_id",
"contact_id": "contact_id"
},
"someId": "asd23f",
"status": "Active",
"perCode": "1",
"idCode": "0987",
"nextCode": "09"
},
{
"fields": {
"service_number": "service_number",
"physical_address": "physical_address",
"account_id": "account_id",
"contact_id": "contact_id"
},
"someId": "789096",
"status": "Active",
"perCode": "1",
"idCode": "076543",
"nextCode": "09"
}
]
I would like to use a for loop in order to add something like a userID before or after the nextCode. Is there any solution to this?
So far, I tried this:
$data = json_decode($file, true);
foreach ($data as $key => $val)
{
foreach ($val as $key=>$c)
{
if (is_array($c))
continue;
$data .= $data . "user_id:$userId";
}
}
Of course it does not make the trick, any ideas please?
There are a few problems.
First, the foreach loop works with a copy of the array it's iterating, so modifying one of the items there won't change the original array.
Then, your inner foreach loop is overwriting the $key from the outer loop. That would cause problems, but it's okay, because you don't actually need the inner loop.
Finally, after you've decoded the JSON string, $data will be an array, so appending to it with .= won't work, and even if it did, you'd just be sticking something on the end of it rather than at the specific point where your loop is.
Just refer to the specific key you need and set the value there.
$data = json_decode($file, true);
foreach ($data as $key => $val)
{
$data[$key]['user_id'] = $userId;
}
$data = json_encode($data);
Another way which is slightly shorter is pass the value by reference &:
$data = json_decode($file, true);
foreach ($data as &$val) {
$val['user_id'] = $userId;
}
https://3v4l.org/ZF4Ve
This is how you can add an element to a parsed JSON and recostruct it back into a JSON:
// parse the JSON into an array
$data = json_decode($file, true);
foreach ($data as $key => $val)
{
// add the userID to each element of the main array
// it will be inserted after the last element (after nextCode)
$data[$key]['userID'] = 'some-id';
}
// if needed, parse the array back to JSON
$data = json_encode($data);
I explain to you
The following expression converts the information from string to array
$data = json_decode ($file, true);
So in your foreach you only have to add the key
$data = json_decode($file, true);
foreach ($data as $key => $val)
{
foreach ($val as $k=>$c)
{
if (is_array($c))
continue;
$data[$k]['user_id'] = $userId;
}
}
I see everybody answered for array options. I don't see why not using object, though.
I would do it like this:
$data = json_decode($file);
foreach ($data as &$field)
{
$field->userID = $userId;
}
$data = json_encode($data);
I am trying to build common function for JSON to CSV Converter. currently for different json file i have to make some changes in existing function.
current code:
function JsonToCSV($jfilename, $cfilename) {
if (($json = file_get_contents($jfilename)) == false)
die('Error reading json file...');
$data = json_decode($json, true);
$fp = fopen($cfilename, 'w');
$header = false;
foreach ($data as $row) {
if (empty($header)) {
$header = array_keys($row);
fputcsv($fp, $header);
$header = array_flip($header);
}
fputcsv($fp, array_merge($header, $row));
}
fclose($fp);
return;
}
Above code is working for below json
[
{
"Id": "1",
"Name": "Juned Ansari",
"Position": "Full Stack Developer",
"Salary": "$99999"
},
{
"Id": "2",
"Name": "Mayur Marolia",
"Position": "Data Analyst",
"Salary": "$6789000"
},
{
"Id": "3",
"Name": "Mitesh Panchal",
"Position": "Team Leader",
"Salary": "$2324540"
}
]
but the problem is if my json structure changed then i have to rewrite above function like it is not working for below json
[
{
"BILLING_SOFTWARE_API_KEY": "ABCD1234",
"DISTRIBUTOR_API_KEY": "11380",
"SALESMANS": [
{
"sm_code": 1,
"sm_name": "DEEPAK MEHTA 7044524144"
},
{
"sm_code": 2,
"sm_name": "Juned Ansari"
}
]
}
]
The problem is that JSON is unstructured, while CSV is structured.
To clear this hurdle you must first of all gather all the JSON fields in all the structure, and since the header must be written first, you need to cycle through the JSON twice.
$columns = [ ];
// This could be a foreach
// foreach($data as $row) { $columns = array_merge($columns, array_keys($row)); }
array_map(function($row) use (&$columns) {
$columns = array_unique(array_merge($columns, array_keys($row)));
}, $data);
// Now columns contain all columns in all rows of the JSON.
$fp = fopen($cfilename, 'w');
fputcsv($fp, $columns);
// Set all empty fields to some default
$template = array_fill_keys($columns, '');
foreach ($data as $row) {
fputcsv($fp, array_values(array_merge($template, $row)));
}
fclose($fp);
The above will not function out of the box for complex data (if a column has sub-information, like in your example). There you need a more complex step:
foreach ($data as $row) {
$collapsed = array_map(function($value) {
if (is_array($value)) {
return implode(', ', $value);
}
return $value;
}, $row);
fputcsv($fp, array_merge($template, $collapsed));
}
Still more complex information in the JSON is a clear indication that you're doing this wrong. Your best bet is then to re-encode the complex value as JSON and store it as is in the CSV field (use json_encode() instead of implode, above).
The Great Column Name Massacre
For those cases where you need to throw bad money after worse, you can implement what I called the Great Column Name Massacre. In its easiest form, you code
{
"address": {
"street": "Piazza Vieusseux",
"number": 2,
"locality" : {
"type": "city",
"name": "Florence"
}
}
}
as
[
"address_street" => "Piazza Vieusseux",
"address_number" => 2,
"address_locality_type" => "city",
"address_locality_name" => "Florence"
]
I'm feeling of two minds about this. Please do not take this wrong, but I'm feeling sort of like you asked me why your Smith&Wesson battery-operated hair dryer is not working, even if you put all six batteries in the drum, pointed it to your head and pulled the trigger.
And I feel like I'm telling you "Oh, there's a safety switch on the side. You need to move it from SAFE to FIRE, or it will not work".
So bearing in mind that this looks like a very bad idea, the folding function I mentioned in the comments is this (you can adapt it to your needs, see later):
function fold($arry, $prefix = '') {
$retval = [ ];
foreach ($arry as $key => $value) {
$newkey = $prefix.$key;
if (is_array($value)) {
$folded = fold($value, $newkey . '_');
foreach ($folded as $subkey => $subval) {
$retval[$subkey] = $subval;
}
} else {
$retval[$newkey] = $value;
}
}
return $retval;
}
Once each element of the array has been folded, it can be analyzed to find out the column names (you can do this while folding) and then everything proceeds like above.
Testing
The folding function works properly with the provided JSON sample, and yields
Array
(
[BILLING_SOFTWARE_API_KEY] => ABCD1234
[DISTRIBUTOR_API_KEY] => 11380
[SALESMANS_0_sm_code] => 1
[SALESMANS_0_sm_name] => DEEPAK MEHTA 7044524144
[SALESMANS_1_sm_code] => 2
[SALESMANS_1_sm_name] => Juned Ansari
)
This of course immediately raises the first problem; "DISTRIBUTOR_API_KEY" is what we would expect from {"DISTRIBUTOR": {"API": {"KEY": 11380}}}. It works, but the decoding is ambiguous.
To overcome this limitation the quickest way is to change the delimiter from '_' to something else, or encode it differently in the keys.
Be warned: there will be no end of problems with this approach. If I had the time, I flatter myself fantasizing that I might end up with a post to rival the Famous Answer.
Having not the time, I hereby decline all responsibility for the frightful damage, loss of property, loss of productivity, loss of time, dereliction of duty, alienation of spouse and family ties and dire health consequences (included, but not limited to, a painful and prolonged death) that I believe are sure to follow.
I use simple client SIDE to convert JSON/HTML/XML to CSV,EXCEL...
because it is easy to download by attaching the file to download attribute of anchor tag...
here is an example you may like ...
The JS FIDDLE
$(document).ready(function(){
$('button').click(function(){
var data = $('#txt').val();
if(data == '')
return;
JSONToCSVConvertor(data, "Vehicle Report", true);
});
});
function JSONToCSVConvertor(JSONData, ReportTitle, ShowLabel) {
//If JSONData is not an object then JSON.parse will parse the JSON string in an Object
var arrData = typeof JSONData != 'object' ? JSON.parse(JSONData) : JSONData;
var CSV = '';
//Set Report title in first row or line
CSV += ReportTitle + '\r\n\n';
//This condition will generate the Label/Header
if (ShowLabel) {
var row = "";
//This loop will extract the label from 1st index of on array
for (var index in arrData[0]) {
//Now convert each value to string and comma-seprated
row += index + ',';
}
row = row.slice(0, -1);
//append Label row with line break
CSV += row + '\r\n';
}
//1st loop is to extract each row
for (var i = 0; i < arrData.length; i++) {
var row = "";
//2nd loop will extract each column and convert it in string comma-seprated
for (var index in arrData[i]) {
row += '"' + arrData[i][index] + '",';
}
row.slice(0, row.length - 1);
//add a line break after each row
CSV += row + '\r\n';
}
if (CSV == '') {
alert("Invalid data");
return;
}
//Generate a file name
var fileName = "MyReport_";
//this will remove the blank-spaces from the title and replace it with an underscore
fileName += ReportTitle.replace(/ /g,"_");
//Initialize file format you want csv or xls
var uri = 'data:text/csv;charset=utf-8,' + escape(CSV);
// Now the little tricky part.
// you can use either>> window.open(uri);
// but this will not work in some browsers
// or you will not get the correct file extension
//this trick will generate a temp <a /> tag
var link = document.createElement("a");
link.href = uri;
//set the visibility hidden so it will not effect on your web-layout
link.style = "visibility:hidden";
link.download = fileName + ".csv";
//this part will append the anchor tag and remove it after automatic click
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
}