This question tries to collect information spread over questions about different languages and YAML implementations in a mostly language-agnostic manner.
Suppose I have a YAML file like this:
first:
- foo: {a: "b"}
- "bar": [1, 2, 3]
second: | # some comment
some long block scalar value
I want to load this file into an native data structure, possibly change or add some values, and dump it again. However, when I dump it, the original formatting is not preserved:
The scalars are formatted differently, e.g. "b" loses its quotation marks, the value of second is not a literal block scalar anymore, etc.
The collections are formatted differently, e.g. the mapping value of foo is written in block style instead of the given flow style, similarly the sequence value of "bar" is written in block style
The order of mapping keys (e.g. first/second) changes
The comment is gone
The indentation level differs, e.g. the items in first are not indented anymore.
How can I preserve the formatting of the original file?
Preface: Throughout this answer, I mention some popular YAML implementations. Those mentions are never exhaustive since I do not know all YAML implementations out there.
I will use YAML terms for data structures: Atomic text content (even numbers) is a scalar. Item sequences, known elsewhere as arrays or lists, are sequences. A collection of key-value pairs, known elsewhere as dictionary or hash, is a mapping.
If you are using Python, using ruamel will help you preserve quite some formatting since it implements round-tripping up to native structures. However, it isn't perfect and cannot preserve all formatting.
Background
The process of loading YAML is also a process of losing information. Let's have a look at the process of loading/dumping YAML, as given in the spec:
When you are loading a YAML file, you are executing some or all of the steps in the Load direction, starting at the Presentation (Character Stream). YAML implementations usually promote their most high-level APIs, which load the YAML file all the way to Native (Data Structure). This is true for most common YAML implementations, e.g. PyYAML/ruamel, SnakeYAML, go-yaml, and Ruby's YAML module. Other implementations, such as libyaml and yaml-cpp, only provide deserialization up to the Representation (Node Graph), possibly due to restrictions of their implementation languages (loading into native data structures requires either compile-time or runtime reflection on types).
The important information for us is what is contained in those boxes. Each box mentions information which is not available anymore in the box left to it. So this means that styles and comments, according to the YAML specification, are only present in the actual YAML file content, but are discarded as soon as the YAML file is parsed. For you, this means that once you have loaded a YAML file to a native data structure, all information about how it originally looked in the input file is gone. Which means that when you dump the data, the YAML implementation chooses a representation it deems useful for your data. Some implementations let you give general hints/options, e.g. that all scalars should be quoted, but that doesn't help you restore the original formatting.
Thankfully, this diagram only describes the logical process of loading YAML; a conforming YAML implementation does not need to slavishly conform to it. Most implementations actually preserve data longer than they need to. This is true for PyYAML/ruamel, SnakeYAML, go-yaml, yaml-cpp, libyaml and others. In all these implementations, the style of scalars, sequences and mappings is remembered up until the Representation (Node Graph) level.
On the other hand, comments are discarded rather early since they do not belong to an event or node (the exceptions here is ruamel which links comments to the following event, and go-yaml which remembers comments before, at and after the line that created a node). Some YAML implementations (libyaml, SnakeYAML) provide access to a token stream which is even more low-level than the Event Tree. This token stream does contain comments, however it is only usable for doing things like syntax highlighting, since the APIs do not contain methods for consuming the token stream again.
So what to do?
Loading & Dumping
If you need to only load your YAML file and then dump it again, use one of the lower-level APIs of your implementation to only load the YAML up until the Representation (Node Graph) or Serialization (Event Tree) level. The API functions to search for are compose/parse and serialize/present respectively.
It is preferable to use the Event Tree instead of the Node Graph as some implementations already forget the original order of mapping keys (due to internally using hashmaps) when composing. This question, for example, details loading / dumping events with SnakeYAML.
Information that is already lost in the event stream of your implementation, for example comments in most implementations, is impossible to preserve. Also impossible to preserve is scalar layout, like in this example:
"1 \x2B 1"
This loads as string "1 + 1" after resolving the escape sequence. Even in the event stream, the information about the escape sequence has already been lost in all implementations I know. The event only remembers that it was a double-quoted scalar, so writing it back will result in:
"1 + 1"
Similarly, a folded block scalar (starting with >) will usually not remember where line breaks in the original input have been folded into space characters.
To sum up, loading to the Event Tree and dumping again will usually preserve:
Style: unquoted/quoted/block scalars, flow/block collections (sequences & mappings)
Order of keys in mappings
YAML tags and anchors
You will usually lose:
Information about escape sequences and line breaks in flow scalars
Indentation and non-content spacing
Comments – unless the implementation specifically supports putting them in events and/or nodes
If you use the Node Graph instead of the Event Tree, you will likely lose anchor representations (i.e. that &foo may be written out as &a later with all aliases referring to it using *a instead of *foo). You might also lose key order in mappings. Some APIs, like go-yaml, don't provide access to the Event Tree, so you have no choice but to use the Node Graph instead.
Modifying Data
If you want to modify data and still preserve what you can of the original formatting, you need to manipulate your data without loading it to a native structure. This usually means that you operate on YAML scalars, sequences and mappings, instead of strings, numbers, lists or whatever structures the target programming language provides.
You have the option to either process the Event Tree or the Node Graph (assuming your API gives you access to it). Which one is better usually depends on what you want to do:
The Event Tree is usually provided as stream of events. It may be better for large data since you do not need to load the complete data in memory; instead you inspect each event, track your position in the input structure, and place your modifications accordingly. The answer to this question shows how to append items giving a path and a value to a given YAML file with PyYAML's event API.
The Node Graph is better for highly structured data. If you use anchors and aliases, they will be resolved there but you will probably lose information about their names (as explained above). Unlike with events, where you need to track the current position yourself, the data is presented as complete graph here, and you can just descend into the relevant sections.
In any case, you need to know a bit about YAML type resolution to work with the given data correctly. When you load a YAML file into a declared native structure (typical in languages with a static type system, e.g. Java or Go), the YAML processor will map the YAML structure to the target type if that's possible. However, if no target type is given (typical in scripting languages like Python or Ruby, but also possible in Java), types are deduced from node content and style.
Since we are not working with native loading because we need to preserve formatting information, this type resolution will not be executed. However, you need to know how it works in two cases:
When you need to decide on the type of a scalar node or event, e.g. you have a scalar with content 42 and need to know whether that is a string or integer.
When you need to create a new event or node that should later be loaded as a specific type. E.g. if you create a scalar containing 42, you might want to control whether that it is loaded as integer 42 or string "42" later.
I won't discuss all the details here; in most cases, it suffices to know that if a string is encoded as a scalar but looks like something else (e.g. a number), you should use a quoted scalar.
Depending on your implementation, you may come in touch with YAML tags. Seldom used in YAML files (they look like e.g. !!str, !!map, !!int and so on), they contain type information about a node which can be used in collections with heterogeneous data. More importantly, YAML defines that all nodes without an explicit tag will be assigned one as part of type resolution. This may or may not have already happened at the Node Graph level. So in your node data, you may see a node's tag even when the original node does not have one.
Tags starting with two exclamation marks are actually shorthands, e.g. !!str is a shorthand for tag:yaml.org,2002:str. You may see either in your data, since implementations handle them quite differently.
Important for you is that when you create a node or event, you may be able and may also need to assign a tag. If you don't want the output to contain an explicit tag, use the non-specific tags ! for non-plain scalars and ? for everything else on event level. On node level, consult your implementation's documentation about whether you need to supply resolved tags. If not, same rule for the non-specific tags applies. If the documentation does not mention it (few do), try it out.
So to sum up: You modify data by loading either the Event Tree or the Node Graph, you add, delete or modify events or nodes in the data you get, and then you present the modified data as YAML again. Depending on what you want to do, it may help you to create the data you want to add to your YAML file as native structure, serialize it to YAML and then load it again as Node Graph or Event Tree. From there, you can include it in the structure of the YAML file you want to modify.
Conclusion / TL;DR
YAML has not been designed for this task. In fact, it has been defined as a serialization language, assuming that your data is authored as native data structures in some programming language and from there dumped to YAML. However, in reality, YAML is used a lot for configuration, meaning that you typically write YAML by hand and then load it into native data structures.
This contrast is the reason why it is so difficult to modify YAML files while preserving formatting: The YAML format has been designed as transient data format, to be written by one application, and then to be loaded by another (or the same) application. In that process, preserving formatting does not matter. It does, however, for data that is checked-in to version control (you want your diff to only contain the line(s) with data you actually changed), and other situations where you write your YAML by hand, because you want to keep style consistent.
There is no perfect solution for changing exactly one data item in a given YAML file and leaving everything else intact. Loading a YAML file does not give you a view of the YAML file, it gives you the content it describes. Therefore, everything that is not part of the described content – most importantly, comments and whitespace – is extremely hard to preserve.
If format preservation is important to you and you can't live with the compromises made by the suggestions in this answer, YAML is not the right tool for you.
I would like to challenge the accepted answer. Whether you can preserve comments, the order of map keys, or other features depends on the YAML parsing library that you use. For starters, the library needs to give you access to the parsed YAML as a YAML Document, which is a collection of YAML nodes. These nodes can contain metadata besides the actual key/value pairs. The kinds of metadata that your library chooses to store will determine how much of the initial YAML document you can preserve. I will not speak for all languages and all libraries, but Golang's most popular YAML parsing library, go-yaml supports parsing YAML into a YAML document tree and serializing YAML document back, and preserves:
comments
the order of keys
anchors and aliases
scalar blocks
However, it does not preserve indentation, insignificant whitespace, and some other minor things. On the plus side, it allows modifying the YAML document and there's another library,
yaml-jsonpath that simplifies browsing the YAML node tree. Example:
import (
"github.com/stretchr/testify/assert"
"gopkg.in/yaml.v3"
"testing"
)
func Test1(t *testing.T) {
var n yaml.Node
y := []byte(`# Comment
t: &t
- x: 1 # anchor
a:
b: *t # alias
b: |
cccc
dddd
`)
err := yaml.Unmarshal(y, &n)
assert.NoError(t, err)
y2, _ := yaml.Marshal(&n)
assert.Equal(t, y, y2)
}
I have been trying for a while now to import this kind of file to phpmyadmin and with external PHP infile load code, however I cant seem to get the result I would like. I'm not too sure whether I am putting the correct data within the format specific options.
Columns separated with: (space character)
Columns enclosed with: -
Columns escaped with: -
Lines terminated with: auto (\n)
Could someone propose a suggestion as what I should do? The snippet of text below is what the text file looks like (without the bullet points).
Perhaps phpmyadmin is not the way to go?
I would show you guys pictures but I don't have the reputation...yet.
If this helps, I have a link from where I got the dataset:
- https://snap.stanford.edu/data/web-Amazon.html
The layout is shown in the link, I would use Python if need be, though I don't have the experience with that language, I'm willing to use a parser which is provided in the link (wouldn't know where to start).
-product/productId- B000068VBQ
-product/title- Fisher-Price Rescue Heroes: Lava Landslide
-product/price- 8.88
-review/userId- unknown
-review/profileName- unknown
-review/helpfulness- 11/11
-review/score- 2.0
-review/time- 1042070400
-review/summary- Requires too much coordination
-review/text- I bought this software for my 5 year old. He has a couple of the other RH software games..
The task is just like the title says: implement in PHP a feature that dynamically sets the delivery cost according to given postcode for United Kingdom. It would be a part of an e-commerce application (precisely: prestashop). It should work more or less like on this page.
My question is: is there any tool I can use? The only thing I've found is a one-table "database" (basically to be downloaded in CSV). But there are 2 problems:
I'm missing relation between a small-scale region (like Aberdeen) and a big-scale region/county/country (like Scotland). I don't consider doing it manually - possibly there's something that could help.
However, as wikipedia says, Royal Mail has given some special postcodes that are not related to regions but to organisations. Therefore I can't be 100% sure that above CSV will cover all possible postcodes that a customer could submit.
Do you know any tool or webservice or whatever else that deals with such functionalities?
Why not use a postcode database from an official source, such as Ordnance Survey's CodePoint Open - ordnancesurvey.co.uk/oswebsite/products/code-point-open - which is also free, and updated four times a year.
CodePoint Open doesn't include the 'specials', because they're not geographically tide (so the valid "SAN TA1" postcode isn't included), but do you really expect to be delivering to the headquarters of the FA or any of those "own postcode" organisations? If you do need those, then you can always add them manually.
We have been developing web application in php.
We need barcode scanner to be integrated into our application.
Our application is divided into two modules, users and merchant.
When user comes and scans the card, merchant should be identified the user by barcode number. Admin will give barcode number and that is being sent to the card manufacturer and the number will be assigned to the magnetic stripe.
As i know scanner can acts as a key board, can you please tell the method to integrate barcode scanner into this php web based application?
PHP can be easily utilized for reading bar codes printed on paper documents. Connecting manual barcode reader to the computer via USB significantly extends usability of PHP (or any other web programming language) into tasks involving document and product management, like finding a book records in the database or listing all bills for a particular customer.
Following sections briefly describe process of connecting and using manual bar code reader with PHP.
The usage of bar code scanners described in this article are in the
same way applicable to any web programming language, such as ASP,
Python or Perl. This article uses only PHP since all tests have been
done with PHP applications.
What is a bar code reader (scanner)
Bar code reader is a hardware pluggable into computer that sends decoded bar code strings into computer. The trick is to know how to catch that received string. With PHP (and any other web programming language) the string will be placed into focused input HTML element in browser. Thus to catch received bar code string, following must be done:
just before reading the bar code, proper input element, such as INPUT TEXT FIELD must be focused (mouse cursor is inside of the input field).
once focused, start reading the code
when the code is recognized (bar code reader usually shortly beeps), it is send to the focused input field. By default, most of bar code readers will append extra special character to decoded bar code string called CRLF (ENTER). For example, if decoded bar code is "12345AB", then computer will receive "12345ABENTER". Appended character ENTER (or CRLF) emulates pressing the key ENTER causing instant submission of the HTML form:
<form action="search.php" method="post">
<input name="documentID" onmouseover="this.focus();" type="text">
</form>
Choosing the right bar code scanner
When choosing bar code reader, one should consider what types of bar codes will be read with it. Some bar codes allow only numbers, others will not have checksum, some bar codes are difficult to print with inkjet printers, some barcode readers have narrow reading pane and cannot read for example barcodes with length over 10 cm. Most of barcode readers support common barcodes, such as EAN8, EAN13, CODE 39, Interleaved 2/5, Code 128 etc.
For office purposes, the most suitable barcodes seem to be those supporting full range of alphanumeric characters, which might be:
code 39 - supports 0-9, uppercased A-Z, and few special characters (dash, comma, space, $, /, +, %, *)
code 128 - supports 0-9, a-z, A-Z and other extended characters
Other important things to note:
make sure all standard barcodes are supported, at least CODE39, CODE128, Interleaved25, EAN8, EAN13, PDF417, QRCODE.
use only standard USB plugin cables. RS232 interfaces are meant for industrial usage, rather than connecting to single PC.
the cable should be long enough, at least 1.5 m - the longer the better.
bar code reader plugged into computer should not require other power supply - it should power up simply by connecting to PC via USB.
if you also need to print bar code into generated PDF documents, you can use TCPDF open source library that supports most of common 2D bar codes.
Installing scanner drivers
Installing manual bar code reader requires installing drivers for your particular operating system and should be normally supplied with purchased bar code reader.
Once installed and ready, bar code reader turns on signal LED light. Reading the barcode starts with pressing button for reading.
Scanning the barcode - how does it work?
STEP 1 - Focused input field ready for receiving character stream from bar code scanner:
STEP 2 - Received barcode string from bar code scanner is immediatelly submitted for search into database, which creates nice "automated" effect:
STEP 3 - Results returned after searching the database with submitted bar code:
Conclusion
It seems, that utilization of PHP (and actually any web programming language) for scanning the bar codes has been quite overlooked so far. However, with natural support of emulated keypress (ENTER/CRLF) it is very easy to automate collecting & processing recognized bar code strings via simple HTML (GUI) fomular.
The key is to understand, that recognized bar code string is instantly sent to the focused HTML element, such as INPUT text field with appended trailing character ASCII 13 (=ENTER/CRLF, configurable option), which instantly sends input text field with populated received barcode as a HTML formular to any other script for further processing.
Reference: http://www.synet.sk/php/en/280-barcode-reader-scanner-in-php
Hope this helps you :)
You can use AJAX for that. Whenever you scan a barcode, your scanner will act as if it is a keyboard typing into your input type="text" components. With JavaScript, capture the corresponding event, and send HTTP REQUEST and process responses accordingly.
I've been using something like this. Just set up a simple HTML page with an textinput. Make sure that the textinput always has focus. When you scan a barcode with your barcode scanner you will receive the code and after that a 'enter'. Realy simple then; just capture the incoming keystrokes and when the 'enter' comes in you can use AJAX to handle your code.
If you have Bluetooth, Use twedge on windows and getblue app on android, they also have a few videos of it. It's made by TEC-IT. I've got it to work by setting the interface option to bluetooth server in TWedge and setting the output setting in getblue to Bluetooth client and selecting my computer from the Bluetooth devices list. Make sure your computer and phone is paired. Also to get the barcode as input set the action setting in TWedge to Keyboard Wedge. This will allow for you to first click the input text box on said form, then scan said product with your phone and wait a sec for the barcode number to be put into the text box. Using this method requires no php that doesn't already exist in your current form processing, just process the text box as usual and viola your phone scans bar codes, sends them to your pc via Bluetooth wirelessly, your computer inserts the barcode into whatever text field is selected in any application or website. Hope this helps.
I am developing a sports website that would be keeping a record of all tournaments in tennis, football and rugby. Now my database structure is built to hold who plays who in which tournament, so it would just be a select to display all the information. The type of workflow that I am talking about is the one that is commonly used in the sports arena where players' names are listed head to head, and the level of that match(knockout,quater final, semifinal, etc.) are also listed. I do not know the correct term for this though. I will give you an example for how it would look.
I am sure this is possible by using web technology, I am just finding it hard on where to start. Any advice or suggestions are much appreciated. Also if there are any libraries I could use for this, that would be immensely helpful.
Depending on how you want to format the information you should be able to do it in a few ways.
You could use GD like you mentioned but that may be a bit tedious once you get larger and larger brackets. (I don't have a lot of exp. with GD but I know the basics)
I have implemented a 256 person ladder or bracket using html and css. This proved to be pretty simple to do and it should be able to scale easily and be easy to make changes to.
Well on a first glance I would see the following data:
Teams
Cups (having Rounds)
Rounds (of Matches)
Matches (of Teams)
You could model that into a relational database, e.g. MySQL.
You can then create models in classes for your application, e.g. in PHP.
You can then create a Web UI to display the data you've entered into the database. You can use GD for that (if it's a need, I think HTML is not that bad for that, would do it with simple text based output first before turning everything into an image).
Maybe that's helpful. Was a bit lengthy for a comment, so I added it as an answer.