Location and lifetime of ondemand strings #2087
Replies: 5 comments 14 replies
-
|
@SpeedyCraftah You may enjoy this code written by @anonrig for the Node.js runtime environment: It basically does what @jkeiser and I were discussing: it grabs raw JSON that does not need to be processed immediately and store it in a structure. It can be parsed later as JSON. As with everything @anonrig does, it is great code. It is going to be stupidly efficient compared to the obvious alternative of deserializing->serializing->deserializing. |
Beta Was this translation helpful? Give feedback.
-
|
I never knew Node.js used simdjson under the hood, at least for some things, that's interesting.
Definitely, that sounds like a great idea to implement into my algorithm. What I'm planning is for the JSON data to be converted into a more efficient structure/format so that it can be executed very efficiently many times and saved as opposed to playing around with JSON every time I want a value, so simdjson is perfect for this as it only allows a value to be consumed once anyways, with some arbitrary values being ignored instead of being parsed. But as it seems there's also room for more optimisations by grabbing raw JSON that I don't need right now and only processing it when it is required, without having to keep the ondemand parser occupied with the entire document. |
Beta Was this translation helpful? Give feedback.
-
Yes. We don't document this approach thoroughly, but it is one the key strengths of simdjson. |
Beta Was this translation helpful? Give feedback.
-
|
Just make sure that when you write the string back out to json, you write it raw: if you use a json writing library, by default it will escape the escape characters, turning |
Beta Was this translation helpful? Give feedback.
-
|
I did forget that it happens both ways 😂 thank you very much for the reminder. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah that would be a doozy, since you won't catch it with ordinary test cases, since you are expecting most input not to have escapes :) Happy to help! |
Beta Was this translation helpful? Give feedback.
-
|
On the plus side, it should be faster when writing without escaping. |
Beta Was this translation helpful? Give feedback.
-
|
I see - thank you for the clarification! I'll take this approach if I cannot grab the string without the escape sequence decoding, which I don't think I need/will make use of as I replied above. |
Beta Was this translation helpful? Give feedback.
-
|
It sounds like you want to keep the strings around from one document while you parse other documents? The parser.unescape() API lets you manage the buffer yourself when the parser's lifetime isn't working. simdjson allocates a single buffer that is large enough to include all possible strings that could be dumped into it. This is probably what you'd want to do as well. You need to hold on to the original pointer so you can free it, of course, but you'll also need a pointer to where simdjson should write the next string. uint8_t* const string_buf = malloc(ENOUGH_SPACE_TO_HOLD_ALL_THE_STRINGS);
uint8_t* string_buf_end = string_buf;Anytime you are about to get a string_view, either by casting to string_view or using string_view name = parser.unescape(json_obj["name"], string_buf_end);
string_view description = parser.unescape(json_obj["description"], string_buf_end);This will unescape name and description into string_buf, one after the other, and modifies string_buf_end to point after the description. Even if the parser is destroyed or used to parse a new document, the string_views remain valid as long as string_buf is valid and doesn't get overwritten. We could make this easier, but I think this will do what you're after if you're really looking for the least number of allocations possible. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for explanations
what if I parse a huge file like 4GB and I dont need to unescape anything, will simdjson allocate 4GB string_buf nevertheless ? |
Beta Was this translation helpful? Give feedback.
-
|
@kirillp What is your operating system? |
Beta Was this translation helpful? Give feedback.
-
|
I developing on windows, but it must also run on linux |
Beta Was this translation helpful? Give feedback.
-
If you specifically as for a raw JSON, then it points that the input ( In other words, if it is possible for us to just point at the input, then we do so. If you are asking for the string value of a JSON string, then, in general, it needs to be escaped (e.g,
You can store them directly into an |
Beta Was this translation helpful? Give feedback.
-
We deliberately do not document it, but you can build simdjson with the UTF-8 validation disabled: https://github.com/simdjson/simdjson/pull/1957/files Of course, then you are responsible to ensuring that the input is UTF-8. |
Beta Was this translation helpful? Give feedback.
-
|
You can definitively grab raw (unescaped) strings if that's what you need. We support that and it is obviously going to be faster. |
Beta Was this translation helpful? Give feedback.
-
I did have a look at
I had a look at this and added a |
Beta Was this translation helpful? Give feedback.
-
|
undefining SIMDJSON_UTF8VALIDATION will not do it as the header file is defining it if not def. Instead define it to be 0 |
Beta Was this translation helpful? Give feedback.
-
|
@lano1106 You are correct. You have to see SIMDJSON_UTF8VALIDATION to false (or 0) at the very start of the compilation cycle. Undefining it won't work. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm looking to parse a JSON object with the ondemand API, but I'm not sure when it comes to string_view's.
The documentation mentions that string_view's may either point to the source JSON string, or to an internal temporary buffer, is there a set of rules or specific operations which stores the string in one or the other?
I need the processing to have minimal memory allocations and copying, and so I'm not sure if I should copy these parsed strings to somewhere else, in my use case, the input JSON string will always outlive the next
iteratecall on another JSON string, so if string_view's were to be pointing to the input JSON string, this wouldn't be a problem, but since I may access different elements in the object and potentially even iterate another JSON string before needing the value of that string_view, I'm worried that if that string does end up in the temporary buffer before I manage to do things with the c 8000 ontents, I would need to copy it somewhere else immediately on reading it, which I want to avoid.Thanks!
Beta Was this translation helpful? Give feedback.
All reactions