Replies: 3 comments
-
|
We expect a padded input irrespective of the alignment. Is there some confusion in the documentation in this respect? We’d love to improve the documentation if some part is confusing. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it is not completely clear whether the code can read up to 63 bytes to the next multiple of 64, or read up to 64 bytes beyond the end. This is very different 8000 . To take a simplified case, I can see how a vector instruction can read 64 bytes out of a 1-byte document, but I don't understand how the code could read 64 bytes beyond the end of the document (in this case, read 65 bytes in total) without being off-by-one due to a software bug. I've looked at the code a bit, and I see great care is taken to ensure the buffer is length + 64, where the code doesn't hesitate to allocate a new buffer with the appropriate padding and copy the data. I don't understand how allocating + copying potentially hundreds of megabytes results in a net performance advantage when the alternative is to process at most 63 bytes byte-by-byte at the end? I'm loading a 100MB document that gets reallocated and copied for padding, just to avoid having to process a few bytes at the end without vector instructions? Also, the apparent scheme prevents me from using memory mapping and passing a view of the file to the parser, even though I know the underlying allocation is page-aligned and page-rounded, because that possible overshoot forces the code to copy the entire file into a new buffer. I can see how a block cipher, for example, could overshoot to the next multiple of the block size when using vector instructions, but not how it could read even a single byte into the next block, which your requirement of 64 extra bytes of padding, as opposed to padding to the next multiple of 64, seems to imply. So my question is: can the code read beyond the next multiple of 64, in other word read beyond at most 63 bytes after the end of the data? As a side note, maybe you should consider removing the padding requirement entirely, removing all internal data copies, accepting a {pv, cb} pair (a pointer and an arbitrary-length) as input, and building a zero-copy parser that operates on any arbitrary block of memory. Your code will be far simpler and ultimately faster, and not trash the heap, which may be decisive in server scenarios. This would allow passing file views and not touch the heap at all to read the data. Then, if you find a way to avoid scanning the entire file unless absolutely necessary, you'll end up with a far faster solution for the average case, where the user only needs part of the data, by letting the OS handle demand paging. |
Beta Was this translation helpful? Give feedback.
-
|
Our expectation is that you either use a padded input or use the free padding trick: https://github.com/simdjson/simdjson/blob/master/doc/performance.md#free-padding As for alternative designs, the library is a community based effort and we invite pull requests. Note that the library has been used in important production systems for many years. If you believe you can improve the design, it might be an impactful contribution. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
There is a lot of information about padding, but I'm left with a doubt: say the padding is 64 bytes.
C 8000 ase 1: The buffer is not 64-aligned, but the buffer size is a multiple of 64.
Case 2: The buffer is 64-aligned, and the buffer size is a multiple of 64.
Is there any circumstance where the parser would read beyond the indicated end of the buffer in either case?
It seems counterintuitive to always add 64 bytes. For example, if the buffer is 63 bytes, adding 1 might be sufficient and less wasteful?
Beta Was this translation helpful? Give feedback.
All reactions