8000
Skip to content

Introducing RenpyPath#6955

Draft
brunoais wants to merge 5 commits intorenpy:masterfrom
brunoais:RenpyPath
Draft

Introducing RenpyPath#6955
brunoais wants to merge 5 commits intorenpy:masterfrom
brunoais:RenpyPath

Conversation

@brunoais
Copy link
Copy Markdown
@brunoais brunoais commented Mar 13, 2026 •
edited
Loading

This is my first part in my 2-part effort to bring RenpyPath to Ren'Py.

This is about having a PathLike class, which is as compatible as viable with Path's API as I reasonably could, while adding extra features which I see as advantageous for its users.

This draft PR already has all the core concepts I want to introduce and the core changes I have planned.

This PR has all the content planned for the first part I planned.

I've been testing and using this in Ren'Py VNs I have access to. Some with a few hundred files such as Tutorial and The Question and a few between 10K-140K file range (voice taking the majority of the count), some with .py files (Python libraries installed and present), some without. Assuming I didn't make any significant copy&paste mistakes between what I used and the master branch, it should work without any known issues at the moment.

This code (except metadata) was fully written by me, however, AI was used. This code was "peer-reviewed" (within its limitations) by AI, most type annotations, unit tests and performance tests (not included) were written by AI, all with my strict supervision. Initial function/method documentation (the one in the first commit here) was some written by me, some by AI.

  • To finish this first part, I still need to write proper documentation which can be picked up by Sphinx and
  • expose RenpyPath to renpy.exports (i.e. making it available either as RenpyPath(preferred) or renpy.RenpyPath.
  • Improve this initial documentation. Make it less repetitive and easier to read.
  • Update code for python 3.14.
  • Move RenpyPath to a separate module.

For the 2nd part, which will be a separate PR and discussion, I intend to replace the majority (but not all) internal calls to renpy.list_files() to use RenpyPath instead. As part of testing if RenpyPath appears to be working right, I already did that in my testing instance of Ren'Py and couldn't find issues.
Regardless, just because I don't find bugs, doesn't mean they aren't there. I may just be testing it wrong.

Resolves #6898

@brunoais brunoais force-pushed the RenpyPath branch 4 times, most recently from 8031918 to e668a1e Compare March 17, 2026 22:09
brunoais added 5 commits March 21, 2026 14:18
@brunoais
RenpyPath: First introductory commit
715746c
@brunoais
Replace TreeEntry and previous Traversable (importer) RenpyPath with …
748f57a 
…TraversableRenpyPath
@brunoais
Simplifying importer which existed for the purposes of previous simpl…
58254f7 
…ified RenpyPath.

This revised code for Renpy importer has revealed 15-20% better performance in "The Question" VN and "Tutorial" VN vs previous code.
Tests were conducted with gc active.
@brunoais
Expose RenpyPath to default store
5126e52 
This makes RenpyPath globally available inside renpy script files
@brunoais
Documentation for RenpyPath
ecff5f5
@brunoais brunoais marked this pull request as ready for review March 21, 2026 14:21
@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 21, 2026 •
edited
Loading

@renpytom It's ready for review.
Note I was unable to place RenpyPathDirEntry in the documentation. I'd like to document it but, at the same time, not export it. I couldn't find how.

@Andykl
Copy link
Copy Markdown
Member
Andykl commented Mar 21, 2026

I do like the idea of RenpyPath. I have a few questions about the implementation:

  1. Why does it inherit PurePosixPath instead of PurePath. I don't undesrtand what are implications of that, but it just a feels wrong.
  2. Ren'Py already normalizes paths to lower-case NFC Unicode. Why do we need that flag for methods?
  3. Do we really need rpy_path, rpyc_path and build_classify? Such utility functions IMO should live outside of class as functions that takes path because it bloat class interface.
  4. Why does it have _FILE_SEPARATOR and _DIR_SEPARATOR? Doesn't it change load order for script?
  5. I think the class should be in separate file that is loaded as early as possible, and it could bring utility functions that it needs with it (import it in loader for backward compatibility).

@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 21, 2026 •
edited
Loading
  1. Why does it inherit PurePosixPath instead of PurePath. I don't undesrtand what are implications of that, but it just a feels wrong.

PurePosixPath is PurePath with Unix flavour. There is one or another optimization which checks for PurePosixPath but, frankly, the biggest optimizations are for the os.posixpath done in the builtin PurePath in the "borrowed" functionality of PurePath which shortcuts the execution if the flavour is os.posixpath.
In its essence, PurePosixPath is PurePath with flavour of os.posixpath
In Python 3.12: https://github.com/python/cpython/blob/v3.12.4/Lib/pathlib.py#L801-L808
In Python 3.15.0a7: https://github.com/python/cpython/blob/v3.15.0a7/Lib/pathlib/__init__.py#L594-L601

  1. Ren'Py already normalizes paths to lower-case NFC Unicode. Why do we need that flag for methods?

I decided, for methods which already exist in Path (directly or indirectly) to copy or extend the call signature whenever possible. My main intent is to make it compatible with as many libraries that exist for pathlib as viable. I also considered keeping enough familiarity with people already used to pathlib.
For me, that's more valuable than keeping a tight signature to the exact behavior (except for the additional methods, not present in Path, of course)

  1. Do we really need rpy_path, rpyc_path and build_classify? Such utility functions IMO should live outside of class as functions that takes path because it bloat class interface.

I don't mind that. Where do you suggest me placing them?
Also, if doing like that, I'd suggest changing build.classify() to support a RenpyPath instead of a new function.
That leaves us with the other 2 to place.

  1. Why does it have _FILE_SEPARATOR and _DIR_SEPARATOR? Doesn't it change load order for script?

I need a better clarification of what you mean related to load order.
I'll try to answer closest to what I understand, if it doesn't answer, please clarify.
_FILE_SEPARATOR and _DIR_SEPARATOR are present as a trick to force total sorting order of files and directories. It enforces strict order where, for each directory, files appear first, then directories and they are all sorted alphabetically. Methods such as _find_ranges and _iter_children rely on that behavior to function.
Using / instead would prevent that because files or directories with any character less than / would break that order. Also, files at the corresponding level would appear in between directories, instead of before directories.
Is this what you wanted to know?

  1. I think the class should be in separate file that is loaded as early as possible, and it could bring utility functions that it needs with it (import it in loader for backward compatibility).

It's not an advantage if this class is loaded earlier than this. Actually, it's only useful when used after scandirfiles() runs because it's scandirfiles() which finds the files to populate it.
It may live in a different file, however, I can't see how it should be done so it's available earlier than if it's in loader.py that, given it's useless before scandirfiles() runs.
I also don't understand what you mean with "backward compatibility". RenpyPath is being introduced here the first time.
If it's in a separate file, something must provide the result of scandirfiles() to RenpyPath or something needs to "trigger" RenpyPath's separate file to load from scandirfiles()'s result.

@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 24, 2026

@Andykl Please respond.

@Andykl
Copy link
Copy Markdown
Member
Andykl commented Mar 25, 2026 •
edited
Loading

I gave it a good thought and I will try to convey my reasoning.

You propose to add RenpyPath as public API, so any decisions you made are gonna stuck for a good amount of years. It is much easier to iterate with functions because you can leave old one with bad interface or something as is and make a new function. But with class, it is more like "should be good first try". That thought was an implicit premise in my questions.

I agree that "make it compatible with as many libraries that exist for pathlib as viable" is a good goal, so making signatues compatible is correct, but that reinforce my objection to inherit from PosixPath, because if library does isinstance(path, PurePosixPath) it would be correct from library POV to assume path represents path on real posix filesystem, which Ren'Py is not.

At the same time, I think RenpyPath should be 100% compatiable with any other Ren'Py API that currently uses str to represent paths, so case-sensitivity and unicode order should be preserved. So renpy.loadable(RenpyPath("test.txt")) and renpy.loadable(RenpyPath("TEST.TXT")) should do the same as currently with strings, and any (Ren'Py or user) function that does any sorting of str paths should sort RenpyPath the same. Because Ren'Py currently does not has an API to list directories your approach (with _DIR_SEPARATOR that looks crypitc, but I can live with it) works, but I am not sure if adding such API is a good thing, but that's a thing to Tom to decide. Another thing that I think should be given explicit thought is are RenpyAllPath("game/file.txt") and RenpyPath("file.txt") equal? (given that gamedir is "game").

So, to sum this up, I think that RenpyPath should be inherited from PurePath with a custom flavor/parser, so libraries don't make incorrect assumptions, even if that flavor is indistinguishable from posixpath. I am sure there are libraries that test for path._flavour == posixpath to see if they can do some operations.

About the class being in separate file, my reasoning is that your PR doubles the size of renpy.loader. You can do something like

renpy/renpypath.py
class RenpyPath:
    def _create_path(...): ...
    def _create_too_early(...):
        raise Exception("RenpyPath can't be instantiated before loader system ready.")
    _create = _create_too_early

renpy/loader.py
from renpy.renpypath import RenpyPath
def scandirfiles():
    ...
    RenpyPath._create = _create_path

or something like that. I don't mind if you leave it as is, that's also something that Tom may decide one way or the other.

@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 25, 2026

You propose to add RenpyPath as public API, so any decisions you made are gonna stuck for a good amount of years. It is much easier to iterate with functions because you can leave old one with bad interface or something as is and make a new function. But with class, it is more like "should be good first try". That thought was an implicit premise in my questions.

With a class, you can leave the old method that had a bad interface as is and make a new method. I don't see how a class makes that worse, at that level. I do agree that methods create an explicit inter-dependency between them. However, this RenpyPath didn't come out of nowhere as the first thing done.

  1. The one I'm producing here is my 7th iteration
  2. This one is my 2nd iteration: https://gitlab.com/brunoais/renpy-RenpyPath/
  3. I have a blueprint and behavior base to work on top of which is Python's native PurePath.

However, that point is valid for the new methods I added. As a middle ground, I can start with separate functions (and using your new module idea, they'd be in the renpypath module) and then, after they show themselves as useful and working as intended, migrate the fixed version or the intended version to become a RenpyPath method.

And on that 3rd one, I made extensive testing (which I didn't include in this PR) to ensure as-perfect behavior parity with PurePath as I could. I ran all tests python has for their PurePath made for posixpath paths and I had AI helping me select which ones to run out of all Path tests.

I agree that "make it compatible with as many libraries that exist for pathlib as viable" is a good goal, so making signatues compatible is correct, but that reinforce my objection to inherit from PosixPath, because if library does isinstance(path, PurePosixPath) it would be correct from library POV to assume path represents path on real posix filesystem, which Ren'Py is not.

I think you meant "my objection to inherit from PurePosixPath". Assuming that:
PurePaths do not represent filesystem. They are memory-level path arithmetic. For those pure string arithmetic methods, PurePosixPath and RenpyPath are interchangeable. If a program does isinstance(path, PurePosixPath) and trusts that result, the outcome is the expected, according to python's own battery of tests.

At the same time, I think RenpyPath should be 100% compatible with any other Ren'Py API that currently uses str to represent paths, so case-sensitivity and unicode order should be preserved. So renpy.loadable(RenpyPath("test.txt")) and renpy.loadable(RenpyPath("TEST.TXT")) should do the same as currently with strings, and any (Ren'Py or user) function that does any sorting of str paths should sort RenpyPath the same.

If we provide support to execute, I'll use the renpy.loadable(RenpyPath("TEST.TXT")) as example, loadable would simply get the path by filename = str(filename) and then run normally without further change. Same for open_file, same for exists. The other functions in loaderexports.py are clearly for strings and providing a RenpyPath makes no sense but we can also support it by calling str with the object instance.

If there's a need to normalize input from user into lower case and unicode folding for RenpyPath, then that's something I'm ok with and can be done at the user APIs for RenpyPath.

Because Ren'Py currently does not has an API to list directories your approach (with _DIR_SEPARATOR that looks crypitc, but I can live with it) works, but I am not sure if adding such API is a good thing, but that's a thing to Tom to decide.

I wonder if that cryptic can be mitigated with documentation. It is not API and it's only used for the internal representation. PurePath has no file order, Path explicitly claims file sort order to be implementation-dependent, RenpyPath makes no claims about sort order of its output. RenpyPath files being sorted in its internal structure (due to implementation detail) and the sorting order of output are both not API. The way list_files sorts files is not documented anywhere either, so I assume that not being API either because Ren'Py's own code that uses it doesn't rely on the fact files are lexicographically sorted.
I don't know what API you refer to in that paragraph excerpt because that comment is about something that is there but is not API.

Another thing that I think should be given explicit thought is are RenpyAllPath("game/file.txt") and RenpyPath("file.txt") equal? (given that gamedir is "game").

I believe you mean them being the same file, as in, calling .samefile(). Because equality in pathlib just means that they have the same pure content., so by PurePath's API, and mimicking their behavior, either RenpyAllPath(...) == RenpyPath(...) must always be False or they are True when they encapsulate the same text. I.e. these 2 are True RenpyAllPath("file.txt") == RenpyPath("file.txt") & RenpyAllPath("game/file.txt") == RenpyPath("game", "file.txt")

Assuming you meant .samefile(), you make a valid point. In the current implementation, a RenpyPath and a RenpyAllPath .samefile() check isn't correctly implemented and I forgot to test that. I may need to fix that before submitting. In this first iteration, I'm only exposing RenpyPath as public API, so I forgot to mix both classes together.

So, to sum this up, I think that RenpyPath should be inherited from PurePath with a custom flavor/parser, so libraries don't make incorrect assumptions, even if that flavor is indistinguishable from posixpath. I am sure there are libraries that test for path._flavour == posixpath to see if they can do some operations.

Given my explanation above, libraries can assume a RenpyPath to be a fully fledged PurePosixPath and behave as one. _flavour is not public API. But even then, it's true libraries break that. But, for everything PurePosixPath-wize, they can use the path._flavour and it will work but only for the subset of PurePath.

However, if you are still uncertain, I'm ok with making a flavour like I did with my 2nd iteration and make it extend PurePath instead.

About the class being in separate file, my reasoning is that your PR doubles the size of renpy.loader. You can do something like

renpy/renpypath.py
class RenpyPath:
    def _create_path(...): ...
    def _create_too_early(...):
        raise Exception("RenpyPath can't be instantiated before loader system ready.")
    _create = _create_too_early

renpy/loader.py
from renpy.renpypath import RenpyPath
def scandirfiles():
    ...
    RenpyPath._create = _create_path

or something like that. I don't mind if you leave it as is, that's also something that Tom may decide one way or the other.

I think we can move to a different file with RenpyPath as a module, no issues.
However, I think a guard like that is unnecessary. Code can only instantiate RenpyPath too early if it's done by someone editing and customizing Ren'Py's source code or by rpe code (archived or not). By the time python early blocks run, scandirfiles had already been called.
It's done when renpy.loader.index_files() is executed, which is right after the multiple load_rpe(path + "/" + fn) calls.

About

_create = _create_too_early

I think sphinx doesn't like that. Complains both methods are the same. That's why I did the partial workaround.

Thank you for the attention.

@Andykl
Copy link
Copy Markdown
Member
Andykl commented Mar 25, 2026

Classes are not only methods. Yes, for extra methods we can always add a new one, but the devil in the details. Things that are tricky to change after class is exposed as public are - public attributes (if any), __new__ and __init__ signatures, and especially pickle methods.

I will take a deeper look at python implementation of pathlib and your library and how it changed over time to see your iterations. It is great that you have a solid ground on the API.

I want to tell that my comments are not deal breaker for your implementation, just my input, based on working with renpy and python overall, so that we can make better decision on what end user will get.

Also I think it's important to note that I am currently working at free time on updating renpy to 3.14. I don't see any blockers for it to be ready for next renpy release, but can't be certain when it will be ready, so if I understand correctly your code will need to change after that.

@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 25, 2026

Classes are not only methods. Yes, for extra methods we can always add a new one, but the devil in the details. Things that are tricky to change after class is exposed as public are - public attributes (if any), __new__ and __init__ signatures, and especially pickle methods.

In the case of RenpyPath (and PurePath) there are no public attributes, so I'm covered there. __new__ and __init__ are just the PurePath ones, so covered there.
About pickling, I went to do a quick test and PurePath's implementation looks correct to me. At least, it seems to work as intended.

I will take a deeper look at python implementation of pathlib and your library and how it changed over time to see your iterations. It is great that you have a solid ground on the API.

I don't have a public git history of most of my iterations. But I can try to find what I have in my backups if you need it.

I want to tell that my comments are not deal breaker for your implementation, just my input, based on working with renpy and python overall, so that we can make better decision on what end user will get.

Thank you for clarifying 🙂. I agree. It's productive to discuss and make sure I didn't forget anything. I did extensive work on it but it's not rare for me to fo 99%, believing I did 100% and then someone calls me out of that 1% which I missed. Rarely it's more than a small thing to fix or some tweak but it happens more often at work than I wish it did.

Also I think it's important to note that I am currently working at free time on updating renpy to 3.14. I don't see any blockers for it to be ready for next renpy release, but can't be certain when it will be ready, so if I understand correctly your code will need to change after that.

There are optimizations I can do for 3.14. Some parts of it, such as glob and rglob can be optimized better, for certain use-cases, for example. With python 3.14, I can remove the path restrictions of .files_matching() and also use glob.translate.

Then I'll need to test again, to make sure all works as expected. But I'm not expecting any regressions.

@brunoais brunoais marked this pull request as draft March 25, 2026 20:56
@brunoais
Copy link
Copy Markdown
Author
brunoais commented Mar 25, 2026

Moved back to draft waiting for python 3.14 and will implement the optimizations I already predicted I could do for 3.13 or higher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ren'PyPath: Integrate PurePath-like behavior into renpy

2 participants

@brunoais @Andykl
0