Plugin Scoring System for Obsidian Plugins #52

ganesshkumar · 2024-12-28T16:04:31Z

ganesshkumar
Dec 28, 2024
Maintainer

Goal

The goal of this scoring system is to provide a fair and consistent method to evaluate Obsidian plugins based on the following factors:

User Engagement: Measures how users interact with and adopt a plugin.
Codebase Activity: Reflects the level of ongoing maintenance and development.
Popularity: Indicates widespread usage and trust among the Obsidian community.

Initial Rules for Scoring

1. Weight Distribution

Description: The sum of all weights must equal 1 to ensure a balanced and fair scoring system.

2. Plugin Age

Description: Plugins created over a year ago are assigned higher scores, as they are considered more mature and stable. Recently created plugins receive slightly lower scores.
Tracks: Codebase Activity

3. GitHub Popularity

Description: Plugins with more stargazers and forks receive higher scores. Due to limited interaction with GitHub among Obsidian users, these metrics have a combined weight of 0.05.
Tracks: User Engagement, Popularity

4. Issue Management

Description: Plugins with a high issue close ratio (closed issues / total issues) are assigned higher scores, reflecting responsiveness to user-reported problems.
Tracks: Codebase Activity

5. Pull Request (PR) Management

Description: Plugins with a high resolved PR ratio ((closed PRs + merged PRs) / total PRs) receive higher scores, indicating efficient handling of contributions and fixes.
Tracks: Codebase Activity

6. Release Activity

Description: Plugins with a higher number of releases in the last year are assigned higher scores, reflecting active development.
Tracks: Codebase Activity

7. Download Popularity

Description: Plugins with a higher number of total downloads are assigned higher scores, as this indicates widespread adoption and user trust.
Tracks: Popularity

8. Recency of Updates

Description: Plugins with a more recent release date (latestReleaseAt) receive higher scores, reflecting active maintenance and compatibility with the latest Obsidian features.
Tracks: Codebase Activity

Scoring Formula

const score =
  (normalized_stargazers * 0.03) +
  (normalized_forks * 0.02) +
  (normalized_closedIssuesRatio * 0.2) +
  (normalized_resolvedPRRatio * 0.15) +
  (normalized_commitCountInLastYear * 0.1) +
  (normalized_totalDownloads * 0.2) +
  (normalized_latestReleaseAt * 0.2) + 
  (normalized_createdAt * 0.1);

Feedback

We are excited to hear your thoughts on this scoring system! Please share feedback, suggest improvements, or discuss how we can refine the weights and rules to better serve the Obsidian community.

Let’s work together to make the plugin ecosystem even more robust and user-friendly!

ganesshkumar · 2024-12-28T16:12:21Z

ganesshkumar
Dec 28, 2024
Maintainer Author

This initial version is mostly a heuristic based approach, but I have tested against well-known plugins.

Top 10 Plugins by Score:

remotely-save - Score: 0.9820722519128287
obsidian-5e-statblocks - Score: 0.9784620503187116
omnisearch - Score: 0.9748685994771505
obsidian-tasks-plugin - Score: 0.9703741732792306
obsidian-textgenerator-plugin - Score: 0.9702995601590902
obsidian-linter - Score: 0.9694185954839998
obsidian-memos - Score: 0.9676575343478576
obsidian-meta-bind-plugin - Score: 0.9634056831731783
todoist-sync-plugin - Score: 0.9610590468028871
copilot - Score: 0.9578306878021283

Plugin score of all the plugins can be found in plugin-scores.json

0 replies

mProjectsCode · 2025-01-02T23:03:48Z

mProjectsCode
Jan 2, 2025

Author of Meta Bind and some other plugins here. I have some questions and concerns about the scoring system.

Regarding the Commits Last Year metric, I feel like a plugin with 74 commits in the last year (that is one commit every 5 days or about one feature or bug fix per week) should earn more than 1 point (with the current weight of 10%). This metric runs into the problem of depending on developer style. Some people commit every small change, some commit every feature/fix, some squash merge PRs, and others commit only once at the end of a day working on their plugin. This metric favors people who commit every 5 lines. You also have to draw a distinction between a small plugin that does one thing well, not needing any updates or bug fixes, and a big plugin that has many features and naturally requires more fixes like Dataview, Tasks, and Meta Bind. Is one necessarily worse? No. Does grading by commits imply that? Probably, to the non-knowable user.
I get what you wanted to express with the Total Downloads statistic, but this metric might hurt smaller better alternatives to big established plugins. This also goes for the Stars and Forks stats, but those have very low weights.
Last Release at suffers from the same point I made above about a small feature complete and well-working plugins needing no updates.
Created also seems to hurt newer alternative plugins. Is maturity a good thing? Yes. But should it be blindly generalized to every plugin without individually reviewing them? I am not sure.

Multiple score categories that each draw upon the metrics with different weights might be a good idea. In general, you should be careful about users blindly trusting the score when selecting a plugin, which is unfair to plugins that got an unjustified lower score due to the reasons I mentioned above. A warning on the website that these scores can be incorrect and shouldn't be taken at face value would be nice.

I can't think of any more constructive feedback at the moment, but maybe I will have some more ideas in the next few days.

Anyway, nice to see some more similar sites to my site: https://www.moritzjung.dev/obsidian-stats/

0 replies

AdamQuadmon · 2025-01-12T14:10:06Z

AdamQuadmon
Jan 12, 2025

LOL - this one scores 65:

https://www.obsidianstats.com/plugins/code-block-plugin

it is 934 days old, it has 10 commits, 32 stars, no issues

I appreciate your efforts but the results are misleading

1 reply

ganesshkumar Jan 12, 2025
Maintainer Author

Agreed. The weights are assigned based on heuristics as of now. I couldn't find any reliable data to learn the weights from.

From the comments on Reddit and Github so far, one emerging point is that people like different ways of scoring system.
I am thinking of letting people add their own scorer function and share it in the community, while continuing to improve what I have put out there!

I am planning to take the issues related to scoring in a batch so that I don't spend a lot of unused iterations! I am open for any inputs

Azeirah · 2025-04-14T00:25:02Z

Azeirah
Apr 14, 2025

It's important to note that the scores can be relatively easily gamed. You can just open a bunch of issues and close them, make a bunch of PRs and commits that add nothing.

1 reply

ganesshkumar Apr 15, 2025
Maintainer Author

Are you referring resolvedPRRatio or commitCountInLastYear?

resolvedPRRatio - since we take only the ratio, opening and closing multiple PRs and equivalent of doing nothing (given 0 existing PRs).

commitCountInLastYear - This can be games. But there are plugins with no activity - I do agree some plugins don't need updates but still most of them have security updates via dependency packages in node ecosystem. So, I didn't want to get rid of this metric. Also, I have given it a low weightage of 0.1.

Ideas to improve it are welcome!

lulunac27a · 2025-04-14T15:16:15Z

lulunac27a
Apr 14, 2025

The outliers are 0.5% for the low and high numbers by rank. Consider the formula for each metric to see if the rank is higher than 0.5%, set the score to 1 + (0.5% - rank) / 0.5%, like 1 + (0.005 - rank) / 0.005

2 replies

ganesshkumar Apr 15, 2025
Maintainer Author

I am not completely clear about what this will achieve. Is this some special form of normalization?

Also, what is the rank here?

lulunac27a Apr 16, 2025

The top 0.5%, so if the rank for each metric is higher than top 0.5% use the formula as 1 + (0.005 - rank) / 0.005 for bonus points for reaching a metric that is ranked within top 0.5%

dbarnett · 2025-09-07T13:18:18Z

dbarnett
Sep 7, 2025

Could you score Health ❤️ and Popularity 🔥 signals more independently?

Right now if I as a user see a low-ish score, the explanations are hard to find and interpret. If you need a single score for ranking, fine, but there are some unmaintained plugins out there with higher scores than healthy ones with lower popularity and that kinda sucks. If you could differentiate the popularity component more people could make better decisions.

For example, when I'm just browsing the entire list of plugins that exist for ideas, sure popularity is a nice signal of what's relevant to a lot of users. But then when I'm comparing one plugin to related alternatives, I don't care what's popular, I care what's high quality.

0 replies

dbarnett · 2025-09-07T13:31:51Z

dbarnett
Sep 7, 2025

Could you give some explanation of the heuristics?

Latest release: 17h ago → 9.0 of 15%
Downloads: 32k → 5.0 of 20%
Stars: 120 → 0.0 of 3%
Commits last year: 157.0 → 3.0 of 10%

Why?

I'm not interested to game it, but it's not at all obvious what sorts of improvements would make those scores higher.

2 replies

lulunac27a Sep 7, 2025

x.x number of max percentage is rounded to the nearest integer instead of actual rounding to the nearest tenth

dbarnett Sep 7, 2025

Right that false precision is another bizarre thing, I shortened to .0 for clarity but it has ".00" on all the contributions, and even the commit count. If 120 stars seriously contributes 0.00, not even one 10000th of the total score, then that's wild…

Plugin Scoring System for Obsidian Plugins #52

Uh oh!

Uh oh!

ganesshkumar Dec 28, 2024 Maintainer

Goal

Initial Rules for Scoring

1. Weight Distribution

2. Plugin Age

3. GitHub Popularity

4. Issue Management

5. Pull Request (PR) Management

6. Release Activity

7. Download Popularity

8. Recency of Updates

Scoring Formula

Feedback

Replies: 7 comments · 6 replies

Uh oh!

ganesshkumar Dec 28, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

ganesshkumar Jan 12, 2025 Maintainer Author

Uh oh!

Uh oh!

ganesshkumar Apr 15, 2025 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

ganesshkumar Apr 15, 2025 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ganesshkumar
Dec 28, 2024
Maintainer

Replies: 7 comments 6 replies

ganesshkumar
Dec 28, 2024
Maintainer Author

ganesshkumar Jan 12, 2025
Maintainer Author

ganesshkumar Apr 15, 2025
Maintainer Author

ganesshkumar Apr 15, 2025
Maintainer Author