```plotly
{
  "data": [
    {
      "x": [1, 2, 3, 4],
      "y": [10, 15, 13, 17],
      "type": "scatter"
    },
    {
      "x": [1, 2, 3, 4],
      "y": [16, 5, 11, 9],
      "type": "scatter"
    }
  ]
}
```

Which generates:

{
  "data": [
    {
      "x": [1, 2, 3, 4],
      "y": [10, 15, 13, 17],
      "type": "scatter"
    },
    {
      "x": [1, 2, 3, 4],
      "y": [16, 5, 11, 9],
      "type": "scatter"
    }
  ]
}

Also another example chart.

```plotly
{
  "data": [
    {
      "x": [1, 2, 3, 4],
      "y": [10, 15, 13, 17],
      "mode": "markers"
    },
    {
      "x": [2, 3, 4, 5],
      "y": [16, 5, 11, 9],
      "mode": "lines"
    },
    {
      "x": [1, 2, 3, 4],
      "y": [12, 9, 15, 12],
      "mode": "lines+markers"
    }
  ],
  "layout": {
    "title": {
      "text": "Line and Scatter Plot"
    }
  }
}
```

This is how it looks like:

{
  "data": [
    {
      "x": [1, 2, 3, 4],
      "y": [10, 15, 13, 17],
      "mode": "markers"
    },
    {
      "x": [2, 3, 4, 5],
      "y": [16, 5, 11, 9],
      "mode": "lines"
    },
    {
      "x": [1, 2, 3, 4],
      "y": [12, 9, 15, 12],
      "mode": "lines+markers"
    }
  ],
  "layout": {
    "title": {
      "text": "Line and Scatter Plot"
    }
  }
}

Yet as revealed in #25623 and #26658, KernelDensity consistently gave incorrect results on data with non-unit variance. In this blog post we will go through the underlying issues and the corresponding fix.

Problem Reproduction

Let us begin with a simple example to reveal the problem. Here we draw samples from the normal distribution, but with different scales (i.e., standard deviations) 1, 0.1, and 10, and perform kernel density estimation respectively.

As we can see from the plots, scikit-learn’s results were far from the underlying distribution for scale=0.1 and scale=10 while scipy gave reasonable estimations.

Problem Analysis

We will start with the univariate case. By definition, let \((x_1,\ldots,x_n)\) be independent and identially distributed samples drawn from some univariate distribution, its kernel density estimator at a given point \(x\) is

where \(K\) is the kernel function and \(h\) is the bandwidth. This is exactly how scikit-learn implemented kernel density estimation, but a good bandwidth should be chosen proportional to the standard deviation of data. If we use Scott’s or Silverman’s rule of thumb (as in the example above) for automatic bandwidth selection, it does not take into account data variance. Thus for scale=0.1, the chosen bandwidth would be too large and over-smoothens the estimation, and for scale=10, the chosen bandwidth would be too small causing the estimation to be too sensitive to noise.

Moreover in scipy’s context, the bw_method parameter (equivalent to scikit-learn’s bandwidth parameter) does not directly give the \(h\) as in the formula. It is scaled to adapt to the variance of data, or equivalently, the data is scaled to unit variance before applying the formula above, which aligns with the analysis above.

What about multivariate data? The formula is similar, such that

except that this time we have a bandwidth matrix. Also similar to the univariate case, a good bandwidth matrix should be chosen proportional to \(\Sigma^{1/2}\) where \(\Sigma\) is the covariance matrix of data. In other words, the bandwidth in each dimension should have unit variance. Otherwise, the same thing as in the univariate case will happen per dimension.

Proposed Solution

The issue is not hard to solve, but to first summarize our analysis, we want the bandwidth parameter to be properly scaled before estimation because:

• We want to respect the rules of thumb, e.g., Scott’s rule and Silverman’s rule.
• We want to be consistent with other scientific Python libraries like scipy and statsmodels to avoid confusion of users.
• We want to make the API easy-to-use. In particular, there is no clear reason to let users manually handle data variance when choosing bandwidth, especially in the multivariate case where users would even need to choose a bandwidth vector if the scaling is not performed internally.

Let \(h\) now be the bandwidth parameter (i.e., before scaling), then we can rewrite the formula into

Compared with the previous (incorrect) implementation

it suffices to (1) input \(\Sigma^{-1/2}\mathbf{x}\) instead of \(\mathbf{x}\), and (2) divide by \(\Sigma^{-1/2}\) for the output. Note that \(\Sigma^{1/2}\) can be computed from the covariance matrix (np.cov) with Cholesky decomposition (scipy.linalg.cholesky), so \(\mathbf{v}=\Sigma^{-1/2}\mathbf{x}\) can be obtained by simply solving \(\Sigma^{1/2}\cdot\mathbf{v}=\mathbf{x}\) (scipy.linalg.solve_triangular).

You may want to check out the actual fix in #27971 as well.

Future Work

• There is another potential issue #27186 in the scikit-learn tree-based implementation, which is not yet confirmed as it has not revealed any unexpected behavior in practice.

• The sample_weight support (originally implemented in #10803) in kernel density estimation is unexpectedly slow. The tree-based implementation of scikit-learn was meant to be faster than the naive implementation of scipy, but with sample_weight it is several or tens of times slower, even if we allow some tolerance.

    2024-07-18
  





  
  

  
    
    
      3 minute read

Yao Xiao recently earned his undergraduate degree in mathematics and computer science. He will be pursuing a Master’s degree in Computational Science and Engineering at Harvard SEAS. Yao joined the scikit-learn team in February 2024.Tell us about yourself.My name is Yao Xiao and I live in Shanghai, China. At the time of interview I have just got my Bachelor’s degree in Honors Mathematics and Computer Science at NYU Shanghai, and I’m going to pursue a Master’s degree in Computational Science and Engineering at Harvard SEAS. My current research interests are in networks and systems (e.g. sys4ml and ml4sys), but this may change in the future.How did you first become involved in open source and scikit-learn?In my junior year I took a course at NYU Courant called Open Source Software Development where we needed to make contributions to an open source software as our final project - and I chose scikit-learn.We would love to learn of your open source journey.I was lucky to get involved in a pretty easy meta-issue when I first started contributing to scikit-learn. I made quite a few PRs towards that issue, familiarizing myself with the coding standards, contributing workflow etc., and during which I gradually explored the codebase and learned a lot from maintainers how to write better code. After that meta-issue was completed, I decided to continue contributing since I enjoyed the experience, and I started looking through the open issues, tried reproducing and investigating them, then opened PRs for those that I was able to solve. It is the process of familiarizing with more parts of the codebase, being able to make more PRs, so on and so forth. While contributing to scikit-learn, sometimes there are also issues to solve upstream, so I also had opportunities to contribute to projects like pandas and pydata-sphinx-theme. Up till today I’m still far from familiar with the entire scikit-learn project, but I will definitely continue the amazing open-source journey.To which OSS projects and communities do you contribute?I have contributed to scikit-learn, pandas, pydata-sphinx-theme, sphinx-gallery. I’m also writing some small softwares that I decide to make open source.What do you find alluring about OSS?It is amazing to feel that my code is being used by so many people all around the world through contributing to open source projects. Well it might be inappropriate to say “my code”, but I do feel like making some actual contributions to the community instead of just writing code for myself. Also OSS makes me care about code quality and so on instead of merely making things “work”, which is very important for programmers but not really taught in school.What pain points do you observe in community-led OSS?Collaboration can lead to better code but also slows down the development process. Especially when there are not enough reviewers around, issues and PRs can easily get stale or forgotten. But I would say it’s more like a tradeoff rather than a pain point.If we discuss how far OS has evolved in 10 years, what would you like to see happen?I couldn’t say about the past 10 years since I’ve only been involved for about one and a half years, but regarding the scientific Python ecosystem I would like to see better coordination across projects (which is already happening). For instance a common interface for array libraries and dataframe libraries would allow downstream dependents to easily provide more flexible support for different input/output types, etc. And as a Chinese I would also hope that open source can thrive in my country some day as well.What are your favorite resources, books, courses, conferences, etc?As for physical books I would recommend The Pragmatic Programmer by Andy Hunt and Dave Thomas, and Refactoring: Improving the Design of Existing Code by Martin Fowler and Kent Back. As for courses I like MIT’s The Missing Semester of Your CS Education. In particular about learning Python, The Python Tutorial in the official Python documentation is good enough for me. By the way I want to mention that documentations of most languages and popular packages are very nice and they are the best place to learn the most up-to-date information.What are your hobbies, outside of work and open source?I would say my largest hobby is programming (not for school, not for work, just for fun). I’ve recently been fascinated with Tauri and wrote a lot of small desktop applications for myself in my spare time. Apart from this I also love playing the piano and I’m an anime lover, so I often listen to or play piano versions of anime theme songs (mostly arranged by Animenz). Tags:

  Open Source


  
 Categories: 


  Team

Updated: 2024-07-18

    2024-12-12
  





  
  

  
    
    
      3 minute read

Author:
Adrin Jalali

    2024-09-02
  





  
  

  
    
    
      1 minute read

Author:
Inessa Pawson

  , 


  
  
  
  

  

  


  François Goupil
  
    
  







  
  
    
    
    2024-08-06
  





  
  

  
    
    
      3 minute read

Author:
Guillaume Lemaitre ,

  Lucy Liu
 







  
  
    
    
    2024-07-24
  





  
  

  
    
    
      8 minute read

Author:
Reshama Shaikh ,

  Adam Li

Dark Theme

PyData theme has built-in support for light and dark themes as well as the theme switcher. In particular we adapted dark theme support for our customized landing page as well, while retaining the old appearance as much as possible. See #28331 for technical details.

Reworked API Pages

The API pages are reworked. In particular, the index page now contains a search-as-you-type table of all APIs powered by DataTables, and the primary sidebar now lists APIs in hierarchy. Moreover within each API page, the secondary sidebar now has links to all methods and gallery examples (if any). All these UI/UX changes are meant to provide better navigation experience. See #28428 for technical details.

Reworked Dropdowns

The dropdowns (i.e., folded sections) were redesigned based on the dropdown feature of Sphinx Design. Except for all its built-in features, we implemented permalink anchor for each dropdowns and the toggle all dropdowns functionality. The latter is because certain browsers (e.g., Firefox) does not search into collapsed dropdowns, so we provide the ability to unfold all with one click. See #28401 for technical details.

Reworked Installation Guide

The installation guide is reworked. In particular, the previous implementation for switching between instructions for different operating systems and package managers was minimal and unpolished. With the tabs feature of Sphinx Design along with some customizations, we worked out the new more native-looking and powerful installation guide. See #28336 for technical details.

Enhanced Gallery UI

Previously the badge links to JupyterLite and Binder launchers as well as the download links of Python source code and Jupyter notebooks lived at the bottom of each page, with a hint at the top of the page linking to the bottom. With the three-column layout of the PyData theme, these information are all moved to the secondary sidebar (on the right side) for better UI/UX. See #28512 for technical details of our temporary workaround and sphinx-gallery/sphinx-gallery#1312 for my better solution upstream.

Conclusion

There are many more small improvements in the website and technical details that I want to share, but due to the limited space, I encourage you to visit the scikit-learn website and explore the new theme yourself. We are still continuously making improvements to our docs, and new versions of dependencies sometimes break out customizations. Feel free to report in the scikit-learn issue tracker if you have nice ideas for improvements or if you have found anything broken. Many thanks again to the theme developers and people involved!