An introduction to Pluto

November 4, 2020

This article was contributed by Lee Phillips

Pluto is a new computational notebook for the Julia programming language. Computational notebooks are a way to program inside of a web browser, storing code, annotations, and output, including graphics, in a single place. They became popular with the advent of the Jupyter notebook, which originally targeted Julia, Python, and R—the names got mashed together to make the word "Jupyter".

Pluto is similar in many ways to Jupyter, which I wrote about earlier. It uses the same basic mode of interaction that is based on input and output cells; both notebook formats are well-suited to exploration, sharing calculations, and education. In an earlier article, I reviewed progress in Julia since v. 1.0 was released two years ago. It also went into some detail about its special attractions of Julia in the area of scientific computing. Readers who are unfamiliar with Julia may want to review some of the earlier articles; here, I concentrate entirely on Pluto.

Like Julia, Pluto has an MIT license. It was created by Fons van der Plas and Mikołaj Bochenski, and has been developed on GitHub since March 2020. Despite its recent vintage, it's already mature enough for serious use; in fact, it's being used in an ongoing open MIT course. But users should keep in mind that the version number, as of this writing, is only 0.12.4; the program's behavior is certainly not set in stone. Pluto advises the user, upon startup, if a fresher version is available in the repository.

Readers who would like to try out Pluto right away and don't have Julia installed can use the Binder service to run a notebook with nothing but a web browser. When a user visits this page, it spins up a Julia instance on a server and opens a notebook interface to it that is ready for experimentation. Whether opened through Binder or running locally, the notebook offers the user an initial page with the choices to open a sample, existing, or new notebook. The series of sample notebooks offers an excellent hands-on introduction to the use of Pluto.

Why another notebook?

Jupyter has become successful because it offers a style of interaction and modes of sharing that are not available through other means. Although it is used most often with Python, the list of kernels—language back-ends that it works with—is astounding in both its size and in the obscurity of some of the languages that it includes.

Given all this, the reader can be forgiven for wondering why we need a completely new notebook project. One answer is that Jupyter presents two problems that work against its intended uses. Briefly, these are its global hidden state and its incompatibility with version control. I'll expand on these two issues below. Although they are widely seen as problems, there is, naturally, disagreement about how serious they are. This YouTube video is an entertaining demonstration of the issues that arise when using Jupyter, presented in the lion's den, the 2018 JupyterCon, by data scientist Joel Grus. Jeremy Howard responded [YouTube] at the 2020 JupyterCon, showing in detail some of the ways to overcome these problems.

At right is a screenshot from my laptop, showing Python 3 running in a Jupyter notebook; it illustrates the hidden state problem. The entire notebook is shown. Notice that a is defined in two contradictory ways, followed by a cell where we ask for the value of this variable. The response from the Python kernel is yet a third value, from an assignment that is nowhere to be found in the notebook because I deleted it.

The problem here is that the visible state of the notebook at any particular time may be inconsistent, because it depends on the order in which the cells were executed and possibly on cells that no longer exist. If the notebook is saved in an inconsistent state and shared, the recipient will open it and see calculations that may not be reproducible from the inputs as shown, or may depend on the cells being executed in some undocumented order.

But what if the notebook worked in a different way, that made these issues moot points? What if it were always consistent, and it didn't matter in what order the cells appeared?

Pluto is different

Pluto appears, at first glance, quite like Jupyter and similar notebooks, but the way it works is rather different.

In Jupyter, when a cell is executed, the result appears, and nothing else happens. Any global variables defined in the cell are available for subsequent execution of other cells. The variables can be redefined freely. The upshot is that the output of any particular cell execution depends on a hidden global state: which cells were previously executed and in what order.

Pluto, instead, analyzes the code in all of the cells and constructs a dependency graph, so that it knows the order in which the cells must be executed; this is based on which cells use which variables. Cells can be grabbed with the mouse and arranged in any order and this has no effect on the results. When the code in a cell is changed, the cell is run; all of the cells that depend on it, and only those cells, are also run, in dependency order. Therefore one is not allowed to define a global variable in more than one cell; an attempt to do so results in an error message.

The Pluto GitHub page makes this promise: "At any instant, the program state is completely described by the code you see." There is no hidden state, and nothing depends on the order in which cells were run.

Execution, as with Jupyter, can be performed with the shift-return key combination or by clicking a button. One difference from Jupyter that jumps out immediately is that results are shown above their input cells, a convention that seems counterintuitive to me. While a cell is executing, its left-hand border turns into an in-progress indicator. One can observe the status of a calculation through the dependency graph by watching as these indicators jump from cell to cell. Below each cell is printed, in unobtrusive grey type, the execution time taken when calculating that cell, which is a nice touch. The following figure illustrates these elements of the interface:

The Pluto creators describe their project as a "reactive notebook", borrowing this term from its emergence as a description of JavaScript libraries, such as React, that allow the creation of web apps where the state of the page automatically reacts to a change made by the user.

In Pluto this means that, when the user executes a change to the value of any global variable, the dependency cascade is triggered, and everything that depends on that variable is automatically updated.

An example calculation

To provide a feel for the use of Pluto notebooks, this section will go through a short but meaningful calculation. Our recent article introduced the use of the DifferentialEquations and Plots packages; here we will use these again.

To begin working with Pluto, one must start the Julia read-eval-print loop (REPL), add the Pluto package if it's not already installed, and then execute using Pluto followed by Pluto.run(port=1234), substituting a different port if required. Current versions will then automatically open a browser window pointing to localhost at this port. The machine running the Julia process and the one running the browser need not be the same; in that case an extra host argument to Pluto.run will allow it to access a remote instance.

Our calculation is the numerical solution of the differential equation for a damped, nonlinear oscillator.

The figure above shows the top of the notebook page and the first four cells. The first two perform the necessary imports. The next cell is a comment describing the equations to be solved, and the fourth cell defines ndo!(), the function that translates this math problem into a form that DifferentialEquations can digest. This function modifies its first argument, du, which holds the derivatives of x and v; the "!" sigil in the name is a Julia convention to indicate argument-modifying functions. Example 2 on this page of the package documentation explains what is being done here.

The crossed-out eye icon to the left of the third cell hides the input, which in this case is a bit of Markdown. It makes sense to keep the input of Markdown cells hidden, unless editing them, to make the notebook easier to read. Here is what this cell looks like with the input exposed:

As this shows, in addition to standard Markdown, Pluto can render LaTeX math. The Markdown can also interpolate the values of variables and calculations by using a dollar sign (e.g. $(2*var)). As with code cells, the Markdown gets automatically updated if any of the variables that it uses happen to change.

This system has three parameters that influence the nature of the solutions: k is the restoring force, or the "spring constant"; α is the nonlinearity, a second term in the restoring force; and d is the strength of the damping, or the amount of friction. The next three cells, as shown in the following figure, contain the time span for the solution (tspan), the initial conditions for the position (u0, which consists of x, and the velocity), and an array (p) holding the three parameters.

The next cell defines an iterator that goes from one to 50 in steps of one; as its name (αs) suggests, I plan to let α take on this list of values.

The final cell in this figure calculates the 50 solutions, one for each value of the nonlinearity parameter α, and plots them all together in a waterfall plot. I use the variable i to offset the plots; since it is initialized outside of the loop, but incremented in the loop, it must be declared global because all blocks in Julia create lexical scopes. The whole cell needs to be wrapped in a begin…end block because, otherwise, Pluto will complain that i is being defined twice. This is the only difference between the Pluto code and the way this would be written in an ordinary Julia file, where this block would be unnecessary.

The other functions in this cell are similar to those we used in the previous article to solve the logistic equation. The exclamation mark on the plot! command is so we can "mutate" the plot by adding a new line to it with each call. Inside the plot! function, the notation sol[1,:] is a convenience interface defined by the package that extracts the first of the two components of the solution, which in this case are x and v.

Here is the result:

I think the plot is rather interesting, as I've not seen the solution space of this equation displayed this way before. It shows how the wavelength varies with time and with the amount of nonlinearity. Julia and Pluto make it easy to play with things and experiment, which can lead to new insights.

This last figure also shows the bottom of the window, with a box for sending instant feedback to the developers, which is a nice idea. At the bottom left is a link that allows the user to opt-in to sending anonymous statistics about the use of Pluto to the developers to help them to improve the product. There is also the button for the "Live Docs" popup window, which is described below.

Integration

The reactive nature of Pluto notebooks can be taken further by using HTML "bindings". Using a simple macro, any variable can be bound to the output of any HTML input type. The following figure shows a screenshot from my laptop where I've bound three variables to three types of widgets.

The illustration on the right shows user interaction with the color picker.

In Jupyter, similar functionality can be had by importing a widget library. In Pluto, an alternative to writing HTML directly is offered by a Julia widget library. Using this library, one can, for example, replace the HTML code for a slider:

    @bind x html"<input type=range min=1 max=5>"

with:

    @bind x slider(1:5)

Those who want to go beyond standard HTML inputs can create any widget they wish using JavaScript because the output of a JavaScript function can be bound to a Julia variable.

Pluto notebooks integrate well into a more general coding environment. Code is not locked into the notebook, and code that exists outside of a notebook can be brought in. One can import (almost) any package into a notebook, just as in normal Julia code. The known exceptions are things that don't make sense in a browser-based notebook, such as REPL-specific packages, or some packages that use macros to radically redefine the language's syntax. Some parallel-processing packages also will not work, specifically those that use Distributed.

It is also possible to include local files, but a notebook that does this will no longer be portable, because those files may not exist on others' machines. So a truly portable notebook should only import from public repositories.

Pluto notebooks are stored as plain text files on disk. Jupyter's ipynb native notebook file format is also text, but there is a crucial difference. Jupyter files include not just the inputs and the code, but also whatever outputs happened to be displayed when the notebook was saved. This has three consequences. First, meaningful diffs between versions are usually impossible to get, which severely limits the usefulness of version control. Second, the person opening a shared notebook is likely to see whatever partially executed state the saved (or auto-saved) file was in before it was shared. Third, the files are not themselves legal code because of the mix of code and output data; execution can only take place through the notebook interface.

In contrast, Pluto notebooks include only the input cells; the order in which cells appear in the file is based on the dependency graph and does not follow the visual order of the cells in the notebook. This presentation order is preserved through comments in the file. Because of this design choice, Pluto notebooks can be usefully managed using version control. Notebooks can also be shared, and the recipient receives a predictable record of the code (and explanatory Markdown cells). Beyond that, notebook files are legal Julia code, so they able to be executed directly by the Julia compiler/interpreter. They can be imported or run as standalone programs, just as any file of Julia code, so they use the same .jl suffix.

This means that it's possible to freely go back and forth between editing code in an editor and in the notebook, creating files that can be imported as any other file in a project, without having to worry about stray bits of output left over in the notebook. One drawback of this design is that if, for example, a notebook contains some graphs or other output that take a long time to generate, it will need to be regenerated each time the notebook is freshly opened; and with Julia's notorious precompilation delays, this can mean waiting for several minutes before getting to work if a new package or a new version needs to be compiled. With Jupyter, in contrast, all of the output, including graphics, is embedded, so there is no delay.

Notebooks can be exported into static PDF or HTML forms with a click on the mysterious triangle-and-circle icon near the top of the page. The order in which the cells are arranged in the notebook is preserved in the exported versions of the notebook, so the visual representation is preserved, regardless of the dependency graph.

Much of Pluto's interactivity and visualization is based on Observable, an interactive notebook for JavaScript, which the Pluto authors credit for being their inspiration. Users of Observable notebooks will immediately recognize the controls for creating and manipulating cells in Pluto. Its reactivity is based on Preact, a JavaScript library that is a lightweight alternative to React. Math rendering is provided by MathJax, and code editing in cells uses CodeMirror.

Rough edges

Despite its polish, Pluto still has a few rough edges, which is not surprising in such a young project.

The current version as of this writing does not work with my daily browser, qutebrowser, but it did work for me in Chrome. This is disappointing, as some previous versions did not have this limitation. The project does claim, however, that Chrome or Firefox are needed, but it is less than ideal for programs to require specific browsers.

The print statement seems to do nothing—but the output is actually being printed in the REPL where Pluto was started. This will probably confuse some users. It certainly surprised me, and I had already read the instructions where this is explained—but I forgot. Some might find this useful, however, as a way to emit logging information and keep it out of the notebook.

A new notebook in Pluto opens in ~/.julia/pluto_notebooks/, with an arbitrary name, instead of the directory where the REPL is running. This seems like an odd behavior, but is an improvement over previous versions that opened new notebooks in /tmp, which could cause the user to lose work if they didn't notice what was going on. Neither of these facts is noted in the online guides to the program. It would seem to be a good idea to enter the desired location of the notebook in the dialog at the top before beginning work.

Certain invocations of parallel processing, such as attempts to run on a multicore graphics processor, which work fine from files or the REPL, will fail from the notebook. This is a problem that is being worked on, but for now there are some simple workarounds.

There is not a huge amount of documentation available, but what exists may be sufficient, because the use of Pluto is fairly intuitive, and the built-in example notebooks are well done.

The GitHub home page serves as an introductory tutorial; I especially recommend the short animation on the page. There is also a FAQ and a good introductory video from this year's JuliaCon.

A help window will pop up when "?" is typed in a cell, similar to the help mode in the REPL. The built-in documentation popup window can be made to persist by clicking on "Live Docs" at the bottom of the window. In this case it will continuously offer definitions for anything that it recognizes as the user types into a cell. Also, the notebook has tab completion which works well, as does the syntax-aware styling of code in cells.

Future plans

The developers are interested in being able to export notebooks that can be used in their fully interactive form without requiring the user to have Julia installed. This would seem to be an ambitious aspiration; the Jupyter developers have been talking about the equivalent for a while, but it still does not exist. However, as noted, the Binder service will host anyone's Pluto notebook, which is helpful for those in school or corporate environments where it may not be possible to install software. The developers plan to include a single button for deployment to Binder in the near future.

The concept of notebooks within notebooks is on the table, as is a modification of the notebook for teacher-oriented features like grading. There may be improved display, debugging, and logging on the way, such as the display of intermediate values during a calculation. There is also interest in making the front-end more responsive, as well as allowing user theming.

One interesting question is whether Pluto, like Jupyter, will ever allow the use of languages other than Julia. As there is nothing online mentioning this, I asked Van der Plas about it. His answer is that they have made a deliberate choice to keep Pluto Julia-only:

Julia-only allows us to do lots of exciting things.[…] I think that the traditional programming experience from the 70s - terminal input, terminal output - is something that we urgently need to leave behind us. […] to get more meaningful interactions with a language, I believe that the polyglot environment is something that you need to leave behind. While Pluto will not become a multiple language environment, we have been working on closer integration with JS, and a lot of powerful stuff is possible already. Cells can output HTML, and that HTML can contain script tags, and Pluto's frontend will execute those. We have a small API layer around vanilla JS to allow you to send values back to Julia (reactively, this is how @bind works), output to the DOM, persist DOM state between cell updates and even change Pluto's UI.

Above we made the case for Pluto by showing how it solves the hidden global state and version control incompatibility problems that occur with Jupyter (and similar notebooks). But this is a third reason for developing a new notebook project: by confining itself to one language, Pluto can focus on being the best possible interactive environment for that language, without making any compromises to allow for a variety of kernels.

Although the dominant method for programming is likely to remain in a text editor like Vim or Emacs, or an integrated development environment, computational notebooks offer an intriguing alternative. They not only offer a unique style of interactive exploration, but can be a good platform for creating documents as well. People have written books using these notebooks, and they may be a way to realize Donald Knuth's concept of literate programming. The great advantage of computational notebooks in writing about programming is that the code can be run and checked before the notebook is exported into the final HTML or PDF form; the author can guarantee that the code runs and does what the book or article says it does.

The computational notebook format began in the 1980s with the proprietary Mathematica system. It was an innovative feat of engineering, but never achieved the widespread adoption of the free, open source, and language-independent Jupyter. However the Mathematica notebook idea has been incarnated by now in several guises in addition to Jupyter. For those using Julia and interested in a notebook interface, Pluto, the latest of these options, is an obvious choice: it offers all of the features of Jupyter, but is a clear advance over the older system. It will be interesting to see how this young project evolves in the coming years.

Index entries for this article
GuestArticles	Phillips, Lee

to post comments

Notebook diffs

Posted Nov 4, 2020 0:28 UTC (Wed) by boog (subscriber, #30882) [Link] (2 responses)

The nbdime package helps in using version control with Jupyter notebooks.

https://nbdime.readthedocs.io/en/latest/

Notebook diffs

Posted Nov 4, 2020 10:23 UTC (Wed) by lobachevsky (subscriber, #121871) [Link] (1 responses)

There is also Jupytext (https://github.com/mwouts/jupytext) for having Jupyter notebooks in plain text format.

Notebook diffs

Posted Nov 4, 2020 16:55 UTC (Wed) by fghorow (subscriber, #5229) [Link]

+1 for Jupytext. It allows a style of programming in Jupyter that I've wanted for years -- where I can fluidly move between a browser based notebook and the editor/IDE of my choice for actually _working_ on the code in a reasonable environment.

Pluto sounds pretty good from that viewpoint also. However, Pluto's "single backend" (i.e. Jupyter kernel) is a dealbreaker for me.

The 'final form'

Posted Nov 4, 2020 9:22 UTC (Wed) by epa (subscriber, #39769) [Link] (3 responses)

before the notebook is exported into the final HTML or PDF form

Perhaps we need to get away from that? The final published form should be the notebook itself, since from the notebook you can generate printed text but the reverse is not true.

I would expect a dead-tree book to include a link to download the notebook, and if reading in electronic form, the best reading interface is the Pluto notebook interface itself, perhaps with a style sheet for prettier display and a read-only mode to avoid accidental changes.

The 'final form'

Posted Nov 4, 2020 10:40 UTC (Wed) by gdt (subscriber, #6284) [Link]

The motivation for PDF is the ability to read it in the future. A reference which is a DOI to a PDF is a far more likely to be found and read than a hyperlink to some ancient software's savefile. Of course, why not both?

The 'final form'

Posted Nov 4, 2020 17:56 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I think the final form should depend on who it's being prepared for. It's great to share a notebook with another data analyst, and the notebook should be included as part of any paper that's published based on it. But a lot of the work in data analysis is performed by analysis specialists for people who don't know enough about programming to make use of a notebook if they had it; most of my- admittedly limited- use has fallen into that category. People like that want a finished product like a PDF or HTML format notebook rather than instructions on how to generate that product.

To use an analogy, the notebook is like a recipe, and the PDF or HTML format is like a prepared dish. It's great that some people are good enough cooks to be able to prepare the dish themselves when presented with ingredients and a recipe, but not everyone is like that. Plenty of people aren't great cooks and just want to eat the dish. A well designed system should be able to accommodate both types.

The 'final form'

Posted Nov 5, 2020 9:56 UTC (Thu) by epa (subscriber, #39769) [Link]

I think it's just the meaning of the word 'final'. I was interpreting it that if the final form is a PDF, you publish the PDF, your work is finished, and you can now delete the notebook. (And that means that if there are any corrections or retractions, those would be published as errata to the PDF rather than as changes to the original notebook.) Whereas I would see the PDF as ephemeral, published so that people can read it easily as you say, but the final record for posterity needs to include the original notebook. Because archaeologists in the future might not have Pluto installed, it makes sense to include a PDF in the final result too, I'll grant you that.

An introduction to Pluto

Posted Nov 4, 2020 9:24 UTC (Wed) by NAR (subscriber, #1313) [Link] (3 responses)

The English word "notebook" is seriously overloaded... Every time I read about this stuff on LWN, my first thought is that this is some kind of physical device, a laptop. The "computational" prefix doesn't really help. Maybe it's not that confusing for native speakers.

An introduction to Pluto

Posted Nov 4, 2020 11:33 UTC (Wed) by roc (subscriber, #30627) [Link]

"Notebook" as the name for a ... book of notes ... long predates and is much more common than its use as "small laptop".

An introduction to Pluto

Posted Nov 5, 2020 0:56 UTC (Thu) by Paf (subscriber, #91811) [Link] (1 responses)

I think this indicates you’ve never been a working scientist (no offense is intended). This is like a *lab* notebook, and as someone who used to be a working scientist, a sort of programmable lab notebook, with calculations included, is an *incredibly* enticing concept.

An introduction to Pluto

Posted Nov 5, 2020 17:55 UTC (Thu) by jezuch (subscriber, #52988) [Link]

Well, I've never been a lab scientist and nevertheless I don't share the confusion. But I also find the concept enticing and I've been previously exposed to Stephen Wolfram's writing, so...

Here's one data point :)

An introduction to Pluto

Posted Nov 4, 2020 10:13 UTC (Wed) by spacefrogg (subscriber, #119608) [Link] (13 responses)

A lot of things don't make sense to me in this article:

The Pluto authors claim, we'd have to leave a polyglot environment behind us just to describe their support for JS and HTML next to Julia. How is that not polyglot?

The Pluto authors claim that the notebook won't have hidden state just to describe that they can keep state in the DOM between notebook updates. How is that stateless? Nobody, who is serious about programming, wants a stateless program. We need obvious (impossible) and precise (hard but achievable) control of the state, not statelessness. So, having stateful program and keeping the program and its results in the same notebook, must result in a stateful notebook. State must be made _manageable_.

Emacs features org-mode and org-babel. They can at least do what Pluto is able to do. (Not to brag but as a reference.) So, there may be no need to change the environment when to change the mode of work.

Why would anyone in their senses resort to a web browser to do their coding? This is why editors actually have the interface they do. Because it's a _good_ interface for coding. Why throw all this work away? What is the gain in the Pluto (or Jupyter) approach? I don't get it.

An introduction to Pluto

Posted Nov 4, 2020 11:38 UTC (Wed) by roc (subscriber, #30627) [Link] (1 responses)

Web browsers give you a cross-platform, standards-based implementation of scriptable, editable documents that support mixing text, graphics, video and more, with extremely well optimized and robust implementations. Also, everyone has a web browser preinstalled. That's hard to beat.

An introduction to Pluto

Posted Nov 5, 2020 9:59 UTC (Thu) by spacefrogg (subscriber, #119608) [Link]

None of this if true, obviously. I now need a ridiculously large code stack, reading input, converting into JS/HTML, sending it over network (!), receiving, maintaining a secure session to the frontend (!), storing in my own data format, maintaining / communicating state between frontend and backend... to implement an editor.

Yes, you seriously, seriously want to have the renderer of the web browser, but that's it. You don't want to have any of the other parts.

Web browsers are notoriously bad at text entry. They are worse than 80's GUIs. And Chrome will never be good at text entry because they lock most of the interesting keyboard shortcuts and text buffer management actions.

Standards-based: Well the author of the article is not and maybe will never be able to use his favorite standards-compliant browser. So, no. It's not standards-based.

Everyone has a web browser: First, as per the article's author, it was easy to beat to not have the right one. Second, try sending your notebook to your colleague and see if they can open it. They shall be able to play with the data, right? They can't, because they still need Pluto (and start it as an extra step) next to the humongous piece of the "right" web browser just to look at your stuff (let alone playing around). How can this be progress?

Cross-platform: Well, it's only as cross-platform as Pluto, because you still need it. Unless you seriously want to work over network. In which case you (aka. someone who likes you enough) still need the second system running Pluto. So, no, the frontend has nothing to do with platform independence.

An introduction to Pluto

Posted Nov 4, 2020 13:19 UTC (Wed) by rschroev (subscriber, #4164) [Link] (1 responses)

By hidden state, they mean something else. I think I can best explain it with an example.

In a Jupyter notebook, make a cell containing "a = 0" and run that cell.
Then make a cell containing "print(a)", and run it. It will output "0";
Make a third cell containing "a = 1" and run it.
Now run the second cell again. It will now output "1".

You now have a notebook looking like this:

a = 0

print(a)
-> 1

a = 1

You can now delete the third cell, making it even more confusing. In this example it's obvious that a might have an unexpected value, but in real notebooks the value of a often just used in calculations, not printed, making it unclear on exactly which values the calculation is based.

The problem disappears of you always run all cells from top to bottom, which is not hard to do, but Jupyter makes it even more easy to run selected cells one by one as users see fit. It's one of the main criticisms of Jupyter. I haven't read the article, but it looks like the Pluto authors have managed to avoid the problem altogether in Pluto.

An introduction to Pluto

Posted Nov 4, 2020 20:15 UTC (Wed) by otaylor (subscriber, #4190) [Link]

The problem disappears of you always run all cells from top to bottom, which is not hard to do, but Jupyter makes it even more easy to run selected cells one by one as users see fit. It's one of the main criticisms of Jupyter.

It's worth noting that (especially for scientific computations) some of the initial steps in your notebook might be very slow, so not always running from the top makes work much faster. Reinteract (https://www.reinteract.org/) tried to achieve stateless notebooks with partial re-execution for Python, but I abandoned it, among other reasons, because it was really hard to get things to work sensibly with the full Python ecosystem. (If a user changes f.write('foo') to f.write('bar') then you have to go back and replay things from the point where you opened a file for writing.)

Julia is likely easier to build this type of system on top of - I'll have to look at the implementation of Pluto one day.

An introduction to Pluto

Posted Nov 4, 2020 13:43 UTC (Wed) by leephillips (guest, #100450) [Link]

You seem to be equating "state" and "hidden state".

An introduction to Pluto

Posted Nov 4, 2020 14:11 UTC (Wed) by leephillips (guest, #100450) [Link]

"The Pluto authors claim, we'd have to leave a polyglot environment behind us just to describe their support for JS and HTML next to Julia. How is that not polyglot?"

HTML and JS are already running in the browser. The idea is to use them to further elaborate the interface. They are not back-end languages.

An introduction to Pluto

Posted Nov 4, 2020 19:35 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (6 responses)

> Why would anyone in their senses resort to a web browser to do their coding?

This is not targeted at software engineers. It's targeted at data scientists, statisticians, and other scientific types who need to do a lot of number crunching. They don't want to spend three hours configuring Emacs, and then another three hours un-learning all the standard keyboard shortcuts used by every text editing program created within the past ~20 years. But it gets worse than that. If you ask them about design patterns (or higher-order functions, or polymorphism, or... insert your favorite concept here), many will have no idea what you're talking about, and those that are familiar with such concepts tend not to use them much in practice. Their code tends to be highly procedural and ugly, so much so that many are too embarrassed to publish it. Switching editors is not going to change that.

Basically, what these people really want is a spreadsheet. Spreadsheets are much more forgiving of weird or irregular organization of program logic, and make it simple and intuitive to reason about how data flows through the system. But regular spreadsheet software isn't powerful enough for their needs, in multiple senses of the word "powerful" (i.e. flexible, computationally efficient, "there isn't a function for that," etc.). So what you end up with is a custom spreadsheet-like-thing that lets you use one or more "real" programming languages. I really don't see anything wrong with that.

An introduction to Pluto

Posted Nov 4, 2020 21:56 UTC (Wed) by khim (subscriber, #9252) [Link] (4 responses)

But you can use "real" programming language with "real" spreadsheet (I know that because I often do that)… so why the ugly substitute?

An introduction to Pluto

Posted Nov 4, 2020 22:09 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Spreadsheets don't really support scientific visualizations, graphing libraries and TeX rendering.

An introduction to Pluto

Posted Nov 5, 2020 1:03 UTC (Thu) by Paf (subscriber, #91811) [Link] (2 responses)

It’s not an ugly substitute. This is fundamentally a tool for dynamic, on the fly, data analysis and visualization. It’s a tool for supporting scientific research and especially for generating scientific papers in a way that others can interact with the results.

Nuggets of random code stuck to spreadsheets and separate data visualizations with the results pasted in to an eventual test document are in fact an ugly substitue for this, for the job this is intended to do.

It’s like, a “light” scientific computation tool. It’s not for heavy duty number crunching in the sense of high performance computation (often simulation) and “big data”. It’s for light to medium weight computational work and presenting data to people.

If it helps, most of these started as tools for biologists. They need to do a *lot* of data manipulation but most of their stuff isn’t heavyweight physical simulation. It’s data processing, and the data sets aren’t necessarily that large. (Though some are.)

An introduction to Pluto

Posted Nov 5, 2020 1:04 UTC (Thu) by Paf (subscriber, #91811) [Link]

Excuse me, *text* document.

An introduction to Pluto

Posted Nov 5, 2020 1:13 UTC (Thu) by Paf (subscriber, #91811) [Link]

Think of this as a way to produce a scientific paper where someone else can reach in and immediately explore and change the data processing.

A step towards a sort of WYSIWYG editor for scientific papers *including the data processing steps within the scope of the editor*. So I can try a new fit function for the data in a plot and it’s just immediately there in its full context. Or something close to that. And I can do it with *your* paper that you sent me.

An introduction to Pluto

Posted Nov 7, 2020 10:55 UTC (Sat) by osma (guest, #6912) [Link]

Software engineer here. I use Jupyter notebooks a lot for data analysis, exploration and prototyping. Eventually I might turn some of the code in the notebooks into "proper" code such as Python modules. But it's just so much easier to do the prototyping in an environment where you can load stuff into memory, try many different ways of exploring it (without having to load it again!), test different variations of an algorithm etc. without having to start your program from scratch all the time and have it fail after 15 minutes of number-crunching due to an edge case you forgot to consider.

Notebooks are also good for explaining algorithms, since you can intermix runnable code with structured documentation, figures etc.

An introduction to Pluto

Posted Nov 4, 2020 13:29 UTC (Wed) by clugstj (subscriber, #4020) [Link] (3 responses)

"In Jupyter, when a cell is executed, the result appears, and nothing else happens. Any global variables defined in the cell are available for subsequent execution of other cells. The variables can be redefined freely. The upshot is that the output of any particular cell execution depends on a hidden global state: which cells were previously executed and in what order.

So, Pluto is a spreadsheet and Jupyter is a broken spreadsheet?

An introduction to Pluto

Posted Nov 4, 2020 13:46 UTC (Wed) by leephillips (guest, #100450) [Link]

Pluto's reactive model is similar to a spreadsheet.

An introduction to Pluto

Posted Nov 5, 2020 11:08 UTC (Thu) by wjt (subscriber, #56250) [Link] (1 responses)

Another implementation of this spreadsheet-style notebook-with-dependency-graph concept is Observable, by the authors of the popular D3.js data visualisation library. JavaScript's not my particular cup of tea, but spending a little time with Observable last year convinced me that the Pluto authors are right: the hidden global state is a fundamental flaw in the Jupyter execution model.

Spreadsheets are probably the most widely-used programming environment, despite their own flaws, so tools which bridge the gap between that and "real" structured programming environments are (IMO) important.

An introduction to Pluto

Posted Nov 9, 2020 14:41 UTC (Mon) by ballombe (subscriber, #9523) [Link]

> the hidden global state is a fundamental flaw in the Jupyter execution model.

Agreed. This makes Jupyter (and maple, etc.) problematic for teaching mathematics because students will invariably manage to put
the worksheet in an inconsistent state and will accept the incorrect result as true 'since the computer says it'.
Then follows an awkward discussion with the teacher.

How is the dependency graph constructed?

Posted Nov 5, 2020 13:01 UTC (Thu) by marcel.oliver (subscriber, #5441) [Link] (1 responses)

First, thanks for the article. The hidden state problem as well as compatibility with version control are huge issues for current notebook interfaces. (In my opinion, poor compatibility with version control is the bigger one - I know there are workarounds, but getting project members to use version control is often already an uphill battle, so any additional hoops that one has to jump through are typically not practical.)

With regards to Pluto, I am still puzzled how the dependency graph is constructed. If order of cells does not matter, that would mean either that changes to a data structure in a different cell need to be done via creation of a new object, or that the dependencies would need to be declared explicitly. From the article ("Therefore one is not allowed to define a global variable in more than one cell"), it seems that it is the former. But it's not only "defining a global variable", any write access to an object would need to be confined to a single cell. So that seems far from being able to write a generic program in Pluto notebook style, or am I missing something? It would also seem to constrain what can be done in interactive exploration. Here I am not sure what the real impact would be, but it certainly requires a change of mindset compared to working in other notebook-style interfaces.

I also wonder how much conceptually depends on specific features of the Julia language, or whether it could be implemented in just the same way for other languages (Scientific Python user here...)

How is the dependency graph constructed?

Posted Nov 5, 2020 13:29 UTC (Thu) by leephillips (guest, #100450) [Link]

Recalculation through the dependency graph is not triggered by mutation of existing objects, so a running a cell that has a[3] = 17 will not have any effect on other cells. One is obligated to, as you say, explicitly assign to an object. This may require additional expressions in the code that would not be present in “normal” code, but those would not change the computation, but only exist to trigger recalculation through the graph. It’s kind of an awkward aspect of the dependency graph model, but easy to get around. It’s the way this type of reactivity works. When using JavaScript reactive libraries, you sometimes have to put things like a = a in your (JavaScript) code after mutating a, to force a recalculation by explicitly having the variable on the LHS of an assignment.

The computational model does not depend on Julia, and in fact is similar to Observable, a notebook for JavaScript that uses the same type of dependency graph analysis. But other aspects, especially, apparently, in the future, may depend more closely on Julia and further ways the developers find to integrate it with JavaScript to evolve the interface.

Oh, and you’re most welcome.

Deploying software used by the notebook

Posted Nov 27, 2020 14:25 UTC (Fri) by civodul (guest, #58311) [Link] (1 responses)

Another limitation of Jupyter (and of Pluto, as I understand it) is that it does not take software deployment into account, although software used by the notebook obviously influences the end result — starting with the Python/Julia version being used.

We proposed a way to annotate a Jupyter notebook with reproducible "deployment info" here: https://hpc.guix.info/blog/2019/10/towards-reproducible-j... .

Is the Pluto project considering ways to address this issue?

Deploying software used by the notebook

Posted Jun 29, 2021 17:12 UTC (Tue) by fonsp (guest, #153010) [Link]

Hi civodul!

This is indeed a major limitation of current systems, and I have been working hard on embedding the Julia package environment (Project.toml and Manifest.toml) directly into the notebook file: https://github.com/fonsp/Pluto.jl/pull/844 This means that packages will be installed or removed automatically as you use them in code, and others can exactly reproduce your package environment. Currently in beta, will be released in the next month!
- fons (author of Pluto 🎈)

Online hosting service

Posted Feb 7, 2021 0:13 UTC (Sun) by EPo (guest, #144662) [Link]

What could help wider adoption is some online service that rendering Pluto notebooks. This probably requires a lot of resources and some corporate support, but would be very welcome. I think of Observanke and many Jupyter-based or similar dervices like Deepnote, Cocalc, Google Colab. Also would binder render a Pluto notebook? If notebook result ifs HTML maybe there can be a clever Github Actions pipeline to show it as well?