How mature is the "Julia" scientific computing language project?

Question

I'm considering learning a new language to use for numerical/simulation modelling projects, as a (partial) replacement for the C++ and Python that I currently use. I came across Julia, which sounds kind of perfect. If it does everything it claims, I could use it to replace both C++ and Python in all my projects, since it can access high-level scientific computing library code (including PyPlot) as well as running for loops at a similar speed to C. I would also benefit from things like proper coroutines that don't exist in either of the other languages.

However, it's a relatively new project, currently at version 0.x, and I found various warnings (posted at various dates in the past) that it's not quite ready for the day to day use. Consequently, I would like some information about the status of the project right now (February 2014, or whenever an answer is posted), in order to help me assess whether I personally should consider investing the time to learn this language at this stage.

I would appreciate answers that focus on specific relevant facts about the Julia project; I'm less interested in opinions based on experience with other projects.

In particular, a comment by Geoff Oxberry suggests that the Julia API is still in a state of flux, requiring the code to be updated when it changes. I would like to get an idea of the extent to which this is the case: which areas of the API are stable, and which are likely to change?

I guess typically I would mostly be doing linear algebra (e.g. solving eigenproblems), numerical integration of ODEs with many variables, and plotting using PyPlot and/or OpenGL, as well as low-level C-style number crunching (e.g. for Monte Carlo simulations). Is Julia's library system fully developed in these areas? In particular, is the API more or less stable for those types of activities, or would I find that my old code would tend to break after upgrading to a new version of Julia?

Finally, are there any other issues that would be worth considering in deciding whether to use Julia for serious work at the present time?

Answer 1

Julia, at this point (May 2019, Julia v1.1 with v1.2 about to come out) is quite mature for scientific computing. The v1.0 release signified an end to yearly code breakage. With that, a lot of scientific computing libraries have had the time to simply grow without disruption. A broad overview of Julia packages can be found at pkg.julialang.org.

For core scientific computing, the DifferentialEquations.jl library for differential equations (ODEs, SDEs, DAEs, DDEs, Gillespie simulations, etc.), Flux.jl for neural networks, and the JuMP library for mathematical programming (optimization: linear, quadratic, mixed integer, etc. programming) are three of the cornerstones of the scientific computing ecosystem. The differential equation library in particular is far more developed than what you'd see in other languages, with a large development team implementing features like EPIRK integrators, Runge-Kutta-Nystrom, Stiff/Differential-Algebraic delay differential equation, and adaptive time stiff stochastic differential equation integrators, along with a bunch of other goodies like adjoint sensitivity analysis, chemical reaction DSLs, matrix-free Newton-Krylov, and full (data transfer free) GPU compatibility, with training of neural differential equations, all with fantastic benchmark results (disclaimer: I am the lead developer).

The thing that is a little mind-boggling about the matured Julia ecosystem is its composibility. Essentially, when someone builds a generic library function like those in DifferentialEquations.jl, you can use any AbstractArray/Number type to generate new code on the fly. So for example, there is a library for error propagation (Measurements.jl) and when you stick it in the ODE solver, it automatically compiles a new version of the ODE solver which does error propagation without parameter sampling. Because of this, you may not find some features documented because the code for the features generates itself, and so you need to think more about library composition.

One of the ways where composition is most useful is in linear algebra. The ODE solvers for example allow you to specify jac_prototype, letting you give it the type for the Jacobian that will be used internally. Of course there's things in the LineraAlgebra standard library like Symmetric and Tridiagonal you can use here, but given the utility of composibility in type generic algorithms, people have by now gone and built entire array type libraries. BandedMatrices.jl and BlockBandedMatrices.jl are libraries which define (Block) banded matrix types which have fast lu overloads, making them a nice way to accelerate the solution of stiff MOL discretizations of systems of partial differential equations. PDMats.jl allows for the specification of positive-definite matrices. Elemental.jl allows you to define a distributed sparse Jacobian. CuArrays.jl defines arrays on the GPU. Etc.

Then you have all of your number types. Unitful.jl does unit checking at compile time so it's an overhead-free units library. DoubleFloats.jl is a fast higher precision library, along with Quadmath.jl and ArbFloats.jl. ForwardDiff.jl is a library for forward-mode automatic differentiation which uses Dual number arithmetic. And I can keep going listing these out. And yes, you can throw them into sufficiently generic Julia libraries like DifferentialEquations.jl to compile a version specifically optimized for these number types. Even something like ApproxFun.jl which is functions as algebraic objects (like Chebfun) works with this generic system, allowing the specification of PDEs as ODEs on scalars in a function space.

Given the advantages of composibility and the way that types can be use to generate new and efficient code on generic Julia functions, there has been a lot of work to get implementations of core scientific computing functionality into pure Julia. Optim.jl for nonlinear optimization, NLsolve.jl for solving nonlinear systems, IterativeSolvers.jl for iterative solvers of linear systems and eigensystems, BlackBoxOptim.jl for black-box optimization, etc. Even the neural network library Flux.jl just uses CuArrays.jl's automatic compilation of code to the GPU for its GPU capabilities. This composibility was the core of what created things like neural differential equations in DiffEqFlux.jl. Probabilistic programming languages like Turing.jl are also quite mature now and make use of the same underlying tooling.

Since Julia's libraries are so fundamentally based on code generation tools, it should be no surprised that there's a lot of tooling around code generation. Julia's broadcast system generates fused kernels on the fly which are overloaded by array types to give a lot of the features mentioned above. CUDAnative.jl allows for compiling Julia code to GPU kernels. ModelingToolkit.jl automatically de-sugars ASTs into a symbolic system for transforming mathematical code. Cassette.jl lets you "overdub" someone else's existing function, using rules to change their function before compile time (for example: change all of their array allocations to static array allocations and move operations to the GPU). This is more advanced tooling (I don't expect everyone doing scientific computing to take direct control of the compiler), but this is how a lot of the next generation tooling is being built (or rather, how the features are writing themselves).

As for parallelism, I've mentioned GPUs, and Julia has built-in multithreading and distributed computing. Julia's multithreading will very soon use a parallel-tasks runtime (PARTR) architecture which allows for automated scheduling of nested multithreading. If you want to use MPI, you can just use MPI.jl. And then of course, the easiest way to make use of it all is to just use an AbstractArray type setup to use the parallelism in its operations.

Julia also has the basic underlying ecosystem you would expect of a general purpose language used for scientific applications. It has the Juno IDE with a built-in debugger with breakpoints, it has Plots.jl for making all sorts of plots. A lot of specific tools are nice as well, like Revise.jl automatically updates your functions/library when a file saves. You have your DataFrames.jl, statistics libraries, etc. One of the nicest libraries is actually Distributions.jl which lets you write algorithms generic to the distribution (for example: rand(dist) takes a random number of whatever distribution was passed in), and there's a whole load of univariate and multivariate distributions (and of course dispatch happens at compile time, making this all as fast as hardcoding a function specific to the distribution). There is a bunch of data handling tooling, web servers, etc. you name it. At this point it's mature enough that if there's a basic scientific thing and you'd expect for it to exist, you just Google it with .jl or Julia and it'll show up.

Then there's a few things to keep in mind on the horizon. PackageCompiler is looking to build binaries from Julia libraries, and it already has some successes but needs more development. Makie.jl is a whole library for GPU-accelerated plotting with interactivity, and it still needs some more work but it's really looking to become the main plotting library in Julia. Zygote.jl is a source-to-source automatic differentiation library which doesn't have the performance issues of a tracing-based AD (Flux's Tracker, PyTorch, Jax), and that is looking to work on all pure Julia codes. Etc.

In conclusion, you can find a lot of movement in a lot of places, but in most areas there is already a solid matured library. It's no longer at a place where you ask "will it be adopted?": Julia has been adopted by enough people (millions of downloads) that it has the momentum to stay around for good. It has a really nice community, so if you ever just want to shoot the breeze and talk about parallel computing or numerical differential equations, some of the best chat rooms for that are in the Julialang Slack. Whether it's a language you should learn is a personal question, and whether it's the right language for your project is a technical question, and those are different. But is it a language that has matured and has the backing of a large consistent group of developers? That seems to be an affirmative yes.

Answer 2

If not, is it possible to give a rough order-of-magnitude estimate for how long I should wait before considering it again?

My rough, order-of-magnitude estimate of how long it takes computational science languages to mature is around a decade.

Example 1: SciPy started in 2001 or so. In 2009, Scipy 0.7.0 was released, and the ODE integrator had an interface to VODE (which is the equivalent of ode15s, roughly; ode15s is NDF-based, VODE is BDF/Adams-Bashforth, depending). With SciPy 0.10.0, an interface to dopri5, which is roughly the equivalent of MATLAB's ode45, a Runge-Kutta 4th order method typically introduced as the first practical numerical integration method to undergraduates. SciPy 0.10.0 was released in December 2011, and it took about 10 years for them to include a feature of MATLAB introduced to every engineering undergrad I know.

Example 2: Mathworks was founded in 1984. In their first release, they used a LAPACK port to C named JACKPAC (after Jack Little, a MathWorks engineer who wrote it). They did not replace it with LAPACK until 2000.

Julia may take less time, but I'd estimate about 10 years from founding to become mainstream. (It's already been out a year or so; maybe 9-10 years, then?)

Is Julia's library system fully developed in these areas? In particular, is the API more or less stable for those types of activities, or would I find that my old code would tend to break after upgrading to a new version of Julia?

I don't use Julia, so take what I say with a grain of salt, since I've only seen Jeff Bezanson give presentations about Julia. They've bent over backwards to make it easy to link and use libraries from C, Python, and Fortran. If you can't find a Julia library that does what you want, write a Julia shim for a library in a more established language. Consequently, I don't think a lack of libraries will be a concern. I think a concern will be making sure that changes to core language features don't bite you in the ass. If you look at the milestones in the Julia Git repo, you'll see that the "breaking" tag is used quite a bit (12 times in the 0.2 release, 5 times in the 0.3 release). To me, that suggests that the core language is still evolving, which is why I hesitate to use the language right now. Julia looks very cool, so I anticipate that I may use it in the future, maybe a few years from now.

EDIT: Aurelius brings up a good point:

What makes you assume Julia will ever actually become mainstream, and not just die off in obscurity like so many other languages? SciPy/numpy had/has the backing of an ever-growing python community, which Julia doesn't have.

In the original answer, I decided to avoid the question of "Will Julia succeed in becoming mainstream?" as much as possible. Failing is easy; success is difficult. I think the best comparison of Julia is to technical computing languages like MATLAB, R, and Octave. HPC languages (Chapel, Fortress, UPC, etc.) have a narrower audience than technical computing languages, and general purpose languages (C, Python, C++, etc.) have a broader audience than technical computing languages.

Something I think that helps Julia is design for performance without sacrificing expressiveness. Julia is much more competitive with compiled languages like C than MATLAB, R, or even Python. This design goal is also a feature that can draw people from existing languages, like:

• People who care a lot about performance and come from languages like C and Fortran, but are willing to sacrifice a tiny bit of performance (maybe a factor of 2ish) to go from compiled language to fewer lines of interpreted language (along with a REPL for more rapid development and testing).
• People who care about high productivity and come from languages like Python, R, and MATLAB, but want more performance. When it comes to execution, pure Python, pure MATLAB, and pure R are slow. Developers in those languages have gone out of their way to wrap libraries in compiled languages, but you can't wrap everything, and at some point, the core language is going to slow you down. Core Julia is faster, which lets you do more science faster.
• People who care about free software. Julia is interpreted and free (so is Python, Octave, etc.); MATLAB is not.

Julia is also attempting to facilitate parallelism; I don't feel terribly qualified to expand upon that point, and I don't think it's the main draw of the language, but I think it's a selling point that they are working on, and I hope others can shed light on it.

However, even with technical merit on their side, the language creators have to do the legwork to promote the language and evangelize. Jeff Bezanson, Alan Edelman, Stephen Karpinski, and Viral Shah are working very hard to make the language succeed. Alan Edelman has deep ties to the computational science community, and he's worked on language-level projects before (notably, the Star-P extension to MATLAB). Jeff Bezanson has been doing the conference circuit to promote Julia to computational scientists and engineers for a while. At MIT, they've been doing a good job of recruiting students and staff (notably, Steven G. Johnson) to contribute by adding libraries to Julia. They've got an article in Wired, and managed to get a Wikipedia article for themselves, all after only a year. Their Git repo has thousands of stars, hundreds of forks, and hundreds of watches, so by open-source standards, their project has been a success. I think they've done all the right things so far, so it's a matter of sustaining that effort and building community. They could still fail, but getting this far suggests to me that they have a reasonable chance at success.

Answer 3

I believe Julia is worth learning. I have used it to produce a few research finite element codes, and produce them very quickly. I have been over all very pleased with my experience.

Julia has enabled a workflow for me that I have found difficult to achieve with other languages. You may use it as a prototyping language like MATLAB, but unlike MATLAB when you have a working code and are going through profiling iterations to speed it up, replacing hotspots with C code is painless. They have made interfacing with C (and Python) a priority in design.

Thus the language has enabled me to very rapidly move from my mathematical formulations to functional code and then from functional code to highly performant code. I think this feature of Julia is undersold, but it adds tremendous value.

In many instances I have obtained C performance (compared to my own C code) in a matter of hours after producing a functional finite element code. So far if I use only Julia features I usually get to within ~3X slower than C. After this I replace hotspots with C functions (Julia comes with a stack sampling profiler to help identify these). Often this requires only replacing the offending hotspot lines of code with a "ccall," with Julia handling any marshalling.

I think the main things Julia is missing right now though that makes me hesitate to consider it for bigger projects is the lack of a fully supported debugger, and poor support for plotting (right now your best bet in plotting is actually just an interface into matplotlib that I've had break more often than not). Immediate feedback on code is really important. I also don't know how to survive without interactive debugging, and in this respect I'm very spoiled by MATLAB and GDB.

Answer 4

From my experience Julia isn't ready for (scientific) everyday use yet (I'm talking about the stabilized version 0.4 of march 2016). The language itself is nice, feature rich and consistent; a refreshing step forward from both matlab and python. There are certainly uses cases where Julia is a good choice even at this early stage. But if you need reliable and mature scientific libraries I cannot recommend making the move now. The main problems I encountered are:

• The packages outside of Julia's core aren't reliable. They break with updates, change, get replaced, sometimes are incomplete or simply broken.
• Julia's parallelism features (imo the most promising potential matlab killer features) are are immature. You will encounter segmentation faults, memory leaks, crashes and disappointing performance. Sometimes they don't play well with other parts of julia, for example some of the solvers for linear systems or packages outside of the core. While these features sound promising, they often enough failed for me. Almost never was it enough to simply write @parallel in front of the for loop, where in theory it should.
• Many little things, small bugs and issues one could live with: not so nice and sometimes erroneous call stack traces, a little buggy interpreter history, slow loading of packages, bad performance of one or another package/function, etc. What annoyed me the most is the PyPlot package, a wrapper for matplotlib, currently there's no alternative to it. It's really great that it's there and it mostly works. But if you need to plot data, be prepared for very slow performance and problems.

This will all get better. I am confident that some day julia will be superior to matlab and python in almost every aspect. Chances are great that innovative packages will be developed for it. It seems like there already is a big community. Even now there is a wealth of packages with use cases ranging from opencl to webservers. Using python or c libraries is very easy, so you probably won't hit a wall in terms of library availability. A great strength of Julia will be the effortlessness with which one can glue together high performance functionality from various sources in a modern and consistent high level language (see answers above). But as a whole I found it to be neither mature nor stable enough to be used productively.