Julia on Google Compute Engine: installation and first steps

This blog entry is part one of a series on using Julia with Google Compute Engine. It was inspired by a bit of narcissism, a burning passion for scientific computing, and lackluster experiences with other comparable languages. I recently had the opportunity to answer a question about running Julia on Google Compute Engine (GCE), and thought I'd dust off this half-finished blog entry to round it all out.

In this entry, we'll focus on installing Julia on a GCE instance.

What is Julia?

Julia is a fairly recent programming language that focuses on high performance and ease of use. It's become rather popular in scientific circles as an alternative to languages like R or MATLAB. It's interactive environment means that you can explore data and models easily, but because Julia programs execute quickly, they scale well, and can be easily parallelized.

I can't do the Julia language justice in one paragraph, so you should check out all the various features on their site. We'll proceed to getting started.

Spinning up an instance

I'm going to use a two-core standard instance for my testing, running Debian, with primarily default settings. However, I want to make sure that I have access to my Google Cloud Storage bucket(s) for the next couple of blog entries, so I'll need to tell GCE to configure the instance with those scopes with the service account (you won't use this yet).

Caveat: this blog is a snapshot of a point in time. Images and command line instructions may become out of date. Please refer to the documentation for current examples.

Using the Developers Console

The Google Developers Console walks you through configuring your machine. There are three primary sections that you're going to want to modify as illustrated:

Instance name

Location and resources

Project access

Using the command line

While you can do this using the Developers Console, I prefer to use the command line utility, gcutil. After installing and configuring it with your project, you can create the instance described above with a simple command:

 gcutil --project="<project name>" addinstance "<instance name>" --zone="us-central1-b" --machine_type="n1-standard-2" --service_account_scopes="https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_write" --image="https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-7-wheezy-v20131120"  

I've created two instances this way, named julia-pkg and julia-src.

Installing Julia on GCE, via package manager

If you're looking to quickly get going with Julia, it's pretty straightforward. There's a Julia package that provides all you need, including dependencies, JIT compiler, etc... However, it is in the unstable branch, so you'll need to do something sort of special.

First, edit your /etc/apt/sources.list file, and add the following line (note, you'll have to run as root):

 deb http://ftp.de.debian.org/debian sid main  

Now, make sure you're pulling from the unstable branch:

 $ sudo apt-get update  

Next, while this is not required, you may want to create a preferences file so that you only take what you need from the unstable branch. Here how I created and populated my preferences file:

 $ echo 'Package: *  
 Pin: release a=stable  
 Pin-Priority: 990  

 Package: *  
 Pin: release a=testing  
 Pin-Priority: 600  

 Package: *  
 Pin: release a=unstable  
 Pin-Priority: 300' | sudo tee /etc/apt/preferences  

Instead of your regular apt-get syntax, you need to tell the package manager that you want to use is in the unstable branch:

 $ sudo apt-get -t unstable install julia  

Follow the prompts, and you'll be ready to use Julia.

Installing Julia on GCE, via compiling source

So you want to be on the bleeding edge of Julia, eh? That's cool. Julia uses quite a few other bits of software, so let's go ahead and install them before we get started:

 $ sudo apt-get install bzip2 g++ gcc gfortran git m4 make ncurses-dev  

Extra credit if you go and research what you're installing first.

Next, you'll want to clone into the GitHub repository:

 $ git clone git://github.com/JuliaLang/julia.git  

Great! Now you're ready to compile. First, step into the directory. Then, run the makefile. Because GCE CPUs use a Xeon architecture the, you'll need to add a couple of flags that help out OpenBLAS:

 $ cd julia  
 $ make OPENBLAS_USE_THREAD=0 OPENBLAS_TARGET_ARCH=NEHALEM  

This might take some time. Go get some coffee/tea/caffeinated substance of your choice. Once the tests all pass, you'll know that your compilation has finished successfully. You might want to create a link or alias to the executable, for ease of use.

Using Julia

Julia is an interpreted language, and as such, can be used interactively with the interpreter in addition to programs stored in files. To call up the interpreter, simply type:

 $ julia   # or your specified alias to ./julia  

To execute a Julia program, pass the filename as an argument to the executable:

 $ julia <filename.jl>  

Test it out! Using your favorite editor (you may have to install it), copy and paste the following code into a file called matrix_mult.jl:

 array1 = randn(8, 8)  
 array2 = rand(8,8)  
 show(*(array1, array2))  

Then execute our matrix multiplication operation by running it:

 $ julia matrix_mult.jl  

You should see an 8x8 array described in your output, similar to this one:

That's it for the first part of the series! Next up, we'll make use of that service account to speed up writing your programs, distributing them, and gathering output. In the meanwhile, now would be a great time to start trying out writing small programs in Julia.

Popular posts from this blog

  • Deep in the dark recesses of Google account settings is one we’d prefer to ignore . It’s one that gives those of us who live our lives online a way to bequeath our digital file cabinet to another. It’s the setting that I recalled -- in a blind panic -- at 4am on this past Halloween. My reaction  @ 4am Rewind The cause for this panic? Well, it started over a year and a half ago in 2013, when I was slotted to attend OSCON. I flew into PDX, excited to attend workshops the next day. I had signed up for a data science with R workshop -- long before I started my flirtation with Julia . While I was unpacking, my back gave out, and I never made it to the workshop. That event kicked off one of the best and worst years of my life (cue the Tale of Two Cities jokes).  My health kept deteriorating, in strange ways. It affected my ability to keep plans with friends, enjoy my life, keep up with day-to-day chores, and eventually my ability to do my job. I remember standing at my des
    Keep reading
  • My first job as a software engineer nearly drove me out of the field forever. Many efforts focus on keeping girls and women interested in technology from elementary school through 4-year computer science programs in order to “fill the pipeline” with qualified female engineers. My all-women's education never dissuaded me from pursuing computer science, but that’s the exception to the norm . Education is but a sliver of a woman’s career. However, what happens after those women graduate and get their first job? What happens to those women in the pipeline? As Julie Pagano candidly states : “The pipeline is leaky and full of acid. The pipeline leads to a sewage treatment plant. The pipeline ends in a meat grinder.” I’d like to talk about one such meat grinder. Meat grinder , not for people The environment My first job out of undergrad was as a software engineer at a major company. When I joined, my immediate team was approximately 15-20 people, all of whom were male. Oddly enough,
    Keep reading
  • This is the second part of the Julia on Google Compute Engine (GCE) series that I started a few weeks back. The first entry addressed how to set up Julia on a standard GCE instance, and how to run simple scripts. Today, we'll be doing less with Julia , and more with setting things up so you can efficiently write Julia programs and scripts, and make it easy to get the results of your computations. Anatomy of a setup There's the typical disclaimer to be had here, that the optimal setup depends largely upon personal preference and what you're trying to accomplish. My general process is to write locally, validate against small data sets, then run on a high power instance with the output in an arbitrary object store. Your process and setup may vary, but it's likely that you'll want to write access files on your virtual machine, and make your results accessible outside of the machine. The easiest way to do this is to use Google Cloud Storage (GCS) , a highly available a
    Keep reading