Tag Archives: IPython

Running My Own Jupyter Notebook Server in a FreeBSD Jail

As a non-computer/data science professional, my fascination of what one can do with data science is growing day by day. I am aware that Enthought Canopy and Continuum Anaconda are both excellent pre-built  packages with intergrated Jupyter notebook and everything that I might need for doing data analysis with Python. I have been using Anaconda on individual computers for over a year and really like it. However I would love to have a central place for all my notebooks, so I don’t have to copy them around while working on my projects on different devices.

There are plenty of advises out there on the web telling people that how difficult and frustrating it is to compile and install Python and those data analysis packages from scratch (e.g. numpy, scipy, matplotlib and pandas just to name a few).  But I wanted to see how hard it really is. So after successfully set up ownCloud in my FreeNAS jail in a recent attempt, I (foolishly?) decided to have a go at setting up my own Jupyter notebook server in another FreeBSD jail instance in my home server.  This documents how I did it.

1. Setup a new FreeNAS jail
2. Install Python
3. Install Jupyter
4. Setup Jupyter server
5. numpy, pandas and matplotlib
6. Some problems and their fixes
7. Installed modules

1. Set up a new FreeNAS jail [^top]

Create Jail: Click the Jails icon from the FreeNAS admin page menu and then Add Jails. Choose Advanced Mode and give the jail a name. Uncheck the box next to VIMAGE then click OK to continue. This will take a while if a jail using the standard template has never been set up before.

Enable SSH: Launch the FreeNAS shell from the treeview menu on the left. Copy all of the files in the /etc/ssh/ to the jail.

If SSH is off by default for FreeNAS, the following line in ssh/sshd_config must be commented out before copying – it can also be edited out later obviously.

Launch a shell for the jail from the Jails tab, enable password and launch sshd service:

Or permanently enable sshd:

Close the shell, SSH into the jail then update pkg and the ports repository:

This may take a while, if there is any problem, delete /var/db/pkg/repo-*.* then redo pkg upgrade.

2. Install Python [^top]
Many of the necessary packages can easily be installed by pkg install [packagename]. Before any attempt of installing from source, try pkg first.

Some of the packages can also be installed using pip, so it’s a good idea to install pip first:

3. Install Jupyter [^top]
pyzmq the Python bindings for ZeroMQ must be installed before jupyter.

This whole installation shouldn’t take more than a few minutes. Before I figured out how to install these quickly, I went down the route of installing jupyter from BSD ports and it took a very long time!

4. Set up Jupyter server [^top]
Jupyter notebook’s online documentation details how to setup and run a secure Jupyter server. To quickly test the server is working properly, the password protection and SSL encryption can be skipped and added later when required. A default configuration can be generated using the following command:

This outputs jupyter_notebook_config.py in the ~/.Jupyter directory. There are a huge amount of options in the configuration file. To simplify things, I just created a jupyter_notebook_config.py with the following lines, instead of using the automatically generated one.

Once the above steps are complete, start the notebook server:

At this point, the server needs to be launched via SSH, it would be very annoying and inconvenient to do that. We need to add this to the crontab:

Now restart the jail and try out the brand new Jupyter Notebook server! Everything should work as expected.

5. numpy, pandas and matplotlib [^top]
Unfortunately, a Jupyter notebook server is not very useful without a few Python modules such as numpy, pandas and matplotlib.

numpy and pandas:

The easiest way to install matplotlib is to follow the installation guide on its website:

The installation takes less than 10 minutes, compared to building and installing from BSD ports, this is like ‘lightening fast’.

6. Some problems and fixes [^top]
When importing numpy, pandas or maplotlib in the notebook, an error occurs with the following lines at the very bottom of the error message.

This can be fixed by setting the LD_LIBRARY_PATH to the required gcc version. This was discussed over in stackoverflow.

If Jupyter is always being called from terminals, adding the following line to /.cshrc will make the LD_LIBRARY_PATH setting permanent:

However, if Jupyter notebook is to be launched automatically after boot by crontab, this fix does not work. I have to modify the /usr/local/bin/jupyter-notebook file to change the system environment before launching the notebook server (highlighted are inserted lines of code):

7. List of all installed modules [^top]
Some of the packages below (e.g. biopython, beautifulsoup4, pypyodbc etc.) are not required but they were installed for future use, when I was stuck and got frustrated :

I have had spent more than a couple of days to make this whole thing working. After I was satisfied with the server, I then created another jail, and made a second attempt looking for ‘better’ ways to set this up. The above procedures take only a couple of hours in total.

To be honest, without a proper step-by-step tutorial at the beginning, for a newbie like myself, it was quite difficult! But I have learned a lot during this process, in the end I think it’s worth it.

[back to ^top]