Toggle the Input Fields of Jupyter Notebooks

Many people have described how to toggle the input <div> tags of Jupyter (IPython) Notebooks. I believe Damian Kao was one of the first to blog about it. It’s pretty simple if you know a bit of jQuery. In fact, you can just use pure vanilla javascript too, but, at least for me, jQuery makes it easy to read; and since it is already an integral part of the Notebook, why not take advantage of it.

Here’s my version of the code:

Basically you select all the <div> tags of ‘input’ class then toggle its visibility and height. I also hide the output prompts too by toggling <div> tags of ‘output_prompt’ class.

You can add this code to your Notebook’s custom.js or even make it into an extension. But for me I only need it to render my exported html pages so I can print/publish them. For convenience, I upload it this script to my website, so I can point the src attribute of <script> to it anywhere.



Scraping Youtube Video Playlist

Youtube is not just a website for cute animal videos. I found it a great place to learn programming (and other interesting skills), especially for non-professional coders like myself. I have been trying to learn more about Python by following a number of great channels like sentdexnoobtoprofessional and CodeGeek etc. As I am progressing my “Youtubeducation”, I have developed the need to track what I have been watching along side other learning activities, I wanted to scrape the links and titles of the videos for my notes. Taking this as an opportunity to practice what I have learned, I have started coding a Python class to scrape any Youtube playlist when given a URL.

Iteration #1:

Without inspecting the HTML of a playlist webpage, I started coding. In hindsight, this was so WRONG in many ways. Anyway, you only learn by practicing and making mistakes.

On every playlist page, take this CodeGeek’s playlist for example, there are no links to other videos except those on the list. Each video is linked twice, one through its thumbnail and the other via the title text, except the first video which has 2 more links at the top as the “Play All” option. The link to a video looks like this:


So I came up with this idea of extracting all links begin with “/watch?v=”. Then make them into a Python set() to remove the duplicates, then I will have the list of URLs to all videos. Here is what the code looked like:

Once this rawList is generated, the indices extracted from “index=” in the URLs are used to sort the final list in order.

Of course there was some parsing going on using the custom function urlSplit() in the middle, but I won’t bore you with that.

This method kind of worked – input a playlist URL, it spit out all the links to its videos. But it didn’t feel right. Especially when using set() to remove duplicates, I lost the order of these videos and had to parse their indices to sort them again – this seemed counter intuitive to me.

Iteration #2:

I finally realized that I must check the HTML code to find out what’s really behind these links. As I mentioned above – I really shouldn’t have started any coding without doing so!

Inspecting HTML elements using Chrome browser is very handy. Right click the element, in this case the link to a video, select Inspect or press Ctrl + Shift + I keys to bring up the page inspection console with the entire <a> tag of the link highlighted.

Oh, Boy! Why didn’t I do it! It’s now so obvious that the video thumbnail and title links, although exactly the same, they are wrapped around <a> tags of different classes. Forget about all the duplication removal and indices, I can simply use Beautiful Soup to extract only links that belong to this pl-video-title-link class. How beautiful is that? Here comes the second iteration of my code:

Only the last line was different, but it’s simpler and more elegant. Because Beautiful Soup is parsing the links from top to bottom, so all of the URLs in the list are in the correct order, there is no need to fiddle with them. In the end I put the codes into a class for ease of usage and added a input URL validation check. Here is the full code:

As you may have spotted in the code, in addition to the URLs, I have also included the titles of the videos in the playlist – it makes sense doesn’t it? This code is packaged and can be downloaded here from github. This code is in the code folder. There is also a file in the same folder showing an example of using this class.

Luminex .csv File Anonymizer v0.1a Download

Luminex csv file anonymiserSolid phase multiplex-bead arrays using the Luminex platform are widely used in many clinical as well as research laboratories. We are relatively comfortable at using our locally generated results despite the fact that inter-assay variability sometimes can be a huge problem.  However, when it comes to compare results with other laboratories, we are not so sure.  Previous studies have shown that by standardized protocols, laboratories could achieve lower inter-laboratory assay variability. This tool was created specifically for clinical laboratories who would like to collaborate, share and compare results anonymously. As its name implies, it does one job and one job only – anonymizes your Luminex .csv output file, renames the patient identifiable labeling of samples to just “Sample_###”.

Download Luminex Anonymizer v0.1a

This app is very simplistic and hopefully easy to use. To install the app, download it from the link above. Unzip the file into any folder on your Windows computer (Windows 7, 8, 10 32 bit/64 bit). To run this app, browser to the folder, and double click LxAnonWin.exe  to launch it. Use the Open File… button to locate the .csv file that you would like to be anonymised. Click Open. If no error occurs, the anonymised file (original file name + anon.csv) will be output in the same folder as your source file.

On my own computer, after I downloaded this zip file, the above warning popped up in Chrome. This may happen on your computer too. If you trust me, please ignore the warning and proceed. All the files inside are generated by cx_Freeze, a set of scripts and modules that convert Python scripts into executables. The app itself has less than 100 lines of code, if you are comfortable with compiling your own executable, or running it in your own Python environment, please get in touch for the source code. I have not tested it in OS X and Linux environments, however, this app only uses standard Python libraries, so hopefully it should work under any OS that supports Python 3.4 and above. This is intended to be a small component of a bigger open source project, however at this preliminary stage I am not ready to release all the codes at the moment.

Installing ownCloud and Lighttpd in FreeNAS (FreeBSD) Jail

I have been using ownCloud as a plugin in my FreeNAS server for about 6 months. I use it to sync the codes that I have been writing on my work desktop with the FreeNAS server at home so I can work on them wherever I go. There are a large amount of data that I am processing, using public services such as DropBox and Google Drive are not very practical and can be quite expensive. To be honest, it has served me well and I really liked it. The version I originally installed was probably version 8.0.

Unfortunately, I have this habit of hitting the “upgrade” link whenever the app prompted me. So far so good, until this Sunday! After upgraded to version 8.2.1, then the mapped folders on my FreeNAS server stopped showing, when I logged into my ownCloud server, even though I can see them correctly mounted when I ssh’ed into the server. When deleting and adding the jail storage for those folder again were of no avail, I ended up deleting the plugin jail and went on and install ownCloud 8.2.1 again in a brand new jail from scratch. Because I am doing it from scratch, I have done a bit of research before I started. This guide and video by Dr.KK have been extremely helpful. I followed his advice and chose the Lighttpd + sqlite of webserver/database combination. Because I am following his tutorials, the steps below are going to be same or very similar, nevertheless, I reproduce them here because some steps in the guides were not so clear to me and I’ve had hard time Googling for the answers I wanted.

1. Create a new FreeNAS jail for ownCloud
When you install ownCloud using the turnkey built-in FreeNAS plugin, it comes with Apache server and MySQL database. These are great packages but at the same time they are also resources hungry. For very few users and relatively few files, they are way overkill. On the other hand, we have very limited control over plugin jails – it can very messy trying to customizing them.

To create a FreeNAS jail, just click the Jails icon from the menu and then Add Jails. Choose Advanced Mode and give your jail a name – I use owncloud here, but obviously it can be anything you like.
Near the bottom of the advanced mode window, uncheck the box next to VIMAGE then click OK to continue. This will take a while if you have not set up a jail using the standard template before. Otherwise it finishes in only a few ten seconds.

2. Compile and install ownCloud
There are a couple of ways to log into the jail that you have just created. Either ways, if you are using Windows, you need to download putty or Bitvise ssh client. Once you are connected to your FreeNAS server via ssh, use jls command to find out the JID of your ownCloud jail then jexec [JID] /bin/csh to log in to your jail as root.
jlsYou can also enable sshd for the jail itself and ssh in directly. To do this, copy the ssh directory in etc/ to your jail in the main FreeNAS command line:

Now go back to your FreeNAS GUI, access the command line of your owncloud jail and perform the following steps:

Now log in your ownCloud jail through whatever route you prefer and:

Update the installed packages then fetch updated list of ports. This will take a while. Once it’s done:

You now have the option to disable MySQL and enable sqlite. Click OK and start the building and installing process with:

This process will take a LONG time, make yourself a coffee sit back and relax until it finishes.

3. Install and configure Lighttpd web server
Lighttpd, pronounced “lighty”, is pretty bad ass, it is very fast and lightweight. Youtube and Wikipedia used to use Lighttpd as their web server.

To install:

Run the following command to clean up the unused installation files at this point.

Configure server:
Open the Lighttpd configuration file:

Make changes to the following lines, save and exit nano:

Enable CGI support:

Uncomment out the line below “plain old CGI (mod_cgi):

In the Youtube video DrKK explained why you should choose “plain old CGI” over other modules.

Start your Lighttpd server to see if it works.

If everything was setup correctly, you should be able to see the log in window when point your browser to the ip address of your ownCloud server.

4. Configure ownCloud
Setup and log into your admin account following the simple steps of the initial ownCloud setup interface. These steps are relatively intuitive and straight forward. It’s not necessary to repeat them here. Once you are logged in, head to the drop down menu at the top right corner to access all the admin tasks.

Unlike previous versions, after this update (8.2.1), to enable mapping your existing Freenas drives/volumes/directories (whatever you call them), you need to install a third party app called External Storage Support. Detailed configuration steps can be found in the ownCloud Admin Manual.

5. Configuring PHP
In the Admin page of ownCloud, you may notice that your upload file size is limited to 513MB. To change this setting you need to edit the file .user.ini resides in /usr/local/www/owncloud, not the php.ini that you are told by many installation guides in the Google search results. They are probably talking about Apache/Nginx servers. For Lighttpd, this file is what you MUST modify to change the limit. Unfortunately, it appears that every time you use a “pkg upgrade” command to upgrade the installed packages, this file may get overwritten. You then have to change the settings again.

6. Create SSL certificate and Enable SSL support
Follow Steps 12 and 13 in DrKK’s guide create and enable SSL for your ownCloud server.
Restart your Lighttpd to test the new settings:

If everything works well:

Congratulations! Finally you have a running ownCloud that is fast and takes up very few resources.

Running Python CGI Scripts on Linux Shared Hosting

Check Python CGI Support
Most of the Linux shared hosting supports some level of python CGI scripting. To check if your hosting service allows Python CGI scripting, you can create a file and name it hello.cgi in the cgi-bin folder of your hosting account. Update hello.cgi with the following code.

If this bit of code works as shown in the screenshot below. Congratulations! Now you can code CGI scripts for your website using Python.


Add Traceback Manager for Your CGI Scripts
The cgitb module provides a special exception handler for Python scripts. This module, when activated, will report back any errors and exceptions in a detailed, formatted webpage or a file if you choose to not show it on the browser.


The report includes a traceback showing excerpts of the source code for each level, as well as the values of the arguments and local variables to currently running functions, to help you debug the problem. To enable traceback manager, add two more lines to your hello.cgi file:

It maybe pretty obvious to some people, but for me it took a while to realise that cgitb can only help you with code after the cgitb.enable() command. For example if you write your import statement like this:

The browser will show nothing that tells you abadmodule does not exist.

How to Find Out Your Current Python Interpreter Version?
To find out your Python interpreter version, modify your code as shown below:

Now you can see on the browser:

If you want to know more about the environment your Python scripts are going to be in, refer to the official Python sys module documentation for details.

Useful Code Snippets to Help You Getting to Know the Environment
What other version(s) of Python is(are) available

List the sys.path environment variable

List the available external modules