Mahabharata Using Recurrent Neural Networks and Long Short Term Memory

Wanted to test RNN [1] on Mahabharata [3]. JCJohnson's implementation [2] a fast implementation recommended by Karpathy.

Mahabharata is available from Project Gutenberg [3]. After data cleanup, ran the experiment to generate some data.

Generating 2000 characters, generates:

A: Drona Parva: Section CCLV 

this protect and approached thy breast. Beholding a friend of Santanu, O king's lronding to that effulgen ances died n it. If the common of this estraining every approving of those persons and 

heart, on his sons; have with husbanded Burnificate of Madhu is palace of the ruler of that daughter. Sharactents, and discoursed to the very sage 

and the carded. Tood (at the basze when they are iking that of Bharata to be milk'Thie kinds of end dragged of Brahmana rids and observarowhere that I regards us 

like that of a 

lore, who consume having who is one's soul, Krishna himself. The hundred intradavas entered thy tiger and religion of Sahadeva for the lord of the broking in the prevent, settled himself, O thou of arms, or Shikhandi, where) resting from pa fierce known into grief is the ye is the mind. Persons, the clouds. A 

auspicious asked his hungly cleansed nook nobodimal our child should as bestood then took along by conduct of gold, with me, K.P been warrior Bhishma most extermination. Thinal. They are bent of arms, weeping, and one with four duties of the great hommaded as only to the act, thy carried that business of great son of Viratishta, I will the quantition that consequently, five the breath (the ocean saying, whose angry 

luled (by all. Salutatir, that man, decority as if from which never hermitad of moon. Arrived ing Brahmanas at deep, O gover by the latter. Thy sires, Ketumishma, entrance. Peneemongst that subthe villages. 

All of wicked roaring in her as a well-knowled at weapons l Pandu's race. 

283:1 There, O thou wilt first 

hurl to this is the bare with great on his sinful achieve the Pitris, the prince, live his minds is matter of are enjoymentible, some immediately 

no dic Siva and Yay the loud behaviour and the driveranged for the inexhaustible. Of the minded them 

which a long of humble slain by mair from their brave vanquished for him all one for his shafts. Aven desirous of fruits. Many fear, that one who used t

RNN generates a good sample. However, data cleaning and formatting the data will make a great sample.

Reference
[1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
[2] https://github.com/jcjohnson/torch-rnn
[3] http://www.gutenberg.org/cache/epub/7864/pg7864.txt

Increasing performance of Octave with OpenBlas and Multiple Threads

Octave [1] is a great platform to perform Machine Learning tasks. Octave run time performance can be improved by adding OpenBlas [2] and by increasing the number of threads Octave can run using. This can be further improved using NVIDIA's cuBLAS [3]. The following steps would work in Ubuntu OS.

For the purposes of this test, the matrix multiplication Octave code from [3] was used. To increase the number of threads and using OpenBlas, run the octave code in a command line in a terminal as:

• export LD_LIBRARY_PATH=/opt/OpenBLAS
• OMP_NUM_THREADS=<NumThreads> LD_PRELOAD=/opt/OpenBLAS/lib/libopenblas.so octave code.m

Results

Matrix Multiplication without OpenBlas. Note: Y-axis is in log scale.

Improved Performance Using OpenBlas. Note: Y-axis is in log scale.

The performance is best when OMP_NUM_THREADS=#coresOnTheMachine (Obviously.).

References

[1] GNU Octave: https://www.gnu.org/software/octave/
[2] OpenBlas: http://www.openblas.net/
[3] Drop-in Acceleration of GNU Octave. https://devblogs.nvidia.com/parallelforall/drop-in-acceleration-gnu-octave/

Object Classification using Deep Learning and Raspberry PI

Of late, I have been playing with multimodal interaction [1] with a Raspberry PI (referred to as an IoT edge device) that is being powered by predictions using Deep Learning. In this post, I discuss how I created the pipeline using voice interaction (voice decoding and speaking) and Raspberry PI camera to identify fruits.

Image Dataset

FID30 [5] image dataset contains 30 fruit category of images. These images were augmented using manipulations of the images such as a 90 degree transformation, and negative transformation of the images. Whenever possible, these images were augmented with images along with the transformations.

Deep Learning

Deep Learning [3] can be used for object recognition in images and for decoding voice. In this demo, I have used Convolutional Neural Networks (CNN) [2] for image analysis. For this demo, I have build the CNN model; and used CMU Sphinx with Linux’s espeak for voice interaction.

Specifically, CNN is a type of feed forward artificial neural network that has a wide applicability in image recognition. The models built using NVidia Digits Framework [6] uses Caffe Framework [7] running an NVidia GPU [8]. For the network part, I have chosen to use AlexNet [4] to generate the model; other networks such as GoogLenet were evaluated too. Typically, CNN require a large dataset of labelled images, and the generated models was overfitting. To have a better accuracy and to reduce the overfitting, a huge repository of images is needed. Creation of such image repository with 100’s of thousands of images is a challenge and cannot be addressed by this post.

IoT Edge Device built using the Raspberry PI module

The edge node was built using Raspberry PI using a $2 microphone, a $34 camera, $10 usb wifi adaptor and a $12 speaker system. The system took about $100 to build. The software tech stack includes Python - PyAudio with models were based on CMU PocketSphinx for voice interaction.

Cloud Tech Stack

The model creation and prediction using the model was done in the cloud. Tech Stack included Linux for OS, Python, Java, MongoDB as the backend storage, NVidia GPU for model generation, Caffe Framework for model generation - object recognition and NVidia’s Digits for model generation.

What you will see in the Demo video

When the word “this” is detected by IoT Device, an image is captured and uploaded to the cloud (my laptop) via a REST interface. On the server, the object recognition module pulls the image, identifies the object and updates the entry via the REST Interface. The Raspberry PI polls the REST interface for the prediction result for the object it just uploaded and announces the object on the speakers.

All human interaction is with the IoT Device - through the speaking and listening; and all image recognition is performed on the server. It can be altered such that an initial prediction can be performed at the edge device making it a ‘true’ IoT device. This demo shows and end-to-end pipeline with CNN.

Apple being identified as Apples:

Lemons being identified as Lemons:

Orange being identified as Apricots:

In some cases, the same fruit with with a different background has been identified incorrectly.

References

[1] Multimodal Interaction. https://en.wikipedia.org/wiki/Multimodal_interaction

[2] Convolutional Neural Networks. https://en.wikipedia.org/wiki/Convolutional_neural_network

[3] Deep Learning. https://en.wikipedia.org/wiki/Deep_learning

[4] Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada.

[5] Škrjanec Marko, Automatic fruit recognition using computer vision. Bsc Thesis, (Mentor: Matej Kristan), Fakulteta za računalništvo in informatiko, Univerza v Ljubljani, 2013.

[6] Digits Framework. https://developer.nvidia.com/digits

[7] Caffe Framework. http://caffe.berkeleyvision.org/

[8] NVIDIA GeForce GTX. http://www.geforce.com/hardware/notebook-gpus/geforce-gtx-760m

Using Apache Zeppelin for Spark, SparkML and SparkSQL

Apache Zeppelin[1] is quite a handy tool for exploratory analysis of data using Spark, SparkML and SparkSQL.

Sample Visualization in Apache Zeppelin generated using SparkSQL:

Create RDD, dataframe and register as a temp table:
Query your data using SparkSQL:
Use SparkML for analytics:

Once the Spark SQLContext has the data, Zeppelin can be used to visualize the data.
[1] Apache Zeppelin: https://zeppelin.incubator.apache.org/

Deep Learning: Identification of Fruits Using Digits, Caffe Framework

Image classification using Neural Networks is a very interesting topic. During my bachelors,  I had a glimpse of Neural Networks - where I specifically worked on "C" code to convert hand written signatures into features that could be fed into the neural networks.

Following is work to take open datasets and classify fruits.

Classification of Orange using the model developed using Digits 2 and Caffe Framework.

Software:

1. CUDA: 7.0
2. Caffe framework [1]
3. NVidia GeForce GTX 860M with 640 cores
4. NVidia Digits 2

DataSet:

1. Fruit DataSet [2]  (30 categories)

Digits does image classification, with 25% of the images are set for validation.

After many modifications and using AlexNet[2], I have received an accuracy of 60%. Anything beyond this has lead to overfitting. Overfitting is part of Convolution Neural Networks. To avoid over fitting, a large amount of images needs to be available.

I posted on multiple forums to see if practitioners/researchers can get larger image data. Following are the next steps that I can envision:

1. Looking at other datasets, to extract the same features from them, to add to the dataset. 
2. Downloading non-copyrighted images from Search Engines. Google's API allows only 64 images to be downloaded. I asked on their forum on how to filter images that are copyrighted.

I would like to create a framework that could create automated image datasets (non-copyrighted images) including translations and transformations. 

References:

[1] Caffe Deep Learning Framework: http://caffe.berkeleyvision.org/
[2] Marko Škrjanec, Fruit Image Data set http://www.vicos.si/Downloads/FIDS30
[3] Convolution Neural Networks, https://en.wikipedia.org/wiki/Convolutional_neural_network

Running CUDA 7 on Ubuntu 14.04: blank screen after CUDA install: Solved

Update: CUDA 8 is awesome; and seems more stable.
--

Installation of CUDA on Ubuntu 14 has resulted in blank screen. After fixing those issues, the GUI login screen shows up, but login would not work. After multiple attempts to fix, either CUDA works or X works.

Trying to use Nouveau driver for GUI and Nvidia driver for CUDA, lead me to a blog [1]. It worked for me and thanks Chen.

Chen suggests to blacklist Nouveau (which CUDA installer does as well), remove and purge nvidia drivers, install CUDA. During this part, install Nvidia driver, but NOT the OpenGL driver.

And everything works. Thanks, Chen.

To fix Ubuntu's blank Screen

If Ubuntu boots to blank screen [2]:

1. Goto command line: ctrl + alt + f1.
2. Stop lightdm: sudo service lightdm stop
3. Disable gpu-manager by backing up /etc/init/gpu-manger.conf and commenting its contents.
4. Switch to prime mode by installing and using sudo prime-select nvidia
5. Get the correct busid for nvidia: lspci | grep NVIDIA
6. Update /etc/X11/xorg.conf with the correct busid. "PCI:01:00.0"
7. Perform sudo chattr +i /etc/X11/xorg.conf
8. Start lightdm: sudo service lightdm start

Reference:

[1] Chen, https://chuanwen.wordpress.com/2015/07/19/run-cuda-on-ubuntu-14-04-2/

[2] http://vxlabs.com/2015/02/05/solving-the-ubuntu-14-04-nvidia-346-nvidia-prime-black-screen-issue/

Caffe Install: ATLAS installation Ubuntu CPU throttling apparently enabled

Caffe Install requires a Math library. Of the 3 options, ATLAS is open source and, hence, the default. Installation of ATLAS was causing issues because intel_pstate governor. Governor does power management for Kernel [1].

To identify if, the intel_pstate is the issue, identify if the governor is intel_pstate. 

1. cpufreq-info will show the governor.
2. If intel_pstate is shown as the governor, disable intel_pstate.
3.  Edit the file /etc/default/grub and add: GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable"
4. Update grub.cfg ala grub-mkconfig -o /boot/grub/grub.cfg

Reference

[1] https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt

[2] http://unix.stackexchange.com/questions/121410/setting-cpu-governor-to-on-demand-or-conservative

[3] https://wiki.archlinux.org/index.php/CPU_frequency_scaling#Scaling_governors