Deep Learning

Deep Learning is a branch of Machine Learning and it seeks to imitate the neural activity of the human brain. Deep learning architectures have been applied to fields of computer vision, speech recognition, NLP, audio recognition, social network filtering, machine translation, bioinformatics and drug design, amongst others. In most cases, results have proven to be comparable, if not superior to human experts.[i]

In the ensuing pages, I’ll break down the current offerings available in deep learning technology and try to give you the pros and cons of each offering. This is, however, technology that changes by the day although I’ll keep the discussion as updated as possible, much of this information will need to be updated in six months. (All the more reason to keep an eye on these editions, I usually update the book once every six months, at minimum!)

In their paper Caffe2 vs. TensorFlow: Which is a Better Deep Learning Framework?[ii], Baige Liu and Xiaoxue Zang focus on the two most used deep learning programs, Caffe2 and Tensorflow, comparing five aspects of the software: the expressiveness, the modeling capability, the performance, help & support, and the scalability.137 The authors chose “TensorFlow because it is currently the most widely-used deep learning framework.”137 The authors recognize that Caffe was an extremely popular framework before TensorFlow was introduced and the Caffe2 framework can build upon that potential, while gaining a lot of user preference in the near future.137

However, as Liu and Zang conclude:

“in many aspects and as a result we find neither of these two has an [sic] dominating advantages over the other. Therefore, in practice, the choice between these two actually depends on the specific user tasks and the user preferences. Overall if the user need [sic] to pursue speed and has limited space restricted by the device, Caffe2 is a better choice since our experiments’ results revealed that Caffe2 has a significant advantage over TensorFlow both in speed and space. Nevertheless, TensorFlow is still powerful and useful because there is a large number official [six] and third-party resources, services, debugging tools, and a big supportive community that makes it easier to find reference codes.”

With software, there is rarely a binary answer, i.e., “Is x piece of software better for my problem than y piece of software?” Caveats abound. Always.  

 

[i] Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffry. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada.

[ii] Liu, Baige, Zang, Xiaoxue. (2017). Caffe2 vs. TensorFlow: Which is a Better Deep Learning Framework? Stanford University. http://cs242.stanford.edu/assets/projects/2017/liubaige-xzang.pdf (Accessed 1 May 2018).

In his article Google Just Open Sourced TensorFlow, Its Artificial Intelligence Engine, Cade Metz explains that at its 2015 Google I/O conference Google open sourced its deep learning engine known as TensorFlow. In open sourcing TensorFlow, Google is freely sharing the underlying code with the world at large.135 “In literally giving the technology away, Google believes it can accelerate the evolution of AI. Through open source, outsiders can help improve on Google's technology and, yes, return these improvements back to Google,” explains Metz.

"What we're hoping is that the community adopts this as a good way of expressing machine learning algorithms of lots of different types, and also contributes to building and improving [TensorFlow] in lots of different and interesting ways," says Jeff Dean, and a key player in the rise of Google’s deep learning technology.

Open sourcing AI has been a common practice over the past few years. Facebook, Microsoft, and Twitter have all made huge strides in AI and some have open sourced software that is similar to TensorFlow, including Torch—a system originally built by researchers in Switzerland—as well as systems like Caffe and Theano. However, Google's move is highly significant because Google's AI engine is considered to be the world's most advanced—and because, well, it is Google after all.

Google, however, isn't giving away all its secrets. As Metz explains:

“At the moment, the company is only open sourcing part of this AI engine. It's sharing only some of the algorithms that run atop the engine. And it's not sharing access to the remarkably advanced hardward infrastructure that drives this engine (that would certainly come with a price tag). But Google is giving away at least some of its most important data center software, and that's not something it has typically done in the past.”

In the past, Google only shared its designs until after it had moved onto other designs, but it had never open sourced code. With TensorFlow, however, the “company has changed tack, freely sharing some of its newest—and, indeed, most important—software.” Google does open source parts of its Android mobile operating system and several other smaller software projects, but this is far different.135 With TensorFlow’s release, “Google is open sourcing software that sits at the heart of its empire,” states Metz.

Deep learning relies on neural networks and Google typically “trains these neural nets using a vast array of machines equipped with GPU chips—computer processors that were originally built to render graphics for games and other highly visual applications, but have also proven quite adept at deep learning,” explains Metx.

GPUs are good at processing lots of little bits of data in parallel, and that's what deep learning need, but after they've been trained, these neural nets run in different ways, often running on “traditional computer processors inside the data center, and in some cases, they can run on mobile phones,” notes Metz. The Google Translate app is a prime example of this. It runs entirely on a mobile device without a data center connection, letting users translate foreign text into their native language.

TensorFlow is a way of building and running neural networks that are required for computations like this, both at the training stage and the execution stage. It is basically a set of software libraries that users “can slip into any application so that it too can learn tasks like image recognition, speech recognition, and language translation.”

The underlying TensorFlow software was built in C++, but “in developing applications for this AI engine, coders can use either C++ or Python, the most popular language among deep learning researchers,” adds Metz.135 Google hopes that developers “will expand the tool to other languages, including Google Go, Java, and perhaps even Javascript, so that coders have more ways of building apps.”

According to Google’s Jeff Dean, “TensorFlow is well suited not only to deep learning, but to other forms of AI, including reinforcement learning and logistic regression.” Tensorflos is twice as fast as Google’s previous system, DistBelief, Dean adds.

In open sourcing the tool, Google provides some sample neural networking models and algorithms, “including models for recognizing photographs, identifying handwritten numbers, and analyzing text.” "We'll give you all the algorithms you need to train those models on public data sets," Dean says.

The major caveat to Google’s seeming geneorisity is that the initial open source version of TensorFlow only runs on a single computer, you can’t train models across a vast array of machines. “This computer can include many GPUs, but it's a single computer nonetheless,” notes Metz. “Google is still keeping an advantage,” Chris Nicholson, Chief Executive of AI startup Skymind, says. “To build true enterprise applications, you need to analyze data at scale,” he adds. “At the execution stage, the open source incarnation of TensorFlow will run on phones as well as desktops and laptops, and Google indicates that the company may eventually open source a version that runs across hundreds of machines,” notes Metz, so the technology is something to keep an eye on in terms of AI and ML options.

So why the change of heart at Google? Well, part of it has to do with the very nature of how the machine learning community operates. “Deep learning originated with academics who openly shared their ideas, and many of them now work at Google—including University of Toronto professor Geoff Hinton, the godfather of deep learning,” explains Metz.

“TensorFlow was built at a very different time from tools like MapReduce and GFS and BigTable and Dremel and Spanner and Borg,” notes Metz. “The open source movement—where Internet companies share so many of their tools in order to accelerate the rate of development—has picked up considerable speed over the past decade. Google now builds software with an eye towards open source,” adds Metz. Many of Google’s earlier tools were just too closely tied to Google’s IT infrastructure to make them easily useful for outside developers.

Unlike its competitors, Google has not handed the open source project to an independent third party, but will manage the project itself at Tensorflow.org.135 The code is shared under an Apache 2 license, meaning anyone can use the code free of copyright issues.

Any goodwill this generates for Google is less important than the projects it could potentially feed. According to Dean, “you can think of TensorFlow as combining the best of Torch and Caffe and Theano. Like Torch and Theano, he says, it's good for quickly spinning up research projects, and like Caffe, it's good for pushing those research projects into the real world.”

However, even some within Google might disagree. “According to many in the community, DeepMind, a notable deep learning startup now owned by Google, continues to use Torch—even though it has long had access to TensorFlow and DistBelief,” notes Metz. But, the writer concludes, “at the very least, an open source TensorFlow gives the community more options. And that's a good thing.”

Even utilizing TensorFlow’s powerful AI and ML capabilities, building a deep learning app still requires some serious analytics and coding skills. But this too may change in the years to come, Metz adds. As Dean points out, “a Google deep-learning open source project and a Google deep-learning cloud service aren't mutually exclusive.”

For now, Google merely wants to generously share the code. As Dean says, “this will help the company improve this code.” At the same time, other benefits will result from this, including improving machine learning as a whole, which will undoubtedly find its way back to the souce—Google. The circle of code continues…

"Google is five to seven years ahead of the rest of the world,” argues Chris Nicholson, adding, “If they open source their tools, this can make everybody else better at machine learning.” 

At its F8 developer conference in San 2018, Facebook announced the launch of Caffe2, an open source framework for deep learning.[i] In his article Facebook-Open Sources Caffe2, a New Deep Learning Framwork, Jordan Novet explains that the announcement “builds on Facebook’s contributions to the Torch open source deep learning framework and more recently the PyTorch  framework that the Facebook Artificial Intelligence Research (FAIR) group conceived.”136 However, Caffe2 does have several differences from PyTorch.136

“PyTorch is great for research, experimentation and trying out exotic neural networks, while Caffe2 is headed towards supporting more industrial-strength applications with a heavy focus on mobile,” explains Yangqing Jia, Facebook AI Platform engineering lead.136 “This is not to say that PyTorch doesn’t do mobile or doesn’t scale or that you can’t use Caffe2 with some awesome new paradigm of neural network, we’re just highlighting some of the current characteristics and directions for these two projects,” notes Jia. “We plan to have plenty of interoperability and methods of converting back and forth so you can experience the best of both worlds,” adds Jia.

In their paper Caffe2 vs. TensorFlow: Which is a Better Deep Learning Framework?[ii], Baige Liu and Xiaoxue Zang focus on Caffe2 and Tensorflow and make compare five aspects of the software: the expressiveness, the modeling capability, the performance, help & support, and the scalability.137 The authors chose “TensorFlow because it is currently the most widely-used deep learning framework.”137 The authors recognize that Caffe was an extremely popular framework before TensorFlow was introduced and the Caffe2 framework can build upon that potential, while gaining a lot of user preference in the near future.137

Liu and Zang discovered that while Caffe2 and TensorFlow do not differ much in expressiveness, modeling capability, and scalability, Caffe2 significantly performs better than TensorFlow in both speed and space aspects, therefore it is a better choice for people who pursue speed or are limited by the device restrictions.137 However, “TensorFlow provides more services and tools, such as Tensorboard, TensorFlow serving, TensorFlow Lite,”137 and it has a strong advantages in help&support.137 “It is a better choice if people want to implement new or complicated models and do not know how to implement exactly yet,” argue Liu and Zang.137

 

[i] Novet, Jordan. (2017). Facebook Open Sources Caffe2, a New Deep Learning Framework. Venturebeat. 18 April 2017. https://venturebeat.com/2017/04/18/facebook-open-sources-caffe2-a-new-deep-learning-framework/ (Accessed 1 May 2018).

[ii] Liu, Baige, Zang, Xiaoxue. (2017). Caffe2 vs. TensorFlow: Which is a Better Deep Learning Framework? Stanford University. http://cs242.stanford.edu/assets/projects/2017/liubaige-xzang.pdf (Accessed 1 May 2018).

© 2017-2018 Intelligencia Limited. All Rights Reserved.

Contact

MACAU:

Rua da Esrela, No. 8, Macau

Macau: +853 6616 1033

 

HONG KONG:

505 Hennessy Road, #613, Causeway Bay

HK: +852 5196 1277

andrew.pearson@intelligencia.co