Jeffrey Luppes

Thoughts on the state of Machine Learning Engineering

2022-10-08T12:00:00+02:00

In 2020 I wrote this piece on becoming a ML Engineer. Since then a lot has changed. The job title of ML Engineer is now a relatively common one - a search in October 2022 for Data Scientist and Machine Learning Engineer openings in my home country, the Netherlands, showed the DS:ML ratio to be roughly 6 to 1. Prior to the title of ML Engineer becoming a thing there were obviously plenty of Data Scientists and Software Engineers doing ML Engineering; It simply wasn't called as such.

Perhaps the tools we used two to three years ago were a little bit more basic, despite already being more numerous than the first five generations of Pokémon put together. We've drifted towards consolidation, though not as much as I had hoped.

Originally I felt that eventually we'd all end up using the tools (e.g. Sagemaker, Vertex) on our cloud vendor of choice with perhaps some companies going for a platform like W&B or ClearML. The world of MLOps would be something that was easily surveyed and picking a platform would be a relatively straightforward choice. That hasn't really happened. While the large chunks of the simple parts of the MLOps work can be covered by platforms there are still areas where small, manoeuvrable companies are the best. Take for example W&B's experiment tracking and compare it to Sagemaker Experiments.

There are a lot of these features that you may need something for. Do you need semantic search? A feature store that actually works? Multi-GPU endpoints? An API to receive feedback from prod? A/B tests? An inference serving option that works well with graph neural networks? A way to monitor drift in literally any NLP system? It goes on and on and you're unlikely to find all of these features in the same platform.

For DPG Media, we eventually designed a platform backed on AWS Sagemaker, with some extra tools added for functionality we found lacklustre on AWS. Every new feature we needed outside of Sagemaker meant another tool, and that's how we ended up with AWS Sagemaker with about 7 other applications and platforms. Of course, at the time, there were just two of us, so we wanted to use managed services as often as possible so we could do other stuff as well. But opting for managed tools means less flexibility and (usually) more costs. It's a thin line between engineering and madness.

The class of 2015 - An analysis in what jobs the people I graduated with ended up

2020-03-25T12:03:38+01:00

Computer Science degrees are notorious for their attrition rate and a lot of people switch degrees. I've switched twice myself, going from CS to AI and then to Software Engineering. Due to the way the Dutch system is organized this almost meant starting from scratch. Frankly, not my best decisions. But I was young.

I figured that it could be interesting to look what everyone was doing now. Most people generally do find their way to the finish line and there is also a substantial part that starts working and doesnt finish their degree. Interestingly, the Hanze University of Applied Science had the same idea and scraped linkedIn profiles of all their alumni - assumingly me included. They found that the top 3 roles for software engineers were:

Developer
Software Engineer
Consultant

I decided to do the same, but the motivation for me is a little different than the Hanze. For GDPR reasons the Hanze can claim their possession of this data is justified: after all, they should know where their graduates end up so they can adjust their curriculum every few decades. I also did not need to redo their work and classify the size of the company someone worked at or the location (North or Randstad et cetera..)

Wait, where's that attendance list?

I still had a grades list from the end of year 1 at which point there were 100 students left. Having made it through the first year, most would go on and graduate. Since scraping is a bit out of the question here, I decided to hit up their linkedin myself (and often saying hi or sending a friend request). I'd only collect their job title from Linkedin, which I added to a CSV.

Sure enough, out of 103 students, there were 78 I could find on LinkedIn. I saw happy faces and a lot of heavy metal shirts. Some people had left IT altogether. Some were doing a master's degree. Some had started a company abroad. It seems that the people I could not find had might've done so to minimize their footprint online, especially not uncommon when one goes into security.

At this point I did some limited post-processing on this list. I removed terms such as medior or senior (side note: hardly anyone was a junior) and tried to merge the various security, consultancy, and dev-related roles together towards a standardized format. Application Developer became Software Developer, but .NET developers and embedded systems developers kept their title. This is a bit arbitrary and opens up this little adventure to bias.

I removed eight non-IT roles, such as caretaker, military, nurse and store manager. For double roles I tried figuring out which was more reasonable. For instance if someone had founded a succesful company but still listed themselves as an developer, I figured that the Founder role was more defining.

Findings

Little to no surprise was that the top three is pretty much the same still the same. The full list is below, but here are some things that stood out to me:

Only two people started a company
There are no data scientists
Likewise, there are no software engineers in BI roles.
Two people are currently doing an internship - most likely as a part of a MSc degree requirement.
There is exactly one Software Architect
In the same vein of requiring experience, there are a handful of managers, but mostly related to products and not teams.
There are several cloud engineers. Back in 2015, this role was pretty much unheard of.
Yours truly is the only Machine Learning Engineer to make the sample.

My curiousity is now satisfied, but not before I made a word cloud of this collection.

Job title	Count (n=70)
Software Developer	13
Software Engineer	10
Consultant	6
.NET Developer	3
Cloud Engineer	3
Full-stack Developer	3
Project Manager	2
Product Owner	2
Founder	2
Security Consultant	2
Network Engineer	2
Intern	2
Back-end Developer	2
Account Manager	2
Service Manager	1
Solution Developer	1
Embedded Software Engineer	1
Functional Application Manager	1
Mobile Developer	1
Systems Engineer	1
Front-end Developer	1
Machine Learning Engineer	1
Scrum Master	1
Web Developer	1
Field Engineer	1
Change Manager	1
Solution Experience Manager	1
Software Architect	1
DBA	1
Sysadmin	1

Building a bird detector from scratch with web scraping and deep learning (part 1)

2020-01-21T19:46:47+01:00

This is part 1 in a series on this project, more posts will be written as the project progresses

I'm an awful birder. While I've always been interested in birds I'm almost completely deaf to identifying them by their calls. I never started memorizing them and my ability to recognize them on the basis of visual cues is poor. It was only when I started kayaking (about five years ago) that I was exposed to more bird-watching, and during a trip to Schotland last year a couple of friends took me on my first bird watching trip.

So given that my human-based detection is obviously lacking I figured that with the plethora of bird data online, would it perhaps not be possible to scrape these and create a bird classifier using CNNs and general Python shenanigans?

Intro

I also wanted to make a funny trinket out of this and deploy it to a Pi. I'm lucky to have family that live on a farm in a rural area and have multiple feeding stations for the birds. Apart from the 30-or so bird species that they've actively tracked over the past year, there are also chickens, hedgehogs, mice, rats, and various predators like hawks and foxes that drop by. Some of the feeding stations are viewable from indoors, while others are between trees and bush. This will be the testing ground.

This is actually a very active research topic in the past couple of years, as ecologists, population biologists and researchers are using deep learning to automate detection. Consider that according to a May 2018 paper, roughly one third of papers on this subject was published in 2017 and 2018 [1]. Focus is particular on video and audio (e.g. bird calls).

As for approaches, it seems that my idea of CNNs and Pis is spot on. [2] Shows a CNN model with skip connections traind on 27 bird species in Taiwan can approach 99% accuracy. There is also another source that shows it is possible to use a Raspberry Pi for classification amongst three species [3]. And that's just scraping the surface: more advanced models are being developed. Further inspiration comes from Ben Hamm and his cat Metric [4]:

One particular website that's a huge hub for biologists and bird-watchers is the Dutch site www.waarneming.nl. It's worth mentioning waarneming employs their own suite of models and automatically classifies uploaded photos. In 2019, they received around eight million photos. With this data they've also put out their own deep learning-powered app: Obsidentify.

But why would anyone do this if there's already apps (and with that probably APIs) that do this? For starters, I like the idea and challenge of bringing this end to end. It's a learning experience for me. Secondly, I like to gift this set up to my in-laws at some point so to have some kind of automated tracking when there's something going on in the garden. Scientifically this is a valid topic: bird classification scales poorly as does droning over photos or video streams is a labour-intensive task.

Why is this a hard problem?

While the training data mostly contains photo's of birds zoomed up and resting on a branch, the real-life data is much more messier. Consider this:

Birds do, in general, not sit still long enough
There might be multiple birds and multiple species in a single shot
The size of a birds vary
Many bird species are dimorphic: males and females look different
Every bird species has different feeding strategies which result in different images, with some more inclined to frequent a station than others.
They might be in an odd point of view (e.g. viewing a bird from the rear)
Training data of the same species from different areas might be too different
The background (branches, fields) is very different from the feeding stations. Consider the "wolf vs background" problem where an AI system trained to distinguish wolves was actually picking up on the presence of snow in the training image to determine whether the photo was of a wolf or dog instead of the actual subject. This is a valid problem in this context
There might be a number of branches and foilage in the way of the birds (consider, for instance the header image of this post)

As for actually doing it: - Training a neural network simply requires a ton of labelled data. - There is a huge difference between photos shot with a telescopic lens of say 250mm+ and a wide-angle over the counter camera such as a the pi camera - The quality of these webcams, action cams and the pi-camera might simply be too bad to do this - Some excellent training data might explictely forbid usage (licence) - Some of the images available are simply cell-phone camera shots through a telescope or of a DSLR-display. - In order to adequately classify birds, I have to recognize (a) an image contains a bird (a binary problem), (b) find where in the image the bird is and predict a bounding box around it and (c) classify the bird.

In all, there's a large potential for a miss-match between the training data and the real-world case I'm trying to work on.

Act 1: Identifying training data

Waarneming.nl has photo's publicly available and the data is relatively easy to access. Since the goal environment (a farm in the Netherlands) and the training data environment (most uploads are from Belgium and the Netherlands) are the same, this was my first choice for training data.

There's also Flickr which even publicly exposes an API for these goals. This might be the best / easiest option if you're not looking for data from Europe. eBird seems to be scrape-able the same way. While searching for 'Staartmees' on Flickr yields only about 2000 hits, checking for 'Long-tailed tit' gives around 62.000 photos.

Lastly, there are two datasets that might be of use. First there's the Caltech-UCSD Birds-200-2011 data set of almost 12000 photos and 200 species. This information may be useful later on for detecting a bounding box (a square to identify where in an image a bird is located) around a bird. Similarly, there is the Japanese Wild Birds in a Wind Farm: Image Dataset for Bird Detection data set for detection, although this is again only useful for detection and not classification - an important distinction.

Act 2: Scraping training data

I set out to scrape the data from waarneming.nl as this is the absolute closest to my real-life use case and the image quality is insanely high, being mostly from birders with professional gear. Futhermore, as the data is community-sourced and often verified, I can directly treat the image labels as based in truth (something that would not be possible with say Flickr). I turned to the gallery:

Waarneming has a gallery function that lists 24 bird photos each time. Despite there being more images on the site than the gallery, only the gallery has the app-ratio-box-image class, so this allows us to collect only these links.

Also note that the html includes links to the image (ending in .jpg) as a page describing an image (ending with photos/<image_id>/. This leads to a page that has meta data on the image - more on that in a bit.

Based on the species list (below) I looked at querying for each specific bird. However, there are many possible species and a simple search might return multiple possible hits (subspecies might be returned). Since I'd have to make a list of each species' latin name anyway, I just searched for an individual species and collected the species' id. I stored these in a dict like below:

{
    'species_name': 'species_x',
    'id': 1
}

For each species in the list I crawled the site. I could not find a robots.txt that disallowed web scraping and the license found on https://waarneming.nl/tos/ explictely permits non-commercial use by individuls. However, because the strain on a website can be considerable it is a good practise to build in a pause of 1 second between requests. I did not initially do this as I had honestly had not considered the strain on their servers.

The below script accomplishes the main scraping goals. I made use of the beautifulsoup and requests libraries to collect the html from the pages and parse them. I also included a call to check which images I already have downloaded, so I don't waste resources fetching the same image twice.

for i in range(1,int(MAXIMAGES/IMAGESPERPAGE)+1):
    # construct the url
    URL = 'https://waarneming.nl/species/'+identifier+'/photos/'
        '?after_date=2018-01-01&before_date=2020-01-19&page='+str(i)

    # fetch the url and content
    page = requests.get(URL)
    soup = BeautifulSoup(page.content, 'html.parser')

    #find the images
    tags=soup.findAll('img',{"class":"app-ratio-box-image"})
    photolinks += tags

    #pause for one second out of courtesy
    time.sleep(1)

# get the photoids we already have scraped from before
photoids = get_photoid_list(species)

metadata = []
# download photos and store them in their new home
for link in photolinks:
    #url without arguments
    url = link['src'].split('?w')[0]
    #obtain filename from url
    filename = url.split('/')[5]

    #check if we have encountered this photo before - will be substantially slower with large n
    if filename.split('.')[0] not in photoids: 

        # we have a new photo, so lets check the metadata first
        meta = get_metadata(filename.split('.')[0])

        if meta:
            meta['photoid'] = filename
            metadata.append(meta)

            if meta['Licentie'] in ALLOWED_LICENSES:
                path = RAWFOLDER+'/'+species+'/'+filename
                get_and_store_image(url, path)

                #also resize the image and store them seperately
                outputpath = PROCESSEDFOLDER+'/'+species+'/'+filename.split('.')[0]+'.png'

                convert_and_store_img(path, outputpath)
                new_photos += 1

                #pause for one second out of courtesy
                time.sleep(1)

The above code includes calls to the following methods and variables: get_photoid_list searches the species directory for any existing photos (because there's no point in downloading the same photo twice if we re-run the script) ALLOWED_LICENSES contains a list of licences that allow us to scrape. get_and_store_image requests the image from an url and stores it to local disk get_metadata requests the page that holds info on a particular photoid and outputs the photo details to a meta object. This way, we can trace which photo's we crawl and who made them, but also the license for an individual photo. convert_and_store_img changes the photo to a set x by y size and also appends each image so all the training data has the same dimensions. time.sleep(1) tells the scrape script to pause for one second. metadata is a list of meta objects I store to disk afterwards.

The attentive reader might notice we're also collecting metadata on images we don't scrape. This is mainly because I want to trace the different licences and other metadata associated with them and to verify that everything works correctly. The metadata is stored to a flat file. Below is the get_metadata call:

def get_metadata(photoid=24691898) -> dict:
    '''Given a photo-id, return metadata in a dict'''

    url = 'https://waarneming.nl/photos/'+str(photoid)
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html.parser')
    tags=soup.find('table',{"class":"table app-content-section"})
    meta =  {}

    if tags:
        # find all table rows
        rows = tags.find_all('tr')
        values = []
        keys = []
        # get the table content and return as two lists
        for row in rows:
            # actual content is listed in the <td>, while <th> holds the keys. 
            descriptions = row.find_all('th')
            cols = row.find_all('td')
            for idx, ele in enumerate(descriptions):
                keys.append(ele.text.strip())
                values.append(cols[idx].text.strip())

        #create a dict out of the data we fetched
        meta = dict(zip(keys, values))  

    return meta

...which returns the following:

The data is contained in a table, and each row only contains a <th> (header) and <tr> (row). Thus, we obtain the keys from the <th> and the values from information enclosed by the <tr> tags. Note that we fail elegantly: if there's nothing in the table (or the table is not there) we return an empty meta. And if we do not have the meta object explictely stating what kind of value we have for a license, we don't download the photo.

With that, we're all set to start scraping. Let's define what birds we want to look for.

Species list

The below lists the species I collected data about.

#	Latin NamE	English Name	Dutch Name
1	Cyanistes caeruleus	Eurasian blue tit	Pimpelmees
2	Parus major	Great tit	Koolmees
3	Aegithalos caudatus	Long-tailed tit	Staartmees
4	Lophophanes cristatus	European crested tit	Kuifmees
5	Fringilla coelebs	Common chaffinch	Vink
6	Turdus merula	Common blackbird	Merel
7	Sturnus vulgaris	Common starling	Spreeuw
8	Passer montanus	Eurasian tree sparrow	Ringmus
9	Passer domesticus	House sparrow	Huismus
10	Emberiza citrinella	Yellowhammer	Geelgors
11	Prunella modularis	Dunnock	Heggenmus
12	Certhia brachydactyla	Short-toed treecreeper	Boomkruiper
13	Chloris chloris	European greenfinch	Groenling
14	Sitta europaea	Eurasian nuthatch	Boomklever
15	Erithacus rubecula	Robin	Roodborstje
16	Dendrocopos major	Great spotted woodpecker	Grote Bonte Specht
17	Pica Pica	Magpie	Ekster
18	Carduelis carduelis	European goldfinch	Putter
19	Troglodytes troglodytes	Eurasian wren	Winterkoning
20	Turdus philomelos	Song thrush	Zanglijster
21	Columba palumbus	Common wood pigeon	Houtduif
22	Streptopelia decaocto	Eurasian collared dove	Turkse Tortel
23	Columba Oenas	Stock dove	Holenduif
24	Motacilla Alba	White wagtail	Witte Kwikstaart
25	Picus Viridis	European green woodpecker	Groene Specht
26	Garrulus glandarius	Eurasian jay	Gaai
27	Fringilla Montifringilla	Brambling	Keep
28	Turdus Iliacus	Redwing	Koperwiek
29	Turdus Pilaris	Fieldfare	Kramsvogel
30	Oriolus Oriolus	Eurasian golden oriole	Wielewaal

Pre-processing data

In order to re-shape the data into a format that a ML model can interpret I performed the following steps.

1) I Resized the image to have a max dimension of 256 by 256 2) Centered the image and padded the sides wherever it was less than 256 3) Cut a 224 x224 section from the middle of the image

This builds heavily on the python version of the opencv library as well as numpy. The result is a 224 by 224 photo from any input photo.

The resizing of images might not be needed for every possible network, but it might be important to some convolutional architectures. I chose 224 by 224 because these are the dimensions that some of the pre-trained networks available in Keras - such as vgg16 [5] - work with. While it would be massively better for training time to pick a smaller size (say 32x32), the bird is generally only a small portion of the pixels in the image. I fear that if I was to limit the size too aggresively I'd limit the usefulness of my training data too much.

The below code snippet shows how the resizing and centering is done.

def center_image(img):
    '''Convenience function to return a centered image'''
    size = [256,256]

    img_size = img.shape[:2]

    # centering
    row = (size[1] - img_size[0]) // 2
    col = (size[0] - img_size[1]) // 2
    resized = np.zeros(list(size) + [img.shape[2]], dtype=np.uint8)
    resized[row:(row + img.shape[0]), col:(col + img.shape[1])] = img

    return resized

I am actually trimming the edges a little as I figured that the bird in any given photo from this data set is likely in the middle of the photo.

#resize 
if(img.shape[0] > img.shape[1]):
    tile_size = (int(img.shape[1]*256/img.shape[0]),256)
else:
    tile_size = (256, int(img.shape[0]*256/img.shape[1]))

#centering
img = center_image(cv2.resize(img, dsize=tile_size))

#output should be 224*224px for a quick vggnet16
img = img[16:240, 16:240]

The actual scraping

So now I have methods defined for going through the gallery, collecting the links along the way. For each link I store the metadata and look up whether the licence is in my ALLOWED?LICENCES list. Roughly 60% of all photo's are blocked by a licence. If there´s a match I download the image.

With the 30 bird species I defined earlier it´s just a simple list of dicts to go through for scraping. Here bird_scraper takes a bird name (used for constructing a folder) and an id to query with:

for s in species:
    bird_scraper(s['name'], s['id'])

    # show a random photo to brighten the day
    print(str(s['name'])+':')
    show_random_img_from_folder(RAWFOLDER+'/'+s['name'])

In the end I show a random image from the samples I collected. I figured that this would be a nice screenshot to include because it demonstrates three problems: * The bird versus background problem (when we are training, are we actually modelling the birds or are we overfitting on their habitat?) * The odd angle * The bird itself might only be a very small portion of the photo

Now, let's see how many photos we captured..

Act 3: Results

The total number of bird photos I scraped per species is shown in the graph below. Apparently the common birds are all a bit even, while I could not query for more than 90 Groenling photos. This might be a bug or some kind of anti-scraping measure, as I always thought these birds were fairly common.

Discussion

After starting out I decided to limit the scraping to a max of 4000 samples per bird species, which I further brought down by instead of scraping until I had 4000 images going for 4000 images and just using how many images were allowed as training data. That meant that I scraped roughly half of that number. My goal is not to build the best possible system, but only 2000 samples per bird will mean its likely difficult to train a model.

While most data scientists and like will argue that scraping is perfectly legal, the legality of web scraping licenced material is much more a grey area. Roughly 60% of the data is licensed with a 'no derivative' variant or 'all rights reserved'.

In total, my scraping netted in 60.149 usable images across 30 species. While some classes have a ton of samples (also because I started off with requesting more than I needed), others have much less samples available. This skew in the data should probably be addressed while constructing a model in the next post in this series.

Disclaimer: this blog post details a project that's still very much underway. I intend to retrain the models before I deploy the 'final' models and further refine the technique, and then also update this blog post. The header image is one of my own photos and I will strive to include many as my own photos as test data.

This blog post benefited from ongoing discussion with experts from a variety of backgrounds. I would like to thank a number of people: Linde Koeweiden, Jobien Veninga and Johan van den Burg. I would also like to thank the people behind Waarneming for their interest in my project and pointing out issues with my approach.

Papers referenced in this post: [1]: Christin, S., Hervet, E., & Lecomte, N. (2019). Applications for deep learning in ecology. Methods in Ecology and Evolution, 10(10), 1632-1644. [2]: Huang, Y. P., & Basanta, H. (2019). Bird image retrieval and recognition using a deep learning platform. IEEE Access, 7, 66980-66989. [3]: Ferreira, A. C., Silva, L. R., Renna, F., Brandl, H. B., Renoult, J. P., Farine, D. R. & Doutrelant, C. (2019). Deep learning-based methods for individual recognition in small birds. bioRxiv, 862557. [4]: Cats, Rats, A.I., Oh My! - Ben Hamm: https://www.youtube.com/watch?v=1A-Nf3QIJjM [5]: Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

Google Cloud Platform Certified Data Engineer

2019-12-22T22:42:45+01:00

Whoooo! I passed the GCP data engineer exam. In truth, I am not really a data engineer.. but it is a role I can fill and I strongly feel that as a data scientist you should be able to have a solid grasp of engineering too, or at least understand it well enough to know what the challenges are. The exam was about 2 hours long and is supposed to be roughly equivalent to the AWS professional level certifications. Funny enough, it actually covers a lot of the ML engineering topics. Perhaps there's a ML exam in the works?

*Update: As of 2020 there is now a GCP Machine Learning certification. I passed it in 2021. You can read more about it here

I suppose I got lucky with the matter. I've been working with GCP for about a year now and I've mostly been involved whenever there's a ML demand.

As for studying strategies: I had a four-day training back in August. After that, I spent about 20 hours doing coursera courses and about 16 studying with the exam guide as my guideline. Particularly the last part was vital.

You can check out the certification here: https://www.credential.net/a015b522-77d3-48c0-b7f7-16c9948a9ac4

Big Data Series

2017-09-02T22:42:45+02:00

I've just added three blog posts I made during the Big Data bachelor course given at the Radboud university. As a master's student I'm allowed to take on one or two bachelor courses if there's a good reason... because no other course really goes into Spark, hadoop and Scala I figured it would be a nice addition to the Python-heavy curriculum. Not that I dislike Python, of course.

There are three posts in total:

Hadoop and the HDFS - an introduction to hadoop and HDFS. Spark - On looking at a Kaggle competition data set in Spark The class project: A solo project about submitting code to a national research cluster and running queries against 1.73 billion web pages.

You can find the posts here: Big Data Series

I learnt a lot and finished the class project with a 9.5, so hoped to share it.

Big Data Series - SurfSara and the Common Crawl

2017-07-07T23:03:48+02:00

This post will have a slightly different angle than the previous posts in the Big Data Course series. The goal for this post is just to detail my progress on a self-chosen, free format project which utilizes the Surfsara Hadoop cluster and the goal is not to solve a problem but rather give an overview of the problems I encountered and the little things I came up with. I intend to post these both on the mini-site for the course and a personal blog, my apologies if my tone is a bit bland as a result. Here we go!

Hathi and Surfsara

SurfSara is a Dutch institute that provides web and data services to universities and schools. Students may know SURF from the cheap software or the internet they provide to high schools. Sara, though is the high performance computing department, and used to be the academic center for computing prior to merging into SURF. They do a lot of cool things with big data which over time has come to include a Hadoop cluster named Hathi.

The Common Crawl

The Hathi cluster hosts a February 2016 collection from the Common Crawl. The Common Crawl is a collection of crawled web pages which comes pretty close to crawling the entirety of the web. The data hosted is in the petabyte range, however we only have access to a single snapshot.. which still takes up a good amount of terabytes and contains 1.73 billion urls. You don't want to download this on your mobile phone's data cap.

The Common Crawl Data is stored in WARC files (Web Archive), an open-source format.

So with all this data, there should be a lot of things to do!

Some ideas I had at this point:

Count the length of all payloads across all pages on the internet and get some statistics.
See how popular certain HTML tags are.
Perform some semantic analysis on pages referring to presidents or politics.
Look at how extreme right communities differ from extreme left communities in terms of vocabularies and word frequencies.
Similarly, compare places like 4chan and Reddit with each other. Who's more vile? There's some easy libraries for sentiment analysis..

And so on.. but what I also played with is something closer to home. I kayak a lot and the kayaking community in the Netherlands is slowly dying: young people are turning away from adventurous sports in general, and kayaking is seen as boring when compared to other, fast-paced water sports (Not necessarily true, but still). Could I try to find places where it's worthwhile to advertise about kayaking perhaps? Or identify communities of people who also kayak, e.g. mountain bikers, sailers, bikers etc? Or perhaps from another perspective, can I try to do some dynamic filtering based on brands or parts of the sport to see what people associate it with?

Plenty of ideas, so let's get started.

Part 2 - Setting up

I'm using my Windows laptop running a (Ubuntu) virtual machine which will be used to connect to SurfSara and develop the code. Similarly to the previous assignment in this series this works with a docker image and lots of command line work. Nothing to be scared of.

Running an example program worked fine on the cluster. But I wanted something more than (redirectable) output in my terminal.

In order to track the jobs given to Hathi a web interface is available. This is not really supported on Windows, but still doable. Using the Heimdall implementation of Kerberos and the Identity Manager I can set up my credentials. I found that I needed to stray from the sort-of specific instructions courtesy of the Uni of Edinbourgh here and actually ended up installing the Heimdall tools fully. I then had to tweak a couple of configurations inside my firefox browser in order to work with Kerberos, but I finally could inspect the progress of my submissions. This seems easy, but in the end was a non-trivial part that took hours to do and even then Firefox was prone to memory leaks.

Part 3 - A local test

I started working with the spark notebook that was provided and after some tweaking around I could run code on a local WARC file containing the course website. This was an iterative process: I started with the grand idea of what I could do but after a few hours I found that I still had made no progress. Following Arjen's suggestion of settling for a simpler challenge when stuck I tried to implement the most basic word count. This was OK-ish, and could be expanded to the full crawl albeit a bit sluggish (slow), which would be decent towards meeting the assignment criteria but I'll let you be the judge of that.

I also avoided SQL this time, as I recall reading that there are some issues when running SQL-queries on something of the order of 100TB. This could complicate things considering our 'stack' already consists of so many applications and tools. Additionally, having worked with MySQL as a teenager I'm still pretty sure that straight up SQL queries on non-indexed text fields is a baaaad idea.

I felt like I still really wanted to do more with the kayaking thing. After some pondering I settled on the following order of battle:

Convert the crawl to text and look for the string kayaking
For the full crawl: figure out how to filter for a specific brand (e.g. bever)
Construct a word count list upon the pages that get returned
Output these to a file so I can work with it
Visualize a word cloud, e.g. using the d3.js method already readily available, or something in python (This is outside the scope of this assignment, and I'll add it later)

So I started with filtering for the text string kayaking after calling Arjen's HTML2Text method (step 1).

    map(wr => HTML2Txt(getContent(wr._2))).
    map(w => w.toLowerCase()).
    filter(w => w.contains("kayaking")).

Now on the basic corpus this returns my own page obviously, as I overshared my love for kayaking a fair bit.

As per Apache Spark's example a word count is implemented with just a few lines of code (step 2):

    textFile.flatMap(line => line.split(" "))
                     .map(word => (word, 1))
                     .reduceByKey(_ + _)

This is a bit crude, as the "words" will include code snippets like the one found on this blog, and random gibberish like solitary punctuation marks. For a full pass over the crawl though I don't think it'll matter, as full words will drown out the noise.

So now I've got a big old list of words with counts. Can I save this? Locally, I can use:

warcl.saveAsTextFile("testje.txt")

I'm guessing this will be different for the full crawl but one problem at a time. This creates a folder (!) with several files: output can be found in one file here. It's interesting that everything to save into a text file was done below the hood without a warning being thrown at my face for not saving to a hdfs!

There are some caveats with this: * In the presentations some people noticed an integer overflow when using word counts, can I figure out something for this? * I need to filter out common words such as "a", "the" and so on. I can do this at a high level or when making the visualisation later on. Will save the problem for now.. * Between the docker container and my Ubuntu host I found that I can copy files using docker cp. What if my files are big, though? And what happens on the full crawl. Write to standard output and just do everything on the cluster? * May I need to purge tags and code from my result file? * How can I easily scale this up to.. say looking at 20 brands at once?

As was shown in the terminal above, it doesn't make sense yet to construct a word cloud from a single page, I suppose though that the same steps go for the full cluster. Let's move on and see if we can export the code to the full crawl!

Part 4 - From Concept to Cluster

The following section will detail the process I went through when exporting the app to the cluster.

Attempt 1 - Top 300 words over all sites containing "Kayaking"

The first attempt is going to go over the entire crawl and look for the term 'kayaking' amongst the payload of all sites. I see some potential issues with this.. mainly because I'm asking for the entire crawl to be parsed through html2text - I reckon that is going to be an immense bottleneck.

The core idea is explained in the following two code snippets..

val warcc = warcf.
    filter{ _._2.header.warcTypeIdx == 2 /* response */ }.
    filter{ _._2.getHttpHeader().statusCode == 200 }.
    filter{ _._2.getHttpHeader().contentType != null }.
    filter{ _._2.getHttpHeader().contentType.startsWith("text/html") }
    .map(wr => HTML2Txt(getContent(wr._2)))
    .map(w => w.replaceAll("[?!,.\"-()]", ""))
    .map(w => w.toLowerCase())
    .filter(_ != "")
    .filter(w => w.contains("kayaking"))
    .cache()

The above snippet checks for non-empty input and skips it. It should be refactored, but I'm still working on more ideas so I felt it should not be a big priority right now.

It also checks for odd characters, e.g. ?! et cetera- we don't want any of that sillyness.

Finally, this big pile of text needs to get filtered for the phrase kayaking - I expect this line just after HTML2Txt to be a huge bottleneck.

The next snippet does the standard MR word count. I've added a sort and top-300 selection.

//now to construct a list with anything in the pages we have found per spark word count example https://spark.apache.org/examples.html
val warcl = warcc.flatMap(_.split(" "))
     .filter(_ != "")
     .map(word => (word, 1))
     .reduceByKey(_ + _)
     .filter( w => !(commonWords.contains(w._1)))
     .sortBy(w => -w._2)
     .take(300)

Lastly, this gets printed to the output.

Filtering for Common words

I browsed around for a solution to the common word problem, as I didn't feel like editing my top 300 list every time. So I found this stackoverflow question about filtering words out of my input, by means of a sequence.

So I still needed a sequence at that point, and I found this list of English stop words.. which brings me to wonder if I'm going to see other languages pop to the top of the list. One problem at a time though. For clarity, here's the complete list.

val commonWords = Set("a", "about", "above", "above", "across", "after", "afterwards", "again", "against", "all", "almost", "alone", "along", "already", "also","although","always","am","among", "amongst", "amoungst", "amount",  "an", "and", "another", "any","anyhow","anyone","anything","anyway", "anywhere", "are", "around", "as",  "at", "back","be","became", "because","become","becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", "between", "beyond", "bill", "both", "bottom","but", "by", "call", "can", "cannot", "cant", "co", "con", "could", "couldnt", "de", "describe", "detail", "do", "done", "down", "due", "during", "each", "eg", "eight", "either", "eleven","else", "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", "fifteen", "fify", "fill", "find", "five", "for", "former", "formerly", "forty", "found", "four", "from", "front", "full", "further", "get", "give", "go", "had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his", "how", "however", "hundred", "ie", "if", "in", "inc", "indeed", "into", "is", "it", "its", "itself", "keep", "last", "latter", "latterly", "least", "less", "ltd", "made", "many", "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "move", "much", "must", "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own","part", "per", "perhaps", "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "serious", "several", "she", "should", "show", "side", "since", "sincere", "six", "sixty", "so", "some", "somehow", "someone", "something", "sometime", "sometimes", "somewhere", "still", "such", "system", "take", "ten", "than", "that", "the", "their", "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "thereupon", "these", "they", "third", "this", "those", "though", "three", "through", "throughout", "thru", "thus", "to", "together", "too", "top", "toward", "towards", "twelve", "twenty", "two", "un", "under", "until", "up", "upon", "us", "very", "via", "was", "we", "well", "were", "what", "whatever", "when", "whence", "whenever", "where", "whereafter", "whereas", "whereby", "wherein", "whereupon", "wherever", "whether", "which", "while", "who", "whoever", "whole", "whom", "whose", "why", "will", "with", "within", "without", "would", "yet", "you", "your", "yours", "yourself", "yourselves", "the")

Getting the code to Hathi

So right now I have a very basic and simple example Scala app which is confined to the notebook. I still need to do some house keeping in order to get it on the cluster.

The first step is exporting it to scala. This opens the file in my browser. I stored the file in a public location on the web (so I could get it via wget from the docker and pushed my updates to it - this allowed me to edit the file using the tools on my own machine and pull it when I want to run it on the cluster. This greatly reduced my effort by reducing my dependency on tools like vim - which, while excellent, do not have the range of capabilities like atom or VS Code do. Again, personal preference.

I then used the skeleton from the example app on the hathi-surfsara image, replacing the original file and deleting the /target/ folder. I made sure to follow the steps needed in the creating a self-contained app tutorial: which meant stripping some code and defining a main method. Additionally, I added jsoup to the libraryDependencies.

Using sbt assembly I then created a fat jar (stored in /target/) and submitted it via

spark-submit --master yarn --deploy-mode cluster --num-executors 300 --class org.rubigdata.RUBigDataApp /hathi-client/spark/rubigdata/target/scala-2.11/RUBigDataApp-assembly-1.0.jar

So for the next 1 minute 40 seconds I was thrilled! Hathi picked up my submission and seemed happy to do it. Then I got a nullPointerException.. turned out I was checking for the contentType before even checking if this wasn't null instead of the other way around.. eager beaver. I had the bright idea to implement a check for it, but did so in the wrong order.

The next big error was regarding my use of saveAsTextFile. Because this would be called many times (once per warc file?) I would get the error that the folder already existed. I took the saveAsTextFile out, and redirected output about the top 300 to the stdout instead.

After this small fix the code was submitted and I went to bed..

After 8 hours, 36 minutes and 45 seconds my code apparently hit an error: potentially having to do with a block being unavailable on the cluster. Just as I was rolling over hugging my pillow, the little cluster named Hathi was in tears. Had it failed the user, or had the user failed it?

User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1263 in stage 0.0 failed 4 times, most recent failure: Lost task 1263.3 in stage 0.0 (TID 3214, worker168.hathi.surfsara.nl, executor 256): org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-16922093-145.100.41.3-1392681459262:blk_1262268700_188594112 file=/data/public/common-crawl/crawl-data/CC-MAIN-2016-07/segments/1454701146196.88/warc/CC-MAIN-20160205193906-00216-ip-10-236-182-209.ec2.internal.warc.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) 
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) 
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) 
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at java.io.DataInputStream.read(DataInputStream.java:149)

I tried to google the error, but found nothing that I could do as a normal user of the cluster. Most of these had to do with missing privs (might be possible) or corruption.

Link to the application details

I've posted an issue, meanwhile I'm going to run it on a single warc segment.

Attempt 2 - One segment

Using the index I've looked for https://www.ukriversguidebook.co.uk/ - a large internet community of kayakers. This gives me a neat JSON output containing the locations of all hits. I just picked one- and added it to my code as "/data/public/common-crawl/crawl-data/CC-MAIN-2016-07/segments/1454701148402.62/*". The rest of the code remained unchanged for the reproducibility of errors. I submitted it with 300 executors and went to get a shave.

15 minutes and 15 seconds later the submission was done, much to my surprise. I had covered 698 tasks. Bear in mind this submission was 1% of the entire crawl, and I stomped through with 300 executors. No error was given, and my glorious output was waiting for me.

The following screenshot shows the inside of the Applicationmanager just after starting. Honestly, glancing over this felt like being inside mission control at NASA.

Now the curious reader will want to know.. what did we get from this?

Earlier I redirected output to stdout: this is where my little frequency list ended up.

womens;849513;
ski;774168;
jackets;738558;
-;703664;
 ;673135;
clothing;614007;
snowboard;581317;
accessories;492069;
pants;475875;
bike;467037;
bikes;463572;
mens;460966;
bags;456382;
shop;430335;
sunglasses;426637;
shoes;423502;

There's still some noise. Apparently I missed a white space and '-'... oh well.

This list seems to indicate that most websites referring to kayaking sell clothing and gear for outdoor activities. That makes sense, given that this is a huge industry with many competitors. Perhaps it would be a good idea to create a second list with words common in retail. It's interesting that words like sea and nature don't appear at all. The word safety - which is at the heart of the sport is ranked #273. Perhaps this is just a batch with a lot of retail sites, but it seems like a decent idea to mine retail terms in order to filter them out for the next iteration.

So I started to work and added another 150 words to the list with all those retail phrases. I refined the method and submitted the jar once more. Nothing was really different apart from a little retouching. Again, the code worked fine and I got a new list!

I then wrote a little bit of javascript to convert the frequency list I had to a payload that could be used for a word cloud (credits: https://github.com/wvengen/d3-wordcloud) and generated the visualization.

The word cloud is pretty cool. Most of the junk has been filtered and we see a lot of sports and outdoor-related terms. I guess that the market for kayaking is the same as the market for bikes and wakeboarding. As a mountain biker myself this is amusing. It also shows Wisconsin. This might be random, but the American state also borders lake Michigan and other large lakes and rivers.

Attempt 3 - Selective filtering, and finding brands!

Lastly I wanted to filter this subsection for specific brands. While I could easily create a list of 50 or so brands of varying popularity I chose Rockpool. Rockpool is a manufacturer of sea kayaks with several models being extremely popular in the expedition kayaking scene. In a year or so, when I graduate.. you can pretty much guess where my pay check is going to. Look at this boat!

Jokes aside, let's find the same word list as for kayaking. I added a brands set at first, but that didn't work out quite well. While I could iterate through it with the following code..

val brands = Set("p&h", "valley", "rockpool", "peakuk")
for (brand <- brands) {
    println(brand+"\n")
        val warcd = warcc /*temp*/
        warcd.cache()     /*lazy evaluation*/
        val warcl = warcd.filter(w => w.contains(brand))
            .flatMap(_.split(" "))
            .filter(_ != "")
            .map(word => (word, 1))
            .reduceByKey(_ + _)
            .filter( w => !(commonWords.contains(w._1)))
            .sortBy(w => -w._2)
            .take(300)

I would continously narrow down my collection. E.g. the first brand would go fine, but the second brand would be filtered from the subsection of the first brand and so on. This is due to Spark's Lazy Evaluation^tm where nothing is actually executed until a reduce operation- and in my code I only used reduceByKey until the end of each brand-specific execution. Regardless, being my favourite kayak manufacturer I chose Rockpool and got the following list:

{text: 'nov', size: 439},
{text: 'mar', size: 407},
{text: 'jan', size: 382},
{text: 'dec', size: 379},
{text: 'apr', size: 369},
{text: 'feb', size: 363},
{text: 'oct', size: 362},
{text: 'jul', size: 355},
{text: 'jun', size: 352},
{text: 'sep', size: 338},
{text: '2006', size: 282},
{text: 'aug', size: 265},
{text: 'ago', size: 216},
{text: 'stay', size: 214},
{text: 'cottage', size: 214},
{text: '13', size: 202},
{text: 'loch', size: 201},
{text: '12', size: 199},
{text: '16', size: 194},
{text: 'holiday', size: 194},
{text: 'house', size: 193},
{text: '21', size: 189},
{text: '11', size: 188},
{text: '17', size: 188},
{text: '20', size: 186},
{text: '22', size: 185},
{text: '15', size: 181},
{text: '14', size: 179},
{text: '27', size: 176},
{text: '19', size: 174},
{text: 'night', size: 174},
{text: '28', size: 169},

While some words are close (e.g. loch) it seems we picked up a lot of calendar or blog contents. After some manual (I'm not going to run this on the cluster and wait another 20 minutes) removal of the nonsense I got the following list:

{text: 'ago', size: 216},
{text: 'stay', size: 214},
{text: 'cottage', size: 214},
{text: 'loch', size: 201},
{text: '16', size: 194},
{text: 'holiday', size: 194},
{text: 'house', size: 193},
{text: 'night', size: 174},
{text: 'home', size: 167},
{text: 'details', size: 165},
{text: '18', size: 157},
{text: 'view', size: 156},
{text: 'min', size: 155},
{text: 'book', size: 155},
{text: 'great', size: 137},
{text: 'views', size: 137},
{text: 'away', size: 127},
{text: 'sea', size: 126},
{text: 'reviews', size: 125},
{text: 'close', size: 118},
{text: 'years', size: 117},
{text: 'sleeps', size: 114},
{text: '5/5', size: 110},

This is more like it. I kept the 16 and 18 as they are both kayak models. Overall, I pruned about 50 words- I might add a regular expression on my next run on the cluster. However, something like 5/5 (a rating, included in the list above) might get lost unintentionally. The word cloud on 'Rockpool' is as follows:

The only downside to this is the small corpus I get. Even though I used 1% of the common crawl, most of the words appear about 200 times. I wish I could run it again to get more data, but I do not want to drain up the entire cluster for a entire day.

EDIT: Full crawl!

I re-submitted the first job that went over the entire crawl. This time I used the retailWords list, as well as filtering for pages that also contained the word sea. I opted to get the top 1000 words instead. The submission was succesful and ended after 10hrs, 47mins, 12sec. In total 69800 jobs were queued. The top 20 words on the entire crawl are:

    'snowboard' 58172829
    'accessories' 47357717
    'bike' 46462542
    'bikes' 45888592
    'shop' 40646038
    'country' 28916820
    'water' 27793604
    'cross' 26669810
    'casual' 26185575
    'gear' 22752323
    'wisconsin' 21410085
    'wakeboard' 21104545
    'travel' 19564475
    'packages' 18880893
    'hiking' 18660510
    'forum' 17253484
    'helmets' 16881583
    'royal' 16747143
    'bindings' 16617302
    'house' 15067835

And the resulting word cloud is as follows:

That concludes this blog post!

Part 5 - Evaluation

In the above post I walked you through my adventures with the Common Crawl and the Dutch National Hathi Hadoop Cluster. I started off with basic examples and tried to solve my own problems as I went. Eventually I formed the idea of generating a word cloud based on the term kayaking. When it apparently was not possible to make a pass over the entire crawl I grabbed a 900 GB partition and worked with 1% of the data. My idea was still to look for how individual brands are viewed: e.g. what words are asociated with brands like Rockpool? Finally, I used javascript and the d3.js library to generate word clouds of my findings.

Though I feel like I had to water down my challenges I feel like there's a lot of things that I can still do with all this data. I'm still in unfamiliar territory and I learnt more each time. I'm still working on this project and I'd like to continue building a few interesting vizualisations. I'm glad I didn't do the standard project, and it just feels better to try out many different things and get something of yourself out of a project like this.

Overal I spent about 40 hours or so on this project.

Dear reader, I thank you for your interest in this blog.

Part 6 - Course Evaluation

Though I already submitted the course evaluation I felt like it would be nice to include a few words on the process I went through for this course. I feel that using git, github and in particular the github pages tool - were enriching and powerful. I'm planning on including this repo with my own website, although I havent updated the latter in over two years. If I was a 2nd or 3rd-year student however, this would have instantly given me a portfolio of sorts which is incredibly useful to have.

The way we went about it, trying to document our struggles with the various tools as we go is much closer to reality than just handing in a polished report that gets written after the product is already done. I would have liked more structure up front to combat the hours of troubleshooting I suppose, but in the end it turned out fine with just the support I found in the issue tracker and google.

Speaking of the issue tracker, I think this was a great addition to the course and I hope it gets included for the students next year. It certainly helped a lot, and it breaks down the hurdles for students to step out and ask for help. I would advise to keep using it!

// Jeffrey

Big Data Series - Give me a spark

2017-05-04T14:46:26+02:00

So the third assignment in this series is running Spark and playing around with it. The first part was basically messing about with the query-processing, the second part is playing with data and dataframes. As these do not actually seem to be part of the required stuff for the post, I have left them out completely.

The way I understand this is that I'm supposed to play with Spark, come up with something new, and write a short blog post detailing my experiences.

Alternatively, you could decide to carry out a small analysis of a different open dataset, of your own choice; and present to your readers some basic properties of that data. You will notice that it is harder than following instructions, and you run the risk of getting stuck because the data does not parse well, etc.

So without further ado, let's explore some datas.

Part 1 - Getting data

Kaggle is one of the top resources for Data Science competitions, where data scientists, analysts, and programmers of all flavors unite and compete for prizes. While IMO prize money mostly seems to go to people who already have top-tier knowledge (like people who work at Deepmind) or just a lot of time/resources behind them (I recall reading some people spend 5 hours a day on a competition, which would probably make the pay-off very poor for their time investment), it's kind of a data geek playground. I have selected the Sberbank competition which was launched recently.

The first step is simply downloading the accepting the conditions of the competition, and downloading a zip file for training and test data. Additional data about Russia's economy is available in different files, and the file description hints that these may be joined together with the proper instructions. All the data is in comma-seperated values. The training data is 44 MB. Because one cay only download the data once authenticated and I'm using a virtual machine, I've hosted the dataset elsewhere before pulling it in with wget as this seems an easier solution than setting up a shared partition.

Part 2 - Import

if you are interested in following along, use wget https://www.dropbox.com/s/4dmsg68lc509q32/train.csv?dl=0. I'll make sure this file stays available for the next few weeks.

Importing the data can be done with the same instructions as any other csv.

val rusdata = spark.read.format("csv").option("header", true).load("notebooks/BigData/data/train.csv?dl=0").cache()

With printSchema I can take a look at the schema of the data set. This also allows for verification that everything got loaded in.

rusdata.printSchema

Which outputs all the fields from the dataset as follows:

root
 |-- id: string (nullable = true)
 |-- timestamp: string (nullable = true)
 |-- full_sq: string (nullable = true)
 |-- life_sq: string (nullable = true)
 |-- floor: string (nullable = true)
 |-- max_floor: string (nullable = true)
 |-- material: string (nullable = true)
 |-- build_year: string (nullable = true)
 |-- num_room: string (nullable = true)
 |-- kitch_sq: string (nullable = true)
 |-- state: string (nullable = true)
 |-- product_type: string (nullable = true)
 |-- sub_area: string (nullable = true)
 |-- area_m: string (nullable = true)
 |-- raion_popul: string (nullable = true)
 |-- green_zone_part: string (nullable = true)
 |-- indust_part: string (nullable = true)
 |-- children_preschool: string (nullable = true)
 |-- preschool_quota: string (nullable = true)
 |-- preschool_education_centers_raion: string (nullable = true)
 ....

 rusdata: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [id: string, timestamp: string ... 290 more fields]

292 fields. Not bad!

Part 3 - Playing

So now let's explore the data! How does a property object look like, actually?

rusdata.show(1)

So it's just a single row in this massive csv. Each house has an ID and 290 other properties that go with it. Finally, the last property is the house price itself, which is used in the competition mainly. Something that catches my eye in both of these is the large amount of features that have a suffix of 500, 1000 or 2000. This could be how many of that particular feature are in a 500 or 1000 meter radius around the object.

Something that is also peculiar is that there are no latitude or longtitude pairs (be it WGS or a Russian format, neither appear) instead the highest "resolution" is the area in which the property is located. E.g in the 'Kremlin' area. This makes data sort of anonymous, but I suspect it would not be hard to identify property objects based on the 292 features that we have if we chose to do this.

So the first thought.. can we find data on the Kremlin itself? From wikipedia I find that the Grand Kremlin Palace was built between 1837 and 1849 and has a sqaura area of

125 metres long, 47 metres high, and has a total area of about 25,000 square metres. It includes the earlier Terem Palace, nine churches from the 14th, 16th, and 17th centuries, the Holy Vestibule, and over 700 rooms

Can we find that?

val potentialKremlin = rusdata.select("id","full_sq","life_sq","floor","build_year","num_room").where("build_year <= 1849")
potentialKremlin.count
904

Now let's filter this again, as we know at that it has 700 rooms.. can we reduce the set of 904 objects to something we can count on our fingers? Let's filter this by selecting objects with more than 10 rooms.

val potentialKremlin2 = potentialKremlin.select("id","full_sq","life_sq","floor","build_year","num_room").where("num_room >= 10")

potentialKremlin2.show()

+---+-------+-------+-----+----------+--------+
| id|full_sq|life_sq|floor|build_year|num_room|
+---+-------+-------+-----+----------+--------+
+---+-------+-------+-----+----------+--------+

So it seems that the Kremlin is not in the data set, despite being in the Kremlin district. Just to check - are there any properties with more than ten rooms in the data set at all?

rusdata.select("id","full_sq","life_sq","floor","build_year","num_room").where("num_room >= 10").show()

+-----+-------+-------+-----+----------+--------+
|   id|full_sq|life_sq|floor|build_year|num_room|
+-----+-------+-------+-----+----------+--------+
|11624|     40|     19|   17|      2011|      19|
|17767|     58|     34|    1|      1992|      10|
|26716|     51|     30|   14|      1984|      17|
|29175|     59|     33|   20|      2000|      10|
+-----+-------+-------+-----+----------+--------+

It seems that having tons of rooms is simply a more modern fad.

Now, let's do some aggregation. Can we get a list of how many properties there are per region? This time we use SQL directly, but we need a dataframe first:

val rusdataDF = rusdata.select("id","full_sq","life_sq","floor","build_year","num_room", "sub_area")
rusdataDF.createOrReplaceTempView("props")
spark.sql("SELECT sub_area, count(sub_area) AS sa FROM props GROUP BY sub_area ORDER BY sa DESC").show(15)

+--------------------+----+
|            sub_area|  sa|
+--------------------+----+
| Poselenie Sosenskoe|1776|
|          Nekrasovka|1611|
|Poselenie Vnukovskoe|1372|
|Poselenie Moskovskij| 925|
|Poselenie Voskres...| 713|
|              Mitino| 679|
|            Tverskoe| 678|
|            Krjukovo| 518|
|             Mar'ino| 508|
|Poselenie Filimon...| 496|
|      Juzhnoe Butovo| 451|
|Poselenie Shherbinka| 443|
|            Solncevo| 421|
|   Zapadnoe Degunino| 410|
|Poselenie Desjono...| 362|
+--------------------+----+

only showing top 15 rows

Note: the sub_areas are supposed to be the 125 raions of Moscow. But are they, actually?

spark.sql("SELECT DISTINCT sub_area FROM props").count
146

So we have more areas than there are raions. This means we should be distrustful of the data: it could at the very least mean that we cannot expect all 125 areas to be present in the data set.

One of the things we might expect to find is negative or zero entries in the build_year column. Can we find these with a simple SQL statement?

spark.sql("SELECT id, build_year FROM props ORDER BY build_year ASC").show

+-----+----------+
|   id|build_year|
+-----+----------+
|11010|         0|
|12811|         0|
|11186|         0|
|10145|         0|
+-----+----------+

uh oh..

spark.sql("SELECT id, build_year FROM props WHERE build_year == 0").count
530

spark.sql("SELECT id, build_year FROM props WHERE build_year >= 2017").count
157

So at the very least there's something funny going on with the building year data. But it could also mean that these building are still being built. A quick look at Kaggle's forum gives the following answer:

What 0 and 1 mean in 'build_year" column of the data?

These are mistakes in the raw data, so we cannot fix it, unfortunately.

And what about the houses that are being built or have been built after the data was collected (as the data is from 2015)

it could be pre-investment (see product type).

Part 4 - Findings

I have added most of the queries I processed in the blog text above, but it would be really nice if I could host the spark notebook somewhere like I would with a ipython notebook.

Overall I'm okay with looking at the data with Spark. For analysis though, I would use Python as it's more established and I can get more support. Additionally, there are a lot of packages available that make development so much more doable.

I did like working with Spark. The SQL-rich syntax makes it easy to learn, and the things that I found gave me plenty of ammunition to start work in node, R or Python.

Big Data Series - Hadoop and HDFS

2017-03-22T20:40:27+01:00

The general idea of this post is to work out a short hello-world type of tutorial. For convenience I'll assume that you have some basic understanding of the idea behind Map-Reduce and why you'd ought to use it. For this post though, I'm not going to go with a very complicated use case, instead sticking to the most basic solution (also I had other deadlines to meet).

Keep in mind that what goes for Shakespeare should also go for the 450-million in words COCA. Maybe something for next week, Arjen? :)

Setting up the environment

Since I'm a stubborn old goat I'm running vagrant to run a virtual Ubuntu distribution. Inside of which I'm running a docker container as was requested per assignment. On my laptop, which runs Windows 8.

Following the instructions I install hadoop version 2.7.3 and set up a hdfs cluster. There is some non-trivial path setting up involved here, but it goes beyond the scope of this assignment.

Getting data

The Complete Shakespeare corpus hosted by Project Gutenberg is only 5.3 MB. Not exactly big data. But for our little tutorial it'll do just fine. For convenience I download it using wget...

The Hello World Example

Considering everyone in the class had to do the same assignment I'm not going to take you by the hand to lead you through the entire Hello World Example on Hadoop again. However, a short summary is in place:

The first steps are setting up the cluster. For the standalone version you don't actually have to do much more. However, we can get Hadoop to run on a single node in a pseudo distributed manner. To first do this, we have to edit the xml config files found under etc/hadoop:

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

After that, we need to get hadoop to format a new HDFS:

bin/hdfs namenode -format
sbin/start-dfs.sh
bin/hdfs dfs -mkdir -p /user/root

Now, we pull the corpus and the java file:

wget http://www.gutenberg.org/ebooks/100.txt.utf-8
wget https://gist.githubusercontent.com/WKuipers/87a1439b09d5477d21119abefdb84db0/raw/c327b9f74d30684b1ad2a0087a6de805503379d3/WordCount.java

And make the directories we need plus the jar we're going to run:

bin/hdfs dfs -mkdir input
bin/hdfs dfs -put 100.txt.utf-8 input
bin/hadoop com.sun.tools.javac.Main WordCount.java
jar cf wc.jar WordCount*.class

Finally, we run the jar on the corpus:

bin/hadoop jar wc.jar WordCount input output
bin/hdfs dfs -get output/part-r-00000
bin/hdfs dfs -rm -r output

We can inspect the results with Nano. This is a bit archaic, but what in computer science isn't?

nano part-r-00000

In the resulting file we have tuples of each token with the cumulative count of their occurence. Don't be alarmed if it looks off: the script does not actually know that "Juliet." and "Juliet?" refer to the same token.

So how does Mapreduce really count these words? The WordCount java code contains a simpel flow consisting of a map-step that emits words (including special characters such as !?,.:) that are deliited by whitespace along with a count of how many times they were encountered. This processing is done one line at a time.

In the reduce step each map is combined so that we get a nice hash map that sums up the values.

Food for thought (or other questions..)

So what happens when you just do the standalone part of the tutorial? A standalone operation just builds a single node. This is probably very handy for programming and debugging. In this mode the commands are handled by just a single node which is kind of defeating the purpose of hadoop!

So what's different in the pseudo-dist case? In the pseudo-distributed case we build a simulated cluster with multiple nodes. This would allow big tasks to be done in an asynchronous way. In the pseudo-distributed case I had to do more set-up and much more debugging.

Really, the standalone variant and the simulated clusters just look like debugging settings for when you can't or will not work on a real cluster.

So who's the most popular kid in town? So there are more ways to determine who's more popular. If we count purely the amount of times Romeo or Juliet spoke, we just have to look for the lines starting with their name. For Romeo this is "Rom." and he has 167 lines. Our tragic Juliet "Jul." only has 117.

But who gets the most mentions? "Juliet" is mentioned 68 times, while "Romeo" in all its forms gets mentioned 152 times.

*Things aren't always fair on fourteen year-olds. *

Sometimes our brains are able to repress bad memories. However, it all came back to me when I saw the Java code... those long days in a hot and sweaty computer lab, trying to understand OO did however make me into the man I am.

Salt on the Road - Tracking Gritters by webscraping

2016-01-16T11:11:20+01:00

Last year, the Dutch Rijkswaterstaat (a part of the Dutch Ministry of Infrastructure and the Environment) released a website where you could track where salt scattering trucks - also known as gritters - moved in real-time. This is particularly useful as Dutch infrastructure always seems to shut down completely during the first days of mild snow and you need to know if there's a chance you might make it to work today.

The Rijkswaterstaat website features a Google Maps widget that shows which trucks are active and moving and which are not.

Getting the data

The website features an API, which while not publicly advertised can be found by opening dev tools in any modern browser and looking at the requests made by the page. This url can then be approached via the URL.

Snooping around I found a nice stream of JSON data:

{"id":"108191021","workcode_id":"34","latitude":"52.053938","longitude":"5.115533"}

The response includes:

coordinate pairs in latitude and longitude (in WGS84!)
a truck identifier (id)
a code whether the truck is active or not. (workcode_id)

Visualizing the data

I quickly wrote a small script to download this JSON from their site - every sixty seconds or so - and started to build my own real-time overview of the gritter truck by converting the data to GeoJSON and plotting it in TNO's Common Sense. This yielded the following image.

At this point I could only see where they were located. If I wanted to get new data I would have to refresh my map, and I'd still be working with point data. I had to find a way to collect historical data of the trucks.

A second version of the script used an array of past values to collect historical data. Every time the script requested the JSON file, I would add the new values to an array I'd connect to each truck, and save it on my local file system. With a little bit of extra code I converted these to GeoJSON LineStrings, which are essentially lines between each coordinate pair I received. Now I had a way to show where trucks had been moving. The following image is aggregated from a day worth of data:

Not surprisingly, it only showed which roads were part of the Rijkswaterstaat's responsibility: main high ways et cetera. What was kind of cool though was that you could see exactly where they were, including ramps.

Wrapping up

That's about it. I found the data, transformed it, and made a pretty image of it. Very little code had to be written thanks to the tools available.

The code can be found on Github for anyone who wants to mess with it.

*Update: The Rijkswaterstaat website has been updated and the API path has been changed. You can find the new JSON file here They also added a nice historical viewing option, which shows you where trucks have been in the past six hours. *