Sparkling Song Recommendations

sparkling song recs

Having taught myself a bit of Python I was keen to start using Spark. Spark is an open source project from Apache building on the ideas of MapReduce. It still uses the HDFS but where Hadoop processes on disk, Spark runs things in memory, which can dramatically increase process speeds. Spark also has a great machine learning library you can use with it. Hive on Hadoop limits you to data processing and manipulation, Spark allows you to build models at scale. Spark can be called with its native Scala, or there are APIs so you can use Python or Java (release 1.4 also includes an R API).

I decided to take the lastfm dataset i have been playing with recently and generate some recommendations for the users based on their listening habits.

Continue reading

Predicting gender from musical tastes

forest cp

Continuing on my mission to get better at Python I’ve been learning about the Pandas and sklearn libraries. I was looking for a challenge to use these libraries on and I had recently come across a nice lastfm data extract. The data contains around 360K users and a little demographic data on them (gender, age, country), and a count of listens they have had for each artist. I decided to try and build a model that could predict someone’s gender based on what they have been listening to.
Continue reading