Author Archives: Gianmario
In-Memory Logical Data Warehouse for accelerating Machine Learning Pipelines on top of Spark and Alluxio
Abstract: Legacy enterprise architectures still rely on relational data warehouse and require moving and syncing with the so-called “Data Lake” where raw data is stored and periodically ingested into a distributed file system such as HDFS. Moreover, there are a … Continue reading
Surfing and Coding in Lanzarote, the Barclays Data Science hackathon
This post has been published on the Cloudera blog and summurises the results and takeaways of a week-long hackathon happened in Lanzarote in December 2015. The goal was to prototype a recommender systems for retail customers of shops in Bristol in Bristol, UK. The article shows how the stack composed by Scala and Spark was great for quickly writing some prototyping code to run locally in a single laptop and at the same time scalable for larger dataset to process in the cluster. Continue reading
Posted in Agile, Machine Learning, Scala, Spark
Tagged als, Functional Programming, hackathon, machine learning, recommender systems, scala, Spark
Leave a comment
Robust and declarative machine learning pipelines for predictive buying
Proof of concept of how to use Scala, Spark and the recent library Sparkz for building production quality machine learning pipelines for predicting buyers of financial products.
The pipelines are implemented through custom declarative APIs that gives us greater control, transparency and testability of the whole process.
The example followed the validation and evaluation principles as defined in The Data Science Manifesto available in beta at http://www.datasciencemanifesto.org Continue reading
Posted in Big Data, Classification, Machine Learning, Scala, Spark
Tagged Functional Programming, machine learning, ML pipelines, predictive analytics, scala, Spark, sparkz
Leave a comment
Thoughts about data operations
Very hard to give guidelines here since that each project have its own deployment process that depends on many factors such as the business context and practical issues associated with it. Continue reading
The balance of exploratory analysis and development
Exploratory analysis should precede and follow any task from the modelling, design and development to the benchmarking. Major problem is how do you share, track and monitor your findings? How do you make your analysis repeatable and scrutinizable from the outside? This is still an open problem. Continue reading
Posted in Agile, Machine Learning, Uncategorized
Tagged exploratory data analysis, machine learning
2 Comments
Coding practices for data products development
Code should be developed in a proper IDE and make use of advanced tools for re-factoring, auto-completion, syntax highlighting and auto-formatters; at least.
Notebooks should use routine libraries from the main codebase. As soon as some code is developed in a notebook and is reusable, it should be moved into a codebase. Continue reading
Posted in Agile, Machine Learning, Software Development
Tagged best practices, coding style, data science, Functional Programming, workflows
2 Comments
The ScrumBan Jira board
Let’s start with one of the core tool of the agile workflow. We use a Jira board for tracking and organizing all of our projects. We developed a custom board which uses the sprints concept of Scrum but in a more flexible way as in Kanban. Continue reading
Functional Data Validation using monads and applicative functors
ETL is probably the most time consuming part of every Data Science project. The quality of extracted and crunched data is one of the major factor affecting the final results. In facts, real world data is always messy and inconsistent. Data Validation … Continue reading
Posted in Big Data, Data Munging, Scala, Spark
Tagged applicative functors, data validation, Functional Programming, monads, scala, scalaz, Spark
Leave a comment
