by Cisco UCS. Virtual servers – automatic installation of RHEL servers. Automatic configuration of systems exemplified by BMC BladeLogic. Hadoop installation with Ambari – Hortonworks HDP. Monitoring: integration with external systems – from Nagios to OP. Security: implementation of Kerberos and LDAP using IdM/IPA. Development plans: integration with Active Directory and a comment on Ranger (importance of privileges management).

in many features that you see in our application today.
Two members of Spotify’s Hadoop squad will reminisce about failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and its evolution over time. We are going to explain how and why we use a number of tools (including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive with Tez) to build features and provide analytics (also worth mentioning are Snakebite and Luigi – two in-house tools created to overcome common frustrations).

them properly. Good visualization of data can lead to important discovery, while bad visualization can lead you astray. Rules enabling proper presentation of data and practical examples.

Spark-like engines. It is as fast as in-memory engines, while providing the reliability of Hadoop. Compatibility with the Hadoop ecosystem (runs on YARN, reads data from HDFS and HBase, and supports mixing existing Hadoop Map and Reduce functions into Flink programs). Flink from a user perspective: APIs and the most interesting design points behind Flink. Outlook on Flink’s development roadmap.

Comment:
Flink contains (1) APIs in Java and Scala for both batch-processing and data streaming applications, (2) a translation stack for transforming these programs to parallel data flows and (3) a runtime that supports both proper streaming and batch processing for executing these data flows in large compute clusters.

Flink’s batch APIs build on functional primitives (map, reduce, join, cogroup, etc), and augment those with dedicated operators for iterative algorithms, and support for logical, SQL-like key attribute referencing (e.g., groupBy(“user.country”). The Flink streaming API extends the primitives from the batch API with flexible window semantics.

Internally, Flink transforms the user programs into distributed data stream programs. In the course of the transformation, Flink analyzes functions and data types (using Scala macros and reflection), and picks physical execution strategies using a cost-based optimizer. Flink’s runtime is a true streaming engine, supporting both batching and streaming. Flink operates on a serialized data representation with memory-adaptive out-of-core algorithms for sorting and hashing. This makes Flink match the performance of in-memory engines on memory-resident datasets, while scaling robustly to larger disk-resident datasets.

with many dead ends and very long shortcuts.
Shape of stored data including:
– Directory structure and it implications for Hadoop cluster performance,
– Importance of declaration of a record scheme,
– Versioning and updating of records aka managing historical data.
Calculation process:
– Make it faster by using proper data structures in every single step,
– Bugs are inevitable so easy recalculation is a must,
– Cluster malfunction/maintenance triggers a recalculation process without human intervention,
– Design for an arbitrary time frame but be prepared for stream data as well.
Be a good citizen:
– Small files and empty directories are an issue,
– Make it as small as possible with data compression,
– YARN resource demand based on real needs,
– CSV files should be considered in the last resort but they are not forbidden.

Hadoop ecosystem surprises, and where to expect them. Based on experience from classic DBRMS and single server apps (such intuition is often misleading in context of Hadoop!):
– Killing a hive process with easy select query,
– Retries that are overloading a cluster,
– Non-deterministic measures of MR performance operating on same data,
– MR reduce phase object reusing,
– Client machine and its capacity may sometimes turn to be a bottleneck,

Comment:
For each troublemaking aspect mentioned I will be providing a sample solution applied or at least evaluated in real life. Problems mentioned above may seem to be a pessimistic and overwhelming vision for applying Hadoop but it is far from truth. We really enjoy working with that framework but I wish to talk about that dark side of Big Data implementation because behind each success story there are hours of problem fixing. I will be more than happy if at least some of you avoid presented pitfalls and successfully use Hadoop ecosystem.

they become a nightmare when your cluster grows. However, even when you realize the problem, you might not see that possible solutions are right under your nose.

How we simplified our data management and process scheduling in Hadoop with useful (but less adopted) open-source tools. How Falcon, HCatalog, Avro, HDFS FsImage, CLI tools and tricks helped us to address typical problems related to orchestration of data pipelines and discovery, retention, and lineage of datasets.

Traditional tools for general-purpose analytics like Hive, Pig or in general MapReduce are not suitable for graph analytics. During the last years, a number of graph-specific processing systems have been released. In this talk, we will look at the general programming paradigm they share, and how it can be leveraged to express graph analytics.