Data lake is like a snowball. Most of us have started with proof of concepts that filled data lake with stream data and batch imports from external data sources. Camus, Gobblin, Spark ingestion, Sqoop, NiFi and more. They all start as shiny snowflakes which get doubled within an eye blink. How to survive the flood on a data lake and successfully solve problems like: small files on HDFS, data retention, auditing and monitoring imports, (near) real-time ingestion, late and out-of-order events?

During panel we are going to discuss best techniques of power full data visualization. Classification data visualization tools according to their strengths and weaknesses would be another stage of discussion.

We will also talk about storytelling and targeting presentations for the audience i.e. customers, stakeholders, students, etc. https://www.youtube.com/watch?v=AdSZJzb-aX8

Plan of discussion:

1. Present yourself: name, current position, experience in data visualization
2. Describe tools for data manipulation and visualization, your favorite ones and most painful.
3. What are the features of good data visualization?
4. Did it happen that customer incorrectly interpret your visualization?
5. What should the correct presentation look like from the perspective of storytelling?
6. Dirty and missing data how to deal with that issue?
7. Reality, Complexity, Simplicity – what is the best strategy for visualizations?

Becoming a scientist date seems to be temptingly easy. Finishing a specialisation at Coursera and waiting for job offers. I would like to share my observations what skills are worth having and what challenges will be set by the reality in the practical implementation of large data science projects

In today’s world, whose functioning is practically based on insights drawn from data, the ability to understand data in depth and communicate findings are increasingly welcome skill sets. Scientific communication is a science in itself, so what does it take to be both a good data scientist/big data engineer AND a good communicator? Should we strain to be both, anyway?

During the discussion we will try to answer this question and touch upon others such as:
– Data visualisation: is it an extra perk or an integral part of a data science project; what are good visualisations
– How to report your data science findings to fully convey the result in a persuasive manner
– What is and how important is exploratory data analysis (EDA)
With this talk I’d like to increase interest in scientific communication techniques and help bridge the gap between the scientist and the audience. We will explore how to fit data storytelling into a data science project development cycle to make it all more practical. Also, we will talk about how rational humans are – and how this (presumed) lack of total rationality influences the techniques of communication.

In this round table, we dive into various topics in the Realm of Artificial Intelligence and Big Data. Understand the driving forces and the technologies powering the AI journey. Especially the growth of data both structured and unstructured in the current times creating a strong synergy between AI and Big Data. How AI and Big Data is instrumental in transformation of businesses? Discuss about applications of deep learning in Big data Analytics and proliferation of conversational AI like chatbots ,voice assistants in enterprises

Hadoop has evolved into a vast Big Data ecosystem of different frameworks and services, which means selecting a distribution that matches one’s needs has become a tricky task. In this session, we discuss the services and features that users should consider when opting for a modern Hadoop distribution. We focus on the main distributions, namely HDP, CDH, MapR, Hops, AWS EMR, Google Dataproc and we discuss how each one would fit their needs based on the following all important aspects:

• Performance
• Security
• Platform Installation & Administration
• Application Monitoring & Control
• Big Data Processing frameworks
• Data Governance
• SQL & Business Intelligence
• Deep Learning

Taboola provides 500 Billion fully personalized content recommendations per month to 1.5 Billion unique visitors of the most prominent publishers across the world. To do that, we are processing 40TB+ a day in real time using Hadoop, Cassandra, Kafka and Spark. A significant part of the system is running SQL queries in Spark. In this table, We want to talk about people’s experience with running SQL on top of Spark, Presto, Drill, etc and share our experience.

Data lake is like a snowball. Most of us have started with proof of concepts that filled data lake with stream data and batch imports from external data sources. Camus, Gobblin, Spark ingestion, Sqoop, NiFi and more. They all start as shiny snowflakes which get doubled within an eye blink. How to survive the flood on a data lake and successfully solve problems like: small files on HDFS, data retention, auditing and monitoring imports, (near) real-time ingestion, late and out-of-order events?

During panel we are going to discuss best techniques of power full data visualization. Classification data visualization tools according to their strengths and weaknesses would be another stage of discussion.

We will also talk about storytelling and targeting presentations for the audience i.e. customers, stakeholders, students, etc. https://www.youtube.com/watch?v=AdSZJzb-aX8

Plan of discussion:

1. Present yourself: name, current position, experience in data visualization
2. Describe tools for data manipulation and visualization, your favorite ones and most painful.
3. What are the features of good data visualization?
4. Did it happen that customer incorrectly interpret your visualization?
5. What should the correct presentation look like from the perspective of storytelling?
6. Dirty and missing data how to deal with that issue?
7. Reality, Complexity, Simplicity – what is the best strategy for visualizations?

Becoming a scientist date seems to be temptingly easy. Finishing a specialisation at Coursera and waiting for job offers. I would like to share my observations what skills are worth having and what challenges will be set by the reality in the practical implementation of large data science projects

When Data Science meets large datasets, this can create a very effective mix and allows us to build more powerful data products. However, this is not always easy and effective.

How to be pragmatic about this topic in order to accelerate Data Science while avoiding pitfalls?

At this round-table, we will discuss a wide range of topics on what makes Data Scientist effective, from exploratory data analysis to the deployment of finished models on production. The questions we will ask ourselves include: Which Big Data tools are the most Data Scientist friendly? When should we use Big Data, and when is it more practical to stay with a single machine? When does it make sense to use distributed machine learning algorithms? How to visualize large datasets? How do you switch from prototyping to deploying scalable models on production?

Let’s share and learn from each other!

Many organisations assume that Business Intelligence will be able to answer all their data related questions. But for that working with the data should not be a bottleneck. For that we need to have a proper BI platform, but what is that and what requirements should it satisfy to provide the most benefits?

During the discussion we will try to answer a couple of questions:

– what is a BI platform?

– who should it serve?

– what are typical requirements on the platform?

– where can we expect the bottlenecks?

I would like to talk with you about your experiences with security issues. Is it possible to implement AAA hadoop security stack without commercial software? How to secure rows of data in Hive? Do you think Java Reflections are safe? Where personal data should be processed? How to prepare data access to logs where the only personal data is cookie? Who should have access to data hash’ing? Kerberized it all – can we enable security in several steps or do we have to start with everything kerberized at the beginning?

When dealing with huge amount of data coming as streams you might not have the possibility to see these data again, in that case you need to use one-pass incremental algorithms.

These algorithms usually trade accuracy for performance. Real life examples are “filtering” using Bloom filter like in Chrome/HBase…, “count distinct” using HyperLogLog like in Spark/Redis/AtScale…), quantiles estimation…

We can have the same trade off in Machine Learning too: algorithms exist for both supervised and unsupervised learning that can learn incrementally on data streams.

The idea for this round table is to discuss real use cases of streaming algorithms/structures and stream mining.

Business requirements for large data systems are growing. Time costs more and more. We will discuss the tools we can use to minimize time of loading and query this data with sub-second OLAP queries and how to scale this architecture to petabytes of data. We will discuss tools such as Druid, Kylin and Clickhouse. We will compare them and exchange experiences.

IoT data often comes with its own unique challenges: lack of structure and the extreme diversity of source devices from light sensors to cars, unreliability around hardware integration with the “internet,” and a lack of an established best practices and ecosystem of tools to process and analyze the data. However, with many of the consumer-side “big data” problems cracked, processing and harnessing the value of IoT data is the next logical progression for the “big data” discipline to figure out on a large-scale. At Springworks, a connected cars platform based in Stockholm, we work with IoT data from cars using a telematic unit and also face many of these challenges. Key questions of this session include:

• What are the best design choices for IoT data, especially to overcome the unreliability around hardware integration component?
• What are similarities/differences with “regular” big data processing?
• Which organizations are the leaders in leveraging IoT data, and what are their learnings?

Apache Spark™ is a fast and general engine for distributed in-memory computations on massive scale. Let’s talk about what’s coming in Apache Spark 2.3 and how to use it for large data processing in batch or streaming modes. Bring all your questions about Apache Spark in general and Spark SQL, Spark MLlib, Spark Structured Streaming in particular. The roundtable is to help you fine-tune existing Spark workloads as well as prepare for future ones