Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems

•

0 j'aime•27 vues

Open Data is about supporting the insights in between what historically have been inaccessible data, protected by companies or other institutions. When data became more opened, it was possible to see a growing sector of activities exploring the opportunities it created. Business Intelligence, Analytics, and Machine Learning are some names very popular around the subject. This data-driven approach quickly became the key to success, and ideas were put to practice. Despite the popularity, the technical solutions of the software systems were yet naive. When handling with large amount of data, specially open data, there’s an important lesson learned: systems got to be flexible, as data are. For that matter, Microservices have been widely adopted in systems that handle open data, due to its support for Extensibility. In this architecture, the system is maintained as a set of small, independent parts. Each part, called a microservice, has a specific responsibility, and the different microservices inter-exchange data to meet the system’s and user’s need. In this talk, we’re going through the most indispensable principles and practices when designing computing systems that are data-driven, perfectly tailored for Open Data. We’ll begin providing a perspective on how to establish the communication and shared data between microservices, and evolve into discussing ways to support seamless incorporation of new data-analysis services.

Technologie

Supporting
Open Data
in large systems
Principles and practices of
Adaptability in Microservices

Supporting
Open Data
in large systems
Principles and practices of
Adaptability in Microservices
João Daniel Eduardo Guerra

drawbacks
of sharing datasources?
What are the

drawbacks
?
Unexpected
overwrites
of sharing datasources?
What are the

drawbacks
?
Unexpected
overwrites
Poorly aided work
(and most likely rework)
of sharing datasources?
What are the

drawbacks
?
Unexpected
overwrites
Poorly aided work
(and most likely rework)
of sharing datasources?
What are the
Local results
instead of accessible

Scenarios
from
Practice
Multiple parts
sharing data

Consuming data
in real time
Scenarios
from
Practice
Multiple parts
sharing data

Multiple parts
sharing data
Scenarios
from
Practice
Consuming data
in real time

Multiple parts
sharing data
Make processing
available to colleagues
Scenarios
from
Practice
Consuming data
in real time

help in this struggle?
How can the
right software
architecture

How can the
right software
architecture
help in this struggle?

How can the
right software
architecture
help in this struggle?
All business logic
in a single deploy unit
Instead of
Monolith

How can the
right software
architecture
help in this struggle?
All business logic
in a single deploy unit
Small, independently
deployed units
Instead of Do this
Micro
Services
Monolith

How can the
right software
architecture
help in this struggle?
Adaptability

Adaptability
How can the
right software
architecture
help in this struggle?

Adaptability
extending what
already exists
How can the
right software
architecture
help in this struggle?

Adaptability
extending what
already exists
making it flexible
to add new things
How can the
right software
architecture
help in this struggle?

Adaptability
Database per Service
Event-Driven
Architecture

Adaptability
Database per Service
Event-Driven Architecture
API Gateway

Adaptability
Database per Service
Event-Driven Architecture
API Gateway
Adapter

Supporting
Open Data
in large systems
Principles and practices of
Adaptability in Microservices
João Eduardo

Supporting
Open Data
in large systems
Principles and practices of
Adaptability in Microservices
João Eduardo
joao.daniel@alumni.usp.br eduardo.guerra@unibz.it
Let’s keep in touch

Contenu connexe

Similaire à Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems

How the Journey to Modern Data Management is Paved with an Inclusive Edge-to-...

Dana Gardner

The IT Intelligence Foundation For Digital Business Transformation Builds fro...

Dana Gardner

The era of big data has ushered in a new paradigm for data analysis, presenting unique challenges and opportunities. This article delves into the world of big data analytics and explores how data analysts work with distributed computing frameworks to handle large and complex datasets. We'll discuss the concept of big data, the challenges it poses, and the evolution of distributed computing frameworks. Furthermore, we'll dive into the role of data analysts, their skills and tools, and the practical applications of big data analytics. By the end of this article, readers will have a comprehensive understanding of how data analysts leverage distributed computing frameworks to extract valuable insights from vast datasets.

How do data analysts work with big data and distributed computing frameworks.pdf

Soumodeep Nanee Kundu

Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...

Dana Gardner

Challenges are consistent in Big Data environments; resource-intensive processes, unwieldy time commitments, and challenging variations in infrastructure. Big Data has grown so large that traditional data analysis and management solutions are too slow, too small and too expensive to handle it. Many companies are in the discovery stage of evaluating the best means of extracting value from it. This Enterprise Tech Journal interview with Kevin Goulet, VP Product Management, CA Technologies, explores the challenges of Big Data, the approach to resolving them. With Big Data environments, the challenges are consistent – resource-intensive processes, unwieldy time commitments, and challenging variations in infrastructure. For more information visit http://www.ca.com/us/products/detail/business-intelligence-and-big-data-management.aspx?mrm=425887

Overcoming Big Data Challenges on System z

CA Technologies

The growing complexity of data science leads to black box solutions that few people in an organization understand. You often hear about the difficulty of interpretability—explaining how an analytic model works—and that you need it to deploy models. But people use many black boxes without understanding them…if they’re reliable. It’s when the black box becomes unreliable that people lose trust. Mistrust is more likely to be created by the lack of reliability, and the lack of reliability is often the result of misunderstanding essential elements of analytics infrastructure and practice. The concept of reproducibility—the ability to get the same results given the same information—extends your view to include the environment and the data used to build and execute models. Mark Madsen examines reproducibility and the areas that underlie production analytics and explores the most frequently ignored and yet most essential capability, data management. The industry needs to consider its practices so that systems are more transparent and reliable, improving trust and increasing the likelihood that your analytic solutions will succeed. This talk will treat the black boxed of ML the way management perceives them, as black boxes. There is much work on explainable models, interpretability, etc. that are important to the task of reproducibility. Much of that is relevant to the practitioner, but the practitioner can become too focused on the part they are most familiar with and focused on. Reproducing the results needs more.

The Black Box: Interpretability, Reproducibility, and Data Management

mark madsen

Business Intelligence for normal people

mark madsen

Data Explosion and Big Data Require New Strategies for Data Management and Re...

Dana Gardner

Test environments are an essential part of any IT environment, providing a secure space where companies can test new software before releasing it to the public. With careful planning and consideration, you can ensure that your IT environment has all the tools and resources, including the correct test data, needed to run successful tests and ensure the quality of your solutions as they progress through the release cycle.

What Are IT Environments, and Which Ones Do You Need?

Enov8

Machine Learning is quickly becoming a ubiquitous technology and expected skill of development teams. In 2019 a stunning 87% of Data Science projects never made it into production. What will you do to make sure that your ML and Data Science projects succeed? This talk will focused on digging deeper into that static and help to explain what goes wrong, why it goes wrong, and what I do to mitigate these issues. You’ll leave with a deep understanding of ML project failures, and some advice on how to improve your projects chance of success. You also will learn how to fail quickly in ML model development and pivot towards a path of success.

Why do most machine learning projects never make it to production

Cameron Vetter

Toward a System Building Agenda for Data Integration (and Data Science) AnHai Doan, Pradap Konda, Paul Suganthan G.C., Adel Ardalan, Jeffrey R. Ballard, Sanjib Das, Yash Govind, Han Li, Philip Martinkus, Sidharth Mudgal, Erik Paulson, Haojun Zhang University of Wisconsin-Madison Abstract We argue that the data integration (DI) community should devote far more effort to building systems, in order to truly advance the field. We discuss the limitations of current DI systems, and point out that there is already an existing popular DI “system” out there, which is PyData, the open-source ecosystem of 138,000+ interoperable Python packages. We argue that rather than building isolated monolithic DI systems, we should consider extending this PyData “system”, by developing more Python packages that solve DI problems for the users of PyData. We discuss how extending PyData enables us to pursue an integrated agenda of research, system development, education, and outreach in DI, which in turn can position our community to become a key player in data science. Finally, we discuss ongoing work at Wisconsin, which suggests that this agenda is highly promising and raises many interesting challenges. 1 Introduction In this paper we focus on data integration (DI), broadly interpreted as covering all major data preparation steps such as data extraction, exploration, profiling, cleaning, matching, and merging [10]. This topic is also known as data wrangling, munging, curation, unification, fusion, preparation, and more. Over the past few decades, DI has received much attention (e.g., [37, 29, 31, 20, 34, 33, 6, 17, 39, 22, 23, 5, 8, 36, 15, 35, 4, 25, 38, 26, 32, 19, 2, 12, 11, 16, 2, 3]). Today, as data science grows, DI is receiving even more attention. This is because many data science applications must first perform DI to combine the raw data from multiple sources, before analysis can be carried out to extract insights. Yet despite all this attention, today we do not really know whether the field is making good progress. The vast majority of DI works (with the exception of efforts such as Tamr and Trifacta [36, 15]) have focused on developing algorithmic solutions. But we know very little about whether these (ever-more-complex) algorithms are indeed useful in practice. The field has also built mostly isolated system prototypes, which are hard to use and combine, and are often not powerful enough for real-world applications. This makes it difficult to decide what to teach in DI classes. Teaching complex DI algorithms and asking students to do projects using our prototype systems can train them well for doing DI research, but are not likely to train them well for solving real-world DI problems in later jobs. Similarly, outreach to real users (e.g., domain scientists) is difficult. Given that we have Copyright 0000 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purpose ...

Toward a System Building Agenda for Data Integration(and Dat.docx

juliennehar

Big Data: Issues and Challenges

Harsh Kishore Mishra

big data Big Things

pateelhs

Uniting traditional GIS and mainstream IT

gssg

Insteddma08 1224025844002745-9

InSTEDD

Getting software released to users can be risky, time-consuming and painful. The solution is the ability to deliver reliable software continuously through build, test and deployment automation, and through improved collaboration between developers, testers and operations. In this tutorial we will present principles and technical practices that enable teams to incrementally deliver software of high quality and value into production whenever they want, and extremely fast. The size of the project or the complexity of its code base does not matter. In the first half of the tutorial we will introduce the concepts of continuous delivery, through continuous integration; and automation of the build, test and deployment process. We will also go through som basic principles and patterns for building automatable applications (architecture). We will cover experiences on team collaboration patterns and lastly; techniques for solving tasks such as an easy and comprehendible version control strategy. The second half of the tutorial we will be working with automated provisioning of agile infrastructure, including the use of tools (puppet) to automate the management of testing and production environments. We will go through some scripting lessons examplifying how to implement zero-downtime deploys (… and rollback – if something goes wrong!), with examples in both bash and Ruby. Along with controlling the start, stop, restart lifecycles during deploys, we will also show some simple techniques for backups, logging, error handling, monitoring and verification of application health that can make the automation more robust. We will also use servers "in the cloud" to demonstrate different techniques, and we hope to make it a fun day and to deliver software (examples) several times throughout the workshop. Required knowledge: Agile/Lean basics, Linux basics, version control basics, maven basics.

Continuous Delivery

Stein Inge Morisbak

ROOTS2011 Continuous Delivery

Ole Christian Rynning

Using AI to Solve Data and IT Complexity -- And Better Enable AI

Dana Gardner

DotNetnuke

kaushal123

With the explosion of new frameworks, a mountain of automation, and our applications distributed across hundreds of services in the cloud, the level of complexity in software development is growing at an insane pace. With increased complexity comes increased costs and risks. When diagnosing unexpected behavior can take days, weeks, or sometimes months, all while our release is on the line, our projects plunge into chaos. In the invisible world of software development, how do we identify what's causing our pain? How do we escape the chaos? Janelle Klein presents a novel approach to measuring the chaos, identifying the causes, and systematically driving improvement with a data-driven feedback loop. Rather than measuring the problems in the code, Janelle suggests measuring the "friction in Idea Flow", the time it takes a developer to diagnose and resolve unexpected confusion, which disrupts the flow of progress during development. With visibility of the symptoms, we can identify the cause—whether it's bad architecture, collaboration problems, or technical debt. Janelle discusses how to measure Idea Flow, why it matters, and the implications for our teams, our organizations, and our industry.

Identify Development Pains and Resolve Them with Idea Flow

TechWell

Similaire à Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems (20)

How the Journey to Modern Data Management is Paved with an Inclusive Edge-to-...

The IT Intelligence Foundation For Digital Business Transformation Builds fro...

How do data analysts work with big data and distributed computing frameworks.pdf

Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...

Overcoming Big Data Challenges on System z

The Black Box: Interpretability, Reproducibility, and Data Management

Business Intelligence for normal people

Data Explosion and Big Data Require New Strategies for Data Management and Re...

What Are IT Environments, and Which Ones Do You Need?

Why do most machine learning projects never make it to production

Toward a System Building Agenda for Data Integration(and Dat.docx

Big Data: Issues and Challenges

big data Big Things

Uniting traditional GIS and mainstream IT

Insteddma08 1224025844002745-9

Continuous Delivery

ROOTS2011 Continuous Delivery

Using AI to Solve Data and IT Complexity -- And Better Enable AI

DotNetnuke

Identify Development Pains and Resolve Them with Idea Flow

Plus de South Tyrol Free Software Conference

The complexity of agricultural droughts requires a consistent, reliable, and systematic method for monitoring and reporting. Amongst the various indices used to monitor this phenomenon, the soil moisture anomaly has been proven to be a more reliable predictor. However, the datasets required for computing this index are often large and computationally demanding. To address this challenge, we have developed SMODEX, a Python package that enables scalable, fast, and open-source standard-compliant computation and visualization of soil moisture anomalies. SMODEX simplifies the computation and visualization of time-series for soil moisture and soil moisture anomalies from high-dimensional climate datasets. It allows for quick and easy parallelization of the computation on a daily, weekly, and monthly timescale. Additionally, SMODEX implements a straightforward workflow for automating the use of FAIR (Findable, Accessible, Interoperable, and Reusable) principles in producing and sharing outputs by leveraging the open source STAC API. The package is extendible and provides information on how to contribute to the project, test suites, test coverage, and a use case for the South Tyrol region, all provided in the package repository. In the future, additional agricultural drought indices and indicators would be included to serve to even larger community of researchers, policy makers, and individual users.

SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...

Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems

Recommandé

Recommandé

Contenu connexe

Similaire à Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems

Similaire à Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems (20)

Plus de South Tyrol Free Software Conference

Plus de South Tyrol Free Software Conference (20)

Dernier

Dernier (20)

Open Data Hub - João Francisco Lino Daniel, Eduardo Martins Guerra BACKUP SE NON FUNZIONA HTML - unibz - Supporting Open Data in large systems