Data transformations. Using kettle transformations - Andriy Kyrylenko,

•

0 j'aime•161 vues

Ruby Meditation

Ruby Meditation #1 February 23, 2013 Kyiv

Technologie

Data transformation.
Using kettle transformation
■ Andriy Kyrylenko (Gera-IT)
How to manage import files from different vendors?
What are the common transformations?
How Kettle can help with managing different
transformations?

Import records standard feeds
NEWS
PRODUCTS
RSS feed
ATOM feed
???

Import products
Vendor1
Vendor2
Vendor3
Vendor4
Products1
Products3
Products4
Products2 Shop
productsShop

Import products: source and file types
FTP
HTTP/HTTPS
Local file
Service API
Source file
file
Records
XML file
Shop Products
DB

Import products: source and file types
Source type1
Source type2
Source type3
Service API
File format1
File format3
Records
File format2
Shop Products
DB

Import products:
several readers
Fieldset1
Fieldset2
FieldsetN
Service API
Converter1
ConverterN
ReaderS
Converter2
Shop Products
DB

Import products:
several ruby classes
Fieldset1
Fieldset2
FieldsetN
Service API
RubyClass1
RubyClassN
RubyClassS
RubyClass2
Shop Products
DB

Converting records:
simple cases
■ Filtering fields
■ Filtering records by field value
■ Cloning records
■ Splitting fields into several fields
■ Joining fields
■ Creating new fields
■ Merging two files

Import products:
several ruby classes
FileSet1
FileSet2
FileSetN
Service API
Kettle1
KettleN
KettleS
Kettle2
Shop Products
DB
http://kettle.pentaho.com

Pentaho Kettle Transformations
■ http://kettle.pentaho.com
■
■ Download data-integration packager
■ Run ”Spoon”
■ Enjoy

Contenu connexe

Tendances

Hack angular wildly

Todd Warren

DataXDay - Real-Time Access log analysis

DataXDay Conference by Xebia

This presentation gives an overview of the dataset description specification developed in the Open PHACTS project (http://www.openphacts.org/). The creation of the specification was driven by a real need within the project to track the datasets used. Details of the dataset metadata captured and the vocabularies used to model this metadata are given together with the tools developed to enable the specification's uptake. Over the course of the last 12 months, the W3C Healthcare and Life Science Interest Group have been developing a community profile for dataset descriptions. This has drawn on the ideas developed in the Open PHACTS specification. A brief overview of the forthcoming community profile is given in the presentation. This presentation was given to the Network Data Exchange project http://www.ndexbio.org/ on 2 April 2014.

Dataset Descriptions in Open PHACTS and HCLS

Alasdair Gray

Highlights of the Projects

Mohammad Rahman

При работе с реляционными СУБД структура данных должна быть определена перед началом работы с ними. Но иногда бывает необходимо сохранять и, что более важно, искать по частично-структурированным данным. А также было бы удобно использовать один и тот же интерфейс для доступа к структурированным и частично-структурированным данным. В рамках доклада будет рассмотрено использование Marten и JSONP для создания IQueryable хранилища документов, а также сравнение данного подхода с другими способами хранения и поиска документов для .NET приложений.

.NET Fest 2017. Константин Проскурдин. Marten как хранилище документов для .N...

NETFest

GraphQL Advanced

LeanIX GmbH

Tendances (6)

Hack angular wildly

DataXDay - Real-Time Access log analysis

Dataset Descriptions in Open PHACTS and HCLS

Highlights of the Projects

.NET Fest 2017. Константин Проскурдин. Marten как хранилище документов для .N...

GraphQL Advanced

Similaire à Data transformations. Using kettle transformations - Andriy Kyrylenko,

Mcneill 01

narsareddy1234

Building bridges - Plone Conference 2015 Bucharest

Andreas Jung

FDMEE Custom Reports

Francisco Amores

ETH Zurich's DOI Desk

ETH-Bibliothek

Phpconf taiwan-2012

Hash Lin

The free and open-source CAT tool OmegaT allows a team of translators to work simultaneously on the same project, sharing translation memories, glossaries and other reference material. This is the presentation by Marco Cevoli (Qabiria.com) at the TriKonf 2015 conference, that explained how to set up and manage the external server required, and analysed a case study on how this powerful feature is helping Qabiria in the localization of a major e-commerce website and blog.

OmegaT "Team Project" feature: a case study

Qabiria

Understanding linport

Andrzej Zydroń MBCS

In this talk we present Zalando's microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing for near-real time business intelligence. Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization. We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach - Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams. We no longer live in a world of static data sets, but are instead confronted with an endless stream of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch. With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL. Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements. On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability. Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.

Stream Processing using Apache Flink in Zalando's World of Microservices - Re...

Zalando Technology

The ETH Zurich DOI Desk

ETH-Bibliothek

Hadoop @ LifeWay

jimforrester11

[WSO2 Integration Summit Nairobi 2019] Role of Integration in an API Driven W...

WSO2

Best Implementation Practices with BI Publisher

Mohan Dutt

10 Reasons ColdFusion PDFs should rule the world

ColdFusionConference

IBM Omnifind Enterprise Portal Seach To Improve Productivity

Francis Ricalde

We've learned about all of the different services that are available in Azure for storage and processing, but when it comes to architecting our solution, it's always a question of which service fits our need best. In this on-demand session, we break down the features of each of the different data services in Azure and answer the question "When to use what." We highlight the advantages each service has and talk through some best practices for designing your application.

Build 2017 - P4016 - Demystifying Cloud Data Services for an App Developer

Windows Developer

CenitHub: Introduction

Miguel Sancho

Introduction to Apache Kafka

Ricardo Bravo

As data volumes grow, managing and scaling data pipelines for ETL and batch processing can be daunting. With more than 13.5 million learners worldwide, hundreds of courses, and thousands of instructors, Coursera manages over a hundred data pipelines for ETL, batch processing, and new product development. In this session, we dive deep into AWS Data Pipeline and Dataduct, an open source framework built at Coursera to manage pipelines and create reusable patterns to expedite developer productivity. We share the lessons learned during our journey: from basic ETL processes, such as loading data from Amazon RDS to Amazon Redshift, to more sophisticated pipelines to power recommendation engines and search services.

Large-Scale ETL Data Flows With Data Pipeline and Dataduct

Sourabh Bajaj

Scaling your application servers is easy with microservices, but the actual scaling and operation challenge is the data. Your database is your bottleneck and the biggest scaling and availability concern. Working with a large scale distributed system entails many challenges in data processing. How do you handle distributed transactions? How to scale your data beyond a single data center and how to handle the eventual consistency state that you may cause by doing that? How do you migrate data and database schemas without downtime? And many more issues when the world of microservices and large scale meets databases. In this talk we’ll try to answer this kind of questions, by exploring some patterns used by Wix.com, which operates hundreds of microservices and petabytes of data across multiple datacenters, as well as multiple clouds on a large scale. Hopefully you can adapt some of these patterns to better handle your data.

Arrested by the cap devoxx uk 2018

Aviran Mordo

Stream processing with Apache Flink @ OfferUp

Bowen Li

Similaire à Data transformations. Using kettle transformations - Andriy Kyrylenko, (20)

Mcneill 01

Building bridges - Plone Conference 2015 Bucharest

FDMEE Custom Reports

ETH Zurich's DOI Desk

Phpconf taiwan-2012

OmegaT "Team Project" feature: a case study

Understanding linport

Stream Processing using Apache Flink in Zalando's World of Microservices - Re...

The ETH Zurich DOI Desk

Hadoop @ LifeWay

[WSO2 Integration Summit Nairobi 2019] Role of Integration in an API Driven W...

Best Implementation Practices with BI Publisher

10 Reasons ColdFusion PDFs should rule the world

IBM Omnifind Enterprise Portal Seach To Improve Productivity

Build 2017 - P4016 - Demystifying Cloud Data Services for an App Developer

CenitHub: Introduction

Introduction to Apache Kafka

Large-Scale ETL Data Flows With Data Pipeline and Dataduct

Arrested by the cap devoxx uk 2018

Stream processing with Apache Flink @ OfferUp

Plus de Ruby Meditation

Talk of Sergey Sergyenko, CEO at Cybergizer, at Ruby Meditation #30 Online 24.10.2020 Next conference - http://www.rubymeditation.com/ Description: I hope that the code I write continues to be used after I die. Which is why I only ever write legacy code" - this is what Patterson mentioned on his Twitter last September. In this talk, I will define metrics and try to distinguish code from Legacy to the one that Survived. Also, I will introduce the so-called "Code Survival" refactoring approach. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30

Ruby Meditation

Talk of Roman Dubrovsky, Backend developer at Datarockets, at Ruby Meditation #29 Kyiv 14.12.2019 Next conference - http://www.rubymeditation.com/ Description: More than two years we’re developing and supporting GraphQL API, tried to make it public and integrated our app with other GraphQL APIs. In this talk, I'll try to summarize our experience, say about good practices, issues we found and can resolve. And which we can’t. I will show you how to set up and how to use them. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...

Ruby Meditation

Talk of Viacheslav Miroshnychenko, Software Engineer at Human Agency, at Ruby Meditation #29 Kyiv 14.12.2019 Next conference - http://www.rubymeditation.com/ Description: Why do we need to think about licenses and check them in our projects? Let's try to figure out it. During this talk, you will learn tools that help you to automate the process of checking licenses for all libraries in your project. I will show you how to set up and how to use them. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29

Ruby Meditation

Talk of Aleksandra Stolyar, Ruby developer at DataArt, at Ruby Meditation #29 Kyiv 14.12.2019 Next conference - http://www.rubymeditation.com/ Description: I will talk about dry-rb ecosystem and it’s major components - dry-validation and dry-schema which are very helpful for validations. This year dry-rb introduced a major update to its’ gems and I faced some problems when decided to marry these updates with a project. This talk will cover some of differences and specifics of dry-validation and dry-schema updates. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...

Ruby Meditation

Talk of Bohdan Parshentsev, senior backend engineer at P2H, at Ruby Meditation #28 Kyiv 26.10.2019 Slideshare - ** Next conference - http://www.rubymeditation.com/ This presentation is connected with presentation How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28 Why we need a Rabbit? Let's try to cook it tasty. And what else? - Live demo of several useful dishes. Be ready to use Telegram to have fun with us. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation * The channel of the organizers of the meetup https://t.me/incredevly

How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28

Ruby Meditation

Talk of Serhiy Nazarov, founder QPARD, system architect, at Ruby Meditation #28 Kyiv 26.10.2019 Next conference - http://www.rubymeditation.com/ This presentation is connected with presentation How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28 Why we need a Rabbit? Let's try to cook it tasty. And what else? - Live demo of several useful dishes. Be ready to use Telegram to have fun with us. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28

Ruby Meditation

Talk of Julik Tarkhanov, senior backend engineer, WeTransfer, Amsterdam, at Ruby Meditation #28 Kyiv 26.10.2019 Next conference - http://www.rubymeditation.com/ It is often a choice, sometimes a whim, and sometimes an act of desperation. We idolise reuse while sometimes the road not taken is just as exciting. Let's chat about where it is appropriate to "do the thing again", take the scenic route and enjoy the view. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation * The channel of the organizers of the meetup https://t.me/incredevly

Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...

Ruby Meditation

Speech of Nihad Abbasov, Senior Software Engineer at Digital Classifieds, at Ruby Meditation 27, Dnipro, 19.05.2019 Slideshare - Next conference - http://www.rubymeditation.com/ How fast is your code? Performance is crucial as your startup grows, and optimizing your application can make a huge impact on user experience. During this talk, you will learn hints, techniques and best practices for improving the overall speed of your Ruby application. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...

Ruby Meditation

Speech of Ruslan Tolstov, Ruby Developer at MLSDev, at Ruby Meditation 27, Dnipro, 19.05.2019 Slideshare - Next conference - http://www.rubymeditation.com/ During this talk, we’ll discuss serverless technologies. In particular, we will talk about popular serverless platforms like AWS Lambda, Google CF, and Microsoft AF and also dwell upon serverless frameworks for managing and deploying your functions. As well, we will touch upon the potential use cases of this new approach in your existing projects. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...

Ruby Meditation

Speech of Orban Botond, Ruby Developer at Toptal, at Ruby Meditation 27, Dnipro, 19.05.2019 Slideshare - Next conference - http://www.rubymeditation.com/ Software development is a domain where everybody can make a beautiful sculpture or can quickly build an unsolvable maze. According to my observations the later happens more often unfortunately. In the title for my presentation both the If Jungle and the Civilised Railway Station are methaphors representing the opposite ends of the scale of quality code. In my presentation I am going to present my personal experiences on how to get out from the trap of the if jungle by making the code to adhere to the SRP and DRY principles. I am also going to show the advantages of the Railway Oriented Programing using the 3 different libraries. The examples are going to be stereotypical errors, fun and easy to follow. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...

Ruby Meditation

Speech of Valentyn Ostakh, Ruby Developer at Ruby Garage, at Ruby Meditation 27, Dnipro, 19.05.2019 Slideshare - Next conference - http://www.rubymeditation.com/ This talk explores basic concepts of GraphQL. The main goal is to show how GraphQL works and of what parts it consists of. From the Ruby side we will look at how to create a GraphQL schema. In addition, we will consider what pitfalls can be encountered at the start of work with GraphQL. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27

Ruby Meditation

Speech of Nihad Abbasov, Backend Engineer at Digital Classifieds, at Ruby Meditation #26 Kyiv 16.02.2019 Next conference - http://www.rubymeditation.com/ What to expect from Rails 6? Review of new features. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26

Ruby Meditation

Speech of Tetiana Chupryna, Backend developer at GitLab, at Ruby Meditation #26 Kyiv 16.02.2019 Next conference - http://www.rubymeditation.com/ We’ll talk about different types of vulnerabilities, scanning tools and the whole process per se. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26

Ruby Meditation

Speech of Dmytro Shapovalov, Infrastructure Engineer at Cossack Labs, at Ruby Meditation #26 Kyiv 16.02.2019 Next conference - http://www.rubymeditation.com/ Most modern applications live in a close cooperation with each other. We will talk about the ways to effectively use the modern techniques for monitoring the health of applications and look on tasks and typical implementation mistakes through the eyes of an infrastructure engineer. And we will also consider the Ruby libraries that help to implement all of this. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...

Ruby Meditation

Speech of Ike Kurghinyan, Senior Software Engineer at VChain LLC, at Ruby Meditation #26 Kyiv 16.02.2019 Next conference - http://www.rubymeditation.com/ «Is there one grand unifying rule to software engineering? Can so-called best practices be somehow derived from that unifying rule? This speech is a humble attempt to explore the subject and provide you with tools for your own explorations.» Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation * The channel of the organizers of the meetup https://t.me/incredevly

Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26

Ruby Meditation

Speech of Alexey Vasiliev, Software Engineer at Railsware, at Ruby Meditation #25 Kyiv 08.12.2018 Next conference - http://www.rubymeditation.com/ In this talk, Alexey will tell about the project in which was necessary to implement A/B testing and what came out of it in result Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25

Ruby Meditation

Speech of Dmytro Herasymuk, WEB developer at Softermii, at Ruby Meditation #25 Kyiv 08.12.2018 Next conference - http://www.rubymeditation.com/ The modern world demands to be faster and faster. Engineers create more powerful CPUs every day, but our app depends on third party apps too often in the WEB world. So, even the fastest CPU can't cope with I/O delays. On the one hand multithreading could help in such cases, but on the other hand, it seems to be a hard decision of using concurrency in production. Dmytro would like to share his experience in this field Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...

Ruby Meditation

Speech of Dmytro Shapovalov, Infrastructure Engineer at Cossack Labs, at Ruby Meditation #25 Kyiv 08.12.2018 Next conference - http://www.rubymeditation.com/ Making secure applications is not easy, especially when encryption tools are difficult and incomprehensible. We will talk about typical data security problems in web apps and how to implement encryption properly. We will review cryptographic approaches and exact tools that ensure that no sensitive data leaks from the application or database. Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...

Ruby Meditation

Rails App performance at the limit - Bogdan Gusiev

Ruby Meditation

GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23

Ruby Meditation

Plus de Ruby Meditation (20)