High performance queues with Cassandra

•

7 j'aime•8,884 vues

Everyone knows that Cassandra is a NoSQL solution for data storage. But often for processing of this data message queues are used with some existing messaging provider. Due to this, there is inconsistency of data sometimes and an additional infrastructure level to maintain. Since one of our services stores all the data in Cassandra, we have developed a solution for message queues that automatically gained a lot of useful features: scalability, high availability and flexibility. This solution I will present in the talk.

Technologie Business

High performance
queues with Cassandra
Mikalai Alimenkou
http://xpinjection.com
@xpinjection

How to
process
data, master
?

Asynchronously!

Some specific requirements
1

Message
Queue

External Service
Provider

Some specific requirements
Message
Queue

1

External Service
Provider

Some specific requirements
2

Message
Queue

External Service
Provider

Some specific requirements
Message
Queue

2
1
Redeliver
after 1 hour

External Service
Provider

Some specific requirements
2

Message
Queue

External Service
Provider

Redeliver
after 1 hour

Redelivery business logic and external service
hourly based usage limits

Message batches “station”
QUEUE
NOW

ALMOST READY

+1 HOUR

WAITING

+6 HOURS

WAITING

+12 HOURS

WAITING

System components: Message
MESSAGE = REAL REQUEST DATA WITH UNIQUE ID
1ST FIELD

3rd FIELD

{ field data}

MESSAGE ID

2nd FIELD
{ field data}

{ field data}

ID FORMAT ALLOWS 4096 MESSAGES
PER MILLISECOND FROM ONE NODE
Timestamp

44 bits

Counter

12 bits

Cluster node ID

8 bits

System components: Batch
• Open for at least 1 second
• Closing if opened for > 10 seconds
• Closing if has > 100 messages
Ascending columns ordering
1ST MESSSAGE ID

2nd MESSAGE ID

3rd MESSAGE ID

BATCH ID { opt message data} { opt message data} { opt message data}

ID FORMAT REQUIRE BATCH TO BE OPENED FOR > 1 SECOND
Timestamp
Rounded to seconds

Cluster node ID + Batch Type
Last 3 digits

System components: Queue
• Similar to batch
• Unlimited
• May have batches with past time

Ascending columns ordering
1ST BATCH ID
QUEUE NAME

2nd BATCH ID

3rd BATCH ID

{ processed at }

{ processed at }

{ processed at }

System components: Broker
batches polling
BROKER
check batch time
process batch
PROCESSOR

QUEUE

lock batch for
processing

ZOOKEEPER

•
•
•
•
•

Natural pre-fetch thanks to batches
Easy to control messages processing
Simple concurrency model
Easy scalable between nodes
No high loading on Cassandra

System components: Processor
PROCESSOR
OK

System components: Processor
PROCESSOR

redeliver
on failure
ANOTHER BATCH

•
•
•
•

Tries to process messages as quickly as possible
On error just redeliver message
Messages are processed concurrently
Any redelivery business logic is easy to implement

Warnings and benefits
• Message and batch must be
checked before processing
• Hard to explain “queue” size
• Separate columns for status
tracking of message
• Perform correct compaction
from time to time

• Expected loading is handled
with single node
• Everything works on
commodity hardware
• Single storage for all data
• System is easily scalable and
reliable (no message was
lost)

@xpinjection
http://xpinjection.com
mikalai.alimenkou@xpinjection.com

Contenu connexe

Tendances

Apache kafka

Rahul Jain

Storm is a realtime fault-tolerant and distributed stream data processing system. Storm is used to run various critical computations in Twitter at scale, and in real-time. This talk was presented at the SIGMOD 2014 conference that accepted our paper Storm@Twitter. It describes the architecture of Storm and its methods for distributed scale-out and fault-tolerance. This presentation also describes how queries (aka. topologies) are executed in Storm, and presents some operational stories based on running Storm at Twitter. We also present results from an empirical evaluation demonstrating the resilience of Storm in dealing with machine failures.

Storm@Twitter, SIGMOD 2014

Karthik Ramasamy

Finite state automata and transducers made it into Lucene fairly recently, but already show a very promising impact on search performance. This data structure is rarely exploited because it is commonly (and unfairly) associated with high complexity. During the talk, I will try to show that automata and transducers are in fact very simple, their construction can be very efficient (memory and time-wise) and their field of applications very broad.

Dawid Weiss- Finite state automata in lucene

Lucidworks (Archived)

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot While the demands for real-time analytics are growing in leaps and bounds, the analytics software must rely on streaming platforms for ingesting high volumes of data that's traveling in lightning speed down the pipeline. We will take a look at 2 powerful open source Apache platforms: Pulsar and Pinot, that work hand-in-hand together to deliver the analytical results which bring great value to your systems. Presenters: Mary Grygleski - Streaming Developer Advocate & Mark Needham - Developer Relations Engineer at StarTree Note: This webinar will be recorded and later posted on our Webinar page (https://altinity.com/webinarspage/) or Altinity official Youtube channel (https://www.youtube.com/@Altinity).

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot

Altinity Ltd

Consuming RealTime Signals in Solr

Umesh Prasad

Understanding and Improving Code Generation

Databricks

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Alluxio, Inc.

Today we will span from edge to any and all clouds to support data collection, real-time streaming, sensor ingest, edge computing, IoT use cases and edge AI. Apache Pulsar allows us to build computing at the edge and produce and consume messages at scale in any IoT, hybrid or cloud environment. Apache Pulsar supports MoP which allows for MQTT protocol to be used for high speed messaging. We will teach you to quickly build scalable open source streaming applications regardless of if you are running in containers, pods, edge devices, VMs, on-premise servers, moving vehicles and any cloud.

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021

StreamNative

고승범(peter.ko) / kakao corp.(인프라2팀) --- 카카오에서는 빅데이터 분석, 처리부터 모든 개발 플랫폼을 이어주는 솔루션으로 급부상한 카프카(kafka)를 전사 공용 서비스로 운영하고 있습니다. 전사 공용 카프카를 직접 운영하면서 경험한 트러블슈팅과 운영 노하우 등을 공유하고자 합니다. 특히 카프카를 처음 접하시는 분들이나 이미 사용 중이신 분들이 많이 궁금해하는 프로듀서와 컨슈머 사용 시의 주의점 등에 대해서도 설명합니다.

카프카, 산전수전 노하우

if kakao

Advanced nGrinder

JunHo Yoon

For many securities organizations, post-trade processing is expensive, cumbersome, and time-consuming. This is in part due to the massive volumes of data required for processing a trade and the limited agility of the technology many organizations rely on today. In order to create efficiencies and move faster, many Financial Services organizations are working with AWS to implement post-trade solutions built with AWS’ storage services (S3 and Glacier) and big data capabilities (Athena, EMR, Redshift, and QuickSight ). In this session, AWS will walk through a trade capture and regulatory reporting solution that utilizes the aforementioned AWS services. We will also provide guidance around obtaining data-driven insights (from pixels to pictures), bolstering encryption with Amazon KMS, and maintaining transparency and control with Amazon CloudWatch and Amazon CloudTrail (which also helps meet SEC Rule 613 that requires the creation of comprehensive consolidated audit trails).

FSI301 An Architecture for Trade Capture and Regulatory Reporting

Amazon Web Services

Most query engines follow an interpreter-based approach where a SQL query is translated into a tree of relational algebra operations then fed through a conventional tuple-based iterator model to execute the query. We will explore the overhead associated with this approach and how the performance of query execution on columnar data can be improved using run-time code generation via LLVM. Generally speaking, the best case for optimal query execution performance is a hand-written query plan that does exactly what is needed by the query for the exact same data types and format. Vectorized query processing models amortize the cost of function calls. However, research has shown that hand-written code for a given query plan has the potential to outperform the optimizations associated with a vectorized query processing model. Over the last decade, the LLVM compiler framework has seen significant development. Furthermore, the database community has realized the potential of LLVM to boost query performance by implementing JIT query compilation frameworks. With LLVM, a SQL query is translated into a portable intermediary representation (IR) which is subsequently converted into machine code for the desired target architecture. Dremio is built on top of Apache Arrow’s in-memory columnar vector format. The in-memory vectors map directly to the vector type in LLVM and that makes our job easier when writing the query processing algorithms in LLVM. We will talk about how Dremio implemented query processing logic in LLVM for some operators like FILTER and PROJECT. We will also discuss the performance benefits of LLVM-based vectorized query execution over other methods. Speaker Siddharth Teotia, Dremio, Software Engineer

Using LLVM to accelerate processing of data in Apache Arrow

DataWorks Summit

Over the years I've spoken many times about what Event Sourcing is and shared many of the good, the bad and the ugly parts of it in blog posts and various talks. However, I've never talked about how to actually build a system based on this architecture style. I keep getting the same questions over and over again. Like when to apply Event Sourcing and at what architectural level. How to deal with transactional boundaries within and outside the domain. How to build projections that are autonomous, reliable and self-supporting. How to deal with upgrades and blue-green deployments. But also on how to handle bugs, design mistakes and crashing projections. Having made a lot of these mistakes myself over these years, it's time to share my current thoughts and opinions about this. Since the .NET space has a pretty rich set of open-source projections to support this, the examples and code will be .NET. But the concepts are universal, so don't let that scare you off.

Event Sourcing from the Trenches (with examples from .NET)

Dennis Doomen

ksqlDB로 시작하는 스트림 프로세싱

confluent

Presto At Treasure Data

Taro L. Saito

다양한 하둡에코 소프트웨어 성능을 검증하려는 목적으로 성능 테스트 환경을 구성해보았습니다. ELK, JMeter를 활용해 구성했고 Kafka에 적용해 보았습니다. 프로젝트에서 요구되는 성능요건을 고려해 다양한 옵션을 조정해 시뮬레이션 해볼수 있습니다. 처음 적용한 뒤 2년 정도가 지났지만, kafka 만이 아니다 다른 Hadoop eco 및 Custom Solution에도 유용하게 활용 가능하겠습니다.

카프카(kafka) 성능 테스트 환경 구축 (JMeter, ELK)

Hyunmin Lee

Spark 의 핵심은 무엇인가? RDD! (RDD paper review)

Yongho Ha

http://flink-forward.org/kb_sessions/scaling-stream-processing-with-apache-flink-to-very-large-state/ The majority of streaming programs is ‘stateful’: Windowed Aggregations, Sessions, Joins, Complex Event Processing, Tables – they all require to keep some form of state across individual events. With the migration of more and more complex batch jobs or data processing pipelines to streaming applications, some streaming programs need to keep terabytes of state. Apache Flink implements a checkpointing-based recovery mechanism that guarantees exactly-once semantics for state also in the presence of failures. The cost of checkpointing and recovery depends on the size of the program’s state. In this talk, we will discuss the current status of stateful processing in Apache Flink, as well as the ongoing efforts to make Flink’s fault tolerance mechanism scale to very large state sizes, supporting frequent checkpoints and faster recovery of large state, without requiring excessive numbers of machines.

Stephan Ewen - Scaling to large State

Flink Forward

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Massive Data Processing in Adobe Using Delta Lake

Databricks

If you’re trying to process financial market data, monitor IoT sensor metrics or run real-time fraud detection, you’ll be thinking of stream processing. Stream processing sounds wonderful in concept, but scaling and debugging stream processing frameworks on distributed systems can be a nightmare. In clustered environments, your logs are scattered across many different computers making errors and strange behaviors are hard to trace. On frameworks like Apache Storm, the many layers of abstraction make it difficult to predict performance and do capacity planning. In micro batching frameworks like Spark Streaming, stateful aggregations can be a hassle. Moreover, in most of the existing frameworks, changing a single line of code requires a full topology redeploy causing operational strain. Concord strives to solve all the challenges above. In this talk, you’ll learn how Concord differs from other stream processing frameworks and how Concord can provide flexibility, simplicity, and predictable performance with help from Apache Mesos. https://databythebay2016.sched.org/event/6EPy/concord-simple-amp-flexible-stream-processing-on-apache-mesos

Concord: Simple & Flexible Stream Processing on Apache Mesos: Data By The Bay...

Concord

Tendances (20)

Apache kafka

Storm@Twitter, SIGMOD 2014

Dawid Weiss- Finite state automata in lucene

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot

Consuming RealTime Signals in Solr

Understanding and Improving Code Generation

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021

카프카, 산전수전 노하우

Advanced nGrinder

FSI301 An Architecture for Trade Capture and Regulatory Reporting

Using LLVM to accelerate processing of data in Apache Arrow

Event Sourcing from the Trenches (with examples from .NET)

ksqlDB로 시작하는 스트림 프로세싱

Presto At Treasure Data

카프카(kafka) 성능 테스트 환경 구축 (JMeter, ELK)

Spark 의 핵심은 무엇인가? RDD! (RDD paper review)

Stephan Ewen - Scaling to large State

Massive Data Processing in Adobe Using Delta Lake

Concord: Simple & Flexible Stream Processing on Apache Mesos: Data By The Bay...

Similaire à High performance queues with Cassandra

Donatas Mažionis, Building low latency web APIs

Tanya Denisyuk

Network protocols and vulnerabilities

G Prachi

Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays. In this talk, we are going to look at some of the most common misconceptions about stream processing and debunk them. - Myth 1: Streaming is approximate and exactly-once is not possible. - Myth 2: Streaming is for real-time only. - Myth 4: Streaming is harder to learn than Batch Processing. - Myth 3: You need to choose between latency and throughput. We will look at these and other myths and debunk them at the example of Apache Flink. We will discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.

Debunking Common Myths in Stream Processing

DataWorks Summit/Hadoop Summit

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

In this talk, we'll explain our journey from having near-zero monitoring to having all of our infrastructure monitored with the necessary metrics and alerts. We will share with the audience some of the mistakes we did and what lessons we have learned. We currently have around 200 instances monitored with a comfortable cost-effective in-house monitoring stack based on Prometheus. We want to demonstrate that you don't need to have a big fleet to embrace Prometheus and that it is a non-expensive solution for monitoring. ---------- ShuttleCloud is a small startup specialized in email and contacts migrations. We developed a reliable migration platform in high availability used by clients like Gmail, GContacts and Comcast. For example, Gmail alone has imported data for 3 million users with our API and we process hundreds of terabytes every month. ------------- Follow us on Twitter: @ShuttleCloud: https://twitter.com/ShuttleCloud @ShuttleCloudEng: https://twitter.com/ShuttleCloudEng ShuttleCloud.com

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016

ShuttleCloud

This presentation was part of EXPERTALKS: Nov 2012 conducted at Equal Experts India on 24th Nov 2012. Visit http://meetup.com/expertalks to know more... ------------------------------------------------------------------------------------- Ever wondered how Facebook is able to support user requests growing at thousands per second without ever crashing ? Any enterprise application that we use today; whether it’s a banking website or an eCommerce app; uses Clustering. Clustering addresses some of the most crucial demands of today’s ever-growing user base; including scalability and 24/7 availability amongst others. This presentation discusses the concept of web application clustering. FOCUS AREAS: * Why clustering * Clustering Types * Load Balancing Algorithms * Fault Tolerance It also has a guide workshop on how to build a Vertical Cluster using Tomcat.

EXPERTALKS: Nov 2012 - Web Application Clustering

EXPERTALKS

Openstack meetup lyon_2017-09-28

Xavier Lucas

Kafka overview v0.1

Mahendran Ponnusamy

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

confluent

Amazon Kinesis is a fully managed, cloud-based service for real-time data processing over large, distributed data streams. Customers who use Amazon Kinesis can continuously capture and process real-time data such as website clickstreams, financial transactions, social media feeds, IT logs, location-tracking events, and more. In this session, we first focus on building a scalable, durable streaming data ingest workflow, from data producers like mobile devices, servers, or even a web browser, using the right tool for the right job. Then, we cover code design that minimizes duplicates and achieves exactly-once processing semantics in your elastic stream-processing application, built with the Kinesis Client Library. Attend this session to learn best practices for building a real-time streaming data architecture with Amazon Kinesis, and get answers to technical questions frequently asked by those starting to process streaming events.

(BDT403) Best Practices for Building Real-time Streaming Applications with Am...

Amazon Web Services

3.2 Streaming and Messaging

振东刘

Ingestion and Dimensions Compute and Enrich using Apache Apex

Apache Apex

2.communcation in distributed system

Gd Goenka University

Scalable IoT platform

Swapnil Bawaskar

OpenStack High Availability

Jakub Pavlik

OpenStack HA

tcp cloud

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, Paul will reveal how he architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. Paul will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from his experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day. Melbourne Big Data Meetup, March 5 2020 https://www.eventbrite.com/e/melbourne-big-data-meetup-realtime-anomaly-detection-with-cassandra-kafka-tickets-93028445585

Melbourne Big Data Meetup Talk: Scaling a Real-Time Anomaly Detection Applica...

Paul Brebner

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Performance

Christophe Marchal

Data Stream Processing with Apache Flink

Fabian Hueske

Similaire à High performance queues with Cassandra (20)

Donatas Mažionis, Building low latency web APIs

Network protocols and vulnerabilities

Debunking Common Myths in Stream Processing

Stephan Ewen - Experiences running Flink at Very Large Scale

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016

EXPERTALKS: Nov 2012 - Web Application Clustering

Openstack meetup lyon_2017-09-28

Kafka overview v0.1

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

(BDT403) Best Practices for Building Real-time Streaming Applications with Am...

3.2 Streaming and Messaging

Ingestion and Dimensions Compute and Enrich using Apache Apex

2.communcation in distributed system

Scalable IoT platform

OpenStack High Availability

OpenStack HA

Melbourne Big Data Meetup Talk: Scaling a Real-Time Anomaly Detection Applica...

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Performance

Data Stream Processing with Apache Flink

Plus de Mikalai Alimenkou

Люди в мире Agile используют Story Points - для Agile коучей и тренеров это самый простой способ объяснить, как следует проводить оценку и планирование в «новом мире». Но тогда эта простая концепция нарушает реальные практические кейсы. В настоящее время команды состоят из очень специализированных людей, работающих над бэкендом, фронтэндом, тестировании, инфраструктуре и прочим. Для них почти невозможно иметь общий уровень сложности. Это только одна из проблем, которые мы собираемся осветить в этом докладе. Чтобы оставаться конструктивным, а не просто старомодным парнем из XP, Николай поделится своим опытом с более точной и прагматичной техникой оценки/планирования - планированием на основе возможностей.

Rise and fall of Story Points. Capacity based planning from the trenches.

Mikalai Alimenkou

We have spent many years testing our applications and systems manually and with test automation tools. During this time many bug root causes have been classified and could be detected automatically with special static analysis tools. Most of them could be applied at the early stages of development even before code is integrated into the main development branch. In this talk, I will go through available solutions and demonstrate what kinds of issues may be detected automatically reducing the time and effort of traditional testing.

Static analysis tools as the best friend of QA

Mikalai Alimenkou

In this talk, we will go through the design process of modern CI/CD for the microservices-based system with Kubernetes support. We will discuss how to verify consistency between microservices, apply different levels of quality gates and promote artifacts between environments. Thanks to Kubernetes we will review different approaches of environment resources optimization for development needs during CI/CD cycles.

Modern CI/CD in the microservices world with Kubernetes

Mikalai Alimenkou

Most of people nowadays think microservices architecture is a great way to build any system. They visit conference talks, read books and review tutorials, where ‘hello world’ applications are built just in several minutes using microservices approach. But the reality is not so wonderful and one of the biggest pain is hidden inside distributed business transactions. In monolith application this topic is almost completely covered with DB level transactions. In distributed world you have to face many issues trying to implement reliable and consistent business logic. In this talk we will review different types of problems related to distributed business transactions, popular SAGA pattern, frameworks and techniques to simplify your life without compromising quality of the system.

Saga about distributed business transactions in microservices world

Mikalai Alimenkou

Effectiveness tips from Kubernetes trenches by Captain Obvious

Mikalai Alimenkou

For a long time DB related testing in Java world has been a real pain and most developers tried to reduce number of such tests as much as possible. With good in-memory database implementations like H2, schema migration solutions like Liquibase or Flyway, containerization with libraries like TestContainers, database management is now much simpler. But test data management is still a pain. Some developers use SQL dumps, others insert data via JPA/JDBC or rely on prepared data sets. Good old DBUnit may be a good option, but it is not so developer friendly and not adopted well for modern annotations driven development style. Database Rider closes the gap between modern Java development environment and DBUnit, bringing DBUnit closer to your JUnit tests, so database testing will feel like a breeze. In addition to flexible data sets management this library provides other useful features: programmatic data sets definition, leak hunting, data sets export, constraints management, etc. As contributor and loyal user for many years, I would like to share my experience with Database Rider and demonstrate how to make database testing a fun again!

Ride the database in JUnit tests with Database Rider

Mikalai Alimenkou

I think almost everybody experienced cases when things are moving very slowly in IT companies or teams. You have many people, talented engineers, Agile process and development speed is still below expectations. We try to focus on performance and efficiency last 10 years, improving our practices and tools. But we are still there in terms of speed when they are applied to real life cases. How is it possible? In this talk we will review the concept of waste circles and understand what are the main sources of time waste in development process. This concept would help you to check your processess, focus on right things and achieve much better results in your organization or team.

Wastful waste or why everything is so slow in development

Mikalai Alimenkou

Nowadays traditional layered monolithic architecture in Java world is not so popular as 5-10 years ago. I remember how we wrote tons of code for each layer repeating almost the same parts for every application. Add unit and integration testing to understand how much time and efforts has been spent on repeatable work. All cool ideas around DDD (domain driven design) and Hexagonal Architecture was just a nice theory because reality hasn’t allow us to implement it easily. Even Dependency Injection with Spring framework was completely focused on traditional layered approach, not even talking about JavaEE platform. Today we have Spring Boot ecosystem covering most of our needs for integration with almost all possible technologies and microservices architectural trend, enabling completely new approach to build Java applications around domain model. It is so natural to build Java domain-oriented services and connect them with external world using ports and adapters, that Hexagonal Architecture is almost enabled by default. You just need to switch your way of thinking…

Hexagonal architecture with Spring Boot

Mikalai Alimenkou

Wastful waste or why everything is so slow in development

Mikalai Alimenkou

DevOps become a buzzword in a last few years. Several companies, development and product teams have achieved quite impressive results in this area making cultural changes, transforming their processes and practices, introducing new roles, tools and techniques. Do you think is achievable for you team or it’s still a bunch of drama? There is no common approach for measuring achievements and understanding how much DevOps’ich the current team/company is. In this talk I will provide attendees with basic checklist to start with and some reliable tools/techniques to monitor progress of “DevOps transformation”.

DevOps checklist or how to understand where is your team in DevOps landscape ...

Mikalai Alimenkou

DevOps is a hot topic during last several years. Some companies, teams and products have achieved quite impressive results in this area making cultural changes, transforming their processes and practices, introducing new roles, tools and techniques. At the same time there is no common approach for measuring achievements and understanding “how DevOps” the current team/company is. In this talk I will provide attendees with basic checklist to start with and some reliable tools/techniques to monitor progress of “DevOps transformation”.

DevOps checklist or how to understand where is your team in DevOps landscape

Mikalai Alimenkou

Почти год мы в Whirl Software разрабатываем систему Медкарта в масштабе целой страны. За это время мы столкнулись с множеством интересных сложностей и проблем, часть из которых успешно победили, а для некоторых хорошего решения до сих пор не найдено. В этом докладе мы поделимся накопленным практическим опытом и некоторыми техническими решениями, которые могут быть полезны в рамках разработки электронных медицинских систем.

Практические трудности в разработке Медкарты для целой страны

Mikalai Alimenkou

Hexagonal architecture with Spring Boot [EPAM Java online conference]

Mikalai Alimenkou

Almost any application or software system manages data. It is hard to imagine test automation that is not affected by this fact. There are many differenct approaches how to prepare system under test, providing predefined test data: use application UI, invoke API methods, run business logic directly, access DB from test scenarios, etc. In this talk we will review most of existing approaches, starting from the easiest and the most popular ones and finishing with really tricky ways to manage your test data for large distributed systems. There is no ideal solution for every case or silver bullet, but I hope your toolset will become wider after visiting this talk.

Bro, manage test data like a pro! [QA Fest 2018]

Mikalai Alimenkou

Вот уже более 10 лет Agile движение шагает по Украине и стучится практически в каждую компанию. Но приносят ли новые процессы, принципы и практики реальную практическую пользу? Получается ли изменить к лучшему команды, проекты, компании? Я в роли консультанта за эти 10 лет поработал с более чем 100 компаниями, поэтому повидал много хорошего и плохого. В данном докладе хочется пробежаться по основным проблемам, сложностям и анти-паттернам в переходе отечественных компаний на "Agile рельсы". Мы рассмотрим какие практики не очень хорошо приживаются, от чего страдает большинство команд, какие основные препятствия встречают на своем пути и как умудряются их обходить. Я надеюсь, доклад поможет зародиться множеству интересных дискуссий.

Agile antipatterns: review after 10 years of practice

Mikalai Alimenkou

Hexagonal architecture with Spring Boot

Mikalai Alimenkou

Bro, manage test data like a pro!

Mikalai Alimenkou

Тестировщики часто говорят о противостоянии и конфликтах с разработчиками. Но ведь есть команды, где все живут в мире и согласии. Видимо что-то тут не так? Я хочу поговорить о том, как тестировщиков видят сами разработчики. В докладе будет проведена забавная классификация. Кроме известного всем тестировщика-обезьянки будут представлены тестировщик-муха, тестировщик-нацист, тестировщик-панда и многие другие герои. Высможете лишний раз задуматься над тем, как вас видят со стороны и, возможно, изменить ситуацию к лучшему. Доклад будет также полезен менеджерам проектов и лидерам команд. Вы сможете быстрее распознавать те или иные шаблоны поведения тестировщикови принимать меры по повышению уровня командной работы. Приходите, будет интересно!

Бытовая классификация тестировщиков с точки зрения разработчика

Mikalai Alimenkou

Usually it is hard to analyze personal effectiveness and detect wastes in development process because developer’s work decomposition is not transparent and available for analysis. As a good sample of ineffective process imagine developer, who spends 1 day on task implementation and then reimplements it several times according to code review notes during next 2 days. Or another developer, who is waiting for code review during 2 days, switching context to other tasks, finally gets notes and switches back to initial task, trying to refresh all details in his head. And so on and so forth… Code review tool usage helps to aggregate lots of useful information about any code change at any stage (static analysis, code review, rework, acceptance, integration into main branch). In this talk I’m going to demontrate how this information could be used for detailed analysis of development effectiveness and wastes detection. Based on mentioned analysis you could implement many improvements for your development process and then measure their success.

Code Review tool for personal effectiveness and waste analysis

Mikalai Alimenkou

During last several years DevOps became strong buzzword used almost in every project, team and company. But almost everywhere it is used in very funny and strange context. For example, existing ops guys are renamed to DevOps just to sell them to the client for more money. Or DevOps is used as new job title for some magically powerful person who is able to operate cloud environment and modern infrastructure related tools, leading team of old school ops and participating in management meetings. In this talk I’m going to review all different anti-patterns and bad practices in DevOps landscape using stories from my personal experience as Delivery Manager and independent consultant.

Funny stories and anti-patterns from DevOps landscape

Mikalai Alimenkou

Plus de Mikalai Alimenkou (20)

Rise and fall of Story Points. Capacity based planning from the trenches.

Static analysis tools as the best friend of QA

Modern CI/CD in the microservices world with Kubernetes

Saga about distributed business transactions in microservices world

Effectiveness tips from Kubernetes trenches by Captain Obvious

Ride the database in JUnit tests with Database Rider

Wastful waste or why everything is so slow in development

Hexagonal architecture with Spring Boot

Wastful waste or why everything is so slow in development

DevOps checklist or how to understand where is your team in DevOps landscape ...

DevOps checklist or how to understand where is your team in DevOps landscape

Практические трудности в разработке Медкарты для целой страны

Hexagonal architecture with Spring Boot [EPAM Java online conference]

Bro, manage test data like a pro! [QA Fest 2018]

Agile antipatterns: review after 10 years of practice

Hexagonal architecture with Spring Boot

Bro, manage test data like a pro!

Бытовая классификация тестировщиков с точки зрения разработчика

Code Review tool for personal effectiveness and waste analysis

Funny stories and anti-patterns from DevOps landscape

Dernier

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Dernier (20)