Map, Grep, and Sort are powerful Perl built-in functions that can simplify your code and improve performance.
New Perl users often skip learning about these three functions because they are considered an advanced topic and may not use them to their full advantage when they do use them. Experienced Perl users exploit these functions in amazing ways, but often the result is a confusing mess of unreadable and thus unmaintainable code. Refactoring existing code into these functions where appropriate can significantly improve the quality of the code *and* improve performance as well.
This talk will present how to take advantage of these powerful functions and have the resulting code understandable.
The document provides an overview of decision trees and machine learning techniques. It discusses how decision trees can be used to classify data and extract patterns, describes the ID3 algorithm for building decision trees using information gain, and notes some extensions to ID3 like C4.5 which allow continuous values and pruning of trees.
This document summarizes a research paper on strategic argumentation and its relationship to defeasible logic. It contains the following key points:
1. Strategic argumentation involves an adversarial dialogue game where players aim to prove or disprove a claim while avoiding playing arguments that could be used against them.
2. Deciding the outcome of an argument at each turn can be computed in polynomial time, but deciding the optimal set of arguments to play (the strategic argumentation problem) is NP-complete.
3. Defeasible logic can be used to model strategic argumentation and compute argument outcomes. The complexity results also apply to defeasible semantics and grounded semantics.
The document discusses principles for writing code that is easy to understand and maintain. It recommends using appropriate naming conventions that clearly convey meaning; consistent formatting and layout to improve aesthetics; comments to explain intent and issues; simplifying loops, logic and expressions; minimizing global variables and scope; and refactoring code into well-organized modules. The goal is to reduce frustration and time for future maintainers to understand the code.
This document provides an overview of regular expressions and the grep command in Unix/Linux. It defines what regular expressions are, describes common regex patterns like characters, character classes, anchors, repetition, and groups. It also explains the differences between the grep, egrep, and fgrep commands and provides examples of using grep with regular expressions to search files.
The document introduces MapReduce, describing how it allows for parallel processing of large datasets. MapReduce works by splitting data into smaller chunks that are processed (mapped) in parallel by worker nodes, and then combining (reducing) the results. The document outlines the Map and Reduce functions, and discusses how Hadoop is an open-source implementation of MapReduce that allows distributed processing of semi-structured data across clusters of machines.
Perl provides many powerful features and modules that allow developers to customize and extend the language. Some popular modules include Moose for object-oriented programming, TryCatch for exception handling inspired by Perl 6, and P5.10 features that backport Perl 6 functionality. While useful, some features like autoboxing and state variables could introduce subtle bugs if misused. Overall, Perl's extensibility makes it a very flexible language that can be adapted to many different use cases.
You Can Do It! Start Using Perl to Handle Your Voyager NeedsRoy Zimmer
This document provides an introduction to the Perl programming language. It discusses Perl nomenclature, basic syntax like variables and data types, control structures, file input/output, regular expressions, and more. The goal is to get readers started using Perl for their needs.
The document provides an overview of decision trees and machine learning techniques. It discusses how decision trees can be used to classify data and extract patterns, describes the ID3 algorithm for building decision trees using information gain, and notes some extensions to ID3 like C4.5 which allow continuous values and pruning of trees.
This document summarizes a research paper on strategic argumentation and its relationship to defeasible logic. It contains the following key points:
1. Strategic argumentation involves an adversarial dialogue game where players aim to prove or disprove a claim while avoiding playing arguments that could be used against them.
2. Deciding the outcome of an argument at each turn can be computed in polynomial time, but deciding the optimal set of arguments to play (the strategic argumentation problem) is NP-complete.
3. Defeasible logic can be used to model strategic argumentation and compute argument outcomes. The complexity results also apply to defeasible semantics and grounded semantics.
The document discusses principles for writing code that is easy to understand and maintain. It recommends using appropriate naming conventions that clearly convey meaning; consistent formatting and layout to improve aesthetics; comments to explain intent and issues; simplifying loops, logic and expressions; minimizing global variables and scope; and refactoring code into well-organized modules. The goal is to reduce frustration and time for future maintainers to understand the code.
This document provides an overview of regular expressions and the grep command in Unix/Linux. It defines what regular expressions are, describes common regex patterns like characters, character classes, anchors, repetition, and groups. It also explains the differences between the grep, egrep, and fgrep commands and provides examples of using grep with regular expressions to search files.
The document introduces MapReduce, describing how it allows for parallel processing of large datasets. MapReduce works by splitting data into smaller chunks that are processed (mapped) in parallel by worker nodes, and then combining (reducing) the results. The document outlines the Map and Reduce functions, and discusses how Hadoop is an open-source implementation of MapReduce that allows distributed processing of semi-structured data across clusters of machines.
Perl provides many powerful features and modules that allow developers to customize and extend the language. Some popular modules include Moose for object-oriented programming, TryCatch for exception handling inspired by Perl 6, and P5.10 features that backport Perl 6 functionality. While useful, some features like autoboxing and state variables could introduce subtle bugs if misused. Overall, Perl's extensibility makes it a very flexible language that can be adapted to many different use cases.
You Can Do It! Start Using Perl to Handle Your Voyager NeedsRoy Zimmer
This document provides an introduction to the Perl programming language. It discusses Perl nomenclature, basic syntax like variables and data types, control structures, file input/output, regular expressions, and more. The goal is to get readers started using Perl for their needs.
The document discusses various techniques for querying databases and generating reports from the query results using Perl. It provides examples of using DBI and SQL to query databases and format output, techniques for binding variables, preparing queries, and fetching and printing rows. Additional examples show merging and transforming tabular data for different output formats.
This document discusses functional programming concepts in Perl. It begins with a brief history of Perl and how functions like sort, grep, and map evolved over time and took on a more functional style. It then demonstrates how to write functional-style code in Perl using lexical scoping, first-class functions, and function composition. Examples include writing dice rolling and dice tower generation functions in a functional way. The document notes some limitations of functional programming in Perl, like lack of purity and difficulty making some concepts simple that are in functional languages, but suggests improvements coming in Perl 6 like static typing and better support for functional patterns.
Perl6 is a powerful programming language that incorporates many programming paradigms including functional, object oriented, reactive, and event based programming. It aims to provide programmers with a toolbox of features to build applications in many different styles. The language includes features like junctions, promises, channels, supplies, sets, roles, and strong support for Unicode. It also allows calling external native libraries and has built-in support for rational numbers, sequences, and lazy evaluation. Perl6 code examples are provided to demonstrate various features like List utilities, junctions, promises, channels/supplies, roles, and native calls. Further reading resources are also referenced.
Perl6 is a powerful programming language that incorporates many programming paradigms including functional, object-oriented, and reactive programming. It has built-in support for concepts like promises to handle asynchronous code, channels for communicating between asynchronous processes, and sets/bags for storing unique values. The language also has features for rational numbers, lazy evaluation, roles for composition, and easy integration with native libraries. Perl6 aims to provide programmers a flexible toolbox to build applications in many different styles.
Python quickstart for programmers: Python Kung Fuclimatewarrior
The document provides an overview of key Python concepts including data types, operators, control flow statements, functions, objects and classes. It discusses lists in depth, covering creation, iteration, searching and common list methods. It also briefly touches on modules, exceptions, inheritance and other advanced topics.
How to train your python: iterables (FR)Jordi Riera
Les iterables de python3 101: Slides axées sur les itérables (list, tuple, dict, etc.) décrivant de bonnes pratiques pour développeurs python débutants
There are a lot of operators in Perl 6, so many that it can be called an OOL: operator oriented language. Here I describe most of them from the angle of contexts, which Perl 6 has also much more than Perl 5.
Dealing with Legacy Perl Code - Peter ScottO'Reilly Media
The document discusses best practices for maintaining Perl code while avoiding burnout. It provides tips on dealing with legacy Perl code, avoiding common myths, testing code, improving code layout and readability, analyzing code for improvements, and handling inherited code. Key recommendations include adopting best practices, writing tests, using tools like perltidy and Devel modules, improving documentation, and refactoring code for clarity and maintainability.
The document discusses the performance implications of using map in void context versus for loops in Perl. A benchmark test was run replacing the number 1 with 'a' in an array of numbers 1-100, using different map and for loop approaches. The results showed that using a for loop with a subroutine reference like "do \&ss($_) for @arr" was the fastest approach, around 90% faster than using map in void context like "map s/1/a/, @arr". In general, for loops outperformed map when used in void context for this task.
The document provides an overview of the basics of Perl programming, including program structure, variables, reading files, matching patterns, and printing. It discusses shebangs, the use of strict and warnings, scalars, arrays, hashes, filehandles, regular expressions, printf formatting, and presents a sample program to read data from files and extract barcode numbers and pass/fail values to print.
This document discusses fundamentals of computing including the lambda calculus and Scheme programming language. It explains how lambda calculus forms the core of functional programming and shows how data types, functions, recursion and other programming concepts can be encoded using lambda calculus. It also demonstrates how Scheme aligns closely with the lambda calculus through its use of variables, functions, recursion and other features. References for further reading on related topics are provided at the end.
This document discusses using the C to Go translation tool c2go to translate C code implementing quicksort algorithms into Go code. It provides examples of translating simple quicksort C code, improving the translation by using a configuration file, and how c2go handles standard C functions like qsort by translating them to their Go equivalents. The examples demonstrate how c2go can generate valid Go code from C code but may require some manual fixes or configuration to handle certain data structures or language differences.
The document discusses how MapReduce can be used for various tasks related to search engines, including detecting duplicate web pages, processing document content, building inverted indexes, and analyzing search query logs. It provides examples of MapReduce jobs for normalizing document text, extracting entities, calculating ranking signals, and indexing individual words, phrases, stems and synonyms.
Why async and functional programming in PHP7 suck and how to get overr it?Lucas Witold Adamus
This presentation describes basic issues related to functional programming with PHP and solution for most of problems served by the library called PhpSlang.
The document provides an introduction to Perl programming and regular expressions. It begins with simple Perl programs to print text and take user input. It then covers executing external commands, variables, operators, loops, and file operations. The document also introduces regular expressions, explaining patterns, anchors, character classes, alternation, grouping, and repetition quantifiers. It provides examples and discusses principles for matching strings with regular expressions.
Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines
Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity.
Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline.
Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches.
Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage.
This session delves into how these challenges were addressed through:
1. Automate the setup of essential dependencies to ensure a consistent testing environment.
2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process.
3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases.
4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments.
5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices.
6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency.
7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues.
These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.
Hyperledger Besu 빨리 따라하기 (Private Networks)wonyong hwang
Hyperledger Besu의 Private Networks에서 진행하는 실습입니다. 주요 내용은 공식 문서인https://besu.hyperledger.org/private-networks/tutorials 의 내용에서 발췌하였으며, Privacy Enabled Network와 Permissioned Network까지 다루고 있습니다.
This is a training session at Hyperledger Besu's Private Networks, with the main content excerpts from the official document besu.hyperledger.org/private-networks/tutorials and even covers the Private Enabled and Permitted Networks.
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Ortus Solutions, Corp
Join us for a session exploring CommandBox 6’s smooth website transition and efficient deployment. CommandBox revolutionizes web development, simplifying tasks across Linux, Windows, and Mac platforms. Gain insights and practical tips to enhance your development workflow.
Come join us for an enlightening session where we delve into the smooth transition of current websites and the efficient deployment of new ones using CommandBox 6. CommandBox has revolutionized web development, consistently introducing user-friendly enhancements that catalyze progress in the field. During this presentation, we’ll explore CommandBox’s rich history and showcase its unmatched capabilities within the realm of ColdFusion, covering both major variations.
The journey of CommandBox has been one of continuous innovation, constantly pushing boundaries to simplify and optimize development processes. Regardless of whether you’re working on Linux, Windows, or Mac platforms, CommandBox empowers developers to streamline tasks with unparalleled ease.
In our session, we’ll illustrate the simple process of transitioning existing websites to CommandBox 6, highlighting its intuitive features and seamless integration. Moreover, we’ll unveil the potential for effortlessly deploying multiple websites, demonstrating CommandBox’s versatility and adaptability.
Join us on this journey through the evolution of web development, guided by the transformative power of CommandBox 6. Gain invaluable insights, practical tips, and firsthand experiences that will enhance your development workflow and embolden your projects.
A neural network is a machine learning program, or model, that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions.
Just like life, our code must adapt to the ever changing world we live in. From one day coding for the web, to the next for our tablets or APIs or for running serverless applications. Multi-runtime development is the future of coding, the future is to be dynamic. Let us introduce you to BoxLang.
The document discusses various techniques for querying databases and generating reports from the query results using Perl. It provides examples of using DBI and SQL to query databases and format output, techniques for binding variables, preparing queries, and fetching and printing rows. Additional examples show merging and transforming tabular data for different output formats.
This document discusses functional programming concepts in Perl. It begins with a brief history of Perl and how functions like sort, grep, and map evolved over time and took on a more functional style. It then demonstrates how to write functional-style code in Perl using lexical scoping, first-class functions, and function composition. Examples include writing dice rolling and dice tower generation functions in a functional way. The document notes some limitations of functional programming in Perl, like lack of purity and difficulty making some concepts simple that are in functional languages, but suggests improvements coming in Perl 6 like static typing and better support for functional patterns.
Perl6 is a powerful programming language that incorporates many programming paradigms including functional, object oriented, reactive, and event based programming. It aims to provide programmers with a toolbox of features to build applications in many different styles. The language includes features like junctions, promises, channels, supplies, sets, roles, and strong support for Unicode. It also allows calling external native libraries and has built-in support for rational numbers, sequences, and lazy evaluation. Perl6 code examples are provided to demonstrate various features like List utilities, junctions, promises, channels/supplies, roles, and native calls. Further reading resources are also referenced.
Perl6 is a powerful programming language that incorporates many programming paradigms including functional, object-oriented, and reactive programming. It has built-in support for concepts like promises to handle asynchronous code, channels for communicating between asynchronous processes, and sets/bags for storing unique values. The language also has features for rational numbers, lazy evaluation, roles for composition, and easy integration with native libraries. Perl6 aims to provide programmers a flexible toolbox to build applications in many different styles.
Python quickstart for programmers: Python Kung Fuclimatewarrior
The document provides an overview of key Python concepts including data types, operators, control flow statements, functions, objects and classes. It discusses lists in depth, covering creation, iteration, searching and common list methods. It also briefly touches on modules, exceptions, inheritance and other advanced topics.
How to train your python: iterables (FR)Jordi Riera
Les iterables de python3 101: Slides axées sur les itérables (list, tuple, dict, etc.) décrivant de bonnes pratiques pour développeurs python débutants
There are a lot of operators in Perl 6, so many that it can be called an OOL: operator oriented language. Here I describe most of them from the angle of contexts, which Perl 6 has also much more than Perl 5.
Dealing with Legacy Perl Code - Peter ScottO'Reilly Media
The document discusses best practices for maintaining Perl code while avoiding burnout. It provides tips on dealing with legacy Perl code, avoiding common myths, testing code, improving code layout and readability, analyzing code for improvements, and handling inherited code. Key recommendations include adopting best practices, writing tests, using tools like perltidy and Devel modules, improving documentation, and refactoring code for clarity and maintainability.
The document discusses the performance implications of using map in void context versus for loops in Perl. A benchmark test was run replacing the number 1 with 'a' in an array of numbers 1-100, using different map and for loop approaches. The results showed that using a for loop with a subroutine reference like "do \&ss($_) for @arr" was the fastest approach, around 90% faster than using map in void context like "map s/1/a/, @arr". In general, for loops outperformed map when used in void context for this task.
The document provides an overview of the basics of Perl programming, including program structure, variables, reading files, matching patterns, and printing. It discusses shebangs, the use of strict and warnings, scalars, arrays, hashes, filehandles, regular expressions, printf formatting, and presents a sample program to read data from files and extract barcode numbers and pass/fail values to print.
This document discusses fundamentals of computing including the lambda calculus and Scheme programming language. It explains how lambda calculus forms the core of functional programming and shows how data types, functions, recursion and other programming concepts can be encoded using lambda calculus. It also demonstrates how Scheme aligns closely with the lambda calculus through its use of variables, functions, recursion and other features. References for further reading on related topics are provided at the end.
This document discusses using the C to Go translation tool c2go to translate C code implementing quicksort algorithms into Go code. It provides examples of translating simple quicksort C code, improving the translation by using a configuration file, and how c2go handles standard C functions like qsort by translating them to their Go equivalents. The examples demonstrate how c2go can generate valid Go code from C code but may require some manual fixes or configuration to handle certain data structures or language differences.
The document discusses how MapReduce can be used for various tasks related to search engines, including detecting duplicate web pages, processing document content, building inverted indexes, and analyzing search query logs. It provides examples of MapReduce jobs for normalizing document text, extracting entities, calculating ranking signals, and indexing individual words, phrases, stems and synonyms.
Why async and functional programming in PHP7 suck and how to get overr it?Lucas Witold Adamus
This presentation describes basic issues related to functional programming with PHP and solution for most of problems served by the library called PhpSlang.
The document provides an introduction to Perl programming and regular expressions. It begins with simple Perl programs to print text and take user input. It then covers executing external commands, variables, operators, loops, and file operations. The document also introduces regular expressions, explaining patterns, anchors, character classes, alternation, grouping, and repetition quantifiers. It provides examples and discusses principles for matching strings with regular expressions.
Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines
Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity.
Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline.
Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches.
Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage.
This session delves into how these challenges were addressed through:
1. Automate the setup of essential dependencies to ensure a consistent testing environment.
2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process.
3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases.
4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments.
5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices.
6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency.
7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues.
These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.
Hyperledger Besu 빨리 따라하기 (Private Networks)wonyong hwang
Hyperledger Besu의 Private Networks에서 진행하는 실습입니다. 주요 내용은 공식 문서인https://besu.hyperledger.org/private-networks/tutorials 의 내용에서 발췌하였으며, Privacy Enabled Network와 Permissioned Network까지 다루고 있습니다.
This is a training session at Hyperledger Besu's Private Networks, with the main content excerpts from the official document besu.hyperledger.org/private-networks/tutorials and even covers the Private Enabled and Permitted Networks.
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Ortus Solutions, Corp
Join us for a session exploring CommandBox 6’s smooth website transition and efficient deployment. CommandBox revolutionizes web development, simplifying tasks across Linux, Windows, and Mac platforms. Gain insights and practical tips to enhance your development workflow.
Come join us for an enlightening session where we delve into the smooth transition of current websites and the efficient deployment of new ones using CommandBox 6. CommandBox has revolutionized web development, consistently introducing user-friendly enhancements that catalyze progress in the field. During this presentation, we’ll explore CommandBox’s rich history and showcase its unmatched capabilities within the realm of ColdFusion, covering both major variations.
The journey of CommandBox has been one of continuous innovation, constantly pushing boundaries to simplify and optimize development processes. Regardless of whether you’re working on Linux, Windows, or Mac platforms, CommandBox empowers developers to streamline tasks with unparalleled ease.
In our session, we’ll illustrate the simple process of transitioning existing websites to CommandBox 6, highlighting its intuitive features and seamless integration. Moreover, we’ll unveil the potential for effortlessly deploying multiple websites, demonstrating CommandBox’s versatility and adaptability.
Join us on this journey through the evolution of web development, guided by the transformative power of CommandBox 6. Gain invaluable insights, practical tips, and firsthand experiences that will enhance your development workflow and embolden your projects.
A neural network is a machine learning program, or model, that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions.
Just like life, our code must adapt to the ever changing world we live in. From one day coding for the web, to the next for our tablets or APIs or for running serverless applications. Multi-runtime development is the future of coding, the future is to be dynamic. Let us introduce you to BoxLang.
The Role of DevOps in Digital Transformation.pdfmohitd6
DevOps plays a crucial role in driving digital transformation by fostering a collaborative culture between development and operations teams. This approach enhances the speed and efficiency of software delivery, ensuring quicker deployment of new features and updates. DevOps practices like continuous integration and continuous delivery (CI/CD) streamline workflows, reduce manual errors, and increase the overall reliability of software systems. By leveraging automation and monitoring tools, organizations can improve system stability, enhance customer experiences, and maintain a competitive edge. Ultimately, DevOps is pivotal in enabling businesses to innovate rapidly, respond to market changes, and achieve their digital transformation goals.
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...kalichargn70th171
Visual testing plays a vital role in ensuring that software products meet the aesthetic requirements specified by clients in functional and non-functional specifications. In today's highly competitive digital landscape, users expect a seamless and visually appealing online experience. Visual testing, also known as automated UI testing or visual regression testing, verifies the accuracy of the visual elements that users interact with.
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Paul Brebner
Closing talk for the Performance Engineering track at Community Over Code EU (Bratislava, Slovakia, June 5 2024) https://eu.communityovercode.org/sessions/2024/why-apache-kafka-clusters-are-like-galaxies-and-other-cosmic-kafka-quandaries-explored/ Instaclustr (now part of NetApp) manages 100s of Apache Kafka clusters of many different sizes, for a variety of use cases and customers. For the last 7 years I’ve been focused outwardly on exploring Kafka application development challenges, but recently I decided to look inward and see what I could discover about the performance, scalability and resource characteristics of the Kafka clusters themselves. Using a suite of Performance Engineering techniques, I will reveal some surprising discoveries about cosmic Kafka mysteries in our data centres, related to: cluster sizes and distribution (using Zipf’s Law), horizontal vs. vertical scalability, and predicting Kafka performance using metrics, modelling and regression techniques. These insights are relevant to Kafka developers and operators.
These are the slides of the presentation given during the Q2 2024 Virtual VictoriaMetrics Meetup. View the recording here: https://www.youtube.com/watch?v=hzlMA_Ae9_4&t=206s
Topics covered:
1. What is VictoriaLogs
Open source database for logs
● Easy to setup and operate - just a single executable with sane default configs
● Works great with both structured and plaintext logs
● Uses up to 30x less RAM and up to 15x disk space than Elasticsearch
● Provides simple yet powerful query language for logs - LogsQL
2. Improved querying HTTP API
3. Data ingestion via Syslog protocol
* Automatic parsing of Syslog fields
* Supported transports:
○ UDP
○ TCP
○ TCP+TLS
* Gzip and deflate compression support
* Ability to configure distinct TCP and UDP ports with distinct settings
* Automatic log streams with (hostname, app_name, app_id) fields
4. LogsQL improvements
● Filtering shorthands
● week_range and day_range filters
● Limiters
● Log analytics
● Data extraction and transformation
● Additional filtering
● Sorting
5. VictoriaLogs Roadmap
● Accept logs via OpenTelemetry protocol
● VMUI improvements based on HTTP querying API
● Improve Grafana plugin for VictoriaLogs -
https://github.com/VictoriaMetrics/victorialogs-datasource
● Cluster version
○ Try single-node VictoriaLogs - it can replace 30-node Elasticsearch cluster in production
● Transparent historical data migration to object storage
○ Try single-node VictoriaLogs with persistent volumes - it compresses 1TB of production logs from
Kubernetes to 20GB
● See https://docs.victoriametrics.com/victorialogs/roadmap/
Try it out: https://victoriametrics.com/products/victorialogs/
What’s new in VictoriaMetrics - Q2 2024 UpdateVictoriaMetrics
These slides were presented during the virtual VictoriaMetrics User Meetup for Q2 2024.
Topics covered:
1. VictoriaMetrics development strategy
* Prioritize bug fixing over new features
* Prioritize security, usability and reliability over new features
* Provide good practices for using existing features, as many of them are overlooked or misused by users
2. New releases in Q2
3. Updates in LTS releases
Security fixes:
● SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4
● SECURITY: upgrade base docker image (Alpine)
Bugfixes:
● vmui
● vmalert
● vmagent
● vmauth
● vmbackupmanager
4. New Features
* Support SRV URLs in vmagent, vmalert, vmauth
* vmagent: aggregation and relabeling
* vmagent: Global aggregation and relabeling
* vmagent: global aggregation and relabeling
* Stream aggregation
- Add rate_sum aggregation output
- Add rate_avg aggregation output
- Reduce the number of allocated objects in heap during deduplication and aggregation up to 5 times! The change reduces the CPU usage.
* Vultr service discovery
* vmauth: backend TLS setup
5. Let's Encrypt support
All the VictoriaMetrics Enterprise components support automatic issuing of TLS certificates for public HTTPS server via Let’s Encrypt service: https://docs.victoriametrics.com/#automatic-issuing-of-tls-certificates
6. Performance optimizations
● vmagent: reduce CPU usage when sharding among remote storage systems is enabled
● vmalert: reduce CPU usage when evaluating high number of alerting and recording rules.
● vmalert: speed up retrieving rules files from object storages by skipping unchanged objects during reloading.
7. VictoriaMetrics k8s operator
● Add new status.updateStatus field to the all objects with pods. It helps to track rollout updates properly.
● Add more context to the log messages. It must greatly improve debugging process and log quality.
● Changee error handling for reconcile. Operator sends Events into kubernetes API, if any error happened during object reconcile.
See changes at https://github.com/VictoriaMetrics/operator/releases
8. Helm charts: charts/victoria-metrics-distributed
This chart sets up multiple VictoriaMetrics cluster instances on multiple Availability Zones:
● Improved reliability
● Faster read queries
● Easy maintenance
9. Other Updates
● Dashboards and alerting rules updates
● vmui interface improvements and bugfixes
● Security updates
● Add release images built from scratch image. Such images could be more
preferable for using in environments with higher security standards
● Many minor bugfixes and improvements
● See more at https://docs.victoriametrics.com/changelog/
Also check the new VictoriaLogs PlayGround https://play-vmlogs.victoriametrics.com/
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻campbellclarkson
Connect with fellow Trailblazers, learn from industry experts Glenda Thomson (Salesforce, Principal Technical Architect) and Will Dinn (Judo Bank, Salesforce Development Lead), and discover how to harness DevOps tools with Salesforce.
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISTier1 app
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...The Third Creative Media
"Navigating Invideo: A Comprehensive Guide" is an essential resource for anyone looking to master Invideo, an AI-powered video creation tool. This guide provides step-by-step instructions, helpful tips, and comparisons with other AI video creators. Whether you're a beginner or an experienced video editor, you'll find valuable insights to enhance your video projects and bring your creative ideas to life.
6. Copyright 2014 Daina Pettit
map, grep, sort – slide 6
General Form—code blocks
Damian Conway in Perl Best Practices*
recommends:
“Always use a block with a map and grep”
This is a syntactic aid suggestion to help you
prevent yourself from making an error with grouping
arguments. Block enclosures actually incur more
overhead. Not much, but some.
*Conway, Damian, Perl Best Practices, O'Reilly Media, Sebastopol, CA, 2005, pp 169-170.
@array = map { exp } @list;
@array = grep { exp } @list;
29. Copyright 2014 Daina Pettit
map, grep, sort – slide 29
Boolean Scalar Context
● Anywhere in perl where a true/false is expected
—if, while, and, or, not, &&, ||, !, etc.
● Evaluation results in 0, “0”, 0.0, “”, or undef then
it is false. Everything else is true.
if ( 0 ) {} # False
if ( 400 ) {} # True
if ( 1 ) {} # True
if ( "false" ) {} # True!
if ( "00" ) {} # True!
undef $x;
if ( $x ) {} # False
30. Copyright 2014 Daina Pettit
map, grep, sort – slide 30
Examples of grep
● Expression can be any valid perl expression.
● Expression is in scalar boolean context.
@ones = grep { $_ < 10 } @numbers;
@dirs = grep { d } @files;
@no_dup = grep { ! $h{$_}++ } @old;
@errors = grep { /error/i } @log;
@true = grep { $_ } @all;
31. Copyright 2014 Daina Pettit
map, grep, sort – slide 31
Sorting Basics
Sort can be called in three ways:
1. With no comparison directives
2. With a subroutine that returns comparison
directives
3. With a code block (an anonymous subroutine) that
returns comparison directives
@sorted = sort @unsorted;
@sorted = sort sub @unsorted;
@sorted = sort { exp } @unsorted;
32. Copyright 2014 Daina Pettit
map, grep, sort – slide 32
Sorting Basics
Sort requires the comparison directives value of -1, 0, or
1 to tell whether any two elements, $a and $b, are in
order (-1), the same (0), or out of order (1).
cmp and <=> conveniently provide this for string or
numeric comparisons, respectively.
We don't have to use cmp and <=>. We just have to
return -1, 0, or 1.
$a <=> $b
40. Copyright 2014 Daina Pettit
map, grep, sort – slide 40
Complicated Sorting
We can sorting with multiple keys such as sort
by year, then by month, then by day even if the
data is mm-dd-yyyy.
@sorted_dates = sort {
( $ma, $da, $ya ) = split //, $a;
( $mb, $db, $yb ) = split //, $b;
$ya<=>$yb || $ma<=>$mb || $da<=>$db;
} @dates;
48. Copyright 2014 Daina Pettit
map, grep, sort – slide 48
Optimizing sort
Now use map to extract just element 0 and we
are back to the original list and sorted by date.
This is known as the Schwartzian Transform.*
*Perl idiom named for Randal Schwartz, author of Learning Perl, coined by Tom Christiansen.
@order =
map { $_>[0] }
sort { $a>[1] <=> $b>[1] }
map { [ $_, M ] } @files;
“x.pl” “file1” “5.dat” “file7” “a.out”
53. Copyright 2014 Daina Pettit
map, grep, sort – slide 53
Optimizing sort—Orcish Maneuver*
Uses “or” cache (in a hash) to remember values
already computed: ||=
● Simpler than ST
● Almost as fast as ST
● Faster if list contains duplicates
*Term coined by Joseph Hall in Effective Perl Programming, Addison-Wesley Professional, Boston, MA, 1998.
@order = sort {
( $cache{$a} ||= M $a ) <=>
( $cache{$b} ||= M $b ) }
@files;
58. Copyright 2014 Daina Pettit
map, grep, sort – slide 58
Optimizing sort—Guttman-Rosler Transform*
This is a tweak on ST. Takes advantage of
substr and sprintf being faster than array
manipulation. Also uses default string sort which
is slightly faster.
*A Fresh Look at Efficient Perl Sorting, Uri Guttman and Larry Rosler, approx. 1999.
@order = map { substr $_, 10 }
sort
map { m#(d{4})/(d+)/(d+)#;
sprintf "%d%02d%02d%s",
$1, $2, $3, $_
} @dates;
60. Copyright 2014 Daina Pettit
map, grep, sort – slide 60
Further List & Sort Options
List::Util
shuffle, reduce, any, first, max, min, ...
List::MoreUtils
uniq, natatime, ...
Sort::Key
May be faster than ST or GRT
Sort::Naturally
Automatically sorts numeric when appropriate
Sort::Maker
Internally uses OM, ST, or GRT.