Slides of the I Workshop on command-line tools with the collaboration of CAG (Center for Applied Genomics - Children's Hospital of Philadelphia) bioinformatics analysts.
1st day
Workshop on command line tools - day 2Leandro Lima
Slides of the I Workshop on command-line tools with the collaboration of CAG (Center for Applied Genomics - Children's Hospital of Philadelphia) bioinformatics analysts.
2nd day
This document provides a cheat sheet for using the UNIX C Shell, covering topics like file manipulation, terminal setup, changing directories, listing and displaying files, copying, moving, removing files, printing files, finding files, comparing files, setting permissions, input/output redirection, C shell specific commands, job control, and special startup files.
This document discusses using sliding windows to aggregate streaming data in MapReduce. It proposes buffering input tuples in mappers until a window is full, then emitting the aggregate. Combiners and reducers combine partial aggregates across windows. Window ranges are initialized and updated during merging to remove outdated data and handle late arrivals. This approach allows streaming aggregation queries to be executed with MapReduce.
This document contains a lecture on working with arrays, scripts, and SSH/SCP in UNIX systems. It discusses special variables used in scripts, how to define and manipulate arrays, examples of useful scripts for renaming files, backing up data, and extracting video files from DVDs, and how to use SSH to securely connect to remote systems and SCP to securely transfer files between systems. It also covers generating and using public/private key pairs for passwordless SSH login.
The document provides an overview of shell scripting in Linux. It discusses that a shell script is a plain text file containing shell commands that can automate tasks. Some key points covered include:
- Shell scripts allow automation of repetitive tasks to save time.
- They provide features like variables, control flow, I/O facilities and more.
- Common shells are Bash, Bourne, and Korn shell, with Bash being the default Linux shell.
- Shell scripts can be used for tasks like customizing the environment, automating daily jobs, and more repetitive system tasks.
The document describes a trash command that provides a recycle bin functionality for Linux similar to Windows. It works by moving deleted files to the $HOME/.trash directory instead of permanently removing them. Users can restore deleted files by running the rm -l command and specifying the file row number. The trash command also checks the trash directory size and automatically deletes the oldest files if the space limit is exceeded.
The document describes how to create a simple character device driver for Linux. It involves writing C code for file operations like open, read, write and close. The code is compiled into a kernel module which is loaded and tested. Key steps include creating files for the driver code, adding a Makefile, building the kernel object, loading the module, creating a device file, and verifying the file operations by reading kernel logs. The module can then be unloaded after testing is completed.
The document provides an introduction to basic UNIX commands written by Razor on January 15, 2000 for new UNIX users. It includes commands for working with files and permissions, such as cp and mv to copy and move files, cd to change directories, pwd to show the current directory, mkdir to create directories, and rm to delete files and directories. The first part focuses on commands for copying, moving, changing directories, viewing the current directory, creating directories, and deleting files and directories.
Workshop on command line tools - day 2Leandro Lima
Slides of the I Workshop on command-line tools with the collaboration of CAG (Center for Applied Genomics - Children's Hospital of Philadelphia) bioinformatics analysts.
2nd day
This document provides a cheat sheet for using the UNIX C Shell, covering topics like file manipulation, terminal setup, changing directories, listing and displaying files, copying, moving, removing files, printing files, finding files, comparing files, setting permissions, input/output redirection, C shell specific commands, job control, and special startup files.
This document discusses using sliding windows to aggregate streaming data in MapReduce. It proposes buffering input tuples in mappers until a window is full, then emitting the aggregate. Combiners and reducers combine partial aggregates across windows. Window ranges are initialized and updated during merging to remove outdated data and handle late arrivals. This approach allows streaming aggregation queries to be executed with MapReduce.
This document contains a lecture on working with arrays, scripts, and SSH/SCP in UNIX systems. It discusses special variables used in scripts, how to define and manipulate arrays, examples of useful scripts for renaming files, backing up data, and extracting video files from DVDs, and how to use SSH to securely connect to remote systems and SCP to securely transfer files between systems. It also covers generating and using public/private key pairs for passwordless SSH login.
The document provides an overview of shell scripting in Linux. It discusses that a shell script is a plain text file containing shell commands that can automate tasks. Some key points covered include:
- Shell scripts allow automation of repetitive tasks to save time.
- They provide features like variables, control flow, I/O facilities and more.
- Common shells are Bash, Bourne, and Korn shell, with Bash being the default Linux shell.
- Shell scripts can be used for tasks like customizing the environment, automating daily jobs, and more repetitive system tasks.
The document describes a trash command that provides a recycle bin functionality for Linux similar to Windows. It works by moving deleted files to the $HOME/.trash directory instead of permanently removing them. Users can restore deleted files by running the rm -l command and specifying the file row number. The trash command also checks the trash directory size and automatically deletes the oldest files if the space limit is exceeded.
The document describes how to create a simple character device driver for Linux. It involves writing C code for file operations like open, read, write and close. The code is compiled into a kernel module which is loaded and tested. Key steps include creating files for the driver code, adding a Makefile, building the kernel object, loading the module, creating a device file, and verifying the file operations by reading kernel logs. The module can then be unloaded after testing is completed.
The document provides an introduction to basic UNIX commands written by Razor on January 15, 2000 for new UNIX users. It includes commands for working with files and permissions, such as cp and mv to copy and move files, cd to change directories, pwd to show the current directory, mkdir to create directories, and rm to delete files and directories. The first part focuses on commands for copying, moving, changing directories, viewing the current directory, creating directories, and deleting files and directories.
The document provides an overview of Unix and shell scripting. It discusses the history and architecture of Unix operating systems. It then covers various Unix commands and utilities for file management, processes, communication, and system administration. Finally, it describes the basics of shell scripting including variables, conditional statements, loops, and here documents.
Shell Script to Extract IP Address, MAC Address InformationVCP Muthukrishna
This script collects the active MAC addresses, IP addresses, and associated hardware vendor information on a system. It uses the arp command to gather this network information and outputs it to an HTML file. The HTML file displays the IP address, MAC address, and includes a hyperlink to lookup the IEEE vendor information based on the first three octets of the MAC address. It also includes an option to email the results in an HTML formatted email.
This document discusses using tracing, awk, and xgraph to analyze network performance parameters from ns2 trace files. It provides details on the wired trace format, examples of awk scripts to calculate link throughput and end-to-end throughput between nodes, as well as a script to calculate average link delay between transmitting and receiving nodes.
1. The document describes an ns-2 tutorial exercise on simulating computer networks using the ns-2 simulator. It provides example scripts for basic network simulations.
2. The example scripts simulate simple network topologies with increasing complexity, including UDP and TCP traffic over droptail and queue configurations.
3. Later examples introduce more complex scenarios like dynamic routing protocols and simulating link failures to observe network behavior.
This document summarizes the key capabilities of Warp 10, a time series data ingestion, processing, and visualization platform:
1. Warp 10 can ingest high volumes of time series data from sensors and other IoT devices via HTTP, WebSockets, and many collection tools in a performant manner.
2. It provides a feature-rich scripting language called WarpScript that allows users to manipulate, analyze, and transform ingested time series data using over 690 functions and frameworks.
3. Warp 10 includes tools to visualize time series data in real-time through widgets that can display charts, images, and more generated from WarpScript. Dynamic tile widgets also enable building configurable
Linux Shell Scripts and Shell Commands✌️Nazmul Hyder
I just shortly describe some Linux shell script and shell commands.Hopefully, it will help you to file edit, make, delete directory operations, grep, pipeline and lots of stuff to your Linux/Mac terminal.
This document provides an overview of Bash scripting concepts including file systems, variables and strings, math operations, file ownership and permissions, users and privileges, processes and subshells, loops, conditional statements, I/O redirection, named pipes, signals, and GUI tools. It also includes examples of Bluetooth file sharing, auto-shutdown scripts, lockscreen notifications, web crawling scripts, and time tracking automation. References are provided for further reading.
This document provides an introduction and overview of shell scripting in Linux. It discusses what a shell script is, when they should and should not be used, examples of common shell scripts, and an introduction to programming features commonly used in shell scripts such as variables, conditionals, loops, command line arguments, and more. Key points covered include that shell scripts allow automating command execution, are useful for repetitive tasks, and come with programming features to customize behavior.
Abstract:
This talk will introduce you to the concept of Kubernetes Volume plugins. We will not only help you understand the basic concepts, but more importantly, using practical examples, we will show how you can develop your own volume plugins and contribute them back to the community of the OSS project as large as Kubernetes.
We will conclude the talk by discussing various challenges one can come across when contributing to a high velocity OSS project of Kubernetes' size which can help you avoid the pain and enjoy the path.
Sched Link: http://sched.co/6BYB
1) The document provides instructions for setting up an AWS account and launching an EC2 instance with an AMI that contains tools and documentation for a hands-on tutorial on NoSQL databases and MongoDB.
2) The tutorial covers basic MongoDB commands and demonstrates how to create, insert, update, and query document data using the mongo shell client. Embedded and nested documents are explored along with geospatial queries.
3) A map-reduce example aggregates historical check-in data to calculate popular locations over different time periods, demonstrating how MongoDB supports batch operations.
Coming Out Of Your Shell - A Comparison of *Nix ShellsKel Cecil
This document provides an overview of several popular shell options including bash, zsh, and fish. It discusses their origins, key features, and popular frameworks used to enhance them. The document encourages exploring options like oh-my-zsh and oh-my-fish to benefit from community configurations while also highlighting capabilities in each shell beyond their initial reputation. The takeaways emphasize that with tweaking, bash is capable of more than assumed, zsh rewards investment in unlocking its power, and fish offers useful features out of the box.
This document discusses shell programming in UNIX/Linux. It covers shell scripts, variables, operators, and logic structures used in shell programming. Shell scripts are text files that contain UNIX commands and do not need compilation. Common shells are Bourne, Bash, Korn and C shells. Variables, operators, and logic structures like if/else, for/while loops, and switch statements allow for programming logic and control flow in shell scripts.
Bash Script Disk Space Utilization Report and EMailVCP Muthukrishna
This bash script generates an HTML disk usage report and emails it. It collects disk usage information using df, formats it into an HTML table, and emails the report. If disk usage exceeds 90%, the row is highlighted red and a critical alert is shown. Usage between 70-80% is highlighted orange. The report is generated daily and emailed to a recipient.
Artimon is a scalable metrics collection and analysis framework. It collects metrics called 'variable instances' that have a name, labels, and timestamped values. Metrics can be exported via a Thrift service and stored in distributed systems like Kafka for later analysis using Groovy scripts. Artimon is designed to collect both IT and business metrics and can adapt to collect from third party sources using agents.
File Space Usage Information and EMail Report - Shell ScriptVCP Muthukrishna
This script generates an HTML report of the top 10 largest files and directories on a server by size and emails the report. It uses the du command to get disk usage information, sorts the results in descending order of size, and outputs the top 10 to a HTML table. The table is written to a file and emailed using sendmail to notify of disk space usage.
"PostgreSQL and Python" Lightning Talk @EuroPython2014Henning Jacobs
PL/Python allows users to write PostgreSQL functions and procedures using Python. It enables accessing PostgreSQL data and running Python code from within SQL queries. For example, a function could query a database table, process the results in Python by accessing modules, and return a value to the SQL query. This opens up possibilities to leverage Python's extensive libraries and expressiveness to expose data and perform complex validation from PostgreSQL.
This document provides instructions for basic Linux commands and administration tasks. It begins by listing commands for checking directory contents and properties, navigating the file system, creating and modifying files and directories. It then covers user administration like adding, modifying and deleting users. Next it discusses group administration and managing permissions on files and directories. Finally it covers partitioning, creating a new partition on /dev/sda, and activating the changes.
Maxym Kharchenko presented ways to manage Oracle databases with Python. He demonstrated a Python tool to ping multiple Oracle databases concurrently and time the execution. The tool reports the status and timing for each database pinged. Python enforces good coding practices and interfaces well with databases, APIs, and other systems. Learning Python helps develop a more Pythonic way of thinking that can improve code quality and productivity.
The document discusses using functional programming techniques in Perl to efficiently calculate tree hashes of large files uploaded in chunks to cloud storage services. It presents a tree_fold keyword and implementation that allows recursively reducing a list of values using a block in a tail-call optimized manner to avoid stack overflows. This approach is shown to provide concise, efficient and elegant functional code for calculating tree hashes in both Perl 5 and Perl 6.
This document provides an overview and introduction to using Linux for a workshop session. It covers working with files and directories, basic commands, process commands, using the vi editor, and a brief introduction to the nano editor. The document also discusses compiling files, debugging in C/C++, and remote access tools for Linux.
1. The document provides examples of common Linux commands and their usage including tar, grep, find, ssh, sed, awk, vim, diff, sort, export, xargs, ls, ifconfig, uname, ps, free, top, df, kill, rm, cp, mv, cat, mount, chmod, chown, passwd, mkdir, ifconfig, uname, whereis, whatis, and locate.
2. Examples shown include how to create, extract, and view tar archives, search files with grep, find files, login remotely with ssh, edit files with vim, compare files with diff, view processes with ps, check storage usage with df, terminate processes with kill, manage files
The document provides an overview of Unix and shell scripting. It discusses the history and architecture of Unix operating systems. It then covers various Unix commands and utilities for file management, processes, communication, and system administration. Finally, it describes the basics of shell scripting including variables, conditional statements, loops, and here documents.
Shell Script to Extract IP Address, MAC Address InformationVCP Muthukrishna
This script collects the active MAC addresses, IP addresses, and associated hardware vendor information on a system. It uses the arp command to gather this network information and outputs it to an HTML file. The HTML file displays the IP address, MAC address, and includes a hyperlink to lookup the IEEE vendor information based on the first three octets of the MAC address. It also includes an option to email the results in an HTML formatted email.
This document discusses using tracing, awk, and xgraph to analyze network performance parameters from ns2 trace files. It provides details on the wired trace format, examples of awk scripts to calculate link throughput and end-to-end throughput between nodes, as well as a script to calculate average link delay between transmitting and receiving nodes.
1. The document describes an ns-2 tutorial exercise on simulating computer networks using the ns-2 simulator. It provides example scripts for basic network simulations.
2. The example scripts simulate simple network topologies with increasing complexity, including UDP and TCP traffic over droptail and queue configurations.
3. Later examples introduce more complex scenarios like dynamic routing protocols and simulating link failures to observe network behavior.
This document summarizes the key capabilities of Warp 10, a time series data ingestion, processing, and visualization platform:
1. Warp 10 can ingest high volumes of time series data from sensors and other IoT devices via HTTP, WebSockets, and many collection tools in a performant manner.
2. It provides a feature-rich scripting language called WarpScript that allows users to manipulate, analyze, and transform ingested time series data using over 690 functions and frameworks.
3. Warp 10 includes tools to visualize time series data in real-time through widgets that can display charts, images, and more generated from WarpScript. Dynamic tile widgets also enable building configurable
Linux Shell Scripts and Shell Commands✌️Nazmul Hyder
I just shortly describe some Linux shell script and shell commands.Hopefully, it will help you to file edit, make, delete directory operations, grep, pipeline and lots of stuff to your Linux/Mac terminal.
This document provides an overview of Bash scripting concepts including file systems, variables and strings, math operations, file ownership and permissions, users and privileges, processes and subshells, loops, conditional statements, I/O redirection, named pipes, signals, and GUI tools. It also includes examples of Bluetooth file sharing, auto-shutdown scripts, lockscreen notifications, web crawling scripts, and time tracking automation. References are provided for further reading.
This document provides an introduction and overview of shell scripting in Linux. It discusses what a shell script is, when they should and should not be used, examples of common shell scripts, and an introduction to programming features commonly used in shell scripts such as variables, conditionals, loops, command line arguments, and more. Key points covered include that shell scripts allow automating command execution, are useful for repetitive tasks, and come with programming features to customize behavior.
Abstract:
This talk will introduce you to the concept of Kubernetes Volume plugins. We will not only help you understand the basic concepts, but more importantly, using practical examples, we will show how you can develop your own volume plugins and contribute them back to the community of the OSS project as large as Kubernetes.
We will conclude the talk by discussing various challenges one can come across when contributing to a high velocity OSS project of Kubernetes' size which can help you avoid the pain and enjoy the path.
Sched Link: http://sched.co/6BYB
1) The document provides instructions for setting up an AWS account and launching an EC2 instance with an AMI that contains tools and documentation for a hands-on tutorial on NoSQL databases and MongoDB.
2) The tutorial covers basic MongoDB commands and demonstrates how to create, insert, update, and query document data using the mongo shell client. Embedded and nested documents are explored along with geospatial queries.
3) A map-reduce example aggregates historical check-in data to calculate popular locations over different time periods, demonstrating how MongoDB supports batch operations.
Coming Out Of Your Shell - A Comparison of *Nix ShellsKel Cecil
This document provides an overview of several popular shell options including bash, zsh, and fish. It discusses their origins, key features, and popular frameworks used to enhance them. The document encourages exploring options like oh-my-zsh and oh-my-fish to benefit from community configurations while also highlighting capabilities in each shell beyond their initial reputation. The takeaways emphasize that with tweaking, bash is capable of more than assumed, zsh rewards investment in unlocking its power, and fish offers useful features out of the box.
This document discusses shell programming in UNIX/Linux. It covers shell scripts, variables, operators, and logic structures used in shell programming. Shell scripts are text files that contain UNIX commands and do not need compilation. Common shells are Bourne, Bash, Korn and C shells. Variables, operators, and logic structures like if/else, for/while loops, and switch statements allow for programming logic and control flow in shell scripts.
Bash Script Disk Space Utilization Report and EMailVCP Muthukrishna
This bash script generates an HTML disk usage report and emails it. It collects disk usage information using df, formats it into an HTML table, and emails the report. If disk usage exceeds 90%, the row is highlighted red and a critical alert is shown. Usage between 70-80% is highlighted orange. The report is generated daily and emailed to a recipient.
Artimon is a scalable metrics collection and analysis framework. It collects metrics called 'variable instances' that have a name, labels, and timestamped values. Metrics can be exported via a Thrift service and stored in distributed systems like Kafka for later analysis using Groovy scripts. Artimon is designed to collect both IT and business metrics and can adapt to collect from third party sources using agents.
File Space Usage Information and EMail Report - Shell ScriptVCP Muthukrishna
This script generates an HTML report of the top 10 largest files and directories on a server by size and emails the report. It uses the du command to get disk usage information, sorts the results in descending order of size, and outputs the top 10 to a HTML table. The table is written to a file and emailed using sendmail to notify of disk space usage.
"PostgreSQL and Python" Lightning Talk @EuroPython2014Henning Jacobs
PL/Python allows users to write PostgreSQL functions and procedures using Python. It enables accessing PostgreSQL data and running Python code from within SQL queries. For example, a function could query a database table, process the results in Python by accessing modules, and return a value to the SQL query. This opens up possibilities to leverage Python's extensive libraries and expressiveness to expose data and perform complex validation from PostgreSQL.
This document provides instructions for basic Linux commands and administration tasks. It begins by listing commands for checking directory contents and properties, navigating the file system, creating and modifying files and directories. It then covers user administration like adding, modifying and deleting users. Next it discusses group administration and managing permissions on files and directories. Finally it covers partitioning, creating a new partition on /dev/sda, and activating the changes.
Maxym Kharchenko presented ways to manage Oracle databases with Python. He demonstrated a Python tool to ping multiple Oracle databases concurrently and time the execution. The tool reports the status and timing for each database pinged. Python enforces good coding practices and interfaces well with databases, APIs, and other systems. Learning Python helps develop a more Pythonic way of thinking that can improve code quality and productivity.
The document discusses using functional programming techniques in Perl to efficiently calculate tree hashes of large files uploaded in chunks to cloud storage services. It presents a tree_fold keyword and implementation that allows recursively reducing a list of values using a block in a tail-call optimized manner to avoid stack overflows. This approach is shown to provide concise, efficient and elegant functional code for calculating tree hashes in both Perl 5 and Perl 6.
This document provides an overview and introduction to using Linux for a workshop session. It covers working with files and directories, basic commands, process commands, using the vi editor, and a brief introduction to the nano editor. The document also discusses compiling files, debugging in C/C++, and remote access tools for Linux.
1. The document provides examples of common Linux commands and their usage including tar, grep, find, ssh, sed, awk, vim, diff, sort, export, xargs, ls, ifconfig, uname, ps, free, top, df, kill, rm, cp, mv, cat, mount, chmod, chown, passwd, mkdir, ifconfig, uname, whereis, whatis, and locate.
2. Examples shown include how to create, extract, and view tar archives, search files with grep, find files, login remotely with ssh, edit files with vim, compare files with diff, view processes with ps, check storage usage with df, terminate processes with kill, manage files
Linux is an open source operating system where everything can be done via commands. It was developed by Linus Torvalds and the first official version 0.02 was released in 1991. The shell is a program that takes commands from the keyboard and gives them to the operating system to perform tasks. Common shells include BASH, KSH, and TCSH. Vi is a basic text editor used on Linux systems to create, edit, and view text files. It has two modes - command mode to enter commands and insert mode to add/edit text. Basic file management, text processing and other useful commands are also described.
Powerful and efficient tool leveraging UNIX commands for streamlined automation, enabling seamless execution of complex tasks and maximizing productivity.
This document provides an overview of basic Linux commands, including man for accessing manual pages, ls for listing directory contents, mkdir for creating directories, cd for changing directories, pwd for printing the working directory, and ~ for accessing the home directory. It also covers commands for copying, moving, removing files, clearing the screen, viewing file contents, searching within files, counting words, piping commands together, using wildcards, and changing file permissions with chmod. The document encourages learning through manual pages and understanding error messages.
This document provides information about various Linux commands. It begins by defining what a command is and explaining the different types of commands - built-in shell commands and external commands. It then discusses command navigation shortcuts and various file manipulation commands like mkdir, rmdir, touch, cp, rm, man, head, tail, cat, tac, more and files. The document also covers the Linux filesystem hierarchy standard and describes the main directories for binaries, configuration, data and memory. Overall, the document serves as a guide to common Linux commands and filesystem structure.
This document provides 50 examples of common Linux/Unix commands organized by command name. It begins with tar, grep, find, ssh, sed, awk, vim, diff, sort, and export examples. The document is intended as a quick reference for users to learn practical uses of fundamental Linux commands.
This document provides 50 examples of common Linux/Unix commands along with brief explanations and usage examples for each command. Some of the commands highlighted include tar, grep, find, ssh, sed, awk, vim, diff, sort, and ls. The examples cover a wide range of tasks from compressing/extracting files to searching/editing text to managing processes and permissions.
This document provides 50 examples of common Linux/Unix commands along with brief explanations and usage examples for each command. Some of the commands highlighted include tar, grep, find, ssh, sed, awk, vim, diff, sort, export, xargs, ls, pwd, cd, gzip, bzip2, unzip, shutdown, ftp, crontab, service, ps, top, df, kill, rm, cp, mv, cat, mount, chmod, chown, passwd, mkdir, ifconfig, and uname. The document is intended to give readers a quick start on frequently used commands.
This document provides instructions for 27 common Linux commands: mkdir, rmdir, ls, cd, cat, touch, wc, who, pwd, rm, mv, chmod, cp, grep, cal, date, vi, tput, ps, export, type, tail, sudo, head, man, clear, and adduser. For each command, it lists the syntax and provides 1-3 examples of common uses. The document is presented over 28 pages with the commands organized topic-by-topic and includes formatting like headings and indentation to aid readability.
The document provides 40 tips for using basic Linux command line commands and tricks. Some key points include: everything in Linux is a file; # and $ denote superuser and normal users respectively; Ctrl+Alt+F1-F6 switch between terminals while Ctrl+Alt+F7 switches to the GUI; tilde ~ denotes the user's home directory; hidden files start with dot; ls -a views hidden files; file permissions use rwx scheme; and variables can store long text for repeated use.
The document provides 40 tips for using basic Linux command line commands and tricks. Some key points include: everything in Linux is a file; # and $ denote superuser and normal users respectively; Ctrl+Alt+F1-F6 switch between terminals while Ctrl+Alt+F7 switches to the GUI; tilde ~ denotes the user's home directory; hidden files start with a dot; ls -a views hidden files; file permissions use rwx notation; and variables can be assigned text for repeated use.
This document provides a summary of Linux command line cheat sheet covering topics such as:
- Simple commands like date, cal, df, free for displaying system information
- Navigation commands like cd, ls
- Exploring and manipulating files and directories using commands like cp, mv, mkdir, rm
- Finding files using locate, find
- Process commands like ps, top, kill
- Networking commands like ping, traceroute, netstat
- Archiving files using gzip, bzip2, tar, zip
- User permissions and ownership with chmod, chown, su
This document provides a cheat sheet for common Linux Bash shell commands. It includes sections on basic commands, file manipulation, process management, user accounts, file permissions, flow redirection, archiving/compressing data, and installing software. The cheat sheet lists over 50 commands and provides examples of how to use each one.
This document provides an overview of Linux file management basics. It discusses permissions for files and directories, essential file management tasks like creating/moving/copying files, using text editors like Nano and Vim, performing operations like viewing file timestamps and disk usage. It also covers Linux wildcards for pattern matching, and tools for finding files like locate, find, whereis etc. I/O redirection and pipes/filters are also mentioned.
This document provides an overview of basic shell commands in Linux/Unix systems. It lists 42 common commands like cat, cd, cp, grep, gzip, kill, ls, man, mv, ps, pwd and tar along with brief descriptions of their usage. The commands allow users to navigate directories, view, edit and manage files, compress and archive files, view system processes and documentation, and more.
The document provides summaries of various Linux commands:
- The man command displays the manual for a command, including its name, synopsis, description, options, exit status, errors, versions, examples, and more.
- The pwd command prints the path of the current working directory.
- The cal command displays the calendar for the current or specified month and year.
The document provides descriptions of Linux commands including their purpose, syntax, and usage. It covers common commands like bc, cal, cat, cd, cp, date, echo, grep, id, last, lastlog, ls, man, mkdir, mv, pwd, rm, rmdir, sort, shutdown, who, whois, more, less, clear, cmp, wc, whoami, head, and tail. For each command, it lists the command name, brief description, and syntax.
The document provides descriptions of Linux commands including their purpose, syntax, and usage. It covers common commands like bc, cal, cat, cd, cp, date, echo, grep, id, last, lastlog, ls, man, mkdir, mv, pwd, rm, rmdir, sort, shutdown, who, whois, more, less, clear, cmp, wc, whoami, head, and tail. For each command, it lists the command name, brief description, and syntax.
Similaire à Workshop on command line tools - day 1 (20)
Estudos genéticos em doenças Mendelianas e complexasLeandro Lima
O documento discute estudos genéticos de doenças complexas, incluindo: (1) estudos de associação em todo o genoma que buscam variantes genéticas comuns associadas a doenças; (2) desafios como explicar apenas uma pequena parte da variação genética e incluir variantes raras; (3) o uso de interações proteína-proteína para entender melhor a arquitetura genética de doenças.
Uso do Cytoscape para Visualização e Análise de RedesLeandro Lima
O documento discute o software Cytoscape, que é usado para visualização e análise de redes biológicas. Ele permite visualizar redes de interação molecular e vias metabólicas, integrando essas redes com dados de expressão gênica e outras informações. O Cytoscape tem ferramentas integradas e plugins para diferentes análises, e permite acessar bancos de dados públicos, exportar imagens e salvar sessões.
Brokers e Bridges (genes em rede de interação proteína-proteína)Leandro Lima
O documento discute bridges e brokers em uma rede. Bridges conectam partes importantes da rede apesar de terem poucas conexões, e são importantes para evitar que partes da rede fiquem desconectadas. Brokers têm muitas ligações e funcionam como corretores, ligando pessoas que não se conhecem, e sua vizinhança fica desconectada se forem removidos da rede.
Int. à Bioinformática (FMU - 08/05/2012)Leandro Lima
- O documento introduz o campo da bioinformática, discutindo o DNA, genoma, sequenciamento, montagem e anotação de genomas. Também aborda alinhamento de sequências usando programação dinâmica e aplicações como estudos de expressão gênica e redes biológicas.
Redes Complexas aplicadas a Redes Sociais (09/05/2012 - FMU)Leandro Lima
O documento discute redes complexas aplicadas a redes sociais. Ele apresenta o palestrante e seu background acadêmico, define o que são redes e como elas podem ser representadas, e discute como redes sociais, biológicas e de influência podem ser analisadas usando métodos de redes complexas.
Malibou Pitch Deck For Its €3M Seed Roundsjcobrien
French start-up Malibou raised a €3 million Seed Round to develop its payroll and human resources
management platform for VSEs and SMEs. The financing round was led by investors Breega, Y Combinator, and FCVC.
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
Using Query Store in Azure PostgreSQL to Understand Query PerformanceGrant Fritchey
Microsoft has added an excellent new extension in PostgreSQL on their Azure Platform. This session, presented at Posette 2024, covers what Query Store is and the types of information you can get out of it.
Top 9 Trends in Cybersecurity for 2024.pptxdevvsandy
Security and risk management (SRM) leaders face disruptions on technological, organizational, and human fronts. Preparation and pragmatic execution are key for dealing with these disruptions and providing the right cybersecurity program.
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
Microservice Teams - How the cloud changes the way we workSven Peters
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemPeter Muessig
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesQuickdice ERP
Explore the seamless transition to e-invoicing with this comprehensive guide tailored for Saudi Arabian businesses. Navigate the process effortlessly with step-by-step instructions designed to streamline implementation and enhance efficiency.
Mobile app Development Services | Drona InfotechDrona Infotech
Drona Infotech is one of the Best Mobile App Development Company In Noida Maintenance and ongoing support. mobile app development Services can help you maintain and support your app after it has been launched. This includes fixing bugs, adding new features, and keeping your app up-to-date with the latest
Visit Us For :
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfVALiNTRY360
Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems.
For more info visit us https://valintry360.com/solutions/health-life-sciences
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
Oracle 23c New Features For DBAs and Developers.pptx
Workshop on command line tools - day 1
1. I Workshop on command-
line tools
(day 1)
Center for Applied Genomics
Children's Hospital of Philadelphia
February 12-13, 2015
2. Arguments
Come after the name of the program
Example:
cat file.txt (1 argument)
cut -f2 file.txt (2 arguments)
The number of spaces between arguments doesn't matter
cut -f2 file.txt
3. man - command manual
man <command>
man cat
man echo
man awk
4. which - which command is being called
which <command>
which cat
which echo
which awk
5. some tips (i)
Use <Tab> to auto-complete your commands or
file/directory names
To search old commands, you can use ↑ and ↓
arrows in your keyboard
6. some tips (ii)
The command history will return a list of your
last commands
Use ! to run the last command starting with…
Example:
!grep
This will run the last command starting with grep
7. Special characters (i)
^ : beginning of line
$ : end of line or beginning of variable name
? : any character (with one occurrence)
* : any character (with 0 or more occurrences)
# : start comments
[ ] : define sets of characters
8. Special characters (ii)
" " : define strings
' ' : define strings
- : start a parameter
` ` : define commands
; : separate commands
| : "pipe" commands
9. Special characters (iii)
~ : home directory
/ : separate internal directories
: escape character
n : new line (Linux)
r : new line (Mac)
t : tab
10. First steps
pwd # where am I?
whoami # who am I?
id <your_username> # what can I do?
date # what time/day is it?
11. cat - concatenate and print text files
cat file1.txt file2.txt > output.txt
cat *.bed > all.bed
cat -n : shows line numbers
cat -e : shows non-printing characters
12. echo - write to the standard output
echo Hello, CAG!
echo -e : prints escape characters
echo -e "CtAtG"
echo -e "CnAnG"
echo -n : prints and doesn't go to a new line
echo -n "CAG"; echo "123"
echo "CAG"; echo "123"
15. ls - list files in directories (i)
ls : list files of current directory
ls workshop : list files in directory workshop
ls -l : in long format
ls -t : list files sorted by time modified
ls -1 : force output to be one entry per line
ls -S : list files sorted by time modified
16. ls - list files in directories (ii)
ls -r : reverse the sorting
ls -a : list hidden files (which begin with a dot)
ls -h : show file size human-readable
ls -G : colors output
We can combine options:
ls -lhrt
17. ssh - secure shell (access remote servers) (i)
ssh <user>@<server>
ssh -t : exits after a list of commands
ssh limal@respublica.research.chop.edu
ssh limal@respublica.research.chop.edu -t top
ssh limal@respublica.research.chop.edu -t ls -lh
ssh limal@respublica.research.chop.edu -t ls -lh >
my_home_on_respub.txt
18. ssh - secure shell (access remote servers) (ii)
ssh -p <port> : access a specific port on server
ssh -X : open session with graphic/display options
(if you need to open a graphic program in a remote
server; e.g. IGV).
19. alias - "shortcut" for commands
alias <alias> : see what is a specific alias
alias ll # ll is not a real command. =)
alias resp='ssh limal@respublica.research.chop.edu'
resp
20. df - report file system disk space usage
df -h : human-readable
21. du - estimate file space usage
du -h : human-readable
22. mkdir - make directory
mkdir bioinfo_files
mkdir workshop_text_files
mkdir workshop123
mkdir -p 2015/February/12
# Suggestion:
# Create names that make sense
23. cd - change working directory
cd bioinfo_files
cd .. # go to directory above
cd ~ # go to home directory
cd - # go to previous directory
24. rmdir - remove empty directories
rmdir workshop123
rmdir 2015 # it will return an error
26. cp - copy files and directories
cp old_file.txt workshop_files
cp error.txt error_copy.txt
# To copy directories with its contents,
# use -r (recursive)
cp -r workshop_files bioinfo_files/
# Now, try...
cp -r workshop_files/ bioinfo_files/
27. scp - secure copy files and
directories in different servers
# Similar to "cp" (in this case, we're uploading)
scp *.txt limal@respublica.research.chop.edu:~/
# To copy directories with its contents,
# use -r (recursive)
scp -r w* limal@respublica.research.chop.edu:~/
# Downloading
scp limal@respublica.research.chop.edu:~/*.txt .
28. rm - remove files and directories
rm old_file.txt error_copy.txt
# Use -r (recursive) to remove
# directories and its contents
rm -r bioinfo_files/workshop_files/
rm -r 2015
29. ln - make links (pointers) of files
(it's good to avoid multiple copies)
# hard links keep the same if the original
# files are removed
ln workshop_files/old_file.txt hard.txt
# symbolic links break if the original
# files are removed
ln -s workshop_files/old_file.txt symbolic.txt
30. testing links
echo "hard" >> hard.txt
echo "symbolic" >> symbolic.txt
head hard.txt symbolic.txt
head workshop_files/old_file.txt
rm workshop_files/old_file.txt
head hard.txt symbolic.txt
32. tar - archiving
Create an archive:
tar -cvf newfile.tar file1 file2 dir1 dir2
tar -cvf BLs.tar bla.txt ble.txt blo.txt
tar -cvzf BLs.tar.gz bla.txt ble.txt blo.txt
Parameters: c (create), v (verbose), z (gzip), f (file)
33. tar - archiving
Extract from an archive:
tar -xvzf GWAS.tar.gz
tar -xvjf XHMM_results.tar.bz2
Parameters: x (extract), v (verbose), f (file),
z (gzip), j (bzip2)
34. gzip - zip files
ls -lh adhd.ped
gzip adhd.ped
ls -lh adhd.ped.gz
# to unzip, run "gunzip adhd.ped.gz"
35. zcat - cat for zipped files
zcat adhd.ped.gz # Ctrl+C to stop
36. less - file visualization
less DATA.xcnv
Use arrows (←↑→↓) to navigate the file
Type / to search
38. head - first lines
# first 20 lines
head -n 20 DATA.xcnv
# all lines, excluding last 2
# (on Linux, not Mac)
head -n -2 DATA.xcnv
39. tail - last lines
# last 20 lines
tail -n 20 DATA.xcnv
# from line 2 to the end
tail -n +2 DATA.xcnv
40. cut - get specific columns of file
# fields 1 to 3 and 6
cut -f 1-3,6 DATA.xcnv
# other examples
cut -f1 adhd.ped
cut -f1 -d' ' adhd.ped # delimiter = space
# other delimiters: comma, tab, etc.
cut -d, -f1-2 …
cut -d't' -f5,7,9 …
41. Using "|" (pipe) to join commands
cut -f 1-3,6 DATA.xcnv | head -n 1
cut -f 1-3,6 DATA.xcnv | less
zcat adhd.ped.gz | less
# Compare (same result? same time?)
zcat adhd.ped.gz | cut -f1 -d' ' | head
zcat adhd.ped.gz | head | cut -f1 -d' '
42. column - columnate lists
# using white spaces to separate
# and fill columns
column -t DATA.xcnv
column -s # choose separator
43. sort - sort lines of text files
sort DATA.xcnv
sort -k : choose specific field
sort -n : numeric-sort
sort -r : reverse
# Exercise: show 10 top CNVs with
# more targets (column 8)
44. uniq - report or filter out repeated lines in a file
cut -f1 DATA.xcnv | sort | uniq
# reporting counts of each line
cut -f5 DATA.xcnv | sort | uniq -c
45. wc - word, line, character and byte count
wc -l : number of lines
wc -w : number of words
wc -m : number of characters
cut -f5 DATA.xcnv | sort | uniq | wc -l
head -n1 DATA.xcnv | cut -f1 | wc -m
46. More exercises
1. What are the top 10 samples with more CNVs?
2. What are the top 5 largest CNVs?
3. What are the top 15 directories using more space?
47. vi/vim (text editor) (i)
vi text_file.txt (open "text_file.txt")
i - start edition mode (remember "insert")
ESC - stop edition mode
:w - save file ("write")
:q - quit
:x - save (write) and quit
48. vi/vim (text editor) (ii)
u - undo
:30 - go to line number 30
:syntax on - syntax highlighting
^ - go to beginning of line
$ - go to end of line
49. vi/vim (text editor) (iii)
dd - delete current line
d2↓ - delete current line and 2 lines below
yy - copy current line
y3↓ - copy current line and 3 lines below
pp - paste lines below current line
50. grep - finds words/patterns in a file (i)
grep word file.txt
Options:
grep -w : find the whole word
grep -c : returns the number of lines found
grep -f : specifies a file with a list of words
grep -o : returns only the match
51. grep - finds words/patterns in a file (ii)
grep -A 2 : also show 2 lines after
grep -B 3 : also show 3 lines before
grep -v : shows lines without pattern
grep --color : colors the match
52. Exercises
1. How many CNVs are located on chrom. 1?
2. How many deletions are there?
3. Which samples finish with character M?
4. Which samples finish with character M or F?
5. How many samples do not have NN in the
name?