SlideShare une entreprise Scribd logo
1  sur  168
Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
What I work on ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object]
Overview ,[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
In the beginning ,[object Object]
In the beginning ,[object Object],[object Object]
In the beginning ,[object Object],[object Object],[object Object]
A world without failure
A world without failure ,[object Object]
A world without failure ,[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Consistency ,[object Object]
User Expectations ,[object Object]
User Expectations ,[object Object],[object Object]
User Expectations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases
Databases ,[object Object]
Databases ,[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What databases are good at ,[object Object]
What databases are good at ,[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object],[object Object],[object Object]
Easy solution to data consistency ,[object Object],[object Object],[object Object]
Revelation #1 ,[object Object]
Revelation #2 ,[object Object]
Revelation #3 ,[object Object]
Revelation #3 ,[object Object],[object Object]
Eat my data ,[object Object]
Where data can be ,[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Simple Application: Save==on disk ,[object Object],[object Object],[object Object]
Saving a simple document ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #1 ,[object Object],[object Object]
Word Processor Saving -1 Bug ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #2, 3 and 4 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object]
File System Integrity ,[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data journaling ,[object Object]
Atomic write(2)
Atomic write(2) ,[object Object]
Atomic write(2) ,[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Eat My Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CRASH
Write to Temp file, rename ,[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Now all is good with the world...
Now all is good with the world... ,[object Object]
Now all is good with the world... ,[object Object],[object Object]
[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
data=ordered ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
other systems ,[object Object],[object Object],[object Object],[object Object],[object Object]
flush and sync ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Flush the buffers! Sync to disk  before  rename
A tale of libxml2 ,[object Object]
A tale of libxml2 ,[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object],[object Object]
so, replace ,[object Object]
gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE  *fp; char  *xmlbuf; int  fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
Nearing Nirvana ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
on fsync, POSIX Says... ,[object Object]
on fsync, POSIX Says... ,[object Object]
POSIX compliant fsync ,[object Object]
POSIX compliant fsync ,[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],gcc
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],pushl  %ebp movl  %esp, %ebp movl  $0, %eax popl  %ebp ret gcc
Tale of a really fast database server ,[object Object]
Tale of a really fast database server ,[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object],[object Object]
fsync() doesn't have to sync ,[object Object]
fsync() doesn't have to sync ,[object Object],[object Object]
Standards are great ,[object Object]
Standards are great ,[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object],[object Object]
#ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier.  Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
#if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ #  define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 #  error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /*  fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
Yes, some OS Vendors hate you ,[object Object]
Big Files
Big Files ,[object Object]
Big Files ,[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object]
Large directories ,[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object]
Where data is after being written ,[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object]
sqlite ,[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
inode Infos Direct blocks Indirect Blocks Double indirect blocks
Extent ,[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object]
Parallel writers ,[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preallocation ,[object Object],[object Object],[object Object],[object Object],[object Object]
Tablespace allocation in NDB ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object]
Library Developers ,[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
There is hope ,[object Object],[object Object],[object Object]
Good Luck!
Good Luck! ,[object Object]

Contenu connexe

Tendances

Hardware Acceleration in WebKit
Hardware Acceleration in WebKitHardware Acceleration in WebKit
Hardware Acceleration in WebKitJoone Hur
 
Algebraic Thinking for Evolution of Pure Functional Domain Models
Algebraic Thinking for Evolution of Pure Functional Domain ModelsAlgebraic Thinking for Evolution of Pure Functional Domain Models
Algebraic Thinking for Evolution of Pure Functional Domain ModelsDebasish Ghosh
 
Android Training (Storing data using SQLite)
Android Training (Storing data using SQLite)Android Training (Storing data using SQLite)
Android Training (Storing data using SQLite)Khaled Anaqwa
 
Multiple choice questions for Java io,files and inheritance
Multiple choice questions for Java io,files and inheritanceMultiple choice questions for Java io,files and inheritance
Multiple choice questions for Java io,files and inheritanceAbishek Purushothaman
 
2 1 Стандартні асинхронні послідовні інтерфейси
2 1 Стандартні асинхронні послідовні інтерфейси2 1 Стандартні асинхронні послідовні інтерфейси
2 1 Стандартні асинхронні послідовні інтерфейсиПупена Александр
 
Android SDK Tutorial | Edureka
Android SDK Tutorial | EdurekaAndroid SDK Tutorial | Edureka
Android SDK Tutorial | EdurekaEdureka!
 
Introduction to Go language
Introduction to Go languageIntroduction to Go language
Introduction to Go languageTzar Umang
 
SOLID, DRY, SLAP design principles
SOLID, DRY, SLAP design principlesSOLID, DRY, SLAP design principles
SOLID, DRY, SLAP design principlesSergey Karpushin
 
JPA and Hibernate Performance Tips
JPA and Hibernate Performance TipsJPA and Hibernate Performance Tips
JPA and Hibernate Performance TipsVlad Mihalcea
 
From Spring Framework 5.3 to 6.0
From Spring Framework 5.3 to 6.0From Spring Framework 5.3 to 6.0
From Spring Framework 5.3 to 6.0VMware Tanzu
 
Reactive programming
Reactive programmingReactive programming
Reactive programmingSUDIP GHOSH
 
Concurrency in Golang
Concurrency in GolangConcurrency in Golang
Concurrency in GolangOliver N
 

Tendances (20)

Memory in go
Memory in goMemory in go
Memory in go
 
Hardware Acceleration in WebKit
Hardware Acceleration in WebKitHardware Acceleration in WebKit
Hardware Acceleration in WebKit
 
Algebraic Thinking for Evolution of Pure Functional Domain Models
Algebraic Thinking for Evolution of Pure Functional Domain ModelsAlgebraic Thinking for Evolution of Pure Functional Domain Models
Algebraic Thinking for Evolution of Pure Functional Domain Models
 
The Rust Programming Language
The Rust Programming LanguageThe Rust Programming Language
The Rust Programming Language
 
Sobrecarga operadores
Sobrecarga operadoresSobrecarga operadores
Sobrecarga operadores
 
Go lang
Go langGo lang
Go lang
 
Android Training (Storing data using SQLite)
Android Training (Storing data using SQLite)Android Training (Storing data using SQLite)
Android Training (Storing data using SQLite)
 
Multiple choice questions for Java io,files and inheritance
Multiple choice questions for Java io,files and inheritanceMultiple choice questions for Java io,files and inheritance
Multiple choice questions for Java io,files and inheritance
 
2 1 Стандартні асинхронні послідовні інтерфейси
2 1 Стандартні асинхронні послідовні інтерфейси2 1 Стандартні асинхронні послідовні інтерфейси
2 1 Стандартні асинхронні послідовні інтерфейси
 
Go Lang Tutorial
Go Lang TutorialGo Lang Tutorial
Go Lang Tutorial
 
Evolution Of Java
Evolution Of JavaEvolution Of Java
Evolution Of Java
 
Android SDK Tutorial | Edureka
Android SDK Tutorial | EdurekaAndroid SDK Tutorial | Edureka
Android SDK Tutorial | Edureka
 
Introduction to Go language
Introduction to Go languageIntroduction to Go language
Introduction to Go language
 
SOLID, DRY, SLAP design principles
SOLID, DRY, SLAP design principlesSOLID, DRY, SLAP design principles
SOLID, DRY, SLAP design principles
 
JPA and Hibernate Performance Tips
JPA and Hibernate Performance TipsJPA and Hibernate Performance Tips
JPA and Hibernate Performance Tips
 
From Spring Framework 5.3 to 6.0
From Spring Framework 5.3 to 6.0From Spring Framework 5.3 to 6.0
From Spring Framework 5.3 to 6.0
 
Actor Model Akka Framework
Actor Model Akka FrameworkActor Model Akka Framework
Actor Model Akka Framework
 
Reactive programming
Reactive programmingReactive programming
Reactive programming
 
Kotlin
KotlinKotlin
Kotlin
 
Concurrency in Golang
Concurrency in GolangConcurrency in Golang
Concurrency in Golang
 

Similaire à Eat my data

Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Michael Fong
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .netMichael Pavlovsky
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5sabtolinux
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy StepsTim Serong
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL databaseDave Stokes
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008guestd9065
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01MongoDB
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016IXIASOFT
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))Papu Kumar
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnosticsguest8212a5
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsguest8212a5
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-operSabbir Alam
 

Similaire à Eat my data (20)

Edubooktraining
EdubooktrainingEdubooktraining
Edubooktraining
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .net
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy Steps
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008
 
Measuring Firebird Disk I/O
Measuring Firebird Disk I/OMeasuring Firebird Disk I/O
Measuring Firebird Disk I/O
 
Ch23 system administration
Ch23 system administration Ch23 system administration
Ch23 system administration
 
file_c.pdf
file_c.pdffile_c.pdf
file_c.pdf
 
Filehandlinging cp2
Filehandlinging cp2Filehandlinging cp2
Filehandlinging cp2
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01
 
File Handling In C++
File Handling In C++File Handling In C++
File Handling In C++
 
Ext 0523
Ext 0523Ext 0523
Ext 0523
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnostics
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-oper
 
Ungooglable
UngooglableUngooglable
Ungooglable
 

Dernier

"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 

Dernier (20)

"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 

Eat my data

  • 1. Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11. A world without failure
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 64.
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
  • 75.
  • 76. Now all is good with the world...
  • 77.
  • 78.
  • 79.
  • 80.
  • 81.
  • 82.
  • 83.
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94. gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE *fp; char *xmlbuf; int fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 101.
  • 102.
  • 103.
  • 104.
  • 105.
  • 106.
  • 107.
  • 108.
  • 109.
  • 110.
  • 111.
  • 112.
  • 113.
  • 114.
  • 115.
  • 116.
  • 117. #ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier. Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
  • 118. #if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ # define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 # error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /* fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
  • 119.
  • 121.
  • 122.
  • 123.
  • 124.
  • 125.
  • 126.
  • 127.
  • 128.
  • 129.
  • 130.
  • 131.
  • 132.
  • 133.
  • 134.
  • 135.
  • 136.
  • 137.
  • 138.
  • 139.
  • 140.
  • 141.
  • 142.
  • 143.
  • 144.
  • 145.
  • 146.
  • 147.
  • 148.
  • 149. inode Infos Direct blocks Indirect Blocks Double indirect blocks
  • 150.
  • 151.
  • 152.
  • 153.
  • 154.
  • 155.
  • 156.
  • 157.
  • 158.
  • 159.
  • 160.
  • 161.
  • 162.
  • 163.
  • 164.
  • 165.
  • 166.
  • 168.