SlideShare une entreprise Scribd logo
1  sur  23
Télécharger pour lire hors ligne
Terminology and Concept                  Roaming Home Directories   Plans for the future




                               Roaming Home Directories

                           Andreas Schneider <mail@cynapses.org>

                                               csync


                                        23rd April 2009




Andreas Schneider <mail@cynapses.org>                                             csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future




Roaming Home Directories


       1 Terminology and Concept
              Terminology
              Concept

       2 Roaming Home Directories
              PAM
              pam csync

       3 Plans for the future
              Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Terminology


csync - roaming home directories


       1 Terminology and Concept
              Terminology
              Concept

       2 Roaming Home Directories
              PAM
              pam csync

       3 Plans for the future
              Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Terminology


Roaming Home Directories




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Terminology


Roaming Home Directories




              We don’t talk about Profiles here
              This is about synchronizing files!
              No merging of content (not yet?)




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Terminology


What is a replica?




              A tree or subtree of a filesystems
              This could be local or remote
              Changes are often made independently and as a result they do
              not contain the same information.




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories     Plans for the future

Terminology


What is a conflict?



      A path is conflicting if it fulfills the following conditions:
          1   it has been updated on one replica
          2   it or any of its decendants has been updated on the other
              replica too, and
          3   its contents are not identical.




Andreas Schneider <mail@cynapses.org>                                              csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Concept


csync - roaming home directories


       1 Terminology and Concept
              Terminology
              Concept

       2 Roaming Home Directories
              PAM
              pam csync

       3 Plans for the future
              Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Concept


What do we need?




      We need a bidirectional file synchronizer
              working with metadata (stat())
              working with a metadata status database
              client only (no specific daemon needed)




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Concept


What do we have?




      csync simple file synchronizer which:
              collects and checks metadata (update detection)
              compares metadata (reconciliation)
              copy the files (propagation)




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Concept


How do we detect updates?



      We compare the modification time of the file with the db entry.
              no entry = new file
              newer modification time = file changed
              equal modification time = no changes
              ...




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories        Plans for the future

Concept


What do we do with conflicts?



      The Merge Algorithm
              doesn’t need any user interaction
              it is used by Microsoft for Roaming Profiles
              if it detects a conflict, it will use the most recent file and
              overwrite the file on the other replica




Andreas Schneider <mail@cynapses.org>                                                 csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

Concept


Which protocols are supported?




              SMB: using Sambas libsmbclient
              SFTP: using libssh (will be in 0.43)




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories      Plans for the future

Concept


For developers



      csync uses:
              a red-black tree to store all information in memory
              a sqlite3 database to store all the metadata for the next
              synchronization
              a virtual IO plugin system to support different protocols
              a unit testing framework




Andreas Schneider <mail@cynapses.org>                                               csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

PAM


csync - roaming home directories


       1 Terminology and Concept
              Terminology
              Concept

       2 Roaming Home Directories
              PAM
              pam csync

       3 Plans for the future
              Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories    Plans for the future

PAM


Pluggable Authentication Module



      Separates the tasks of authentication into four independent
      management groups:
              authentication management
              password management
              account management
              session management




Andreas Schneider <mail@cynapses.org>                                             csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

pam csync


csync - roaming home directories


       1 Terminology and Concept
              Terminology
              Concept

       2 Roaming Home Directories
              PAM
              pam csync

       3 Plans for the future
              Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

pam csync


pam csync does ...




              authentication management to get the password and pass it to
              session management to synchronize the files




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                 Roaming Home Directories   Plans for the future

pam csync


pam csync features




              pam winbind support to get the Profile directory from the
              ADS
              kerberos suppport
              exclude users




Andreas Schneider <mail@cynapses.org>                                            csync
Roaming Home Directories
Terminology and Concept                  Roaming Home Directories   Plans for the future

Features users or developers asked for


csync - roaming home directories


       1 Terminology and Concept
                Terminology
                Concept

       2 Roaming Home Directories
                PAM
                pam csync

       3 Plans for the future
                Features users or developers asked for



Andreas Schneider <mail@cynapses.org>                                             csync
Roaming Home Directories
Terminology and Concept                  Roaming Home Directories    Plans for the future

Features users or developers asked for


GUI support




       What’s needed?
               A conflict algorithm
               Callback that for the GUI to display and modify conflicts




Andreas Schneider <mail@cynapses.org>                                              csync
Roaming Home Directories
Terminology and Concept                  Roaming Home Directories   Plans for the future

Features users or developers asked for


More protocols




               WEBDAV: maybe possible with libneon
               RSYNC: maybe we can talk to rsync
               own daemon with mimetype support to merge special contents




Andreas Schneider <mail@cynapses.org>                                             csync
Roaming Home Directories
Terminology and Concept                         Roaming Home Directories   Plans for the future

Features users or developers asked for




                                           Questions & Answers


                                         Interested in development?

               Poject page: http://www.csync.org/




Andreas Schneider <mail@cynapses.org>                                                    csync
Roaming Home Directories

Contenu connexe

Similaire à csync talk at SambaXP 2009

Realizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamRealizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamDataWorks Summit
 
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...Amazon Web Services
 
An introduction to Apache Kafka and Kafka ecosystem at LinkedIn
An introduction to Apache Kafka and Kafka ecosystem at LinkedInAn introduction to Apache Kafka and Kafka ecosystem at LinkedIn
An introduction to Apache Kafka and Kafka ecosystem at LinkedInDong Lin
 
Simplify Governance of Streaming Data
Simplify Governance of Streaming Data Simplify Governance of Streaming Data
Simplify Governance of Streaming Data confluent
 
CD presentation march 12th, 2018
CD presentation march 12th, 2018CD presentation march 12th, 2018
CD presentation march 12th, 2018Ran Levy
 
Realizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache BeamRealizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache BeamDataWorks Summit
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBDDan Frincu
 
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...Amazon Web Services
 
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)Amazon Web Services Korea
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...KRamasamy2
 
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...Amazon Web Services
 
Good practices to design and implement IT architecture based on AWS
Good practices to design and implement IT architecture based on AWSGood practices to design and implement IT architecture based on AWS
Good practices to design and implement IT architecture based on AWSLCloud
 
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...Jacek Biernat
 
Efficient Data Formats for Analytics with Parquet and Arrow
Efficient Data Formats for Analytics with Parquet and ArrowEfficient Data Formats for Analytics with Parquet and Arrow
Efficient Data Formats for Analytics with Parquet and ArrowDataWorks Summit/Hadoop Summit
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceDataWorks Summit/Hadoop Summit
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...VAISHNAVI MADHAN
 
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko GlobalLogic Ukraine
 
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud EcosystemAmazon Web Services
 

Similaire à csync talk at SambaXP 2009 (20)

Realizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamRealizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache Beam
 
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...
AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DA...
 
An introduction to Apache Kafka and Kafka ecosystem at LinkedIn
An introduction to Apache Kafka and Kafka ecosystem at LinkedInAn introduction to Apache Kafka and Kafka ecosystem at LinkedIn
An introduction to Apache Kafka and Kafka ecosystem at LinkedIn
 
Simplify Governance of Streaming Data
Simplify Governance of Streaming Data Simplify Governance of Streaming Data
Simplify Governance of Streaming Data
 
CD presentation march 12th, 2018
CD presentation march 12th, 2018CD presentation march 12th, 2018
CD presentation march 12th, 2018
 
Realizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache BeamRealizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache Beam
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBD
 
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...
AWS re:Invent 2016: Design Patterns for High Availability: Lessons from Amazo...
 
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
 
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...
(HLS402) Getting into Your Genes: The Definitive Guide to Using Amazon EMR, A...
 
Good practices to design and implement IT architecture based on AWS
Good practices to design and implement IT architecture based on AWSGood practices to design and implement IT architecture based on AWS
Good practices to design and implement IT architecture based on AWS
 
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...
Dobre praktyki projektowania architektury i wdrażania systemów IT dla chmury ...
 
Efficient Data Formats for Analytics with Parquet and Arrow
Efficient Data Formats for Analytics with Parquet and ArrowEfficient Data Formats for Analytics with Parquet and Arrow
Efficient Data Formats for Analytics with Parquet and Arrow
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open Source
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
 
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
 
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
 
Ad fundamentals
Ad fundamentalsAd fundamentals
Ad fundamentals
 
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
 

csync talk at SambaXP 2009

  • 1. Terminology and Concept Roaming Home Directories Plans for the future Roaming Home Directories Andreas Schneider <mail@cynapses.org> csync 23rd April 2009 Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 2. Terminology and Concept Roaming Home Directories Plans for the future Roaming Home Directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 3. Terminology and Concept Roaming Home Directories Plans for the future Terminology csync - roaming home directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 4. Terminology and Concept Roaming Home Directories Plans for the future Terminology Roaming Home Directories Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 5. Terminology and Concept Roaming Home Directories Plans for the future Terminology Roaming Home Directories We don’t talk about Profiles here This is about synchronizing files! No merging of content (not yet?) Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 6. Terminology and Concept Roaming Home Directories Plans for the future Terminology What is a replica? A tree or subtree of a filesystems This could be local or remote Changes are often made independently and as a result they do not contain the same information. Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 7. Terminology and Concept Roaming Home Directories Plans for the future Terminology What is a conflict? A path is conflicting if it fulfills the following conditions: 1 it has been updated on one replica 2 it or any of its decendants has been updated on the other replica too, and 3 its contents are not identical. Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 8. Terminology and Concept Roaming Home Directories Plans for the future Concept csync - roaming home directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 9. Terminology and Concept Roaming Home Directories Plans for the future Concept What do we need? We need a bidirectional file synchronizer working with metadata (stat()) working with a metadata status database client only (no specific daemon needed) Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 10. Terminology and Concept Roaming Home Directories Plans for the future Concept What do we have? csync simple file synchronizer which: collects and checks metadata (update detection) compares metadata (reconciliation) copy the files (propagation) Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 11. Terminology and Concept Roaming Home Directories Plans for the future Concept How do we detect updates? We compare the modification time of the file with the db entry. no entry = new file newer modification time = file changed equal modification time = no changes ... Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 12. Terminology and Concept Roaming Home Directories Plans for the future Concept What do we do with conflicts? The Merge Algorithm doesn’t need any user interaction it is used by Microsoft for Roaming Profiles if it detects a conflict, it will use the most recent file and overwrite the file on the other replica Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 13. Terminology and Concept Roaming Home Directories Plans for the future Concept Which protocols are supported? SMB: using Sambas libsmbclient SFTP: using libssh (will be in 0.43) Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 14. Terminology and Concept Roaming Home Directories Plans for the future Concept For developers csync uses: a red-black tree to store all information in memory a sqlite3 database to store all the metadata for the next synchronization a virtual IO plugin system to support different protocols a unit testing framework Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 15. Terminology and Concept Roaming Home Directories Plans for the future PAM csync - roaming home directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 16. Terminology and Concept Roaming Home Directories Plans for the future PAM Pluggable Authentication Module Separates the tasks of authentication into four independent management groups: authentication management password management account management session management Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 17. Terminology and Concept Roaming Home Directories Plans for the future pam csync csync - roaming home directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 18. Terminology and Concept Roaming Home Directories Plans for the future pam csync pam csync does ... authentication management to get the password and pass it to session management to synchronize the files Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 19. Terminology and Concept Roaming Home Directories Plans for the future pam csync pam csync features pam winbind support to get the Profile directory from the ADS kerberos suppport exclude users Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 20. Terminology and Concept Roaming Home Directories Plans for the future Features users or developers asked for csync - roaming home directories 1 Terminology and Concept Terminology Concept 2 Roaming Home Directories PAM pam csync 3 Plans for the future Features users or developers asked for Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 21. Terminology and Concept Roaming Home Directories Plans for the future Features users or developers asked for GUI support What’s needed? A conflict algorithm Callback that for the GUI to display and modify conflicts Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 22. Terminology and Concept Roaming Home Directories Plans for the future Features users or developers asked for More protocols WEBDAV: maybe possible with libneon RSYNC: maybe we can talk to rsync own daemon with mimetype support to merge special contents Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories
  • 23. Terminology and Concept Roaming Home Directories Plans for the future Features users or developers asked for Questions & Answers Interested in development? Poject page: http://www.csync.org/ Andreas Schneider <mail@cynapses.org> csync Roaming Home Directories