SlideShare une entreprise Scribd logo
1  sur  18
Télécharger pour lire hors ligne
Writable FDW

KaiGai Kohei <kaigai@kaigai.gr.jp>
          Tw: @kkaigai



       JPUG PostgreSQL Unconference - 20130216
Overview of FDW

FDW: Foreign Data Wrapper
External data source as if regular tables
  file_fdw     ... for flat files
  postgres_fdw ... for remote PostgreSQL
Extension performs as if a part of executor
Read-only, right now,




             JPUG PostgreSQL Unconference - 20130216   2
How does it work? (1/2)
                  Query
                   Tree                                                FDW Extension
pg_plan_queries



                     set_base_rel_sizes                            GetForeignRelSize

                    set_base_rel_pathlists                             GetForeignPaths

                      create_scan_plan                                 GetForeignPlan




                    ExecInitForeignScan                            BeginForeignScan
ExecuteQuery




                          ForeignNext                              IterateForeignScan

                    ExecEndForeignScan                                 EndForeignScan

                             JPUG PostgreSQL Unconference - 20130216                     3
How does it work? (2/2)


  ResultSet


                                     Join


                                                                   Performs as if
                 Join
                                                                    data format
                                                                     converter
TupleTableSlot          TupleTableSlot         TupleTableSlot


                          Foreign                  Foreign
  Regular                  Table                    Table          Remote           Flat
   Table                                                            Table           Files

                        FDW driver              FDW driver


                         JPUG PostgreSQL Unconference - 20130216                       4
Learn from existing logic to modify tables

                                                 ExecScan            ExecQual

              ExecModifyTable
                                               TupleTableSlot
 TupleTableSlot                                                    with “ctid”
                 ExecInsert
                ExecUpdate                                          system
JunkFilter      ExecDelete                       TableScan          column


ctid                HeapTuple




  Ctid addresses
  the target row
  to be modified.
                         JPUG PostgreSQL Unconference - 20130216                 5
Issues towards writable foreign table

Way to identify a remote row to be modified
  “rowid” to identify a particular remote row
  “pseudo-column” to move values out of foreign
  table definition
New methods relevant to ExecModityTable
  Methods relevant to ...
     ExecInitModifyTable
     ExecInsert, ExecUpdate, ExecDelete
     ExecEndModifyTable
Transaction control with remote data source
  Responsibility of FDW drivers

               JPUG PostgreSQL Unconference - 20130216   6
rowid pseudo-column (1/3)

 Calculation of                ExecScan
  newer tuple                                              HeapTuple      + rowid

                             TupleTableSlot
ExecModifyTable
                                                                 Remote Table
                                Foreign
   ExecInsert
                                 Table
  ExecUpdate
  ExecDelete


                              FDW driver

HeapTuple’        + rowid
                                  Update a row
                                                                Primary Key column
                               identified with rowid
                      JPUG PostgreSQL Unconference - 20130216                        7
rowid pseudo-column (2/3)

 Calculation of                                                                resjunk=true
  newer tuple

                          ExecScan                                 A   B   C    D    rowid
                                                   tts_values                       
ExecModifyTable                                     tts_isnull             
                        TupleTableSlot
   ExecInsert
  ExecUpdate
  ExecDelete                                               CREATE FOREIGN TABLE
                           Foreign
                                                           (
                            Table
                                                              A   int,
      TupleTableSlot
                                                              B   int,
                                                              C   text,
HeapTuple’        + rowid FDW driver                          D   date
                                                           ) SERVER loopback;



                         JPUG PostgreSQL Unconference - 20130216                         8
rowid pseudo-column (3/3)
AttrNumber
GetForeignRelWidth(PlannerInfo *root,
                   RelOptInfo *baserel,
                   Relation foreignrel,
                   bool inhparent,
                   List *targetList);


A new GetForeignRelWidth() method
   It returns width of TupleTableSlot, even if it overs
   number of columns in table definition.
   if targetList contains “rowid” column, FDW driver
   has to expand the width of TupleTableSlot.
   Extra fields are acquired to store values of pseudo-
   columns

               JPUG PostgreSQL Unconference - 20130216    9
Why multiple pseudo-column is allowed?

  SELECT X, Y, (X-Y)^2 from ftable;
           
  SELECT X, Y, Pcol_1 from ftable;

  Just reference to               SELECT X, Y, (X-Y)^2 AS Pcol_1
the calculated result               from remote_data_source;
 in the remote side
                                                              Remote Query
  Calculation off-loading
       GetForeignRelWidth() can replace expression node of
       the given TargetEntry with pseudo-column reference,
       instead of complex calculation.
       Utilization of external computing resource
           GPU, remote host, ...

                        JPUG PostgreSQL Unconference - 20130216              10
New APIs
List *PlanForeignModify(PlannerInfo *root,
                        ModifyTable *plan,
                        Index resultRelation,
                        Plan *subplan);
void BeginForeignModify(ModifyTableState *mtstate,
                        ResultRelInfo *resultRelInfo,
                        List *fdw_private,
                        Plan *subplan,
                        int eflags);
TupleTableSlot *ExecForeignInsert(ResultRelInfo *rinfo,
                                 TupleTableSlot *slot);
TupleTableSlot *ExecForeignUpdate(ResultRelInfo *rinfo,
                                  const char *rowid,
                                  TupleTableSlot *slot);
bool ExecForeignDelete(ResultRelInfo *rinfo,
                       const char *rowid);
void EndForeignModify(ResultRelInfo *rinfo);

                JPUG PostgreSQL Unconference - 20130216    11
Working Example (1/2)
postgres=# CREATE FOREIGN TABLE ft1 (a int, b text, c float)
                SERVER loopback OPTIONS (relname 't1');
CREATE FOREIGN TABLE
postgres=# SELECT * FROM ft1;
 a | b | c
---+-----+-----
 1 | aaa | 1.1
 2 | bbb | 2.2
 3 | ccc | 3.3
 4 | ddd | 4.4
 5 | eee | 5.5
(5 rows)
postgres=# EXPLAIN (verbose) SELECT * FROM ft1 WHERE b like '%b%';
                             QUERY PLAN
-------------------------------------------------------------------
 Foreign Scan on public.ft1 (cost=100.00..100.01 rows=1 width=44)
   Output: a, b, c
   Filter: (ft1.b ~~ '%b%'::text)
   Remote SQL: SELECT a, b, c FROM public.t1
(4 rows)


                     JPUG PostgreSQL Unconference - 20130216     12
Working Example (2/2)
postgres=# UPDATE ft1 SET b = b || '_updt‘
                      WHERE a % 2 = 0 RETURNING *;
 a |     b    | c
---+----------+-----
 2 | bbb_updt | 2.2
 4 | ddd_updt | 4.4
(2 rows)

UPDATE 2
postgres=# EXPLAIN (verbose, costs off)
    UPDATE ft1 SET b = b || '_updt' WHERE a % 2 = 0 RETURNING *;
                      QUERY PLAN
----------------------------------------------------------------
Update on public.ft1
   Output: a, b, c
   -> Foreign Scan on public.ft1
         Output: a, (b || '_updt'::text), c, ctid, ft1.*
         Remote SQL: SELECT a, b, c, ctid FROM public.t1
                   WHERE (((a OPERATOR(pg_catalog.%) 2)
                              OPERATOR(pg_catalog.=) 0)) FOR UPDATE
(5 rows)
                     JPUG PostgreSQL Unconference - 20130216     13
Transaction Control
postgres=# BEGIN;
BEGIN
postgres=# UPDATE ft1 SET b = b || '_updt' WHERE a % 2 = 1;
UPDATE 3
postgres=# ABORT;
ROLLBACK
postgres=# SELECT * FROM t1;
 a |     b    | c
---+----------+-----
 2 | bbb      | 2.2
 4 | ddd      | 4.4
 1 | aaa_updt | 1.1
 3 | ccc_updt | 3.3
 5 | eee_updt | 5.5
(5 rows)


    Transaction control is responsibility of FDW drivers
    Register(Sub)XactCallback gives extensions chance to get
    control on transaction events, but I didn’t cover it yet.
                     JPUG PostgreSQL Unconference - 20130216    14
Does someone implement DLM?
postgres=# WITH ch AS (
    UPDATE t1 SET b = 'changed' RETURNING a
  ) UPDATE ft1 SET b = 'new val‘
    FROM ch WHERE ch.a = ft1.a;
^CCancel request sent
^CCancel request sent


   Right now, we have no distributed lock manager :-)
   Be careful not to acquire exclusive lock on same table
   In above example...
      This session acquires exclusive lock on table t1.
      table t1 is external data source of ft1 via loopback connection.
      UPDATE on ft1 also requires exclusive lock on t1 on remote session.




                       JPUG PostgreSQL Unconference - 20130216              15
Current Status




JPUG PostgreSQL Unconference - 20130216   16
FDW, be writable!




JPUG PostgreSQL Unconference - 20130216   17
その他のネタ




JPUG PostgreSQL Unconference - 20130216   18

Contenu connexe

Tendances

The Ring programming language version 1.2 book - Part 57 of 84
The Ring programming language version 1.2 book - Part 57 of 84The Ring programming language version 1.2 book - Part 57 of 84
The Ring programming language version 1.2 book - Part 57 of 84Mahmoud Samir Fayed
 
Drizzles Approach To Improving Performance Of The Server
Drizzles  Approach To  Improving  Performance Of The  ServerDrizzles  Approach To  Improving  Performance Of The  Server
Drizzles Approach To Improving Performance Of The ServerPerconaPerformance
 
Integrate Hive and R
Integrate Hive and RIntegrate Hive and R
Integrate Hive and RJunHo Cho
 
C - aptitude3
C - aptitude3C - aptitude3
C - aptitude3Srikanth
 
R hive tutorial - apply functions and map reduce
R hive tutorial - apply functions and map reduceR hive tutorial - apply functions and map reduce
R hive tutorial - apply functions and map reduceAiden Seonghak Hong
 
Tutorial de forms 10g
Tutorial de forms 10gTutorial de forms 10g
Tutorial de forms 10gmiguel
 
Morel, a Functional Query Language
Morel, a Functional Query LanguageMorel, a Functional Query Language
Morel, a Functional Query LanguageJulian Hyde
 
05 pig user defined functions (udfs)
05 pig user defined functions (udfs)05 pig user defined functions (udfs)
05 pig user defined functions (udfs)Subhas Kumar Ghosh
 
Hive Object Model
Hive Object ModelHive Object Model
Hive Object ModelZheng Shao
 
Class 17: Golden Sneezewort
Class 17: Golden SneezewortClass 17: Golden Sneezewort
Class 17: Golden SneezewortDavid Evans
 
Looping the Loop with SPL Iterators
Looping the Loop with SPL IteratorsLooping the Loop with SPL Iterators
Looping the Loop with SPL IteratorsMark Baker
 
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 WinterHuahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 WinterRyu Kobayashi
 
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...Uehara Junji
 

Tendances (15)

The Ring programming language version 1.2 book - Part 57 of 84
The Ring programming language version 1.2 book - Part 57 of 84The Ring programming language version 1.2 book - Part 57 of 84
The Ring programming language version 1.2 book - Part 57 of 84
 
Drizzles Approach To Improving Performance Of The Server
Drizzles  Approach To  Improving  Performance Of The  ServerDrizzles  Approach To  Improving  Performance Of The  Server
Drizzles Approach To Improving Performance Of The Server
 
Integrate Hive and R
Integrate Hive and RIntegrate Hive and R
Integrate Hive and R
 
C - aptitude3
C - aptitude3C - aptitude3
C - aptitude3
 
R hive tutorial - apply functions and map reduce
R hive tutorial - apply functions and map reduceR hive tutorial - apply functions and map reduce
R hive tutorial - apply functions and map reduce
 
Tapp 13-talk
Tapp 13-talkTapp 13-talk
Tapp 13-talk
 
Tutorial de forms 10g
Tutorial de forms 10gTutorial de forms 10g
Tutorial de forms 10g
 
Morel, a Functional Query Language
Morel, a Functional Query LanguageMorel, a Functional Query Language
Morel, a Functional Query Language
 
05 pig user defined functions (udfs)
05 pig user defined functions (udfs)05 pig user defined functions (udfs)
05 pig user defined functions (udfs)
 
Hive Object Model
Hive Object ModelHive Object Model
Hive Object Model
 
Class 17: Golden Sneezewort
Class 17: Golden SneezewortClass 17: Golden Sneezewort
Class 17: Golden Sneezewort
 
Looping the Loop with SPL Iterators
Looping the Loop with SPL IteratorsLooping the Loop with SPL Iterators
Looping the Loop with SPL Iterators
 
Stack queue
Stack queueStack queue
Stack queue
 
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 WinterHuahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
 
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...
Let's go Developer 2011 sendai Let's go Java Developer (Programming Language ...
 

En vedette

PG-Strom - GPU Accelerated Asyncr
PG-Strom - GPU Accelerated AsyncrPG-Strom - GPU Accelerated Asyncr
PG-Strom - GPU Accelerated AsyncrKohei KaiGai
 
20150318-SFPUG-Meetup-PGStrom
20150318-SFPUG-Meetup-PGStrom20150318-SFPUG-Meetup-PGStrom
20150318-SFPUG-Meetup-PGStromKohei KaiGai
 
Row-level Security
Row-level SecurityRow-level Security
Row-level SecurityKohei KaiGai
 
PG-Strom - A FDW module utilizing GPU device
PG-Strom - A FDW module utilizing GPU devicePG-Strom - A FDW module utilizing GPU device
PG-Strom - A FDW module utilizing GPU deviceKohei KaiGai
 
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)Kohei KaiGai
 
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)Kohei KaiGai
 
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~Kohei KaiGai
 
SQL+GPU+SSD=∞ (English)
SQL+GPU+SSD=∞ (English)SQL+GPU+SSD=∞ (English)
SQL+GPU+SSD=∞ (English)Kohei KaiGai
 
SQL+GPU+SSD=∞ (Japanese)
SQL+GPU+SSD=∞ (Japanese)SQL+GPU+SSD=∞ (Japanese)
SQL+GPU+SSD=∞ (Japanese)Kohei KaiGai
 
20170127 JAWS HPC-UG#8
20170127 JAWS HPC-UG#820170127 JAWS HPC-UG#8
20170127 JAWS HPC-UG#8Kohei KaiGai
 
An Intelligent Storage?
An Intelligent Storage?An Intelligent Storage?
An Intelligent Storage?Kohei KaiGai
 
pgconfasia2016 plcuda en
pgconfasia2016 plcuda enpgconfasia2016 plcuda en
pgconfasia2016 plcuda enKohei KaiGai
 
20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_Place20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_PlaceKohei KaiGai
 
pgconfasia2016 lt ssd2gpu
pgconfasia2016 lt ssd2gpupgconfasia2016 lt ssd2gpu
pgconfasia2016 lt ssd2gpuKohei KaiGai
 
GPGPU Accelerates PostgreSQL (English)
GPGPU Accelerates PostgreSQL (English)GPGPU Accelerates PostgreSQL (English)
GPGPU Accelerates PostgreSQL (English)Kohei KaiGai
 
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsPL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsKohei KaiGai
 
PG-Strom - GPGPU meets PostgreSQL, PGcon2015
PG-Strom - GPGPU meets PostgreSQL, PGcon2015PG-Strom - GPGPU meets PostgreSQL, PGcon2015
PG-Strom - GPGPU meets PostgreSQL, PGcon2015Kohei KaiGai
 
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsPL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsKohei KaiGai
 

En vedette (20)

PG-Strom - GPU Accelerated Asyncr
PG-Strom - GPU Accelerated AsyncrPG-Strom - GPU Accelerated Asyncr
PG-Strom - GPU Accelerated Asyncr
 
20150318-SFPUG-Meetup-PGStrom
20150318-SFPUG-Meetup-PGStrom20150318-SFPUG-Meetup-PGStrom
20150318-SFPUG-Meetup-PGStrom
 
Row-level Security
Row-level SecurityRow-level Security
Row-level Security
 
PostgreSQL with OpenCL
PostgreSQL with OpenCLPostgreSQL with OpenCL
PostgreSQL with OpenCL
 
PG-Strom - A FDW module utilizing GPU device
PG-Strom - A FDW module utilizing GPU devicePG-Strom - A FDW module utilizing GPU device
PG-Strom - A FDW module utilizing GPU device
 
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)
Custom Scan API - PostgreSQL Unconference #3 (18-Jan-2014)
 
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)
Technology Updates of PG-Strom at Aug-2014 (PGUnconf@Tokyo)
 
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~
GPGPU Accelerates PostgreSQL ~Unlock the power of multi-thousand cores~
 
PG-Strom
PG-StromPG-Strom
PG-Strom
 
SQL+GPU+SSD=∞ (English)
SQL+GPU+SSD=∞ (English)SQL+GPU+SSD=∞ (English)
SQL+GPU+SSD=∞ (English)
 
SQL+GPU+SSD=∞ (Japanese)
SQL+GPU+SSD=∞ (Japanese)SQL+GPU+SSD=∞ (Japanese)
SQL+GPU+SSD=∞ (Japanese)
 
20170127 JAWS HPC-UG#8
20170127 JAWS HPC-UG#820170127 JAWS HPC-UG#8
20170127 JAWS HPC-UG#8
 
An Intelligent Storage?
An Intelligent Storage?An Intelligent Storage?
An Intelligent Storage?
 
pgconfasia2016 plcuda en
pgconfasia2016 plcuda enpgconfasia2016 plcuda en
pgconfasia2016 plcuda en
 
20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_Place20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_Place
 
pgconfasia2016 lt ssd2gpu
pgconfasia2016 lt ssd2gpupgconfasia2016 lt ssd2gpu
pgconfasia2016 lt ssd2gpu
 
GPGPU Accelerates PostgreSQL (English)
GPGPU Accelerates PostgreSQL (English)GPGPU Accelerates PostgreSQL (English)
GPGPU Accelerates PostgreSQL (English)
 
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsPL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
 
PG-Strom - GPGPU meets PostgreSQL, PGcon2015
PG-Strom - GPGPU meets PostgreSQL, PGcon2015PG-Strom - GPGPU meets PostgreSQL, PGcon2015
PG-Strom - GPGPU meets PostgreSQL, PGcon2015
 
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database AnalyticsPL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
PL/CUDA - Fusion of HPC Grade Power with In-Database Analytics
 

Similaire à Writable Foreign Data Wrapper (JPUG Unconference 16-Feb-2013)

Apache PIG - User Defined Functions
Apache PIG - User Defined FunctionsApache PIG - User Defined Functions
Apache PIG - User Defined FunctionsChristoph Bauer
 
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)Serban Tanasa
 
Using R on Netezza
Using R on NetezzaUsing R on Netezza
Using R on NetezzaAjay Ohri
 
Lazy vs. Eager Loading Strategies in JPA 2.1
Lazy vs. Eager Loading Strategies in JPA 2.1Lazy vs. Eager Loading Strategies in JPA 2.1
Lazy vs. Eager Loading Strategies in JPA 2.1Patrycja Wegrzynowicz
 
ClickHouse new features and development roadmap, by Aleksei Milovidov
ClickHouse new features and development roadmap, by Aleksei MilovidovClickHouse new features and development roadmap, by Aleksei Milovidov
ClickHouse new features and development roadmap, by Aleksei MilovidovAltinity Ltd
 
Virtual Separation of Concerns (2011 Update)
Virtual Separation of Concerns (2011 Update)Virtual Separation of Concerns (2011 Update)
Virtual Separation of Concerns (2011 Update)chk49
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12Andrew Dunstan
 
Writing A Foreign Data Wrapper
Writing A Foreign Data WrapperWriting A Foreign Data Wrapper
Writing A Foreign Data Wrapperpsoo1978
 
エンタープライズ・クラウドと 並列・分散・非同期処理
エンタープライズ・クラウドと 並列・分散・非同期処理エンタープライズ・クラウドと 並列・分散・非同期処理
エンタープライズ・クラウドと 並列・分散・非同期処理maruyama097
 
EuroPython 2015 - Big Data with Python and Hadoop
EuroPython 2015 - Big Data with Python and HadoopEuroPython 2015 - Big Data with Python and Hadoop
EuroPython 2015 - Big Data with Python and HadoopMax Tepkeev
 
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...IndicThreads
 
Big data shim
Big data shimBig data shim
Big data shimtistrue
 
Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School Flink Forward
 
User-Defined Table Generating Functions
User-Defined Table Generating FunctionsUser-Defined Table Generating Functions
User-Defined Table Generating Functionspauly1
 
Apache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customizationApache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customizationBartosz Konieczny
 
MAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxMAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxHARIKRISHNANU13
 
Hadoop_Pennonsoft
Hadoop_PennonsoftHadoop_Pennonsoft
Hadoop_PennonsoftPennonSoft
 

Similaire à Writable Foreign Data Wrapper (JPUG Unconference 16-Feb-2013) (20)

Apache PIG - User Defined Functions
Apache PIG - User Defined FunctionsApache PIG - User Defined Functions
Apache PIG - User Defined Functions
 
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)
Get up to Speed (Quick Guide to data.table in R and Pentaho PDI)
 
Using R on Netezza
Using R on NetezzaUsing R on Netezza
Using R on Netezza
 
Lazy vs. Eager Loading Strategies in JPA 2.1
Lazy vs. Eager Loading Strategies in JPA 2.1Lazy vs. Eager Loading Strategies in JPA 2.1
Lazy vs. Eager Loading Strategies in JPA 2.1
 
ClickHouse new features and development roadmap, by Aleksei Milovidov
ClickHouse new features and development roadmap, by Aleksei MilovidovClickHouse new features and development roadmap, by Aleksei Milovidov
ClickHouse new features and development roadmap, by Aleksei Milovidov
 
Virtual Separation of Concerns (2011 Update)
Virtual Separation of Concerns (2011 Update)Virtual Separation of Concerns (2011 Update)
Virtual Separation of Concerns (2011 Update)
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12
 
Writing A Foreign Data Wrapper
Writing A Foreign Data WrapperWriting A Foreign Data Wrapper
Writing A Foreign Data Wrapper
 
エンタープライズ・クラウドと 並列・分散・非同期処理
エンタープライズ・クラウドと 並列・分散・非同期処理エンタープライズ・クラウドと 並列・分散・非同期処理
エンタープライズ・クラウドと 並列・分散・非同期処理
 
EuroPython 2015 - Big Data with Python and Hadoop
EuroPython 2015 - Big Data with Python and HadoopEuroPython 2015 - Big Data with Python and Hadoop
EuroPython 2015 - Big Data with Python and Hadoop
 
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...
 
Big data shim
Big data shimBig data shim
Big data shim
 
Hachiojipm11
Hachiojipm11Hachiojipm11
Hachiojipm11
 
Hadoop + Clojure
Hadoop + ClojureHadoop + Clojure
Hadoop + Clojure
 
Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School
 
User-Defined Table Generating Functions
User-Defined Table Generating FunctionsUser-Defined Table Generating Functions
User-Defined Table Generating Functions
 
Apache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customizationApache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customization
 
MAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxMAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptx
 
Hadoop_Pennonsoft
Hadoop_PennonsoftHadoop_Pennonsoft
Hadoop_Pennonsoft
 
Flink internals web
Flink internals web Flink internals web
Flink internals web
 

Plus de Kohei KaiGai

20221116_DBTS_PGStrom_History
20221116_DBTS_PGStrom_History20221116_DBTS_PGStrom_History
20221116_DBTS_PGStrom_HistoryKohei KaiGai
 
20221111_JPUG_CustomScan_API
20221111_JPUG_CustomScan_API20221111_JPUG_CustomScan_API
20221111_JPUG_CustomScan_APIKohei KaiGai
 
20211112_jpugcon_gpu_and_arrow
20211112_jpugcon_gpu_and_arrow20211112_jpugcon_gpu_and_arrow
20211112_jpugcon_gpu_and_arrowKohei KaiGai
 
20210928_pgunconf_hll_count
20210928_pgunconf_hll_count20210928_pgunconf_hll_count
20210928_pgunconf_hll_countKohei KaiGai
 
20210731_OSC_Kyoto_PGStrom3.0
20210731_OSC_Kyoto_PGStrom3.020210731_OSC_Kyoto_PGStrom3.0
20210731_OSC_Kyoto_PGStrom3.0Kohei KaiGai
 
20210511_PGStrom_GpuCache
20210511_PGStrom_GpuCache20210511_PGStrom_GpuCache
20210511_PGStrom_GpuCacheKohei KaiGai
 
20210301_PGconf_Online_GPU_PostGIS_GiST_Index
20210301_PGconf_Online_GPU_PostGIS_GiST_Index20210301_PGconf_Online_GPU_PostGIS_GiST_Index
20210301_PGconf_Online_GPU_PostGIS_GiST_IndexKohei KaiGai
 
20201128_OSC_Fukuoka_Online_GPUPostGIS
20201128_OSC_Fukuoka_Online_GPUPostGIS20201128_OSC_Fukuoka_Online_GPUPostGIS
20201128_OSC_Fukuoka_Online_GPUPostGISKohei KaiGai
 
20201113_PGconf_Japan_GPU_PostGIS
20201113_PGconf_Japan_GPU_PostGIS20201113_PGconf_Japan_GPU_PostGIS
20201113_PGconf_Japan_GPU_PostGISKohei KaiGai
 
20201006_PGconf_Online_Large_Data_Processing
20201006_PGconf_Online_Large_Data_Processing20201006_PGconf_Online_Large_Data_Processing
20201006_PGconf_Online_Large_Data_ProcessingKohei KaiGai
 
20200828_OSCKyoto_Online
20200828_OSCKyoto_Online20200828_OSCKyoto_Online
20200828_OSCKyoto_OnlineKohei KaiGai
 
20200806_PGStrom_PostGIS_GstoreFdw
20200806_PGStrom_PostGIS_GstoreFdw20200806_PGStrom_PostGIS_GstoreFdw
20200806_PGStrom_PostGIS_GstoreFdwKohei KaiGai
 
20200424_Writable_Arrow_Fdw
20200424_Writable_Arrow_Fdw20200424_Writable_Arrow_Fdw
20200424_Writable_Arrow_FdwKohei KaiGai
 
20191211_Apache_Arrow_Meetup_Tokyo
20191211_Apache_Arrow_Meetup_Tokyo20191211_Apache_Arrow_Meetup_Tokyo
20191211_Apache_Arrow_Meetup_TokyoKohei KaiGai
 
20191115-PGconf.Japan
20191115-PGconf.Japan20191115-PGconf.Japan
20191115-PGconf.JapanKohei KaiGai
 
20190926_Try_RHEL8_NVMEoF_Beta
20190926_Try_RHEL8_NVMEoF_Beta20190926_Try_RHEL8_NVMEoF_Beta
20190926_Try_RHEL8_NVMEoF_BetaKohei KaiGai
 
20190925_DBTS_PGStrom
20190925_DBTS_PGStrom20190925_DBTS_PGStrom
20190925_DBTS_PGStromKohei KaiGai
 
20190909_PGconf.ASIA_KaiGai
20190909_PGconf.ASIA_KaiGai20190909_PGconf.ASIA_KaiGai
20190909_PGconf.ASIA_KaiGaiKohei KaiGai
 
20190516_DLC10_PGStrom
20190516_DLC10_PGStrom20190516_DLC10_PGStrom
20190516_DLC10_PGStromKohei KaiGai
 
20190418_PGStrom_on_ArrowFdw
20190418_PGStrom_on_ArrowFdw20190418_PGStrom_on_ArrowFdw
20190418_PGStrom_on_ArrowFdwKohei KaiGai
 

Plus de Kohei KaiGai (20)

20221116_DBTS_PGStrom_History
20221116_DBTS_PGStrom_History20221116_DBTS_PGStrom_History
20221116_DBTS_PGStrom_History
 
20221111_JPUG_CustomScan_API
20221111_JPUG_CustomScan_API20221111_JPUG_CustomScan_API
20221111_JPUG_CustomScan_API
 
20211112_jpugcon_gpu_and_arrow
20211112_jpugcon_gpu_and_arrow20211112_jpugcon_gpu_and_arrow
20211112_jpugcon_gpu_and_arrow
 
20210928_pgunconf_hll_count
20210928_pgunconf_hll_count20210928_pgunconf_hll_count
20210928_pgunconf_hll_count
 
20210731_OSC_Kyoto_PGStrom3.0
20210731_OSC_Kyoto_PGStrom3.020210731_OSC_Kyoto_PGStrom3.0
20210731_OSC_Kyoto_PGStrom3.0
 
20210511_PGStrom_GpuCache
20210511_PGStrom_GpuCache20210511_PGStrom_GpuCache
20210511_PGStrom_GpuCache
 
20210301_PGconf_Online_GPU_PostGIS_GiST_Index
20210301_PGconf_Online_GPU_PostGIS_GiST_Index20210301_PGconf_Online_GPU_PostGIS_GiST_Index
20210301_PGconf_Online_GPU_PostGIS_GiST_Index
 
20201128_OSC_Fukuoka_Online_GPUPostGIS
20201128_OSC_Fukuoka_Online_GPUPostGIS20201128_OSC_Fukuoka_Online_GPUPostGIS
20201128_OSC_Fukuoka_Online_GPUPostGIS
 
20201113_PGconf_Japan_GPU_PostGIS
20201113_PGconf_Japan_GPU_PostGIS20201113_PGconf_Japan_GPU_PostGIS
20201113_PGconf_Japan_GPU_PostGIS
 
20201006_PGconf_Online_Large_Data_Processing
20201006_PGconf_Online_Large_Data_Processing20201006_PGconf_Online_Large_Data_Processing
20201006_PGconf_Online_Large_Data_Processing
 
20200828_OSCKyoto_Online
20200828_OSCKyoto_Online20200828_OSCKyoto_Online
20200828_OSCKyoto_Online
 
20200806_PGStrom_PostGIS_GstoreFdw
20200806_PGStrom_PostGIS_GstoreFdw20200806_PGStrom_PostGIS_GstoreFdw
20200806_PGStrom_PostGIS_GstoreFdw
 
20200424_Writable_Arrow_Fdw
20200424_Writable_Arrow_Fdw20200424_Writable_Arrow_Fdw
20200424_Writable_Arrow_Fdw
 
20191211_Apache_Arrow_Meetup_Tokyo
20191211_Apache_Arrow_Meetup_Tokyo20191211_Apache_Arrow_Meetup_Tokyo
20191211_Apache_Arrow_Meetup_Tokyo
 
20191115-PGconf.Japan
20191115-PGconf.Japan20191115-PGconf.Japan
20191115-PGconf.Japan
 
20190926_Try_RHEL8_NVMEoF_Beta
20190926_Try_RHEL8_NVMEoF_Beta20190926_Try_RHEL8_NVMEoF_Beta
20190926_Try_RHEL8_NVMEoF_Beta
 
20190925_DBTS_PGStrom
20190925_DBTS_PGStrom20190925_DBTS_PGStrom
20190925_DBTS_PGStrom
 
20190909_PGconf.ASIA_KaiGai
20190909_PGconf.ASIA_KaiGai20190909_PGconf.ASIA_KaiGai
20190909_PGconf.ASIA_KaiGai
 
20190516_DLC10_PGStrom
20190516_DLC10_PGStrom20190516_DLC10_PGStrom
20190516_DLC10_PGStrom
 
20190418_PGStrom_on_ArrowFdw
20190418_PGStrom_on_ArrowFdw20190418_PGStrom_on_ArrowFdw
20190418_PGStrom_on_ArrowFdw
 

Dernier

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 

Dernier (20)

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 

Writable Foreign Data Wrapper (JPUG Unconference 16-Feb-2013)

  • 1. Writable FDW KaiGai Kohei <kaigai@kaigai.gr.jp> Tw: @kkaigai JPUG PostgreSQL Unconference - 20130216
  • 2. Overview of FDW FDW: Foreign Data Wrapper External data source as if regular tables file_fdw ... for flat files postgres_fdw ... for remote PostgreSQL Extension performs as if a part of executor Read-only, right now, JPUG PostgreSQL Unconference - 20130216 2
  • 3. How does it work? (1/2) Query Tree FDW Extension pg_plan_queries set_base_rel_sizes GetForeignRelSize set_base_rel_pathlists GetForeignPaths create_scan_plan GetForeignPlan ExecInitForeignScan BeginForeignScan ExecuteQuery ForeignNext IterateForeignScan ExecEndForeignScan EndForeignScan JPUG PostgreSQL Unconference - 20130216 3
  • 4. How does it work? (2/2) ResultSet Join Performs as if Join data format converter TupleTableSlot TupleTableSlot TupleTableSlot Foreign Foreign Regular Table Table Remote Flat Table Table Files FDW driver FDW driver JPUG PostgreSQL Unconference - 20130216 4
  • 5. Learn from existing logic to modify tables ExecScan ExecQual ExecModifyTable  TupleTableSlot TupleTableSlot with “ctid” ExecInsert ExecUpdate system JunkFilter ExecDelete TableScan column ctid HeapTuple Ctid addresses the target row to be modified. JPUG PostgreSQL Unconference - 20130216 5
  • 6. Issues towards writable foreign table Way to identify a remote row to be modified “rowid” to identify a particular remote row “pseudo-column” to move values out of foreign table definition New methods relevant to ExecModityTable Methods relevant to ... ExecInitModifyTable ExecInsert, ExecUpdate, ExecDelete ExecEndModifyTable Transaction control with remote data source Responsibility of FDW drivers JPUG PostgreSQL Unconference - 20130216 6
  • 7. rowid pseudo-column (1/3) Calculation of ExecScan newer tuple HeapTuple + rowid TupleTableSlot ExecModifyTable  Remote Table Foreign ExecInsert Table ExecUpdate ExecDelete FDW driver HeapTuple’ + rowid Update a row Primary Key column identified with rowid JPUG PostgreSQL Unconference - 20130216 7
  • 8. rowid pseudo-column (2/3) Calculation of resjunk=true newer tuple ExecScan A B C D rowid tts_values     ExecModifyTable tts_isnull   TupleTableSlot ExecInsert ExecUpdate ExecDelete CREATE FOREIGN TABLE Foreign ( Table A int, TupleTableSlot B int, C text, HeapTuple’ + rowid FDW driver D date ) SERVER loopback; JPUG PostgreSQL Unconference - 20130216 8
  • 9. rowid pseudo-column (3/3) AttrNumber GetForeignRelWidth(PlannerInfo *root, RelOptInfo *baserel, Relation foreignrel, bool inhparent, List *targetList); A new GetForeignRelWidth() method It returns width of TupleTableSlot, even if it overs number of columns in table definition. if targetList contains “rowid” column, FDW driver has to expand the width of TupleTableSlot. Extra fields are acquired to store values of pseudo- columns JPUG PostgreSQL Unconference - 20130216 9
  • 10. Why multiple pseudo-column is allowed? SELECT X, Y, (X-Y)^2 from ftable;  SELECT X, Y, Pcol_1 from ftable; Just reference to SELECT X, Y, (X-Y)^2 AS Pcol_1 the calculated result from remote_data_source; in the remote side Remote Query Calculation off-loading GetForeignRelWidth() can replace expression node of the given TargetEntry with pseudo-column reference, instead of complex calculation. Utilization of external computing resource GPU, remote host, ... JPUG PostgreSQL Unconference - 20130216 10
  • 11. New APIs List *PlanForeignModify(PlannerInfo *root, ModifyTable *plan, Index resultRelation, Plan *subplan); void BeginForeignModify(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, List *fdw_private, Plan *subplan, int eflags); TupleTableSlot *ExecForeignInsert(ResultRelInfo *rinfo, TupleTableSlot *slot); TupleTableSlot *ExecForeignUpdate(ResultRelInfo *rinfo, const char *rowid, TupleTableSlot *slot); bool ExecForeignDelete(ResultRelInfo *rinfo, const char *rowid); void EndForeignModify(ResultRelInfo *rinfo); JPUG PostgreSQL Unconference - 20130216 11
  • 12. Working Example (1/2) postgres=# CREATE FOREIGN TABLE ft1 (a int, b text, c float) SERVER loopback OPTIONS (relname 't1'); CREATE FOREIGN TABLE postgres=# SELECT * FROM ft1; a | b | c ---+-----+----- 1 | aaa | 1.1 2 | bbb | 2.2 3 | ccc | 3.3 4 | ddd | 4.4 5 | eee | 5.5 (5 rows) postgres=# EXPLAIN (verbose) SELECT * FROM ft1 WHERE b like '%b%'; QUERY PLAN ------------------------------------------------------------------- Foreign Scan on public.ft1 (cost=100.00..100.01 rows=1 width=44) Output: a, b, c Filter: (ft1.b ~~ '%b%'::text) Remote SQL: SELECT a, b, c FROM public.t1 (4 rows) JPUG PostgreSQL Unconference - 20130216 12
  • 13. Working Example (2/2) postgres=# UPDATE ft1 SET b = b || '_updt‘ WHERE a % 2 = 0 RETURNING *; a | b | c ---+----------+----- 2 | bbb_updt | 2.2 4 | ddd_updt | 4.4 (2 rows) UPDATE 2 postgres=# EXPLAIN (verbose, costs off) UPDATE ft1 SET b = b || '_updt' WHERE a % 2 = 0 RETURNING *; QUERY PLAN ---------------------------------------------------------------- Update on public.ft1 Output: a, b, c -> Foreign Scan on public.ft1 Output: a, (b || '_updt'::text), c, ctid, ft1.* Remote SQL: SELECT a, b, c, ctid FROM public.t1 WHERE (((a OPERATOR(pg_catalog.%) 2) OPERATOR(pg_catalog.=) 0)) FOR UPDATE (5 rows) JPUG PostgreSQL Unconference - 20130216 13
  • 14. Transaction Control postgres=# BEGIN; BEGIN postgres=# UPDATE ft1 SET b = b || '_updt' WHERE a % 2 = 1; UPDATE 3 postgres=# ABORT; ROLLBACK postgres=# SELECT * FROM t1; a | b | c ---+----------+----- 2 | bbb | 2.2 4 | ddd | 4.4 1 | aaa_updt | 1.1 3 | ccc_updt | 3.3 5 | eee_updt | 5.5 (5 rows) Transaction control is responsibility of FDW drivers Register(Sub)XactCallback gives extensions chance to get control on transaction events, but I didn’t cover it yet. JPUG PostgreSQL Unconference - 20130216 14
  • 15. Does someone implement DLM? postgres=# WITH ch AS ( UPDATE t1 SET b = 'changed' RETURNING a ) UPDATE ft1 SET b = 'new val‘ FROM ch WHERE ch.a = ft1.a; ^CCancel request sent ^CCancel request sent Right now, we have no distributed lock manager :-) Be careful not to acquire exclusive lock on same table In above example... This session acquires exclusive lock on table t1. table t1 is external data source of ft1 via loopback connection. UPDATE on ft1 also requires exclusive lock on t1 on remote session. JPUG PostgreSQL Unconference - 20130216 15
  • 16. Current Status JPUG PostgreSQL Unconference - 20130216 16
  • 17. FDW, be writable! JPUG PostgreSQL Unconference - 20130216 17