クローラーをテストしよう！

•

0 j'aime•709 vues

Kazuki Morozumi

クローラーの品質を維持するためにテストを整備した話

Logiciels

2019/02/26
https://scouty.co.jp
Kazuki Morozumi

Amazon 
DynamoDB
Amazon 
RDS
Amazon  
ElastiCache
Amazon 
SQS
Scrapy
spider middleware
pipeline
Request
Output
Input
Response

Amazon 
DynamoDB
Amazon 
RDS
Amazon  
ElastiCache
Amazon 
SQS
Scrapy
spider middleware
pipeline
Request
Output
Input
Output
Response
Input

Amazon 
DynamoDB
Amazon 
RDS
Amazon  
ElastiCache
Amazon 
SQS
Scrapy
spider middleware
pipeline
Request
Output
Input
Input/Output
Response
(middleware)
DB localstack

scouty CI/CD
Amazon ECS
Docker compose •
• unittest
• Integration test
•
$ git push git push $ ecs-goploy

https://scouty.co.jp
scouty s
(Scouty → scouty)

Recommandé

Atlantisで実現するTerraformのGitOps理弘山崎

자바를 잡아주는 GURU가 있다구!? - 우여명 (아이스크림에듀) :: AWS Community Day 2020 AWSKRUG - AWS한국사용자모임

IVS CTO Night And Day 2018 Winter - AWS Startup Tech Office HoursAmazon Web Services Japan

20180309 DLIもくもく会　Deep Learning on AWSYasuhiro Matsuo

AWS Fargate でお手軽開発ブランチデプロイTatsuo Yokoyama

Octocatは技術的負債の夢を見るか？treby

WordPressに足りないモノMasaki Komagata

Внедрение SDLC в боевых условиях / Егор Карбутов (Digital Security)Ontico

Recommandé

Atlantisで実現するTerraformのGitOps理弘山崎

자바를 잡아주는 GURU가 있다구!? - 우여명 (아이스크림에듀) :: AWS Community Day 2020 AWSKRUG - AWS한국사용자모임

IVS CTO Night And Day 2018 Winter - AWS Startup Tech Office HoursAmazon Web Services Japan

20180309 DLIもくもく会　Deep Learning on AWSYasuhiro Matsuo

AWS Fargate でお手軽開発ブランチデプロイTatsuo Yokoyama

Octocatは技術的負債の夢を見るか？treby

WordPressに足りないモノMasaki Komagata

Внедрение SDLC в боевых условиях / Егор Карбутов (Digital Security)Ontico

AmebaPico 裏側の技術やAWSの利用についてKohei Morino

Open Source Monitoring Toolsm_richardson

2018512 AWS上での機械学習システムの構築とSageMakerYasuhiro Matsuo

hbstudy@bpstudy#50 配布用Toshiaki Baba

플렉스팀 프론트엔드 기술 스택의 이해: `lint`, `build`, `run`Heejong Ahn

Amazon Aurora로 안전하게 migration 하기Jesang Yoon

クラウドで災害対策Serverworks Co.,Ltd.

개발자가 알아두면 좋은 5가지 AWS 인공 지능 서비스 깨알 지식 (윤석찬, AWS 테크에반젤리스트) :: AWS DevDay 2018Amazon Web Services Korea

forteeに脆弱性検査をかけてみた VAddy編ichikaway

Apache Spark v3.0.0Jean-Georges Perrin

Angular2 + AWS Lambdaでサーバサイドレンダリングしてみた暁三宅

CodeStarではじめるRailsアプリのCodeDeploy化Keishi Hosoba

グラフデータベース Neptune 使ってみたYoshiyasu SAEKI

AWS におけるサーバーレスの基礎からチューニングまで崇之清水

Economies of Scaling SoftwareJoshua Long

20181027 deep learningcommunity_awsHirokuni Uchida

Spark cassandra connector.API, Best Practices and Use-CasesDuyhai Doan

Serverless in production, an experience report (FullStack 2018)Yan Cui

マイクロにしすぎた結果がこれだよ！mosa siru

Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCPDaniel Zivkovic

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfsteffenkarlsson2

APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795

Contenu connexe

Similaire à クローラーをテストしよう！

AmebaPico 裏側の技術やAWSの利用についてKohei Morino

Open Source Monitoring Toolsm_richardson

2018512 AWS上での機械学習システムの構築とSageMakerYasuhiro Matsuo

hbstudy@bpstudy#50 配布用Toshiaki Baba

플렉스팀 프론트엔드 기술 스택의 이해: `lint`, `build`, `run`Heejong Ahn

Amazon Aurora로 안전하게 migration 하기Jesang Yoon

クラウドで災害対策Serverworks Co.,Ltd.

개발자가 알아두면 좋은 5가지 AWS 인공 지능 서비스 깨알 지식 (윤석찬, AWS 테크에반젤리스트) :: AWS DevDay 2018Amazon Web Services Korea

forteeに脆弱性検査をかけてみた VAddy編ichikaway

Apache Spark v3.0.0Jean-Georges Perrin

Angular2 + AWS Lambdaでサーバサイドレンダリングしてみた暁三宅

CodeStarではじめるRailsアプリのCodeDeploy化Keishi Hosoba

グラフデータベース Neptune 使ってみたYoshiyasu SAEKI

AWS におけるサーバーレスの基礎からチューニングまで崇之清水

Economies of Scaling SoftwareJoshua Long

20181027 deep learningcommunity_awsHirokuni Uchida

Spark cassandra connector.API, Best Practices and Use-CasesDuyhai Doan

Serverless in production, an experience report (FullStack 2018)Yan Cui

マイクロにしすぎた結果がこれだよ！mosa siru

Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCPDaniel Zivkovic

Similaire à クローラーをテストしよう！ (20)

AmebaPico 裏側の技術やAWSの利用について

Open Source Monitoring Tools

2018512 AWS上での機械学習システムの構築とSageMaker

hbstudy@bpstudy#50 配布用

플렉스팀 프론트엔드 기술 스택의 이해: `lint`, `build`, `run`

Amazon Aurora로 안전하게 migration 하기

クラウドで災害対策

개발자가 알아두면 좋은 5가지 AWS 인공 지능 서비스 깨알 지식 (윤석찬, AWS 테크에반젤리스트) :: AWS DevDay 2018

forteeに脆弱性検査をかけてみた VAddy編

Apache Spark v3.0.0

Angular2 + AWS Lambdaでサーバサイドレンダリングしてみた

CodeStarではじめるRailsアプリのCodeDeploy化

グラフデータベース Neptune 使ってみた

AWS におけるサーバーレスの基礎からチューニングまで

Economies of Scaling Software

20181027 deep learningcommunity_aws

Spark cassandra connector.API, Best Practices and Use-Cases

Serverless in production, an experience report (FullStack 2018)

マイクロにしすぎた結果がこれだよ！

Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP

Dernier

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfsteffenkarlsson2

APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795

Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980

Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...Abortion Clinic

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1KnowledgeSeed

Studiovity film pre-production and screenwriting softwareinfo611746

JustNaik Solution Deck (stage bus sector)Max Lee

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy

AI Hackathon.pptxERRORhackerboy

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Andrea Goulet

IT Software Development Resume, Vaibhav jha 2024vaibhav130304

AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171

Crafting the Perfect Measurement Sheet with PLM IntegrationWave PLM

What need to be mastered as AI-Powered Java DevelopersEmilyJiang23

Secure Software Ecosystem Teqnation 2024Soroosh Khodami

CompTIA Security+ (Study Notes) for cs.pdfFurqanuddin10

The Impact of PLM Software on Fashion ProductionWave PLM

5 Reasons Driving Warehouse Management Systems DemandCanary7-Warehouse Management System

Dernier (20)

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf

APVP,apvp apvp High quality supplier safe spot transport, 98% purity

Mastering Windows 7 A Comprehensive Guide for Power Users .pdf

Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1

Studiovity film pre-production and screenwriting software

JustNaik Solution Deck (stage bus sector)

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)

AI Hackathon.pptx

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...

IT Software Development Resume, Vaibhav jha 2024

AI/ML Infra Meetup | Perspective on Deep Learning Framework

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf

Crafting the Perfect Measurement Sheet with PLM Integration

What need to be mastered as AI-Powered Java Developers

Secure Software Ecosystem Teqnation 2024

CompTIA Security+ (Study Notes) for cs.pdf

The Impact of PLM Software on Fashion Production

5 Reasons Driving Warehouse Management Systems Demand

クローラーをテストしよう！

1. 2019/02/26 https://scouty.co.jp Kazuki Morozumi

4. scouty

5. scouty

6. 2018 8 27 scouty

7. scouty

9. scouty

10. scouty

11.

12. scouty

13. scouty

14. scouty • • •

15. scouty ( )

16. scouty ( )

17.

18.

19. Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Scrapy spider middleware pipeline Request Output Input Response

20. Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Scrapy spider middleware pipeline Request Output Input Output Response Input

21. Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Scrapy spider middleware pipeline Request Output Input Input/Output Response (middleware) DB localstack

22. scrapy

23. • • DB

24.

25.

26. TDD

27. scouty CI/CD Amazon ECS Docker compose • • unittest • Integration test • $ git push git push $ ecs-goploy

28. scouty ChatOps

29. scouty

30.

31.

32.   scouty 1 1   scouty     176   176 U X

33. SNS SNS   SNS   SNS 1 1 scouty      

34.

35.

36.

37. https://scouty.co.jp scouty s (Scouty → scouty)