2019 05-28 SRE Consul Criteo Meetup

•Télécharger en tant que PPTX, PDF•

0 j'aime•273 vues

This presentation explains the challenges we face at Criteo on discovering machines and services. Criteo is using HashiCorp's Consul to discover services. We explain what is Consul, how it works, what are the challenges we faced and how we improved it. We also explain how using it in combinaison with consul-templaterb can allow us performing Inversion Of Control for the whole infrastructure allowing us to use it as a database and iterating faster.

Internet

Pierre Souchay pierresouchay
Meetup S.R.E Tech Talk – Criteo Paris 05/28/2019
Consul Discovery:
Challenges &
Opportunities

2 •
EJB / 3 tier architecture
1/2 Web
1 App
1 Database
Early SOA (Service Oriented Architecture)
Discovery’s history from 2001 to 2010

3 •
SOA Step 2
1 Service / Machine
1 Apps needs several services
Discovery’s history early 2010’s

4 •
SOA Step 3: µ-services
1 container = 1 service(s)
10’s of services / machine
1000’s of services
Discovery now
3,000+ services
200,000+ instances
2,000+ instances/service
35,000+ machines
9 DCs

6 •
Open-Source, 2014
No SPOF / Fault Tolerant
Distributed Agents on all machines
Services Oriented / DC aware
Updates in Real-Time of Services
Distributed toolbox (Locks, K/V…)
Easy to integrated (DNS support)
Can Work on any IP network
Consul is a Discovery Database with fault-tolerance

8 •
5 people
Create SDK for other teams (JVM, C#, python, ruby)
Handle all infrastructure, on-call 24/24 7/7
Architecture patterns
1st worldwide contributors to Consul
The team

10 •
“Criteo has probably the most intensive usage of Consul in the world” (M. Hashimoto)
Discover all instances of all systems in our applications
All the Load-Balancing, the DNS provisioning
Metrics
Alerting systems
Used it with bare-metal (Windows, Linux), Mesos, Kubernetes, Hadoop
One of the biggest installations of Consul in the world

11 •
RPC query/s/DC from ~1.5k/s to 9k/s (up to 1300 qps on a single service)
1 change/sec on a large service of many instances
→ 2k req/sec if 2k observers #RealLife
Consul At Scale

12 •
~100 PRs, 70+ merged upstream
~15 PR for features (Service.Meta, weights…)
~30 PR merged for performance (DNS, watches)
~10 PR merged for safety (node registration, memberlist…)
~2 PR fixing security bugs
OSS UI: https://github.com/criteo/consul-templaterb/
Our Pull request

13 •
Bandwidth: from 1Gb/s to 12k/s
CPU: from 32/32 CPUs at 100% to 1/32 CPU at 100%
From 3/4 notifications/s to 1 notification/10 min for 1 service
From 1 incident / 10 days to no incident in 6 months
From a fragile tool to a database for the whole infrastructure
Prometheus improvements
Metadata for services…
Improvements from our merged PRs

14 •
A OSS Scalable UI for Consul (consul-templaterb)

16 •
Load-Balancing unification
F5 / HaProxy / DNS / Service-Mesh
Weight issue
Snowball effect
Load-Balancing in Consul

17 •
Services expose semantics:
I want HTTPS
I speak Swagger
Call someone when I lost 40% of capacity
Tools observe, react and provision systems: Consul is an infrastructure database
Inversion of Control

22 •
Infrastructure Service Mesh (Details)

23 •
GitHub:
- Criteo’s patches: http://github.com/criteo-forks/consul/
- UI and template system: http://github.com/criteo/consul-templaterb/
Video resources of previous presentations
- HashiConf ‘18: https://www.hashicorp.com/resources/criteo-containers-consul-connect
- HashiTalks (Operating Consul at Scale): https://www.youtube.com/watch?v=x9MiaV0WdSs
- Devoxx ‘19: https://www.youtube.com/watch?v=aQb2_WrmED0
- Consul-Timeline (our UI): https://youtu.be/zLzrLGLLl4Q
Q&A + Resources

Contenu connexe

Tendances

Serverless for the Cloud Native Era with Fission

NATS

Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service

Weaveworks

Spring Cloud and Netflix Components

Dharshan Sastry B N

Building applications that leverage blockchain data can be cumbersome, but it’s not impossible. Because blockchains don’t typically have RESTful APIs, delivering data to your web app can be complicated. However, we found a way around this challenge by deploying a blockchain node to an EC2 instance. With this approach, we were able to request data and deliver it to our web app using Lambda. In this presentation, you will learn: -How we used AWS Lambda and blockchain to build a robust web app -Best practices for delivering blockchain data to your web app -Cost-effective ways of serving blockchain data to your application Learn more in this blog post: https://www.verypossible.com/blog/how-to-serve-blockchain-data-on-the-web

How to Serve Blockchain Data with AWS Lambda

Very

My graduation project on Opsschool ('the elephant')

Liora Kamil

In mission-critical real time applications, using machine learning to analyze streaming data are gaining momentum. In those applications Apache Kafka is the most widely used framework to process the data streams. It typically works with other machine learning frameworks for model inference and training purposes. In this talk, our focus is to discuss the KafkaDataset module in TensorFlow. KafkaDataset processes Kafka streaming data directly to TensorFlow’s graph. As a part of Tensorflow (in ‘tf.contrib’), the implementation of KafkaDataset is mostly written in C++. The module exposes a machine learning friendly Python interface through Tensorflow’s ‘tf.data’ API. It could be directly feed to ‘tf.keras’ and other TensorFlow modules for training and inferencing purposes. Combined with Kafka streaming itself, the KafkaDataset module in TensorFlow removes the need to have an intermediate data processing infrastructure. This helps many mission-critical real time applications to adopt machine learning more easily. At the end of the talk we will walk through a concrete example with a demo to showcase the usage we described.

Real Time Streaming Data with Kafka and TensorFlow (Yong Tang, MobileIron) Ka...

confluent

Service mesh is a dedicated layer of a company's infrastructure which should simplify communication between services and make it reliable, secure and observable. In this talk, we'll go deep into Booking.com's case study of introducing service mesh. We will discover the reasons and objectives of the project. Why Envoy was selected as the base rather than other available options. Find out what is the setup and features of the homegrown control plane. We will expand on what service is provided for developers and how they safely deploy potentially dangerous configuration changes. Finally, we will talk about pitfalls met along the way.

Introducing envoy-based service mesh at Booking.com

Ivan Kruglov

Presented by Tom Wilkie at PromCon 2017 With Speaker Notes: https://goo.gl/V9qGva At PromCon 2016, I presented "Project Frankenstein: A multitenant, horizontally scalable Prometheus as a service". It's now one year later, and lots has changed - not least the name! This talk will discuss what we've learnt running a Prometheus service for the past year, the architectural changes we made from the original design, and the improvements we've made to the Cortex user experience.

Cortex: Prometheus as a Service, One Year On

Kausal

Building Microservices with Spring Cloud and Netflix OSS

Semih Hakkıoğlu

As many startups of the last decade, SoundCloud’s architecture started as a Ruby-on-Rails monolith, which later had to be broken into microservices to cope with the growing size and complexity of the site. The microservices initially ran on an in-house container management and deployment platform. Recently, the company has started to migrate to Kubernetes. With the introduction of microservices, the existing conventional monitoring setup failed both conceptually and in terms of scalability. Thus, starting in 2012, SoundCloud invested heavily into the development of the open-source monitoring system Prometheus, which was designed for large-scale highly dynamic service-oriented architectures. Migrating to Kubernetes, it became apparent that Prometheus and Kubernetes are a match made in open-source heaven. The talk will demonstrate the current Prometheus setup at SoundCloud, monitoring a large-scale Kubernetes cluster.

Monitoring a Kubernetes-backed microservice architecture with Prometheus

Fabian Reinartz

GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...

wallyqs

Kong in 1.x Territory

Thibault Charbonnier

Talk given at OpenResty Con 2017 in Beijing. Kong (https://getkong.org) is a widely-adopted open source API Gateway built with OpenResty. It aims at helping secure, manage, and extend microservices-based architectures with minimal effort from the user, while ensuring platform agnosticism. In this talk, we will explore the challenges we encountered developing such an OpenResty application, and how we overcame many of them by way of libraries and contributions back to the OpenResty community. We will cover topics such as clustering OpenResty nodes, inter-workers communication, DNS resolution, typical pitfalls OpenResty developers should avoid, and much more.

Developing a user-friendly OpenResty application

Thibault Charbonnier

Monitoring Kubernetes is vital to understanding the health and performance of a cluster, but which metrics are most important to add to your dashboards and alert on? Jacob will discuss how to most effectively monitor and visualize your Kubernetes cluster using the Grafana Kubernetes plugin and PromQL. Some of the topics of discussion include(1) how and what to collect when monitoring Kubernetes, (2) how to bring your Grafana dashboards to the next level by using Kubernetes as a data-source, and (3) what to do when managing multiple clusters. All of these topics and more will be discussed to help people get the most out of their Kubernetes monitoring.

What Does Kubernetes Look Like?: Performance Monitoring & Visualization with ...

InfluxData

Cinder Updates - Liberty Edition

OpenStack Foundation

The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion. In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

The RED Method: How to monitoring your microservices.

Grafana Labs

Manage your APIs and Microservices with an API Gateway

Thibault Charbonnier

How to make your mobile... to call? Is it easy to create another Skype or Whatsapp? This talk will walk you through the various aspects of creating "the calling app". Starting with what the VoIP actually is, going through some theory of WebRTC connections, and ending with the CallKit of iOS vs Android way of handling calls with the React Native. It will give you a sneak peek of what is required to make a production app that handles outgoing and incoming calls and what kind of infrastructure you need. The talk is based on the experience taken from the phone app made for sales and support teams around the world.

React Native EU 2021 - Creating a VoIP app in React Native - the beginner's g...

Wojciech Kwiatek

Consul scale

Ariel Moskovich

A year ago, our technical debt was incurring compound interest and we weren’t going to be able to keep making the minimum payments. We had to take a hard look at how we built our infrastructure—we had a code base that had evolved over time, with no design patterns and little structure. Changes in one Chef cookbook would have cascading failures in seemingly unrelated places elsewhere. We had dependencies upon dependencies. We faced a critical decision: fix what we have in place or start over again using what we’d had before as a guide. We chose the latter. And so, over the past 12 months, we rewrote the majority of our infrastructure code, upgraded the operating system, and rebuilt every single server in our public cache fleet from the ground up—without anyone noticing. Previously, it took Fastly about six people over six weeks to install a new POP. Now, it requires just one person and a single week. We attribute our ability to rapidly scale, both safely and reliably, to a new approach, which we'll discuss in this talk. We will relive the design and redeployment of our public cache fleet, the challenges faced, lessons learned, and benefits gained from our new architecture.

Altitude NY 2018: Don't let the weeds overwhelm the garden

Fastly

Tendances (20)

Serverless for the Cloud Native Era with Fission

Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service

Spring Cloud and Netflix Components

How to Serve Blockchain Data with AWS Lambda

My graduation project on Opsschool ('the elephant')

Real Time Streaming Data with Kafka and TensorFlow (Yong Tang, MobileIron) Ka...

Introducing envoy-based service mesh at Booking.com

Cortex: Prometheus as a Service, One Year On

Building Microservices with Spring Cloud and Netflix OSS

Monitoring a Kubernetes-backed microservice architecture with Prometheus

GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...

Kong in 1.x Territory

Developing a user-friendly OpenResty application

What Does Kubernetes Look Like?: Performance Monitoring & Visualization with ...

Cinder Updates - Liberty Edition

The RED Method: How to monitoring your microservices.

Manage your APIs and Microservices with an API Gateway

React Native EU 2021 - Creating a VoIP app in React Native - the beginner's g...

Consul scale

Altitude NY 2018: Don't let the weeds overwhelm the garden

Similaire à 2019 05-28 SRE Consul Criteo Meetup

Consul administration at scale

Pierre Souchay

Microservices, Service Discovery and Registration have been heading towards the peak of inflated expectations on the Gartner Hype cycle for over the last year or so, but there has often been a lack of clarity as to what these are, why are they needed or how to implement them well. Service discovery and registration are key components of most distributed systems and service oriented architectures. In this session we will talk about what, why and how of service registration and discovery in distributed systems in general and OpenStack in particular. We will talk about some of the technologies that address this challenge like Zookeeper, Etcd, Consul, Mesos-DNS, Minuteman, SkyDNS, SmartStack or Eureka. We will also address how these technologies as well as existing OpenStack projects can be used to solve this problem inside OpenStack environments.

Service Discovery and Registration in a Microservices Architecture

PLUMgrid

Scylla Summit 2016: Scylla at Samsung SDS

ScyllaDB

BonAHA framework - IEEE CCNC 2009

Suman Srinivasan

CNCF Singapore - Introduction to Envoy

Harish

Beyond static configuration

Stefan Schimanski

How to deploy & optimize eZ Publish (2014)

Kaliop-slide

One of the critical factors for development velocity is software correctness. Our ability to develop and ship new features fast is bounded by our ability to validate several aspects of the change: * Does the feature meet the requirements? * How does the feature affect existing code, and how can it affect the production environment? With continues codebase growth and new features being added, naturally our productivity decreases, and our need to improve the guarantees for quality and correctness increase. In this talk, I’ll focus on testing environments: why developers need a self-serve platform to create a full functioning environment on-demand, how such environments should be managed, and how can one restore part of the lost velocity. I’ll cover an internal system we use at AppsFlyer called ‘Namespaces’ that addresses the issue with the help of Mesos / Marathon, Docker, Traefik, and Consul.

Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...

DevOpsDays Tel Aviv

Today, every one of us wants to get things done fast. The fact of the matter is Serverless is a fantastic platform for doing things fast. Because, with Serverless, you really don’t have time to waste in terms of delivering your business value. Turns out you can with the right cloud services. In this talk we’ll create a microservice using Azure Functions and also get introduced to bigger picture of serverless computing. I presented this session in Global Azure Bootcamp 2019 in Dublin. #GlobalAzure #AzureFunctions #Serverless

Develop in ludicrous mode with azure serverless

Lalit Kale

Apricot2017 Request tracing in distributed environment

Hieu LE ☁

Running High-Speed Serverless with nuclio

iguazio

Docker practical solutions

Kesav Kumar Kolla

Cytoscape CI Chapter 2

bdemchak

Workflow on Hadoop Using Oozie__HadoopSummit2010

Yahoo Developer Network

Logging/Request Tracing in Distributed Environment

APNIC

One of the most fundamental challenges of monitoring modern cloud applications is the ability to see all the components and their dependencies. The lack of visibility is a critical problem that is worsening as the instance lifespan reduces, components span private and public clouds, and external service dependency increases. As the pace of software development and complexity of application continues to increase, the visibility challenge for operations teams can be summarized as “driving at 100mph with blindfolds!”

Application Mapping & Techniques

Steffi Li

.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...

Javier García Magna

In this talk, Till Rohrmann and Addison Higham discuss how Flink allows for ambitious stream processing workflows and how Pulsar and Flink enable new capabilities that push forward the state-of-the-art in streaming. They will also share upcoming features and new capabilities in the integrations between Flink and Pulsar and how these two communities are working together to truly advance the power of stream processing.

Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote

StreamNative

RTI Data-Distribution Service (DDS) Master Class 2011

Gerardo Pardo-Castellote

Puppet devops wdec

Wojciech Dec

Similaire à 2019 05-28 SRE Consul Criteo Meetup (20)

Consul administration at scale

Service Discovery and Registration in a Microservices Architecture

Scylla Summit 2016: Scylla at Samsung SDS

BonAHA framework - IEEE CCNC 2009

CNCF Singapore - Introduction to Envoy

Beyond static configuration

How to deploy & optimize eZ Publish (2014)

Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...

Develop in ludicrous mode with azure serverless

Apricot2017 Request tracing in distributed environment

Running High-Speed Serverless with nuclio

Docker practical solutions

Cytoscape CI Chapter 2

Workflow on Hadoop Using Oozie__HadoopSummit2010

Logging/Request Tracing in Distributed Environment

Application Mapping & Techniques

.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...

Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote

RTI Data-Distribution Service (DDS) Master Class 2011

Puppet devops wdec

Dernier

20240510 QFM016 Irresponsible AI Reading List April 2024.pdf

Matthew Sinclair

𓀤Call On 7877925207 𓀤 Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready For Sex At Your Doorstep Call Mr Chirag Ji +91-7877925207 NORTH/SOUTH INDIANS HIGH CLASS MODELS/COLLEGE GIRLS/ HOUSEWIVES/BODY TO BODY MASSAGE, SANDWICH MASSAGE, SHOWER BATHING, BLOW JOB, LICKING, FRONT AND BACK SHOT, RUBBING, KISSING SMOOCHING ETC ETC... UNLIMITED FUN Call 24/7. Your assistance for your service... ☛ ✔✔ secure✔✔ 100% safe ✔✔ Vip Callgirl Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ A02524SW ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife • Collage Going Girls. • Travelling Escorts. • Ramp-models • Foreigner And Many More.. ★OUR BEST SERVICES: - FOR BOOKING ★ A-Level (5 star escort) ★ Strip-tease ★ BBBJ (Bareback Blowjob)

𓀤Call On 7877925207 𓀤 Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...

Neha Pandey

VIP Call Girls Pollachi 7001035870 Whatsapp Number, 24/07 Booking Escorts Service Available Whatsapp SABANA ☎️ : [+91-7001035870] Escorts Service are always ready to make their clients happy. Their exotic looks and sexy personalities are sure to turn heads. You can enjoy with them, including massages and erotic encounters. Our area Escorts are young and sexy, so you can expect to have an exotic time with them. They are trained to satiate your naughty nerves and they can handle anything that you want. They are also intelligent, so they know how to make you feel comfortable and relaxed Independent Escorts Service They know all the sex positions and can satisfy you in any way that you desire. They can even give you erotic massages to help you relax before your session. This is essential, because a man who is stressed won’t be receptive to the pleasures of sex. They also know how to play with your sexy organs, so you’ll have plenty of foreplay and cuddling. P252024SS SERVICE ✅ ❣️ ⭐➡️HOT & SEXY MODELS // COLLEGE GIRLS HOUSE WIFE RUSSIAN , AIR HOSTES ,VIP MODELS . AVAILABLE FOR COMPLETE ENJOYMENT WITH HIGH PROFILE INDIAN MODEL AVAILABLE HOTEL & HOME ★ SAFE AND SECURE HIGH CLASS SERVICE AFFORDABLE RATE ★ SATISFACTION,UNLIMITED ENJOYMENT. ★ All Meetings are confidential and no information is provided to any one at any cost. ★ EXCLUSIVE PROFILes Are Safe and Consensual with Most Limits Respected ★ Service Available In: - HOME & HOTEL Star Hotel Service .In Call & Out call SeRvIcEs : ★ A-Level (star escort) ★ Strip-tease ★ BBBJ (Bareback Blowjob)Receive advanced sexual techniques in different mode make their life more pleasurable. ★ Spending time in hotel rooms ★ BJ (Blowjob Without a Condom) ★ Completion (Oral to completion) ★ Covered (Covered blowjob Without condom ★ANAL SERVICES.

VIP Call Girls Pollachi 7001035870 Whatsapp Number, 24/07 Booking

dharasingh5698

Call Girl In Delhi ☎92055#41914 ¶¶ Indian,Russian Best Quality full Educated And Full Cooperative Independent Call Girls Escort Services In New Delhi- I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls and Escorts, We Are Located in 3* 4* 5* Hotels in Delhi. Safe & Secure High Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Delhi High class luxury and premium escorts agency Indian Russian Call Girls In Delhi Booking Good High Profile Escorts (Call Girls) In Delhi 5 Star Hotel ,Incall Service,OutCall Service, We provide services by Call Girls,College Girls,Modals Get High Profile queens,Well Educated,Good Looking,Full Cooperative Model, Russian Models,Punjabi Girls Kashmeri Girls Services etc… We Provide Hottest Female With Safe And Consensual With Most Limits Respected Complete Satisfaction Guaranteed…Service. Call Me Spacial For Including Incall//outcall Service In New Delhi Indian Russian Escorts Service

Hire↠Young Call Girls in Tilak nagar (Delhi) ☎️ 9205541914 ☎️ Independent Esc...

Delhi Call girls

VIP Model Call Girls NIBM ( Pune ) Call ON 8005736733 Starting From 5K to 25K High Profile Escorts In Pune Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 (V030524]N) 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

VIP Model Call Girls NIBM ( Pune ) Call ON 8005736733 Starting From 5K to 25K...

SUHANI PANDEY

Ganeshkhind ! Call Girls Pune - 450+ Call Girl Cash Payment 8005736733 Neha Thakur Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔$V15 ✔✔To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

Ganeshkhind ! Call Girls Pune - 450+ Call Girl Cash Payment 8005736733 Neha T...

SUHANI PANDEY

Call Girls Ludhiana Just Call 98765-12871 Top Class Call Girl Service Available

Seo

➥🔝 7737669865 🔝▻ mehsana Call-girls in Women Seeking Men 🔝mehsana🔝 Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , India,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At #K09 Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

➥🔝 7737669865 🔝▻ mehsana Call-girls in Women Seeking Men 🔝mehsana🔝 Escorts...

nirzagarg

Yerawada ] Independent Escorts in Pune - Book 8005736733 Call Girls Available 24 Hours In All City Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔$V15 ✔✔To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

Yerawada ] Independent Escorts in Pune - Book 8005736733 Call Girls Available...

SUHANI PANDEY

在线制作约克大学毕业证（yu毕业证）在读证明认证可查【Q/微信741003700】原版仿制全套留学文凭材料（毕业证/成绩单（GPA成绩修改）/文凭学历证书/在读证明、毕业完成信、Offer录取通知书)；（真实可查）教育部学历认证、留信网认证、文凭认证、diploma、certificate、Degree、Transcript（实体公司，专业可靠）。 1:1完美还原海外各大学证书上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700。留学生归国服务中心面向美国英国澳洲加拿大留学生提供以下服务: 一、办理毕业证成绩单（学校原版1：1高仿真制作）二、留信认证（留学回国人员学历认证，网上存档可查，查到后付款）三、教育部学历认证（中国教育部留服中心存档可查,查到后付款）办理流程 1、收集客户办理信息； 2、客户付定金下单、公司确认到账转制作点做电子版； 3、电子版做好发给客户确认、电子版确认好转成品部做成品； 4、成品做好拍照或者视频确认再付余款； 5、快递给客户（国内顺丰，国外DHL）。本公司诚聘美国、加拿大、英国、新西兰、澳洲、法国、德国、新加坡各地代理人员，如果你有业余时间有兴趣就请联系我们校园代理，报酬丰厚。真诚期待您的加盟。请联系本公司学历认证顾问(QQ：741003700 微信：741003700)欢迎咨询！最专业的学历顾问，最有经验的顶尖OP为广大留学生的归国之路保卫护航！

在线制作约克大学毕业证（yu毕业证）在读证明认证可查

ydyuyu

Top Rated Pune Call Girls Daund ⟟ 6297143586 ⟟ Call Me For Genuine Sex Service At Affordable Rate Booking Contact Details WhatsApp Chat: +91-6297143586 pune Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts pune understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide - 01-may-2024(v.n)

Top Rated Pune Call Girls Daund ⟟ 6297143586 ⟟ Call Me For Genuine Sex Servi...

Call Girls in Nagpur High Profile

Microsoft Azure Arc Customer Deck Microsoft

AanSulistiyo

20240509 QFM015 Engineering Leadership Reading List April 2024.pdf

Matthew Sinclair

Russian Call Girls Pune (Adult Only) 8005736733 Escort Service 24x7 Cash Payment Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔$V15 ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

Russian Call Girls Pune (Adult Only) 8005736733 Escort Service 24x7 Cash Pay...

SUHANI PANDEY

Katraj ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready For Sex At Your Doorstep Booking Contact Details WhatsApp Chat: +91-6297143586 pune Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts pune understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide - 02-may-2024(v.n)

Katraj ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready For S...

tanu pandey

Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Independent Escorts service Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔$V15 ✔✔To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Ind...

SUHANI PANDEY

Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R. Why you Choose Us- +91-8264348440 I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls Delhi and Escorts, We Are Located in 3* 4* 5* Hotels. Safe & Secure High Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Delhi High class luxury and premium escorts agency. ★ CALL US High Class Luxury and Premium Escort Service We Provide Well Educated,Royal Class Female,High-Class Escort Service Offering a Top High Class Escort Service In The & Several Nearby All Places of . ★ To Enjoy With Hot and Sexy Girls . ★ We Are Providing :- • 3* 5* 7*Hotels Service • Hygienic Full AC Neat and Clean Rooms Avail. • Daily New Escorts Staff Available • Minimum to Maximum Range Available. Our escort service •BJ (Blowjob) •FK (French kissing) •Kissing with tongue •O-Level (Oral sex) •Long Silky Brown Hair •69 (69 sex) •A-Level (5 start Escort) •Dating

Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R.

soniya singh

Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready For Sex At Your Doorstep Booking Contact Details WhatsApp Chat: +91-6297143586 pune Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts pune understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide - 02-may-2024(v.n)

Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready...

tanu pandey

Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵 (Simran) Call Girls in Chandigarh- 9053900678Escorts Service 50% Off with Cash ON Delivery★ 😋 Chandigarh Call girls ✂️ Service Hello my dear gentleman ❤️ Are you looking for a smart, beautiful Girl ? I will seduce you and intoxicate you from head to toe, with beautiful green eyes, soft pout, curly hair, big natural breasts, and flawless curves. I'm flirtatious, I exude confidence and incredible attractiveness, and I've mastered the art of seduction, which makes me the perfect companion for any occasion that calls for a little depraved play. Unforgettable appointment at your hotel or in my apartment. Call Girls in Chandigarh Chandigarh Call Girl Service call girl Chandigarh Chandigarh call girl escort in Chandigarh Chandigarh escort. Chandigarh escort service, Escort service in Chandigarh, Chandigarh escorts service, Chandigarh escorts agency. I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls and Escorts, We Are Located in 3* 4* 5* Hotels in Chandigarh. Safe & Secure High-Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Chandigarh High-Class luxury and Premium Escorts ServiCe. Hot And Sexy Independent Model Escort Service In Chandigarh Unlimited Enjoy Genuine 100% Profiles And Trusted Door Step Call Girls Feel Free To Call Us Female Service Hot Busty & Sexy Party Girls Available For Complete Enjoyment. We Guarantee Full Satisfaction & In Case Of Any Unhappy Experience, We Would Refund Your Fees, Without Any Questions. Feel Free To Call Us Female Service Provider Hours Open Thanks. ★ CALL US High Class Luxury and Premium Escorts ServiCe We Provide Well Educated, Royal Class Female, High-Class Escorts Offering a Top High-Class Escorts Service In the & Several Nearby All Places Of . ★ Get The High Profile, Bollywood Queens , Well Educated , Good Looking , Full Cooperative Model Services. You Can See Me at My Comfortable Hotels or I Can Visit You In hotel Our Service Available IN All SERVICE, 3/4/5 STAR HOTEL , In Call /Out Call Services.24 hrs , ★ To Enjoy With Hot and Sexy Girls . ★ We Are Providing :- • Models • Vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife • Collage Going Girls. • Travelling Escorts. • Ramp-Models • Foreigner And Many More.. Incall & Outcall Available… • INDEPENDENT GIRLS / HOUSE WIFES ★ I Guarantee You To Have An Unforgettable Experience With Me.A Curvy Body, long Hair and Silky Smooth Skin. She Is an Independent Escorts Model Will Give You More Pleasure & Full Satisfaction. ★OUR BEST SERVICES: – FOR BOOKING CALL Call/WhatsApp ☎+91-9115573837☎ ★ A-Level (5 Star Escort) ★ Strip-Tease ★ BBBJ (Bareback Blowjob) ★ Spending time in my rooms ★ BJ (Blowjob Without a Condom) ★ COF (Come On Face) ★ Completion ★ (Oral To Completion) ★ Special Massage ★ O-Level (Oral s

Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵

Chandigarh Call girls 9053900678 Call girls in Chandigarh

Call Girls Dubai Dubai Call Girls Agency Vip Call Girls Dubai Hot Call Girls Dubai Out Call Girls Dubai Best Call Girls Dubai Indian And Pakistani Call Girls Dubai Independent Call Girls Dubai Dubai Indian Call Girls Agency Class Call Girls In Dubai #First Class Call Girls In Dubai #Full Massage Services Call Girls In Dubai Full Night Call Girls Dubai Hourly Call Girls Dubai Call Girls Deira Dubai Dating Call Girls In Deira Dubai Silicon Oasis Call Girls Dubai International City Call Girls Dubai Al Jaddaf Al Satwa Al Quoz Al Barsha South Media City Discovery Garden Call Girls Dubai Al Jaffiliya,Business Bay,Al Karama,Bur Dubai,Deira,Dubai,Palm Jumeirah,Al Wasl,Trade Centre,Dubai Mall,JBR,JVC,JLT,Discovery Garden Al Nahda Call Girls Dubai Al Qusais Call Girls Dubai Dating Call Girls Dubai Dubai Call Girls Services Provide In Ajman_Dubai_RAK_UMQ_Fujairah_Abu_Dhabi#Indian #Tamil #Kerala #Russian #Philippine #Morocco #Thailand #English Models In Dubai #If You Want Serv#Dubai Pakistani Call Girls Agency #Beautiful Call Girls in Dubai #High ices Just Send Me Text On Whatsapp #

Real Escorts in Al Nahda +971524965298 Dubai Escorts Service

Escorts Call Girls

Dernier (20)