TDC2018SP | Trilha Go - Processando analise genetica em background com Go

1. Processando análise genética em
background com Go
Marco Singer
Trilha Go - TDC 2018

2. Um pouco de contexto

7. Monolítico > Microserviços > Binário

8. ~ vsa
vsa stands for *Very Simple Annotator*. This package annotates variants in VCF format
Usage:
vsa [command]
Available Commands:
dx Annotate and classify variants given a VCF file
help Help about any command
server Run vsa server at port 9090
train Prepare features for training or standardization
Flags:
-h, --help help for vsa
Use "vsa [command] --help" for more information about a command.

9. Simplificar a forma de
processamento dos exames

10. Premissas do
VSA-Worker ● Receber os exames que precisamos processar
● Delegar a execução para o nosso binário
● Notificar em caso de erro no processamento

11. Tecnologias
● RabbitMQ
● Apache Kafka
● Amazon SQS
● Sidekiq

12. Tecnologias
● RabbitMQ (exemplo)
● Apache Kafka
● Amazon SQS
● Sidekiq

13. Tecnologias
● RabbitMQ (exemplo)
● Apache Kafka (canhão)
● Amazon SQS
● Sidekiq

14. Tecnologias
● RabbitMQ (exemplo)
● Apache Kafka (canhão)
● Amazon SQS (não funciona offline)
● Sidekiq

15. Tecnologias
● RabbitMQ (exemplo)
● Apache Kafka (canhão)
● Amazon SQS (não funciona offline)
● Sidekiq (mais uma linguagem na stack)

16. Vamos no simples

17. Faktory
https://github.com/contribsys/faktory
● Criador do Sidekiq
● Feito em Go
● Clientes em Go e Ruby "oficiais"
● Usa RocksDB para persistência
● Vem com um dashboard built-in
● Retry automático
● Pode agendar execução de um job

18. Server

19. Rodando Faktory
docker pull contribsys/faktory
docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory

20. Rodando Faktory
docker pull contribsys/faktory
docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory
comunicação cliente / servidor

21. Rodando Faktory
docker pull contribsys/faktory
docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory
porta da interface web

22. http://localhost:7420

23. Criando um job

24. Payload
Estrutura de um job
{
"jid": "123861239abnadsa",
"jobtype": "SomeName",
"args": [1, 2, "hello"]
}

25. Payload
Estrutura de um job
{
"jid": "123861239abnadsa",
"jobtype": "SomeName",
"args": [1, 2, "hello"]
}
ID único do job

26. Payload
Estrutura de um job
{
"jid": "123861239abnadsa",
"jobtype": "SomeName",
"args": [1, 2, "hello"]
}
nome da função

27. Payload
Estrutura de um job
{
"jid": "123861239abnadsa",
"jobtype": "SomeName",
"args": [1, 2, "hello"]
}
argumentos da função

28. Executando VSA através do Faktory

29. ~ vsa dx --help
Annotate and classify variants given a VCF file.
Usage:
vsa dx [flags]
Flags:
--assembly string Supported assemblies: GRCh37 or GRCh38.
--vcf string VCF file address.

30. Criando um job
go get -u github.com/contribsys/faktory/client
import (
faktory "github.com/contribsys/faktory/client"
)
func send(){
client, err := faktory.Open()
if err != nil {
// tratamento de erro
}
defer client.Close()
job := faktory.NewJob("dx", "GRCh37", "/path/to/file")
if err := client.Push(job); err != nil {
// tratamento de erro
}
}

31. faktory.NewJob("dx", "GRCh37", "/path/to/file")

32. faktory.NewJob("dx", "GRCh37", "/path/to/file")
payload argspayload jobType

33. ~ vsa dx --help
Annotate and classify variants given a VCF file.
Usage:
vsa dx [flags]
Flags:
--assembly string Supported assemblies: GRCh37 or GRCh38.
--vcf string VCF file address.

34. client.Push(job)

35. http://localhost:7420

36. http://localhost:7420

37. Processando um job

38. go get -u github.com/contribsys/faktory_worker_go

39. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)

40. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
}

41. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
svc.Register("dx", dxWorker)
}

42. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
svc.Register("dx", dxWorker)
} payload jobType

43. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
svc.Register("dx", dxWorker)
}
?

44. // github.com/contribsys/faktory_worker_go
func (mgr *Manager) Register(name string, fn Perform) {
}
type Perform func(ctx Context, args ...interface{}) error

45. func dxWorker(ctx worker.Context, args ...interface{}) error {
}

46. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
svc.Register("dx", dxWorker)
}

47. package worker
import (
worker "github.com/contribsys/faktory_worker_go"
)
func Run() {
svc := worker.NewManager()
svc.Register("dx", dxWorker)
svc.Run()
}

48. Juntando todos os conceitos

49. Premissas do
VSA-Worker ● Receber os exames que precisamos processar
● Delegar a execução para o nosso binário
● Notificar em caso de erro no processamento

50. main.go
package main
import (
// ...
)
func main() {
// running workers in background
go func() {
worker.Run()
}()
// configure API handlers
svc := api.New()
http.Handle("/", svc.Handler())
logrus.WithField("port", config.Port).Info("starting server")
if err := http.ListenAndServe(":"+config.Port, nil); err != nil {
logrus.WithError(err).Fatalf("something went wrong")
}
}

51. main.go
package main
import (
// ...
)
func main() {
// running workers in background
go func() {
worker.Run()
}()
// configure API handlers
svc := api.New()
http.Handle("/", svc.Handler())
logrus.WithField("port", config.Port).Info("starting server")
if err := http.ListenAndServe(":"+config.Port, nil); err != nil {
logrus.WithError(err).Fatalf("something went wrong")
}
}

52. main.go
package main
import (
// ...
)
func main() {
// running workers in background
go func() {
worker.Run()
}()
// configure API handlers
svc := api.New()
http.Handle("/", svc.Handler())
logrus.WithField("port", config.Port).Info("starting server")
if err := http.ListenAndServe(":"+config.Port, nil); err != nil {
logrus.WithError(err).Fatalf("something went wrong")
}
}

53. api.go
package main
import (
// ...
)
func (s Service) Handler() *mux.Router {
r := mux.NewRouter()
r.PathPrefix("/api")
r.HandleFunc("/v1/import", s.importHandler).Methods("POST")
return r
}

54. api.go
package main
import (
// ...
)
func (s Service) Handler() *mux.Router {
r := mux.NewRouter()
r.PathPrefix("/api")
r.HandleFunc("/v1/import", s.importHandler).Methods("POST")
return r
}

55. api.go
type ImportInput struct {
Assembly string `json:"assembly,omitempty"`
Vcf string `json:"vcf,omitempty"`
}
func importHandler(w http.ResponseWriter, r *http.Request) {
var input ImportInput
json.NewDecoder(r.Body).Decode(&input)
client, _ := faktory.Open()
defer client.Close()
job := faktory.NewJob("dx", input.Assembly, input.Vcf)
client.Push(job)
w.WriteHeader(http.StatusOK)
}

56. api.go
type ImportInput struct {
Assembly string `json:"assembly,omitempty"`
Vcf string `json:"vcf,omitempty"`
}
func importHandler(w http.ResponseWriter, r *http.Request) {
var input ImportInput
json.NewDecoder(r.Body).Decode(&input)
client, _ := faktory.Open()
defer client.Close()
job := faktory.NewJob("dx", input.Assembly, input.Vcf)
client.Push(job)
w.WriteHeader(http.StatusOK)
}

57. api.go
type ImportInput struct {
Assembly string `json:"assembly,omitempty"`
Vcf string `json:"vcf,omitempty"`
}
func importHandler(w http.ResponseWriter, r *http.Request) {
var input ImportInput
json.NewDecoder(r.Body).Decode(&input)
client, _ := faktory.Open()
defer client.Close()
job := faktory.NewJob("dx", input.Assembly, input.Vcf)
client.Push(job)
w.WriteHeader(http.StatusOK)
}

58. main.go
package main
import (
// ...
)
func main() {
// running workers in background
go func() {
worker.Run()
}()
// configure API handlers
svc := api.New()
http.Handle("/", svc.Handler())
logrus.WithField("port", config.Port).Info("starting server")
if err := http.ListenAndServe(":"+config.Port, nil); err != nil {
logrus.WithError(err).Fatalf("something went wrong")
}
}

59. worker.go
package worker
import (
//
)
func Run() {
logrus.Info("starting workers in background mode")
svc := worker.NewManager()
logrus.WithField("jobType", "dx").Info("register job")
svc.Register("dx", dxWorker)
logrus.Info("waiting for jobs")
svc.Run()
}

60. func dxWorker(ctx worker.Context, args ...interface{}) error {
var (
assembly = args[0].(string)
vcf = args[1].(string)
)
cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf)
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
rerr, _ := ioutil.ReadAll(stderr)
if err := cmd.Wait(); err != nil {
return fmt.Errorf("%s", rerr)
}
return nil
}

61. func dxWorker(ctx worker.Context, args ...interface{}) error {
var (
assembly = args[0].(string)
vcf = args[1].(string)
)
cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf)
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
rerr, _ := ioutil.ReadAll(stderr)
if err := cmd.Wait(); err != nil {
return fmt.Errorf("%s", rerr)
}
return nil
}

62. func dxWorker(ctx worker.Context, args ...interface{}) error {
var (
assembly = args[0].(string)
vcf = args[1].(string)
)
cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf)
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
rerr, _ := ioutil.ReadAll(stderr)
if err := cmd.Wait(); err != nil {
return fmt.Errorf("%s", rerr)
}
return nil
}

63. func dxWorker(ctx worker.Context, args ...interface{}) error {
var (
assembly = args[0].(string)
vcf = args[1].(string)
)
cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf)
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
rerr, _ := ioutil.ReadAll(stderr)
if err := cmd.Wait(); err != nil {
return fmt.Errorf("%s", rerr)
}
return nil
}

64. http://localhost:7420

65. func dxWorker(ctx worker.Context, args ...interface{}) error {
var (
assembly = args[0].(string)
vcf = args[1].(string)
)
cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf)
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
rerr, _ := ioutil.ReadAll(stderr)
if err := cmd.Wait(); err != nil {
return fmt.Errorf("%s", rerr)
}
return nil
}

66. http://localhost:7420

67. Exemplo de call
// payload.json
{
"assembly":"GRCh37",
"vcf":"/home/marcosinger/files/FXM281-001.vcf.gz"
}

68. Exemplo de call
curl -XPOST
-H "Content-Type: application/json"
--data @payload.json
http://host:port/api/v1/import

69. 15 exames/hora
5 máquinas c4.4xlarge
Amazon SQS
MongoDB
Elasticsearch

70. 200 exames/hora
1 máquina t2.large
Faktory
MongoDB

71. Obrigado!
Marco Singer
@mahsinger
linkedin.com/in/marcosinger
github.com/marcosinger