5. What is our problem?
• 개발자 수준 편차가 크다
• 조직 수준은 최하위 개발자 수준의 의해 결정된다
6. Our status
• Lack of Safety code
• Not Readable code
• Lack of Testing (Unit & Integration)
• Heavy oriented to JPA
• Lack of experience to Async/Non-Blocking
Programming
7. Our environments
• 1일 데이터 처리량 : 천만건(2016) -> 10억건(2018)
• MySQL (Aurora) : Replication delay120 sec (max)
• Backend Stack in 2016
• MySQL, RabbitMQ, Hive
• Spring 3, JPA
8. Our Challenges
• Akka Cluster with Lagom Framework (CQRS)
• Move to NoSQL (Cassandra, HBase)
• Migrate RabbitMQ to Kafka
• Adopting Scala (Pilot)
• Akka, Spark …
9. Failure
• Adopting Scala
• Learning Curve (Almost Junior)
• Lack of Tech Leaders
• Try to develop too complex system
• Adopting Kafka
• Simultaneous development (duplicate same error)
• Offset managements
10. To pursue change
• 객관적 시각 유지
• History 및 현실 상황 대한 이해 (기술부채)
• 현 조직의 개발 역량 객관적 평가
• 동기부여 - 필요성 설득보다는 자각할 수 있도록 자극
• 충분한 학습 시간
• 변화 경험 공유
• ASP -> C#, Java -> Scala
11. Prepare
• Code Quality 중요성 자각 -> Code Review
• Enterprise Application Commons
• Patterns 교육, Library 제공
• Enterprise Architecture 학습
• Asynchronous Programming 학습
• Kotlin Language 학습
• Kotlin is language for Android ?
12. Adopting Kotlin
• Safety Code / Readable Code
• Simple grammar (Easy to learn)
• Interoperability between Java and Kotlin
• Extension methods
• Asynchronous Programming by Coroutines
13. Environments 준비
• 개발 Tool 지원 - IntelliJ IDEA
• Static code analysis - ktlint, detekt
• Test code coverage - jacoco (sample)
• Sonarcube - sonar-kotlin plugin
19. Our usages
// Length (Inch)
IN("in", BigDecimal.ONE),
FT("ft", 12.toBigDecimal()),
YD("yd", 36.toBigDecimal()),
// Weight (gram)
MG("mg", 0.001.toBigDecimal()),
G("g", BigDecimal.ONE),
KG("kg", 1000.toBigDecimal()),
T("t", 1_000_000.toBigDecimal()),
OZ("oz", 28.349523.toBigDecimal()),
LB("lb", 453.59237.toBigDecimal()),
•Define Units by Enum
•Not support conversions
•Not support various units
•Not support operations
•Not readable code
20. Features for units
• Readable code like this
• 5.kg() + 400.gram() => 5.4.kg()
• 100.cm() + 2.m() => 2.1.cm()
• Convert various units
• 5.kg().inGram() -> 5000.gram
• 2.5.Kb().inBytes() -> 2500.bytes()
• Convert SI Units <-> US Units
21. Use Kotlin extension methods
fun Double.millimeter(): Length = lengthOf(this, MILLIMETER)
fun Double.centimeter(): Length = lengthOf(this, CENTIMETER)
fun Double.meter(): Length = lengthOf(this)
fun Double.kilometer(): Length = lengthOf(this, KILOMETER)
22. 1. Define Unit factors
enum class LengthUnit(val unitName: String, val factor: Double) {
MILLIMETER("mm", 1.0),
CENTIMETER("cm", 10.0),
METER("m", 1e3),
KILOMETER("km", 1e6);
}
23. Define Unit class
data class Length(val value: Double = 0.0): Comparable<Length>, Serializable {
operator fun plus(other: Length): Length = Length(value + other.value)
operator fun minus(other: Length): Length = Length(value - other.value)
operator fun times(scalar: Int): Length = Length(value * scalar)
operator fun times(scalar: Long): Length = Length(value * scalar)
operator fun times(scalar: Double): Length = Length(value * scalar)
operator fun times(other: Length): Area = Area(value * other.value)
operator fun div(scalar: Int): Length = Length(value / scalar)
operator fun div(scalar: Long): Length = Length(value / scalar)
operator fun div(scalar: Double): Length = Length(value / scalar)
operator fun unaryMinus(): Length = Length(-value)
Override Operators
24. 2. Korean Tokenizer
• 중복상품 Merge 위한 Tokenizer 필요 (명사 위주)
• Twitter 에서 개발한 open-korean-text 를 Customizing
• Scala vs Kotlin 성능 비교
• Kotlin version is 1.5 ~ 3X faster with Coroutines
• 효과
• Full Indexing per Day 부하 감소 : 30%
• Elastic Search 질의 부하 : 80%
25. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)-
트렁크
3928
95
제테스
남성
26. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)-
트렁크
3928
95
제테스
남성
27. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)
L -
트렁크
3928
95
제테스
남성
28. Tokenizer Benchmark
은전한닢 Twitter RATIO
RPS 73.2 429.5 5.87 X
Avg Latency 626 ms 106 ms 5.91 X
Avg CPU Load 90% 55% 35 % Down
Twitter Tokenizer 는 한음절을 분석하지 못하는 단점이 있다
29. Scala Kotlin RATIO
Tokenize 668.620 ms 197.632 ms 3.38 X
Phrase extractor 644.902 ms 212.500 ms 3.13 X
구문 : “동해물과 백두산이 마르고 닳도록“
Benchmark by jmh
Linux Mint 18.2 Intel I7, 32GB, SSD
30. Why Kotlin is faster?
• Scala 의 loop는 느리다 - 아주 느릴 수 있다
• eclipse-collections 사용
• 메모리 절약
• Primitive type collection 지원
• Kotlin Coroutines
• 비동기 / Non-Blocking
31. 3. Kafka Client - Wilson
• 동기
• 안정된 Kafka 사용을 위해 Offset 관리가 필수
• 각 팀별 중복된 Client 구현 및 삽질
• 효과
• Message 중복 / 유실 없음 (Latest strategy)
• Retry / Circuit breaker 지원
• Metrics 를 활용한 Ack 지원
• 전사 확대 적용 중
32. Wilson message flows
Producer ConsumerKafka
Retry
Circuit Breaker
Metrics
Retry
Circuit Breaker
Metrics
Dead letters
Sending box
Redis
MySQL
Couchbase
Dead letters
Received box
Last sent timestamp Kafka Offset Manager
Message Managements
• Metrics
• Recovery / Retry
• Deadletter handling
34. 4. Audit Tool
• 상품 정보 Audit System
• developers : 1 senior, 2 junior developer
• Software stack
• React
• Spring Boot 1.5 on Vitamin Framework
• jOOQ (향후 requery로 변환 예정)
• Pilot 로 시작, 개발자들의 노력으로 정식 시스템으로 승격
36. 5. Creation Pipeline System
• 상품 정보 생성 프로세스 관리 시스템
• Features
• Workflow (Heavy use Kafka)
• Asynchronous / Non-blocking System
• Need High throughput
45. Spring MVC + Cassandra
Spring WebFlux + Cassandra Reactive
출처: Reactive Java Performance Comparison
46. 6. Tagging System
• 상품 정보에서 원하는 특성을 추출하는 시스템
• Matching 과 Refinement 에서 따로 관리
• 범용화 필요 (Configuration, Plugin)
• 기존 Java & Scala 혼용
• 장애 대응에 문제
• High Throughput
48. 7. spring-data-requery (2018)
• RequeryOperations
• Wrap EntityDataStore
• RequeryTransactionManager for TransactionManager
• Support Spring @Transactional
• Better performance than spring-data-jpa
• when exists, paging, not load all entities
https://www.slideshare.net/debop/alternatives-of-jpahibernate
https://www.slideshare.net/debop/requery-overview
https://www.slideshare.net/debop/spring-data-requery
49. spring-data-requery
• Repository built in SQL
• ByPropertyName Auto generation methods
• @Query for Native SQL Query
• Query By Example
• Not Supported
• Association Path (not specified join method)
• Named parameter in @Query (just use `?`)
51. Use @Query in Repository
interface DeclaredQueryRepository extends RequeryRepository<BasicUser, Long> {
@Query("select * from basic_user u where u.email = ?")
BasicUser findByAnnotatedQuery(String email);
@Query("select * from basic_user u where u.email like ?")
List<BasicUser> findAllByEmailMatches(String email);
@Query("select * from basic_user u limit ?")
List<BasicUser> findWithLimits(int limit);
@Query("select * from basic_user u where u.name=? and u.email=? limit 1")
BasicUser findAllBy(String name, String email);
@Query("select u.id, u.name from basic_user u where u.email=?")
List<Tuple> findAllIds(String email);
@Query("select * from basic_user u where u.birthday = ?")
List<BasicUser> findByBirthday(LocalDate birthday);
}
52. Query By Example
BasicUser user = RandomData.randomUser();
user.setName("example");
requeryTemplate.insert(user);
BasicUser exampleUser = new BasicUser();
exampleUser.setName("EXA");
ExampleMatcher matcher = matching()
.withMatcher("name", startsWith().ignoreCase())
.withIgnoreNullValues();
Example<BasicUser> example = Example.of(exampleUser, matcher);
Return<? extends Result<BasicUser>> query = buildQueryByExample(example);
BasicUser foundUser = query.get().firstOrNull();
assertThat(foundUser).isNotNull().isEqualTo(user);
53. Query by Property
List<User> findByFirstnameOrLastname(String firstname, String lastname);
List<User> findByLastnameLikeOrderByFirstnameDesc(String lastname);
List<User> findByLastnameNotLike(String lastname);
List<User> findByLastnameNot(String lastname);
List<User> findByManagerLastname(String name);
List<User> findByColleaguesLastname(String lastname);
List<User> findByLastnameNotNull();
@Query("select u.lastname from SD_User u group by u.lastname")
Page<String> findByLastnameGrouped(Pageable pageable);
long countByLastname(String lastname);
int countUsersByFirstname(String firstname);
boolean existsByLastname(String lastname);
Note: Association Path is not supported
Note: Association Path is not supported
54. Future works
• Coroutines for Async JDBC Operations (working)
• Support Named parameter
• Support `@Param` in spring data
• Requery for SQL on Hadoop
• Apache Phoenix (HBase) (Done)
• Apache Hive, Apache Drill …
55. 8. Rule Engine (2018)
• Embedded
• Support Language
• MVEL (almost like Java spec)
• KotlinScript
• Javascript
• Support Multi Rule combination
• Rule Definition DSL (Kotlin)
• Rule Editor in WEB (in construction)
56. Rule Engine
A rule engine can be viewed as a sophisticated interpreter for if/then
statements, where the statements themselves are known as rules.
57. Rule Definition
val rule = rule {
name = "myRule"
description = "myDescription"
priority = 3
condition {
condition1.evaluate(it)
}
action {
action1.execute(it)
}
action {
action2.execute(it)
}
}
val rule = RuleBuilder()
.name("myRule")
.description("myDescription")
.priority(3)
.whenever(condition)
.then(action1)
.then(action2)
.build()
@Rule(name = "weather rule", description = "if it rains then take an umbrella")
class WeatherRule {
@Condition
fun itRains(@Fact("rain") rain: Boolean): Boolean = rain
@Action
fun takeAnUmbrella() {
println("It rains, take an umbrella")
}
}
58. 9. Wakanda
• Catalog 정보 관리 시스템의 핵심
• Read, Write Side 분리 제공
• 전사 통합에 의해 Write side만 존치
• Fully Reactive / Non-Blocking
62. 반성해야 하는 점
• 기본기 학습 - 닥치고 코딩 (X)
• Java와 차이점 및 Kotlin best practices 검토 필요
• 첫 술에 배부를 수 없다
• 실망하지 말라, Refactoring 은 필수다
• Coroutines 에 대한 학습 및 테스트
• 어차피 비동기는 어렵다.
• Coroutines는 하나의 방법일 뿐이다
63. 안정적 도입을 위한 Tips
• 충분한 학습 기회 & 실습
• Code Quality 중요성 인식
• Upsource 전사 활용 중 (문화가 중요)
• 강력한 동기 부여
• Tech Leader의 지속적인 Leading & Coach
• 성공 사례 만들기 (작은 것부터)
64. 효과
• Safety Code
• Readable Code
• 성능 향상
• Latency 감소, High Throughput 달성
• Asynchronous/Non-Blocking 적용 용이
• 유지보수성 향상 (장애 감소)