11. Get Open Data Sources on the
internet browser
– Restful API
• https://ci.taiwan.gov.tw/dsp/environmental.aspx
• https://mis.twse.com.tw/stock/api/getStockInfo.jsp?ex_ch=tse_23
30.tw&json=1&delay=0
• https://api.dictionaryapi.dev/api/v2/entries/en/similar
• https://sta.ci.taiwan.gov.tw/STA_AirQuality_v2/v1.0/Datastreams?
$expand=Thing,Observations($orderby=phenomenonTime%20des
c;$top=1)&$filter=name%20eq%20%27PM2.5%27%20and%20Thing/
properties/authority%20eq%20%27%E8%A1%8C%E6%94%BF%E9%99
%A2%E7%92%B0%E5%A2%83%E4%BF%9D%E8%AD%B7%E7%BD%B2%
27%20and%20substringof(%27%E7%A9%BA%E6%B0%A3%E5%93%81
%E8%B3%AA%E6%B8%AC%E7%AB%99%27,Thing/name)&$count=tr
ue
13. Get Open Data Sources on
Advanced Rest Client
• Advanced Rest Client run on chrome
• https://chrome.google.com/webstore/detail/ad
vanced-rest-
client/hgmloofddffdnphfgcellkdfbfbjeloo?page=
1&hl=zh-tw&itemlang=sv
15. Basic knowledge of
CLI(commandline)
• Install needed packages whenever login in the
google colab
• apt-get install jq
• cd : Change directory
• pwd : Show the current directory
• cd .. : Leave the current directory
• cd [Directory name] : Enter the [Directory name]
• wget [URL] : Download the file from the URL
• curl [URL] : Powerfully download the anything from
the URL
16. Basic knowledge of google
colab CLI(commandline)
• Install needed packages whenever login in the google colab
• !apt-get install jq
• % : Use % if the command needs to be run on the specific
notebook
• ! : Use ! if the command is run in a separate shell.
• %cd : Change directory
• !pwd : Show the current directory
• !cd .. : Leave the current directory
• !cd [Directory name] : Enter the [Directory name]
• !wget [URL] : Download the file from the URL
• !curl [URL] : Powerfully download the anything from the URL
17. Retrieve Open data on Linux CLI
• curl -s
'https://mis.twse.com.tw/stock/api/getStockInfo.jsp?ex_ch=tse_23
30.tw&json=1&delay=0' | jq '.["msgArray"][0]["h"] '
• curl -s 'https://api.dictionaryapi.dev/api/v2/entries/en/similar'
• curl -s
'https://sta.ci.taiwan.gov.tw/STA_AirQuality_v2/v1.0/Datastreams?
$expand=Thing,Observations($orderby=phenomenonTime%20des
c;$top=1)&$filter=name%20eq%20%27PM2.5%27%20and%20Thing/
properties/authority%20eq%20%27%E8%A1%8C%E6%94%BF%E9%99
%A2%E7%92%B0%E5%A2%83%E4%BF%9D%E8%AD%B7%E7%BD%B2%
27%20and%20substringof(%27%E7%A9%BA%E6%B0%A3%E5%93%81
%E8%B3%AA%E6%B8%AC%E7%AB%99%27,Thing/name)&$count=tr
ue' | jq '.["value"][0]["Observations"][0]["result"] '
18. API Key
• Proprietary Restful API needs to register a API Key
• https://newsapi.org/
• Registered Newsapi API Key
• 1ac7db45547b4a6d928a6dedb7296d4e
• Get data from any browser :
• https://newsapi.org/v2/top-
headlines?country=us&apiKey=1ac7db45547b4a6d92
8a6dedb7296d4e
• Get data from Colab CLI:
• !curl -s 'https://newsapi.org/v2/top-
headlines?country=us&apiKey=1ac7db45547b4a6d92
8a6dedb7296d4e' | jq '.["articles"][]["content"]?'
19. Find the whole headlines
with Trump in all media
• Step 1 : Read the available news contents in the NewsApi
• curl ‐s 'https://newsapi.org/v2/top‐
headlines?country=us&apiKey=1ac7db45547b4a6d928a6dedb7296d4e'
• Step 2 : List every article of the whole media from the above
JSON data
• jq '.["articles"][]["content"]?'
• Step 3 : Filter out the articles relevant to “Trump”
• grep -i 'Trump'
• Process in batch
• curl ‐s 'https://newsapi.org/v2/top‐
headlines?country=us&apiKey=1ac7db45547b4a6d928a6dedb7296
d4e' | jq '.["articles"][]["content"]?' | grep ‐i 'Trump'
24. Regular Expression
• What is regular expression?
• An expression of a pattern in a string using
special characters and words.
• When and where we use it?
• Regular expression is used to parse an output
from a software , for example, BLAST, or used
to extract information you need from a text file.
When a string | line matches the pattern, it is
extracted. Therefore, it is extremely useful.
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
25. Download the materials
• Move to AIEMBA directory
• Move to user’s home directory
$ cd ~
• Create the AIEMBA directory
$ mkdir AIEMBA
• Download the materials
$ wget
"https://drive.google.com/uc?export=download&id=1Rq3
h3rdRYLoEJsAei_0H7VnXI0XjfYI-" -O AIEMBA_20201111.zip
• Extract the zip file
• unzip AIEMBA_20201111.zip
26. What Is a Regular Expression?
• A regular expression (regex) describes a set of possible
input strings.
• Regular expressions descend from a fundamental concept in
Computer Science called finite automata theory
• Regular expressions are endemic to Unix
– vi, ed, sed, and emacs
– awk, tcl, perl and Python
– grep, egrep, fgrep
– compilers
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
27. Regular Expressions
• The simplest regular expressions are a
string of literal characters to match.
• The string matches the regular expression if
it contains the substring.
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
28. grep Family
• Syntax
grep [-ilnv] [-e expression] [filename]
egrep [-ilnv] [-e expression] [-f filename] [expression]
[filename]
– -i Ignore case
– -l List only filenames containing matching lines
– -n Precede each matching line with its line number
– -v Negate matches
– -e expression
– -f filename
Specify expression as option
Take the regular expression (egrep) or
a list of strings (fgrep) from filename
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
29. Test document
"Open Source" is a good mechanism to develop programs.
apple is my favorite food.
Football game is not use feet only.
this dress doesn't fit me.
However, this dress is about $ 3183 dollars.
GNU is free air not free beer.
Her hair is very beauty.
I can’t finish the test.
Oh! The soup taste good.
motorcycle is cheap than car.
This window is clear.
the symbol '*' is represented as start.
Oh! My god!
The gd software is a library for drafting programs.
You are the best is mean you are the no. 1.
The world is the same with "glad".
I like dog.
google is the best tools for search keyword.
goooooogle yes!
go! go! Let's go.
# I am VBird
CLI $ grep 'good' regular_express.txt
Print out the line with “good” in the regular_express.txt
30. UNIX Tools rocks.
match
UNIX Tools sucks.
match
UNIX Tools is ock.
no match
regular expression c k s
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
31. Keyword matching
Ex : Find the lines with 'the' in the file
CLI $ grep ‐n 'the' regular_express.txt
8:I can't finish the test.
12:the symbol '*' is represented as start.
15:You are the best is mean you are the no. 1.
16:The world is the same with "glad".
18:google is the best tools for search keyword.
32. Case insensitive
Ex : Find the lines with case insensitive ‘the’ in the file
CLI $ grep ‐in 'the' regular_express.txt
8:I can't finish the test.
9:Oh! The soup taste good.
12:the symbol '*' is represented as start.
14:The gd software is a library for drafting programs.
15:You are the best is mean you are the no. 1.
16:The world is the same with "glad".
18:google is the best tools for search keyword.
33. Regular Expressions
• A regular expression can match a string in
more than one place.
Scrapple from the apple.
match 1 match 2
regular expression a p p l e
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
34. Character Classes
• Character classes [] can be used to match
any specific set of characters.
beat a brat on a boat
match 1 match 2
regular expression b [eor] a t
match 3
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
35. Character Classes
• Ex:
• t[ae]st == tast OR test
≠ taest
CLI $ grep ‐n ‘t[ae]st’ regular_express.txt
8:I can't finish the test.
9:Oh! The soup taste good.
36. Find the line with ‘oo’
• Ex:
CLI $ grep ‐n 'oo' regular_express.txt
1:"Open Source" is a good mechanism to develop programs.
2:apple is my favorite food.
3:Football game is not use feet only.
9:Oh! The soup taste good.
18:google is the best tools for search keyword.
19:goooooogle yes!
37. Negated Character
Classes
• Character classes can be negated with the
[^] syntax.
beat a brat on a boat
match
regular expression b [^eo] a t
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
38. Negative lookahead
•Find the lines without g in front of oo.
•不想要 oo 前面有 g
• Ex:
CLI $ grep ‐n '[^g]oo' regular_express.txt
2:apple is my favorite food.
3:Football game is not use feet only.
18:google is the best tools for search keyword.
19:goooooogle
39. More About Character Classes
– [aeiou] will match any of the characters a, e, i, o,
or u
– [kK]orn will match korn or Korn
• Ranges can also be specified in character classes
– [1-9] is the same as [123456789]
– [abcde] is equivalent to [a-e]
– You can also combine multiple ranges
• [abcde123456789] is equivalent to [a-e1-9]
– Note that the - character has a special meaning in a
character class but only if it is used within a range,
[-123] would match the characters -, 1, 2, or 3
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
40. Named Character Classes
• Commonly used character classes can be
referred to by name (alpha, lower, upper,
alnum, digit, punct, cntrl)
• Syntax [:name:]
– [a-zA-Z]
– [a-zA-Z0-9]
– [45a-z]
[[:alpha:]]
[[:alnum:]]
[45[:lower:]]
• Important for portability across languages
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
41. Find a span
•The digits inside the brackets can be any
numbers or span of numbers from 0 to 9.
• Ex:
CLI $ grep ‐n '[0‐9]' regular_express.txt
5:However, this dress is about $ 3183 dollars.
15:You are the best is mean you are the no. 1.
42. Negative matching
•Find the ASCII characters which is not in the
range from a to z
• Ex:
CLI $ grep ‐n '[^a‐z]oo' regular_express.txt
3:Football game is not use feet only.
43. Anchors
• Anchors are used to match at the beginning or end
of a line (or both).
• ^ means beginning of the line
• $ means end of the line
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
• Ex:
CLI $ grep ‐n '^the' regular_express.txt
12:the symbol '*' is represented as start.
44. beat a brat on a boat
regular expression ^ b [eor] a t
match
regular expression b [eor] a t $
beat a brat on a boat
match
^$
^word$
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
45. Define the beginning
• Ex:
•Find the lowercase characters which exists in the
beginning of the line
CLI $ grep ‐n '^[a‐z]' regular_express.txt
2:apple is my favorite food.
4:this dress doesn't fit me.
10:motorcycle is cheap than car.
12:the symbol '*' is represented as start.
18:google is the best tools for search keyword.
19:goooooogle yes!
46. Complex regex pattern
CLI $ grep ‐n '^[^a‐zA‐Z]' regular_express.txt
1:"Open Source" is a good mechanism to develop programs.
20:# I am VBird
47. Match the special characters
• ! $ % ^ & * ( ) _ + | ~ - = ` { } [ ] : “ ; ‘ < > ? , . /
• Slash symbol ‘’ can escape the function of the special
characters
CLI $ grep ‐n '.$' regular_express.txt
1:"Open Source" is a good mechanism to develop programs.
2:apple is my favorite food.
3:Football game is not use feet only.
4:this dress doesn't fit me.
10:motorcycle is cheap than car.
11:This window is clear.
12:the symbol '*' is represented as start.
15:You are the best is mean you are the no. 1.
16:The world is the same with "glad".
17:I like dog.
18:google is the best tools for search keyword.
48. Regular Expressions
• The . regular expression can be used to
match any character.
For me to poop on.
match 1 match 2
regular expression o .
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
49. Repetition
• The * is used to define zero or more
occurrences of the single regular expression
preceding it.
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
50. I got mail, yay!
match
regular expression y a * y
For me to poop on.
regular expression o a * o
match
.*
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
51. Match length
Scrapple from the apple.
no
yes
regular expression a . * e
• A match will be the longest string that
satisfies the regular expression.
no
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
52. Repetition Ranges
• Ranges can also be specified
– { } notation can specify a range of repetitions
for the immediately preceding regex
– {n} means exactly n occurrences
– {n,} means at least n occurrences
– {n,m} means at least n occurrences but no
more than m occurrences
• Example:
– .{0,} same as .*
– a{2,} same as aaa*
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
53. Colab grex {} bug
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
CLI in Colab: !grep ‐Eio '[a‐z]{{2,}}' regular_express.txt
• {} should be repeated as {{ }}
• Ranges can also be specified
– {{}}notation can specify a range of repetitions for the
immediately preceding regex
– {{n}}means exactly n occurrences
– {{n,}}means at least n occurrences
– {{n,m}}means at least n occurrences but no more than m
occurrences
CLI $ grep ‐Eio '[a‐z]{2,}' regular_express.txt
54. Subexpressions
• If you want to group part of an expression so that
* or { } applies to more than just the
previous character, use ( ) notation
• Subexpresssions are treated like a single character
– a* matches 0 or more occurrences of a
– abc* matches ab, abc, abcc, abccc, …
– (abc)* matches abc, abcabc,
abcabcabc, …
– (abc){2,3} matches abcabc or
abcabcabc
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
55. 找出 g 後面接 2 到 5 個 o
[root@test root]# grep -n 'go{2,5}g' regular_express.txt
18:google is the best tools for search keyword.
56. .* 是什麼
[root@test root]# grep -n 'g.*g' regular_express.txt
1:"Open Source" is a good mechanism to develop programs.
14:The gd software is a library for drafting programs.
18:google is the best tools for search keyword.
19:goooooogle yes!
57. Escaping Special Characters
• Even though we are single quoting our regexs so the shell won’t
interpret the special characters, some characters are special to grep (eg
* and .)
• To get literal characters, we escape the character with a (backslash)
• Suppose we want to search for the character sequence
• a*b*
• –Unless we do something special, this will match zero or more ‘a’s followed by
zero or more ‘b’s, not what we want
• – a*b* will fix this - now the asterisks are treated as regular characters
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
58. Egrep: Alternation
• Regex also provides an alternation character | for matching
one or another subexpression
– (T|Fl)an will match ‘Tan’ or ‘Flan’
– ^(From|Subject): will match the From and Subject lines
of a typical email message
• It matches a beginning of line followed by either the characters
‘From’ or ‘Subject’ followed by a ‘:’
• Subexpressions are used to limit the scope of the alternation
– At(ten|nine)tion then matches “Attention” or “Atninetion”,
not “Atten” or “ninetion” as would happen without the parenthesis -
Atten|ninetion
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
59. | 字元
• 用或( or )的方式找出數個字串
• 範例：egrep -n 'gd|good' regular_express.txt
搜尋 gd 或 good 這兩個字串，注意，是『或』！ 所以，
第 1,9,14 這三行都可以被列印出來喔！那如果還想要找出
dog 呢？就這樣
CLI $ egrep ‐n 'gd|good|dog' regular_express.txt
60. () 字元
• 找出『群組』字串
•
搜尋 (glad) 或 (good) 這兩個字串，因為 g 與 d 是重複
的，所以， 我就可以將 la 與 oo 列於 ( ) 當中，並以 | 來
分隔開來，就可以啦！
此外，這個功能還可以用來作為『多個重複群組』的判別
喔！舉例來說：
echo 'AxyzxyzxyzxyzC' | egrep 'A(xyz)+C'
上面的例子當中，意思是說，我要找開頭是 A 結尾是 C ，
中間有一個以上的 "xyz" 字串的意思～
CLI $ egrep ‐n 'g(la|oo)d' regular_express.txt
61. Egrep: Repetition
Shorthands
• The * (star) has already been seen to specify zero
or more occurrences of the immediately preceding
character
• + (plus) means “one or more”
abc+d will match ‘abcd’, ‘abccd’, or ‘abccccccd’ but
will not match ‘abd’
Equivalent to {1,}
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
63. Egrep: Repetition
Shorthands cont
• The ‘?’ (question mark) specifies an optional character,
the single character that immediately precedes it
July? will match ‘Jul’ or ‘July’
Equivalent to {0,1}
Also equivalent to (Jul|July)
•
•
The *, ?, and + are known as quantifiers because they
specify the quantity of a match
Quantifiers can also be used with subexpressions
– (a*c)+ will match ‘c’, ‘ac’, ‘aac’ or ‘aacaacac’ but will not
match ‘a’ or a blank line
[Ref]:https://www.slideshare.net/vtriqn/regular-expressions-grep-and-egrep
66. SWI-Prolog
• SWI-Prolog is a good, standard Prolog for Windows and Linux
• Can be installed on Macintosh with a little more effort
(requires X11 and Mac developer tools)
• It's licensed under GPL, therefore free
• Downloadable from:
http://www.swi‐prolog.org/
66
67. Syllogisms
• “Prolog” is all about programming in logic.
• Aristotle described syllogisms 2300 years ago
• Sample syllogism:
• Socrates is a man.
• All men are mortal.
• Therefore, Socrates is mortal.
• Syllogisms are a form of logic. Can Prolog do them?
• Note: If a word or term is in red, you should learn
and remember its meaning
67
68. Forward and backward
reasoning
• A syllogism gives two premises and a conclusion
• Forward reasoning: Given some premises, ask “What can we
conclude?”
• Forward reasoning is inefficient when you are trying to get a particular
conclusion
• Backward reasoning: Given some premises and a
conjectured conclusion, try to derive the conclusion from the
premises
• You start from the conclusion and try to work backward to prove it
• You use Prolog by asking it specific questions
• This is backward reasoning -- from (potential) conclusions to facts 68
70. Facts, rules, and queries
• Fact: Socrates is a man.
• man(socrates).
• Rule: All men are mortal.
• mortal(X) :‐ man(X).
• Query: Is Socrates mortal?
• mortal(socrates).
• Queries have the same form as facts
70
71. Variables and atoms
• Variables begin with a capital letter:
X, Socrates, _result
• A variable can have a value
• An atom is a value; it just stands for itself
• Atoms do not begin with a capital letter:
x, socrates
71
72. Running Prolog I
• Create your “database” (program) in any editor
• Save it as text only, with a .pl extension
• If you have Perl installed, you may have to use
the .pro extension instead
• Google swi prolog file extension for
instructions
• Here's the complete program:
72
man(socrates).
mortal(X) :‐ man(X).
73. Running Prolog II
• SWI-Prolog is interpreted and completely interactive
• You may be able to run your program by double-
clicking your .pl file
• Here are two ways you can run the interpreter:
• Double-click on the swipl file, or
• If your PATH is set correctly, enter swipl at the command line
• At the ?‐ prompt in the interpreter, enter:
• ?‐ consult('Complete path to your .pl file').
• Then, ask your question at the prompt:
• ?‐ mortal(socrates).
• Prolog responds:
• true. 73
74. Prolog is a theorem prover
• Prolog’s true. means “I can prove it”
• Prolog’s false. really means “I can’t prove it”
• It does not mean “I can prove it is untrue.”
• ?‐ mortal(plato).
false.
• This is the closed world assumption: the Prolog program knows
everything it needs to know
• Prolog supplies values for variables when it can
• ?‐ mortal(X).
X = socrates 74
75. Structures
• A structure consists of a name and zero or more arguments.
• Omit the parentheses if there are no arguments
• Example structures:
• sunshine
• man(socrates)
• path(garden, south, sundial)
75
76. Base Clauses
• A base clause is just a structure, terminated with a period.
• A base clause represents a simple fact.
• Example base clauses:
• debug_on.
• loves(john, mary).
• loves(mary, bill).
76
77. Nonbase Clauses
• A nonbase clause is a structure, a turnstile :- (meaning “if”),
and a list of structures.
• Example nonbase clauses:
• mortal(X) :‐ man(X).
• mortal(X) :‐ woman(X).
• happy(X) :‐ healthy(X), wealthy(X), wise(X).
• The comma between structures means “and”
• “X is happy if X is healthy, wealthy, and wise.”
77
78. Predicates
• A predicate is a collection of clauses with the same
functor (name) and arity (number of arguments).
• loves(john, mary).
loves(mary, bill).
loves(chuck, X) :‐ female(X), rich(X).
• The scope of a variable (such as X) is the single
clause in which it occurs.
78
79. Programs
• In Prolog, a program is just a collection of predicates.
• Predicates can be in any order.
• Clauses within a predicate are used in the order in
which they occur.
79
80. Atoms
• You can make an atom containing any characters at all by
enclosing it in single quotes:
• 'C:My Documentsexamples.pl'
• If you use double quotes, you will get a list of ASCII values
• ?‐ X = "Hello".
X = [72, 101, 108, 108, 111].
• You probably don’t want this!
• In a quoted atom, a single quote must be doubled or backslashed:
• 'Can''t, or won't?'
• Backslashes in file names must also be doubled:
• 'C:My Documentsexamples.pl'
• Better yet, use forward slashes in paths; every OS, including Windows,
understands this 80
81. Common problems
• Capitalization is meaningful!
• No space is allowed between a functor and its
argument list:
man(socrates), not man (socrates).
• Double quotes indicate a list of ASCII character
values, not a string
• Don’t forget the period! (But if you do, you can put it
on the next line.)
81
83. Backtracking and Beads
• Each Prolog call is like a “bead” in a string of
beads:
• Each structure has four ports: call, exit, redo, fail
• Exit ports connect to call ports;
fail ports connect to redo ports
83
call
fail
exit
redo
86. Readings
• A clause can be read declaratively (as a statement of fact)
or procedurally (as a list of things to try to do)
• loves(chuck, X) :‐ female(X), rich(X).
• Declarative reading: Chuck loves X if X is female and rich.
• Approximate procedural reading:
To find an X that Chuck loves:
• First try to find a female X (fail and backtrack if you can’t)
• Given a particular value for X, try to show that X is rich (fail
and backtrack if you can’t)
• Declarative readings are almost always preferred. 86
90. Lots of solution available
Data
Source
Data
Collectio
n/Ingesti
on
Stream
Data
Processing
Data
Storage
Data
Presentatio
n
/Analysis
Communicati
on
Protocol
Communication Data format:
JSON, CSV, BSON, YAML, XML
107. Foundations of Computer
Science
• http://infolab.stanford.edu/~ullman/focs.html
• Chapter 1 Computer Science: The Mechanization of Abstraction
• Chapter 2 Iteration, Induction, and Recursion
• Chapter 3 The Running Time of Programs
• Chapter 4 Combinatorics and Probability
• Chapter 5 The Tree Data Model
• Chapter 6 The List Data Model
• Chapter 7 The Set Data Model
• Chapter 8 The Relational Data Model
• Chapter 9 The Graph Data Model
• Chapter 10 Patterns, Automata, and Regular Expressions
• Chapter 11 Recursive Description of Patterns
• Chapter 12 Propositional Logic
• Chapter 13 Using Logic to Design Computer Components
• Chapter 14 Predicate Logic
108. Download the materials on
20201202.zip
• Move to AIEMBA directory
• Move to user’s home directory
$ cd ~
• Download the materials
$ wget
"https://drive.google.com/uc?export=download&id=1izf2QN
xLIH-gx95h7ecQNgtlNiakW7Xw" -O AIEMBA_20201202.zip
• Extract the zip file
• unzip AIEMBA_20201202.zip
111. Tutorials in Github
[Book / github / Data /Beginner ]
https://github.com/jakevdp/PythonDataScienceHandbo
ok
[github / Data /Beginner ]
https://github.com/Pierian-Data/Complete-Python-3-
Bootcamp
► [ Book / github / Data / Math ] Data science from scratch
► [ Book / github / Finance ] Python for finance
► [Collections] https://iter01.com/158569.html
112. Data science from scratch
► https://github.com/joelgrus/data-science-from-scratch
1. Introduction
2. A Crash Course in Python
3. Visualizing Data
4. Linear Algebra
5. Statistics
6. Probability
7. Hypothesis and Inference
8. Gradient Descent
9. Getting Data
10. Working With Data
11. Machine Learning
12. k-Nearest Neighbors
13. Naive Bayes
14. Simple Linear Regression
15. Multiple Regression
16. Logistic Regression
17. Decision Trees
18. Neural Networks
19. [Deep Learning]
20. Clustering
21. Natural Language Processing
22. Network Analysis
23. Recommender Systems
24. Databases and SQL
25. MapReduce
113. 課程比重
Case 1 Introduction to python Python 速成 VVVVV
Case 2 Visualizing Data 數據視覺化 VVV
Case 3 Linear Algebra 線性代數 V
Case 4 Statistics 統計學 V
Case 5 Probability 機率 V
Case 6 Hypothesis and Inference 假設與推論 V
Case 7 Gradient Descent 梯度遞減 V
Case 8 Getting Data 取得數據資料 VVV
Case 9 Working With Data 處理數據資料 VVV
Case 10 Machine Learning 機器學習 VV
Case 11 k-Nearest Neighbors k最近鄰 V
Case 12 Naive Bayes 單純貝氏 V
Case 13 Simple Linear Regression 簡單線性迴歸 VVV
Case 14 Multiple Regression 多元迴歸 VVV
Case 15 Logistic Regression 邏輯迴歸 VVV
Case 16 Decision Trees 決策樹 VVV
Case 17 Neural Networks 神經網路 V
Case 18 [Deep Learning] 深度學習 V
Case 19 Clustering 集群
Case 20 Natural Language Processing 自然語言處理 VV
Case 21 Network Analysis 網路分析 V
Case 22 Recommender Systems 推薦系統 V
Case 23 Databases and SQL 資料庫與SQL VV
Case 24 MapReduce MapReduce
Case study :
Data science
114. Knowledge System of Data Science
• {Atom} Data type/set manipulation ; Data collection
• {Relation/Propositions; Predicate Logic/Rule/Calculation }
• Linear algebra [scikit-learn] [scipy-lectures]
• Statistics [scipy]
• Data processing ; [scipy]
• Data transformation [scipy]
• FFT; DFT[scipy]
• Image [scipy-lectures]
• { Modal ; Probabilistic logic }
• Data visualization
• Probability
• { Modeling }
• Decision tree
• PCA
• linear
• Clustering biclustering
115. Python for finance
► https://github.com/yhilpisch/py4fi2nd
1. Python And Finance
2. 1.0 WHY PYTHON FOR FINANCE
3. 2.0 PYTHON INFRASTRUCTURE
4. Mastering The Basics
5. 3.0 DATA TYPES AND STRUCTURES
6. 4.0 NUMERICAL COMPUTING WITH NUMPY
7. 5.0 DATA ANALYSIS WITH PANDAS
8. 6.0 OBJECT-ORIENTED PROGRAMMING
9. Financial Data Science
10. 7.0 DATA VISUALIZATION
11. 8.0 FINANCIAL TIME SERIES
12. 9.0 INPUT/OUTPUT OPERATIONS
13. 10.0 PERFORMANCE PYTHON
14. 11.0 MATHEMATICAL TOOLS
15. 12.0 STOCHASTICS
16. 13.0 STATISTICS
17. Algorithmic Trading
18. 14.0 THE FXCM TRADING PLATFORM
19. 15.0 TRADING STRATEGIES
20. 16.0 AUTOMATED TRADING
21. Derivative Analytics
22. 17.0 VALUATION FRAMEWORK
23. 18.0 SIMULATION OF FINANCIAL MODELS
24. 19.0 DERIVATIVES VALUATION
25. 20.0 PORTFOLIO VALUATION
26. 21.0 MARKET-BASED VALUATION
117. Python for finance
► https://github.com/yhilpisch/py4fi2nd
1. 3.0 DATA TYPES AND STRUCTURES
from decimal import Decimal
a = 1.115
Decimal(a) # Decimal(a) 的值和想像中的不同
Decimal('1.1149999999999999911182158029987476766109466552734375')
b = 1.125
Decimal(b)
# Decimal(b) 的值則和想像中一樣
Decimal('1.125')
from decimal import Decimal
a = 1.115
Decimal(str(a)) # Decimal(a) 的值和想像中一樣了
Decimal('1.115')
b = 1.125
Decimal(str(b)
# Decimal(b) 的值則和想像中一樣
Decimal('1.125')
118. Sympy for symbolic
► Jypyther is easy to show Latex
► Ipython latex rendering needs markdown
► https://nbviewer.jupyter.org/github/ipython/ipython/blob
/2.x/examples/Notebook/Display%20System.ipynb#LaTeX
► https://blog.csdn.net/halazi100/article/details/79706172
► Latex(r"[ lim_{x to 0} frac{3x^2 +7x^3}{x^2 +5x^4} = 3.]")
► online Latex editor
► https://www.codecogs.com/latex/eqneditor.php
► https://www.mathcha.io/editor
119. Google Cloud Latex Rendering
import sympy as sym
from IPython.display import Latex
x = sym.symbols('x')
I = sym.integrate(1/(1+x**2), (x, 0, 1))
a = sym.latex(I)
Latex(r'[begin{eqnarray*}' + a + r'end{eqnarray*}]')
a = Integral(cos(x)*exp(x), x)
s= Eq(a, a.doit())
a2 = sym.latex(s)
Latex(r'[begin{eqnarray*}' + a2 + r'end{eqnarray*}]'
120. Run conda on Google Cloud Console
► If interactive python fails, do the following steps:
► remove ./.local
► /usr/bin/python3 -m pip install -U jupyter --user --ignore-
installed
► restart VS
► conda initiate
► source /home/hwchiu0810/miniconda3/bin/activate
► source /home/hwchiu0810/miniconda3/bin/deactivate
► Kernel Environment Selection for terminals and ipython
144. Difficult decision and simple results
• Who makes the decision?
• Human or Data?
• From very complicated to very simple:
• 1 or 0
• True or false
• Pass or Fail
• Yes or no.
• Do it or not to do it.
157. Computational Statistics in
Python from Duke
• https://people.duke.edu/~ccc14/sta-663-2018/
• https://github.com/cliburn/sta-663-
2019/tree/master/notebook
158. Statistical Data Modeling
• https://www.statsmodels.org/stable/examples/notebooks/ge
nerated/statespace_cycles.html
