AkkomaGang/mfm-parser
1
1
Fork 1
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

Code review part 3

Browse source 
* <https://ilja.space/notice/ALpd6nux5hT2nsfetM>
    * Previous commit: `|> fill_props(token) after the cond` in the parser
    * Previous commit: Don't use intention-specific data in the tokens (e.g. left is reverse, x is mfm-spinX)
* <https://ilja.space/notice/ALpcK6W59UjkIUofU8>
    * This commit: Use less files
* Previous commit: Change nested if-statement in mfm.ex to `cond do`

I also added some more and better info to the README.md and moduledocs.
This commit is contained in:
Ilja 2022-07-25 13:46:11 +02:00
parent a8dd3dd719
commit 1bf36d1f52
32 changed files with 269 additions and 183 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
README.md
View file

@ -40,3 +40,52 @@ You can also convert the tree into HTML.
]
iex> MfmParser.Parser.parse("$[twitch.speed=5s 🍮]") |> MfmParser.to_html()
"<span style=\\"display: inline-block; animation: 5s ease 0s infinite normal none running mfm-twitch;\\">🍮</span><style>@keyframes mfm-twitch { 0% { transform:translate(7px,-2px) } 5% { transform:translate(-3px,1px) } 10% { transform:translate(-7px,-1px) } 15% { transform:translateY(-1px) } 20% { transform:translate(-8px,6px) } 25% { transform:translate(-4px,-3px) } 30% { transform:translate(-4px,-6px) } 35% { transform:translate(-8px,-8px) } 40% { transform:translate(4px,6px) } 45% { transform:translate(-3px,1px) } 50% { transform:translate(2px,-10px) } 55% { transform:translate(-7px) } 60% { transform:translate(-2px,4px) } 65% { transform:translate(3px,-8px) } 70% { transform:translate(6px,7px) } 75% { transform:translate(-7px,-2px) } 80% { transform:translate(-7px,-8px) } 85% { transform:translate(9px,3px) } 90% { transform:translate(-3px,-2px) } 95% { transform:translate(-10px,2px) } to { transform:translate(-2px,-6px) }}</style>"
## Reading
### The Parser
A [parser](https://en.wikipedia.org/wiki/Parsing#Parser) takes in structured text and outputs a so called "tree". A tree is a data structure which can be more easily worked with.
A parser typically consists of three parts
* a Reader
* a Lexer (aka Tokeniser)
* the Parser
A Reader typically has a `next` function which takes the next character out of the input and returns it.
A `peek` function allows it to peek at the next character without changing the input.
There's also some way of detecting if the eof (End Of File) is reached.
Depending on the needs of the parser, it may be implemented to allow asking for the nth character instead of just the next.
A Lexer uses the Reader. It also has a `peek` and `next` function, but instead of returning the next (or nth) character, it returns the next (or nth) token.
E.g. if you have the MFM `$[spin some text]`, then `$[spin`, `some text`, and `]` can be considered three different tokens.
The parser takes in the tokens and forms the tree. This is typically a data structure the programming language understands and can more easily work with.
### The Encoder
Once we have a good data structure, we can process this and do things with it.
E.g. an Encoder encodes the tree into a different format.
### The code
The code can be found in the *lib* folder. It contains, among other things, the Reader, Lexer, Parse, and Encoder modules.
The *test* folder contains the unit tests.
## License
A parser/encoder for Misskey Flavoured Markdown.
Copyright (C) 2022 Ilja
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.

24
lib/encoder.ex
View file

@ -2,6 +2,30 @@ defmodule MfmParser.Encoder do
alias MfmParser.Parser
alias MfmParser.Node
@moduledoc """
An encoder who can turn a tree into HTML.
It only works for the MFM specific tags of the form $[name.opts content].
Other parts of MFM (html, Markdown and [KaTeX](https://katex.org/)) are out of scope for this project.
It can directly take input from function `MfmParser.Parser.parse`.
## Examples
iex> [
...> %MfmParser.Node.MFM.Twitch{
...> children: [%MfmParser.Node.Text{props: %{text: "🍮"}}],
...> props: %{speed: "5s"}
...> }
...> ]
...> |> MfmParser.Encoder.to_html()
"<span style=\\"display: inline-block; animation: 5s ease 0s infinite normal none running mfm-twitch;\\">🍮</span><style>@keyframes mfm-twitch { 0% { transform:translate(7px,-2px) } 5% { transform:translate(-3px,1px) } 10% { transform:translate(-7px,-1px) } 15% { transform:translateY(-1px) } 20% { transform:translate(-8px,6px) } 25% { transform:translate(-4px,-3px) } 30% { transform:translate(-4px,-6px) } 35% { transform:translate(-8px,-8px) } 40% { transform:translate(4px,6px) } 45% { transform:translate(-3px,1px) } 50% { transform:translate(2px,-10px) } 55% { transform:translate(-7px) } 60% { transform:translate(-2px,4px) } 65% { transform:translate(3px,-8px) } 70% { transform:translate(6px,7px) } 75% { transform:translate(-7px,-2px) } 80% { transform:translate(-7px,-8px) } 85% { transform:translate(9px,3px) } 90% { transform:translate(-3px,-2px) } 95% { transform:translate(-10px,2px) } to { transform:translate(-2px,-6px) }}</style>"
iex> MfmParser.Parser.parse("$[twitch.speed=5s 🍮]") |> MfmParser.Encoder.to_html()
"<span style=\\"display: inline-block; animation: 5s ease 0s infinite normal none running mfm-twitch;\\">🍮</span><style>@keyframes mfm-twitch { 0% { transform:translate(7px,-2px) } 5% { transform:translate(-3px,1px) } 10% { transform:translate(-7px,-1px) } 15% { transform:translateY(-1px) } 20% { transform:translate(-8px,6px) } 25% { transform:translate(-4px,-3px) } 30% { transform:translate(-4px,-6px) } 35% { transform:translate(-8px,-8px) } 40% { transform:translate(4px,6px) } 45% { transform:translate(-3px,1px) } 50% { transform:translate(2px,-10px) } 55% { transform:translate(-7px) } 60% { transform:translate(-2px,4px) } 65% { transform:translate(3px,-8px) } 70% { transform:translate(6px,7px) } 75% { transform:translate(-7px,-2px) } 80% { transform:translate(-7px,-8px) } 85% { transform:translate(9px,3px) } 90% { transform:translate(-3px,-2px) } 95% { transform:translate(-10px,2px) } to { transform:translate(-2px,-6px) }}</style>"
"""
def to_html(tree) when is_list(tree) do
{html, styles} = to_html_styles(tree)

23
lib/mfm_parser.ex
View file

@ -1,23 +0,0 @@
defmodule MfmParser do
@moduledoc """
`MfmParser` is a parser for [Misskey Flavoured Markdown](https://mk.nixnet.social/mfm-cheat-sheet).
It can parse MFM and return a tree. It can also turn a tree into HTML.
It only works for the MFM specific tags of the form $[name.opts content].
Other parts of MFM (html, Markdown and [KaTeX](https://katex.org/)) are out of scope here.
## Examples
iex> MfmParser.Parser.parse("$[twitch.speed=5s 🍮]")
[
%MfmParser.Node.MFM.Twitch{
children: [%MfmParser.Node.Text{props: %{text: "🍮"}}],
props: %{speed: "5s"}
}
]
iex> MfmParser.Parser.parse("$[twitch.speed=5s 🍮]") |> MfmParser.Encoder.to_html()
"<span style=\\"display: inline-block; animation: 5s ease 0s infinite normal none running mfm-twitch;\\">🍮</span><style>@keyframes mfm-twitch { 0% { transform:translate(7px,-2px) } 5% { transform:translate(-3px,1px) } 10% { transform:translate(-7px,-1px) } 15% { transform:translateY(-1px) } 20% { transform:translate(-8px,6px) } 25% { transform:translate(-4px,-3px) } 30% { transform:translate(-4px,-6px) } 35% { transform:translate(-8px,-8px) } 40% { transform:translate(4px,6px) } 45% { transform:translate(-3px,1px) } 50% { transform:translate(2px,-10px) } 55% { transform:translate(-7px) } 60% { transform:translate(-2px,4px) } 65% { transform:translate(3px,-8px) } 70% { transform:translate(6px,7px) } 75% { transform:translate(-7px,-2px) } 80% { transform:translate(-7px,-8px) } 85% { transform:translate(9px,3px) } 90% { transform:translate(-3px,-2px) } 95% { transform:translate(-10px,2px) } to { transform:translate(-2px,-6px) }}</style>"
"""
end

67
lib/node.ex Normal file
View file

@ -0,0 +1,67 @@
defmodule MfmParser.Node.Text do
defstruct props: %{text: ""}
end
defmodule MfmParser.Node.Newline do
defstruct props: %{text: "\n"}
end
defmodule MfmParser.Node.MFM.Blur do
defstruct props: %{}, children: []
end
defmodule MfmParser.Node.MFM.Bounce do
defstruct props: %{speed: "0.75s"}, children: []
end
defmodule MfmParser.Node.MFM.Flip do
defstruct props: %{v: false, h: false}, children: []
end
defmodule MfmParser.Node.MFM.Font do
defstruct props: %{font: nil}, children: []
end
defmodule MfmParser.Node.MFM.Jelly do
defstruct props: %{speed: "1s"}, children: []
end
defmodule MfmParser.Node.MFM.Jump do
defstruct props: %{speed: "0.75s"}, children: []
end
defmodule MfmParser.Node.MFM.Rainbow do
defstruct props: %{speed: "1s"}, children: []
end
defmodule MfmParser.Node.MFM.Rotate do
defstruct props: %{}, children: []
end
defmodule MfmParser.Node.MFM.Shake do
defstruct props: %{speed: "0.5s"}, children: []
end
defmodule MfmParser.Node.MFM.Sparkle do
defstruct props: %{}, children: []
end
defmodule MfmParser.Node.MFM.Spin do
defstruct props: %{axis: "z", direction: "normal", speed: "1.5s"}, children: []
end
defmodule MfmParser.Node.MFM.Tada do
defstruct props: %{speed: "1s"}, children: []
end
defmodule MfmParser.Node.MFM.Twitch do
defstruct props: %{speed: "0.5s"}, children: []
end
defmodule MfmParser.Node.MFM.Undefined do
defstruct props: %{}, children: []
end
defmodule MfmParser.Node.MFM.X do
defstruct props: %{size: nil}, children: []
end

3
lib/node/mfm/blur.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Blur do
defstruct props: %{}, children: []
end

3
lib/node/mfm/bounce.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Bounce do
defstruct props: %{speed: "0.75s"}, children: []
end

3
lib/node/mfm/flip.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Flip do
defstruct props: %{v: false, h: false}, children: []
end

3
lib/node/mfm/font.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Font do
defstruct props: %{font: nil}, children: []
end

3
lib/node/mfm/jelly.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Jelly do
defstruct props: %{speed: "1s"}, children: []
end

3
lib/node/mfm/jump.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Jump do
defstruct props: %{speed: "0.75s"}, children: []
end

3
lib/node/mfm/rainbow.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Rainbow do
defstruct props: %{speed: "1s"}, children: []
end

3
lib/node/mfm/rotate.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Rotate do
defstruct props: %{}, children: []
end

3
lib/node/mfm/shake.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Shake do
defstruct props: %{speed: "0.5s"}, children: []
end

3
lib/node/mfm/sparkle.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Sparkle do
defstruct props: %{}, children: []
end

3
lib/node/mfm/spin.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Spin do
defstruct props: %{axis: "z", direction: "normal", speed: "1.5s"}, children: []
end

3
lib/node/mfm/tada.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Tada do
defstruct props: %{speed: "1s"}, children: []
end

3
lib/node/mfm/twitch.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Twitch do
defstruct props: %{speed: "0.5s"}, children: []
end

3
lib/node/mfm/undefined.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.Undefined do
defstruct props: %{}, children: []
end

3
lib/node/mfm/x.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.MFM.X do
defstruct props: %{size: nil}, children: []
end

3
lib/node/newline.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.Newline do
defstruct props: %{text: "\n"}
end

3
lib/node/text.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Node.Text do
defstruct props: %{text: ""}
end

79
lib/parser.ex
View file

@ -3,6 +3,26 @@ defmodule MfmParser.Parser do
alias MfmParser.Node
alias MfmParser.Lexer
@moduledoc """
`MfmParser` is a parser for [Misskey Flavoured Markdown](https://mk.nixnet.social/mfm-cheat-sheet).
It can parse MFM and return a tree. It also has an encoder who can turn a tree into HTML.
It only works for the MFM specific tags of the form $[name.opts content].
Other parts of MFM (html, Markdown and [KaTeX](https://katex.org/)) are out of scope for this project.
## Examples
iex> MfmParser.Parser.parse("$[twitch.speed=5s 🍮]")
[
%MfmParser.Node.MFM.Twitch{
children: [%MfmParser.Node.Text{props: %{text: "🍮"}}],
props: %{speed: "5s"}
}
]
"""
def parse(input, tree \\ [], is_end_token \\ fn _ -> false end) do
case Lexer.next(input) do
:eof ->
@ -83,8 +103,65 @@ defmodule MfmParser.Parser do
end
defp fill_props(node = %{props: props}, %{content: content}) do
new_props = props |> Map.merge(Token.MFM.to_props(content))
new_props = props |> Map.merge(to_props(content))
node |> Map.merge(%{props: new_props})
end
def to_props(opts_string) when is_binary(opts_string) do
cond do
opts_string =~ "." ->
Regex.replace(~r/^.*?\./u, opts_string, "")
|> String.trim()
|> String.split(",")
|> Enum.reduce(%{}, fn opt, acc ->
acc
|> Map.merge(
cond do
opt =~ "speed" ->
%{speed: String.replace(opt, "speed=", "")}
opt =~ "v" ->
%{v: true}
opt =~ "h" ->
%{h: true}
opt =~ "x" ->
%{axis: "x"}
opt =~ "y" ->
%{axis: "y"}
opt =~ "left" ->
%{direction: "left"}
opt =~ "alternate" ->
%{direction: "alternate"}
true ->
if Regex.match?(~r/^\$\[font/, opts_string) do
%{font: opt}
else
%{}
end
end
)
end)
opts_string =~ "$[x" ->
%{
size:
case opts_string |> String.replace("$[x", "") |> String.trim() do
"2" -> "200%"
"3" -> "400%"
"4" -> "600%"
_ -> "100%"
end
}
true ->
%{}
end
end
end

16
lib/token.ex
View file

@ -3,3 +3,19 @@ defmodule MfmParser.Token do
token |> Map.put(:content, content <> new_char)
end
end
defmodule MfmParser.Token.Text do
defstruct content: ""
end
defmodule MfmParser.Token.Newline do
defstruct content: ""
end
defmodule MfmParser.Token.MFM.Open do
defstruct content: ""
end
defmodule MfmParser.Token.MFM.Close do
defstruct content: ""
end

58
lib/token/mfm.ex
View file

@ -1,58 +0,0 @@
defmodule MfmParser.Token.MFM do
def to_props(opts_string) when is_binary(opts_string) do
cond do
opts_string =~ "." ->
Regex.replace(~r/^.*?\./u, opts_string, "")
|> String.trim()
|> String.split(",")
|> Enum.reduce(%{}, fn opt, acc ->
acc
|> Map.merge(
cond do
opt =~ "speed" ->
%{speed: String.replace(opt, "speed=", "")}
opt =~ "v" ->
%{v: true}
opt =~ "h" ->
%{h: true}
opt =~ "x" ->
%{axis: "x"}
opt =~ "y" ->
%{axis: "y"}
opt =~ "left" ->
%{direction: "left"}
opt =~ "alternate" ->
%{direction: "alternate"}
true ->
if Regex.match?(~r/^\$\[font/, opts_string) do
%{font: opt}
else
%{}
end
end
)
end)
opts_string =~ "$[x" ->
%{
size:
case opts_string |> String.replace("$[x", "") |> String.trim() do
"2" -> "200%"
"3" -> "400%"
"4" -> "600%"
_ -> "100%"
end
}
true ->
%{}
end
end
end

3
lib/token/mfm/close.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Token.MFM.Close do
defstruct content: ""
end

3
lib/token/mfm/open.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Token.MFM.Open do
defstruct content: ""
end

3
lib/token/newline.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Token.Newline do
defstruct content: ""
end

3
lib/token/text.ex
View file

@ -1,3 +0,0 @@
defmodule MfmParser.Token.Text do
defstruct content: ""
end

2
test/encoder_test.exs
View file

@ -4,6 +4,8 @@ defmodule MfmParser.EncoderTest do
alias MfmParser.Encoder
alias MfmParser.Node
doctest MfmParser.Encoder
describe "to_html" do
test "it handles text" do
input_tree = [%Node.Text{props: %{text: "chocolatine"}}]

4
test/mfm_parser_test.exs
View file

@ -1,4 +0,0 @@
defmodule MfmParserTest do
use ExUnit.Case
doctest MfmParser
end

33
test/parser_test.exs
View file

@ -2,6 +2,8 @@ defmodule MfmParser.ParserTest do
use ExUnit.Case
alias MfmParser.Parser
doctest MfmParser.Parser
describe "single element input" do
test "it can handle an empty string as input" do
input = ""
@ -586,4 +588,35 @@ defmodule MfmParser.ParserTest do
]
end
end
describe "to_props/1" do
test "it returns speed in the list of parameters" do
assert %{speed: "5s"} = Parser.to_props("$[blabla.speed=5s")
assert %{speed: "0.5s"} = Parser.to_props("$[blabla.speed=0.5s")
end
test "it returns v and h in the list of parameters" do
assert %{v: true} = Parser.to_props("$[blabla.v")
assert %{v: true, h: true} = Parser.to_props("$[blabla.h,v")
end
test "it returns fonts" do
assert %{font: "some_font"} = Parser.to_props("$[font.some_font")
end
test "it returns a size for an x element" do
assert %{size: "200%"} = Parser.to_props("$[x2")
assert %{size: "400%"} = Parser.to_props("$[x3")
assert %{size: "600%"} = Parser.to_props("$[x4")
assert %{size: "100%"} = Parser.to_props("$[xqsdfqsf")
end
test "it returns an empty list when there are no parameters" do
assert %{} = Parser.to_props("$[blabla")
end
test "it ignores unknown parameters" do
assert %{} = Parser.to_props("$[blabla.idk")
end
end
end

34
test/token/mfm_test.exs
View file

@ -1,34 +0,0 @@
defmodule MfmParser.MFMTest do
use ExUnit.Case
alias MfmParser.Token.MFM
test "it returns speed in the list of parameters" do
assert %{speed: "5s"} = MFM.to_props("$[blabla.speed=5s")
assert %{speed: "0.5s"} = MFM.to_props("$[blabla.speed=0.5s")
end
test "it returns v and h in the list of parameters" do
assert %{v: true} = MFM.to_props("$[blabla.v")
assert %{v: true, h: true} = MFM.to_props("$[blabla.h,v")
end
test "it returns fonts" do
assert %{font: "some_font"} = MFM.to_props("$[font.some_font")
end
test "it returns a size for an x element" do
assert %{size: "200%"} = MFM.to_props("$[x2")
assert %{size: "400%"} = MFM.to_props("$[x3")
assert %{size: "600%"} = MFM.to_props("$[x4")
assert %{size: "100%"} = MFM.to_props("$[xqsdfqsf")
end
test "it returns an empty list when there are no parameters" do
assert %{} = MFM.to_props("$[blabla")
end
test "it ignores unknown parameters" do
assert %{} = MFM.to_props("$[blabla.idk")
end
end