docs: some initial groundwork #2

Merged
multisamplednight merged 26 commits from :initial-docs into main 2024-01-20 19:00:52 +00:00
13 changed files with 767 additions and 40 deletions

3
.envrc
View file

@ -1 +1,2 @@
use flake . --impure
use flake . --impure
export TYPST_ROOT="$(pwd)/docs"

1
.gitignore vendored
View file

@ -3,3 +3,4 @@
/target
.pre-commit-config.yaml
*.pdf
*.png

10
CODE_OF_CONDUCT.md Normal file
View file

@ -0,0 +1,10 @@
See https://www.rust-lang.org/policies/code-of-conduct.
The current maintainers, that is,
- [@Schrottkatze](https://forge.katzen.cafe/schrottkatze)
- [@multisamplednight](https://forge.katzen.cafe/multisamplednight)
are the entities to email, message or talk to if you feel like any interaction in the context of
iOwO is not okay. We'll try to answer as soon as we can.
Please do **not** open an issue. Notify the maintainers privately instead.

120
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,120 @@
# Contributing to iOwO
Before we get started, thank you for thinking about doing so!
## Through an issue
- Be excellent to each other. Adhere to the [code of conduct].
- About the title: If you had 5 seconds to tell someone the essence of the issue, what would it be?
### Bugs
- Write out in detail which steps in which order are necessary to reproduce the bug.
- Include environmental information as well, in specific:
- How did you install iOwO?
- What version of iOwO are you running?
- What operating system are you running?
In the case of a Linux distro, mention the specific distro and when you last update as well.
- If the bug causes a crash, try to get a backtrace or in worse cases, a coredump.
### Feature requests
- Be sure to include a motivation in which case your intended feature would be used
even if it seems obvious to you.
- Estimate what would be needed to implement the feature:
- Is it an addition to the language itself?
- Is it just a new command?
- Does it ground-breakingly change how iOwO works?
## Through a PR
1. Clone the repo.
2. Switch to a new appropiately named branch for what you want to do, using `git switch -c`.
3. Implement your code changes with your favorite code editor.
4. Try them with `cargo run`.
5. If there are errors or warnings, go to step 3. Commit occasionally.
6. Otherwise,
- if you have an account at https://forge.katzen.cafe,
1. fork the repo
2. set it up as a remote using `git remote add`
3. push using `git push @ -u`
- if you don't,
1. combine your patches using `git diff --patch` and throw them in a file
2. send that file to one of the maintainers per email
- alongside with a description of what it does
3. also mention in the mail that we should consider GitHub and GitLab mirrors,
referring to this line
### Tech stack
The techstack we operate on is
- [typst] for documents and concrete proposals
- [Rust] for the actual code
So if you want to contribute functionality, take a look at [The Rust Programming Language book]!
If you want to contribute thoughts and technical designs, then consider taking a ride through
[typst's excellent tutorial]!
For creative things, we suggest using whatever **you** are comfortable with.
Otherwise, we suggest these:
- [Inkscape], [GIMP] and [Blender] for promotional material like logos and posters
- [Penpot] for layouting prototypes and the like
[typst]: https://typst.app
[Rust]: https://www.rust-lang.org
[The Rust Programming Language book]: https://doc.rust-lang.org/book/
[typst's excellent tutorial]: https://typst.app/docs/tutorial
[Inkscape]: https://inkscape.org/
[GIMP]: https://www.gimp.org/
[Blender]: https://www.blender.org/
[Penpot]: https://penpot.app/
## Politics
- Current maintainers are defined as the entities listed in the [code of conduct]
### PRs
- Every PR requires an approving review from a maintainer (that is not the author) before merge
- Maintainers can merge their own PRs
- But only after approval
### Major decisions
- All current maintainers have to agree **unanimously**.
- Agreement must be based on [informed consent].
- In effect, a maintainer has to understand what they agree to.
# Interacting with PRs
Remember, be respectful.
Entities invest their free time and motivation into making these changes,
treat them appropiately.
- Since we mostly work based on forks, [git's remotes] work fairly good.
- Replace things in pointy brackets (`<>`) respectively (and remove the pointy brackets).
## Initial steps for a new contributor or new local checkout
```sh
git remote add <contributor-name> https://forge.katzen.cafe/<contributor-account>/iowo.git
git remote update <contributor-name>
```
## After setting up the remote
- You can repeat this step anytime you want to switch branches or update your local checkout.
- The PR branch is visible just below the PR title on Forgejo, after the colon (`:`).
```sh
git switch <pr-branch>
git pull
```
[code of conduct]: ./CODE_OF_CONDUCT.md
[informed consent]: https://en.wikipedia.org/wiki/Informed_consent
[git's remotes]: https://git-scm.com/docs/git-remote

16
README.md Normal file
View file

@ -0,0 +1,16 @@
# iOwO
Tired of remembering if the argument order matters on this ImageMagick command?
Tired of having shell parsing interfere with what you actually want to do?
Fear no more, with iOwO (pronounced iowo (
we believe at least (we don't know what we're doing [please help]!)
)) you can just write a plain pipeline with [nushell]-ish syntax!
Or that's what we want it to be at least.
So, uh, grab a seat and a beverage, sit down, and maybe take a look around.
Although there's not much here — _yet_.
[please help]: ./CONTRIBUTING.md
[nushell]: https://www.nushell.sh/

1
docs/compiled/.gitkeep Normal file
View file

@ -0,0 +1 @@
the just script will place compiled pdfs in here. thank you for your understanding.

188
docs/design/iowo-design.typ Normal file
View file

@ -0,0 +1,188 @@
#import "../template.typ": conf
#import "util/graphics.typ"
#import graphics: arrow
#show: conf.with(
title: [iOwO design],
subtitle: [don't worry, we're just dreaming],
)
= Evaluation stages
#graphics.stages-overview
iOwO operates in stages.
This has a number of benefits and implications:
- One can just work on one aspect of the stages without having to know about the rest.
- Bugs are easier to trace down to one stage.
- Stages are also replacable, pluggable and usable somewhere else.
- For example,
one could write a Just-In-Time compiler as a new executor to replace the runtime stage,
while preserving the source #arrow() graph IR step.
However, this also makes the architecture somewhat more complicated. So here we try our best to describe how each stage looks like. If you have any feedback, feel free to drop it on #link("https://forge.katzen.cafe/katzen-cafe/iowo/issues")[the issues in the repository]!
== Source <source>
```iowo
open base.png >|
open stencil.png >|
mask
|> show
|> (invert | show)
```
Functional and pipeline-based.
However, in contrast to classic shell commands,
functions can have multiple outputs and multiple inputs.
=== Functions
`open`, `invert`, `mask`, `show` and friends.
The foundation of actually "doing" something.
- Can have any, even infinite, amount of inputs and outputs.
- Their amounts may or may not be equal.
- Inputs and outputs must have their types explicitly declared.
- An function with
- at least one output is called a streamer.
- at least one input is called a consumer.
- _both_ at least one input and at least one output is _both_ a streamer and a consumer,
and culminatively called a modifier.
- May also contain spaces in its name.
==== Inputs <input>
- Based on position.
- Inputs can be provided both through the pipeline and ad-hoc.
- Ad-hoc inputs are called arguments.
- So all of these are equivalent:
```iowo
add 1 2 -- all inputs as arguments
[ 1 2 ] | add -- all inputs through the pipeline
1 | add 2 -- one input through the pipeline, one as argument
```
==== Outputs
- Also based on position.
=== Pipelines <pipeline>
- Exchange data between streamers and consumers.
==== Simple forwarding
In the simplest case,
where inputs map to outputs bijectively#footnote[one to one],
pipelines are just pipes and forward unchanged:
```iowo
open owo.png | invert | save owo.png
```
==== Splitting <splitting>
To handle each output of a streamer individually, they can be _split_:
```iowo
mask
|> show -- would show the masked part
|> invert -- would invert the unmasked part
```
==== Combination <combination>
To throw multiple streamers into the inputs of a consumer,
they can be _combined_:
```iowo
open base.png >|
open stencil.png >|
mask
```
However, since lists are automatically spliced into inputs,
this is equivalent to the above:
```iowo
[
open base.png,
open stencil.png,
]
| mask
```
=== Comments
Done with any of `--` or `//`.
=== Data types
==== Lists
- Signified by `[]` braces.
- If thrown into a pipeline, it automatically works like a streamer.
- Can be delimited by commas.
- Must be delimited by commas if a contained streamer takes an input.
- May have a trailing comma.
- Outputs of streamers are spliced into the contained list.
- In effect, they are automatically flattened.
== Graph IR
#graphics.graph-example
The parsed representation of the source, and also what the runtime operates on.
In a way, this is the AST, except that it's not a tree.
It is represented in iOwO using adjacencies, where essentially
the vertices#footnote[Nodes or functions in this case.]
and their edges#footnote[Connections or pipes in this case.]
are stored separately.
=== Optimizer
Merges and simplifies functions in the graph IR.
== Runtime <runtime>
Runs through all functions in the graph IR.
It does not have any significantly other representation,
and despite its name there's _no_ bytecode involved.
Different runtimes are called executors.
Executors operate on instructions instead of functions.
=== Scheduler
Looks at the graph IR and decides when the VM should execute what.
=== VM <vm>
= Open questions
- @input
- At which position are arguments injected into function inputs?
- How could that be controlled if set to e.g. the end by default?
- Not all inputs are order-independent, e.g. `div`
- Should inputs and outputs really be positional?
- Could make more complex functions hard to read
- But keyworded could also make code very noisy
- Maybe a middle ground, such that at most 1 input is allowed to be positional?
- @pipeline
- We need some way to reshuffle and reorder outputs and inputs in a pipeline
- @splitting
- How would one split different outputs into a list?
- @runtime
- What is the difference between an instruction and a function?
- Should outputs that are not thrown into a consumer
be automatically displayed in some kind of debug view?
- Or should that be done instead using a debug `show` function or the like?
- Should consumers be called sinks instead?
- Shorter
- More ambiguous if only looking at the first char though

View file

@ -1,10 +0,0 @@
#import "../template.typ": conf
#show: doc => conf(
doc
)
= meow nya
nyanyanya
#lorem(50)

View file

@ -0,0 +1,251 @@
#import "@preview/cetz:0.1.2"
#import cetz.draw: *
// quick reference
// - `graphic` is for inline icons/images
// - `canvas` is for centered more prominent stuff
// - `group` can be used in either of ^, but not standalone
#let graphic(what) = box({
cetz.canvas({
// any preamble-ish stuff can go here
set-style(
mark: (angle: 90deg),
stroke: (cap: "round", join: "round"),
)
what
})
})
#let canvas(what) = {
align(center, graphic(what))
}
// smaller stuff
#let arrow(length: 0.4cm, lift: 3pt, stroke: 1pt) = graphic({
line((0, lift), (rel: (length, 0)), mark: (end: ">", stroke: stroke))
// hack for the bounding box bottom
// so that `lift` even has any effect
line((0, 0), (0, 0), stroke: none)
})
// larger stuff
#let sequence(
distance: 3cm,
arrow-spacing: 0.15cm,
// cetz will support rounded rects in 0.2.0
style: (frame: "rect", padding: 0.1cm),
..labels,
) = group({
let labels = labels.pos()
// draw each label itself
for (i, label) in labels.enumerate() {
if i != 0 {
set-origin((distance, 0))
}
content((0, 0), name: "label-" + str(i), label, ..style)
}
// then draw an arrow from each to each
// since an arrow is between two, the last one can't be connected with the "next-to-last" one
// so we leave it out
for i in range(labels.len() - 1) {
line(
(rel: (arrow-spacing, 0), to: "label-" + str(i) + ".right"),
(rel: (-arrow-spacing, 0), to: "label-" + str(i + 1) + ".left"),
mark: (end: ">"),
)
}
})
#let stages-overview = canvas({
sequence(
[Source],
[Graph IR],
[Runtime],
)
})
// A few commands to help demonstration in the docs.
// Supply a string to mark the input or output as simple.
// (fwiw in typst, parenthesis around a single expression just evaluate the expression, and don't put it into an array)
#let cmds = (
"const": (
inputs: (),
outputs: ("data",),
),
"open": (
inputs: ("path",),
outputs: ("data",),
),
"save": (
inputs: ("data", "path"),
outputs: (),
),
"show": (
inputs: ("data",),
outputs: (),
),
"invert": (
inputs: ("base",),
outputs: ("",),
),
"mask": (
inputs: ("base", "stencil"),
outputs: ("masked", "rest"),
),
)
#let opposite(anchor) = {
(
"bottom": "top",
"top": "bottom",
)
.at(anchor)
}
#let sockets(
start,
stop,
sockets,
socket-size: (0.5, 0.1),
socket-shape: "circle",
parent-name: "",
label-anchor: "bottom",
) = {
for (i, socket) in sockets.enumerate() {
let x-ratio = (i + 1) / (sockets.len() + 1)
let center = (start, x-ratio, stop)
let socket-name = parent-name + "/" + socket
let common-args = (name: socket-name, fill: black)
if socket-shape == "rect" {
rect(
(rel: ((0, 0), -0.5, socket-size), to: center),
(rel: socket-size),
..common-args,
)
} else if socket-shape == "circle" {
circle(
center,
radius: socket-size.at(1),
..common-args,
)
} else {
panic("unknown socket shape: `" + socket-shape + "`")
}
set-style(fill: none)
// don't ask why, I don't know myself
let use-opposite-anchor = socket-shape == "circle"
content(
socket-name + "." + if use-opposite-anchor { opposite(label-anchor) } else { label-anchor },
anchor: opposite(label-anchor),
box(inset: 0.25em, text(8pt, socket)),
)
}
}
#let node(
at,
size: (3, 1.5),
ty: none,
body: none,
name: "unnamed",
) = {
set-origin(at)
let label = [#ty]
if body != none {
label += [\ ] + text(0.7em, font: "IBM Plex Mono", body)
size.at(1) += 0.5
}
rect((0, 0), (rel: size), name: name)
content(((0, 0), 0.5, size), align(center, label))
// input and output sockets
if ty == none { return }
let ty = cmds.at(ty)
let sockets = sockets.with(parent-name: name)
sockets(
((0, 0), "|-", size),
size,
ty.inputs,
)
sockets(
(0, 0),
((0, 0), "-|", size),
label-anchor: "top",
ty.outputs,
)
// helper text
let helper(base, label, where) = {
if not type(base) != list or base.len() != 0 {
content(
name + "." + where + "-left",
anchor: "right",
box(inset: 0.25em, text(fill: luma(75%), label))
)
}
}
helper(ty.inputs, [in], "top")
helper(ty.outputs, [out], "bottom")
// reset the origin transform so other nodes can still work in the global coord system
// can't use groups since otherwise the anchors are not exported
set-origin(((0, 0), -1, at))
}
#let connect(from, to, bend: 1.5, mark-cfg: (size: 0.25, offset: 0.1)) = {
bezier(
from,
to,
(rel: (0, -bend * 1cm), to: from),
(rel: (0, bend * 1cm), to: to),
)
mark(
(rel: (0, mark-cfg.size + mark-cfg.offset), to: to),
(rel: (0, -mark-cfg.size)),
symbol: ">",
)
}
#let graph-example = canvas({
let x = 3
let y = -3
node((-x, -0.75 * y), ty: "const", body: "\"base.png\"", name: "base")
node((x, -0.75 * y), ty: "const", body: "\"stencil.png\"", name: "stencil")
node((-x, 0), ty: "open", name: "a")
node((x, 0), ty: "open", name: "b")
node((0, y), ty: "mask", name: "c")
node((-x, 2 * y), ty: "invert", name: "d")
node((-x, 2.75 * y), ty: "show", name: "e")
node((x, 2.75 * y), ty: "show", name: "f")
connect("base/data", "a/path")
connect("stencil/data", "b/path")
connect("a/data", "c/base")
connect("b/data", "c/stencil")
connect("c/masked", "d/base")
connect("d/", "e/data")
connect("c/rest", "f/data", bend: 2.5)
})
// literally just for standalone display of the graphics alone
#import "../../template.typ": conf
#show: conf
#set page(width: auto, height: auto)
#graph-example

View file

@ -1,12 +1,152 @@
// would also be interesting to try out IBM Plex Mono/Sans sometime
#let atk = "Atkinson Hyperlegible"
#let fonts = (
main: (font: atk, size: 12pt),
title: (font: atk, size: 20pt),
subtitle: (font: atk, size: 10pt),
heading: (font: "Montserrat", weight: "regular"),
)
#let expand(it) = {
("("
+ upper(it.first())
+ "|"
+ it.first()
+ ")"
+ it.clusters().slice(1).join()
+ "s?")
}
#let singlify(it) = {
it = lower(it)
if it.ends-with("s") {
it = it.slice(0, -1)
}
it
}
#let terminate-recursion(it) = {
let clusters = it.text.clusters()
clusters.insert(1, "\u{FEFF}")
clusters.join()
}
// even though it looks like this could be automated with a `query`,
// this'd require wrapping the whole document in a show rule
// at which point `query` doesn't find anything anymore
#let terms = (
"source",
"AST",
"graph IR",
"runtime",
"executor",
"optimizer",
"scheduler",
"VM",
"function",
"instruction",
"pipeline",
"input", "argument", "consumer",
"output", "streamer",
"modifier",
)
// yes, the shadowing is intentional to avoid accidentally using the list
#let terms = regex("\\b(" + terms.map(expand).join("|") + ")\\b")
#let conf(
doc
title: none,
subtitle: none,
doc,
) = {
set text(font: "Atkinson Hyperlegible");
show heading: it => [
#set text(font: "Montserrat", weight: "regular")
set page(
numbering: "1 / 1",
margin: 3.25cm,
header: locate(loc => {
datetime.today().display()
#it
]
if counter(page).at(loc).first() > 1 {
// on all pages other than the first, the title is useful to have at hand
h(1fr)
title
}
}),
)
set par(justify: true)
set text(..fonts.main)
set heading(numbering: "A.1")
show heading: it => text(..fonts.heading, it)
// color links
show link: text.with(fill: blue)
// prettify codeblocks
show raw.where(block: true): box.with(
fill: luma(95%),
inset: 1.25em,
radius: 0.75em,
width: 45em,
)
show raw.where(block: false): box.with(
fill: luma(95%),
outset: (y: 0.25em),
inset: (x: 0.15em),
radius: 0.25em,
)
show raw.where(block: true): it => {
// don't try to put codeblocks into blocktext
set par(justify: false)
it
}
// semi-strategically place pagebreaks for better orientation
let pagebreak-before(it) = pagebreak() + it
show heading.where(outlined: true): it => {
if it.level <= 2 {
pagebreak(weak: true)
}
it
}
// highlight important terms in bold
show raw: it => {
// avoid making terms in codeblocks bold
show terms: terminate-recursion
it
}
show terms: strong
// color codeblocks
// haskell hl seems to work ok for this
show raw.where(lang: "iowo", block: true): it => {
raw(it.text, lang: "haskell", block: true)
}
// document title
if title != none {
align(
right,
text(..fonts.title, title)
+ v(-12pt)
+ text(..fonts.subtitle, subtitle)
)
v(0.25cm)
}
// outline and other prelude info
outline(
indent: auto,
fill: line(
length: 100%,
stroke: (
cap: "round",
join: "round",
thickness: 0.5pt,
paint: luma(75%),
),
),
)
// content itself
doc
}

View file

@ -8,11 +8,11 @@
"pre-commit-hooks": "pre-commit-hooks"
},
"locked": {
"lastModified": 1699541523,
"narHash": "sha256-Rv0ryuBC5KtA/3YqwIEe58Tabu71qSGnGcGRd67mMUY=",
"lastModified": 1704835383,
"narHash": "sha256-SoC0rYR9iHW0dVOEmxNEfa8vk9dTK86P5iXTgHafmwM=",
"owner": "cachix",
"repo": "devenv",
"rev": "14fdefc0bb80c3d6f3a18a491e33429b4064c371",
"rev": "18ef9849d1ecac7a9a7920eb4f2e4adcf67a8c3a",
"type": "github"
},
"original": {
@ -27,11 +27,11 @@
"rust-analyzer-src": "rust-analyzer-src"
},
"locked": {
"lastModified": 1699510895,
"narHash": "sha256-eaOkJUvHeYNW/xEoRotz0rHkKihKoQdWB1ctX4q1MTQ=",
"lastModified": 1704867811,
"narHash": "sha256-pG4O1vPpNSMjz7p/5x+/OH4tXC0thzAPbJ55kI/W5dU=",
"owner": "nix-community",
"repo": "fenix",
"rev": "8eeef23f2c8d092227af40eff98afe5b41891e3b",
"rev": "93e89638c15512db65e931f26ce36edf8cfbb4a5",
"type": "github"
},
"original": {
@ -186,11 +186,11 @@
},
"nixpkgs_2": {
"locked": {
"lastModified": 1699099776,
"narHash": "sha256-X09iKJ27mGsGambGfkKzqvw5esP1L/Rf8H3u3fCqIiU=",
"lastModified": 1704538339,
"narHash": "sha256-1734d3mQuux9ySvwf6axRWZRBhtcZA9Q8eftD6EZg6U=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "85f1ba3e51676fa8cc604a3d863d729026a6b8eb",
"rev": "46ae0210ce163b3cba6c7da08840c1d63de9c701",
"type": "github"
},
"original": {
@ -202,16 +202,16 @@
},
"nixpkgs_3": {
"locked": {
"lastModified": 1699291058,
"narHash": "sha256-5ggduoaAMPHUy4riL+OrlAZE14Kh7JWX4oLEs22ZqfU=",
"lastModified": 1704722960,
"narHash": "sha256-mKGJ3sPsT6//s+Knglai5YflJUF2DGj7Ai6Ynopz0kI=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "41de143fda10e33be0f47eab2bfe08a50f234267",
"rev": "317484b1ead87b9c1b8ac5261a8d2dd748a0492d",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-23.05",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
@ -231,11 +231,11 @@
"nixpkgs-stable": "nixpkgs-stable"
},
"locked": {
"lastModified": 1688056373,
"narHash": "sha256-2+SDlNRTKsgo3LBRiMUcoEUb6sDViRNQhzJquZ4koOI=",
"lastModified": 1704725188,
"narHash": "sha256-qq8NbkhRZF1vVYQFt1s8Mbgo8knj+83+QlL5LBnYGpI=",
"owner": "cachix",
"repo": "pre-commit-hooks.nix",
"rev": "5843cf069272d92b60c3ed9e55b7a8989c01d4c7",
"rev": "ea96f0c05924341c551a797aaba8126334c505d2",
"type": "github"
},
"original": {
@ -255,11 +255,11 @@
"rust-analyzer-src": {
"flake": false,
"locked": {
"lastModified": 1699451299,
"narHash": "sha256-7HJMyp62fCS6/aCCCASz8MdJM2/M8d1pBNukyLmPdwA=",
"lastModified": 1704833483,
"narHash": "sha256-Ox01mpYmjapNYaqOu4fMS/4Ma9NLd2rVNz6d4rJmcf4=",
"owner": "rust-lang",
"repo": "rust-analyzer",
"rev": "7059ae2fc2d55fa20d7e2671597b516431129445",
"rev": "ae6e73772432cfe35bb0ff6de6fdcfa908642b67",
"type": "github"
},
"original": {

View file

@ -1,6 +1,6 @@
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
systems.url = "github:nix-systems/default";
devenv.url = "github:cachix/devenv";
fenix.url = "github:nix-community/fenix";
@ -36,7 +36,7 @@
rustfmt.enable = true;
};
packages = [];
packages = with pkgs; [just nushell typst];
})
];
};

View file

@ -1,2 +1,11 @@
doc-design:
typst compile docs/design/*.typ --root=docs
# Compile all documentation as in proposals and design documents, placing them under `docs/compiled`.
docs:
#!/usr/bin/env nu
glob **/*.typ --exclude [**/{template.typ,util/**}] | par-each { |source|
typst compile $source --root=docs
let pdf = (
(echo $source | path dirname)
| path join (echo $source | path basename | str replace ".typ" ".pdf")
)
mv $pdf docs/compiled
} | ignore