\start
Date: Wed, 12 Oct 2011 12:12:59 -0400
From: Camm Maguire
To: Tim Daly
Subject: Re: odd looping behavior

Greetings!  Sorry I did not see this earlier.  In case this is still
extant:

Tim Daly writes:

> Camm,
>
> I am seeing some odd behavior in loops.
>
> If I write
>
> (let (result)
>   (setq a 
>     (do ((x somelist (cdr x)) (result nil))
>         ((atom x) (reverse result))
>       (setq result (cons (fn x) result)))))

This passes each cons of somelist to fn, not just the cars.  The outer
result binding is shadowed by the equivalent binding in the do.

>
> or, in a simple form as
> (let (result)
>   (setq a
>    (dolist (x somelist (nreverse result))
>      (setq result (cons (fn x) result)))))
>

This passes the each car of somelist to fn, and will give different
results thereby.

> I do not get the same behavior as
> (setq a 
>  (prog ()
>   (return
>    (do ((x somelist (cdr x)) (result nil))
>        ((atom x) (reverse result))
>     (setq result (cons (fn x) result))))))
>

This is equivalent to the first.

Please let me know if this does not clear this up, and/or if you find
different behavior w.r.t. any other lisp.

\start
Date: Fri, 14 Oct 2011 15:20:50 -0400
From: Tim Daly
To: David Mould
Subject: Re: Axiom CAS

Axiom has its own internal "browser-like" front end called
hyperdoc. I am in the process of moving the information from
hyperdoc to traditional web pages.

The browser pages are all built automatically and live in
$AXIOM/doc/hypertex and the root page is:
$AXIOM/doc/hypertex/rootpage.xhtml

Communication with a running Axiom is started by typing
)browse
at the Axiom command line. Axiom will now listen on
localhost:8085 and serve pages to the same port.

This is still under development. It will eventually allow
Sage notebook-style interactions (See "Any Command").
Unfortunately there is still much work to be done but
the basics exist.

Tim Daly


On Fri, 2011-10-14 at 20:09 +0100, David Mould wrote:
> Hello
>
> 
>
> I download and install the Axiom Scientific Computation System from
> time to time usually when I update my Fedora Linux system. I note
> that, for the past year or so, Axiom has been provided with a
> graphical front end through the Firefox web browser. Unfortunately, it
> is not abundantly clear to me how the Firefox GUI is invoked from the
> details provided on your web-site. Is it invoked through a statement
> similar to notebook() used at the command line in the Sage CAS system
> or by some other method?
>
> 
>
> I note that on http://en.wikipedia.org/wiki/Axiom_%
> 28computer_algebra_system%29, volume 11 of the listed documentation
> describes the compilation of the web pages apparently from source code
> but this is too =E2=80=98heavy duty=E2=80=99 for the occasional user. I=
 therefore
> wondered whether any precompiled material is available (open-source)
> that will allow Axiom to be used within Firefox without too much
> difficulty.

\start
Date: Sun, 16 Oct 2011 12:22:10 -0500
From: Tim Daly
To: Ralf Hemmecke
Subject: Bootsys and Depsys history

I created Bootsys to be a stand-alone boot-to-lisp conversion program.
We needed this because people were writing portions of the system in BOOT
that needed to be in Common Lisp. The Bootsys image did the translations
before the whole system existed. (Axiom no longer has any BOOT code so
Bootsys no longer exists.)

I created Depsys to contain the necessary macros and functions to compile
any file. I helped translate Scratchpad from MacLisp and LispVM to Common
Lisp and a portion of that translation involved writing macros that worked
like their MacLisp and LispVM versions (e.g. fileactq, qcar, etc). These
need to be loaded before compilation.

So in theory you should be able to run Bootsys on any .boot file and end up
with a Common Lisp file. Then you should be able to compile this Common Lisp
file using Depsys.

As more things are moved to BOOT and functionality is added to Bootsys you 
begin to reach a state where you need the Depsys macros which creates a circle.

Scratchpad developers did work in full interpsys images and getting it all
to compile cleanly was my problem. Scratchpad could never build "standalone".
It always needed a running interpsys during the build of interpsys. It was
not until Axiom became open source that I untangled the build graph.

I do not know what your particular problem is but I thought I'd give you
some history about the different images and why they exist. 

\start
Date: Sun, 16 Oct 2011 19:36:35 +0200
From: Ralf Hemmecke
To: Tim Daly
Subject: Re: Bootsys and Depsys history

Thank you, Tim,

it's always good to learn about the historic aspects.

> I do not know what your particular problem is but I thought I'd give you
> some history about the different images and why they exist.

Recently, Waldek removed BOOT in favour of SHOE. Then he switched to 
compile the .boot files with bootsys. While seemingly FriCAS compiles 
fine, libaxiom.al currently does not.

The first problem was missing parentheses in ax.boot. But there is 
another one which I am currently trying to track down. If I didn't have 
"git bisect run" I would give up. But so it's just waiting for the 
computer to show the bad commit.

PS: Yes, I'm still hoping that one day Aldor becomes free. ;-)

\start
Date: Sun, 16 Oct 2011 13:15:24 -0500
From: Gabriel Dos Reis
To: Ralf Hemmecke
Subject: Re: Bootsys and Depsys history

On Sun, Oct 16, 2011 at 12:36 PM, Ralf Hemmecke wrote:

> Recently, Waldek removed BOOT in favour of SHOE. Then he switched to compile
> the .boot files with bootsys.

Some data points: OpenAxiom had removed depsys for years and
exclusively used what is being called SHOE (bootsys in OpenAxiom) to
build the entire compiler and interpreter suite.  One of the earliest
things I did was to have a simple diff tool that compared the output
of both old and new bootsys, to spot syntactic differences.  My own
experience is that SHOE feels much better to work with than depsys or
equivalent.  Once you remove depsys, you would realize that the system
is much simpler in structure than one might realize.  It is an
unnecessary complexity.  Your mileage may vary.

> While seemingly FriCAS compiles fine, libaxiom.al currently does not.

Until the recent release, OpenAxiom used to have the Aldor
interoperability boot files. They were removed from the last release.
My suspicion is that it would be much easier to work with Aldor 2
(when it comes to life) with a sound existing AXIOM system than try to
work backward.  Your mileage may vary.

> The first problem was missing parentheses in ax.boot. But there is another
> one which I am currently trying to track down. If I didn't have "git bisect
> run" I would give up. But so it's just waiting for the computer to show the
> bad commit.

I used to maintain an "incompatibility" list between new boot and old
boot.  I also built into the SHOE translator a compatibility detector
which was very handy.  Experience taught me it would be unwise to
attempt the move without assistance from tools.  After the conversion,
and after two releases (I think), I also removed the list because at
the time OpenAxiom was the only AXIOM that made the move to build the
entire system with SHOE.

\start
Date: Sun, 16 Oct 2011 14:14:36 -0500
From: Tim Daly
To: Ralf Hemmecke
Subject: Aldor

Ralf,

> PS: Yes, I'm still hoping that one day Aldor becomes free. :-)

I think you over-estimate the value of a "free Aldor" and underestimate
the costs.

Aldor is a fairly complex compiler. It has its own version of various
unix utilities (e.g. tar-like files). It has an "interesting" layering
that involves its own build system. The internals are not well documented.
Worse yet, it serves 3 masters: it compiles to binary, it has a built-in
interpreter, and it compiles to code that you want to run in AXIOMsys.

So if Aldor became "free" as of today your problem is only beginning.
You would be at the same state that Axiom was in 2001, a huge pile of
code without the original authors involvement (well, not counting myself).

As you can see, Aldor is currently "abandon-ware" and it is suffering from
code-rot. Releasing it as open source will do nothing to stop the rot. The
compiler will need to change to track changes in the AXIOMsys image (as you
can already see, this can be painful to debug). AXIOMsys changes might have
incompatible syntax and semantics introduced which will really cause havoc.

Somebody would have to make maintaining Aldor their primary life
goal. I can tell you from experience that this is extremely costly in
terms of time. There is much to learn, there will be hard bugs to find
and fix, you will always be "chasing" AXIOMsys compatible syntax and
semantic changes, and the original authors will not be available to
answer questions (in Axiom's case, some of the authors are dead).

You will be faced with hard choices such as remaining compatible with
"old" Aldor code or embracing the "new" Aldor code that follows
AXIOMsys. Of course, you'd have to rewrite the compiler to support the
"new" semantics in the binary, interpreter, and cross-compiler. That
would be a lot of hard work, especially if you disagreed with the
"new" semantics anyway. You will have 2 AXIOMsys images to chase which
might have incompatible semantics as the projects don't make an effort
to communicate. Fortunately, Axiom has given up on Aldor and the
support code no longer exists so life is just a bit easier.

You should have a realistic view of the time involved as just doing the
compiler maintenance will take up a large chunk of your life. I spend
about 20 to 30 hours a week and all of my vacation time working on Axiom.

Axiom's changelog lists every file changed (since August 30, 2003
which skips a couple years of history before I introduced changelog). 
It has 13357 lines over 2969 days so far, netting out to 4.5 files 
changed per day on average.  That doesn't include the many thousands 
of emails in the mailing list, some of which took hours to compose.

If you're even mildly successful there will be people who disagree
with your goals and will cause you many sleepless nights, potentially
ending in forks. 

Making Aldor "free" will not keep it alive. You'll discover, as I
have, that keeping software alive means that people will be able to
maintain and modify it after you leave the project. Ultimately this
comes down to documentation. And the best tool for that, in my
opinion, is literate programming. I think we agree on this point. But
the hardest part of literate programming is the documentation. Even in
new code (such as the algebra being added in the last few years) there
is no real effort to document the theory or the issues involved in
reducing the theory to code. The tools are there but the will is not.

Oh, and there are other costs. I maintain 2 servers (axiom-developer.org
and axiom-developer.com) which costs about $300 per month. I travel to
conferences on my own money. I buy books (e.g. Luke's Special Functions
which set me back about $150) that are expensive as I don't have access
to a good library. I buy hardware (a high end laptop and an SSD drive)
to support continuous builds for testing. So it costs me between $4000
and $6000 per year to maintain Axiom (which is challenging as I'm currently
unemployed). Thus "free Axiom" has cost me between $40k and $60k over the
last 10 years. If you're at a university you can bury these costs and also
use students but I don't have those advantages. So be prepared to pay
actual cash to keep up your "open source hobby".

I'm impressed with your tenacity but be careful what you wish for.
It will cost you a huge chunk of your life. It will make people you
respect end up disliking you. Even Linus Torvolds has his detractors.

\start
Date: Sun, 16 Oct 2011 22:27:41 +0200
From: Ralf Hemmecke
To: Tim Daly
Subject: Re: Aldor
Cc: Stephen Watt

On 10/16/2011 09:14 PM, Tim Daly wrote:
>> PS: Yes, I'm still hoping that one day Aldor becomes free. :-)
>
> I think you over-estimate the value of a "free Aldor" and underestimate
> the costs.

Sure, I do. ;-) But a non-free Aldor cannot serve the scientific 
community at all. All CAS people that were interested in Aldor already 
lost interest.

Better it is free and there is a little chance that some day someone 
picks it up) than non-free and unmaintained.

It's completely clear to me that it will take quite some time to build a 
community around Aldor. The longer it takes to get it free, the higher 
the chances that Aldor will be completely forgotten.

Well, it's Stephen's choice to let his baby die. I don't have the 
copyright so I can do nothing but ask.

\start
Date: Mon, 17 Oct 2011 10:15:23 +0100
From: Martin Baker
To: list
Subject: Re: Aldor

On Sunday 16 Oct 2011 21:27:41 Ralf Hemmecke wrote:
> Better it is free and there is a little chance that some day someone
> picks it up) than non-free and unmaintained.

I don't understand the technical issues and I agree that, in most cases, 
choice is a good thing.

However, in the case of pan-axiom there already seems to be far too much 
choice (i.e. fragmentation). What would be the effect of making Aldor free? 
Would each of the 3 lead developers get behind it and abandon their current 
projects or would it cause yet more fragmentation and complexity?

Given the small number of active developers then the current level of 
duplication already seems absolutely mad, 3 lots of release cycles, 3 lots of 
documentation (or lack of it), 3 email lists, and so on. Also all the time 
that is wasted translating from one flavor to another, if only all this wasted 
effort could have been spent adding Aldor-like features to Axiom.

I think this situation must be putting off potential new Axiom users, I would 
be more encouraged to use and contribute to a project if it had a big and 
active community rather than choose between 3 (or presumably 4 if Aldor were 
free) small projects.

Each of the 3 lead developers seems to have an ideology and a long term plan 
to get more users, I hope it works but it seems far from certain? At the 
moment, if any of the lead developers stopped working on their project I can't 
see how their code would survive as an active project. I fear for the long 
term survival of the whole pan-Axiom project.

I haven't put the level of commitment into Axiom that you guys have but I do 
strongly value pan-Axiom and appreciate all the work that has gone into it. I 
guess I don't have much influence but I would like to use any influence that I 
do have to ask you to lobby for less fragmentation rather than more 
fragmentation.

\start
Date: Mon, 17 Oct 2011 11:31:41 +0200
From: Ralf Hemmecke
To: Martin Baker
Subject: Re: Aldor

Hi Martin,

>> Better it is free and there is a little chance that some day someone
>> picks it up) than non-free and unmaintained.

> I don't understand the technical issues and I agree that, in most cases,
> choice is a good thing.

This is not about choice or fragmentation. Simply look at the error 
messages you get from any of the pan-axiom systems. Aldor is not 
perfect, but the compiler is much more mature than any of the axiom-like 
systems.

Understand me correctly, I want to Aldor free in order to write code in 
a properly specified language (which SPAD unfortunately isn't (yet)). 
(You've certainly read the Aldor User Guide.)

I'm not sure whether any of the system will take Aldor as their primary 
compiler, but at least for FriCAS I want to make sure that I can use it 
the day that the Aldor compiler becomes free.

\start
Date: Mon, 17 Oct 2011 05:35:05 -0500
From: Gabriel Dos Reis
To: Martin Baker
Subject: Re: Aldor

On Mon, Oct 17, 2011 at 4:15 AM, Martin Baker wrote:
> On Sunday 16 Oct 2011 21:27:41 Ralf Hemmecke wrote:
>  [...] if only all this wasted  effort could have been spent adding
> Aldor-like features to Axiom.

like per/rep, or import Foreign to OpenAxiom? ;-)

\start
Date: Mon, 17 Oct 2011 20:39:33 -0400
From: Tim Daly
To: Ralf Hemmecke
Subject: Re: Aldor

Well I spent the night thinking about it and, frankly, I'd
say that Haskell has taken over the niche where Aldor could
have been. Haskell has an interactive mode, a strong compiler,
strong type checking, multi-core support, and a much larger
user population. I don't see that Aldor offers any advantages
at this point.

Its been on my queue of things to do to figure out how to
generate haskell using the same mechanisms we use to generate
Fortran. Something like )set output haskell

On Sun, 2011-10-16 at 22:27 +0200, Ralf Hemmecke wrote:
> On 10/16/2011 09:14 PM, Tim Daly wrote:
> >> PS: Yes, I'm still hoping that one day Aldor becomes free. :-)
> >
> > I think you over-estimate the value of a "free Aldor" and underestimate
> > the costs.
> 
> Sure, I do. ;-) But a non-free Aldor cannot serve the scientific 
> community at all. All CAS people that were interested in Aldor already 
> lost interest.
> 
> Better it is free and there is a little chance that some day someone 
> picks it up) than non-free and unmaintained.
> 
> It's completely clear to me that it will take quite some time to build a 
> community around Aldor. The longer it takes to get it free, the higher 
> the chances that Aldor will be completely forgotten.
> 
> Well, it's Stephen's choice to let his baby die. I don't have the 
> copyright so I can do nothing but ask.

\start
Date: Wed, 19 Oct 2011 07:16:17 -0400
From: Tim Daly
To: list
Subject: Re: [sage-devel] Multiple databases and sage

On Wed, 2011-10-19 at 10:44 +0200, Paul-Olivier Dehaye wrote:
> Dear all,
> 
> I am currently participating in a sage-based project that aims to
> integrate a lot of number theory databases (some of you know it, or
> are even participating!).  Some of the goals of the project are to
> display, search across or perform further computations on archived
> objects that come from very different sources. The variety of sources
> can be due to human factors (more than one person has data, or people
> have overlapping but not identical interest), historical reasons (a
> legacy table has since been expanded but should be kept for
> compatibility reasons), and most importantly mathematical reasons (the
> archived objects can be computed using very different constructions
> that current mathematical results do not know how to unify, but all
> these objects should really be looked at through the same lens
> sometimes).
> 
> I expect these to be very common problems when trying to integrate
> mathematical data from various sources into sage, regardless of the
> area of mathematics.
> 
> Does this sound familiar to anyone?
> 
> Some participants of the sage-combinat project are pushing for the
> concept of "Categories" (<> mathematical categories) as a way to
> exploit object-oriented programming to its fullest and organize code
> in a flexible and efficient way. "Categories" are then used to
> incorporate purely mathematical information (of the type "a Field is a
> Ring"), that Python can also understand and use to build a whole class
> hierarchy.

Axiom is based on a Category concept. Categories give a mathematical
structure to the results that make it possible to use the algebra
as a scaffold to organize the software. It has proven to be very
effective.

Can you point me at the sage-combinat project?

\start
Date: Thu, 20 Oct 2011 16:53:31 +0100
From: Martin Baker
To: list
Subject: Kernel documentation

Frustrated at the minimal documentation for the Kernel domain I have attempted 
to start documenting it myself here:

https://github.com/martinbaker/fricas/blob/master/src/algebra/kl.spad.pamphlet

I have added lines: 216:272

I don't consider myself an expert on this subject but perhaps its better than 
just the minimal inline comments. If so, you would be welcome to include it in 
Axiom/FriCAS/OpenAxiom, please also feel free to correct and extend it.

\start
Date: Thu, 20 Oct 2011 12:09:52 -0400
From: Tim Daly
To: Martin Baker
Subject: Re: Kernel documentation

Excellent. 

I would comment that Axiom now supports "examples" of functions.
You add examples to documentation using ++X lines. For example,

pop_! : % -> S
  ++ pop the first element off the array stack
  ++ It destructively modifies its argument
  ++
  ++X a:ArrayStack INT:= arrayStack [1,2,3,4,5]
  ++X pop! a
  ++X a

The ++X comments are printed when someone displays the operation:

)d op pop!
There are 4 exposed functions called pop! :
   [1] ArrayStack D1 -> D1 from ArrayStack D1 if D1 has SETCAT
   [2] Dequeue D1 -> D1 from Dequeue D1 if D1 has SETCAT
   [3] D -> D1 from D if D has SKAGG D1 and D1 has TYPE
   [4] Stack D1 -> D1 from Stack D1 if D1 has SETCAT

Examples of pop! from ArrayStack

a:ArrayStack INT:= arrayStack [1,2,3,4,5] 
pop! a 
a


Examples of pop! from Dequeue

a:Dequeue INT:= dequeue [1,2,3,4,5] 
pop! a 
a


Examples of pop! from StackAggregate

a:Stack INT:= stack [1,2,3,4,5] 
pop! a 
a


Examples of pop! from Stack

a:Stack INT:= stack [1,2,3,4,5] 
pop! a 
a


On Thu, 2011-10-20 at 16:53 +0100, Martin Baker wrote:

> Frustrated at the minimal documentation for the Kernel domain I have
> attempted to start documenting it myself here:
> https://github.com/martinbaker/fricas/blob/master/src/algebra/kl.spad.pamphlet
> I have added lines: 216:272 I don't consider myself an expert on
> this subject but perhaps its better than just the minimal inline
> comments. If so, you would be welcome to include it in
> Axiom/FriCAS/OpenAxiom, please also feel free to correct and extend
> it.

\start
Date: Sat, 22 Oct 2011 14:37:43 -0500
From: Tim Daly
To: list
Subject: [Hamming problems]

http://www.paulgraham.com/hamming.html

In this essay Hamming makes some very interesting points. One of the
points is that we need to know what are the important problems in our
field. He used fridays to discuss only the important problems.

Which raises the question, 

  What are the important problems in computational mathematics?

I'd like to have some thoughts and discussion on this topic.
Please include [Hamming problems] in your subject line so we can
isolate the discussion.

\start
Date: Sat, 22 Oct 2011 14:43:05 -0500
From: Tim Daly
To: list
Subject: [Hamming problems]

What are the important problems in computational mathematics?

I believe that we need to merge proof technology with computational
mathematics. We need to build a platform of proofs, possibly starting with
things like Axiom Categories.

For instance, can we prove that an Axiom Ring fulfills the ring axioms?

We could probably build an annotated bibliography of research papers and
websites. I will start one and host it on the Axiom website.

\start
Date: Tue, 25 Oct 2011 07:30:46 -0700 (PDT)
From: Paul-Olivier Dehaye
To: list
Subject: Re: [sage-devel] Multiple databases and sage

The sage-combinat project is mostly hosted at 
http://wiki.sagemath.org/combinat , and parts are constantly merged into 
sage. There is also the sage-combinat-devel group. 

\start
Date: Wed, 26 Oct 2011 14:06:01 -0500
From: Tim Daly
To: list
Subject: Literate programming

I see that my Literate Programming session is beginning to gain some
traction. I would encourage you to bring examples. We can discuss the
merits and possibly gain some new insights. If nothing else, please
sign up for the Literate Software session at Clojure-Conj. I promise
to keep it short.

Literate programming can take various forms. I am working on a survey
of literate software. I came across an interesting non-latex example
worth sharing:

http://jashkenas.github.com/coffee-script/documentation/docs/nodes.html

Notice how well they have documented an apparently simple line as in:

  exports.Base = class Base

  The Base is an abstract base class for all nodes in the syntax tree.
  Each subclass implements the compileNode method, which performs the
  code generation for that node. To compile a node to JavaScript, call
  compile on it, which wraps compileNode in some generic extra smarts,
  to know when the generated code needs to be wrapping up in a
  closure. An options hash is passed and cloned throughout, containing
  information about the environment from higher in the tree (such as
  if a returned value is being requested by the surrounding function),
  information about the current scope, and indentation level.

Notice how this is not only giving trivial information (e.g. Base is
an abstract base class) but WHY it exists (..as a base for all nodes in
the syntax tree). It gives operational information (to compile a node..)
as well as information about the effect (..which wraps...). It shows
how global information is used (An options hash..) and WHY (containing
information about the environment...)

Code only tells you HOW something is done at the time it is done.
It's like having a recipe without an idea what you would make.

If our standards of documentation were raised to this level then large
systems like Axiom, Clojure, and ClojureScript would be much easier to
maintain and modify in the long term.

If you want your code to live beyond you, make it literate.

\start
Date: Thu, 27 Oct 2011 13:58:52 -0400
From: Tim Daly
To: list
Subject: Re: Literate programming

On Thu, 2011-10-27 at 00:17 -0700, Mark Engelberg wrote:
> Tim,
> 
> I recall that at some point you described your setup for doing Clojure
> literate programming, and if I recall correctly, you were primarily
> working in LaTeX, relying on incremental compilation to test little
> snippets of code as you wrote them.

Yes, the idea is to write a literate version of Clojure, see
http://daly.literatesoftware.com/clojure.pamphlet
http://daly.literatesoftware.com/clojure.pdf
similar in style to "Lisp in Small Pieces". (The effort has been
stalled temporarily while I try to find new employment.)

>From the above document you extract the Makefile, type 'make'
and end up with the newly formatted PDF and a running REPL.
So any edits to the document text are immediately reflected in
the PDF and any edits to the code are immediately reflected in 
the compiled results.

> 
> Have there been any new developments on the literate programming /
> Clojure front, in terms of tools that leverage existing build tools,
> test suites, generating meaningful line numbers for stack traces and
> debugging?
> 

The question is ill-posed in sense that literate programming is not
a tool or technology but a change in mindset. To quote Knuth:

  I believe that the time is ripe for significantly better documentation
  of programs, and that we can best achieve this by considering programs
  to be works of literature. Hence, my title "Literate Programming".

  Let us change our traditional attitude to the construction of 
  programs: Instead of imagining that our main task is to instruct
  a computer what to do, let us concentrate rather on explaining to
  human beings what we want a computer to do.
    -- Donald Knuth "Literate Programming (1984) CSLI 1992 pg. 99

So imagine a world where the eloquence of Rich Hickey was expressed in
book form. Imagine the whole immutability lecture written down and 
decorated with the actual running code as illustration. Imagine that
the original authors clearly explained software transactional memory
and illustrated it with the actual code. Imagine a discussion of
argument destructuring with running code illustrations.

How many more people would be able to dive into the details of Clojure
to maintain and modify the code? How many people would find it much
easier to understand prototypes, defrecords, macros, and all of the
other struggles that populate the current email forums? How are
infinite sequences supported and what code supports it? What does
NIL mean and why was it defined that way?

We do not capture the ideas. We do not rewrite the explanations to
clarify the fine points of confusion. We do not illustrate how the
ideas of Red-Black tries are moved "from ideas to implementation".
We do not communicate the ideas to each other. We code for the machine.

So we end up with the traditional "tree of sand" pile of little files.
In order to overcome this we construct tools (IDEs) that know how to
navigate this pile, tools to store the changes (SVN, git, hg), tools
to put the sandpiles together (#include), tools to build guardrails,
tools, tools, tools... (You can, of course, keep each chapter and
section and subsection in separate files and "include" them in the
book.)

Now imagine that the "book" is multimedia where you can include
animation (watch the tree balance), video, and a running REPL.
Think Khan Academy for Clojure.

This community is willing to overthrow the traditional ideas of lisp
in order to make progress on a new path. Yet we continue to structure
the development as though we worked on a PDP 11/40 with 4k file size
limits. We continue to let the most valuable information which is the
meat of Clojure disappear while we keep only the code, the bones of
Clojure.

Open the source code. Stare at it. Ask yourself if you understand
exactly why it was needed, why it is structured that way, what would
happen if you changed it and what else depends on this code. Imagine
your job is to maintain and modify it but Rich is not available for
questions and answers.

Ultimately that is what matters. In the long term the code will be
the only remaining artifact after Rich leaves the project. Look at
Sourceforge and you will see thousands of dead projects that will
never be picked up because they are just trees of code, dead code.
Et tu, Clojure?

Literate programming is about making code live. 
I like Clojure and I really want it to live.

Think long term. Imagine a better way.

\start
Date: Thu, 27 Oct 2011 23:52:21 -0400
From: Tim Daly
To: list
Subject: Re: Literate programming

On Thu, 2011-10-27 at 20:11 -0400, Larry Johnson wrote:
> My two favorite articles on Literate Programming are both from Donald
> Knuth's book Literate Programming.  One is "Computer Programming as an
> Art", and the other is "Literate Programming".  When I was preparing
> to interview Knuth a bit over a year ago I re-read the entire book.  I
> expected it to be a somewhat outdated description of WEB, TANGLE, and
> WEAVE.  On the contrary it was wonderfully timeless.  When I mentioned
> that to Knuth he sort of grumbled something to the effect of "Well,
> yes, some things in computer science have a long shelf life" (that's a
> paraphrase, but it was something like that).

Knuth's invention of literate programming is one of the unrecognized
pearls of computer science. I have yet to see a programming team that
has an Editor-in-Chief who does patch-review for clarity, sentence
structure, punctuation, relevance, location in the book, proper
citations and index terms, etc. Maybe someday.

> 
> I haven't been working with it for awhile, but I did a somewhat
> primitive modification to the XML Docbook markup language (I just
> added a few appropriate tags for "tangling" the executable source
> code, and "weaving" the well formatted article documenting the code)
> which I used as the source language, then wrote a tangle and weave in
> perl.  I got the idea from Norman Walsh's article Literate Programming
> in XML which can be found at
> http://nwalsh.com/docs/articles/xml2002/lp/paper.html
> 
> The advantage of this was that given the array of tools for rendering
> Docbook "weaving" was a piece of cake, and perl had a good range of
> modules for doing the "tangle".

Any means of publication can be the medium for literate programming.
As I rule I prefer Latex but anything will do.

All you need is a distinguished means of quoting and naming the
chunks. In html this could be as simple as:
   <pre id="somename">
      your code
   </pre>
and you need a program, often called "tangle", to extract the chunk
   tangle mywebpage.html somename >mysomename.file

The machinery of literate programming is dirt simple. 
Poof! You're done. 

The hardest part of literate programming is the mindset.

In order to do literate programming you need to change your focus
from traditional programming to writing for humans and, as a side
effect, writing for the machine.

> 
> As I stated, I'm very new to clojure, but I've always been fascinated
> with LP, and I'm very happy to see this discussion going on here.

Java has taken the PDP 11/40 sand files to their logical extreme where
we have wired the name of the tiny file to the name of the tiny object.
We have packaged the automobile into a crate with labeled bags of
screws, hoses, switches, etc. and are expected to understand the car.

Clojure is a very early adoptive, open minded community willing to 
challenge old assumptions. It helps to highlight what those old
assumptions are, as Rich has done for Lisp, since they can be 
difficult to see and hard to displace. I am hoping the community
will disrupt the tiny-files, javadoc, IDE, code-for-the-machine,
mindset and start communicating with humans. Rich has great ideas
in Clojure and all we get is the dried bones of source code.

We can do so much better.

\start
Date: Fri, 28 Oct 2011 12:33:42 -0400
From: Tim Daly
To: Tassilo Horn
Subject: Re: Literate programming

On Fri, 2011-10-28 at 11:59 +0200, Tassilo Horn wrote:
> Hi Tim,
> 
> while I agree that good documentation is important for maintaining and
> developing further a given code base, I always wonder how literate
> programming deals with refactoring and larger restructuring.  I mean, in
> basically all software projects I'm involved in, developers have a hard
> time in keeping at least the most integral documentation up-to-date,
> e.g., docstrings, JavaDocs, etc.
> 
> Now I consider a some literate code base, i.e., a book that describes
> the complete system in a very detailed manner including the reasons to
> design it that way.  That's awesome for developers joining the project,
> but doesn't it hinder any further development?
> 
> For example, I'm currently reworking some paper that includes code for
> some model transformation.  If I had the right tools, I could extract
> the source code from the latex files and have a running transformation.
> But now, I had to do some minor modifications to the code which took me
> about 5 minutes, but adapting the surrounding text takes me hours over
> hours.  I think, that's an experience many people have made: changing
> source code of a system that is well-designed and understood by the
> developer is much easier to do than changing a cohesive text in natural
> language.  In code, I have an automated, picky lector that always reads
> the complete "book" after each change I make (also known as compiler).
> In plain text, only a human can verify consistency and only in very
> narrow borders.
> 
> Well, that's more text than I intended to write. ;-) So short story
> long: how do you manage conistency of docs and code in literate code
> bases?

Hard work? Book authors face the same problem. Every new change to
Clojure makes more work for the authors.

One side-effect of Literate Programming is that you re-arrange your
code to fit the presentation. As a result you bring together chunks
that are all related (e.g. the code to support Red-Black tries).

I would claim that refactoring "in the large" is unlikely in a
literate program because you have to think through the code in order
to explain it clearly. But if you do a large refactor then it is 
just hard work. For small refactorings you might not have to change
the explanation at all.

Clojure is pretty well "firmed up" at this point. I don't know if
there is going to be a large rewrite of the fundamentals.

\start
Date: Thu, 27 Oct 2011 12:45:46 -0700 (PDT)
From: Daniel Jomphe
To: clojure@googlegroups.com
Subject: Re: Literate programming

I would gladly pay for such a thing to materialize on my screen; if it only 
took money to get that, I'm sure we'd all be willing to finance such an 
effort however we can.

On Thursday, October 27, 2011 1:58:52 PM UTC-4, TimDaly wrote:
>
> So imagine a world where the eloquence of Rich Hickey was expressed in
> book form. Imagine the whole immutability lecture written down and 
> decorated with the actual running code as illustration. Imagine that
> the original authors clearly explained software transactional memory
> and illustrated it with the actual code. Imagine a discussion of
> argument destructuring with running code illustrations.
>
> How many more people would be able to dive into the details of Clojure
> to maintain and modify the code? How many people would find it much
> easier to understand prototypes, defrecords, macros, and all of the
> other struggles that populate the current email forums? How are
> infinite sequences supported and what code supports it? What does
> NIL mean and why was it defined that way?
>
> We do not capture the ideas. We do not rewrite the explanations to
> clarify the fine points of confusion. We do not illustrate how the
> ideas of Red-Black tries are moved "from ideas to implementation".
> We do not communicate the ideas to each other. We code for the machine.
>
> ... 

> Now imagine that the "book" is multimedia where you can include
>
> animation (watch the tree balance), video, and a running REPL.
> Think Khan Academy for Clojure.
>
> This community is willing to overthrow the traditional ideas of lisp
> in order to make progress on a new path. Yet we continue to structure
> the development as though we worked on a PDP 11/40 with 4k file size
> limits. We continue to let the most valuable information which is the
> meat of Clojure disappear while we keep only the code, the bones of
> Clojure.

\start
Date: Fri, 28 Oct 2011 00:30:03 -0700 (PDT)
From: Stefan Kamphausen
To: list
Subject: Re: Literate programming

just to be sure: are you are aware of Marginalia?
https://github.com/fogus/marginalia

\start
Date: Fri, 28 Oct 2011 11:59:46 +0200
From: Tassilo Horn
To: list
Subject: Re: Literate programming

while I agree that good documentation is important for maintaining and
developing further a given code base, I always wonder how literate
programming deals with refactoring and larger restructuring.  I mean, in
basically all software projects I'm involved in, developers have a hard
time in keeping at least the most integral documentation up-to-date,
e.g., docstrings, JavaDocs, etc.

Now I consider a some literate code base, i.e., a book that describes
the complete system in a very detailed manner including the reasons to
design it that way.  That's awesome for developers joining the project,
but doesn't it hinder any further development?

For example, I'm currently reworking some paper that includes code for
some model transformation.  If I had the right tools, I could extract
the source code from the latex files and have a running transformation.
But now, I had to do some minor modifications to the code which took me
about 5 minutes, but adapting the surrounding text takes me hours over
hours.  I think, that's an experience many people have made: changing
source code of a system that is well-designed and understood by the
developer is much easier to do than changing a cohesive text in natural
language.  In code, I have an automated, picky lector that always reads
the complete "book" after each change I make (also known as compiler).
In plain text, only a human can verify consistency and only in very
narrow borders.

Well, that's more text than I intended to write. ;-) So short story
long: how do you manage conistency of docs and code in literate code
bases?

\start
Date: Fri, 28 Oct 2011 02:35:15 -0700 (PDT)
From: Michael Jaaka
To: list
Subject: Re: Literate programming

Maybe COBOL already solves the problem which Literate Programming want to 
solve?


