debian/0000755000000000000000000000000012127346576007203 5ustar debian/control0000644000000000000000000000116112127346576010605 0ustar Source: tcl-sugar Section: interpreters Priority: extra Maintainer: Andrew Shadura Build-Depends: debhelper (>= 9) Standards-Version: 3.9.4 Homepage: http://wiki.tcl.tk/11155 Package: tcl-sugar Architecture: all Depends: tcl (>= 8.4), ${misc:Depends} Description: Lisp-like macro system for Tcl Sugar is a macro system for the Tcl programming language, with a design very similar to Lisp macros. It provides a way to create Tcl procedures using the [sugar::proc] command instead of the normal Tcl [proc] command, and a way to define macros that are expanded inline when the procedure is created. debian/install0000644000000000000000000000003212127256440010555 0ustar sugar0.1 /usr/share/tcltk debian/docs0000644000000000000000000000003712127272735010051 0ustar API.txt README debian/tutorial debian/README.source0000644000000000000000000000041512127274143011350 0ustar Sugar tutorial -------------- In the subdirectory tutorial, a tutorial from the project's page at Tclers Wiki is included in a WubWikit text format. A script, get-wiki-docs, is included to update these pages to their most recent versions as published at wiki.tcl.tk. debian/changelog0000644000000000000000000000021112127257063011037 0ustar tcl-sugar (0.1-1) unstable; urgency=low * Initial release. -- Andrew Shadura Thu, 04 Apr 2013 13:04:18 +0200 debian/source/0000755000000000000000000000000012127254145010471 5ustar debian/source/format0000644000000000000000000000001412127254145011677 0ustar 3.0 (quilt) debian/copyright0000644000000000000000000000511512127256242011126 0ustar Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: Salvatore Sanfilippo Upstream-Contact: antirez@invece.org Source: http://www.hping.org/tclsbignum/sugar-0.1.tar.gz Files: * Copyright: 2004 Salvatore Sanfilippo License: Tcl Files: debian/* Copyright: 2013 Andrew Shadura License: Tcl License: Tcl This software is copyrighted by the Regents of the University of California, Sun Microsystems, Inc., Scriptics Corporation, and other parties. The following terms apply to all files associated with the software unless explicitly disclaimed in individual files. . The authors hereby grant permission to use, copy, modify, distribute, and license this software and its documentation for any purpose, provided that existing copyright notices are retained in all copies and that this notice is included verbatim in any distributions. No written agreement, license, or royalty fee is required for any of the authorized uses. Modifications to this software may be copyrighted by their authors and need not follow the licensing terms described here, provided that the new terms are clearly indicated on the first page of each file where they apply. . IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. . THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. . GOVERNMENT USE: If you are acquiring this software on behalf of the U.S. government, the Government shall have only "Restricted Rights" in the software and related documentation as defined in the Federal Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2). If you are acquiring the software on behalf of the Department of Defense, the software shall be classified as "Commercial Computer Software" and the Government shall have only "Restricted Rights" as defined in Clause 252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the authors grant the U.S. Government and others acting in its behalf permission to use and distribute the software in accordance with the terms specified in this license. debian/examples0000644000000000000000000000001212127256344010726 0ustar example/* debian/rules0000755000000000000000000000003712127260717010253 0ustar #!/usr/bin/make -f %: dh $@ debian/tutorial/0000755000000000000000000000000012127346740011037 5ustar debian/tutorial/Sugar_syntax_macros.txt0000644000000000000000000001401712127272670015636 0ustar * Section 0 - '''[Sugar]''' * Section 1 - '''[Sugar command macros]''' * Section 2 - '''[Sugar syntax macros]''' (what you are reading) * Section 3 - '''[Sugar transformers]''' '''Syntax macros and syntax sugar''' Syntax macros are exactly like the command macros already described, with the only difference that *every syntax macro is called for every command in the application*. Actually you can immagine a command macro as a syntax macro that works as an identity transformation if the first argument does not match a given command name. Because syntax macros are called for every command, they have the ability to add new syntax to Tcl: they can inspect every argument of the whole script and do substitutions on it. You can do with syntax macros many syntax manipulations previously explored with [unknown], with the following differences: * Expand to native code. No performance penality. * Don't need that the first argument is unknown. Always work. * The expansion happens when the procedure is created, so it's possible to use syntax macros for general static checks. [unknown] is very interesting to explore many [Radical Language Modification] ideas, but to add syntax in a reliable way and/or with acceptable performances is often impossible. I'll show syntax macros with the implementation of the following syntax glue. Instead to write: ====== puts [expr {$a+$b}] ====== Let's the user write: ====== puts ($a+$b) ====== or ====== puts ( $a + $b ) ====== or any mix. Well that's *NOT A GOOD IDEA* of course ;) see TIP 174 ([Math Operators as Commands]) to see another alternative to [expr], that I think is a good idea, but this syntax glue it's a decent exercise for syntax macros: ====== sugar::syntaxmacro sugarmath args { set argv $args set start_idx -1 set end_idx -1 for {set i 0} {$i < [llength $argv]} {incr i} { set tok [lindex $argv $i] if {[string index $tok 0] eq {(}} { set start_idx $i } } if {$start_idx == -1} { return $argv } set level 0 for {set i $start_idx} {$i < [llength $argv]} {incr i} { set tok [lindex $argv $i] foreach char [split $tok {}] { if {$char eq {(}} {incr level} if {$char eq {)}} { incr level -1 } } if {$level == 0 && [string index $tok end] eq {)}} { set end_idx $i } } if {$end_idx == -1} { return $argv } # The syntax looks ok, perform the expansion. lset argv $start_idx "\[expr \{[string range [lindex $argv $start_idx] 1 end]" lset argv $end_idx "[string range [lindex $argv $end_idx] 0 end-1]\}\]" return $argv } ====== That's it. With this macro the following code: ====== puts (($a+$b)*($a*$b)) lindex $mylist ($idx-1) ====== is translated to ====== puts [expr {($a+$b)*($a*$b)}] lindex $mylist [expr {$idx-1}] ====== But you can't use it in string interpolation. Because this macro expands to a command substitution form where actually there wasn't one, it can't be used inside strings. This will not work: ====== puts "($a+$b)" ====== Will just print ($a+$b) verbatim. Instead one should use: ====== puts "[format ($a+$b)]" ====== Third good thing about macros: '''3) Macros allows to create syntax sugar without to cause cancer to the semicolon.''' Macros are a good place where we can syntax-overload our preferite language to specialize it and solve a specific problem with less typing. If, instead, we add syntax to Tcl itself, we have a less general, more complex language, that may make a task like the creation of a macro system much more hard. Note that syntax sugar may expand to other macros that are recursively processed, so adding syntax with macros does not limit the range of actions of other macros. '''More static source code checks''' Another example of syntax macro is a macro to output a warning about procedures called with a bad number of arguments. This is very short and simple and will detect a bad usage only if the called procedure is defined *before* of the bad usage (to detect bad usage of commands that are defined later, it needs to be a bit more complex and to collect unresolved calls in a global variable). ====== syntaxmacro aritychecker args { set argv $args set command [lindex $argv 0] set ns [sugar::currentProcNamespace] set procname $ns\::$command if {[catch {info args $procname} cmdargs]} { set procname $command if {[catch {info args $procname} cmdargs]} { return $argv } } set call_argnum [expr {[llength $argv]-1}] # Calculate the min and max number of arguments. set max_argnum [llength $cmdargs] set min_argnum [llength $cmdargs] foreach a $cmdargs { if {[info default $procname $a _]} { incr min_argnum -1 } elseif {$a eq {args}} { incr min_argnum -1 } } # If the last argument is 'args', set a fake max at 1000 if {[lindex $cmdargs end] eq {args}} { set max_argnum 1000 } # Check if it's ok. if {$call_argnum < $min_argnum || $call_argnum > $max_argnum} { puts stderr "*** Warning, procedure '$command' may be called with bad number of arugments in [sugar::currentProcName]" } return $argv } ====== This kind of macros are pretty slow (until the parser is implemented in C at least) being called for every command in the script (possibly more times for every command, because the script is re-processed if some macro expansion happened in order to expand macros produced by macros), so it's better to use it only during development, being not useful at all in a finished program that's correct enough to don't produce warnings. Actually, if [Sugar] will be of some value in my day-to-day programming (and I think it will), it will be reimplemented in C once the API is stable. But for now I'll take in pure Tcl to hack it faster if needed. [TP] 2011/05/18 - Fixed above macros to work with the latest version of [Sugar]. Continue with '''[Sugar transformers]''' <> Tutorialdebian/tutorial/Sugar_transformers.txt0000644000000000000000000003767612127272672015513 0ustar * Section 0 - '''[Sugar]''' * Section 1 - '''[Sugar command macros]''' * Section 2 - '''[Sugar syntax macros]''' * Section 3 - '''[Sugar transformers]''' (what you are reading) '''Tail call optimization''' We saw two kind of macros: command macros, and syntax macros. They actually are the same thing, just the second type can be registered to be called with any command name. This kind of macros are already quite powerful, but their environment is limited to commands. To perform more complex works we need a more general source transformation tool, that can see a whole script at once, process it, and return a new one. In [Sugar] this kind of macros, are called transformer macros, and are exported to the user in two different ways: the first is an API with just two commands: * sugar::scriptToList - translates the text of a script into a Tcl list representation of this script. * sugar::listToScript - translates a list representation of a script, in the script text. There are other procedures that helps to work with the script in its list representation, but the core are this two commands. Using this API the programmer may create new versions of [proc] that are able to do some kind of transformation, to create more complex static checkers, optimizations, and many other tasks about analysis and transformation of source code. The second way to use transformers, is to register a transformer macro with the sugar::transformermacr command. It's very similar to a syntax macro, that instead to be called for every command processed, is called for every script. We will see transformer macros in the next section, for now we will see how to directly use the API to write custom versions of [proc]. To show transformers we will develop a new version of [proc], called [[tailrec_proc]], able to translate normal Tcl code that contains tail recursive calls, in a semantically equivalent script that runs with a different space complexity (and actually even a bit faster). To write transformers is exactly like to write macros, we need before to figure how to do the transformation, so the first question is if it's possible to find an algorithm to automatically find and translate tail recursive calls in Tcl code. I guess that if you are still reading this document, you perfectly know what tail recursion is, but it's better to go incremetally to be more clear. We start writing a recursive call that counts from N to 0. ====== proc counter n { puts $n if {$n} { counter [expr {$n-1}] } } ====== [[counter 3]] will output 3 2 1 0, and so on. The procedure has a recursive tail call at the end of the if block. as you can see, the [[counter $n]] call is the last call that will be executed in the procedure. This means that the procedure is semantically equivalent to this (with an immagination effort, I'll use [goto] in Tcl): ====== proc counter n { start: puts $n if {$n} { set n [expr {$n-1}] goto start } } ====== Because Tcl does not have [goto], is it still possible to jump at the start of the procedure to reiterate in some way? It is, using [while], we can write a (this time valid) Tcl procedure with the tail call optimized away: ====== proc counter n { while 1 { puts $n if {$n} { set n [expr {$n-1}] continue } break ; # note the final break } } ====== Now that the [while] 1 is sorrounding the procedure code, we can use [continue] to jump at the start of the procedure. And it will work inside conditionals: we at least need to ensure that tail calls will be optimized inside [if] branches, as nested as the programmer like, if there are tail recursive call inside. There is another problem, as we did with the 'n' variable in the previous example, we need to setup the formal arguments of the procedure to the right value before to call [continue] and jump at the start. The basic work to do is to set every parameter to the value passed at it's position in the tail call, so if the recursive tail call of a procedure foobar accepting {x y z} as arguments is called with A B C, like in: ====== foobar A B C ====== We will translate it to: ====== set x A set y B set z C continue ====== Actually, it's not enough. The following is the recursive GCD algorithm: ====== proc gcd {a b} { if {$b == 0} { return $a } else { gcd $b [expr {$a%$b}] } } ====== According to the previous rule, we may translate the recursive call to: ====== set a $b set b [expr {$a%$b}] continue ====== But this will not work, because $a gets replaced with $b before [expr] can compute $a%$b. To avoid this problem, we need to use some temp variable, and translate the script in this way: ====== set __t0 $b set __t1 [expr {$a%$b}] set a $__t0 set b $__t1 continue ====== Note that the creation of variables before a tail call is unlikely to create collision problems because even if the procedure were using a __t0 variable, it's value is no longer useful, but still in the real implementation you may want to call the temp variables __tailcall__t0 or something like this. Finally we know how our transformation will look like: * take the procedure body (fake step.. ;) * recursively search for tail calls at "top level" and inside conditionals. * translate every call as commands to set formal parameters and then call [continue] * add a 'break' command at the end of the code * use it as [while] 1 script argument And we are done. It's as simple as a nightmare to do without a macro system capable to translate the script in a simple to process form, and then back again to a script, that's why [Sugar] provides this API. Now you may wonder what's this easy to process form that is returned by sugar::scriptToList. Actually it is a list (representing the whole script) of lists (representing every command), where every element of a command is represented by a two-elements list, the first element being the type, and the second the value. The following types are possible: SPACE - a valid argument separator in a Tcl script, like " ", or "\t". TOK - a valid argument in a Tcl script, that can be just a string, or a command or variable substitution form, or any mix of they. EOL - a valid command separator in a Tcl script, like "\n", or ";". Spaces are reported varbatim, so a transformer is able to process the script without to mess with the identation. Commands may be empty, with just an element of type EOL. This happens when there are newlines or ";" between differnet commands: ====== # like this puts foo puts bar ; ; ; puts foobar ====== This means that a transformer is able to create semantic from spaces in extreme cases, like in a transformer that simulates the [Python] way to create blocks of code. That's an example of output of sugar::scriptToList. For the following code: ====== { puts [string length $foo] puts $foo$bar if {$j} { command arg arg } } ====== the equivalent translation to a list is: {EOL { }} {SPACE { }} {TOK puts} {SPACE { }} {TOK {[string length $foo]}} {EOL { }} {SPACE { }} {TOK puts} {SPACE { }} {TOK {$foo$bar}} {EOL { }} {SPACE { }} {TOK if} {SPACE { }} {TOK {{$j}}} {SPACE { }} {TOK {{ command arg arg }}} {EOL { }} {EOL {}} The API guarantees that the opposite transformation from list to a script is performed by mere concatenation of all the tokens, so "types" are information only useful for the transformer, but not for sugar::listToScript. There is an important thing to note in the example output. The code of [if] is not automagically converted to a list. This is intentional, because if you want to, you can call again scriptToList against it, or better use a normal registered transformer macro instead to use the API directly. It depends on your transformation, sometimes it's better to have only the first level, and recusively transform to scripts only the wanted parts. When instead we want to do some processing in every part of the Tcl program that is a script, we register a transformer macro, that like command and syntax macros are automatically called against arguments that are known to be scripts (thanks to the macros we already shown). (Note: there is a middle point between this two extremes: the programmer may want to perform a transformation that's not global to the whole Tcl program, so doesn't want to register a transformer macro but to use the low level API, but for the spirit of the transformation it can be useful to call it recursively in all the tokens that are scripts. Actually it's possible to register a transformer macro, then call sugar::expand $myscript, and then unregister the macro, but I'll experiment with other ways as well in the future. Another solution is to add some redundant code inside the transformer that checks if the command name is while/if/for/switch/ .... and so on, or write a function that is able to automatically recognize such commands) In the case of our transformer for tail recursive procedures the low-level API is just fine. Now we are ready to show the actual [tailrec_proc] implementation: ====== proc tailrec_proc {name arglist body} { # Convert the script into a Tcl list set l [sugar::scriptToList $body] # Convert tail calls set l [tailrec_convert_calls $name $arglist $l] # Add the final break lappend l [list {TOK break} {EOL "\n"}] # Convert it back to script set body [sugar::listToScript $l] # Add the surrounding while 1 set body "while 1 {$body}" # Call [proc] uplevel proc [list $name $arglist $body] } # Convert tail calls. Helper for tailrec_proc. # Recursively call itself on [if] script arguments. proc tailrec_convert_calls {name arglist code} { # Search the last non-null command. set lastidx -1 for {set j 0} {$j < [llength $code]} {incr j} { set cmd [lindex $code $j] if {[sugar::indexbytype $cmd TOK 0] != -1} { set lastidx $j set cmdidx [sugar::indexbytype $cmd TOK 0] } } if {$lastidx == -1} { return $code } set cmd [lindex $code $lastidx] set cmdname [lindex $cmd $cmdidx 1] if {$cmdname eq $name} { set c 0 set recargs [lrange [sugar::tokens $cmd] 1 end] foreach v $recargs { set t [list [list TOK set] [list SPACE " "] \ [list TOK __t$c] [list SPACE " "]\ [list TOK $v] [list SPACE " "]\ [list EOL "\n"]] set code [linsert $code $lastidx $t] incr c incr lastidx } set c 0 foreach a $arglist { set a [lindex $a 0] set t [list [list TOK set] [list SPACE " "] \ [list TOK $a] [list SPACE " "]\ [list TOK \$__t$c] [list SPACE " "]\ [list EOL "\n"]] set code [linsert $code $lastidx $t] incr c incr lastidx } lset code $lastidx [list [list TOK continue] [list EOL "\n"]] } elseif {$cmdname eq {if}} { for {set j 0} {$j < [llength $cmd]} {incr j} { if {[lindex $cmd $j 0] ne {TOK}} continue switch -- [lindex $cmd $j 1] { if - elseif { incr j 2 } else { incr j 1 } default { set script [lindex $code $lastidx $j 1] set scriptcode [sugar::scriptToList [lindex $script 0]] set converted [tailrec_convert_calls $name $arglist $scriptcode] lset code $lastidx $j 1 [list [sugar::listToScript $converted]] } } } } return $code } ====== Well.. I wrote a lot, to show this, but actually is very simple as you can see. Usually transformers have to deal with a particular construct, and use recursion to deal with nested levels. This transformer uses sugar::token command, it's very simple, just it takes a list representing a command and returns only the values of the elements of type TOK as a list. With this list as input: {SPACE { }} {TOK puts} {SPACE { }} {TOK {[string length $foo]}} {EOL {}} It returns this list: ====== puts {[string length $foo]} ====== It's just a facility to have access to the meaningful parts of the command in a simple way. That's a more complex example of what the macro does. Writing the following ackermann function implementation: ====== tailrec_proc ack {m n} { if {$m == 0} { return [expr {$n + 1}] } elseif {$n == 0} { ack [expr {$m - 1}] 1 } else { ack [expr {$m - 1}] [ack $m [expr {$n - 1}]] } } ====== The procedure body is translated to: ====== while 1 { if {$m == 0} { return [expr {$n + 1}] } elseif {$n == 0} { set __t0 [expr {$m - 1}] set __t1 1 set m $__t0 set n $__t1 continue } else { set __t0 [expr {$m - 1}] set __t1 [ack $m [expr {$n - 1}]] set m $__t0 set n $__t1 continue } break } ====== Of the three calls to [[ack]], only one is left (the only that is not tail-recursive). Note that it's possible to extend the macro to optimize recursive tail calls inside [switch], but actually [if] is the thing that's really needed to write most tail recursive functions. Other two examples of tail calls: ====== tailrec_proc dohanoi {n to from using} { if {$n > 0} { dohanoi [expr {$n-1}] $using $from $to moveit $from $to dohanoi [expr {$n-1}] $to $using $from } } ====== and ====== tailrec_proc show_list l { if {[llength $l]} { puts [lindex $l 0] show_list [lrange $l 1 end] } } ====== The programmer should care to don't use [return], because the macro for semplicity does not check for "return [call ...]": with tail recursive calls return is always not need. '''4) Transformer macros allows to implement features that may otherwise require changes to the compiler itself. It's possible, for example, to translate a program in one semantically equivalent, but with different space complexity.''' '''[if] with C-like block indentation''' While it's quite ideal to build a specialized version of [proc], the API from transformer should not be called to perform general transformations in the whole source code at all levels. Instead, it's possible to register a transformer macro with this API: ====== sugar::transformermacro foobar list { ... do something with the list representation of the script ... return $list } ====== The transformer macro will be called for every part of the script that is recognized as a script, and is re-called after other macro expansions (actually all the macros are guaranteed to be called in turn until there is something to expand, so you don't have to care about the registered order, at the same time you *can't* write different macros conceived to run at a given order, the behaviour of every macro must be self-contained). The following is an example of a simple transformer macro that translates occurrences of [if] indented with a newline before the expression and the body: ====== if {$test} { set a [+ 1 2]; # Of course, + can be a simple command macro. } ====== The above code will be translated to: ====== if {$test} \ { set a .... } ====== It's a very *bad* example of what you can do with transfomers :) The macro doesn't implement support for elseif/else and so on, but it's trivial to add. That's the implementation: ====== sugar::transformermacro sillyindentation list { for {set i 0} {$i < [llength $list]} {incr i} { set tokens [::sugar::tokens [lindex $list $i]] if {[llength $tokens] == 2 && [lindex $tokens 0] eq {if}} { set nexttokens [::sugar::tokens [lindex $list [expr {$i+1}]]] if {[llength $nexttokens] == 1 && [string index [lindex $nexttokens 0] 0] eq "\{"} { lset list $i end 1 " " } } } return $list } ====== That's all for now. Positive and negative feedbacks are welcomed. <> Tutorialdebian/tutorial/Sugar_command_macros.txt0000644000000000000000000005654012127272667015743 0ustar *** NOTE *** The API changed in two ways, you should be aware of this if you want to try with the current release of sugar what is shown in this tutorial. First Change - Now macros are expanded only inside procedures defined with ::sugar::proc Second Change - Now macros get a list of arguments like normal procedures, but the first argument is the macro name itself. All the macros in this tutorial will work substituting the 'argv' argument with 'args'. * Section 0 - '''[Sugar]''' * Section 1 - '''[Sugar command macros]''' (what you are reading) * Section 2 - '''[Sugar syntax macros]''' * Section 3 - '''[Sugar transformers]''' '''What is a Tcl macro'''. A macro is an operator implemented by transformation. Macros are procedures that generate a Tcl program at compile time, and substitute it where the programmer used thier name as command. It's a simple concept if explained by examples. Suppose you want to write a [[clear]] command, that set the variable name passed as it's unique argument, to a null string. Implemented as a procedure using [upvar] it is like this: ====== proc clear varName { upvar 1 $varName var set var {} } ====== Every time the user type ====== clear myvar ====== in a program, the [[clear]] procedure will be called, and the variable myvar of the caller procedure will be set to a null string. As an alternative to call a procedure that is able to alter the caller's execution environment, we may want to automatically substitute every occurrence of the command [[clear ]] with [[set {}]]. So basically we want that when we write ====== clearn myvar ====== in a program, it is substitute with ====== set myvar {} ====== as if the programmer had really typed "set myvar {}" instead of "clear myvar". That's the goal of the simplest form of a [Sugar]'s macro. The definition of a new macro is very similar to the creation of a procedure. The following is the implementation of [[clear]] as a macro: ====== sugar::macro clear argv { list set [lindex $argv 1] {{}} } ====== It means: "If you encounter a command called 'clear' inside the source code, call the following procedure putting all the parts of which the command is composed in $argv, and substitute the occurrence of the clear command and arguments, with what the procedure will return." Again, with other words: So, what happens is that when a procedure is compiled, for every occurrence of the [[clear]] command inside the procedure, the above procedure is called, with $argv set to a list that represents the arguments used to call the macro (including the macro name itself as first argument). The result value of the function, that should be a list of the same form, is substituted in place of the original macro call. To make the example more concrete, see the following code: ====== proc foobar { set x 10 clear x } ====== Before to compile the procedure, Tcl will call the macro we defined with $argv set to the two elements list {clear x} (verbatim). That procedure returns ====== list set [lindex $argv 1] {{}} ====== so for the argument {clear x} it will return the list {set x {}}. This return value will be substituted in place of "clear x". Actually, after the proc was defined, we can use [[info body]] to check what happened with the macro: ====== info body proc ====== will output ====== set x 10 set x {} ====== At this point it's possible to use the [[clear]] macro as it was a Tcl procedure. But Tcl has [[uplevel]] and [[upvar]], so for what macros are useful? Fortuantely they allows for many interesting things not possible at all otherwise, still, this example shows the first big advantage of macros: '''1) Macros makes Tcl faster, without to force the user to inline code by hand.''' The [[clear]] command implemented as macro runs 3 times faster in my Tcl 8.4. Also, being [[upvar]] one of the biggest obstacles in the ability of the Tcl compiler to optimize Tcl bytecode, it's not impossible that at some point Tcl will be able to run much faster if the user will ensure a given procedure is never target of [[upvar]]. Simple commands that involve the use of upvar can be even more simple to write implemented as macros. The following are four examples: ====== # [first $list] - expands to [lindex $list 0] sugar::macro first argv { list lindex [lindex $argv 1] 0 } # [rest $list] - expands to [lrange $list 1 end] sugar::macro rest argv { list lrange [lindex $argv 1] 1 end } # [last $list] - expands to [lindex $list end] sugar::macro last argv { list lindex [lindex $argv 1] end } # [drop $list] - expands to [lrange $list 0 end-1] sugar::macro drop argv { list lrange [lindex $argv 1] 0 end-1 } ====== [Sugar] supports three types of macros. We are dealing with the simplest and more common macros: command macros. The other two types, syntax macros, and transformers, will be covered later. For now let's go to create a more complex macro. '''A more complex example''' Good macros do source code transformation in a smart way, they turn a form that is undestood by the programmer, to one that is also understood by the compiler, that's hard to type and use in raw form without the macro support, but optimal otherwise. Ideally a macro should expand to a single command call (possibly including many other nested), and should not expand to code that magically creates variables at runtime to store intermediate results all the times it can be avoided (because there may be collisions with variables in the function, or created by other bad macros. Btw, in the TODO list of sugar there is a way to generate unique local variable names). If the macro is well written, then the programmer can use it like any other command without to care much. We will see a more real example of macro that implements an very efficient [[lpop]] operator. It accepts only one argument, the name of a variable, and returns the last element of the list stored inside the given variable. As side effect, [[lpop]] removes the last element from the list. (it's something like the complementar of [lappend]). A pure-Tcl implementation is the following: ====== proc lpop listVar { upvar 1 $listVar list set res [lindex $list end] set list [lrange $list 1 end] return $res } ====== This version of lpop is really too slow, in fact when [[lrange]] is called, it creates a new list object even if the original one, stored in the $list variable, is going to be freed and replaced by the copy. To modify the list on-place is far better. The [[lrange]] implementation is able to perform this optimization if the object in "not shared" (if you don't know about this stuff try to read the Wiki page about the [K] operator before to continue) So it's better to write the proc using the [K] operator. The lrange line should be changed to this: ====== set list [lrange [K $list [set list ""]] 1 end] ====== With K being: ====== proc K {x y} { return $x } ====== But even to call [K] is costly in terms of performace, so why don't inline it also? To do it requires to change the previous lrange line to this: ====== set list [lrange [lindex [list $list [set list ""]] 0] 1 end] ====== That's really a mess to read, but works at different speed, and even more important, at different time complexity!. Writing a macro for [[lpop]] we can go even faster, and, at the same time, we can have this code more easy to maintain and read. Actually macros are allowed to expand to commands containing other macros, and so on. This means that we can write a macro for every single step of [[lpop]]. We need the [[first]] [[last]] and [[drop]] macros already developed, and a macro for [K]: ====== sugar::macro K argv { foreach {x y} $argv break list first "[list $x $y]" } ====== Note that we used [foreach] instead of two calls to [lindex] that's probably faster. But remember that macros don't have to be fast in the *generation* of the expanded code. This will expand [[K $x $y]] into [[first [[list $x $y]]]], that will be expanded in [[lindex [[list $x $y]] 0]]. We have one last problem. Even after the optimization and the use of [[K]] inline, the procedure above required a local variable 'res' to save the last argument of the list before to modify it, and use $res later as return value for the procedure. We don't want to create local vars into the code that calls the [[lpop]] macro, nor we want to expand to more than a single command. The [K] operator can help us to do so. Instead to use: ====== set res [lindex $list end] set list [lrange [lindex [list $list [set list ""]] 0] 1 end] return $res ====== why don't just write: ====== K [lindex $list end] [set list [lrange [lindex [list $list [set list ""]] 0] 1 end]] ====== That's ok, but what an unreadable code! Thanks to amcros we can abstract from the fact that to call procedures is slow, so we just write: ====== [K [last $list] [set list [rest [K $list [set list ""]]]]] ====== Will not win the clean-code context this year, but it's much better than the previous. Ok... now we want a macro that, every time we type "lpop $list", will expand in the above line: ====== sugar::macro lpop argv { set varname [lindex $argv 1] set argv [list \ K \ {[last $%varname%]} \ {[set list [drop [K $%varname% [set %varname% ""]]]]} \ ] foreach i {1 2} { lset argv $i [string map [list %varname% $varname] [lindex $argv $i]] } return $argv } ====== There are few things to note about this code. The macro returns a list, where every element is a token of a Tcl command in the source code. This does not mean we have to transform in lists even arguments that happens to represent a script. Also note that the input list of the macro is just a list of tokens that are *exactly* what the user typed they in the source code, verbatim. What follows is that the tokens are already quoted and valid representations of a procedure argument. We don't need to care about the fact that they must be interpreted as a single argument like if we were generating code to pass to [eval]. This allows the macro developer to use templates for macros, in fact the [[lpop]] macro is just using a three argument template, and the final foreach will substitute the arguments that needs to refer to the variable name, with that name. You don't have to care what that variable name is. It can be a complex string formed by more commands, vars, and so on [[like]][[this]]$and-this. If it was a single argument in the source code, it will be in the macro after the expansion. Another interesting thing to note is that we don't really have to return every token as a different element of the list. In pratice we can return it even as a single-element list. The rule is that the macro expander will care to put an argument separator like a tab, or a space, for every element of the list, and put a command separator like newline or ";" at the end. If we put spaces ourself, we can just return a single element list. So, the lpop macro can also by written in this way: ====== sugar::macro lpop argv { set varname [lindex $argv 1] set cmd [format { K [last $%varname%] [set list [drop [K $%varname% [set %varname% ""]]]] } $varname $varname $varname] return [list $cmd] } ====== This is much more simple and clean, and actually it's possible to use this style. The difference is that returning every token as a different element of a list makes [Sugar] macros able to left the indentation of the original code unaltered. This is helpful both to take procedure error's line numbers correct, and to see a good locking output of [info body]. But as long as most macros are about commands that are just typed in the same line together with all the arguments, for many macros is just a matter of tastes. If you are implementing control structures that are: ====== indented {in} { this way } ====== It's another question, and it's better to return every token as a list element. '''Number of argument and other static checks in macros''' Macros expand to code that will raise an error if the number of arguents is wrong in most cases, but it's possible to add this control inside the macro. Actually it's a big advantage of macros because they are able to signal a bad number of arguments at run time: this can help to write applications that are more reliable. It's even possible to write a macro that expands to exactly what the user typed in, but as side effect does a static check for bad number (or format) of arguments: ====== sugar::macro set argv { if {[llength $argv] != 3 || [llength $argv] != 2} { error "Bad number of arguments for set" } return $argv } ====== This macro returns $argv itself, so it's an identity transformation, but will raise errors for [[set]] with a bad number of arguments even for code that will never be reached in the application. Note that the previous macro for set is a bit incomplete: to get it right we should add checks for arguments that starts with [{*}], for this reason [Sugar] will provide a function to automatically search for a bad number of arguments in some next version. Note that {*} introduces for the first time the possibility for a command to get a number of arguments that is non evident reading the source code but computed at runtime. Actually {*} is an advantage for static checks because before of it the way to go was [[eval]], that does totally "hide" the called command postponing all the work at run-time. With {*} it's always possible to say from the source code that a command is called with *at least* N arguments. Still, to add new syntax to Tcl will probably not play well with macros and other form of source code processing. Identity macros are very powerful to perform static syntax checks, they can not only warn on bad number of arguments, but with the type of this arguments. See for example the following identity macro for "string is": ====== proc valid_string_class class { set classes {alnum alpha ascii control boolean digit double false graph integer lower print punct space true upper wordchar xdigit} set first [string index $class 0] if {$first eq {$}} {return 1} if {$first eq {[}} {return 1} if {[lsearch $classes $class] != -1} {return 1} return 0 } sugar::macro string argv { if {[lindex $argv 1] eq {is} && [llength $argv] > 2} { if {![valid_string_class [lindex $argv 2]]} { puts stderr "Warning: invalid string class in procedure [sugar::currentProcName]" } } return $argv } ====== Thanks to this macro it's possible to ensure that errors like to write [[string is number]] instead [[string is integer]] are discovered at compile-time. In this respect macros can be seen as a programmable static syntax checker for Tcl. We will see how "syntax macros" are even more useful in this respect. This is the second feature that macros add to Tcl: '''2) Macros are a powerful programmable static checker for Tcl scripts.''' Actually I think it's worth to use macros even only for this during the development process, and than flush they away. '''Conditional compilation''' That's small and neat: we can write a simple macro that expands to some code only if a global variable is set to non-zero. Let's write this macro that we call [[debug]]. ====== sugar::macro debug argv { if {$::debug_mode} { list if 1 [lindex $argv 1] } else { list } } ====== Than you can use it in your application like if it was a conditional: ====== # Your application ... debug { set c 0 } while 1 { debug { incr c if {$c > 100} { error "Too many iteractions..." } } .... do something .... } ====== All the [[debug {someting}]] commands are compiled as [[if 1 {something}]] if the ::debug_mode variable is true. Instead if this var is false, they will not be compiled at all. That's the simplest example, you can write similar macros like [[ifunix]], [[ifwindows]], [[ifmac]], or even to expand to different procedures call if a given command is called with 2, 3 or 4 arguments. The limit is the immagination. '''New control stuctures''' Not all the programming languages allow to write new control structures. Tcl is one of this better languages that don't put the programmer inside a jail, but, not all the programming languages that allows to write new control structures, are able to make they efficient. Tcl macros can make new control sturctures as fast as byte compiled control structures, because user defined ones are usually syntax glue for code transformations. Being macro transformers that translates a from to another, that's a good fit for macros. That's a macro for the ?: operator. ====== # ?: expands # ?: cond val1 val2 # to # if $cond {format val1} {format val2} sugar::macro ?: argv { if {[llength $argv] != 4} { error "Wrong number of arguments" } foreach {_ cond val1 val2} $argv break list if $cond [list [list format $val1]] [list [list format $val2]] } ====== The macro's comment shows the expansion performed. Being it translated to an [if] command, it's as fast as a Tcl builtin. '''How macros knows what's a script?''' I Tcl there are no types, nor special syntaxes for what is code and what is just a string, so you may wonder why macros are not expanded in the following code: ====== puts { set foo {1 2 3}; [first $foo] } ====== But they are expanded in this: ====== while 1 { set foo {1 2 3}; [first $foo] } ====== I guess this is one of the main problems developers face designing a macro system for Tcl, and even one of the better helpers of the idea that a good macro system for Tcl is impossible because you can't say what is code and what isn't. [Sugar] was designed to address this problem in the simplest possible of the ways: because it can't say if an argument is a script or not, macro expansion is not performed in arguments, so in theory [Sugar] will not expand the code that's argument to [puts], nor [while]. But of course, in the real world for a macro system to be usable, macros should be expanded inside the [while], and not expanded in [puts], so the idea is that for commands you know the argument is a script, you write a macro that returns the same command but with script arguments macro-expanded. It is very simple and in pratice this works well. For example that's the macro for [while]: ====== sugar::macro while argv { lset argv 1 [sugar::expandExprToken [lindex $argv 1]] lset argv 2 [sugar::expandScriptToken [lindex $argv 2]] } ====== That's the macro for [if]: ====== sugar::macro if argv { lappend newargv [lindex $argv 0] lappend newargv [sugar::expandExprToken [lindex $argv 1]] set argv [lrange $argv 2 end] foreach a $argv { switch -- $a { else - elseif { lappend newargv $a } default { lappend newargv [sugar::expandScriptToken $a] } } } return $newargv } ====== As you can see [Sugar] exports an API to perform expansion in Tcl scripts and Expr expressions. There are similar macros for [switch], [for], and so on. If you write a new conditional or loop command with macros, you don't need it at all because the macro will translate to code that contains some form of a well known built-in conditional or loop command, and we already have macros for this (remember that macros can return code with macros). If you write any other command that accept as arguments a Tcl script or expr expression, just write a little macro for it to do macro expansion. This has a nice side effect: ====== proc nomacro script { uplevel 1 $script } ====== Don't write a macro for nomacro, and you have a ready-to-use command that works as a barrier for macro expansion. Continue with section 2 - '''[Sugar syntax macros]''' ---- [WHD]: This is very cool, but I have to ask--why not allow macros to have a standard Tcl argument list? That is, ====== sugar::macro mymacro {args} {...} ====== Gives the behavior you describe here, while ====== sugar::macro mymacro {a b c} {...} ====== explicitly creates a macro that takes three arguments and will generate a standard error message if you supply some other number? ---- [SS]: This can be a good idea, being always possible to use 'args' as only argument to have the current behaviour. I used a single list as input mainly because the same macro can have more then a name, and in order to have the same interface for both command macros and syntax macros. For example: ====== sugar::macro {* + - /} argv { list expr [list [join [lrange $argv 1 end] " [lindex $argv 0] "]] } ====== Will handle * + - / with the same code. Macros with more than a name may in extreme cases even give different meanings for arguments in the same position. Btw there is 'args' for this case. So I can change the API to something like this: ====== sugar::macro {name arg1 arg2 ...} {...} ====== That's like a Tcl proc, but with the name that was used to call the macro as the first argument. For syntax macros this format actually may not have a lot of sense, but there is still 'args'. I'll include this chage in the next version if I'll not receive feedbacks against it. Thanks for the feedback WHD. [WHD]: I think that on the whole I prefer the previous syntax for command macros; the macro can always have an implicit argument that is the macro name. For example, ====== # Identity macro sugar::macro {+ - * /} {args} { return "$macroname $args" } ====== [JMN]: I'd just like to add my vote for removing the macroname as first argument syntax. From my hacking about, it seems easy to make it implicitly available more or less as WHD suggests. (I don't *think* I broke anything.. ) ---- [SS]: For a different question about the sugar API, I wonder if Tclers interested in this macro system feel better the current redefinition of [proc], or if it's better to provide a sugar::proc procedure that's exactly like [proc] but with macro expansion. If the API will remain the current with [proc] redefined, I'll add in the wrapper an option -nomacro that will just call the original proc. Please add your name with optional motivation below. Yes, I think it's better to wrapper the real [proc]: * Put your name here if you are for this solution. No, I want macro expansion only using sugar::proc: * [SS] (avoid to waste CPU time for procs that don't use macros, this can be a big difference if you [package require] sugar before Tk or other big packages) * [DKF]: Avoiding overriding the real [proc] allows packages to use sugar if they want without surprising packages that don't expect it. Packages that do want it can just do [[[namespace import] ::sugar::proc]] into their own private workspace. [WHD]: Since you have to override the standard control structures to make macros work, it seems to me that what you really need is a pair of commands: ====== sugar::configure -enabled 1 # Macros expanded in body proc myproc {a b c} {....} # Macros expanded in expression and body while {$a > $b} {....} sugar::configure -enabled 0 # Macros no long expanded. ====== [SS]: Actually [Sugar] does not override nothing! (so it will expand all at compile time, no run-time overhead). It does expansion inside control structures just using macros for [while] and so on. In this page this is better explained in the section: '''How macros knows what's a script?'''. So to override [proc], or to provide a private proc-like command is just a matter of design (or tastes), all will work in both the cases. <> Dev. Toolsdebian/tutorial/Sugar.txt0000644000000000000000000000441312127272665012667 0ustar '''News''' 0.1 version released. This is the first real release in form of a Tcl package. See the download section for the download URL. '''Introduction''' [Sugar] is a macro system for the Tcl programming language, with a design very similar to [Lisp] macros. This document will try to explain what [Sugar] macros are, how they can be applied, and why I think they improve the Tcl language. Every time I refer to the term [macro] in this document, I mean a [Sugar] Tcl macro if not better specified. Note that the idea of a macro system for Tcl is not new, but as far as I know, there is no implementation that is comparable with [Sugar] in terms of usage and design principles. Something of similar in the spirit, is the [Tmac - a Tcl macro processor package], but it works by pattern matching and substitution, requires the user to use a special syntax, does not guarantee the expansion to happen only where actually a command can be called, and can't be used to add arbitrary new syntax to Tcl (syntax macros feature of [Sugar]), nor complex code transformation (see for example the transformer macro to optimize tail calls in this document). '''Programmer API''' The following sections will explain the basic [Sugar] API with some detail, but it's worth to say now that macro expansion happens when the [proc] command is called. Actually [Sugar] is a wrapper for proc, an API to write source code transformations, and some command to register this transformations in the modified version of [proc]. '''Sections''' * Section 0 - '''[Sugar]''' (what you are reading) * Section 1 - '''[Sugar command macros]''' * Section 2 - '''[Sugar syntax macros]''' * Section 3 - '''[Sugar transformers]''' You may also want to check the [Sugar macros collection] page. '''Authors''' I'm ([Salvatore Sanfilippo]) currently the only author of [Sugar]. '''Download''' You can get the latest version of [Sugar] from http://www.hping.org/tclsbignum/sugar-0.1.tar.gz If you have suggestions, bugfixes or API changes requests, please drop me an email at antirez at invece dot org. See also [run]. '''Disambiguation''' http://www.sugarcrm.com/crm/%|%SUGAR%|% is also an open source CRM (Customer Relationship Management) application. <> Application | Dev. Tools | String Processingdebian/get-wiki-docs0000755000000000000000000000070512127272660011571 0ustar #!/bin/sh resolve_wiki_name () { wget -O/dev/null http://wiki.tcl.tk/$1 2>&1 | grep -o http.\* | tail -n1 } HOMEURL=http://wiki.tcl.tk/11155 mkdir -p debian/tutorial cd debian/tutorial rm -f *.txt for d in $(wget $HOMEURL.txt -O- 2>/dev/null | sed -n -e '/Section /{s/.*\[\(.*\)\].*/\1/;s/ /_/g;p}') do echo Receiving $d.txt >&2 wget $(resolve_wiki_name $(echo $d | tr _ +)).txt -O$d.txt 2>/dev/null done rm -f $(basename $HOMEURL.txt) debian/compat0000644000000000000000000000000212127254176010373 0ustar 9