(david-mcneil.com :blog)

Understanding Datomic?

Sat, 17 Mar 2012 10:56:00 -0500

In the Datomic unsession last night at ClojureWest Stu and Rich hosted a Q&A on Datomic. I appreciated their time and I now feel I have a better grasp of what they are doing with Datomic.

In particular I felt enlightened by Stu’s question: “which provides faster data access, a fast local spinning drive or an SSD attached via a fat network pipe?”

I hadn’t focused on this aspect of their approach. In particular I was not aware that Amazon’s DynamoDB is SSD based.

So the way I understand what Datomic does is that it allows the traditional database server to be distributed across many server machines (i.e. Datomic peers). Writes still go through a single server for consistency, but reads can go against a cluster of machines. This is possible because Amazon’s SSD backed DynamoDB offers network access to storage at speeds that historically required local disks. So as I understand it:

Datomic is at the front of a technology wave that is crashing. Spinning disks are out and SSDs are in.
We can practically think of a distributed SSD cluster on a fat network pipe as the equivalent of a shared hard drive in terms of performance.
This means that we can fire up many servers to read from that “shared & distributed” drive.
Once we can start up many servers we have a distributed database server that offers ACID writes through a single server with failover.
Now that we have a distributed data server we have an abundance of processing on the database server.
With an abundance of database server processing power we can move the application code onto the database servers.

To my way of thinking the description above makes it more clear to me what Datomic is enabling and why it is possible. It seems many people that talk about Datomic get hung up on thinking of Datomic peers as traditional clients. To me it seems that Datomic peers are more like traditional database servers.

I wonder if my description above is correct? I wonder if positioning peers as servers would be an easier “sell”?

Using Datomic in Clojure: Baby Steps

Sat, 17 Mar 2012 10:12:00 -0500

I ported over the first few steps of the in-memory Datomic tutorial to Clojure: https://gist.github.com/2060731

To run this code do something like:

download the datomic zip
follow the datomic README to install the datomic jar to your local maven repo (in the mvn incantation change “datomic.jar” to match the local file name from the datomic zip)
setup a lein project something like (I looked in the datomic pom.xml file to see what version of Clojure was required):

(defproject datomic-tutorial "1.0.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.4.0-beta3"]
                 [com.datomic/datomic "0.1.2753"]]
  :dev-dependencies [[swank-clojure "1.4.0-SNAPSHOT"]])

At first I tried to run this on a version of Clojure 1.3 and nothing worked.

I assume that there are Clojure protocols that we can use instead of the Java interfaces but I haven’t tracked them down yet.

Clojure Robot DSL

Sat, 04 Feb 2012 13:28:00 -0600

The code from my “Building a DSL in Clojure For Controlling a Lego MindStorm / Arduino Robot” talk at Lambda Lounge on February 2, 2012 is up on github.

The talk had a few points:

playing with Lego MindStorm and Arduino based robots is great fun
they are very effective tools for teaching kids to program and to start to understand engineering
a DSL built in Clojure can give kids the the same level of “tinkerability” in the programming language as they have with the hardware
creating a new type in Clojure (a List object that is associative) is a way to get a featureful DSL with very little code

The first file to checkout is op_demo.clj. This file introduces the new Clojure type that the DSL is built on, a Clojure “operator”. An operator is similar to a Clojure record, except it presents itself as a List instead of as a Map.

With that as our base, we can now introduce the operators that will make up our language.

Let’s see what we can do with these operators:

This example shows a technique for creating a DSL that nestles tightly down into Clojure.

No parser is needed; the Clojure reader reads the program.
No compiler is needed; the Clojure evaluator will walk the tree of expressions.
We get macros for free.
Our programs are data that can be programatically accessed.
We can implement multiple targets for the language by implementing the operators in multiple namespaces.

With respect to teaching kids to program, this approach is a natural fit to the robots. The kids can:

Define the language they will use. This can be made simpler for younger kids and more advanced for older kids.
Implement the operators for their language.
Learn that programs are just data to be generated and manipulated.
Go as deep as they care to with creating a real compiler by defining operators for control-flow, variable bindings, etc.
Get hooked on Lisp!

Clojure: Creating a Custom Map Type

Thu, 26 Jan 2012 14:47:00 -0600

As presented at today’s St. Louis Clojure Cljub meeting.

Clojure defrecord2 code update

Sat, 07 May 2011 19:17:47 -0500

I updated the defrecord2 code. It now includes the following:

record zipper support
record matchure support
dissoc2 preserves record type
universal record constructor
defined record? predicate

defrecord improvements - feedback

Wed, 06 Apr 2011 19:37:00 -0500

In the course of doing Clojure development I have made extensive use of records and have extended them in many was. So I was excited to see a proposal from the Clojure Core team for defrecord improvements. It looks like a good start, below are my thoughts and questions about the proposal.

1) Variety of forms

One question that arises is how to understand the difference between the various forms. For example these two forms:

#myns.MyRecord[1 2]

and

(MyRecord. 1 2)

As best I can understand it, the first form would have the validation function applied but the second would not.

Furthermore as a literal form, the first can only include constants. So for instance the following use of a function call would not be valid in the first syntax:

(MyRecord. 1 (+ 1 1))

If you need to write expressions to compute field values and want the validation function to be applied then you use one of the factory functions:

(myns/->MyRecord 1 (+ 1 1))

I assume that the position forms will require all of the fields to be provided? If so, the initialization values will only be relevant to the map forms.

My understanding of the proposal is that the literal record syntax is a general syntax for Java objects. So the object created by

(java.util.Locale. "en" "US")

could be expressed in the literal syntax as

#java.util.Locale["en" "US"]

However, it doesn’t seem to me to be generally possible to know which contructor to use for a given object so I am not sure when this literal syntax will be used for printing (non-record) Java objects.

2) Factory function naming

I realize that the names in the proposals are just placeholders… my preference is to not use “->” (which could easily be confused with the threading macro “->”) in the names and to name the functions with lower-case-dashed versions of the record names instead of CamelCase versions. For my sense of aesthetics this makes the use of record factory functions look like “normal” function calls. Furthermore, I value the ability to specify the name of the factory functions at the time the record is defined.

3) Validation

One of the forms of validation that we have found particularly helpful is to validate the names of the fields passed into the map factory function. If the map contains a key that is not a record field name then an exception is thrown. It is possible to add additional, non field-name keys, with assoc.

In addition I think it is useful to allow a validation function to be defined as part of the defrecord.

4) Writing records

I value an option to exclude the namespace from the printed from of the records. Instead of this:

(myns/->MyRecord 1 2)

They would optionally print as:

(->MyRecord 1 2)

This is useful when printing deeply nested trees of records because it trims a potentially long namespace identifier from every object’s output.

Finally, we have found it useful to suppress nil values when printing records with the map factory form. Again this makes the output less verbose.

5) Mutators

Beyond the initial creation of records we have found it useful to provide functions to create new record objects from existing objects. For example, the syntax could be:

(def x (myns/map->MyRecord {:a 1}))
(myns/map->MyRecord x {:b 2})
;; -> (myns/->MyRecord 1 2)

By virtue of going through the factory function the validations are applied. As opposed to using assoc directly in which case the validations are not applied.

Related to this is the idea of a universal constructor, e.g. named “new-record”:

(def x (myns/map->MyRecord {:a 1}))
(new-record x {:b 2})
;; -> (myns/->MyRecord 1 2)

This allows a new record object to be created from a record object without knowing the type of the object. We have found this useful for writing generic code to handle record objects.

Finally we have found it useful to define a dissoc function that removes a key from a record object, but produces a record object as the result.

So instead of this default behavior from Clojure:

(class (dissoc (map->MyRecord {:a 1 :b 2}) :b))
;; -> clojure.lang.PersistentArrayMap

We would get:

(class (dissoc2 (map->MyRecord {:a 1 :b 2}) :b))
;; -> myns.MyRecord

6) record?

We have found it useful to define a record? predicate function that reports whether a given object is a record object:

(record? (map->MyRecord {:a 1 :b 2}))
;; -> true
(record? "hello")
;; -> false

7) walking records

We have extended defrecord to define prewalk and postwalk support and this has proven useful (despite the fact that pre/postwalk are semi-deprecated).

8) zipper support

We have extended defrecord to generate multi-method implementations for each record class to participate in a zip-record function that allows zippers to be used to navigate record trees. We have used this feature extensively in our product to manipulate record trees.

9) matchure support

We have extended defrecord to support matchure. Specifically all records participate in a multi-method that allows them to be used with a “match-record” of our creation that delegates to matchure if-match . For example:

(match-record [(map->MyRecord {:a 1 :b ?b})
               (map->MyRecord {:a 1 :b 2})]
                           b)
;; -> 2

Clojure Protocol Adapters

Thu, 24 Feb 2011 21:13:00 -0600

Once you start using Clojure protocols to capture abstractions it is natural to want to define implementations of higher level protocols in terms of the lower level protocols. But, Clojure does not allow protocols to be extended to other protocols. At the Clojure Conj in 2010 Rich Hickey mentioned an approach to this problem in which a protocol is extended to Object as a “catch all”. Then if the protocol is used with an object that satisfies protocol X dynamically extend the class of the object to protocol Y.

For example, consider a low-level protocol for a Dog:

(defprotocol Dog
  (bark [_]))

And a higher level protocol for an Animal:

(defprotocol Animal
  (speak [_]))

We would like to be able to write something like the following to define how a Dog can participate in the Animal protocol.

(adapt-protocol Dog Animal
                (speak [dog]
                       (bark dog)))

I have written a module, named clojure-adapt, that provides this adapt-protocol capability.

The adapt-protocol call registers the adapter functions in a global map that is keyed by the protocols Animal and Dog. The Animal protocol is extended to the base Object class with implementation functions that consult the global adapter map and dynamically extend the Animal protocol to the classes of objects that satisfy the Dog protocol.

If we have an object that satisfies the Dog protocol, for example, a String:

(extend-protocol Dog String
                 (bark [s] (str "arf " s)))

Then we can use the Animal functions on the Dog:

(speak "Fido")
=> "arf Fido"

The adapters are only used if the object does not satisfy the protocol. So if a protocol is extended to a class, then the adapters are not used on that class even if objects of the class satisfy the protocol being adapted.

For example, if the Animal protocol and the Dog protocol are extended to the Date class the adapter is not used.

(extend-protocol Dog java.util.Date
                 (bark [d] (str "dog as of " (.getTime d))))

(extend-protocol Animal java.util.Date
                 (speak [d] (str "animal as of " (.getTime d))))

(bark (java.util.Date. (long 100)))
=> "dog as of 100"

(speak (java.util.Date. (long 100)))
=> "animal as of 100"

When the Animal function, speak, is called on a Date object the adapter is not used even though the Date satisfies the Dog protocol. This is because the Date class is already participating in the Animal protocol.

The adapt-protocol call is tricky to use during development because once an adapter is “installed” for a class subsequent calls to adapt-protocol do not affect the class. One work-around to this is to refine an adapter by using extend-protocol with a test class. Once the adapter is working properly then register it for use via adapt-protocol.

Implementation "inheritance" in Clojure

Wed, 03 Nov 2010 20:36:00 -0500

Clojure records and protocols eschew inheritance. For details see the “Datatypes and protocols are opinionated” section on the Clojure datatypes page.

If you are used to Java style type inheritance you might be surprised that there is no explicit record/protocol mechanism for defining one type as a “sub-type” of another and inheriting the super-type’s implementation. You might even think that Clojure datatypes are less powerful than Java clasess… but you would be wrong.

The trick is that Clojure allows the implementation of a protocol to be specified as a map. Just a simple, standard Clojure map that maps function names to implementations. Since implementations are expressed as maps, the surprising place to look for information on implementation “inheritance” is just the functions that operate on maps. For example: assoc and merge.

Notice in the example that a TrainedDog contains a Dog as a member. This, combined with an implementation of to-dog that returns the TrainedDog’s Dog, allows the functions defined in base-behavior to also operate on TrainedDogs.

Clojure maps are constructed using the full power of the Clojure language. Maps can be combined in ways that emulate Java’s single inheritance model or more complex multiple inheritance models and mixins. Combine maps in any way needed to do what you need.

This is a great example of embracing standard Clojure datatypes as the means of abstraction for a problem (in this case, the problem is “how to specify type implementations”) and gaining the full power of Clojure as a consequence.

(This was presented at the October 2010 Clojure Cljub )

Clojure dynamic variables and Java ThreadLocals

Sat, 30 Oct 2010 08:54:05 -0500

At Clojure Cljub we talked about dynamic variables being like Java ThreadLocals. Which led to the question of whether they were really ThreadLocals. After a cursory look at the Clojure 1.2 source, the answer appears to be: “yes”. Each binding is not itself a ThreadLocal, but the dynamic values of a Var appear to be rooted in a ThreadLocal named “dvals”.

From src/jvm/clojure/lang/Var.arg

public final class Var extends ARef implements IFn, IRef, Settable{
  ...
  static ThreadLocal<Frame> dvals = new ThreadLocal<Frame>(){
  ....
  public static Associative getThreadBindings(){
    Frame f = dvals.get();
    IPersistentMap ret = PersistentHashMap.EMPTY;
    for(ISeq bs = f.bindings.seq(); bs != null; bs = bs.next())
      {
        IMapEntry e = (IMapEntry) bs.first();
        Var v = (Var) e.key();
        Box b = (Box) e.val();
        ret = ret.assoc(v, b.val);
      }
      return ret;
    }

Create default xmodmap

Thu, 28 Oct 2010 13:29:54 -0500

From http://blacketernal.wordpress.com/set-up-key-mappings-with-xmodmap/

Run

xmodmap -pke > default-modmap
xmodmap -pm >> default-modmap

Then edit the end of the default-modmap file to be formatted like this:

add shift =      Shift_L  Shift_R
add lock  =      Caps_Lock

Then the resulting file can be loaded via:

xmodmap default-modmap

This is useful for restoring the default key maps after you have messed them up. This is particularly useful if your custom xmodmap file is not idempotent.

clojure-conj day 2 - notes

Mon, 25 Oct 2010 10:19:00 -0500

Aaron Bedra - Clojure in the field

A couple of the main issues to address:

Web frameworks - too many to pick from, need to coordinate these and need better docs
Dependency management - Lein is a good start, but we need more. Also Clojars is too hard to search.

Stuart Sierra - Macro Club

slides

clojure.test uses macros. This is confusing at times. For example, “thrown?” is not a var. It is interpreted by the “is” macro. This is surprising because it looks like a function call when it is used and this is one of the down-sides of macros: surprising behavior.

lazytest is Stuart’s replacement for clojure.test. He went through macros, protocols, etc. looking for good abstractions for tests and ended up falling back to the powerful, core Clojure constructs:

functions - represent tests
sequences - represent test suites

He is still struggling with test contexts that perform setup/teardown around tests. The problem is having a good way to get the context state into the tests. There are macros to accomplish this but they are surprising because they establish new scoping rules.

In summary:

use Clojure’s core types as abstractions (functions are the most powerful)
use macros for sugar
don’t violate user’s expectations

Lightning Talks

Zach from Runa talked about Aleph. An asynchronous web server built on “channels” which are unidirectional pub/sub queues.

Alex Miller from Revelytix described how to use Clojure zippers to process trees of Records. Great talk.

Infer from Bradford Cross is a very approachable machine learning library for Clojure.

limelight is a rich client GUI framework.

Hazelcast is a Java dynamic clustering library.

Mark McGranaghan - Ring

Model web requests and responses with Clojure concepts:

app( request-map ) -> response-map
the app is a Clojure function
the request and response are Clojure maps
“middleware” is implemented as higher-order-functions

They produced ~120 line “spec” and from there an entire ecosystem of libraries and frameworks has grown. Now it is possible to deploy a Ring app that mixes in components from a variety of locations and they all work together in one stack.

The essence of Clojure is composable abstractions

If you want to build powerful apps you need to get the underlying abstractions right.

Rich Hickey - Keynote - Step away from the computer

When was the last time you thought hard about a problem for an hour, day, month, year?

When have you confidently sat down to implement something for the first time?

The point he made by the end of the talk was something like: if you will think hard and well about a problem and write down what you learn then you will position yourself to confidently implement a solution on your first attempt

We commonly talk about the cheapest place to fix bugs is at development time. But, it is cheaper to fix bugs at design time. The biggest problems in software are ones of misconception. The developer has wrong ideas about the problem or the solution. Automated testing cannot provide a direct answer for the question of “whether this is a good idea?”

Analysis and Design

not UML, not a process, not waterfall, step before “go do it!”
identify and state the problem (verbally or in written form)
critically assess proposed solutions

We should be solving problems, not adding features. Will the features solve a problem? Or will the features create a problem? Avoiding a problem is not the same as solving a problem.

Problem solving is a skill which can be learned and practiced. See “How to Solve It” Is it better to be good at problem solving or methodology X?

Understand the problem:

facts, context, constraints (known and unknown), questions to find unknown unknowns, related problems
understand previous attempts to solve the problem
goal is to work incrementally ahead of the best known solution to the problem
write this down

Be discerning:

are there problems in your solution? If so, solve these.
identify tradeoffs. Often stated tradeoffs just identify deficiencies in a solution. To properly understand tradeoffs it is necessary to have at least two alternative solutions in hand
write this down

Focus is needed to think well:

the computer is a distraction
balls will be dropped while you are focusing deeply on a problem. This is part of the process.
feed your background mind by thinking a lot about the problem with the waking mind. This will establish an agenda for the background mind to work.
since the mind can only focus on ~7 items at a time, write down the aspects of a problem. This will allow you to quickly swap some of the items in and out of your mind. Maybe use pictures for this (not prescribing UML). Go over this repeatedly so you internalize it and can think about it long and hard.
closing your eyes and removing other sensory inputs facilitates thinking well
… wait for a solution to come
since it is not always practical to wait, pipeline. Get several problems going at once. Switch tasks when you are stuck.

Once an idea for a solution has come:

try it
code it
this should involve little typing
since you have thought about it enough this should be confident coding
feedback (e.g. from test results) is important but don’t lean on it

David Liebke - Concurrency to parallelism

The slides are very well down and contain much of the content of the presentation.

Concurrency is about managing access to shared state.

Parallelism is about making use of multiple processors.

pmap

parallel meets lazy
weakness - if there is a slow consumer then it degenerates (because the worker threads stop working ahead) [is that really a weakness if there is no consumer to use the results yet?]
weakness - one slow processor can slow the entire process (maybe this is more of a theoretical problem?)
it is more useful on long-running functions

fork join

based on work of Doug Lea
each worker has a double ended queue (deque) of work
workers can “steal” from other workers when they are out of work. This provides some load balancing across threads.
implemented in Clojure in fjvtree
in order to make pvfilter on a pvreduce work it is necessary to make the reduce function associative and it needs to handle intermediate results (which might be in a different from that the leaf inputs)

Chas Emerick - Continuous deployment

slides

First principle - get your beautiful creating into the hands of your customers

Paths to deploying your apps:

“system admin” - unrepeatable, manual, undocumented, high priesthood <- horrible
“roll your own” automated system - typically the default solution, not really a solution, just yak-shaving
community standards - this is where you want to be, stay close to the herd

Varying levels of community standards:

vertically intergrated solution. This is the best, use it if you can. e.g. Google App Engine, Heroku, Azure
more general tools. e.g. Chef or Puppet. But you need to use a non-Clojure language to express tasks.
Clojure specific tools: jclouds (with Clojure wrappers), Pallet

Pallet

pure Clojure
uses jclouds
similar to Chef
“crates” (instead of recipes) are idempotent tasks that are “just” Clojure and totally composable

Don’t deploy to production on Jetty (ok to use it for development). Instead use Tomcat, JBoss, or Glassfish, etc.

Automate the build, test, and deploy process.

Stuart Halloway - Simplicity

slides

The word “simple” has been severely abused:

it does not mean counting things (i.e. “this is simple because it has ”)
it does not make sense for it to mean: how productive can a newbie be (e.g. kazoo vs violin). This is no way to build a career.
it does not mean that something is familiar to us
it does not mean that something is minimal (although this is often a valuable characteristic)
it does not mean uncomplicated

So is the definition hopelessly subjective? No. It has an objective definition that should inform our designs and is fundamental to what we are trying to do.

Simple means: not compound

See “Studies in Words” by C.S. Lewis in which he talks about verbicide and how the meaning of words change over time. Specifically it examines the history of the meaning of the word “simple”

[… I had to leave early, so I missed the 2nd half of the presentation.]

Edit - Added link to Chas’ slides. Edit - Added link to Stuart Halloway’s slides.

clojure-conj day 1 notes

Sun, 24 Oct 2010 19:48:27 -0500

My rough notes of the messages I heard during day 1 (October 22, 2010) of clojure-conj.

Fogus - Roots of Clojure

slides

A list (some of it vaguely chronological) of the language environment in which Clojure was created:

1994 - Rich praised C++ strong typing
Perl arose as a language that tied the web together
Paul Graham sang the praises of Lisp
Java was born. Took developers half way to Lisp, but ended up on the wrong half.
Ruby, Python, and JavaScript opened the minds of many to dynamic languages
The Kingdom-of-Nouns is dominant, although alternative views are present in languages like Dylan, CLIPS, and Lisp
Rich’s “Are we there yet?” talk challenged the predominance of OO thinking. Referenced the work of Alfred North Whitehead’sf “Process and Reality”.
Prolog and Datalog
Test Driven Development - Should everything be test-driven? Test Driven Dentistry?
Haskell has the important idea of laziness
MonetDB (“an open source column-oriented database… designed to provide high performance on complex queries against large databases, e.g. combining tables with hundreds of columns and multi-million rows”)
Databases offer ** Multi-version concurrency control ** Snapshot isolation ** Transactions
ML - idea of isolating the point of change
Erlang - shares many ideas with Clojure except Clojure is more open because it separates data from the functions that operate on it
Rich has read “all” of the papers from the 1970’s and 1980’s

Luke VanderHart - Zippers

Good talk that made me appreciate the beauty of zippers.

Zipper overview:

Generic tree walking/editing
Purely functional, fast, and elegant
tree of data + focus on specific location = zipper
edits happen at the point of focus of the zipper. It is not necessary to re-create the tree to that point. It is like an “inside out glove”

Zipper implementation:

Zippers are implemented as a thin wrapper around existing Clojure structures.
Creating a zipper on a tree does not copy the tree into a new data structure. Instead zipper functions are applied to the existing tree.
Code demonstrates effective use of metadata, first class functions, and destructuring
The “rest of the tree” (beside the node that is focused on) is stored in a “path”
Metadata stores zipper functions (children, branch?, make-node)
Zipper data structure is very easy to explore (not a byzantine structure)

Zippers in use:

Luke used them to create a word processor in which each version of the document was stored in a zipper. This made it easy to get to the past state of the document at any point in time.

Christophe Grand - DSLs != macros

slides

Talk was packed with API design tips based on real-world experience.

Christophe is the creator of Enlive, Moustache, and Parsely.

Writing DSLs is hard
Try to use macros
Now you have two problems!

Is it a DSL or an API? In a sense every API is a DSL. It is a continuum, but the key attributes of a DSL are:

succinct notation for data or logic
usually declarative

For his products:

users complained about macros ** behavior was too surprising ** they were not composable
users don’t want a DSL, then want data values
values + functional-core = DSL success ** values - allow users to define their data values

So:

rely on Clojure for lexical scope (avoid creating new scope types via macros) and control flow
use closures and higher order functions. Don’t worry if it is ugly. Icing can be added later
no macros - macros are like premature optimization. They provide too much rope to hang yourself with.
start with data
give domain-specific semantics to Clojure’s datatypes (e.g. vectors, maps, etc.). But don’t try and give domain-specific meanings to lists or symbols.

e.g. - Enlive - he removed 1/2 of the macros and the result was a cleaner, faster system.

Two legitimate cases for macros. But even for these avoid them:

control flow - instead use closures and delays
binding - decouple binding via capturing [I don’t understand this point, but Christophe mentioned he might blog more details about this later.]

Christophe provided an example of a DSL for building regexes

In the example, the function “regex” is like eval for the DSL
Helper functions in the DSL are like macros in the DSL
Make the regex function idempotent so it can be called on input and if necessary the input will be “eval’d” to a regex. This is analogous to how the core sequence functions convert their args into sequences.
Small tip - when the implementation of the DSL is optimizing the input data structure be sure to check vars not symbols to recognize the DSL’s functions. This avoids colliding with other functions of the same name.

Tom Faulhaber - Flow

Flow is the opposite of feeling frustrated and passive. Instead things feel easy and natural.

Lisp promotes flow:

homoiconic - computer as a canvas, manipulate the language itself
symbolic - “
REPL - machine becomes a partner

Functional programming promotes flow:

immutability
composition
horizontal abstractions - [I think this refers to things like sequences which are pervasive in Clojure code.]

These allow you to build new abstractions with familiar mechanisms. It is easy to pick up tools from the Clojure toolbox and start manipulating data.

The focus in on dataflow, not flow-charts.

Good examples of flow in Clojure embody flow in both their implementation and their use. This means:

consistent granularity of data
consistent, comfortable toolset

Code examples that embody flow:

fill-queue ** treat asynchronous events like a seq
enlive ** read the source to see mind blowing code

Sean Devlin - Protocols

Examples of using protocols to get homogeneous behavior on heterogeneous types.

treating dates, strings, longs, etc. as “dates”
making variants of map, etc. that return the same type of object they accept. So calling map on a String produces a String.

When to use Protocols over multi-methods:

for performance
to get different behavior in different namespaces

Chouser - Finger Trees

slides

Great talk.

Implementation of ideas expressed in an academic paper in Haskell code. Finger trees are another persistent collection type that in some cases are more suitable than the built in Clojure types. In particular they are customizable and allow the creation of different kinds of data structures.

For example, you can build a double-list with finger trees:

a list you where you can use (first, rest, pop, peek) both ends without losing the type of the structure
get the count
lookup items in O(n)
split/replace/remove/insert in O(log n)

These are built by defining the “meter” to use for the finger-tree. A single meter function is allowed but it can effectively be the composition of several functions. The meter defines how to split the tree and how to access indexed items in the tree.

The finger tree implementation in Clojure is not finished yet. Still needs to handle metadata and equality.

technomancy - Leiningen

slides

Additional lein features:

checkouts - like multi-module projects, but ad-hoc and opt-in. You can tell lein to use some files in a local “checkouts” sub-directory to satisfy a dependency.
shell wrappers - [for starting a project-less swank?]
test selectors - allow conditional test execution - e.g. for tagging long-running integration tests
tasks - just functions. Use eval-in-project to get JVM isolation between user tasks and lein itself
hooks - use Robert Hooke to customize tasks that are outside of your control (otherise just use functions to customize it)
plug-ins - jars that define tasks or include hooks
lein int - an interactive shell for lein tasks

Community building:

identify low-hanging fruit to make it easy for people to contribute
provide a developer guide
accept patches promptly
make commit rights easy to get

Laurent Petit - Counterclockwise

They now have 80% of paredit.

Rich Hickey - New Clojure Features

Focus of new features is performance, but the results also offer better semantics (!). They are breaking changes. A key to the performance improvements is to produce code that Hotspot will optimize.

Unified primitives and boxed numbers

both act like primitives unless you use bigint operands
bigints are contagious
local numbers are represented as either longs or doubles (not shorts or ints)
+, -, … are non-promoting operators
+’, -‘, … are promoting operators <- don’t use these, they are just hear to keep people from whining, instead use contagion (as an aside, ’ is now a valid character in identifiers)

Align map key equality with number equality

the problem is that if a map key is Long(42) a lookup on Int(42) will fail even though Long(42) is numerically equal to Int(42)
this has been a nagging problem
the roots of the problem are in Java. This is how map equality is defined and there is no way to fix the root problem
the solution is: when maps are accessed via Java they will provide the expected (bad semantics), but when they are accessed via Clojure functions they will make map equality match numerical equality

Bindings & threads

current problem is that bindings only apply to the thread that creates them. So if a thread pool is used somewhere down the call stack (e.g. with pmap) then the bindings are lost.
vars have root values
bindings create a mapping from that var (they don’t change the root binding) to a “box”. Using set! the value in the box can be changed. Every time a new binding is executed, a new box is created.
bindings assume a single-threaded model
the key to making bindings work across threads is to make the “boxes” thread safe
the solution is: the thread that established the binding can use set! to change the value, all other threads get read-only access to the binding. So sub-threads (i.e. threads enlisted in part of an overall “task”) can point to the same binding map as their “parent” thread. NOTE: They are not really child threads because typically they are threads from a pool.
this is intimately related to the “scopes” problem in which we would like to create resource scopes (e.g. to close a file, etc) around lazy sequences, in which case the resource needs to be closed in a location far from the lexical scope where the resource is used.

Pay for what you use & dynamic variables

the current “def” is too powerful because it always creates vars which can be dynamically bound. This is “unfair” in that even if this dynamicity is not needed, the user still incurs a performance cost every time the variable is accessed.
most defs are functions that are not going to be rebound
every var access incurs some costs to determine if the variable has been dynamically bound
solution: require the developer to explicitly identify defs that need to be dynamic (stylistically - the names of these variables and functions should include earmuffs, e.g. foo)
Clojure will “presume stability”
but, to accomodate the development-time use-case of redefining functions regularly: at the start of each function invocation a single check will be made against a var-universe-counter to make sure that no vars have been refdefined (any time a var is changed the var-universe-counter will be incremented). For long running functions they should be invoked with #’ to trigger a check of the var-universe-counter.

Functions & primitives

Currently pulling code out to a function causes the args and return values to be boxed.
solution: add type hints to the args and return values to indicate primitives. The return value type hint is placed just before the arg vector to accommodate multiple arities.
future work will handle making higher order functions (e.g. map, reduce) preserve primitive performance.

Enhanced Clojure records (part 2)

Sun, 15 Aug 2010 12:47:00 -0500

Following up on my efforts to enhance clojure records I put the second version of the code on github. Now record fields are printed in the same order they were defined. I also made it print nil values that are added to the record as long as they are not one of the “native” fields in the record.

Ubuntu systray icons missing

Mon, 09 Aug 2010 17:08:27 -0500

I spent some time today tracking down an issue in which several systray icons disappeared from my Ubuntu system. The solution was to add the Notification Area item back to the panel.

http://ubuntuforums.org/showthread.php?t=1072833

Enhanced Clojure records

Sat, 03 Jul 2010 09:11:00 -0500

Clojure records are a mechanism for creating data abstractions.

Records are instantiated with a list of the field values:

Explicit nils are required to create a record without some field:

Records are immutable, but new ones can be created by mutating an existing record

Multiple fields can be set by merging a map into a record

Notice that records print in a format that cannot be evaluated. Evaluating the printed form throws an exception.

My understanding is that a future Clojure will address some of the current shortcomings with records, but in the meantime…

The code below provides a generic mechanism for enhancing records in the following ways:

Generate constructor functions that take maps.
Print records in a form that can be evaluated.

The new defrecord2 macro is invoked like defrecord:

This generates a constructor function that takes a map. Using named parameters is clearer than the positional constructors that are provided by plain records.

It is not necessary to pass nils in for the missing fields.

The enhanced records can be mutated by passing in a map of new values:

And of course the records are now printing in a form that can be evaluated. This works both from print and from pprint:

A final benefit of defrecord2 is that the records can be used without importing the underlying record class. This avoids a bit of import complexity.

Wrapping Clojure in a Java API

Sat, 12 Jun 2010 14:41:00 -0500

How to wrap some Clojure code in a Java API? Googling on this leads first to a solution that involves invoking methods on clojure.lang.RT from Java. This might make sense for launching a Clojure program from Java, but it is strikes me as a crude tool for trying to build an API.

Protocols and Datatypes

Clojure protocols and datatypes create Java interfaces and classes behind the scenes so they are a possible solution:

An issue I encountered with this approach is that the type hints are not included in the Java interface.

javap -classpath target/classes demo.impl.boat.Boat

Even when I provide type hints to define the method arguments and return values the resulting interface has these all typed as Objects. So this fails to provide a nice interface for Java apps to consume.

gen-class and gen-interface

Clojure provides the gen-class and gen-interface macros to produce Java classes and interfaces. Typically the gen-class usage is shown embedded in a namespace definition like this example from the Clojure site:

For my use this feels a bit cumbersome because it’s most natural progression would lead to a Clojure file per Java class/interface. I prefer to avoid that sprawl by invoking gen-class and gen-interface directly.

This gives tight control of the Java classes and interfaces produced:

Java constructs are not just side effects of a Clojure construct, but rather are directly produced
definitions are not spread out across files unnecessarily
namespaces of Clojure files do not need to match the Java packages of the resulting classes/interfaces.

For example, the resulting interface looks like this:

javap -classpath target/classes demo.api.Store

Overall this is a very satisfying solution. The real beauty of the solution is that a precise Java API is produced without leaving the comfort of Clojure!

Here is a sample of how the API looks from Java:

Protocols and Datatypes Revisited

The syntax for defining a datatype is nicer than the magic-coincidental-naming of the gen-class approach. So another viable choice is to define a datatype that implements a Java interface. This combines precise control of the Java interface with the nice datatype syntax.

Further Investigation

As I write this I am asking questions on the Clojure mailing list and it seems there are more options to explore with definterface and a new feature, under development, that allows functions to be defined as static…

"… “ns” and “in-ns” have special evaluation rules. In general, they..."

Fri, 30 Apr 2010 11:33:00 -0500

“… “ns” and “in-ns” have special evaluation rules. In general, they don’t work as you’d expect in block expressions such as “do” or “let”. If you want to create namespaces programatically, use “create-ns” and “intern”.”

- http://stuartsierra.com/ on http://groups.google.com/group/clojure

Clojure - repeated function application

Sun, 21 Mar 2010 15:57:00 -0500

In Clojure I was looking for a compact syntax to repeatedly apply the
result of a function to a series of arguments. This led me to explore
various mechanisms:

”->” threads a value through as the 2nd value in a series of expressions.

(-> 20 (+ , 1) (* , 2) (/ , 21) (println ,))
;; prints “2” (which is ((20+1)*2)/21)

Clojure treats the “,” as whitespace, so that is simply added so the
reader of the code can see where the previous value will be plugged
into each expression. The println call is added to demonstrate that
this macro works not just with mathematical operators, but with any
function call.

“»” similarly threads a value through but does it to the 3rd position
rather than the 2nd.

(-» 20 (+ 1 ,) (* 2 ,) (/ 168 ,) (println ,))
;; prints “4” (which is (168 / ((20+1)*2)))

“comp” composes functions and allows for similar functionality, but
with more power and more syntax:

((comp #(/ 168 %) #(* 2 %) #(+ 1 %)) 20)
;;=> 4

Partial functions can be used instead of lambdas:

((comp (partial / 168) (partial * 2) (partial + 1)) 20)
;;=> 4

However, none of these accomplish what I want, which is to repeatedly
apply the result of a function to the arguments. For example, for
dealing with functions like this:

(def foo
(fn [x]
(fn [y]
(fn [z]
(+ x y z)))))

I find it best to not tie my head in knots trying to think of a
function that returns a function that returns… Rather, just think of
it as something which can be applied repeatedly and with each
application fill in the next nested level of arguments:

(((foo 1) 2) 3)
;;=> 6

And this is the kind of expression I am trying to simplify. I want a
cleaner syntax for it. The idea is to make a general version of a
function like this:

(defn apply-repeatedly-v0 [f x y z]
(((f x) y) z))

(apply-repeatedly-v0 foo 1 2 3)
;;=> 6

Of course it needs to handle a variable number of arguments:

(defn apply-repeatedly [f & args]
(loop [f f args args]
(if (first args)
(recur (f (first args)) (rest args))
f)))

All of the following are equivalent:

((((apply-repeatedly foo) 1) 2) 3)
(((apply-repeatedly foo 1) 2) 3)
((apply-repeatedly foo 1 2) 3)
(apply-repeatedly foo 1 2 3)
;;=> 6

For nicer syntax the function can be given a shorter name:

(defn » [& args] (apply apply-repeatedly args))

Repeated applications can be expressed as:

(» foo 1 2 3)
;;=> 6

For example, this can be used to express S/K combinator code more
nicely. If K and S are defined as:

(def K
(fn [x]
(fn [y] x )))

(def S
(fn [x]
(fn [y]
(fn [z]
((x z) (y z))))))

Then instead of writing code like this:

(((S K) K) 13)
;;=> 13

Write:

(» S K K 13)
;;=> 13

And “I” can be defined as:

(defn I [x] (» S K K x))

Or in the cleaner, if more obscure, point-free style as:
- Hide quoted text -
(def I (» S K K))

(I 27)
;;=> 27

Clojure - destructuring nested maps

Sat, 20 Mar 2010 15:02:50 -0500

I was writing a Clojure function that was passed a nested map like this:

(def input {“results”
  {“data”
   [{“x” { “value” 100}
   ”y” { “value” 105}}
   {“x” { “value” 203}
   ”y” { “value” 219}}
   {“x” { “value” 345}
   ”y” { “value” 378}}]}})

I was pleased to write the following, compact code to tease it apart and extract the data:

(let [{{rows “data”} “results”} input]
  (for [{x “x”
       y “y”} rows]
       ((juxt x y) “value”)))

;;=> ([100 105] [203 219] [345 378])

The first line shows nested map destructuring.

Then the for loop shows regular map desctructuring.

Finally the juxt call was used to access the value of each of the fields.

Out of the Tarpit : comments

Sun, 14 Mar 2010 19:31:00 -0500

This technomancy blog entry links to a video of Rich Hickey talking at a user conference. It includes the thought provoking quote (taken out of context) along the lines of “clojure is not bug-free, but it is not bad… considering it has no tests”. Rich makes the assertion that his time is better spent thinking about implementing his code correctly rather than writing tests around it. To better understand his position, I followed the link from technomancy to the Out of the Tarpit paper.

The paper is interesting. It helped clarify my thoughts on managing state and accidental complexity. I particularly appreciated the description of how stateful logic contaminates the functions it is called from. Below are specific points I noted:

Abstract Data Types considered harmful

p 52 - Abstract data types are inevitably tailored to a particular purpose. They hinder referential transparency (the attribute of a function in which it only depends on data passed in at the top) because they cause the caller to pass in more data than is really needed.

Separate Infrastructure from Application Specification

Build an infrastructure, then build a specification for the application and execute the specification. A variation on the theme of: build a DSL for problem space, then implement your app in that language.

Essential State

The only essential state is that state input directly from the user. All other data can be derived from this.

Separate state and behavior

p 44 - The authors advocate separating the state and behavior of the system (obviously, in direct conflict with one of the core OO tenets).

Power Corrupts

P 12 - Language power corrupts. An argument for language constraints.

Separation Pure Logic from Control
P 34 - Separate the pure logic (code with no state) from the “control”. One of the motivations for doing this is so that these separate parts of the application can be coded in different languages. The power of the language used for each can be restricted. This facilitates reasoning about the application (e.g. state can be ignored in the “logic” portions of the application). This culminates in the diagram on p 35 that shows 3 top-level components of the system.