分类： LINUX
2008-06-03 08:33:27
I recently found lisp. (That is that teaching computer programming language) I found lisp by reading too many of and by getting too inspired by him. Curious, I started down the path of giving it a fair evaluation, which meant: getting a decent implementation of Common Lisp (), getting a good book () and a good development environment ( and ). A is not required but appreciated. Try if you are new to linux. You will also need the apache (version 1 or 2) webserver up and running as well as a piece of software called (which is a module or plugin for apache). Lastly you will need the database server. (MySQL will do if it has to.)
I will present this tutorial as a series of blog posts. This page will be the growing index until I'm finished. What you need to do to play along at home is get all of the above set up on your computer before you start.
I will cover the following problem: How do you set up lisp so that it can be used to serve webpages with. A required subtitle is How do you do all this if you are unfamiliar with lisp in the first place. I doubt whether experienced lispers will learn anything new, but their suggestions will be most definately welcomed. I am really just trying to document a way of starting on the road to lisp. I did webpramming before in the so anyone else in this situation might follow the same route.
In order to learn lisp I decided to write a web application. In order to finish it, I decided to make it as simple an application as possible. No fancy stuff, just enough of everything to serve as a testbed for evaluating Common Lisp. So. I decided to write an online forum engine.
First I will define that database structure that we will use for our application. It is so simple that adapting this to just about anything else will be trivial. Here is the script that creates the database in PostgreSQL
create sequence topicid;create table topics ( topicid integer unique not null primary key, time timestamp not null, name varchar(255) not null ); create sequence postid;create table posts ( postid integer unique not null primary key, topicid integer references topics( topicid ) on delete cascade, time timestamp not null default 'now', name varchar(255) not null, post text not null);
What this means is that you would have a topic, and each topic would have 0 or more posts associated with it. That's it. That is all the information that this forum engine will carry.
In the next topic I will present the part of the code that interfaces between the database and the rest of the application.
The next bit is the part of the application that handles the database connectivity. In order to connect to a database I used the clsql package.
Sidenote 1: If you don't know about it allready, the is the website where I go to most regularly when I'm looking for new information. It is a wiki and a lot of people have contributed very helpful information there.
Sidenote 2: You need to figure out how to get other people's libraries to work with your application. If you are using Debian or a debian-based linux (like Ubuntu) then you will have access to the common lisp controller. This means that there will be a way of installing lisp packages that is integrated with your OS's packaging mechanism. I also know gentoo linux has the same or similar capability with it's portage system. You can also install directly and use that to install lisp packages with. Even so; Some packages might not be in your specific package management system, and then you also have a problem. Your best bet is figuring out how to use asdf properly, since this has been ported to a lot of lisp implementations and packages.
Get the clsql packge loaded into your current lisp session. Then define your package as follows. This gives your package a name (ZA.CO.PB.FORUMS) and declares that it uses the COMMON-LISP package (which you should get with your lisp implementation by default) and the CLSQL package. The symbols in these packages will also be locally available and will save you having to type a qualifier in front of every library symbol's name.
(defpackage :ZA.CO.PB.FORUMS (:use :COMMON-LISP :CLSQL))
Sidenote 3: Qualifiers: If you did not :use a packge then you can still use the symbol (provided it has been loaded) with its qualifier. If for example you wanted to connect to the database you would do it like this:
(clsql:connect +db-connect-param+)
+db-connect-param+ is a constant that is defined in the package. In my (disconnected from the internet) machine i defined it like this:
(defconstant +db-connect-param+ '("" "clsql" "" ""))
If you read the clsql documentation at the psql section you will find that I have specified the connection string to only contain the database name. If your database required authentication information then specify it in the appropriate places in that list of strings.
;; inserts a new topic into the database;; returns the id of the new topic(defun make-new-topic* (name) (let* ((newid (car (query "select nextval('topicid')" :field-names nil :flatp t))) (sql (format nil "insert into topics ( topicid, name, time ) values ( ~a, '~a', 'now' )" newid (make-safe-db-string name)))) (execute-command sql) newid)) ;; insert a new post into the database;; returns the id of the post(defun make-new-post* (topicid name post) (let* ((newid (car (query "select nextval('postid')" :field-names nil :flatp t))) (sql (format nil "insert into posts ( topicid, postid, name, post, time ) values ( ~a, ~a, '~a', '~a', 'now' )" topicid newid (make-safe-db-string name) (make-safe-db-string post)))) (execute-command sql) newid))
For both of these functions you pass strings as parameters. That will be the information that you want persisted in the database. Some observations:
So you now have the capability to insert data into the database. These function are completely testable from the command line of your lisp environment.
Next is the function that will do most of the work of getting the information back out of the database and into an usable form.
(defun get-list-of-hashes-from-query (sql) (let ((result nil)) (multiple-value-bind (datas fields) (query sql) (dolist (data datas) (let ((short-result (make-hash-table :test 'equal))) (dotimes (i (length fields)) (setf (gethash (elt fields i) short-result) (elt data i))) (push short-result result)))) (nreverse result)))
Update: 28 September 2005: Dave Roberts points out in his comment that the get-list-of-hashes-from-query defun is highly inefficient. This is true. You can read his suggestion and my reply in the comments section of this post.
clsql:query returns multiple values. The first is the list of results. Each query row is returned as a list itself. The second value is the names of the columns in the query. The order thereof is the same as the order in which each row's columns are returned. What the get-list-of-hashes-from-query> function does is to transform each row list into a hashtable with the column name as the key-value. The next couple of functions use this function as their own backends.
;; returns all the information associated with a topic in a hash(defun get-topic (topicid) (first (get-list-of-hashes-from-query (format nil "select * from topics where topicid = ~a" topicid)))) ;; returns all the information associated with a post in a hash(defun get-post (postid) (first (get-list-of-hashes-from-query (format nil "select * from posts where postid = ~a" postid)))) ;; get all the post's that belong to a topic in list(defun get-posts-for-topic (topicid) (get-list-of-hashes-from-query (format nil "select * from posts where topicid = ~a order by time asc" topicid))) (defun get-all-topics () (get-list-of-hashes-from-query "select * from topics order by time asc" ))
These are self-explanatory. So the only thing left is the make-safe-db-string function.
;; returns a string that is safe to be inserted into the postgres db(defun make-safe-db-string (string) (let ((result string)) (setf result (regex-replace-all "\\" result "\\\\")) (setf result (regex-replace-all "'" result "\\\\'")) result))
For this method to work you also need the ppcre package. This is a perl-compatible regular expressions library. Since lisp compiles to native code, ppcre actually outperforms perl's own regex implementation so is regex libs go, this one is quite good. What this function does is to replace all single backslashes with double backslashes. Then replace all single quotes with a backslash and a single quote. So why so many backslashes? The lisp loader escapes one set of backslashes so you are left (in the first line) with (regex-replace-all "\" result "\\")) after this round of escaping. And that shows you hat single backslashes will be scaped by doubles.
Next time I'll introduce the tbnl package.
is a Common Lisp library that makes development of web applications easier when using Common Lisp. To understand what it does, one first needs to know how mod_lisp goes about its business.
If you are coming from PHP you are familiar with the idea that each page request starts an instance of the PHP interpreter, which loads the specific script that was requested, interprets it and sends the output to the requesting http client. So any variable has value only while the script "lasts" (meaning: is running within the interpreter). This creates a lot of problems with state. (In fact, this problem is related to the HTTP protocol and not to PHP which merely drapes itself around it.)
In contrast to this, you have the mod_lisp model. This model works by assuming you have a lisp instance running with your application loaded in it. The mod_lisp apache module uses a socket to connect apache with your lisp instance. So your application is now a normal application. (Think of an application written as a deamon. You start the application at boot time, and stop it at shutdown time.)
Whether this architecture is better or worse than the one that PHP uses depends more on what application you are writing than on your opinion.
However. This is all that mod_lisp provides. Just a socket where communications may potentially take place. On the lisp side, you still have to provide code that will read from the socket, and write back to it. And this is a pain, since it forces you to do a lot of low-level stuff, which while important, happens to impede progress in your main application.
This is where TBNL comes in. TBNL wraps all of this up so that you can pretend you don't care how communication happens between apache and lisp. It gives access to GET and POST variables, as well as the name of the script, headers sent from the client etc. It really provides you with a very nice framework to do web development with. And we will use it in the forum application we are writing.
It has been some time since I posted on this series last. Not much is left to do. We basically need to connect some HTML and some PostGresQL database queries with the requests from mod_lisp. As we saw last time, TBNL will actually be handling that part, so we just need to get some presentation and some glue code going.
I used cl-who to do the HTML generation. Make sure that it gets loaded into your running lisp image. cl-who allows you to generate valid HTML by using s-expressions. This is important since it allows the reader to check your HTML syntax for you.
For example: In PHP you would do something like this to generate some HTML code:
print "Some Title Some body stuff
";
Notice that the HTML is embedded in a PHP string value, and besides from making sure that the quotes match, the PHP syntax checker cannot really help you with making sure that your HTML is correct. In lisp you can program some intelligence into the reader (by way of macro) and it will now be able to help you check your HTML syntax. cl-who does this with the (with-html-output-to-string) form, and the above HTML will be generated by the following lisp code:
(with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Some Title")) (:body "Some body")))
With this short piece of code it might seem more verbose, but notice that no closing tags are explicitly coded. They are inserted automatically. Also notice that the reader has no knowledge of what the meaning is of a :head or a :body or anything. It handles all (:
text Some intelligence makes sure that when you have a single element HTML tag, that it is handled properly.
The most important bit in the application is the structure that tells TBNL where to get the HTML for which pages from. This is handled through a global variable called *dispatch-table*. We set it in the following way:
(setq *dispatch-table* (nconc (mapcar (lambda (args) (apply #'create-prefix-dispatcher args)) '(("/tbnl/index" index) ("/tbnl/show-topic" show-topic-main) ("/tbnl/new-topic" new-topic) ("/tbnl/new-post" new-post))) (list #'default-dispatcher)))
From this you can see that our web application consists of 4 different "pages", namely /tbnl/index, /tbnl/show-topic, /tbnl/new-topic and /tbnl/new-post. Then there is also the default page handler, which is a good idea to have. I set my mod_lisp up so that every request that comes for the tbnl directory gets served by the lisp instance.
Before I can show the four functions that serve their various page requests, I need to show the macro that they are written with.
(defconstant +all-locations+ '("index" "new-topic")) ;; here we can open the page and as well present;; the main layout of the webpage. Give the name of the page.;; this will be the title field(defmacro with-menu-html (name &body body) `(with-html-output-to-string (*standard-output* nil :prologue t :indent t) (:html (:head (:title ,name) (:link :rel "Stylesheet" :type "text/css" :href "/main.css")) (:body (:div :class "allcontainer" (make-main-menu) (:div :class "contentcontainer" ,@body (:div :class "sourcelink" (:a :href "/forums.lisp" "source"))))))))
The +all-locations+ constant is used to generate the menu with. Then this macro defines the wrapping HTML within which the actual unique content of each page will be embedded. The (with-html-output-to-string) form is in fact the macro that we showed above and comes from the cl-who package.
You will notice that it calls the (make-menu) form. Here it is:
(defun make-main-menu-item (itemname selected) (if selected (with-html-output (*standard-output* nil :indent t) (:li :class "mainmenuselected" (:a :class "selected" :href (format nil "/tbnl/~a" itemname) (str itemname)))) (with-html-output (*standard-output* nil :indent t) (:li :class "mainmenuunselected" (:a :class "unselected" :href (format nil "/tbnl/~a" itemname) (str itemname)))))) (defun make-main-menu () (with-html-output (*standard-output* nil :indent t) (:div :class "mainmenucontainer" (:ul :class "mainmenu" (dolist (item +all-locations+) (make-main-menu-item item (string-equal (format nil "/tbnl/~a" item) (script-name))))))))
This is some very simple jiggery-pokery to make a nice unordered list so that we can use css later to style menu with.
Next we have the four functions that will handle these individual requests. They operate by dumping the output text into stdout. TBNL then sends this text back to apache and then from there it goes to the client. The first one is index.
;; browsing start point. shows all topics, allows to browse to topics(defun index () (with-menu-html "Main forums page" (:h1 "All Topics") (dolist (topic (get-all-topics)) (make-topic-row topic)))) (defun make-topic-row (topic) (with-html-output (*standard-output* nil :indent t) (:div :class "topicrow" (:span :class "topictime" (str (gethash "time" topic))) (:span :class "topicname" (:span :class "topiclink" (:a :href (format nil "/tbnl/show-topic?topicid=~a" (gethash "topicid" topic)) (str (gethash "name" topic))))))))
As you can see the index page lists all different topics in the database. It wraps each topic into some HTML elements with the call to (make-topic-row)
Next is the (show-topic-main) form, which displays a chosen topic, with all posts associated with the topic. It also adds a HTML form at the bottom, so that new posts can be added to the current topic. Notice the (get-parameter) call which gets a GET parameter. (Here "topicid")
(defun show-topic-main () (let* ((topicid (get-parameter "topicid")) (topic (get-topic topicid))) (with-menu-html (gethash "name" topic) (show-topic topicid)))) (defun show-topic (topicid) (let ((topic (get-topic topicid))) (with-html-output (*standard-output* nil :indent t) (:h1 (str (gethash "name" topic))) (:h2 (str (gethash "time" topic))) (dolist (post (get-posts-for-topic topicid)) (make-post-row post)) (:h2 "New Post") (make-new-post-form topicid)))) (defun make-post-row (post) (with-html-output (*standard-output* nil :indent t) (:div :class "postcontainer" (:div :class "postheader" (:span :class "postname" (str (gethash "name" post))) (:span :class "posttime" (str (gethash "time" post)))) (:div :class "postbody" (:span :class "postpost" (str (gethash "post" post))))))) (defun make-new-post-form (topicid) (with-html-output (*standard-output* nil :indent nil) (:div :class "formcontainer" (:form :method :post :enctype "multipart/form-data" :action "/tbnl/new-post" (:div :class "formnamefield" (:p "Name:" (:input :type :text :name "name"))) (:div :class "formpostfield" (:p "Body:" (:br) (:textarea :name "post" :rows 10 :cols 30 ""))) (:div :class "formsubmitfield" (:p (:input :type :hidden :name "topicid" :value topicid) (:input :type :submit)))))))
The new-topic page is only one function long. Here it is. Nothing fancy about it either.
(defun new-topic () (let ((isposted (parameter "isposted"))) (if isposted (let ((name (parameter "name"))) (make-new-topic* name) (redirect "/tbnl/index")) (with-menu-html "Make a new Topic" (:div :class "pageheader" (:h1 "Make new Topic")) (:div :class "formcontainer" (:form :method :post :enctype "multipart/form-data" (:div :class "formnamefield" (:p "Name:" (:input :type :text :name "name"))) (:input :type :hidden :name "isposted" :value "true") (:div :class "formsubmit" (:input :type :submit))))))))
Notice the (redirect) form which is a shorthand for sending the "Location: " header.
Lastly we have the (new-post) form which handles the HTML FORM that takes the parameters for the new post as well as the (make-safe-html-string) form which makes the FORM parameters reasonble safe:
(defun make-safe-html-string (string) (let ((result string)) (setf result (escape-for-html result)) (setf result (regex-replace-all (string #\Newline) result (format nil "~a
" #\Newline))) result)) (defun new-post () (let ((topicid (parameter "topicid")) (name (parameter "name")) (post (parameter "post"))) (when (and topicid name post) (make-new-post* topicid (make-safe-html-string name) (make-safe-html-string post)) (redirect (format nil "/tbnl/show-topic?topicid=~a" topicid))) (redirect "/tbnl/index")))
The (make-safe-html-string) form uses the cl-ppcre library. This library is a perl compatible regular expression library. It is actually quite good, but I only use it here to do text replacements with. A little bit of overkill maybe.
(defun start-forums () (connect +db-connect-param+) (tbnl:start-tbnl)) (defun stop-forums () (tbnl:stop-tbnl) (disconnect))
I used these two forms to start and end the forums engine with.
And that is it! Everything that you need to know so that you know how to write a little application in Common Lisp.