UniServe : Developer's Guide

Author: SOFTINNOV / Nenad Rakocevic
Date: 02/10/2004
Version: 0.9.9
Comments: [email protected]

Table of Contents

1. Basics
2. First Steps
      2.1. Installation
      2.2. Testing your UniServe
      2.3. Directory tree
3. Starting UniServe
4. Services API
      4.1. Service definition
      4.2. Incoming data handling
      4.3. Properties
      4.4. Events
      4.5. Methods
      4.6. Local values isolation
      4.7. Shared space
      4.8. Installing a service
      4.9. Services at runtime
      4.10. Example: micro-httpd
5. Protocols API
      5.1. Protocol definition
      5.2. Incoming data handling
      5.3. Properties
      5.4. Events
      5.5. User events
      5.6. Methods
      5.7. Global functions
      5.8. Local values isolation
      5.9. Example: wget
6. Logging/Debugging


1. Basics

UniServe is a multi-protocol client/server framework for network programming. It's built on a port! asynchronous I/O multiplexing engine and is inspired by the MEDUSA framework idea.

Its purpose is to offer a simple but powerful solution for programming client and servers applications that can be combined with View interfaces easily.

UniServe consists in :

Services are loaded at UniServe's start and can be changed, removed and loaded at runtime too. They can also be instancied multiple times to listen on several ports number at the same time.

Protocols are loaded also at start and are available to use with an API similar to the native REBOL port! API. They can be used to create clients very easily that will run simultaneously with services and View events inside a REBOL's event loop.

Services and protocols API is event oriented. This means that implementing a new service or a new protocol is just a matter of filling one or more callback function with appropriate code. (Something close to what VisualBasic provides for GUI programming)


2. First Steps


2.1. Installation

Unzip you UniServe archive where you want in your system. The installation is finished !

Installing a new software shoudn't be more difficult than that, don't you think so ? ;-)

Folder with spaces in their path
Sometimes, having spaces in installation path can cause troubles in your REBOL apps, so avoid the desktop folder or any other folders having spaces in their path.


2.2. Testing your UniServe

Run the %starter.r script. A console should popup with the list of all the services and protocol loaded and UniServe will enter in an event loop waiting for clients to connect. If you see an error line in the console saying that the HTTPd (or other) service cannot be launched, it's probably because you already have a server running on the same port. Either shutdown your local server, or try to run a new instance of the conflicting UniServe service on another port using the Monitor application (in the %client folder).

Open your favorite web browser and try the following address: http://localhost. If all goes well, you should see a "Welcome" web page with Softinnov's logo. You've just successfully tested UniServe's HTTPd service !

If you have a REBOL version with an active /Shell component, you can try to click on the "Example CGI Link". You should now see a CGI environment dump.

Your UniServe installation is OK, so you can now build your next killer-app ! ;-)


2.3. Directory tree

UniServe needs its own directory tree in order to work properly. Do not change the names of these directories : 

*root*/                 Installation folder
    uni-engine.r        UniServe kernel
    starter.r           All services launcher script
    ...
    libs/               Library folder
        log.r
        headers.r
    services/           Active Services folder
        ...
        store/          Services repository
            ...
    protocols/          Active Protocols folder
        ...
        store/          Protocols repository
            ...
    modules/            Modules repository
        ...
Use the store folder to "plug" or "unplug" services before launching UniServe. All valid rebol scripts found in %services/ will be loaded as services when UniServe starts. Just copy a service file to %store to disable it.

A service may need other files to work. These additional service's files should be installed in a sub-folder with a name matching the service name.

Ex: 

services/
    ...
    httpd.r
    httpd/
        wwwroot/
            ...
        mime.types
        httpd.conf
        ...
NB: Runtime service management is available through the 'Admin service or through the 'control shared function. (see "Services at runtime" section)


3. Starting UniServe

To load the engine : 

>> do %uni-engine.r
The engine is loaded in a specific context named UniServe.

To start the engine : 

>> UniServe/boot
By default, UniServe will load all the services found in the %services/ sub-directory and all the protocols found in the %Protocols/ sub-directory.

The boot function can take optionnaly two refinements :

Example: 

UniServe/boot/no-loop/with [
    services  [HTTPd FTPd...]
    protocols [HTTP FTP...]
]
(NB: You can omit one of the keywords ('services or 'protocols) instead of providing an empty block)

Start path information
If you're starting UniServe from a different directory than the current one, you need to specify an additionnal path by setting the UniServe-path word. Path must end with a slash. You need to setup the path before calling the 'boot function.

Example: 

>> UniServe-path: %/library/UniServe/
>> UniServe/boot

Once booted, if the /no-loop refinements was specified, UniServe needs an event loop to be able to work. This event loop can be provided by a call to 'view function or directly with a 'do-events call :

To start an event loop, you can choose between the following alternatives : 

>> do-events
or 

>> view ...


4. Services API


4.1. Service definition

A new service has to be written in a new file with a name matching the service's name plus ".r" extension. (see %services folder in UniServe archive)

Each service is defined using the install-service function. A REBOL header is required. (even if you leave it empty) 

install-service [
    ...service definition...
]
(NB: You do not need to 'do or 'load any scripts to use this function.)

Put all the service related code in the service definition. You can include external libraries of code if needed. Everything that's not inside the service definition block will be globally BIND-ed.


4.2. Incoming data handling

UniServe provides to the developer a unique system for implementing network protocols. The management of incoming data is handled at low-level by UniServe.

To be able to decode every incoming logical data packet (not the physical IP packet), you must know when the packet ends. There's two ways to acheive that :

UniServe automatically manages incoming data using one of these methods. It's up to the developer to set the correct mode using the service's stop-at property.

Once one of these two end is reached, UniServe will call the on-received function in your service definition with the received data as argument.

Example: 

Remote host sends the string "Hello. I'm ok." to your service
[stop-at: #"."] will generate the following calls :
-> on-received "Hello."
-> on-received " I'm ok."
[stop-at: 1] will generate the following calls :
-> on-received "H"
-> on-received "e"
-> on-received "l"
-> on-received "l"
-> on-received "o"
-> on-received "."
-> on-received " "
-> on-received "I"
-> on-received "'"
-> on-received "o"
-> on-received "k"
-> on-received "."
[stop-at: 4] will generate the following calls :
-> on-received "Hell"
-> on-received "o. I"
-> on-received "'m o"
                    <- notice here that we miss "k." bytes because there's not
                    enough data in the incoming buffer to trigger a new call
                    to 'on-receive !! A new event can be generated if new data
                    comes (the "k." is waiting in UniServe's internal buffer)
stop-at can be changed at any time in your service definition, allowing you to implement the most complex protocols (switching from one receive mode to another). Its new value becomes effective as soon as the current event function ends.

You can optionnaly decide to disengage this system by setting stop-at to none! value. In this mode, UniServe will call the on-raw-received function in your service definitione each time your application receives a physical (TCP or UDP) packet from the remote host.

Received data type
Remember that data is always passed as binary! value to your 'on-received and 'on-raw-received functions. You will often need to convert it to string! in order to handle it correctly.


4.3. Properties

The following list of properties are available for use in your service definition code. Some of them are mandatory to write a valid service.

Word Value Mandatory? Description
name word! yes Name that uniquely identifies the service
port-id integer! yes The default listen port number
stop-at integer!
string!
char!
binary!
none!		 no Defines when the 'on-received event will be generated.
Default value is none.
scheme word! no Default value is 'tcp, you can change it to 'udp
module word! no The default module name for background processing
hidden logic! no (reserved for future use)
client port! no Property set at runtime pointing to the current client port! value
shared object! no Access to the UniServe shared space. You're free to add your own data here, so that other services can use them.

Client property
Client word points to the current port! value connected to the remote client. You can use the usual port! internal words to access all the connection informations.

For example, if you want to know the IP address of the client : 

client/remote-ip


4.4. Events

Events are special functions called by UniServe kernel to process network events. By default, these words are set to none. It's up to the developer to choose and implement the appropriate events.

Word Prototype Description
on-load func [] Called when the service is loaded by UniServe. Place your setup code here if needed.
on-new-client func [] Called when a new client connection is accepted by UniServe
on-close-client func [] Called when the client or the service closes the connection.
on-received func [data [binary!]] Called when the desired sequence or length defined by stop-at is reached.

data : all data received from the client since the last call to 'on-received. data is cleared once the function is evaluated, so use 'copy if you want data to survive the function.
on-raw-received func [data [binary!]] Called when a raw TCP or UDP packet is received.

data : all data received from the client since the last call to 'on-raw-received. data is cleared once the function is evaluated, so use 'copy if you want data to survive the function.

At least one !
You should implement at least one of these events if you want your service to do anything useful !

Avoid blocking code in events
Developers should take care about event functions implementation and be sure not doing blocking tasks (like connecting to a remote host in sync mode, doing long calculations, etc...). This will directly affect UniServe's performance. If you need to do heavy or blocking tasks, use the 'task-master service and background processes.


4.5. Methods

There's several functions pre-defined in the service context :

Word Prototype Description
write-client data [string! binary! file!] Send data back to the client. This function is not blocking, so you can use it safely in event functions. (Data is not sent at once, but queued for latter processing)

data : data to send.
close-client /now Close the client connection. This function is not blocking, so you can use it safely in event functions. (The client port will be closed when there's no more pending data in the output queue)

/now : Close the client connection at once and clear its output data queue.
share values [block!] Add values to the shared space. (see "Shared space" section)

values : block of name/value pairs. (name [set-word!], value [any-type])

Serving files
If you use a file! as argument to write-client, the file won't be loaded in memory. UniServe will send it in parts to the client, reading on the disk each time only the data required. This allows serving files of unlimited size without impact on UniServe's memory or performances.


4.6. Local values isolation

Service definition code had to be re-entrant, that means that it can be called with different client context without corrupting data or mixing data between clients. To avoid issues, you have to store client session values in client/user-data.

Example: 

client/user-data: context [
    id: 3432
    state: 'logged
    etc...
]
All words defined in the service context are "globals" for all clients. Words stored in client/user-data are local to that client.

For example, if you want to maintain a count of current clients, you could do : 

install-service [
    ...
    count: 0
    on-new-client: does [count: count + 1]
    on-close-client: does [count: count - 1]    
]
There's other ways to save the session values. You're free to choose whatever method you want to isolate client specific data.


4.7. Shared space

The global shared space is just an object! value that can be use at any time by any service.

All services can add their own values in the shared object at load time using the share function. 

share [
    a: "hello"
    pool: 50
    do-task: func [...]...
    ...
]
All services have access to this object through the shared word at runtime. 

probe shared/a
shared/a: 2
shared/do-task args...
The default shared object comes with only one method defined : 

shared: context [
    control: func [...]...      ; UniServe administration function. 
]


4.8. Installing a service

Once created, your new service goes in the %services directory and will be loaded automatically by UniServe at the next start.

Manually loading services
If you need to manually load some services, you should do it before starting the main event loop.

Example: 

do %uni-engine.r
UniServe/boot/no-loop
install-service [...]
or 

#include %services/my-service.r     ; for PREBOL preprocessor
then 

do-events   ; or view...


4.9. Services at runtime

When a new client connects to one of the active services, UniServe creates a port! value for handling the connection. UniServe uses the port's locals word to store all the informations needed to process the port events.

Port/locals
All ports managed by UniServe have their locals word reserved for internal use. Don't change anything in port/locals unless you know exactly what you're doing !

The interesting thing to notice here is that port/locals/handler will point to your service definition.

Stopping a running service can be done either remotely through the admin:// protocol or locally by using the UniServe shared control function (access path is shared/control)

Here a short description of the control function : 

control
    name        [word!]     Service's name
    id          [integer!]  Service listening port number
    /start                  Launch a new instance of the service listening at the :id port
        /all                Launch all loaded services with their default port number
    /stop                   Stop a running service
    /list                   Return a list of the running services
    /install                (not implemented)
        file                (not implemented)


4.10. Example: micro-httpd

Here's the source code for a minimal web server adapted from the %webserver.r script (You can find the original script here)

UniServe's keywords are colored in red.

'micro-httpd service source code
install-service [

    name: 'micro-httpd
    port-id: 81

    mime: htfile: file: none

    stop-at: join crlf crlf

    wwwpath: %wwwroot/  ; path must end with a slash !

    html: "text/html" gif: "image/gif"  jpg: "image/jpeg"

    not-found: [
        "404 Not Found" 
        [<HTML><STRONG><H1>ERROR</H1></STRONG><H4><p><p>
         "REBOL Webserver error 404"<BR>"File not found"</html>]
    ] 

    bad-perms: [
        "400 Forbidden" 
        [<HTML><STRONG><H1>ERROR</H1></STRONG><H4><p><p>"REBOL Webserver" 
         "error 400"<br>"You do not have permission to view the file"</html>]
    ]

    http-head: func [type mime][
        reform ["HTTP/1.0" type newline "Content-type:" mime newline newline]
    ]

    error: func [err][
        write-client append http-head err/1 html err/2
    ]

    on-received: func [data][
        if "HTTP" = htfile: third parse data "/" [htfile: "index.html"]
        mime: parse htfile "." 
        if error? try [mime: get to-word mime/2][mime: html]
        any [all [not exists? file: join wwwpath htfile error not-found]
             all [error? try [read/binary/part file 1] error bad-perms]
             all [
                write-client http-head "200 OK" mime
                write-client file
             ]
        ] 
        close-client
    ]
]

Remarks :


5. Protocols API


5.1. Protocol definition

Protocols are the client equivalent of services. Their purpose is to provide a client-side network protocol implementation that can be instanciated to build network clients. You'll see a lot of similarities between protocols and services API, so that if you know how to write a service, you also know how to write a client protocol ! :-)

A new protocol has to be written in a new file with a name matching the protocol's name plus ".r" extension. (see %protocols folder in UniServe archive)

Each protocol is defined using the install-protocol function. A REBOL header is required. (even if you leave it empty) 

install-protocol [
    ...protocol definition...
]
(NB: You do not need to 'do or 'load any scripts to use this function.)

Put all the protocol related code in the protocol definition. You can include external libraries of code if needed. Everything that's not inside the protocol definition block will be globally BIND-ed.


5.2. Incoming data handling

Incoming data is handled exactly in the same way as with services.


5.3. Properties

The following list of properties are available for use in your protocol definition code. Some of them are mandatory to write a valid protocol.

Word Value Mandatory? Description
name word! yes Name that uniquely identifies the protocol
port-id integer! yes The default connection port number
events block! yes List of user-defined event names
stop-at integer!
string!
char!
binary!
none!		 no Defines when the 'on-received event will be generated.
Default value is none.
scheme word! no Default value is 'tcp, you can change it to 'udp
server port! no Property set at runtime pointing to the server port! value
connect-retries integer! no Sets the maximum number of retries for connecting to a server before aborting. Default value is 5

Server property
Server word points to the current port! value connected to the remote server. You can use the usual port! internal words to access all the connection informations.

For example, if you want to know the IP address of the server : 

server/remote-ip


5.4. Events

Events are special functions called by UniServe kernel to process network events. By default, these words are set to none. It's up to the developer to choose and implement the appropriate events.

Word Prototype Description
on-init-port func [port [port!] url [url!]] Called before opening the connection. Useful for last minute parameter changes like user name or password.
on-connected func [] Called when the connection is established with the remote host. (before sending or receiving any data)
on-close-server func [] Called when the client or the server closes the connection.
on-received func [data [binary!]] Called when the desired sequence or length defined by stop-at is reached.

data : all data received from the client since the last call to 'on-received. data is cleared once the function is evaluated, so use 'copy if you want data to survive the function.
on-raw-received func [data [binary!]] Called when a raw TCP or UDP packet is received.

data : all data received from the client since the last call to 'on-raw-received. data is cleared once the function is evaluated, so use 'copy if you want data to survive the function.
on-error func [why [word!] port [port!]] Called when a problem occurs during connect time.

The why argument in the on-error event can receive the following values :

 unknown-hostThe hostname cannot be resolved !
 unreachableThere's no available route to reach the server !
 connect-failedThe server is not responding after several attempts to connect !


5.5. User events

In order to make your protocols usable, you have to define a local API using custom events called here "user events". It's called "user", because it's the interface that your protocol implementation will provide to the protocol user.

Protocols are, in fact, just a layer between UniServe's events and user events !

These user events are defined in your protocol using the events keyword. You just have to give a list of event names, in no particular order. You call, then, these words as if they were functions in your protocol implementation at strategic positions. That's all !

Example: 

events: [
    on-login-ok
    on-message
    ...
]
You can use any name you want. In UniServe, events names are prefixed with on- to easily recognize them in source code. Use meaningful names that reflect the current state in the communication with the server or the object you're waiting to receive, etc...

If you need to pass arguments to your user events, just do it! There's nothing special to do to allow your events to take arguments. Remember to document your user events header somewhere ! For now, we add a comment line for each event specifying the expected header. (We may add a kind of autodoc feature for user events documentation in future versions)

More complete example : 

install-protocol [
    name: 'pop
    ...
    on-received: func [data][
        ...
        on-login-ok server
        ...
        ...
        on-message server import-email msg
        ...
    ]

    events: [
        on-login-ok     ; [port [port!]]
        on-message      ; [port [port!] msg [object!]]
        ...
    ]
]
NB: It's useful (but not mandatory) to pass the port value as first argument. It gives to the protocol user an easy access to all the port values including port/user-data local context.


5.6. Methods

There's several functions pre-defined in the protocol context :

Word Arguments Description
write-server data [string! binary! file!]

/with port [port!]		 Sends data to the server. This function is not blocking, so you can use it safely in event functions. (Data is not sent at once, but queued for latter processing)

data : data to send.

port : port value to use to send data.
close-server   Closes the connection to the server. This function is not blocking, so you can use it safely in event functions. (The server port will be closed when there's no more pending data in the output queue)

Sending files
If you use a file! as argument to write-server, the file won't be loaded in memory. UniServe will send it in parts to the server, reading on the disk each time only the data required. This allows sending files of unlimited size without impact on UniServe's memory or performances.


5.7. Global functions

This set of functions will allow you to manage UniServe protocols at runtime, in a very close way as you do with classics REBOL's built-in protocols. There are globally defined (in REBOL's global context), so they can be used from anywhere. Remember that these functions are not blocking !

Word Arguments Description
open-port url [url!] events [block!]

/with spec [block!]		 Creates and returns a new port! working in asynchronous mode.

url : a valid url specifying the scheme, host, target, etc...
events : a block of events functions.

spec : extends the port value with additionnal definitions. It consists in a list of name: value pairs.
insert-port port [port!] data [any-type!] Sends data through the port.
close-port port [port!] Closes the connection and calls the on-close-server events.

The events argument of 'open-port takes an event definition block using the following format : 

open-port url [
    event1: func ...
    event2: func ...
    ...
]
Each event name has to be supported by the protocol implementation used referenced by the scheme in the url.
Don't add any other kind of value to the event block!

The 'open-port /with refinement is useful to store local values that can be easily reached from within the events.

Customizing 'insert-port
The insert-port function is basically a wrapper on write-server. That means that the data will be sent "as is" without any formatting or encoding process. If you want to add a layer to process data before sending (dialect interpretation, compression, encoding, etc...), this can be achieved by defining a new function in your protocol definition that will be automatically called by insert-port.

Your new custom insert-port function will be named new-insert-port and has to be defined like that : 

new-insert-port: func [port [port!] data [...]]...
If you define 'new-insert-port, remember to call write-server to send your processed data to the server.


5.8. Local values isolation

The same rules described in the services API, apply to protocol writing.

You can also use the /with refinements of 'open-port function to add local values to your port.


5.9. Example: wget

Here's the source code for a simple HTTP GET method implementation using UniServe's protocol API.

UniServe's keywords are colored in red.

'wget protocol source code
install-protocol [

    name: 'wget
    port-id: 80

    stop-at: header-end: join crlf crlf

    on-connected: does [
        write-server rejoin [
            "GET " server/target
            " HTTP/1.0" crlf 
            "Host: " server/host
            crlf crlf
        ]
        print ["connecting to:" server/host]
    ]
    
    on-received: func [data /local pos][
        either integer? stop-at  [
            on-response server to-string data
            stop-at: header-end    ; Reset stop-at for next request
        ][  
            ; HTTP header received          
            either pos: find/tail data "Content-Length:" [
                ; Extract document size
                data: to-string trim/all copy/part pos find pos newline
                stop-at: to-integer data
            ][
                on-failed server
            ]
        ]
    ]
    
    events: [
        on-response     ; [port [port!] result]
        on-failed       ; [port [port!]]
    ]
]

Usage example in console : 

>> UniServe/boot/no-loop/with [protocols [wget]]
[UniServe] Async Protocol wget loaded
>> open-port wget://www.rebol.net [
    on-response: func [port result][
        print ["Page size =" length? result]
    ]
]
>> do-events
connecting to: www.rebol.net
Page size = 4375
Remarks:


6. Logging/Debugging

All UniServe's parts: kernel, services, protocols uses the logger object (located in the %libs/log.r file) to generate informations at runtime. You can also use this feature in your own services and protocols easily. The logger object is loaded with UniServe's kernel, so its available to all your source code. It offers a global function called log with the following specs : 

log msg
    /info       ; display an information
    /warn       ; display as a warning
    /error      ; display as an error (doesn't break the program)
The msg argument can be any value. It will be REJOIN-ed by the log function.

The logger object also allows you to control the output redirection using its logger/level value. This value can be one of the following : 'screen (default), 'file, 'both, 'csv. ('both = 'screen + 'file)

To allow or not the output, we usually use a special word in all contexts derivated from the logger object : verbose. By setting it to an integer value, you can control the amount of output generated. (zero should mean : no output).

Each service and protocol in UniServe's archive is built upon the logger object, which adds advanced logging abilities. Set the verbose value to 0 to disable all logging. Set it to 4 to allow maximum output.

For example, to see how does the kernel work in realtime, do the following : 

uniserve/verbose: 2
The previous 'wget demo in console would then give you : 

>> do-events
[uniserve] Connecting to www.rebol.net : 80
[uniserve] Connected to 209.167.34.214 port: 80
[uniserve] Calling >on-connected<
connecting to: www.rebol.net
[uniserve] << low-level writing: 39
[uniserve] >> low-level reading: 1420
[uniserve] Calling >on-received< with "HTTP/1.1 200 OK^M^/Date: Sat, 09 Oct 2004 21:37:30 G"
[uniserve] >> low-level reading: 1420
[uniserve] >> low-level reading: 1420
[uniserve] >> low-level reading: 412
[uniserve] Calling >on-received< with {<HTML>
<!--Page generated by REBOL-->
<HEAD>
<TITL}
Page size = 4375
[uniserve] Connection closed by peer 209.167.34.214
[uniserve] Calling >on-close-server<
[uniserve] Port closed : 209.167.34.214

Copyright 2004-2007 SOFTINNOV All Rights Reserved.
Formatted with REBOL Make-Doc 0.9.6.1 on 13-Nov-2009 at 14:30:24