Most often, such data can be organized in a table with rows of information, each column representing a different kind of data. For example, in the case of processes running on a computer system, rows might be sorted according to their unique process identifier, with columns containing values such as CPU usage, memory usage, owner's name, time of creation, ...
The software used to view this type of information comes in different forms and shapes. Unix users might be familiar with the "top" application which presents rows of process data as lines of text, whereas RMON (Remote MONitoring) SNMP software usually uses multiple windows with graphical displays, curves, pie charts, multiple configuration dialog boxes, even 3D visualization modules to visualize network traffic, connection matrices, ...
In most cases, data comes from one or more tables. A common interface, graphical with menus, drag'n'drop capability, table widgets, textual and graphical data viewers such as multiple line graphs, bar and pie charts, could be used. The user could then sort table rows, select one or more cells, rows, columns, create views such as other tables, charts, ... best suited to the way data should be presented. Once optimized, the data viewers layout and configuration could be saved for later reuse as a dashboard. In effect, what is needed is a spreadsheet that is capable of dealing with dynamically changing data.
Moodss (Modular Object Oriented Dynamic SpreadSheet) is an attempt at answering these needs. It is composed of a main part (the core) and an unlimited number of modules, loaded as the application is launched, each module interfacing to a specific type of data. The core is written in the great Tcl language (at http://www.scriptics.com/) using object oriented techniques thanks to the stooop and scwoop packages (at http://www.multimania.com/jfontain/). The module function is to describe the data that it is also in charge of retrieving and formatting. Modules can be written in plain Tcl or use dynamically linked libraries written in the C language (modules are packages in the Tcl sense, so any language that can interface with Tcl is supported).
Modules are loaded when moodss is started. Several modules can be handled concurrently (starting with moodss version 3.0). This way, you may monitor data coming from any number of heterogeneous sources. Module names are specified in the command line and cannot be dynamically unloaded.
Versions 4.0 and up add a dashboard functionality: the current configuration (modules, viewers, poll time, windows sizes and placement, ...) can be saved in a file at any time, for later reuse (see the -f (--file) command line switch documentation).
Versions 4.3 and up support asynchronous modules (no polling needed as module data may change on its own). Note that any number of asynchronous and regular (synchronous) modules can be simultaneously loaded.
Versions 5.0 and up add a free text viewer, which can be used for comments, and which can also embed live data cells in text form.
Versions 5.3 and up support viewer type mutation through viewer icon drag'n'drop, and viewer quick destruction through trash icon drag'n'drop operation. A new menu for empty viewer creation was also added.
Versions 6.0 and up support command line arguments per module, data table column anchoring in module configuration and automatic module discovery.
Versions 6.1 and up support HTML formatted data for modules.
Since module data access is entirely customizable (through C code, Tcl, HTTP, ...) and since several modules can be loaded at once, applications for moodss become limitless. For example, comparing a remote database server CPU activity and traffic load from a network probe on the same graph becomes possible.
As features are added to moodss, different ways of viewing data will be made available while the module structure will stay the same. The goal of moodss is to become a nice feature packed generic way of viewing data. Moodss can be used to monitor any type of data, since the simplest cases can fit in one table with a single row, with the most complicated requiring loading several multiple table modules.
As moodss is written in Tcl and uses well supported extensions (Tktable and BLT), it will run on Tcl/Tk supported platforms: UNIX and Windows (I do not know if Tktable and BLT are available for the MacIntosh). Obviously, some modules may be specific to a platform, but the core is guaranteed to run on them all.
After reading and understanding this document, you should be able to write your own modules in order to monitor the data that you are interested in.
Moodss is free software. You can redistribute it and/or modify it under the terms described in the COPYRIGHT file or use the main window Help Copyright menu for more information.
For the current version (6.4), the following packages must be installed before attempting to install moodss (make sure to check the INSTALL file for the latest information):
The pie widgets, stooop and scwoop libraries are included in the
self contained moodss application file. Therefore, it is not required
to install the stooop, scwoop and tkpiechart packages, unless you want
to work on the moodss source code itself. However, should you want to get
more information on those extensions, you will find the latest versions:
The core will load one or more modules, whose names were passed as command line parameters or come from a save file, and will start displaying module data in one or more tables. The tables are then updated at the frequency defined by the poll time, which the user may change, or asynchronously for the relevant modules. For example, to launch moodss with the random module, just type:
$ wish moodss randomAll the module code and data are kept in a separate namespace. The module data is stored is a single array including some configuration data used when the module is loaded by the core, and variable data (displayed in the application table and eventual graphical viewers). If a module is synchronous, it must start updating its data when requested by the core. If a module is asynchronous, its data may be updated at any time. The synchronous or asynchronous nature is specified in the configuration data for the module.
The initial data tables represent the first data views, from which any number of cells can be selected. Data viewers can be created by dropping cells through a drag'n'drop operation into a graph, bar chart, pie chart, summary table or free text iconic site. In turn, these viewers can display more table cells, which when dropped into the viewer, result in the creation of corresponding data graph lines, chart bars, pie slices, table rows or text cells. Cells or rows can be removed from existing viewers, by simply selecting them and dropping them in the trash iconic site (a bullet hole).
Any viewer can be mutated (its type changed) by dropping a viewer icon into it. For example, create a stacked data bar viewer from several cells, then drag the 3D pie drag'n'drop icon into it. Any viewer can also be destroyed in one shot by dropping the trash icon into it.
Any draggable data can be dropped in any valid drop site at any time. It is thus possible to drag several data cells from any table or any viewer into other ones, the trash, ... even if the data comes from different modules.
All data viewers can be moved and resized at will with the help of a simple internal window manager.
The current configuration (loaded modules, tables and viewers coordinates, sizes, poll time, main window size, ...) can be saved in a file at any time, so that an identical dashboard can be relaunched at will.
The window title shows the name of the loaded module(s) along with the poll time.
The File menu contains the Save, Save As and Exit menu entries, used to save the current configuration (modules and viewers), or gracefully quit the moodss application.
The Options menu (may not exist, see below) contains the Poll time entry which when selected launches the corresponding dialog box, as shown below:
When several modules are used, the minimum poll time is the greater of the minimum poll times of all modules. The default poll time (used when moodss is started) is the greater of the default poll times of all modules. The available choices in the poll time dialog box is the combination of all modules poll times.
The Poll time menu entry is available only when needed, which is not the case if all the loaded modules are asynchronous. If this case, the Options menu itself is not displayed.
The New menu (only visible when not running in read-only mode, see Command line), allows the creation of empty viewers of any type.
Any data displayed in a table can be sorted at any time by simply clicking
on a column title. Clicking on the same column title again sorts the data
in opposite order, thus toggling between increasing and decreasing orders.
When sorting, the selected column is used as a reference, meaning that
all rows will be rearranged so that the selected column appears sorted,
with values either increasing or decreasing.
A little arrow indicator is placed to the right of the reference column
title label, pointing up or down depending on whether the sorting order
is decreasing or increasing.
Table columns can be interactively resized by holding the first mouse
button down on a column border. The mouse cursor is changed to an horizontal
double arrow on column borders to show this capability.
Aside from the main tables, graphical and textual viewers can be created for viewing table cell data behavior over time. Viewers can also be deleted, data views (such as pie slices, curves, ...) can be added or removed from existing viewers, ... These functions are all implemented using the drag and drop functionality described below.
Graphical viewers available at this time are BLT graph viewers (such as can be seen below), side-by-side bars charts, overlapped bars charts (only available when the BLT version 2.4 library is used), stacked bars charts, 2D pie charts and 3D pie charts*.
*note: if you know of any other nice viewers (like 3D graphs) that work with Tcl, please let me know so I can integrate them. Many thanks in advance...
|
|
|
|
|
|
|
|
|
The summary table displays for each row the cell label, the current, average, minimum and maximum values since the row was created. Data cells can be inserted one or several at a time through a simple drop. Rows can be deleted by selecting any cell in the row then dropping any number of them into the trash drop site.
The free text viewer is an editable Tcl text widget with any number (including zero) of embedded data cell windows. Data cells can be inserted one or several at a time through a simple drop, as with the other viewers. New data cell windows are inserted at the current insertion cursor position. Data cells can be deleted by selecting then dropping any number of them into the trash drop site. They can also be deleted using the keyboard Delete and Backspace keys, which also work on the regular text, as well as the expected other key bindings. When dropping data cells, each data cell window is preceded by a relevant label text for the cell, which can later be edited at any time.
|
|
|
Here is another shot featuring a free text viewer with loaded cpustats and memstats modules:
Only valid drop sites for the data being dragged are highlighted when the mouse cursor passes over them, thus guaranteeing error free operations (if there are no bugs, that is :).
In summary, data cells can be dragged from any table or any viewer into any viewer drop site icon, any viewer or the trash can.
Once a viewer exists, it also acts as a drop site for data cells, which may be dragged from any table or other viewers. Dropping one or more cells directly in the viewer results in corresponding lines, bars, slices or rows being created and automatically updated. Each new graphical element is assigned a new and different color.
You may delete one or more viewer elements (graph lines, bar chart bars,
pie charts slices, summary table rows or free text cell window) from a
viewer by selecting them (using the first mouse button) through their labels.
Several elements can be selected by depressing the control key as the first
mouse button is pressed. The selection can also be extended by depressing
the shift key along with the first mouse button. The pie slices can also
be directly selected by clicking on the slices themselves.
Then dragging from the viewer to the trash drop site (the bullet hole)
on the upper right side of the main window and releasing the first mouse
button result in the corresponding viewer elements to be destroyed. When
there are no remaining elements, the viewer itself (graph, bar chart, pie
or summary table) can be destroyed by dropping it into the trash. The free
text viewer can only be deleted this way when completely emptied of any
text and data cell window.
Any viewer can be deleted in one shot by dropping from the trash icon into it.
Any viewer also acts as a drop site for viewer type data, which allows
viewer mutation by just dropping from the new viewer type icon into the
existing viewer. It is much quicker than destroying the existing viewer
and create a new one of the new type, while remembering which data cells
were monitored by the viewer of the old type.
When mutating, if some cells in the current viewer no longer exist
(they may belong to a disappeared summary table), they are not made a part
of the new viewer, and a warning message is flashed to the user in the
message area.
Any viewer is also a drag site. It requires selecting one or more viewer elements before initiating the drag operation from any selected element in the viewer. If there are no selected elements, dragging is impossible: the mouse cursor is not changed into the drag circular cursor.
If a viewer contains no elements, then the viewer itself can be dragged and dropped into the trash.
All viewer icons (below the menu bar) are drag sites for viewer type data, which allows quick viewer mutation (see mechanism description in Drop sites).
The trash icon is also a drag site of the killing action type, which allows viewer destruction in one shot.
Once moodss has been launched with one or several modules and tables have been moved, resized, viewers created, moved and resized, the current configuration can be saved in a .moo file, and later reused by passing the corresponding file name with the -f (--file) command line switch.
For moodss version 4.0 and above, the following information is saved in the file (which is human readable):
The File Save As menu behaves as expected, with the user always having to choose a file name.
Once a file name has been specified (either through the command line or the file selector dialog box), that file name is reused whenever the File Save menu is used. The File Save As menu may be used at any time to change the file name.
$ wish moodss randomor, for 2 modules at once:
$ wish moodss ps cpustatsYou may not specify the same module more than once in the command line.
You may eventually specify a poll time in seconds using:
$ wish moodss -p 25 randomNote that when all the specified modules are asynchronous, the poll time option specifies the preferred interval for graph viewers.
Once saved through the File Save menus (for example in save.moo) , the configuration can be retrieved using:
$ wish moodss -f save.moowhich would result in the same modules being loaded, the same viewers displayed at the same positions and sizes, the same poll time being used, as well at the same application window size. New modules data displays can be added at any latter time to existing dashboards by specifying modules on the command line after the -f (--file) switch / value pair.
Command line options include:
For example, the following command:
$ moodss -p 15 random --asynchronous arp --remote jdoe@foo.bar --numeric route --numericcauses the random module to update asynchronously, the arp module to collect data from the foo.bar host under the jdoe login name and not attempt to lookup symbolic names for hosts, with the last module route doing the same.
Note the setting the application poll time to 15 seconds does not interfere with the module options.
The moodss core checks the validity of module options according to the
information provided by the module programmer. Any invalid option / value
combination for the module is detected, reported on the standard error
channel before the application exits.
package provide random 1.1This line simply states that this is the 1.1 version of the random package. Please note that the package name must also be used as the module namespace name (see below).
namespace eval random {
array set data {
...
}
proc update {} {
...
}
}
The update procedure is not needed when the module is asynchronous.
All the module configuration data is stored as array members of the array named data within the module namespace. For example:
namespace eval random {
array set data {
updates 0
0,label name 0,type ascii 0,message {user name}
1,label cpu 1,type real 1,message {cpu usage in percent}
2,label disk 2,type integer 2,message {disk usage in megabytes}
3,label command 3,type dictionary 3,message {most time consuming command} 3,anchor left
pollTimes {10 5 20 30 60 120 300}
sort {1 decreasing}
indexColumns {0 3}
helpText {
This is a simple demonstration module ...
}
}
...
}
The updates member is a counter used to keep track of the number
of times that the module data was updated, and is also used by the core
to detect when module data display should be updated (see variable
data for more information).
The columns member is obsolete and no longer used after and including moodss version 3.5.
The n,label members define the text to be displayed as column
titles. There must be as many n,label members as they are columns.
The n,type members define the type of the corresponding column
data. Valid types are simply those that the Tcl lsort command can
handle: ascii, dictionary, integer and real.
There must be as many n,type members as they are columns.
The n,message members define the text of the help message to
be displayed in the message area (see User Interface)
as the user moves the mouse pointer over column titles. It should be composed
of only a few words, just enough to actually help the user understand what
the column data means. There must be as many n,message members as
they are columns.
The n,anchor member is optional. Column data is either centered
by default, tucked to the left or right side of the column. Valid values
are center, left or right.
Note that column numbers start at 0. There must not be any hole in
the column numbers sequence.
The pollTimes member is a list of valid poll times (in seconds)
for the module. The list is not ordered, as its first element represents
the default poll time value to be used when the moodss application starts.
This value may be overridden by a command line argument. The smallest value
in the list is used by the core as the lowest possible poll time and checked
against when the user enters a new value through the poll time dialog box.
The list must not be empty.
Note that the list is also used by moodss as a set of possible choices
in the dialog box used to set the new poll time. The user may still directly
input any value as long as it is greater than or equal to the minimum value.
If the module is asynchronous (data can be updated at any time and not in response to update procedure invocations (no polling required)), the pollTimes member must be a single negative integer value representing the preferred time interval for viewers that require one (only graphs at this point). For example, if you wish graph viewers to have a display interval of 10 seconds, use:
namespace eval random {
array set data {
...
pollTimes -10
...
}
...
}
In this case, the graph viewers time range (knowing that they feature 100
time points) would be 1000 seconds. I guess that the value that you should
specify as the pollTimes member should be the expected average update interval
for your asynchronous data. Note that the graphical viewers x axis always
display properly labeled absolute time ticks in any case.
When several asynchronous modules are loaded with no synchronous modules, the interval used for all relevant viewers is the average (in absolute value) of all module intervals. For example, if you load 2 asynchronous modules, one with a pollTimes member of -10 and the other of -20, then a 15 seconds interval value is retained. Note that the interval can be forced through the --poll-time command line argument.
If at least one synchronous module is loaded concurrently with any number of asynchronous modules, the actual application poll time (the one that can be set with the then available poll time dialog box) is used.
The visibleColumns member is an optional list that specifies the table columns that should be displayed. If not specified, all the table columns are visible.
The sort list defines the index of the column which should be initially used as a reference for sorting the data table rows, and in which order (increasing or decreasing) the rows should be sorted. The column index for sorting works like the -index Tcl lsort command option, that is rows are sorted so that that specific column appears sorted in the specified order. The specified column must be visible (see visibleColumns member documentation above).
The indexColumns list specifies the columns required to uniquely identify a row in the table. In database talk, it represents the table key. To maintain backward compatibility, it is optional and defaults to 0, the leftmost column. The index columns are used when creating data viewer elements: their label is built by concatenating the key value for the cell row with the cell column title. The key value is the concatenation of the index column values for the cell. When specified, all the columns in the list must be visible (see visibleColumns member documentation above).
The helpText member specifies a text of any length, to be displayed when the user requests help information on the current module from within the help menu. The text can be HTML formatted or plain. HTML formatted help requires the <HTML> and <BODY> tags to be present, while tables and frames are not supported (and many other tags: stick to formatted text at the moment). The core will render HTML formatted or plain text in the module help window according to the help text contents.
The views member is optional. If specified, it defines one or more views to be used in place of the default view. One table will be displayed per view. Each view is a list of array members, suitable as an argument to an "array set" command. For each view, 2 members must be defined: visibleColumns and sort (syntax and usage are identical to the default table members).
The switches member is optional. If specified, it defines a list of switch / boolean pairs. A switch is a single letter or a string prepended with the - or + sign. The boolean value (0 or 1) specifies whether the switch takes an argument. If the switches member exists, an appropriate initialize procedure must be provided by the module (see Initialization). The core will take care of parsing the command line and reject any invalid switch / value combination for the module. The switches value may not be changed in the initialize procedure.
For example, a recent random module configuration is as follows:
array set data {
updates 0
0,label name 0,type ascii 0,message {user name}
1,label cpu 1,type real 1,message {cpu usage in percent}
2,label disk 2,type integer 2,message {disk usage in megabytes}
3,label memory 3,type integer 3,message {memory usage in kilobytes}
4,label command 4,type dictionary 4,message {command name}
pollTimes {10 5 20 30 60 120 300}
sort {1 decreasing}
indexColumns {0 4}
helpText {...}
views {
{visibleColumns {0 1 3 4} sort {1 decreasing}}
{visibleColumns {0 2 4} sort {2 decreasing}}
}
switches {-a 0 --asynchronous 0}
}In this case, data is presented in 2 tables: one for the CPU and memory usage, the other for disk usage.
In order for a module to accept command line arguments, the initialize procedure must exist (more information below). Otherwise, if the module can take no arguments, it stays optional. In that case, it is rather redundant with inline module namespace code (namespace code outside of any module namespace procedure). When the module takes no arguments, so does the initialize procedure if it exists.
The initialize procedure is mandatory when the module supports command line arguments, and in such a case takes an array name as sole argument. The array contains the switched options values, indexed by switch. For example, if the command line was:
$ moodss random --asynchronous --other-option value -x 1234the options array will contain:
options(--asynchronous) = options(--other-option) = value options(-x) = 1234Note that the --asynchronous member value is empty as that switch takes no argument. For the above example, switches would have been defined as:
switches {-a 0 --asynchronous 0 --other-option 1 -x 1}
In all cases, data members other than updates and switches can be set or
updated in the initialize procedure, and successfully taken into account
by the core.
Example for a module that take arguments:
proc initialize {optionsName} {
upvar $optionsName options
variable asynchronous
variable data
set asynchronous [info exists options(--asynchronous)]
if {$asynchronous} {
array set data {pollTimes -10}
}
}
The options array is accessed using the upvar Tcl command technique
(used when passing arguments (such as arrays) by name). The module namespace
variable asynchronous is set according to the --asynchronous
command line switch existence. Then, if operating asynchronously, a preferred
poll time is set in the module data.
The core waits for the initialize procedure to be completed before initializing the next module. For modules likely to initialize slowly and/or susceptible to initialization failure, it is advised to allow the user interface to be updated using the Tcl vwait command (technique used in all the remote capable modules shipped with moodss). For example:
proc initialize {} {
...
set file [open "| /usr/bin/rsh -nl $remote(user) $remote(host) cat /proc/net/arp"]
fileevent $file readable {set ::arp::remote(busy) 0}
vwait ::arp::remote(busy)
...
}
Since the file to be read sits in a remote machine on an eventual slow
link, the initialize procedure is blocked until the file becomes readable,
but without hanging the user interface, using the Tcl fileevent
and vwait commands combination.
Similar techniques must be used in such cases within the module update procedure, so that the user interface and eventual modules running concurrently are not prevented to update. The remote capable moodss modules contain various coding techniques achieving that functionality, which I am sure you can improve on.
In case of a synchronous module, the core invokes the module update procedure (which obviously must exist) when it is time to refresh the data display (tables and eventually graphical viewers). At this time, the update procedure may update the tabular data straight away (synchronous operation) or launch a request for later data update (asynchronous operation).
In case of an asynchronous module, variable data may be updated at any time. The update procedure may not exist.
For all module types, it actually does not matter when the data is updated.
The core will know that fresh data is available when the updates
array member is set (actually incremented as it also serves as a counter
for the number of updates so far).
It is the module programmer's responsibility to increment this counter
right after all tabular data has been updated.
For example, retrieving information for the processes running on a machine
is a local operation that can be achieved in a reasonably small amount
of time. In such a case, data would be updated immediately and the updates
variable incremented at the same time.
But if the data has to be retrieved from across a network, waiting
for it to come back would cause a delay that the user would certainly notice,
as the application would not respond to mouse or keyboard input during
the whole time that it would take to fetch the whole data. In such cases,
it is easier to let the update procedure return immediately without setting
the updates variable, which would be incremented at a later time,
only when the data would become available. For example, when waiting for
data to come across a network connection, the Tcl fileevent command
could be used on a non blocking channel, where the script to be evaluated
when the channel becomes readable would increment the updates array
member.
For example:
namespace eval random {
...
proc update {} {
variable data
array set data "
0,0 john 0,1 1234 0,2 4567 0,3 cc
1,0 william 1,1 8901 1,2 2345 1,3 xedit
2,0 anny 2,1 6789 2,2 0123 2,3 ps
4,0 peter 4,1 4567 4,2 8901 4,3 ls
6,0 laura 6,1 2345 6,2 6789 6,3 emacs
3,0 robert 3,1 1234 3,2 5678 3,3 top
"
incr data(updates)
}
}
The tabular data array index is the row number followed by the column
number separated by a comma. The column number must start from 0
up to the total number of columns minus 1 (no holes are allowed in the
column sequence).
When all rows (or only those table cells that have changed) have been updated, the updates member array must be incremented so that the core knows that it can update the table data display.
The random module source code can be made to function asynchronously: please look into the random.tcl file.
package ifneeded random 1.1 "source [file join $dir random.tcl]"The line above says that if the random package is needed, the Tcl core should source the random.tcl module source code from the directory where it was installed. 1.1 is the version number for the package.
Modules can be installed at any valid place that the Tcl core allows (look at the pkg_mkIndex manual page for more information).
When you unpack moodss, you will find the sample modules in sub directories. The current directory (.) is appended to the auto_load global list variable so that sample modules can be found when moodss is run from the unpacking directory.
For example, if you unpacked moodss in /home/joe/moodss-X.x/, you will find the random module package in /home/joe/moodss-X.x/random/ so that the following will work:
$ cd /home/joe/moodss-X.x/ $ wish moodss randomYou can install your new modules in the default location: /usr/local/lib/ on Unix. For example, if you move the files in /home/joe/moodss-X.x/random/ to /usr/local/lib/random/, moodss (actually Tcl :^) will still be able to find the random module (again, look at the pkg_mkIndex manual page for more information).
Please take a look at the INSTALL file for the latest information on how to install the moodss application itself.
In such a case, no index column is required and as a matter of fact rather gets in the way. The trick is to use an empty column 0, use as index by the core by default, and to prevent it from being displayed by using 1 or more views.
Please look at the cpustats module code for a working example.
Send your comments, complaints, ... to Jean-Luc
Fontaine.