Stubs: A dynamic linking mechanism for Tcl, Tk and extensions. ============================================================== What does it do ? ----------------- 1) Adds support for backlinking to all platforms, thus making it possible for static executables to dynamically load extensions on any platform. 2) Eliminates all the operating system specific problems associated with dynamically linking an extension to Tcl/Tk and any other library which exports a stub interface. This may sound like a wild claim but it is true simply because the operating system does not have to do anything. 3) Extensions and libraries built with different compilers will work together even if it is not possible to link them normally. The only requirement for them to be able to work together is that function calls are compatible. 4) As extensions do not have a hard coded reference to the library it is possible to use them with any library which is compatible. e.g. an extension which was originally built for Tcl 8.0 should work with Tcl 8.1, and an extension built for Tcl 8.1 could work with Tcl 8.0. How does it do it ? ------------------- Basically the stub mechanism provides a cross platform dynamic linking mechanism using tables of function pointers. o A library which wishes to provide a stub interface populates a function table with the addresses of its interface functions. o A pointer to that table is then made available to any extension which needs to access that library's interface. o Once an extension has obtained the pointer it simply uses it to get the address of the function it wants to call and calls it. o At no time in this process does the operating system become involved. To see the differences take a look at the following which describes what happens when Tcl loads two different extensions. o Loading an extension which is linked directly to Tcl. 1) Tcl asks the operating system to load the extension. 2) The operating system loads the extension and then tries to resolve any undefined symbols. This process is very operating system dependent but involves one or more of the following steps. 1) Resolve any symbols which are defined in the current process context. (Backlinking). 2) Find and load any libraries that the extension is dependent on. This involves searching paths defined through a variety of operating system dependent methods. 3) Resolve any symbols which are defined in the new libraries. 3) Tcl then calls the extension's initialisation entry point. o Loading an extension which uses Tcl's stub interface. 1) Tcl asks the operating system to load the extension. 2) The operating system does so, resolving any non Tcl symbols. 3) Tcl calls the extension's initialisation entry point. 4) The extension obtains the pointer(s) to Tcl's stub table(s) and uses that to call Tcl. Where is it ? ------------- ftp://www.neosoft.com/pub/tcl/sorted/packages-8.0/devel/StubPatch8041.tar.gz How to apply it ? ----------------- The StubPatch8041.tar.gz file contains the following files. Tcl8.0-StubPatch.diff The patch for Tcl. Tk8.0-StubPatch.diff The patch for Tk. README This file. Unpack them as usual. e.g. gunzip -c StubPatch8041.tar.gz | tar -xvf - These patches should only be applied to vanilla 8.0.4. Move into the Tcl directory which contains the generic directory and apply the patch making sure to specify -p1. e.g. cd Tcl8.0.4 patch -p1 < ../Tcl8.0-StubPatch.diff And do the same for Tk. e.g. cd Tk8.0.4 patch -p1 < ../Tk8.0-StubPatch.diff You are now free to build it. How to build it ? ----------------- o Unix o Tcl There is one new configure option for Tcl. --enable-stub If this option is disabled (default) then you will simply get a normal tcl installation plus a few enhancements. If it is enabled then Tcl will provide a stub interface for extensions to use. It will also build an archive library called libtclstub*.a which is used to initialise and access Tcl's stub interface. You should delete the following files (if they exist) if you have changed the value of this option. tclBasic.o o Tk There are two new configure options for Tk; both of the require that Tcl is built with stub support. --enable-tcl-stub If this option is disabled then Tk will link to Tcl in the normal way. If it is enabled then Tk will use Tcl's stub interface. You should delete the following files (if they exist) if you have changed the value of this option. tkMain.o tkConsole.o --enable-stub If this option is disabled then Tk will not provide a stub interface. If it is enabled then Tk will provide a stub interface for extensions to use. It will also build an archive library called libtkstub*.a which is used to initialise and access Tk's stub interface. You should delete the following files (if they exist) if you have changed the value of this option. tkWindow.o o Windows o Tcl Two new makefiles have been provided, one for Borland and one for Visual C++. makefile.vc.stub (The Visual C++ version) makefile.bc.stub (The Borland version) These makefiles build a version of Tcl which provides a stub interface for extensions to use and will also build a static library tclstub80.lib which is used to initialise and access Tcl's stub interface. You should delete the following files if they exist when building with these makefiles for the first time. tclBasic.obj o Tk Four new makefiles have been provide, two for Borland and two for Visual C++. makefile.vc.stub (The Visual C++ version) makefile.bc.stub (The Borland version) These makefiles build a version of Tk which uses Tcl's stub interface. You should delete the following files if they exist when building with these makefiles for the first time. tkMain.obj tkConsole.obj (Visual C++ users will need to delete all .obj files due to name mangling done when functions are declared as __declspec(dllimport)) makefile.vc.stub2 (The Visual C++ version) makefile.bc.stub2 (The Borland version) These makefiles build a version of Tk which uses Tcl's stub interface and also provides one of its own, it will build a static library tkstub80.lib which is used to initialise and access Tk's stub interface. You should delete the following files if they exist when building with these makefiles for the first time. tkMain.obj tkConsole.obj tkWindow.obj o Macintosh At the moment there is no 'makefile' available to build a Macintosh version. However the code has been designed to work with the Macintosh and therefore it should not be too difficult to create one. If anyone would like to try then I will be happy to provide them with some more details about what needs to be done. Compatability of the stub interfaces with the standard interfaces ? ------------------------------------------------------------------- The stub interfaces of Tcl and Tk provide access to almost all of the external and internal functions both generic and platform dependent. The exceptions to these rules are ones which it makes no sense to make available through the stub interface. These include all Tcl command functions plus the following functions: Tcl_Main Tcl_AppInit Tk_Main Neither Tcl nor Tk provides access to the global variables which are available with the standard interface. This is for two reasons, the first being that it is bad practice to access global variables and the second being that they could only really be accessed using macros which can be very dangerous, especially in C++. On Windows and Macintosh the emulated X calls are included as part of the Tk stub interface. How to enable an extension to use a stub interface ? ---------------------------------------------------- Using Tcl's stub interface. 1) Before the extension calls any Tcl function is must call Tcl_Required. This is essentially the same as Tcl_PkgRequire except that the package name is implicitly "Tcl". As this function is supported by both the standard and the stub interface the call to it does not need to be conditional. The stub interface version checks that stubs are supported and initialises the stub table pointers. 2) The makefile needs to be modified to use the stub library (libtclstub*.a) instead of the standard library, and the compile flags need to be changed to define ACCESS_TCL_THROUGH_STUB. 3) An inline versions of the stub interface is available, it uses macros in C and inline functions in C++ to access Tcl's interface. These will be used if INLINE_TCL_STUB is defined before including tcl.h, alternatively you can include tclStubInls.h yourself. The stub library is still needed to ensure that the stub initialisation is done correctly. Using Tk's stub interface. 1) Before the extension calls any Tk function it must call either Tk_Required or Tk_Present. These are essentially the same as Tcl_PkgRequire and Tcl_PkgPresent respectively except that the package name is implicitly "Tk". These functions are implemented as a macro in the standard version of the interface because Tk may not be present when they are called. The stub interface version checks that stubs are supported and initialises the stub table pointers. 2) The makefile needs to be modified to use the stub library (libtkstub*.a) instead of the standard library, and the compile flags need to be changed to defined ACCESS_TK_THROUGH_STUB. Inline versions of the stub interfaces are available, they uses macros in C and inline functions in C++ to access the library's interface. The Tcl inline interface is used if INLINE_TCL_STUB is defined before including tcl.h, or you can include tclStubInls.h directly. Similarly for Tk. The stub library is still needed to ensure that the stub initialisation is done correctly. Unix users should take a look at the tclConfig.sh and tkConfig.sh file to see what variables are available to help configuring stubs. How to create an extension with a different compiler ? ------------------------------------------------------ All that is needed to do this is to build the stub library with your compiler. e.g. assuming that you have a Borland compiler and want to build an extension which works with a Visual C++ built core you need to do the following. 1) Get a patched version of the source and make tclstub80.lib. e.g. make -f makefile.vc.stub tclstub80.lib 2) Build your extension making sure that you use the Borland version of the stub library. Although it is possible to link objects built with different compilers together in most cases, each combination of compilers requires a different process to do it and starts to get very complicated when you have to link an extension to two or more libraries which have been built with a combination of compilers and compiler versions. This method should work for all combinations of compilers because it only requires that function calls are compatible; if they are not then the compilers cannot be used together anyway. Changes ------- The following lists the changes made to Tcl. o Improved the support for AIX shared libraries. By making use of export/import files I was able to create Tcl and Tk libraries which are shared objects rather than archives containing shared objects. This makes it possible to dynamically load Tk on AIX now. Extensions which are correctly configured should not need to make any changes, extensions which are also libraries will need to create an export file and use that instead of the library when linking. See the Tk configure.in file to see how it does it. o Extended the package mechanism to support stubs, and added the ability to determine whether a package has been loaded without causing it to be loaded. See the PkgRequire.3 man page for more details. o Added ByteArray type. There is a major problem when porting extensions which wish to handle binary data from 8.0 to 8.1. In 8.0 they can use string objects to hold the binary data, unfortunately this will not work on 8.1 because the internal representation of strings is now UTF. Binary data in 8.1 is handled using ByteArray objects. Therefore I have added them to improve the upward compatibility. o For each function which takes a variable number of arguments I have added a function which takes a va_list instead. This is so that I can create wrapper functions around them. These new functions names are the original function names appended with "VA" The following lists the changes made to Tk. o Modified Tk_Main and TkConsoleCreate to work properly with stubs. The problem is that those functions and the other Tk functions that they use should not actually be in the standard Tk library because they are only really needed by applications. The problem that they cause with stubs is that they are called before the Tk initialisation function is called and therefore the pointers to the Tcl stub tables have not been properly set up in the Tk library. The solution was to make them take a Tcl_Interp parameter (which is used to get the Tcl stub table pointers) and to get them to initialise the pointers as well. The changes to Tk_Main will not break any existing applications unless Tk is built to use Tcl's Stub interface; in which case a simple rebuild is all that is required to fix the problem. The changes to TkCreateConsole should be totally backwardly compatible. Future ------ There is one more thing that I have left to do which will make this mechanism even more useful; that is to make the libraries use their stub interface to access their own functions. This will allow dynamic extensions to change the behavior of the libraries, either to fix a problem or to add some function. Plus Patch ---------- The plus patches also have a stub interface available for them although it only supports a subset of Tcl functions and one Tk function. They are available from Acknowledgements ---------------- Jean-Claude Wippler For the initial seed idea from which this all came from and also for his continued support and feedback during development. Jan Nitjmans For his ideas and feedback which improved the final result and made me think. Contact ------- Paul Duffin