Copyright 1995, 1996 Andrew Worsley May only be redistributed for free PSPTOOL Documentation RELEASE 0.5 psptool is a tool to automate collection of some simple but very useful statistics on a programmer's work so they can improve their programming method. By using psptool the programmer can with only a modest amount of effort gain many insights into their programming that will help them manage and improve their programming process. It's my experience that by making a modest effort to collect data (this tool helps) and properly using code and design reviews a programmer can gain the following: o 10 fold reduction in defects per line of code (LOC). You can feel confident in your code. Compare this to how you might feel if you just removed 10+ bugs in the test phase. Achieving levels of 10-20/1000 lines of code at the test phase is quite practical. o A change from spending most of the time in the debugging phase to spending most of the time in the design phase. o Detailed statistics on time, defects(bugs) which can be used to track and improve your programming. e.g. learn your most common types of bugs and take action against them. o Make good estimates for the time to implement software based on no. lines of code. With in 30-40% if project is not too different to previous work. o All this for only at most modest extra time spent. I don't have much data on the old ways because I don't want to go back to them. Features: o Automates many calculations - you only need to enter data for: Lines of code measures, bug details and click to mark entering the next phase. o Automatically estimates time and defects based on LOC estimate and historical data. o Supports code and design reviews by calculating many metrics such as Yield, Defect removal leverages so you can see how much benefit reviews give you. o Records time spent in each phase of a process o Records details of each bug including time spent fixing o GUI interface with one click required for common actions o One click pause/continue button for all timers including bug timers o Ability to edit times in all timers. o Supports multiple concurrent Bugs o All text messages/buttons configurable (for different language support) o Editable file of Bug types with unlimited ability to refine categories o One easy command installation o Written in highly portable Tcl/Tk Requires: (At the moment) Unix workstation Tcl/Tk installed (is a wish script) Installation: 1. Extract the files i. If received via mail as a uuencoded file feed the mail into uudecode to reveal psptool0.?.tar.gz thus: uudecode < mailitem ii. Un-gzip using gunzip and untar the files thus: gunzip < psptool0.?.tar.gz | tar xfv - Should yield doc.txt psptool os.tk psp.tk reports.tk pspdefs initpsptool 2. If you are happy to install it (it creates file ~/.pspcnf and directory ~/.psp) in your home directory, then run the installation script thus: ./initpsptool It should report creating the PSP directory and installing several .tk files in there. Take note of this directory for it is where all the collected data is stored 3. Try it out by running the script thus: ./psptool It should create a window with a "New Project" button and report on creating a statefile which is used to save the current state of the project. i. Click this button: It will become a New Project window. ii. Type in the name of the new project (no spaces or '.'s allowed) and press return or click Okay to start a new project. The Quit button exits the psptool, saving it's state. The Cancel button reduces the window down to the small New Project button. I plan to add configuration options to here. In particular to allow details like the programming language and the Software Process to be selected. Creating a new project will create a file in your psp directory (~/.psp/.log where is the name you gave your project) in which the details of the project are logged to. It will also report writing out a bugtypes file. If a bug type file doesn't already exist it creates a default one for you. It will also report there is no "to date" information for a lot of fields. This is because it has no historical data for you (which it uses to calculate projections). After finishing the first project it will save the results in a todate file which subsequent projects will read in. It will also create two windows, a small psptool window and a very large (and slow to create) Plan Summary window. The psptool window is the control panel of the psptool. Click on the Close button of the Plan Summary window to shut it down for the moment, so we can look at the smaller window. The psptool window has two rows. The top row contains 3 buttons which contain respectively, the name of the current project, the phase you are currently in and the word "Bug". The first button (project name) is actually a pull down menu which contains 3 entries. A pull right menu (to enter a project phase), a plan summary action (to create the Plan Summary window if it doesn't exist) and a Quit entry to save the state and exit the psptool. The second button, containing the current phase will be showing "plan" the first phase of your software process. Clicking on it will advance to the next phase, but don't do that just yet as we should complete our project plan. The third, "Bug", button is used to capture Bug (or defect) information. When we find a bug we click on this and record the it's details for later analysis. Underneath, is the second row, which consists of the Run/Pause button (now Run which means clicking on it will make it Run) and next to it the current "time in phase". The Run button will be highlighted in a different colour. iii Click on Run. The Run button will change colour and become the "pause" button. The current time will start to accumulate. iv. Click on pause The pause button will change back to Run and the time will stop accumulating. You can edit the current time at any stage by clicking on the current time and it will pop up a window to allow you to edit it. If you changed your mind about editing it, just click on the cancel button to cancel the edit. Pressing the key on an edit time field will move you to the next field (for minutes) or complete the time edit. v. Plan Summary. Before we move on to the next phase by clicking on the phase button (which will showing the phase name "plan") let's complete our plan. It will only take a few minutes and will help improve our estimating ability which is vital for project planning, otherwise we will always be chasing deadlines and we can't produce good quality code in those situations. First make sure the time is accumulating by clicking on the Run button if it you are not accumulating time. 1. Re-create the Plan Summary window again by pulling down the project menu (top row first button) and selecting the "Plan Summary" entry. This window summaries all the metrics collected over one project. The sunken fields indicate where it expects some data to be entered. We will ignore most of the entries for the moment (all we would be here all day explaining them). 2. About half way down on the left side is the Program Size (LOC) section with "sunken" several entries. The LOC means Lines of Code, which is what we use for size estimation. It doesn't much matter what you use for your LOC as long as it is consistent and linear (i.e. 2 LOC = 1 LOC + 1 LOC). Making it consistent does require work (e.g. Coding standard and a line counting standard) but we'll assume this for the moment. We refer to the book at the end for more details on how to do this. It tedious to count lines by hand so people write programs to do this which is not a trivial matter as general intelligent line counter can end up being a little compiler! If you don't have any line counter program I suggest you use something like the "wc" utility and realise you are counting comments as lines of code. 3. Draw up an estimate of your project - again the book gives a detailed way to do this. For the moment just make a rough estimate based on how many functions or classes X an average size number. 4. If you are modifying an existing program, measure it's size (e.g. use 'wc' on all its files) and type in that number into the "Base (B)" entry area. Type 0 if you are starting from scratch. Pressing return will automatically move you on to the next field. I use Ctrl-H key to delete the previous character in an entry, one day I might add a binding to make the "Del" key do the same. For example enter 100 and press return. This will move the cursor to the next entry below. 5. Estimate how many lines you will need to delete from this "Base" code for you project. Again entry 0 if you are not deleting any lines or starting from scratch. For example enter 25 6. Estimate the number of lines you are going to be changing, alternatively you can count these lines as deleted and then add back in as new lines - it doesn't matter. For the example enter 25 again. 7. The Added lines will be calculated from the New & Changed lines later so Added is raised and can not be entered. 8. If you keep a re-use library, a very sensible policy these days, enter in the LOC of re-used code. Note this should not contain any of the lines you recorded in your Base estimate (think of Base as files which you are modifying and Re-use as libraries you linking in). If there you don't have a re-use library, just enter zero into the "Reused (R)" entry. For the example enter 0. 9. "New & Changed (N)" field should be your modified and added code. i.e. just add your estimate of modified lines to you estimate of new lines you add to give this number. If you are generating new code for your re-use library (e.g. a new re-usable class) don't put it here it goes in the next field. For the example enter 100. 10. "Total New Reused" this contains the LOC of new classes you are creating for this project, but which will be stored in your re-use library. Don't count this as part of your "New & Changed" code or you will be double counting. For the example enter 0. 11. This will take you to the next section to the right "Time in Phase". It has only one sunken entry, the Plan total. Later when the psptool has some historical data from you it can estimate this field automatically (just a linear projection) based on your LOC numbers. For the example enter 240 (minutes = 4 hours) and press return. 12. The next entry will be the "Defects Injected in Phase" which means how many bugs. This is at the bottom right - so you will probably have to use the scroll bar to get to it. Use the mouse to drag the scroll bar to the bottom. For out example enter 10 and press return. A few seconds later it some calculated numbers will appear for those fields it can calculate. 13. In particular the LOC section will display 75 added (100 New and Changed lines less 25 lines modified). The Total will be 150 in the Total field (which means the new total LOC) will be 150. i.e. 100 less 25 deleted plus 75 added gives 150. 14. On the top left (may need scroll bar) this gives you 25 Loc/hr (new and changed = 100 / 4 hrs) productivity rate. The total Defects/KLOC (KLOC = 1000 LOC) is 10 defects / 100 New & Changed = .1 Defect/LOC or times 1000 LOC = 100 Defects/KLOC. i.e. your plan would require you to achieve a rate of 25 LOC per hour and only 1 Defect per 10 LOC. 15. Now you have finished the plan Close the Plan Summary window or just click on the Plan button of the psptool window and go into the design phase. If you want to alter your plan some more click on the sunken entry you want to change and press the re-calculate button to calculate the new numbers. vi. Click on "project phase button" now showing the word "plan". The phase button changes to the next phase "design" and the current time is logged into the project log file (named above as ~/.psp/.log) and the current time reset to zero. vii. This is the phase you would design your program (on paper say) before typing it in. If you "don't design" then just click on the phase button again to move straight through. One extension will allow people to choose or even create there own software process which includes selecting their own set of phases they would use. For the moment we have lots of phases so people who don't use them will have to just click through the ones they don't use. viii. If some one interrupts you or you want to stop and do something else - just click on the pause button. If you want to avoid recording any time in a phase (say you want to click right through with out recording any time) just click the phase when the psptool is paused. If the current time (for a phase) is zero it won't even record an entry for it in the log file. Alternatively you can just use the pull right menu on the project name to go right to the phase you want, but that's sometimes a little more effort. People have asked for a "go back" button, which should be easy to add - but I just haven't gotten around to it. ix. When you have finished designing (or if you are just following the example just edit the design time and change it to what ever seems reasonable to you) click on the phase button to enter the next phase. This will be "Design-Review". Reviews are a disciplined process of checking for mistakes using a defined check list and set of defined checks. Again refer to the book for a whole chapter on how to do these properly. One feature I plan to add is a way of automating the check list so you can see what check you are up to and indicate the checks you've made. We won't say much more about them, except that they are key to achieving the 10 fold reduction in defects present in the final code. It turns out that finding bugs in reviews is faster than in testing phase but this counter intuitive result is only demonstratable if you record how long you spend in each phase and do reviews correctly. So most people never do reviews, the psptool helps you break into reviewing by automatically collecting data that shows which approach is best. Almost all people (including myself) don't believe this until they collect data on themselves. They are not convinced by data from other people showing this. x Defects/Bugs. While we are on the topic of bugs let's see how to record one. First make sure your time is accumulating e.g. by clicking on the "Run" button if time is not accumulating, and then click on the "Bug" button. Up should pop a Bug reporting window. It has a timer as well which will have a different time to that in the psptool window if you waited before clicking on the Bug button. xi. Click on the "pause" button. Both timers stop. xii. Click on the Run button again Both timers restart. You can also edit the each timer individually if you want to by clicking on the time field. The bug window has two rows. The top row (from left to right) has the bug no., the current time against the bug, a type menu button (showing "select type" until the type is selected) and a cancel button. The bottom row has a prompt "symptom" with an entry field next to it for you to type in a description of the problem. xiii. Pull down the type menu and select a type. It will display a pull right menu of different types. Some types have further refinements of the types. It is set to display the types verbosely but there is another more compact setting for the types. You have to edit the pspstate field at the moment to select this (see later) but one day it will be a project configuration option. The type file lists all the types it will display. The User can edit this file and hence refine or replace existing types with their own ones. The User can add new subtypes or refine existing types into sub-categories. To add a new type to the existing category select the last field in that menu (Add new bug type) and a new bug type window will pop-up. You can cancel the addition by clicking on the cancel button. If you Click Okay with entering reasonable bug name it will print an error message. The name must be a a single word made of letters - numbers - underscores (no spaces or '.'s. type return and enter a long description and press return (or click Okay) to actual create the type. It will report that it wrote out the new bug list file out. The selected bug type will automatically be selected as the bug type. If you want to refine an existing bug type into sub-categories select the bug type from the menu. Then click on the menu button with the right button and up will pop a new bug type entry. Except that now the bug menu entry for original parent type is a pull right menu with the old parent bug type as the first entry and the child entry as the next. See later for more details of the numbering scheme and editing the bug type file by hand. Note that refining bug types like this is helpful to track down the cause of bugs. Basically you keep refining a bug type until it corresponds to a single cause and then you can work on eliminating this cause. When all bugs are lumped together its hard to discriminate the causes and hence to work on them. When a type is selected the type pull down menu changes it's name into the type selected and two other buttons are added to the window. They are the injected phase menu button (labeled "plan") and the fixed button (labeled "fixed" and raised). The injected phase - will be the name of the phase where the bug was "injected" into the software phase recorded for this bug. Clicking on the Fixed button will mark the bug as fixed and record it's details in the log file. The injected phase is the previous phase by default and is a pull down menu which you can use to select another phase. In addition you can record a brief comment about the bug. Initial a symptom, describing the appearance of the bug and when it's found what was the problem. These are useful if you ever review you bugs and want to be able to remember what they actually were. You can leave them blank otherwise. xiv. pull down the "plan" menu button (phase injected setting) and select external. The menu button changes it's name to external. I use external to record bugs injected from sources external to the project (e.g. library or compiler error). xv. Click on Fixed The bug window disappears and an entry is made in the project log file recording the details of the bug. xvi. Repeat these actions to record the time and bugs found in each phase. The Code phase is meant for typing code into the computer (unless you don't design in which case it is the time you spend up to the first compile). Record each line that you actually delete and those you modify, perhaps by a mark for each deleted or modified line. You will need these later. Compile is the time you spend until it first compiles with out error and test if the phase you spend debugging it until it's passed all your tests (or you are happy it's working). xvii. You will finally move into the postmortem phase, in which the giant plan summary window will pop-up again. This time there is less fields to fill in as the defects and time have already been recorded. Fill in the LOC section. Now the middle column of entries will be sunken. This column corresponds to the actual values (as opposed to the Planned values) that occurred during the project. Fill in: Base (B) - put the size of the pre-modified code (it may be different from planned if you needed to modify more code than you planned). Deleted (D) and Modified (M) - these you should have from your recordings during the coding phase. If not you are going to have to guess from what you have information you have, which can be pretty hard. Reused (R) - again the size of the re-used code from the re-use library. Total (T) - This is the total size of the project and will be used to calculate the added value by subtracting off the Base and adding the Deleted and subtracting the Reused. The New Reused - The new re-usable code developed during this project. The other values are calculated from these as A = T - B + D - R and N = A + M. xviii. From these values many useful numbers are calculated in the Summary section in the top left corner of the Plan Summary window. Including the Loc/hr (Lines of Code per hour), Defects / KLOC (1000 lines of Code) which is a measure of buggyness the out going code is likely to have. A reasonable estimate of how many bugs left in the code is that it will be the same order of magnitude as that found in the test phase. Another very useful number is the Yield - which is the percentage of bugs removed before the compilation was attempted. This is likely to be very low unless you do reviews (code and/or design reviews). Will good training and practice it has been reported that most people can achieve 70% or higher yields. On the top right hand side are summaries of the "defect removal leverage efficiencies". Basically these measure how many bugs per hour are removed in various phases and emphasise how much better than testing (which usually has the lowest rate) reviews are at finding defects. The DRL values are the same rates expressed as multiples of the rate in the test phase. i.e. how many times faster than testing. If you don't do reviews the only phase to compare it with is compiling. Finally the time and defect areas show the break down of time and bugs by project phase. Typically if you do not do reviews then 40-60% of the time is in the test phase. When you combine this with the fact that test has the lowest rate of removing bugs it provides compelling evidence that doing reviews is cost efficient. The "To Date" fields represent the values summed over the projects to date (which for the first project will be the same as the Actual entries). I will briefly mention the other fields in the Summary section for completeness but read the book listed below for a detailed description of them. Planned and Actual time are just that. They are used to compute the CPI (Cost Performance Index) which is the ratio of the total Planned time to the total actual time. As your estimating improves this will move near to 1. The % Reused and % New Reused are the percentage of the project's code which is from the re-use library and % of the project code entering the re-use library. High values for this mean you are getting more leverage out of your re-use library and so hence saving time in your project. The Appraisal COQ (Cost of Quality) and Failure COQ are the percentage of time spent preventing problems (Review times) and repairing problems, e.g. Compiling (i.e. getting it to compile) plus testing. The COQ A / F Ratio (is the ratio of the two. The book described below suggests a good balance to aim for is around 2. xiii Pull down the project menu button and select exit. The tool exits having recorded the current entry in the log file xix. Restart the psptool as by typing: ./psptool The psptool comes back in the psptool window state, ready to continue work where it left off. xx. Click on the project phase button ("test") The project is finished so it reverts back to the "New Project" window ready to start on the New project. xxi. When you start a new project you only have to fill in the LOC entries of the Planned field and the other entries are estimated based on the historical data. Basically it takes the ratio of the to date value and the LOC to date and multiplies by the estimated LOC (The New & Changed (N)) value to estimate the value. For values which are broken down by phase (such as defects and time) it estimates the total this way and then uses the percentage by phase to estimate values for each phase. Hence the estimates have fractional values which really represent a mean value. i.e. what's .5 of a bug? That's basically all there is to the tool at the moment but there are many areas for improvement. By distributing it freely and documenting it I hope to stimulate others to either enhance the tool (send me you extensions if you would like comments and hopefully to fold it back in to the tool) or develop other tools. Obvious extensions to my mind are: 1. Automate the estimation process further. If the user were to nominate a LOC counting program, the user could merely type in the names of the files and psptool could automatically compute the Base and Total LOC values. Further extensions might allow automating the counting of Modified and Deleted lines (perhaps using "diff" or some counter button). 2. Automate the review process so the user can keep track of which reviews they have done so far and prompt them about what to check next. 3. Extend the estimation to provide estimating ranges (based on linear regression methods). 4. Allow multiple concurrent projects and moving between them. 5. Allow visiting completed projects, perhaps to add new bugs found after the testing phase. 6. Allow printing of the summary sheet. 7. Provide tools to further analyse the data. e.g. Pareto charts for bug types. Or control charts on defects/LOC (by type?) by phase to see if there is any significance to a rate change. 8. Record changes/suggestions to the Software Process. 9. Allow changes to the Software process (e.g. new phases or sub-cycles) 10. Scheduling / Mile stones for larger projects. Details: .pspcnf: The configuration file (~/.pspcnf) defines many configuration variables which are labeled for you to edit. In all files a leading # makes the line a comment. Blank lines are also skipped. In the configuration parameters are defined by lines of the format: