Tokenizer/DeTokenizer changes from Version 1.0.1 to Version 1.0.2

1. One of my colleagues, who is maintaining a very large and convoluted body of Legacy code, asked me questions like:  "Where is such-and-so definition coming from?"  "Which version of it is being used here?" "How can I find where exactly that is in the Binary?"
   
   I decided the Tokenizer should supply the answers.  For the first and second question, I beefed up the Tracing feature both to give more information at the time a Traced Symbol is created, and to give a Trace-Note Message every time a Traced Symbol is invoked.  It is not fooled by aliases, and a failed attempt to create or invoke a Traced Symbol will also cause a Message to be displayed.  The differences in TokBrack/TokBrkTst_04.Log, TokeErrs/DupNams.Log and TokMisc/MulDev_02.Log show this.  For the third, I modified the Message format to show the current Output Position (and possibly the position relative to the end of a PCI Header).  I do not show this here; it can be seen in just about any of the .Log files in the Test Results directory.
   


2. The emit-fcode command, in the previous version, was held to the same restrictions as the next-fcode command, i.e., limited to the range of user-defined FCodes 0x800 to 0xFFF.  The new version permits numbers in the entire range of valid FCodes, 0x010 to 0xFFF as arguments to the emit-fcode command.  This is shown in TokeErrs/MiscFeatErrs.Log


3. It came to my attention that not all platforms correctly process the Return-Stack operators in Interpretation mode, (even though the Standard requires them to), and that a developer of Device Drivers intended for the full range of possible platforms would need to avoid using those.  I added a Special-Feature Flag called Ret-Stk-Interp to allow the user to select whether such a usage should be permitted (flag enabled) or treated as an Error.  The differences shown for TokeErrs/DecodProp.nrsi.Log show the difference, in Version 1.0.2 (rather than comparing with Version 1.0.1, which does not recognize this flag), between using the flag in its default (enabled) form (to allow Return-Stack operators in Interpretation mode), and disabling it.  The "normal" comparison in TokMisc/BatchTst.FHelp.Log shows the help-message for this flag.


4. In the course of my colleague's work, a need arose to devise a way to track the progress, not only of the execution of the sequence, but also of its interpretation.  The [function-name] directive would be well-suited for this need, if it were allowed to persist after the end of the current colon-definition.   The change in TokMisc/MiscFeatures.Log shows the result.


5. My colleague's work involved detokenizing a body of FCode that had been created by another, non-Standard, tokenizer; it included a Vendor-Specific FCode token for which the the "Additional FCodes" facility was not sufficient.
   
   It also uncovered a subtle error:  In the previous version of the DeTokenizer, if, while 8-bit offsets were in effect, an unexpected "end" byte followed by an Invalid FCode Start Byte was encountered, the remainder of the detokenization would automatically (and unannouncedly) revert to 16-bit offsets.  When the sequence came back into sync, new, unexpected and unnecessary errors would be announced.  The "old" side of TokMisc/VSFCtest.DeTok shows how this happens; it starts with the byte at offset 18, and manifests in the change at offset 28.
   
   To cope with the non-standard Vendor-Specific FCode token, I introduced an enhancement to the "Additional FCodes" facility:  predefined Special Functions with reserved names and complex non-standard behavior. In this Version, only one such is defined: double(lit) --  but the infrastructure is there to introduce others as they might be needed.  The changes in TokMisc/VSFCtest.VSfc.DeTok show how it works.
   
   You can examine the "Additional FCodes List" file used in this test-case by looking at TokMisc/VendSpecFCodes in the Test Suite Sources ("Latest/testsuite") directory.


6. Finally, a slight change to the Local Values Support file, visible in LocValSupport/LocalValuesSupport.fth, causes the constant named _local-storage-size_ to be defined under all options, so that it will remain available for examination during development and testing.