lfcbase:
- compiler warnings fixed for class Chain::toUpper and toLower
( dynamic wText allocation )
cego:
- procedure cache
- Added admin command set maxpagedelete to set up maximum number
of pages marked for delete before forcing a checkpoint.
- CegoQueryManager: _authUser has to be stored locally,
otherwise setUser method looses user information, if auth was not enabled
- new admin command object usage to show usage of tableset objects
to detect any usage leaks and to get an overview of objects in
use by running database threads
- backupManager: indicate number of logs to recover up to crash
for the corresponding backup id.
- Fix added in CegoRecoveryManager::recoverCurrentTransactionLog
- Added list transaction command
- Added cgclt command show transaction to indicate detailed
transaction information
- in CegoPredicate::Constructor, for LIKE-Constructor, the case
ISNOTLIKE was not treated with the right condition
( ! isNegated && isCaseSensitive => isNegated && isCaseSensitive )
- fix in CegoBTreeValue::valueToFVL
- log mesage cleanup done for CegoRecoveryManager, CegoQueryManager
and CegoTableManager.
- Changed return code of CegoBufferPool::writeCheckPoint to int to
return the exit code of the optional escape command.
This exit code then is indicated in the admin console
- Consolidation work done for CegoTableManager and CegoAuthManager
- Code reorganization for CegoAuthManager, CegoQueryManager and
CegoTableManager: TableManager contains low level data access methods,
QueryManger contains high level access and AuthMenager contains
authorized access methods
- Post patch for this version : In CegoFieldValue::castTo, changed
__dateFormatLock to writeLock. This is needed, since with readlock
the _dateFormatList could be traversed in parallel by the Datetime
constructor. This may lead to concurreny issues
Author: Björn Lemke <lemke@lemke-it.com>
- btree dump implementation to CegoAction::execDumpBTree
- cleanup of dump statements ( moved dump table / btree statements from
CegoAdm.def to Cego.def ).
Dump still supported for table objects
- In CegoFunction::evalFieldValue, case USERDEFINED,
removed cout debugging statements
- A recovery issue occured when using user function is expression list
for update operations. For expressions containing user functions,
the values for CegoQueryManager and tabSetId have to be setup explicit,
since this is not done by redo log decoding. The corresponding setup
method setQueryManager has been added for all involved classes
( CegoExpr, CegoTerm, CegoFactor, CaseCaseCond, etc )
- new global variable CegoFieldValue::__recoveryTS has been introduced.
During transaction recovery, this variable is setup to the
corresponding recovery timestamp. So the constant datatime value
sysdate is setup to the current recovery timestamp value
( otherwise the current timestamp is used )
- Introduced crash recovery autocorrect option ( direct and post )
for index rebuild either direclty after invalidating index during
recovery or at at the end of the recovery procedure.
- indirect advantage : in case of many insert operations during recovery,
this might lead to a faster recovery time, since index is
invalidated and not treated
direct advantage : in case of index oriented update operations,
this might lead to a better recovery performance
Author: Björn Lemke <lemke@lemke-it.com>
lfcbase:
- Improved error message for File::open with strerror details added
- In File::operator = and File:operator +=, after file copy operation,
source file descriptor was not closed.
This might lead to file descriptor leaks.
cego:
- Many fixes for crash recovery and startup, write logs, etc.
- rework tableset verification procedure
- removed logback.xml to avoid logging side effects
- Added admin command "verify locks" to perform a test aquire
of all lock manager locks read and write
- Fix CegoQueryHelper::maxFieldSize.
The returned size must be at least MAX_NULL_LEN
- CegoTableManager::checkBTreeIntegrity support for allowDuplicateNull
feature ( multiple null values in unique btree )
- CegoLockHandler, check for unlock methods to check for lockId == 0
- Completed verify locks command with missing pool, querycache, tablecache,
db and xml locks
- CegoClient: query abort via interrupt character fixed
- fix CegoAdminThread::srvVerifyTableSet for view loop iterator
- Before decoding values via CegoQueryHelper::decodeFVL,
the btree schema has to be sorted in terms of field id.
Otherwise, decoding might be not complete in case of
non-ascending btree attributes.
fix in CegoBtreeNode::verifyLeafFull
lfcbase:
- Screen class regarding, some int to unsigned fixes
cego:
- patch in CegoAction::execProcCall and CegoAction::execFuncCall,
fix locking in case of exceptions
- CegoAdmScreen to support current backup and restore features.
Also added online help
- added dbcheck/failcheck to also check correct handling of failed queries
- Fixes added for CegoFieldValue:castTo. In case of thrown data
conversion exceptions, is is ensured, that _pV is set to _staticBuf
after freed, otherwise memory fault cause by double free in desctructor
may occur
- initfile handling fixed
- CegoAdminThread::executeRestore improvements
with online docs updated for the restore procedures
- Added patches to CegoBufferPool, CegoLogManager and CegoArchiveThread.
- database xml file will be updated during checkpointing runs.
- Removed PIDFILE attribute from db xml.
PIDFILE should be specified by database startup with pidfile option
- remove distributed mode, was never used in production (2.53.x -> 2.54.x)
- Changed tableset ticket handling for backup recovery
- big rework for crash recovery
- Added point in time recovery option to restore to a specified lsn.
- Cleanup and reduction of logging in CegoRecoveryManager
- fix in CegoSelect::nextJoinTuple
in case of result ordering, the check for unknown order attribute
has been removed. The attribute list returned with getAttrRefList
may also contain attributes for nested ( or parent ) query attributes
( attr not found is returned in any case,
if there is an invalid attribute reference ).
check125 has been added to verify this case.
lfcbase:
- Completed integer cleanup for Pager class,
replacement of some asInteger with asUnsigned method calls
( BigInteger, NetHandler, etc. )
cego:
- Added several patches for trigger objects
( check self refering tables in trigger block to avoid recursion,
object invalidation handling fixed )
- In CegoRecoveryManager::recoverCurrentTransactionLog,
completed recovery procedure for trigger and alias objects.
For drop operations, added CegoDatabaseManager::removeObject calls,
to remove corresponding objects from object cache
- integer cleanup, all int values will be replaced by unsigned
or unsigned long, where is no need of nagative values
- for those ports:
devel/lfcbase
devel/lfcxml
databases/cego
databases/cegobridge
databases/p5-DBD-cego
cego:
- Fix added in CegoDbHander::acceptSession and CegoSerial::readChain
to treat empty or invalid session requests
- fix in CegoDbThreadPool for bad message handling
p5-DBD-cego:
- Fix added for execute return value and $sth->rows value to
return affected rows for update statements
lfcbase:
- Warning elimination in several code modules.
-Wall should print no warnings anymore (with clang)
cego:
- Added patch for CegoJoinObject for correct and complete decoding.
This is needed for database recovery procedure if nested join select
occurs in update statement, e.g.
update t1 set b1 = ( select max(b2) from t2 inner join t3 on t2.a2 = t3.a3 );
- Warning elimination for several code modules.
In CegoContentObject encoding and decoding method,
the obsolete key flag in the schema information was removed.
This leads to a incompatibel datastructure and requires tableset
export/import for an uplift ot this version
- To toChain method of CegoTableObject, CegoJoinObject and
CegoAliasObject has been improved to treat undefined tabSetId
argument ( defTabSetId = 0 ).
lfcbase:
- Added patch in File::hasData to check return code of pool library call
- Added fix in basecheck ReplaceTest to return valid results
cego:
- For CegoClient and CegoAdmin changed argument evaluation to handle
pipe input data in a better way.
Pipe mode is just used, if no batch or command mode ist setup and
if data is available at stdin.
- This should also work for FreeBSD, when cgclt or cgadm is used
via cronjob. It has been observed, that if a program is called by
cron, poll returns data available at stdin also if there is no data
available.
- In CegoAdminThread, for tableset and table import procedure added
switch to next backup branch.
This is required, since during import, logging is disabled.
So after import, a new backup is needed.
A new uses 'sudo' has been added to transperently depends of the
proper flavor of sudo 'default' or 'sssd' depending on the default
version set by the user.
If you prefere to use 'sssd' flavor of sudo, add the following
to your make.conf:
DEFAULT_VERSIONS+=sudo=sssd
All sudo consummer ports has benn updated to USES=sudo
Reviewed by: 0mp
Approved by: mat
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D52160
NOTE: Since this patch changed content object encoding,
the upgrade requires a tableset export/import !
- Fix added for CegoLogReader. __dateTimeFormat has to be setup
to print date values in log data
- In CegoContentObject, encoding and decoding was not done for tableAlias.
This might lead query errors during crash or backup recovery for insert
or update statements with nested selects.
Table Alias has been added to encoding.
- Fix added in CegoAggration decoding constructor.
_pExpr must be set to 0, otherwise this might result in seg fault
during tableset recovery.
- Added stdin mode for cego admin console cgadm, so instead
of using option --batchfile, admin commands can be feed by pipe
- Cleanup and completion work done for CegoLogReader ( cglog ).
For this also the abstract method CegoDecodable::getFormatted
has been changed to getInfo, which provides a more compact output format.
cglog is used to print ( binary ) redo logfiles in a human readable form.
- For CegoLogReader added stdin feature, so log file can be piped
via stdin to cglog
- Extended backup list to indicate most recent redo log timestamp.
For this, the backupManager utility had to be adapted to retrieve
last redo log timestamp from arch and tape directory
- Added CegoXMLSpace::adjustPageOffset method to adjust page offset
attribute after tableset recovery. This is needed, if a recovery
is made from scratch with a new initialized database xml file
- Changed method CegoXMLSpace::nextTSID to generic.
Instead of calculating the next tableset id on XML_MAXTSID_ATTR,
the id is calculated based on all defined tablesets.
In case of a from scratch recovery ( with a new initialized db xml file ),
this ensures id consistency
- improved utility scripts for online backup to tools directory
( logManager, backupManager, backupExecuter )
- more stabilzation for online backup recovery from scratch,
e.g. checking for existing tableset set entry in db xml during recovery
and allow non existing entry.
Perhaps it might be useful to check pagesize of backup datafiles
and defined pagesize in db xml ...
- added tableset define parameter branchid.
This parameter can be used to set up an appropriate backup branchid
for from scratch recovery procedures
- patch added for scratch recovery.
If a tableset is defined for backup recovery, the tableset id
must be adjusted to the corresponding id in the datafile.
For this, new methods CegoTableManager::adjustTableSetId and
CegoFileHandler::readTableSetId have been introduced and added
to CegoRecoveryManager::recoverTableSet
- In CegoRecoveryManager::recoverTableSet: shifted ticket recovery
before file registration.
So all needed tableset id values are set up correctly and there's
no more need to adjust any ids.
Just root path and branch id must contain appropriate values
for newly defined tableset to be recovered
- Fix added in CeegoXMLSpace::getTableSetBackupBranch,
removed a double V() operation which caused XML locking issue
- Changed BRANCHID attribute to NEXTBRANCHID in XML root tag,
to clarify the meaning.
- Changed Backup Branch Id from timestamp value to
database wide unique numeric id.
For this CegoXMLSpace::setTableSetBackupBranch method
has changed to nextTableSetBackupBranch
- Block variables in CegoDistCursor still have not been handled
- For this, the constructor has been expanded to take a block variable
as an argument. The block variable then is propagated to the
corresponding eval methods ( of CegoPredicate objects )
- check124 has been added to illustrate a case, where this patch is needed.
- Improved input handling in CegoClient.
For interactive mode, trailing LF, CR and multiple semicolons are cut now.
- Added command line option lockexpire with integer parameter seconds
exceeded.
If set, this option will check for the configured daemon lockfile
and if the timestamp of the lockfile is older than now - seconds
exceeded, the lock file is removed and the database is started.
Database crash recovery seems to be stable, so this option is made available.
- In CegoDbThread::serveRequest, added further tableset states (
BACKUP, CHECKPOINT ) where sessions are accepted
- fix performance for CegoPredicate
- CLI: threadinfo db, lastAction was added to the threadlist to
indicate the query currently processing
- format improvements for CegoAction::getPlanString to indent subqueries
- handle join master predicate in the most optimal way.
To show several condition cases, check123.sql has been added to
demonstrate the join optimization for different join cases
- fixed some cases for master predicate handling
example:
plan select cpid from companyview cv where lower(cv.cpname) like '%%'
or debnr = 3;
shows the execution plan for the query.
lfcbase:
- Fixed BigDecimal class for internal constructor to
normalize zero values to "0" and in scaleTo method to treat
zero values correctly
- Added class Replacer to support string replacement based
on regular expressions ( like unix sed )
- Fixed BigDecimal, scale handling still not correct for some cases
- Replacer: The match token has been changed to $ char to treat
database integration
cego:
- Performance optimization added in CegoSelect and CegoDistCursor
for or-condition handling:
If an or-condition is recognized, the predicate is propagateed
to the corresponding join level cursor and the evaluation is handled
by the cursor.
This can avoid large table cursor result sets for the embedding
join and might improve overall performance of the query.
- For the new introduced query optimization, several code extensions
have been added. In CegoPredicate, new methods map, getReduced,
mapExpr and reduceExpr have been added to convert the "master"
predicate to an appropriate form, which can be handled by CegoDistCursor.
Also for for CegoDistCursor, the distSetup methods have been expanded
to propagate the master predicate
- Added fix in CegoAction for the following query:
select invid from invoice where status = ( select 'OPEN' );
In CegoAction::noFromOption, for _coListStack has to be pushed empty coList
- Added verify010 db check with a small invoice er model. This is
used for advanced query verification with multilevel nested views
and other dynamic stuff which is also relevant to proof valid results
from query cache
lfcbase:
- BigDecimal::scaleTo: added RoundMode parameter which can either be
UP ( round up )
DOWN ( round down )
HALFUP ( round up if neigbours equidistant or higher ) or
HALFDOWN ( rond down if neighbors are equidistant or lower )
cego:
- Added support for lfcbase-1.21.8 ( BigDecimal::RoundMode ),
Round mode ist setup in dbxml via database ROUNDMODE attribute
( either UP, DOWN, HALFUP, HALFDOWN )
- Added performance patch to CegoSelect::buildJoinConditions. pPred
is set to checked if analyzed as complete, also if the predicate
is external ( setup via setViewCond ) In the meantime, external
conditions are fully handeled via CegoDistCursor
- In CegoFieldValue::toChain, for case DATETIME resulting string
was corrected. In the meantime, we avoid date value construction
via date/scandate function, either the required value is calculated
via cast. So just the date string value is provided in the defined
datetime format.
- Added datetime cast operator to convert from string value to
datetime. If used, this avoids expensive real time cast operations
for queries and should accelerate execution time
- Fix added in CegoQueyHelper::evalAttrCond, COMPLETE value must
be returned in flArray loop and if not found PARTIAL must be returned
( return was missing )
- Patch in CegoTerm::getSelectQueryList to retrieve subselects
also for case conditions and nested conditions. So this affects
also class CegoFactor, CegoCaseCond and CegoCondition which have
to provide the getSelectQueryList method. This extension is required
to retrieve appropriate table sets for query cache decicisions
- Added tableset online check in CegoDbThread::serveRequest.
This avoids query execution of any connected client during recovery phase
- Improved log trace in CegoDistManager::startTableSet to log complete
stack trace in case of tableset startup failure
- In CegoAdminHandler::getUserInfo, increased size
of NUMREQ and NUMQUERY from 10 to 12, for long running systems
with > 10^9 requests
- Added pack mode in CegoClient to improve the display of
large string column values. Only the longest length of all values
in <n> rows is displayed, not the full length of the field itself.
Example: If the table definition for a string is 300 chars,
and the values are only 4-10 chars wide, the table is only
displayed with 10 chars.
- Fix added in CegoDistCursor::joinSetup:
for outerCond calculation, null value comparisons are also used,
since this leads to correct join results and can be optimized by
using btrees.
( set attrCond.getFilterCond(outerSchema, false) to (outerSchema, true)
for left outer and right outer join)
- In CegoSelect::getPlan, added parentJoinBuf setup for nested queries in expression list to evaluate join plan correctly
( parentJoinBuf is required for analysis in CegoQueryHelper::checkAttrCond)
- Add duplicate check for table aliases in CegoSelect::evalReferences
- Fix in CegoSelect constructors to initialize _joinLevel to 0.
A problem may arise, if the reset method is called without
a previous select analyse which happens in nested subselects
lfcbase:
- Added improvements to Pager class, e.g. case insensitive search and
message box at end of search
cego:
- Small adaptions made for Debian
( in cgwtest.c, cast added to malloc for CGClob and CGBlob allocation ).
There are still required some changes for the testsuite shell scripts
to satisfy Debian flavour ( expimpcheck )
- Patch added in CegoCondition and CegoPredicate for arithmetic
expression evaluation. The recent grammar modification limits
enclosed arithmetic expression to logic expressions. This leads
confusing results e.g. for
select ( a + b ) / 2 from t1 since ( a + b )
is evaluated as a condition with a logical value as the result
( true or false )
( caused by grammar production : Factor -> LB Condition RB )
Arithmetic expressions are just parsed with grammar production
Predicate -> Expr ( expression only ), so for this case now a
special handling has been added
( with new introduced method CegoPredicate::getExprOnlyValue )
lfcbase:
- To support several keyboard layouts, changed some window control
sequences from Cntrl-Key to ESC key
- Added virtual method hasData to Pager class. The method must be
implemented to indicate, if an execution returns a result set. If
no result set is returned, just a message box is shown and edit
mode is left ). This seems to be useful for DML queries ( insert,
udate, delete, create, drop, ...)
- In Pager class, added buffer scrolling in editor part. Preparation
started for Pager command history management
- Added history feature to Pager class. Command history can be loaded
and saved with readHistory and writeHistory methods.
Navigation through command history in Edit mode with Cntrl-P and Cntrl-N
cego:
- Adaptions made for pager and pager history feature
lfcbase:
- more patches for the Pager class
Removed init_color with custom color for select cursor since the
custom color seems to be poor supported for curses. Fixes added
to Pager::writeFooter
- edit-feature in Pager (must discuss this with Bjoern 8-)
- ignore SIGINT in Pager
- change from unsigned long long to long long value in Datetime class
- pattern search feature for Pager class
cego:
- Small fix in CegoBufferPool.h
- CegoClient adapted to new Pager API
- Queries now can be interrupted with Cntrl-C also in pager mode
- adapt to Datetime long long values ( e.g. date2str from int to long long )
lfcbase:
- Rework of Pager class, which is a API change
cego:
- Made some adaptions for reworked lfcbase Pager class
rows with multiline entries are more readable
- Added check script check119.sql to verify grammar expansion
for stored procedure condition
- CegoAction::procArg1() and CegoAction::procArg2() to check
duplicate parameter names
- Added check118.sql to demonstrate explicit table locking for nested
delete operation within a cursor loop
- Fix added in CegoFactor::getAggregationList:
for case QUERY and case FETCH, the not-supported exceptions
have been removed, otherwise this leads a useless exception
in CegoAttrCond::asConjunctionList where aggregations are just checked
- Added grammer change in Cego.def to ensure operator precedence
for AND and OR condition operator.
AND has higher precedence if no enclosing braces are given.
The result for the following queries should be the same
select * from t1 where a = 42 or a = 1 and b = 'X' ;
select * from t1 where a = 1 and b = 'X' or a = 42;
select * from t1 where a = 42 or ( a = 1 and b = 'X' );
- With cego version before 2.51.7, braces have been mandatory to
ensure operator precedence.
With the new grammar extension, the AND operator has a higher presedence
- CegoFactor::evalFieldValue, QUERY case, after query
evaluation, the parentJoinBuf has to be reset, otherwise an invalid
reference is used later to CegoSelect::getQueryId. This might result
in core dumps, if query cache is enabled
- Added querycache verification to db check routines
- Added querycache mode to command line options to set up query cache by batch
lfcbase:
- BigDecimal::div null value check
cego:
- CegoFactor::toChain to add braces for condition case
- CegoSelect::initAggregation, for sum and avg aggregation, aggregation
value is initialized with 0 and appropriate datatype ( instead of
null value ). This leads to numeric result 0 in case of empty result
sets ( instead of null value )
- Fix made in CegoDistCursor::distSetup to support FLA setting with
no cursor condition. This is needed for query plan evaluation for
nested inner/left outer/right outer joins
See check114 as a sample
devel/lfcbase 1.18.11 -> 1.19.0
- Changed Datetime::asLong to return signed long values ( long long ).
This is needed to support timestamp values before 01.01.1970,
which are signed negative
databases/cego 2.50.6 -> 2.51.0
- consolidate grammar productions for query conditions and
procedure conditions
- Since CegoPredicate::CompType has received new type EXPRONLY,
this requires a tableset export/import for upgrading to this version
( for predicate encoding / decoding )
- Several changed in CegoFieldValue to switch from Datetime integer
type unsigned long long to long long. This is required to support
datetime comparison for values before 01.01.1970
databases/cegobridge 1.8.1 -> 1.9.0
- adapt to cego 2.51.0
Björn Lemke
Xavier Beaudouin