- Some improved exception handling in CegoTableManager::checkIntegrity
( more detail information about violated object )
- Code cleanup CegoTableManager, drop/create/alter operations inside
a transaction no more allowed
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Further recovery fix in CegoTableManager::insertDataTable In case
of an insert exception ( e.g. duplicate pkey ), a checkpoint is
forced now after deleteData to ensure file consistency
In CegoObjectManager::insertPageData a second checkpoint is forced
after data copy, if a previous checkpoint has occured. Otherwise
this insert might be ignored during recovery because of lower LSN
- Fix in CegoBTreeManager::checkDuplicate, the check for uflag (
returned CegoBTreeNode::valueExists ) to check for high value is
not enough if, entries have been deleted from node.
Instead, all subsequent pages have to be checked until a value higher is found
- Introduced CegoBufferPool::getCPCount() to return to current
checkpoint count. This method is used by CegoTableManager,
CegoObjectManager and CegoTransactionManager to decide, if a
consitency checkpoint is required
- Version passed checkCrashRecovery with the following parameters
NUMRUN=100
DOTRANSACTION=-dotransaction
ARANGE=10000
AINTERVAL=500
ACOUNT=30000
PRIMARY=primary
POOLSIZE=191
All relevant checkpoint situations have been observed over all run
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix added in CegoTransactionManager::commitUpdate, btree must be
inserted with insertBTreeWithCommit
- Fix added in CegoTransactionManager::getCrashAffectedTables,
schema entry must be searched with Find(CegoField(Chain(),
SYS_RB_TABLE))
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoRecovery:::recoverCurrentTransactionLog, for LOGREC_INSERT
and LOGREC_DELETE, the corresponding index have to be invalidated
with CegoTableManager::invalidateIndexForTable. This logic was
already implemented but removed in context of more sophisticated
transaction handling ( patch 2.39.11 and around )
- Small fix in CegoObjectCursor constructor, changed _isEOC = false
to _isEOC = ( pageId == 0 ). This causes getFirst to return false
in case of truncated objects ( can occur for RBSEG objects )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoAction::readChain() to support "\r" escape character
Fix in CegoBTreeCursor::getFirst, in case of pAttrCond = 0, getNext
has to be called if current leaf page is empty, otherwise cursor
trace is incomplete ( just used for CegoDistManager::verifyTable )
- Fix in CegoBTreeNode::getChildPage, parameter traceMin has been
introduced, which is needed to trace btree with
CegoBTreeManager::deleteBTree.
Otherwise, it might happen that non-unique btree objects are not found
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Removed legacy method CegoAggregation::setInit/isInit
- Fix in CegoQueryHelper::aggregateTuple, since the field value
reference for aggregation field value might be no more valid during
the aggregation run, a local copy is created via
CegoFieldValue::getLocalCopy.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added method CegoClient::treateEscape to handle escacpe charaters
in command strings for interactive and batch mode ( newline, carriage
return und tab )
- Further optimization in CegoFunction::getReturnTypeLen, if the
length argument in left, right substr function is a constant, this
value is used for schema definition, otherwise the length of the
source is used. Example:
select left(a,3) from t1 => length 3 is used for schema
select left(a,b) from t1 => definition of a is used for schema
Further optimization done in CegoSelect and CegoAction for schema
evaluation.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- In CegoSelect::evalSelection, optimization provided in a way,
that schema is not evaluated explizit anymore. Rather the previous
analysed schema ( prepare method, _evalSchema ) is used. Since for
dynamic field length, the resulting schema must be estiamated, this
approach is ok.
- Fix added in CegoFunction::getReturnTypeLen, evalFieldValue for
expression values could fail, since FLA ist still not set up. For
this case, now the cont value RETVAL_LEN is returned as an estimation.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added fix in CegoAttrComp::setup, _posSetup variable was not setup
correctly for case BTWN / VALUE2VALUE which might lead to invalid
join array access.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added fix in CegoExpr::evalField and CegoTerm::evalField to provide
correct result type. This is required for fastserial protocol usage.
- Also added fix in CegoFieldValue::operators +-*/ to cast to the
largest value
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoTableManager::deleteDataTable, before resolving external
table references via evalExtTableReferences, the subquery must be
prepared, to resolve local attributes first
- Optimization added in CegoBufferPool::bufferFix. If there is no
available slot in the bufferpool, a checkpoint is forced for the
corresponding tableset ( already implemented ). Now if still no
available slot, we force checkpoint for all other online tableset
of the database. This might be useful, if the bufferpool is heavily
occupied by other tablesets with dirty pages.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
devel/lfcxml: update 1.2.6 -> 1.2.10
databases/cego: update 2.39.16 -> 2.44.1
databases/cegobridge: update 1.4.0 -> 1.5.0
databases/p5-DBD-cego: update 14.0 -> 1.5.0
- Warning: storage format has changed
Export to xml format before upgrade and re-import after the upgrade
See UPDATING
- recompile all applications linked to libcego
- Lots of changes, among them:
o improved crash recovery
o fixes to SQL expected behaviour
o better CDATA handling
o fixes primary key handling design issue
o changes to serialisation for export/import, XML export/import is still
possible
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
Changelog: http://www.lemke-it.com/litexec?request=pubnews&user=&lang=en&numlog=1000
databases/cego: update 2.39.15 -> 2.39.16
lfcbase:
- Introduced new method File::flush to force synchronization of data to disk
cego:
- Fix in CegoObjectManager::insertPageData, new data entry is checked
now for maximum available space in page. This is done if a new
data page has to be allocated and the data entry still fits not
into page
- Added File::flush method to CegoLogManager and CegoFileHandler
to synchronize log data and datafile data to disk
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in dbcheck/expimpcheck, test plan was not set up correctly
In CegoXPorter and CegoExpImpStream, row export information was
added again to indicate overall row export for each table
- Fix added in CegoDbThread::serveRequest ( GETBLOB and GETCLOB case ) and
CegoTableManager::getBlobData / getClobData chunkSize calculation. The
existing calculation did not treat every boundary case.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoXPorter::writeRow and CegoXPorter::readRow for binary
export and import of tables containing lob null values. lob null
values must be identified by colLen unsigned long long = 0 instead
of integer = 0
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Introduced CegoTransactionManager::getCrashAffectedTables method.
In case of a crash recovery, all transaction affected tables have
to be analysed for index objects. Before finishing transactions,
the index objects have to be invalidated ( since they might be
corrupted ) and after completion of transactions they have to be
recreated. This is done now in CegoTableManager::finishOpenTransaction
- Fixes in CegoLogManager, CegoRecoveryManager, CegoDistManager and
CegoTableManager for LSN handling. Instead of saving the next LSN,
now the current written LSN is recorded in CegoLogManager. This
seems to be more natural, since a lot of methods set up the LSN to
a value + 1. Also this fixes a mismatch between commited lsn and
current lsn, which leads to a tableset recovery during startup in
any case ( occured with version 2.39.9 )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Patch in CegoClient to catch format exceptions like
"Invalid datetime string <0> for format <%d.%m.%Y %H:%M:%S>"
This may happen, if on server side bad union selects are defined like
create view v1 as
select a as a, b as b from t1
union all
select a as a, 0 as b from t2;
where b is a datetime data type. In this case, invalid formatted
strings occur on the client side, which are catched now.
- Patch in CegoDatabaseManager::useObject to support lock
delay values > 1 sec. The DBM_LOCKDELAY value in CegoDefs has now
increased to 2500 msec. In some cases for heavy updates
( e.g. clob updates ) this might be useful to avoid ugly exclusive
lock delay messages in database log file
- Fix in CegoTableManager::updateTuple. If the update fails ( this
might happen, if another transaction actually updates the corresponding
tuple ), any allocated lob values are freed now. Otherwise the
data pages for the lob values were allocated with no more references.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Improvements for log handling, the logIt method was moved from
CegoTableManager to CegoBufferPool. This allows a more adequate
locking of the logging actions. For this the tsLock array has been
removed from CegoTableManager and lmLock array has been set up in
CegoBufferPool. Since checkpoint writing occurs in CegoBufferPool
( which is also relevant for loging operations ), the logging method
is now part of CegoBufferPool.
- The logAction method in CegoLogManager has also been improved in
a way, that a log buffer is not allocated for each logging operation
anymore. Instead, a logBuf array has been defined for all tablesets
and the logBuf is reallocated, if the current logging record exceeds
the log buffer.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added fixes for recovery procedure in CegoTableManager. Now the
LSN is allocated at the beginning of critical operation to avoid
double operations in case of a forced checkpoint. For this the
CegoLogManager::nextLSN method has been introduced. Now it is no
more ensured that the log is written with increasing LSN, so some
logical parts of the recovery procedure in CegoRecoveryManager has
been changed.
- In CegoBufferPool, the writeCheckPoint method still returned int
value. This has been changed to unsigned long long
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: 2.39.7 -> 2.39.8
lfcbase:
- Fix for memory leak in TreeT and AVLTreeT.
The = operator implementation did not empty the tree before copy.
- Additional fix to this release : Added regfree call to Matcher
destructor. Missing this call seems to cause a small memory leak
cego:
- Fix for memory leak in CegoPredDesc, pMatcher has not been cleaned up
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoTableManager::deleteDataTable, pBlock has to be checked
for null value, otherwise crash recovery might crash. ( bug came
with 2.39.0 to check for pBlock->getTriggerValueList )
- Some clob related fixes in CegoQueryHelper::decodeNativeFVL /
decodeFVL relevant for recovery handling
- Fix in CegoRecoveryManager::recoverCurrentTransactionLog for the
INSERT case. For the blob/clob handling, the lobREf values have not
been set up correctly. This might lead to problems for table recovery
with more than one lob row ( e.g. table t1 ( a blob , b clob )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Bug fix in CegoAction::reallocateStringBuf, in the memcpy call,
just the previous stringBufLen has to be copied ( _stringBufLen -
MAXSTRINGLEN ). This bug might lead to seg fault in case of large
parser input ( e.g. clob strings )
- In CegoQuery for insert and insert-by-select case, allocated blob
and clob data ( by CegoQueryHelper::prepareFieldValue ) is cleaned
now if any exception occurs
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Small bug fix in CegoDistManager, which came up with version
2.39.0. In method deleteLocalDataTable, the variable delCount was
declared two times, which lead to a result of 0 tuples deleted in
any case
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Small improvement in CegoDbThreadPool. For a better db thread
load value calculation, the request queue is no longer locked while
waiting for incoming requests. Instead the QUEUEDELAY parameter has been
increased to a default value of 200 msec to reduce db thread cpu
load while idle. Now the db thread has a good chance to report
idle time to the db thread pool.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cegobridge: update 1.3.0 -> 1.4.0
cego:
- Fix in CegoTableManager::updateTuple, while setting up expression
list, field list array must be setup BEFORE block ist set, since
field list is needed by block setup ( in case of subqueries for
prepare )
cegobridge:
- Adaptions for cego-2.39
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Introduced table locking statements to set an explicit lock on a
table ( e.g. for update synchronisation ). This feature replaces
the "set update sync on/off" command
- Trigger implementation basically completed
- Fix in CegoQueryHelper::encodeFVL, for blob/clob values lists,
the corresponding index variables ( blobidx/clobidx ) have not been
increaed while encoding lob data. This might lead to invalid results
and seg faults in case of multiple lobs values in one insert/update
operation.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- For heavy update operations on tables with btrees using embedding
transactions, duplicate btree errors may occur. This is caused
by not checking corresponding tuple states. To solve this problem,
the following fixes have been done :
o Fix in CegoTableManager::checkBTreeIntegrity : The tuple state for
found entries has to be checked if state = COMMITTED
o Fix in CegoBTreeNode:checkForAffected ( was checkForDeleted before ) :
The tuple state has already checked for state
INSERTED if tid != dataTid
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in dbcheck/check065.sql for union sql in view.There must be
set up aliases for attributes in selection now
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoTableManager::createForeignKey, for empty tables,
referenced attributes names have not been checked. This might lead
to invalid key objects.
- Fix in CegoSelect::prepare, expression alias in select list are
checked now for union selects. Alias definition in select expression
list should be identical for all select statements in union
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Increased NETMNG_QUEUE_DELAY to 1000000 ( 1 ms ) to minimize CPU load
- Introduced QUEUEDELAY ( in usec ) config parameter to adjust queue
delay in database xml file. This allows to throttle down database
response time for incoming connection requests and minimize CPU
load
- Added dbcheck/check072.sql. This implements the calculation of
fibonacci numbers using stored procedures as a demonstration for
massive recursive procedure calls
- Optimization for procedure load in CegoFunction::evalFieldValue
( case USERDEFINED ), procedure is just loaded as a dedicated
instance ( via loadProcedure method ), if pMasterBlock is set. This
avoids ( expensive ) dedicated load, if the procedure is still not
in use by the corresponding thread. So just for recursive procedure
calls, the dedicated load happens.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.38.12 -> 2.38.14
lfcbase:
- Stability patch for strptime implementation ( MinGW only )
cego:
- Fixed memory lead in CegoDistManager destructor ( _pPA was not deleted )
- Further optimization done in CegoBTreeCursor, introduced methods
traceLow and inRange to treat btree comparison in a more efficient way
- Fix in CegoAttrCond::getIndexCond, like and no like comparisons
must be filtered out for new btree cursor tracing logic
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Small optimization in CegoAttrCond::checkIndex, removed primary
flag, since this is done now by attribute ordering. This leads
to full index trace match instead of trace with index support and
so, additional unnecessary predicate evaluation is avoided.
- More rework for btree evaluation in CegoBTreeCursor
- Optimization in CegoBTreeCursor::getNext. Instead of calling
getNext in a recursive way, this is now done in a iterative way.
This might avoid heavy stacking in case of special query constraints
( e..g multi dimensional btrees and range conditions like a > 400
and b betweeen 'AAA' and 'DDD' )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.38.9 -> 2.38.10
lfcbase:
- Fixes in TreeT class, changed methods First, Next, isEmpty and
compare operators to const
cego:
- Fix in CegoDistManager, we have to use a dedicated parser for
loadView and loadProc methods. Otherwise compilation may fail in
case of nested objects ( e.g. a view which references another view
object)
- Fix in CegoBTreeCursor, CegoAttrCond and CegoAttrComp, for
conflicting cursor conditions ( e.g. a = 4 and a < 1 ), the cursor
evaluation failed, since just the primary condition was treated.
- For CegoBTreeCursor, the method fullMatch has been introduced,
to make an final evalution in case of multi conditions.
- Also introduced: a ordering for CegoAttrComp set in
CegoAttrCond to treat more constraint conditions at the beginning
( EQUAL before LESS_THAN before NOT_EQUAL and so on )
- Changed CegoAttrCond compSet from SetT to TreeT.
This is required, since we need the ordering in the set.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Introduction of new functions ascii ( get ascii character for
numeric value ) , ldiv and lmod ( div and mod operation for long
values )
- Modification in CegoFunction::evalFieldValue. To allow recursive
calls of user defined functions and procedures, dedicated procedure
instances are created for each call. This is done by parsing the
procedure text via CegoAction
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.38.6 -> 2.38.7
lfcbase:
- Improvements made for dedicated strptime implementation. The
function did not return null in case of unparsable date values.
This code is just used for MinGW compiles since there is no
implementation in the Standard C library available.
cego:
- Fix in CegoXPorter::readTableObject, the defintion for MAXROW BUF
still was defined locally and rowLen was noch checked.
- Optimization in CegoFieldValue::decode, memory was allocated via
malloc, rather _staticBuf should be used for performance reasons.
_isLocalCopy was not set to true
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.38.5 -> 2.38.6
lfcbase:
- More detailed exception messages for BigDecimal and BigInteger
cego:
- Fix in CegoFunction::getId and CegoSelect::getQueryId : with
enabled query cache, the following select is cached
select nextcount(mycount);
- Since nextcount is a modifying operation, this is not a good idea.
To avoid caching, CegoFunction now throws an Exception if
nextcount / setcount function calls are detected during getQueryId.
For these cases, the query id ignored for caching
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoBTreeValue::valueFromSchema, introduced method
getReservedLength to return fixed value length for type fixed and
decimal. Since for decimal and fixed type just the dimension is
stored in CegoField::getLength, we must define a reserved area
for those types. This is done now with constant definition
RESERVED_BTREE_FLOATLEN in CegoDefs.h
- Improved ordersize calculation in CegoOrderSpace:insertTuple. Now
the following formula is used :
int s = sizeof(fv);
if ( fv.getLength() > STATICFIELDBUF )
s += fv.getLength();
orderEntryLen += s;
The catches the base memory usage for the instance but also the
dynamic allocated part
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoXMLSpace::setTSSortAreaSize, changed type from int to
unsigned long long, otherwise overflow may occur for large sort
area size configurations
- Performance optimization done in CegoOrderSpace. Instead of storing
the CegoField list for the result tuple in CegoOrderNode, just the
values are stored there. The order cursor now gets the schema with
the constructor and build the CegoField List in the getFirst /
getNext methods. This seems a huge amount of heap space, which is
dynamically allocated during the sorting procedure ( sortareasize
parameter ).
Especially for large ordering result sets this saves significantly
memory and improves performance.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoPredDesc::decode, loop variable i for IN/NOTIN case
was not initialized
- Fix in CegoAction::execCheckCreate, check contraints are checked
for contained subselects. subselects are ( still ) not supported
since the CegoCheckOject::decode method can not provide a valid
tablemanager instance ( it is called from CegoObjectManager )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cegobridge: update 1.2.1 -> 1.3.0
cego:
- Fix in CegoSelect::clone, pUnionSelect was not cloned, which might
lead to seg faults
- Introduced predicate clause for select .. in ( expr, expr, ... ),
e.g. select * from t1 where a in ( 1, 2, 3 ); This was still not
implemented but is part of standard SQL.
cegobridge:
- Adaptions made for modified cego-2.38.0 API with extended
CegoDatabaseFormatter::formatPred method ( exprList was added for
new introduced select in ( expr list ) predicate )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoClient, the new comment logic by using Tokenizer does
not work ( '-' characters are detected as comment tokens ). Code
replaced by using posStr chain function
- Adaption in CegoMain to also parse comment lines in a more
sophisticated way ( same as in CegoMain ), just relevant for server
batch mode
- Optimization in CegoDistCuror::joinSetup, the join predicate is
now analysed in a way, that a condition list is created ( via makeCNF
and createCondition methods in CegoQueryHelper ). The condition
list then is analysed to achieve a better cursor condition which
results in improved index usage
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.36.4 -> 2.37.0
databases/p5-DBD-cego: update 1.3.0 -> 1.4.0
databases/cegobridge: update 1.1.6 -> 1.2.0
lfcbase:
- Layout improvements in Pager class
- Changed constructor for Net and NetHandler class to setup maxSendLen.
Since this changes the API, minor release level has been increased
cego:
- Improvements for cgblow simulation mode added
- Improvements for cgclt, added pipe mode to read input from stdin
Now comments are allowed also after delimiter token, e.g.
create table t1(a int); -- a sample table
- Support for lfcbase-1.11.x with new Net API.
The parameter maxSendLen ist still a constant in CegoDefs.h
Improvements added for CegoXMLSpace::setPerm, for existing permissions,
just tableset, filter or right can be set up now,
e.g set permission p1 with right=WRITE for role1 just sets the right
value for permission p1 to value WRITE, all other values are unchanged
- Added show parameter admin command to show all tableset independent
database parameters
p5-DBD-cego:
- support changed API with lfcbase-1.11.0
cegobridge:
- support changed API with lfcbase-1.11.0
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: update 2.36.3 -> 2.36.4
databases/cegobridge: update 1.1.5 -> 1.1.6
cego:
- Fix in CegoSelect::nextTuple, before adding an entry to the query
cache, it has to be checked, if any tables from foreign tableset
are referenced in the query. If so, no cache entry is made, since
table changes for foreign tables are not detected.
- Improved error messages for invalid database objects
- Decreased shutdown delay time by reducing net delays for all
thread pools and optimized mediator thread wait procedure
- Completely removed the nologging option for import actions.
Logging is disabled in any case now.
- Optimization added for log handling. During ( xml ) import,
logging is completely deactivated, after import is completed,
log is started again and a checkpoint is written.
Since import can be repeated in case of a system during import,
nothing gets lost. Deactivating logging increases import speed and
avoids side effects
cegobridge:
- Speed up imports by changing the way the parser reads the input
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Changed recovery strategy for existing btree / avl index objects.
Before creating an index object, it is checked if exists. If exists,
it is dropped and recreated. This seems to be a more stable strategy,
since it may occur, that index objects still exist for several
reasons.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoFactor::setFieldListArray, the _flaCached variable has
to be set to false, if _pFLA is set to a different value. Otherwise,
we might refer to invalid memory, which results in core dump
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added signal handler to CegoAdmAction. Now, ( long ) running
tableset export and import requests can be aborted in a controlled
way via Ctrl-C command.
- Fix in CegoXPorter, export file will be removed, if any exception
occurs. So it is ensured, that the written export file is consistent
and complete
- fixes in CegoXPorter for CLOB handling ( was not checked for plain exports )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
