Synopsis of MonetDB package releases

MonetDB Jun2010-SP2 release

Release "Jun2010-SP2" consists of the following updated components. If you're using the Mercurial clone, use the Jun2010 branch and the Jun2010_5 tag.

Name Version Subdirectory Note/Comment

MonetDB Common 1.38.5 MonetDB

MonetDB Clients 1.38.5 clients

MonetDB4 Server 4.38.5 MonetDB4

MonetDB4/XQuery 0.38.5 pathfinder

MonetDB5 Server 5.20.5 MonetDB5

MonetDB5/SQL 2.38.5 sql

MonetDB5/SQL/GIS 0.18.5 geom

MonetDB Java 1.38.5 java

MonetDB Testing 1.38.5 testing (developers only)

NOTE:
Before upgrading to the latest "Jun2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.

Jun 2010 SP2 bug-fix release
Jun 2010 SP1 bug-fix release
Jun 2010 feature release

Jun 2010 SP2 bug-fix release

This is the second bug fix release of the Jun2010 version. This version can be installed "on top of" the Jun2010 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

Client Packages

mclient now complains about NULL bytes in the input when in interactive mode.
Slight rendering improvements to mclient's tabular output when rendering results larger than the available screen width, headers were previously unnecessarily squeezed.
Fix bug #2650, a too small buffer caused the active database as reported by mclient's welcome message to be truncated
Add --version option to mclient.
In case of an incomplete line from the server, add a newline. This fixes bug 2619.
Avoid using SQLROWSETSIZE and SQLROWOFFSET. This fixes bug 2558.

MonetDB Common

Fixed a bug where some internal reference counting was done incorrectly, causing an infinite loop when exiting the server due to it being killed. This bug probably only manifested itself in MonetDB/XQuery. Bug 2658.

MonetDB Server 4

Fixed a crash when calculating certain aggregates on platforms where non-aligned data access is not allowed (e.g. Sparc).
Slight change to Fedora, Debian, Ubuntu installers: the database directory now has the group setuid bit set so that new databases inherit the group ownership (monetdb).

MonetDB Server 5

Fixed a crash when calculating certain aggregates on platforms where non-aligned data access is not allowed (e.g. Sparc).
A bug was fixed where on 32 bit systems (or 64 bit systems using 32 bit OIDs), values were sometimes written as 32 bits but read as 64 bits. This fixes bugs 2644 and 2654.
If extensions such as SQL and GEOM are properly installed, they are loaded automatically when mserver5 starts. This fixes bug 2522.
Fixed bug 2557. There was a bug in the mergetable optimizer which was triggered by multi column (at least 32 columns).
Added missing multiplex version of MAL str.stringlength(). This improves performance of SQL length().
Protect dataflow against multi-assignments. This fixes bugs 2626 & 2614.
Slight change to Fedora, Debian, Ubuntu installers: the database directory now has the group setuid bit set so that new databases inherit the group ownership (monetdb).
Restored genuine original mitosis logic by disabling incorrect octopus dominance (even when octopus was not enabled); basically a selective back-port of changesets http://dev.monetdb.org/hg/MonetDB/rev/2a358751a4b6 http://dev.monetdb.org/hg/MonetDB/rev/692eff15bea0 from the default branch. This fixes bug 2596.

SQL

Fixed a case where the optimizer incorrectly removed an expression. This fixes bug 2602.
Fix a crash in prepared statements when a parameter is on the left-hand side of a binary operator. This fixes bug 2599.
Fixed reporting of a violated foreign key constraint. This fixes bug 2598.
Certain schema altering queries didn't report success, even though they did succeed. This fixes bug 2589.
Fixed a crash when a non-existing table was used in an IN clause. Fixes bug 2604.
Fixed bug 2633. Adding a LIMIT clause could, in certain conditions, cause a crash.
A bug was fixed where updates were missing in large transaction. This fixes bug 2543.
A memory leak was fixed which caused the server to grow when loading and emptying a table. This was bug 2539.
Fixed bug 2643 added more defensive code, when an aggregation function doesn't exist
Fixed bug 2651 properly handle dead code elimination with op_semi/op_anti and references
Fixed bug 2652. Correctly list all columns of a 'IN' query with 'EXCEPT'
Fixed bug 2353. The relational optimizer didn't handle range join expressions properly.
fixed bug 2354. Improved function resolution.
Fixed bug 2645: mat.pack+algebra.slice should be replaced by mat.slice for 'limit 1' (when the default_pipe is used)
Fixed bug #2641, The SQL server now handles Unicode BOM sequences occurring in any place. Previously an "unexpected character (U+FEFF)" error would be returned.
Fix Bug 2611. Fixed check for multiple functions without parameters.
Fixed bug 2569 (except/union/intersect right after insert/delete/update. We now correctly fallback to the more general subquery case (not only simple selects (SQL_SELECT)).
Fixed bug in handling 'WITH' and row_number() (Bug 2631). The cardinality of the row_number expression was incorrect.
Fixed ORDER BY over UNION etc. (bug 2606) by automatically adding select * around select x union select y order by z.
On installs with (very) long prefixes, the UNIX domain sockets could get truncated, causing merovingian to become unavailable to monetdb and mclient. Similarly, a fall back to a regular TCP socket for mapi connections is used for forked mservers in this case.
Really shutdown when an argument to merovingian was given, instead of ending up in some inconsistent state. This solves all weird behaviour observed in bug #2628.
Report MonetDB release on --version flags and in the logs.
Fixed bug 2624: function returning decimal returned result that was scaled incorrectly.
fixed bug 2622 "LIMIT & OFFSET ignored on 64-bit big-endian when combined with GROUP BY"
Remove references to master/slave settings, since replication isn't yet fully implemented in Jun2010 branch.

The following bug tracker items were fixed since the Jun2010-SP1 release:

2265: SQL: concurrent tracelog()
2319: SQL: got assertion for type-mismatch
2333: SQL: crash on where in query
2341: load_idxcolumn: Assertion `kc->c' failed
2342: bat_storage.mx:822: dup_delta: Assertion `bat->ibid' failed.
2352: SQL: in (test + 1)
2353: SQL: exp_bin: Assertion `0'
2354: SQL: concatination length is based on last column
2529: main BAT fallback creates collision
2548: Connect to default segfault
2557: SQLException:assert:INSERT INTO: FOREIGN KEY constraint violated on 64-bits
2558: unixODBC-2.3.0 removed SQLROWOFFSET and SQLROWSETSIZE
2574: msqldump does not export GRANT information upon export
2576: copy into resulted in HEAPsave_intern: Assertion `0' failed.
2577: not in (... union ...) differs from: except (... union ...)
2589: CREATE TABLE & ALTER TABLE no longer produce "Operation successful"
2591: missing/not-found file: "could not read createdb.sql"
2592: test sql/src/test/BugTracker-2009/Tests/dumping_tables.SF-2776908--optimizers.sql produces different output
2596: some TPCH tests fail with assertion in opt_mergetable on Windows
2597: test sql/src/test/Tests/rank.sql produces varying (wrong?) output on various platforms
2598: Wrong constraint in error message when delete prohibited due to foreign key
2599: server crashes when prepare statement has expression like "a>?+1"
2602: WHERE clause with AND NOT is broken
2604: segfault having non existing table IN (...)
2606: ORDER BY over UNION / EXCEPT / INTERSECT is ignored
2611: creating second function with existing signature succeeds
2613: SQL creates wrong MAL code a.o. with GDK_nr_threads=1
2614: Update statement in function ignored
2619: io.printf without newline hangs
2621: Some *.py tests produce "strange(ly formatted)" stderr output on Windows
2622: LIMIT & OFFSET ignored on 64-bit big-endian when combined with GROUP BY
2624: function returning decimal with fraction returns too large integer
2626: Incorrect NULL function results with multi-thread execution
2627: java/merovingian fails to compile on Debian 4.0
2628: Cannot reconnect to same database twice
2631: Combination of WITH and numbering function crashes Mserver
2633: crash in rel2bin with complex query and LIMIT
2641: Unicode BOM upsets the scanner
2643: sum over a column type date
2644: multi-threaded execution instable on 32-bit systems & 64-bit systems with 32-bit OIDs
2645: SQL: mat.pack+algebra.slice should be replaced by mat.slice for 'limit 1'
2650: Name of database is truncated
2651: rel_bin.mx:1571: rel2bin_project: Assertion `0' failed.
2652: Select * from aliased subquery join doesn't return all columns
2653: segfault sql_mvc.mx:847
2654: extremely slow query
2657: test sql/src/test/Skyserver/Tests/Skyserver_v6.SQL.py: sql/src/common/sql_mem.mx:53: sql_ref_dec: Assertion `r->refcnt > 0' failed.
2658: Mserver busy in BBPExit() for more than 1 minute upon Mserver kill

Jun 2010 SP1 bug-fix release

This is a bug fix release of the Jun2010 version. This version can be installed "on top of" the Jun2010 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Most importantly, the bug that was the immediate reason for retracting the Jun2010 release has been fixed.

Upgrading from the Jun2010 Release

Upgrading directly to the Jun2010-SP1 release from the Feb2010 release or one of the Feb2010 bug fix releases should go smoothly. No problems are known at this time. If, however, you had upgraded to the retracted Jun2010 release, you will need to take some extra steps on each database that the Jun2010 version was used on. Note that this is only relevant for MonetDB/SQL. MonetDB/XQuery was not affected by the bug.

In case still running, stop any MonetDB server.
It is always a good idea to make sure you have an up-to-date filesystem backup (using your favorite filesystem backup software) of your database(s) (i.e., your dbfarm('s)).
Upgrade MonetDB to the Jun2010-SP1 release.

For each database, execute the following steps (once):

Start a Jun2010-SP1 MonetDB/SQL server

Execute the following 4 SQL commands via an mclient session:

update _tables set schema_id=(select id from sys.schemas where name='sys')
    where schema_id not in (select id from sys.schemas) and schema_id <> 0;
update functions set schema_id=(select id from sys.schemas where name='sys')
    where schema_id not in (select id from sys.schemas) and schema_id <> 0;
update types set schema_id=(select id from sys.schemas where name='sys')
    where schema_id not in (select id from sys.schemas) and schema_id <> 0;
update sequences set schema_id=(select id from sys.schemas where name='sys')
    where schema_id not in (select id from sys.schemas) and schema_id <> 0;

Stop the MonetDB/SQL server

Now your databases should be upgraded and consistent, and you can continue working as usual.

In case you did upgrade your Feb2010(|-SP(1|2))-generated SQL databases with the retracted Jun2010 release, and then, after restarting the MonetDB/SQL server, experience problems like missing tables or alike, please follow the same instructions as given above.

In case you tried to upgrade your Feb2010(|-SP(1|2))-generated SQL databases with the retracted Jun2010 release, but that failed with a server crash (segfault), please contact us via one of our public mailing lists or via info@monetdb.org.

Other Changes in Jun2010-SP1

Some of the other bugs that were fixed:

Client Packages

Implemented dumping of GRANT statements. This fixes bug 2574.
Fix implementation of mapi_mapiuri to deal with UNIX socket urls properly. This fixes one part of Bug #2567.

MonetDB Server 5

Fixed bug 2597: This bug manifested itself in rank queries in SQL but was a bug in the mergetable optimizer.
Return a correct URI for local connection from Sabaoth when the connection is a UNIX domain socket. Partial fix for bug #2567.
Various performance fixes in grouping and grouped aggregation code (MonetDB5/src/modules/kernel/group.mx, MonetDB5/src/modules/kernel/aggr*.mx) to reduce the execution time of the following query that mimics a two-column primary key check over the ~5 billion tuple "neighbors" table of the Skyserver database from 26 hours to 1.5 hours (on a 64 GB machine): SELECT count(c), sum(c), min(c), max(c) FROM (SELECT count(*) AS c FROM "neighbors" GROUP BY "objID","NeighborObjID") AS t;

SQL

Make queries like (SELECT ...) UNION ALL (SELECT ...) ORDER BY ...; work, again, that were broken since Jun 22 2010 triggering errors like "ORDER BY: missing select operator"
Removed false connection warning about missing SQL script ("could not read createdb.sql") received by the client upon first connect on a newly created database. Bug #2591
Improved error message for certain type-related problems by including the affected column name.
Make TEXT a separate keyword, separating it from CLOB, such that we can sloppily allow TEXT to appear as a column name, since it seems not to be in the standard as reserved keyword.
Fixed bug 2581. Completed the implementation of handling Boolean types in prepare statements.
Fix bug 2582. Statements with 'constant in ( )' are now handled properly.
Fixed bug 2583 + added test. The assert was incorrect.
Fixed bug in zero_or_one
Fixed bug in dead code elimination for projections with distinct
Fixed bug handling join with constant values on both sides (like group results and constants)
fixed bug in UPDATE TABLE when updating multiple rows
fixed bug 2564: in case group by column is not found as alias in projection list, fall back to check plain input columns in order to find the underlying BAT and check its sortedness

Jun 2010 feature release

The Jun2010 release has been retracted. There is a serious bug in the code that deals with databases that were created with older releases. See the bug report for details. After an upgrade and a restart of the database, the SQL catalog is no longer usable, and so it looks like all tables in the sys schema have disappeared. Note however that not all is lost. The database can be repaired without loss of data once we have issued the Jun2010-SP1 bug fix release. Also, as long as you don't make any schema changes (i.e., no new tables, functions, etc.) you can repair the database using the following queries and a restart of the server:

select id from schemas where name = 'sys';

Note the result of this query and use it in the next queries in place of RESULT:

update _tables set schema_id = RESULT where schema_id not in
       (select id from schemas) and schema_id <> 0;
update functions set schema_id = RESULT where schema_id not in
       (select id from schemas) and schema_id <> 0;
update types set schema_id = RESULT where schema_id not in
       (select id from schemas) and schema_id <> 0;
update sequences set schema_id = RESULT where schema_id not in
       (select id from schemas) and schema_id <> 0;

Packaging

Linux

We now package for Fedora 11, 12, and 13, for Debian 5.0 (Lenny), and for Ubuntu 9.10 (Karmic Koala) and 10.04 (Lucid Lynx).

MonetDB Common

Updated Vendor information.
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

MonetDB Server 4

Updated Vendor information.
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

MonetDB Server 5

When libxml2 is available, the XML module is automatically loaded.
Updated Vendor information.
Created a new RPM MonetDB5-server-rdf for the optional MonetDB/RDF module.
Added the Z-order module to simplify future manipulation of arrays.
fixed BUG #2994521 "mat.slice unable to cope with only empty BAT arguments" by making MATpackSliceInternal() handle empty input BATs correctly
Select <col> from <t> limit <n> has been improved by introducing mat.slice().
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
The MAL debugger list command has been extended with an optional hash '#', which produces line numbers for each of reference and analysis of variable span.
The dataflow scheduler has been revamped to allow for more parallelism to be exploited.
The garbage collection administration has been changed. Every variable record now comes with an end-of-life field (eolife), which denotes the instruction after which the BAT variable reference counter can be decremented. The garbage collector is never called automagically on MAL functions, because there may be other pressing needs to retain them. This means that MAL functions defined and used in the context of SQL, and which are not inlined, may cause a leakage. The garbage collection has become part of each interpreter step. The new approach makes the SQL/MAL plans half the size as before.
Removed stethoscope from MonetDB5 sources. New location is in the clients repository.
Added the compression optimizer as an example of how to gain access to foreign file formats deep down in the kernel and transfer them just in time into a temporary BAT.
Renamed configure argument --with-console to --enable-console. Default remains console being enabled.
Fixed cleaning the user module context upon session end. Fixes bug #2956664
The config variable mapi_usock can now be used to instruct the server to listen for connections on a local UNIX domain socket on UNIX-like systems.
The MAL interpreter has been extended with an operation admission policy to control the memory claims of all concurrent running interpreters. Instructions are hold up unto there is sufficient resource or the query plan can not avoid its execution anymore.
The join path optimizer has been extended with searching for join*, semijoin*, and leftjoin* paths. Furthermore, it avoids duplicate work by factoring out all common simple join paths.

SQL

make it possible to use '?' in offset and limit
Added include and .lib files to Windows installers that are needed to compile client programs.
Updated Vendor information.
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

XQuery

Updated Vendor information.
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

Merovingian

Fixed an issue where Merovingian's strict umask was applied too early causing files to be inaccessible. Now the .merovingian_control socket file is accessible to both user and group of the merovingian process, and the .mapi_socket file is accessible to everyone like a TCP socket is. Fixes bug #2550.
Properly clean up .merovingian_control file in dbfarm.
Removed database pre-starting (mis-)feature, any argument to merovingian now yields in a version message followed by a server exit.
Sort output returned by discover, status and get commands based on url or dbname.
Implemented UNIX domain socket listener for Merovingian, found as mapi_socket in the dbfarm. Client connections can be made over it like normal TCP connections.
When a proxy connection is made to a local UNIX domain socket, pass on the filedescriptor instead of proxying.
Use mserver5's UNIX domain socket support when Merovingian is supposed to proxy the connection to the database.
Enlarged buffer to avoid incomplete output being sent to monetdb(1), causing commands like status and get to fail on certain configurations.
Avoid running over the allocated buffer space resulting in crashes when a client connects which IP address cannot be resolved into a hostname, bug #2964247.

Client Packages

Updated Vendor information.
The MonetDB ODBC driver now no longer depends on its own copy of the standard ODBC include files but instead depends on system include files (unixODBC on Linux and Microsoft SDK on Windows).
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
Changed \d output of mclient to no longer list (internal use only) system tables. Administrator users who like to inspect these tables can use SELECT * FROM tables; instead
Implemented the changes required for properly running ODBC on a 64-bit platform.
Added stethoscope, an utility to profile MonetDB5 instances, originally from the MonetDB5 repository.

Geom Module

Updated Vendor information.
Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

MonetDB Jun2010-SP2 release

Contents

Jun 2010 SP2 bug-fix release

Client Packages

MonetDB Common

MonetDB Server 4

MonetDB Server 5

SQL

Jun 2010 SP1 bug-fix release

Upgrading from the Jun2010 Release

Other Changes in Jun2010-SP1

Client Packages

MonetDB Server 5

SQL

Jun 2010 feature release

Packaging

Linux

MonetDB Common

MonetDB Server 4

MonetDB Server 5

SQL

XQuery

Merovingian

Client Packages

Geom Module