Malformed string firebird error

asked Nov 27, 2012 at 17:22

Malformed string error can come from data and you can correct this with switch -fix_fss_d but it can also come from metadata and you have to use the switch -fix_fss_m

-FIX_FSS_D[ATA]

This switch forces gbak to fix malformed UNICODE_FSS character data
during a restore.

This switch, and the following one, should not be required under
normal circumstances. However, if a restore operation fails with a
«malformed string» error, the message output from gbak will refer the
user to one or both of these switches to fix the malformed UNICODE_FSS
data or meta data as appropriate.

-FIX_FSS_M[ETADATA]

This switch forces gbak to fix malformed UNICODE_FSS metadata during a
restore.

This switch, and the preceding one, should not be required under
normal circumstances. However, if a restore operation fails with a
«malformed string» error, the message output from gbak will refer the
user to one or both of these switches to fix the malformed UNICODE_FSS
data or meta data as appropriate.

link to firebird gbak utility

RBARBA

12.2k16 gold badges77 silver badges125 bronze badges

-?

This switch displays the command line options and switches.
It replaces the old method in which you had to supply an invalid switch (see -help below) in order to see the list of valid ones.

-FE[TCH_PASSWORD] <password file name> | stdin | /dev/tty

This switch causes the password for the appropriate user to be read from a file as opposed to being specified on the command line.
The file name supplied is not in quotes and must be readable by the user running gbak.
If the file name is specified as stdin, then the user will be prompted for a password.
On POSIX systems, the file name /dev/tty will also result in a prompt for the password.

-M[ETA_DATA]

This switch causes your data to be ignored and not backed up or restored.
In a backup, only the database meta data are backed up.
When used in a restore, any data in the dump file will not be restored.
This switch can be used when creating database clones which are required to contain only the tables, indices etc but none of the data.

-PAS[SWORD] <password>

Supplies the password for the username specified above.
This need not be supplied if ISC_PASSWORD environment variable exists and has the correct value.

-RO[LE] <SQL role name>

Allows the specification of the role to be used by the connecting user.
Not of much practical use and is not normally used in practice.

-U[SER] <username>

Allows the username of the SYSDBA or database owner user to be specified if the database is to be backed up, or, in the case of a restore (with the -c[reate] switch specified), any valid username can be specified.
This need not be supplied if ISC_USER environment variable exists and has a correct value for the username.

Databases can only be backed up by SYSDBA or the database owner.
A restore can also be carried out by SYSDBA or the database owner, however, if the -c(reate) switch is used, any validated username can create a new database from a backup file.

-V[ERIFY]

Normally gbak operates quietly with no information written to the display.
This switch reverses that situation and causes lots of information to be displayed.
The default is to display the output to the screen, but you can redirect the output to a log file using the -y switch.

-Y <filename> or the text "suppress"

Used in conjunction with the -v[erify] switch to redirect status messages to a file or device, rather than the screen, or to suppress them altogether.

If -y suppress is used, then no information will be written to screen regardless of whether -v[erify] is specified.

If a filename is given and the -v[erify] switch is specified, the file will be written to record progress, errors etc.

All known documentation on this switch mentions that the text should be "suppress_output" rather than "suppress". This is an error in the documentation as the source code for gbak shows that the switch must be "suppress".

The log file must not exist. If it does, the backup or recovery operation will fail: tux> rm employee.log tux> gbak -backup employee.fdb employee.fbk -y employee.log -v tux> ls -l employee.log -rw-r--r-- 1 firebird firebird 21610 2010-08-04 10:22 employee.log tux> gbak -backup employee.fdb employee.fbk -y employee.log -v gbak:cannot open status and error output file employee.log

-Z

This option displays some information about the version of gbak being used, and optionally, a database.
To obtain the version of gbak only, run the command as follows:

tux> gbak -z

gbak:gbak version LI-V2.5.0.26074 Firebird 2.5
gbak: ERROR:requires both input and output filenames
gbak:Exiting before completion due to errors

tux> echo $?
1

This displays the current version of gbak, and after displaying a couple of error messages, exits with an error code of 1 as shown above by the echo command.
This method does not attempt to backup any databases and does not require a username and password to be defined or supplied.

If you wish to display the version number of the gbak utility and also details of database, you must supply a valid database name and backup filename, as follows:

tux> gbak -z employee employee.fbk -user sysdba -password secret

gbak:gbak version LI-V2.1.3.18185 Firebird 2.1
gbak:    Version(s) for database employee
Firebird/linux (access method),version LI-V2.1.3.18185 Firebird 2.1
Firebird/linux (remote server),version LI-V2.1.3.18185
Firebird 2.1/tcp (tux)/P11
Firebird/linux (remote interface), version LI-V2.1.3.18185
Firebird 2.1/tcp (tux)/P11
on disk structure version 11.1

tux> echo $?
0

You will note from the above that a valid username and password must be defined on the command line, or by the use of environment variables ISC_USER and ISC_PASSWORD.
This version of the command will exit with a error code of zero.

This method of calling gbak will make a backup of the database. If your database is large, this can take some time to complete and the backup file specified will be overwritten if it already exists. Beware.

The output above has been slightly abbreviated to allow it to fit the page width for a pdf.

-help

Help is actually not a valid option, but can be used to display the following screen of information as output from gbak in Firebird 2.0:

gbak:legal switches are:
        -B[ACKUP_DATABASE]    backup database to file
        -BU[FFERS]            override page buffers default
        -C[REATE_DATABASE]    create database from backup file
        -CO[NVERT]            backup external files as tables
        -E[XPAND]             no data compression
        -FA[CTOR]             blocking factor
        -G[ARBAGE_COLLECT)    inhibit garbage collection
        -I[NACTIVE]           deactivate indexes during restore
        -IG[NORE]             ignore bad checksums
        -K[ILL]               restore without creating shadows
        -L[IMBO]              ignore transactions in limbo
        -M[ETA_DATA]          backup metadata only
        -MO[DE] <access>      "read_only" or "read_write" access
        -N[O_VALIDITY]        do not restore database validity conditions
        -NOD[BTRIGGERS]       do not run database triggers
        -NT                   Non-Transportable backup file format
        -O[NE_AT_A_TIME]      restore one table at a time
        -OL[D_DESCRIPTIONS]   save old style metadata descriptions
        -P[AGE_SIZE]          override default page size
        -PAS[SWORD]           Firebird password
        -R[ECREATE_DATABASE]  [O[VERWRITE]] create (replace if O[VERWRITE] used)
                                database from backup file
        -REP[LACE_DATABASE]   replace database from backup file
        -RO[LE]               Firebird SQL role
        -SE[RVICE]            use services manager
        -T[RANSPORTABLE]      transportable backup -- data in XDR format
        -USE_[ALL_SPACE]      do not reserve space for record versions
        -USER                 Firebird user name
        -V[ERIFY]             report each action taken
        -Y  <path>            redirect/suppress status message output
        -Z                    print version number

The explanation of the -m[eta_data] switch implies that it is useful in a backup situation only. This is not the case as it can also be used on a restore.

From Firebird 2.5 onwards, there is a new -? switch to display the list of valid options.
The output has slightly different layout and a couple of new options have been added:

gbak:legal switches are:
        -B(ACKUP_DATABASE)    backup database to file
        -C(REATE_DATABASE)    create database from backup file (restore)
        -R(ECREATE_DATABASE) [O(VERWRITE)] create (or replace if OVERWRITE used)
                                database from backup file (restore)
        -REP(LACE_DATABASE)   replace database from backup file (restore)

gbak:backup options are:
        -CO(NVERT)            backup external files as tables
        -E(XPAND)             no data compression
        -FA(CTOR)             blocking factor
        -G(ARBAGE_COLLECT)    inhibit garbage collection
        -IG(NORE)             ignore bad checksums
        -L(IMBO)              ignore transactions in limbo
        -NOD(BTRIGGERS)       do not run database triggers
        -NT                   Non-Transportable backup file format
        -OL(D_DESCRIPTIONS)   save old style metadata descriptions
        -T(RANSPORTABLE)      transportable backup -- data in XDR format

gbak:restore options are:
        -BU(FFERS)            override page buffers default
        -FIX_FSS_D(ATA)       fix malformed UNICODE_FSS data
        -FIX_FSS_M(ETADATA)   fix malformed UNICODE_FSS metadata
        -I(NACTIVE)           deactivate indexes during restore
        -K(ILL)               restore without creating shadows
        -MO(DE) <access>      "read_only" or "read_write" access
        -N(O_VALIDITY)        do not restore database validity conditions
        -O(NE_AT_A_TIME)      restore one table at a time
        -P(AGE_SIZE)          override default page size
        -USE_(ALL_SPACE)      do not reserve space for record versions

gbak:general options are:
        -FE(TCH_PASSWORD)     fetch password from file
        -M(ETA_DATA)          backup or restore metadata only
        -PAS(SWORD)           Firebird password
        -RO(LE)               Firebird SQL role
        -SE(RVICE)            use services manager
        -USER                 Firebird user name
        -V(ERIFY)             report each action taken
        -Y  <path>            redirect/suppress status message output
        -Z                    print version number

The parentheses shown in the above indicates how much of the switch name you need to use in order to avoid ambiguity.
Once you have specified the absolute minimum — the part before the opening ‘[‘ — you can use as much of what follows as you wish.
For example, to use the -b[ackup_database] switch the minimum you must supply on the command line is -b but anything between -b and -backup_database will be accepted.

Using the -help switch like this, or any other invalid switch, will cause gbak to exit with an error code of 1 on Linux and Windows.

-B[ACKUP_DATABASE]

This switch is used whenever you wish to take a backup of a database.

-CO[NVERT]

This switch causes any tables, defined as external, to be backed up as if they were normal (non-external) tables.
When this dump file is used to restore a database, the tables that were external in the original database will no longer be external.

-E[XPAND]

Normally, gbak will compress the output file.
This switch prevents that compression from taking place.

-FA[CTOR] <block size>

If backing up to a physical tape device, this switch lets you specify the tape’s blocking factor.

-G[ARBAGE_COLLECT]

The use of this switch prevents Firebird’s garbage collection from taking place while gbak is running.
Normally gbak connects to the database as any other connection would and garbage collection runs normally.
Using this switch prevents garbage collection from running during the course of the backup.
This can help speed up the backup.

-IG[NORE]

This switch causes gbak to ignore bad checksums in the database.
This can be used to attempt to backup a database that failed due to checksum errors.
There is no guarantee that the data will be usable though, so it is best to take other precautions to preserve your data.

-L[IMBO]

If you have a two-phase transaction (across two different databases) that failed because a server died before the commit, but after the changes were made, you have a limbo transaction.
This switch forces the backup to ignore data from such broken transactions.
It should not be used for normal backups and only used, like the -ig[nore] switch to attempt to recover from a failure.

-M[ETA_DATA]

See above.

-NT

This switch turns off the -t[ransportable] switch (which is on by default) and causes the dump file to be created using platform dependent formats.
If you use this switch to create a backup then you can only restore the backup on a similar platform.
You cannot, for example, take a dump file from Linux over to a Windows server.

-OL[D_DESCRIPTIONS]

This switch is unlikely to be used.
It has been deprecated.
Its purpose is to force the backup to be made using the older InterBase versions’ format of meta data descriptions.

-PAS[SWORD] <password>

See above.

-RO[LE] <role name>

See above.

-SE[RVICE] <servicename>

This switch causes gbak to backup a remote database via the service manager.
This causes the backup file to be created on the remote server, so the path format and filename must be valid on the remote server.
The servicename is currently always the text service_mgr.

You can use this option to connect to a locally hosted database as well. If you do, taking a backup using this option can run quicker than accessing the database directly. See the section below on speeding up backups.

-T[RANSPORTABLE]

The default dump file format is transportable.
Transportable backup files are written in a format known as external data representation (XDR) format and it is this format which allows a dump taken on a server of one type to be successfully restored on a server of another type.

-USER <username>

See above.

-V[ERIFY]

See above.

-Y <full name of log file> or the text "suppress"

See above.

-BU[FFERS] <number of buffers>

This switch sets the default database cache size (in number of database pages) for the database being restored.
If a database is being overwritten then this setting will overwrite the previous setting for the cache size.

-C[REATE_DATABASE]

This switch causes a new database to be created from the backup file.
The database file must not exist or the restore will fail.
Either this switch or -rep[lace_database] or -r[ecreate_database] must be specified.

-FIX_FSS_D[ATA]

This switch forces gbak to fix malformed UNICODE_FSS character data during a restore.

This switch, and the following one, should not be required under normal circumstances.
However, if a restore operation fails with a «malformed string» error, the message output from gbak will refer the user to one or both of these switches to fix the malformed UNICODE_FSS data or meta data as appropriate.

-FIX_FSS_M[ETADATA]

This switch forces gbak to fix malformed UNICODE_FSS metadata during a restore.

This switch, and the preceding one, should not be required under normal circumstances.
However, if a restore operation fails with a «malformed string» error, the message output from gbak will refer the user to one or both of these switches to fix the malformed UNICODE_FSS data or meta data as appropriate.

-I[NACTIVE]

This switch can be used to restore a database when a previous restore attempt failed due to index errors.
All indexes in the restored database will be inactive.

-K[ILL]

This switch restores the database but doesn’t recreate any shadow files that existed previously.

-M[ETA_DATA]

See above.

-MO[DE] <access>

This switch allows the database being restored to be set to the given access mode when opened.
By default, the mode is taken from the database that was dumped.

-N[O_VALIDITY]

This switch is similar to the -i[nactive] switch above, except, it disabled all check constraints in the restored database.

-NOD[BTRIGGERS]

New switch from Firebird 2.1 which prevents the database triggers from firing on a restore.
Database triggers are a new feature in Firebird 2.0 onwards and are different from table triggers which will continue to fire.

-O[NE_AT_A_TIME]

This switch causes the restore to restore one table at a time.
This can be useful when a previous restore failed due to data errors.
Normally, a restore takes place in a single transaction with a single commit at the end of the restore.
If the restore is interrupted for any reason, an empty database is the end result.
Using the -o[ne_at_a_time] option uses a transaction for each table and commits after each table has been restored.

-P[AGE_SIZE] <new page size>

Use this switch to change the default database page size.
By default, the database is restored using a page size the same as the one that was in use when the database was dumped.

-PAS[SWORD] <password>

See above.

-R[ECREATE_DATABASE] [O[VERWRITE]]

New from Firebird 2.x.
Create (or replace if o[verwrite] is used) the named database from the backup file.
The database filename should not already exist or an error will occur.
This is not the case if the o[verwrite] option is also used.

This is a new switch and is deliberately abbreviated to -r to try to prevent unsuspecting DBAs from overwriting an existing database thinking that the -r was abbreviated from -restore.
Now, it takes special effort to manage this especially as -restore was never actually a valid switch; -r was in fact an abbreviation of -replace_database and it did this by first deleting the existing database file and then recreating it from the backup.

Using -r[ecreate_database] o[verwrite] is effectively the same as using -rep[lace_database].

-REP[LACE_DATABASE]

Replace database from backup file.
This switch used to be abbreviated to -r in previous (to Firebird 2.x) versions.
This switch will be removed in a version of Firebird later than 2.1.3 (where it still exists).
You are advised to use the -r[ecreate_database] o[verwrite] switch instead.

-SE[RVICE] <servicename>

Use the services manager on a remote database to restore a remote database.
The servicename is currently always the text service_mgr.

You can use this option to connect to a locally hosted database as well. If you do, restoring a backup using this option can run quicker than accessing the database directly. See the section below on speeding up restores.

-USE_[ALL_SPACE]

This switch forces the restore to use 100% of each database page and thus not leave any room for changes.
If you omit this switch, some space will be kept free for subsequent changes.
Using this switch is likely to be only of practical use where the database is created and used in read-only mode and no updates to existing data are required.

Once a database has been restored with this option specified, all database pages will be filled to 100% and no free space will be left for updates. Use of this switch set a flag in the database header page to signal that all pages are to be filled to 100% — this applies to any new pages created after the restore. You can override this setting, using gfix -use full | reserve database_name where full uses 100% of each page and reserve reserves some space for subsequent updates. See the gfix manual for more details.

-USER <username>

See above.

-V[ERIFY]

See above.

-Y <name of log file>

See above.

tux> gbak -backup -garbage_collect employee /backups/employee.fbk

tux> gbak -backup -service tux:service_mgr employee /backups/employee.fbk

The source text of stored procedures (and several other types of object, such as CHECK constraints) is stored in a blob, as is the «compiled» BLR code.
When you restore a database, the BLR is not recreated: the same BLR is used until next time you recreate or alter the object.

Historically, the engine did not do the right thing regarding the transliteration of strings embedded in the source and the BLR.
In v.2.1 and 2.5 a lot of work was done to address the international language issues, as you probably know.
A side effect of this was that everything that was read from data and meta data became subject to «well-formedness» checks.
Hence, on restoring, those previously stored source and BLR objects are throwing «malformed string» errors when gbak tries to read and write the data in these system table records.
This very old bug affects user blobs, too, if they have been stored using character set NONE and the client is configured to read a specified character set to which the stored data could not be transliterated.

In v.2.1 there were scripts in ../misc that you could run to repair the meta data blobs and also use as a template for repairing the similar errors in blobs in your user data.
The repair switches were added to the gbak restore code in v.2.5 to do the same corrections to meta data and data, respectively, during the process of restoring a database for upgrade.

tux> gbak -replace -service tux:service_mgr /backups/employee.fbk employee

tux> gfix -shut -tran 60 employee

tux> # Backup the database.
tux> gbak -backup employee /backups/employee.fbk

tux> # Restore the database.
tux> gfix -shut -tran 60 employee
tux> gbak -replace /backups/employee.fbk employee

tux> #Backup only the database metadata.
tux> gfix -shut -tran 60 employee
tux> gbak -backup -meta_data employee employee.meta.fbk

tux> #Restore only the database metadata.
tux> gbak -create employee.fbk mytest.fdb -meta_data

tux> # Backup the database to multiple files.
tux> gbak -backup employee /backups/emp.a.fbk 600m /backups/emp.b.fbk 600m

tux> # Restore the database from multiple files.
tux> gfix -shut -tran 60 employee
tux> gbak -replace /backups/employee.a.fbk /backups/employee.b.fbk employee

tux> setenv_firebird 2.0
Firebird environment set for version 2.0.

tux> # Check current ODS version (as root user!)
tux> gstat -h employee|grep ODS
        ODS version             11.0

tux> # Backup the (old) database.
tux> gbak -backup employee /backups/employee.2_0.fbk

tux> setenv_firebird 2.1
Firebird environment set for version 2.1.

tux> # Recreate the database and upgrade the ODS.
tux> gfix -shut -tran 60 employee
tux> gbak -replace /backups/employee.2_0.fbk employee

tux> # Check new ODS version (as root user!)
tux> gstat -h employee|grep ODS
        ODS version             11.1

tux> # Check current cache size (as root user!)
tux> gstat -h employee | grep -i buffer
        Page buffers            0

tux> # Restore the database & change the cache size.
tux> gfix -shut -tran 60 employee
tux> gbak -replace -buffer 200 /backups/employee.fbk employee

tux> # Check the new cache size (as root user!)
tux> gstat -h employee | grep -i buffer
        Page buffers            200

tux> # Check current page size (as root user!)
tux> gstat -h employee | grep -i "page size"
        Page size               4096

tux> # Restore the database & change the page size.
tux> gfix -shut -tran 60 employee
tux> gbak -replace -page_size 8192 /backups/employee.fbk employee

tux> # Check the new page size (as root user!)
tux> gstat -h employee | grep -i "page size"
        Page size               8192

tux> # Backup the production database.
tux> gbak -backup employee /backups/employee.fbk

tuxrep> # Scp the dump file from tux.
tuxrep> scp [email protected]:/backups/employee.fbk ./
Using keyboard-interactive authentication.
Password:
employee.fbk              |         19 kB |  19.3 kB/s | ETA: 00:00:00 | 100%

tuxrep> # Restore the employee database as read-only.
tuxrep> gfix -shut -tran 60 employee
tuxrep> gbak -replace -mode read_only employee.fbk employee

tuxrep> # Check database mode (as root user)
tuxrep> gstat -h employee|grep -i attributes
        Attributes              no reserve, read only

tux> # Clone a test database to the same server, without requiring a dump file.
tux> gbak -backup emptest stdout | gbak -replace stdin emptest_2

tux> # Check current shadows, use isql as gstat is broken.
tux> isql employee

Database:  employee
SQL> show database;
Database: employee
        Owner: SYSDBA
 Shadow 1: "/opt/firebird/shadows/employee.shd1" manual
 Shadow 2: "/opt/firebird/shadows/employee.shd2" manual
...

SQL> quit;

tux> # Restore the database preserving shadow files.
tux> gfix -shut -tran 60 employee
tux> gbak -replace overwrite /backups/employee.fbk employee

tux> # Check shadows again, use isql as gstat is broken.
tux> isql employee

Database:  employee
SQL> show database;
Database: employee
        Owner: SYSDBA
 Shadow 1: "/opt/firebird/shadows/employee.shd1" manual
 Shadow 2: "/opt/firebird/shadows/employee.shd2" manual
...

SQL> quit;


tux> # Restore the database killing shadow files.
tux> gfix -shut -tran 60 employee
tux> gbak -replace overwrite -kill /backups/employee.fbk employee

tux> # Check shadows again, use isql as gstat is broken.
tux> isql employee

Database:  employee
SQL> show database;
Database: employee
        Owner: SYSDBA
...

SQL> quit;

tux> # Backup the reporting database on remote server tuxrep.
tux> gbak -backup -service tuxrep:service_mgr employee /backups/remote_backup.fbk

tux> # Restore the read-only reporting database on remote server tuxrep.
tux> gbak -replace -mode read_only -service tuxrep:service_mgr 
            /backups/remote_backup.fbk employee

tux> # Backup the employee database on this server, but pseudo-remotely!
tux> gbak -backup -service tux:service_mgr employee /backups/remote_backup.fbk

tux> # Restore the employee database on this server, but pseudo-remotely!
tux> gbak -replace -service tux:service_mgr /backups/remote_backup.fbk employee

TCP

When using TCP networks, the parameter separator is a colon, as in -service server_name:service_mgr.

Named pipes

When using named pipes, the parameter requires two leading back slashes and the separator is another back slash, as in -service \server_nameservice_mgr.

tux> # Clone a test database to a different server, without requiring a dump file.
tux> gbak -backup employee stdout | 
ssh [email protected] "gbak {...} -replace stdin emptest"

tux> # Clone a test database to a different server, without requiring a dump file.
tux> gbak -backup employee stdout | 
ssh [email protected] "gbak {...} -replace stdin emptest"

[email protected]'s password:

tux> isql {...} tuxrep:emptest

Database:  tuxrep:emptest

SQL> show database;

Database: tuxrep:emptest
        Owner: SYSDBA
PAGE_SIZE 4096
...

tux> ssh [email protected] "gbak -backup {...} emptest stdout" | 
gbak -create stdin data/tuxrep_emptest.fdb

[email protected]'s password:

tux> ls data

employee.fdb  tuxrep_emptest.fdb

tux> isql data/tuxrep_emptest.fdb

Database:  data/tuxrep_emptest.fdb

SQL> quit;

• The database alias must start in column one.

• There must be a single space before the equals sign (=).

• There must be a single space after the equals sign (=).

• Double quotes around the database filename is not permitted — it doesn’t work for the Firebird utilities either.

• Databases are all single file databases.

tux> cat /opt/firebird/aliases.conf
# ---------------------------------------------------------
# WARNING: Backup Standards require that:
#          The database name starts in column 1.
#          There is a single space before the equals sign.
#          There is a single space after the equals sign.
#          The path has no double quotes (they don't work!)
# ----------------------------------------------------------
employee = /opt/firebird/examples/empbuild/employee.fdb

tux> # Backup the production employee database using gzip.
tux> gfix -shut -tran 60 employee
tux> DBFILE=`grep -i "^employee =" /opt/firebird/aliases.conf | cut -d" " -f3`
tux> gzip -9 --stdout $DBFILE > /backups/employee.fdb.gz

tux> # Restore the production employee database from a gzip backup.
tux> gfix -shut -tran 60 employee
tux> DBFILE=`grep -i "^employee =" /opt/firebird/aliases.conf | cut -d" " -f3`
tux> gunzip --stdout /backups/employee.fdb.gz > $DBFILE

tux> # Make sure firebird can see the file.
tux> chown firebird:firebird $DBFILE

tux> # Generate unique dump and logfile name.
tux> FILENAME=employee_`date "+%Y%m%d_%H%M%S"`

tux> # Shut down and Backup the database
tux> gfix -shut -tran 60 employee
tux> gbak -backup employee /backups/${FILENAME}.fbk -y /logs/${FILENAME}.log -v