Using "iconv" to Address Conversion and Migration Issues

Info & Tags

Article #:
974
Created:
06/24/08
Modified:
12/28/09
Tags:
conversion, iconv, migration, missing, unicode, utf-8

Related Articles

  1. MKS to Perforce Conversion
  2. Cross-Platform Perforce Server Migration
  3. Unicode File Type Handling Changes In 2009.2
  4. Internationalization and Localization
  5. CVS to Perforce

Links

  1. Documentation for iconv tool
    Third Party Site
  2. iconv tool
    Third Party Site

PROBLEM

Missing Files During Verify

After migrating the Perforce database and versioned files, running p4 verify produces errors similar to: 

//depot/foo/verrŸckt.c#1 - edit change 46102 (text) 252F3099E90B3F453841BE2800EC7D08 MISSING!

You confirm that the versioned file is present on the Perforce server, but the file name is displayed differently: 

/p4root/depot/foo/verrückt.c,v

Perforce cannot find this versioned file because it is looking for "verrŸckt.c" instead of "verrückt.c ". This is most common after moving a checkpoint from a Windows server to a Unix based server -- the file names with special characters are encoded as LATIN1 under Windows, which might not match the UTF-8 encoded file names.

Cannot Convert Server to Unicode Mode

Regardless of server platform, file names submitted from Windows clients using extended characters can cause a Perforce server error when switching to Unicode mode: 

Perforce Server Error:
Database has N tables with non-UTF8 text and can't be switched to Unicode mode.

SOLUTION

You can use the iconv utility to convert extended characters to UTF-8 Unicode encoding. The iconv tool is available for UNIX and Windows from the following ftp site:

http://ftp.gnu.org/pub/gnu/libiconv/

It is also typically installed by default on most UNIX based operating systems.

Windows Installation
Using "iconv"
  • Make a new checkpoint of the server, and copy it to a temporary location.

  • Convert this new checkpoint using iconv. The following is an example conversion command for converting Windows encoding to UTF-8: 
    iconv -f LATIN1 -t utf-8 checkpoint.1234 > checkpoint.utf8
  • Move your Perforce database (db.* files) from your P4ROOT directory and put them in a safe location.

  • Recover from the converted checkpoint. This recreates the Perforce database with Unicode characters in place of the high ASCII characters.

At this point, you can verify that the "MISSING" files are present, or you can switch the server to Unicode mode without error.

Note: The iconv tool is not supported by Perforce.