For God’s sake, don’t try sorting a linked list.

– Steve Yegge

Why would it be a bad idea to sort a linked list? A few things come to mind:

1. To locate the kth item you have to traverse the k-1 items that precede it. This makes the formulation of many sorting algorithms unusable for linked lists: the basic “swap" operation no longer takes constant time.
2. Linked lists are inefficient in general: going over the items in a list means jumping around memory in random. This is bad for the CPU caches, and it might be faster to copy the linked list into an array, and then sort the array.

But let’s not be discouraged: in defiance of Steve, let’s sort a linked list, and do it efficiently, with minimal cache misses!

The Algorithm

What we can do is use a modified merge sort algorithm. The basic idea behind merge sort is:

1. If the input list is empty or has exactly one element, return because it’s already sorted.
2. Otherwise, break the list into two roughly equal sized parts.
3. Sort the two parts recursively using merge sort.
4. Merge the two sorted lists into one sorted list.

Since we don’t know how long the list is, breaking the list into two equal parts is not trivial. It can be done in a single pass over the items, but that means traversing many links, each of which is a potential cache miss. Instead we’ll use the following variation of the algorithm: sort first the first 2 items of the list with the merge operation, then the next 2, merge these so that the first 4 items are sorted, repeat for the next 4 elements and merge so that the first 8 items are sorted, and so on until the end. In pseudocode:

1. k :=1, p := head[list].next
2. while p ≠ nil:
3.     Sort k items of the list that starts at p
4.     Merge two lists of size k that start at head[list] and at p
5.     k := 2k, p := next item of the last node that was merged

The Implementation

The above pseudocode looks simple, but can be tricky to implement. For example, you have to remember that when you sort a list its head moves to the middle of the list, so the sort operation has to return the new head of the list somehow. This may seem obvious, but if you start coding without thinking things through it may come as a surprise!

First we’ll need some representation for linked list nodes. A class that holds the item and a pointer to the next node does nicely:

    static class Node<T> {
        T value;
        Node<T> next;
    }

The merge operation as we use it has to return both the first node of the merged list, and the node past the end of the merged list. Let’s call this construct a bracket:

    private static class Bracket<T> {
        Node<T> top;
        Node<T> bot;
    }

Now let’s define the topmost sort method, following the pseudocode:

    public static <T> void sort(Comparator<T> cmp, 
                                Node<T> head) {
        Bracket<T> bracket = new Bracket<T>();
        int k = 1;
        Node<T> p = head.next;
        while (p != null) {
            bracket.top = p;
            sort(cmp, k, bracket);
            merge(cmp, head, k, bracket.top, k, bracket);
            head = bracket.top; p = bracket.bot;
            k = 2*k;
        }
        return head;
    }

What’s left is the method that sorts the first k items in a linked list with a recursive merge sort, and the merge operation that actually does the work of putting items in order. Both return a bracket from the head of the sorted list until item past the end of the sorted list. No surprises here, pretty standard stuff. I’ll use whitespace creatively to save space though, so you don’t have to scroll so much to skip ahead:

private static <T> void sort(Comparator<T> cmp, int k,
                             Bracket<T> bracket) {
    if (k < 2) { bracket.bot = bracket.top.next; return; }
    sort(cmp, k/2, bracket);
    if (bracket.bot != null) {
        Node<T> top = bracket.top;
        bracket.top = bracket.bot;
        sort(cmp, k/2, bracket);
        Node<T> bot = bracket.top;
        merge(cmp, top, k/2, bot, k/2, bracket);
    }
}

private static <T> void merge(Comparator<T> comparator, 
                              Node<T> top, int ctop, 
                              Node<T> bot, int cbot,
                              Bracket<T> bracket) {
    int count = ctop+cbot;
    Node<T> head=null, tail=null, next;
    while (ctop + cbot > 0) {
        if (cbot == 0)      { next = top; top = top.next; ctop--; }
        else if (ctop == 0) { next = bot; bot = bot.next; cbot--; }
        else {
            int cmp = comparator.compare(top.value, bot.value);
            if (cmp > 0) { next = bot; bot = bot.next; cbot--; }
            else         { next = top; top = top.next; ctop--; }
        }
        if (head == null) head = next;
        else tail.next = next;
        tail = next;
        if (bot == null) cbot=0;
    }
    tail.next = bot;

    bracket.top=head;
    bracket.bot=tail.next;
}

The Analysis

Like any serious comparison-based sorting algorithm, this one makes O(n log n) comparisons between list items when sorting a list of length n. Merge sort can be implemented in constant space as shown by Simon Tatham, but this implementation will end up using O(log n) space due to the use of the stack for recursive calls. How about our goal of minimizing cache misses?

List items are being traversed only in the merge method, so that’s the only place where cache misses can be a problem. (Let’s assume for simplicity that the local variables can fit in registers, or never leave the cache). The algorithm will make:

• n/2 merges of size 2, for a total of n potential cache misses
• n/4 merges of size 4, for a total of n potential cache misses
• …
• 1 merge of size 2^{log₂ n}, for a total of n potential cache misses

As there are log n levels of merges, in total there are n log n potential cache misses. Every merge sort on linked lists has to do this same number of merges (with the exception of natural sort perhaps), in addition to any other list traversal they may do, so the algorithm presented here is optimal for CPU cache usage.

Whether this algorithm is measurably better than the many alternatives (see for example Tatham, GeeksForGeeks) remains to be seen.

It is pretty simple to calculate the number of ways to choose k items out of n: There are n ways to choose the first item, n-1 ways to choose the second item, and so on, until there are (n-k+1) ways to choose the kth item. By the rule of product there are n (n-1)···(n-k+1) ways to choose the k items in total.

But hold on, this process considers picking first A and then B distinct from picking first B and then A. If you don’t care about in which order the items are picked you have to remove the duplicates by dividing by the number of ways there are to order k items: this is k! = k·(k-1)···3·2·1. So, the number of ways you can choose k items out of n is:

\[
\binom{n}{k} = \frac{n(n-1) \cdots (n-k+1)}{k(k-1)\cdots 3 \cdot 2 \cdot 1} = \frac{n!}{k!(n-k)!}
\]

Of course, implementing this formula on a computer as it is is not a very good idea. If you use integers, the multiplications rapidly overflow fixed sized integers, and then the division doesn’t return the correct value because integer division is not properly defined modulo a power of two. If you use floating point types you’ll have a different set of problems. So let’s devise an algorithm that calculates \(\binom{n}{k}\) correctly whenever it fits into a computer word, or in general, modulo a power of two.

We’ll replace the division with multiplication by the inverse modulo 2^m. Here it is important to keep in mind that even numbers don’t have a multiplicative inverse, so we remove any powers of 2 from the numerator and denominator and keep track of the exponent of 2 separately. Done this, we can safely accumulate the denominator and numerator through multiplication and calculate only one modular inverse, in the end.

Without further talk, here’s a C program that implements the algorithm for 32 bit ints.

#include <stdio.h>
#include <stdlib.h>

unsigned int inv(unsigned int n);   /* multiplicative inverse mod 2^32 */
int ord2(int n);                    /* exponent of 2 in n */

unsigned int binom(int n, int k)
{
    unsigned int num=1, den=1, aux, i;
    int ord=0;
    if (n-k < k) k = n-k;
    for (i = 1; i <= k; i++) {
        aux = ord2(i);
        ord -= aux;
        den = den*(i>>aux);

        aux = ord2(n+1-i);
        ord += aux;
        num = num*((n+1-i) >> aux);
    }
    return num*inv(den) << ord;
}

int main(int argc, char *argv[])
{
    int a=3, b=1;
    if (argc > 2) {
        a = atoi(argv[1]);
        b = atoi(argv[2]);
        printf("C(%d,%d) = %u (mod 2^32)\n", a, b, binom(a,b));
    } else  {
        printf("%s a b - calculate C(a,b) mod 2^32", argv[0]);
    }
    return 0;
}

unsigned int inv(unsigned int n) {
    int x = -1;
    x = x*(2-x*n); /* Newton's method */
    x = x*(2-x*n); /* 5 steps enough for 32 bits */
    x = x*(2-x*n);
    x = x*(2-x*n);
    x = x*(2-x*n);
    return x;
}

int ord2(int n) {
    int i, e;
    i = n & -n; /* isolate rightmost 1-bit */
    for (e=0; i > 1; e++) i >>= 1;
    return e;
}

Lately we’ve discussed how integers work in computers when their size is limited to a fixed number of bits. What if there is no such limitation and we can use as many bits as we want?

As computers get more and more powerful we’d like to derive this case by looking at what happens when we add more and more bits to the binary representation. For example, if you use 2 bits, -1 = 11₂ because 1+11₂ = 100₂, and 100₂ ≡ 0 (mod 2²). When you move to 3 bits and more you start to see a pattern emerge:

m=3:  -1 =    111 because    111 + 1 =    1000 ≡ 0 (mod 2³) 
m=4:  -1 =   1111 because   1111 + 1 =   10000 ≡ 0 (mod 2⁴) 
m=5:  -1 =  11111 because  11111 + 1 =  100000 ≡ 0 (mod 2⁵)
m=6:  -1 = 111111 because 111111 + 1 = 1000000 ≡ 0 (mod 2⁶)

So, if we have an infinite number of bits, would -1 be …111111₂? Can a number that extends infinitely to the left, an infinitely large number, have any meaning?

It turns out that, given the right definitions, it can.

Introduction

Recall that m-bit integers are integers modulo 2^m. This means that two m-bit integers are equal if their difference is a multiple of 2^m; for example -1 is 2^m-1. When we use m+1 bits this changes: the numbers are equal if their difference is 2^m+1. To make the jump from m bits to m+1 bits we define a new concept of distance that makes two numbers “close” if their difference is a high power of two: the greater the power the closer they are. In fact, if the difference is 2^k, we make the distance 2^-k. This distance makes m-bit numbers approximations to m+1 bit numbers: 1111₂ is a good approximation to -1 because their distance is 2^-5 = 1/32. Adding one bit, 11111₂ is even better: its distance to -1 is 2^-6 = 1/64, twice as close.

It turns out that this way of defining a distance between two numbers, though it sounds strange at first, fulfills all the usual requirements we ask of distance functions (metrics in mathematics). Having a metric means it becomes meaningful to talk about an extension of integers where numbers like …111111₂ arise as limits to sequences such as (1₂,11₂,111₂,1111₂,…)^(*). We call this extended number set the 2-adic integers^(*).

Arithmetic

Arithmetic with these “infinite” numbers works pretty much like you would expect. You add and multiply finite integers just like before. To add infinite integers you use the exact same methods, but since there’s an infinite number of digits you can never finish, but you can calculate the result up to any number of digits you like. If this seems objectionable, consider a more familiar case: if you calculate 1/3 in decimal numbers you get 0.333333…: you never finish, but you can calculate as many digits as you like.

For reference, calculating the sum of two 2-adic is simple:
\[
\sum_{k=0}^\infty a_k 2^k + \sum_{k=0}^\infty b_k 2^k = \sum_{k=0}^\infty (a_k+b_k) 2^k
\]

Calculating the product is slightly more complicated since you have to collect the powers of two:
\[
\left(\sum_{i=0}^\infty a_i 2^i \right)\left(\sum_{i=0}^\infty b_i 2^i\right)
= \sum_{k=0}^\infty \left(\sum_{i+j=k} a_i b_j \right) 2^k
\]

Example

To demonstrate, Let’s calculate the sum of …111101₂ and 3 = 11₂, and the product of …0101011₂ and = 3:

  ...111111111111111111111101           ...10101010101010101011
+ ...000000000000000000000011         × ...00000000000000000011
= ...000000000000000000000000         = ...10101010101010101011 
= 0                                   + ...01010101010101010110
                                      = ...00000000000000000001 = 1

It turns out that the sum is 0, so …111101₂ = -3. Also, the product is 1, so in a sense …0101011₂ = 1/3.

Representation of negative integers

In the example we found a way to represent -3 using binary digits. Let’s see if we can generalize this.

Suppose x is a positive integer and we want to find a representation for -x. One way to approach this problem is using the identity …111111₂ = -1 from our introduction: -x = -1-(x-1) = (…111111₂-x)+1. The subtraction is easy to calculate because all the bits in …111111₂ are 1: there is never need to borrow.

In fact, you can see …111111-x as the bitwise complement operation “~x” which returns x with all of its bits flipped: where x has a 1, …111111-x has 1-1 = 0, and where x has a 0, …111111-x has 1-0 = 1. Therefore the 2′s complement rule extends to integers in general: again we have -x = ~x+1.

For example, 1234 = 10011010010₂, so -1234 is …111101100101110₂:

-1234 =  ...111111111111111                    To verify:
        -...000010011010010 + 1                  ...111101100101110
      =  ...111101100101101 + 1                +        10011010010
      =  ...111101100101110                    =                  0

Practical applications

Suppose you are to create a program that requires arbitrarily large integers, and for whatever reason don’t want to use an existing “bigint” implementation, such as BigInteger in Java or .Net. Like the people that design computer hardware, you have to decide how to represent signed integers. I feel that the 2-adic representation is optimal for this purpose because it extends the well known 2′s complement method in a natural way, and because approaching a problem from a larger perspective often permits you to use more powerful tools, even if the tools make no sense in the problem domain.

This is a common pattern in mathematics: to solve a problem you may have to express it in a more general setting (for example, generalized Fourier transforms), and then you come a full circle when applying the results in the original setting.

One example of this is the calculation of multiplicative inverses. Remember how in m-bit integers every odd number has a multiplicative inverse: for each integer x there is an integer x^-1 that fulfills x^-1x = 1. Working with m bits, that is, in integers modulo 2^m, the inverse can be calculated using the extended Euclidean algorithm, but if we consider the larger setting of 2-adic integers, a result called Hensel’s Lemma implies the inverse can be calculated using Newton’s method much faster than the Euclidean algorithm.

Footnotes

1. This is the same process we use to define the real numbers using sequences of rational numbers, except that in this case the numbers we construct extend infinitely to the left rather than to the right.
2. You can do this derivation in any base p, giving the p-adic integers. For everything to work correctly p has to be a prime number.

The problem is that often we need operations that can’t be well defined on general rings. For example, we want to compare two numbers and see which is “greater.” In particular we’re interested in knowing if a given integer is greater or less than zero. These operations have to be easy to implement in hardware if they are to be efficient, so the simpler the better.

Recall that computers represent small negative integers are represented by really large integers. Using m bits for integers, numbers over 2^m-1 have their highest bit set to one. This gives us the idea of identifying the sign with this bit. Now the range of signed integers becomes (-N/2,N/2-1): half are negative, half nonnegative. To check if a number is negative you just have to look at its most significant bit. If you chose to use a different range, like (-100, N-101), you would need to investigate much more than a single bit to decide if a number is less than zero.

As you can read on Wikipedia there are many ways to represent signed integers on computers, differentiated by how one obtains the representation of -x from x. Let’s see how one goes about changing the sign of in our system.

Let’s denote the number obtained by flipping all bits of x by ~x: where x has a zero, ~x has a one, and where x has a one, ~x has a zero. This means that x + ~x will consist of all ones. But the number whose binary representation consists of all ones is 2^m-1, so x + ~x = 2m-1. From here we obtain -x = ~x-2m+1. Since we’re working mod 2^m, we have -x = ~x+1. So our derivation of signed integers coincides with the common 2′s complement representation.

By the logic presented here 2′s complement is the only sane way of representing signed integers. All other methods break in some way: 1′s complement for example has two distinct representations for 0, and requires separate circuits for addition and subtraction. Why would anyone bother with any other way of representing numbers?

The Wikipedia article seems to suggest the reason is usability: The early programmers had to work with machine code and binary memory dumps all day, and seeing negative numbers as sign-magnitude, or at least as 1′s complement, would have made their jobs a little bit easier.

In the end the 2′s complement system won because it was simpler and cheaper to create in hardware. On the other hand usability became a non-issue thanks to programming tools getting better.

The main problem with having a finite number of bits is that we have a finite number of integers, which means that if you keep adding 1s the sequence you get 1,2,3,4,… at some point starts to repeat. Still we’d like to define addition and multiplication so that all the familiar rules still hold: we expect that for every a, b, c we have a+(b+c)=(a+b)+c, a*(b*c)=(a*b)*c, a*(b+c)=a*b+a*c, and so on. Mathematical objects that do this are called rings.

A useful way to define a finite ring of N elements is to take the integers modulo N. Formally this works by defining an equivalence relation for integers, where a and b are equivalent when their difference is a multiple of N:

a ~ b if and only if a – b ≡ 0 (mod N).

For example, for N = 12, 7 ~ 19 because 7-19 = -12 ≡ 0 (mod 12).

It turns out that this equivalence partitions ℤ into N subsets: those that are multiples of N, those that are a multiple of N plus 1, and so on. These subsets, also called congruence classes, will form the elements of the ring we’re defining. Now we have to define + and * on them in a meaningful way.

Let’s denote the congruence class of a by [a]. We have:

[a] = {b : a ~ b} = {a + kN : k ∈ ℤ}. Note that [a] = [a + kN] for every integer k.

For example, for N = 12, [-5] = [7] = [19] = {…, -5, 7, 19, 31, …}.

It turns out we can define addition and multiplication on these classes based on the arithmetic of integers, and everything checks out:

[a] + [b] = [a + b], and [a]*[b] = [ab]. Since [a]=[a+N], these operations “wrap around” after N elements.

For example, for N = 12, [7]+[6] = [13] = [1]. [5]*[8] = [40] = [4].

To represent classes we can pick a number from each class: [a] is represented by an arbitrarily chosen a. To encode classes on a computer we pick a from the set {0, 1, …, N-1}, stored in which ever way is appropriate for the machine. This gives us the unsigned integers. For example with N = 2³² we have [2948326455] + [1346640853] = [4294967308] = [12], so a computer working with 32-bit integers calculates 2948326455 + 1346640853 = 12. I’ll be using N = 2³² in all of the following examples.

But what about signed integers? Since -[a] = [-a] = [N-a], negative integers close to 0 will be stored as large positive numbers. For example we have -[1] = [4294967295], so a computer would store -1 as 4294967295.

Because the arithmetic is defined on congruence classes rather than integers immediately we see that the range of integers we represent on a computer can be chosen arbitrarily. Commonly we choose to work with numbers [0, N-1] or [-N/2, N/2-1], which gives us the unsigned and signed integers, respectively, but we might as well choose to work with any other range that suits our needs, like [-100, N-101], as long as it contains a representative of each of the N classes. As far as the ring operations of addition, subtraction and multiplication are concerned it really doesn’t matter. (Operations not generally defined on rings, like division or order, are a different matter.)

The benefits of this theoretical approach is that we can immediately apply well known results from mathematics. For example, every integer in {1, 2, …, N-1} has a multiplicative inverse modulo N provided it has no common factors with N. For our computers with N = 2^m this means division by one odd integer can be performed as multiplication by another. For example, to divide by 3 you can multiply by -1431655765: -1431655765*15 = 5.

Notice that we haven’t made any assumptions of how computers actually store numbers as bits, or of how signed integers could be encoded. We only assumed that the number of bits available is finite and that addition and multiplication should be defined in a sane way. In upcoming posts I’ll discuss how this approach links to the common 2′s complement representation of signed integers, and how the theory behind arbitrary sized integers (bignums) could be explained.

Most of the time you can’t tell the difference between the two: As far as basic arithmetics is concerned both are zero, and the equality operator has been defined so that +0.0 = -0.0. Pretty much the only place where you see a difference between the two is when you divide by zero: 1/-0.0 results in negative infinity.

The MySQL floating point types also have positive and negative zeros, and it treats them as equals for selecting data:

mysql> create table t ( f float );
mysql> insert into t values (0.1), (-0.1);
mysql> update t set f = round(f);
mysql> select f from t where f = 0;
+------+
| f    |
+------+
|    0 |
|   -0 |
+------+

However, GROUP BY considers them distinct!

mysql> select count(*), from t group by f;
+----------+------+
| count(*) | f    |
+----------+------+
|        1 |    0 |
|        1 |   -0 |
+----------+------+

This is something to keep in mind if you are using MySQL floating point types for scientific computing.

One trick is to take a large number as the denominator. For the numerator you multiply the floating point number with that large number and round the result. But this is suboptimal because we would get for example 66667/10000; never the expected 20/3. Can we do any better?

Yes, yes we can.

The best way to do this is with continued fractions, that is, fractions of the form:

\[a_0+\cfrac{1}{a_1+\cfrac{1}{a_2+\cfrac{1}{\ddots}}}\]

It turns out that continued fractions allow the construction of a sequence of rational numbers that approximate a given real number x really well. The first approximation is \(a_0\), the next is \(a_0+1/a_1\), the next \(a_0+\frac{1}{a_1+1/a_2}\), and so on. The process to calculate the sequence of a’s is really simple: \(a_0\) is just the integral part of x, floor(x). Then we calculate the inverse of the fractional part. The integral part of the result is \(a_1\), and we can use the fractional part to repeat the process. A recurrence relation for the numerator and denominator allows us to update them in each step rather than having to calculate them using the above formula.

Why it works

Intuitively you can think of the process of calculating the a’s as lopping off as much of the error as possible in each step while limiting ourselves to inverses of integers: the error is the fractional part left over in each step. Indeed, for rational numbers eventually we would get fractional part of 0, meaning we have found the continued fraction representation of the number.

Formally, there’s a theorem that states that each approximation is nearer to the true value than any other fraction whose denominator is less than that of the approximation. This means that the approximations produced by this process are as good as they get: other ratios are either longer or less precise.

Demo

Convert a decimal number to a fraction:

Conclusions

Continued fractions have many interesting properties: Euler used them to prove that e is irrational. The continued fractions that eventually repeat are exactly the quadratic irrationals. The numerators and denominators produced by the process are coprime. The continued fraction representation of the golden ratio \(\varphi\) is all 1′s, which means it’s the number that is worst approximated by continued fractions. In a sense this makes \(\varphi\) the most irrational number of all.

You can find out more about continued fractions on the Wikipedia page.

This little JavaScript function encodes a string as an array of integers using UTF-8 encoding while taking surrogate pairs into account:

create table t (
    i integer auto_increment,
    other_field varchar(100),
    primary key(id)
);

There some features and “features” that you have to keep in mind when using AUTO_INCREMENT, though.

AUTO_INCREMENT never gives a value that is less than one already present in the table.

If you add a row with ID explicitly set to a higher value, the next value produced by AUTO_INCREMENT will be one higher. Yes, this means that there will be gaps. Yes, this means that if for some reason the greatest possible value of the field is inserted, the following inserts will fail:

mysql> insert into t values (2147483647);
Query OK, 1 row affected (0.05 sec)
mysql> insert into t values ();
ERROR 1062 (23000): Duplicate entry '2147483647' for key 'PRIMARY'

How to reset the AUTO_INCREMENT counter.

The values are produced by a counter associated with the table, and the counter only grows. Except that sometimes you may want to reset the counter to some old value.

alter table t auto_increment = 0;

If you find yourself doing this after emptying a testing table with DELETE, try using the TRUNCATE statement next time. It deletes all the data, and resets the auto increment counter.

If records are deleted, AUTO_INCREMENT may jump back on server restart.

In the InnoDB storage engine the auto increment counter is only stored in memory, not on disk. This means that the auto increment counter is initialized to the maximum value the column has when MySQL is restarted. This may be a problem for applications that use one table to archive data that has been deleted from another: suddenly IDs are repeated.

A table can have only one AUTO_INCREMENT field, and it has to be key.

The auto increment column does not have to be the primary key; it’s enough that there’s an index of some kind on it. It does not even have to be a unique index.

In the MyISAM and BDB storage engines you can have the auto increment column as a secondary column in a multi-column index, and then the auto increment starts from 1 for each group:

create temporary table t (
    v varchar(100), 
    i integer(2) auto_increment, 
    primary key(v, i)
) engine=myisam;
insert into t (v) values ('a'),('a'),('b'),('b'),('b');
select * from t;
+---+---+
| v | i |
+---+---+
| a | 1 |
| a | 2 |
| b | 1 |
| b | 2 |
| b | 3 |
+---+---+

Example

This Java program lists files from the current directory and tells you if they exist. It demonstrates that when Java encounters a file with a problematic name it does report it in listFiles, but any further operations on the file fail.

import java.io.File;
import java.io.IOException;

class Ls {
    public static void main(String[] args) throws IOException {
        File d = new File(".");
        for (File f : d.listFiles()) {
            System.out.printf("%s: %b\n", f.getName(), f.exists());
        }
    }
}

For example, when it encounters a file with a name encoded in latin1, this is what happens:

$ ls -b
ni\361o
$ java Ls
ni�: false

You can download Ls.java with an example file here.

Setting the default character encoding

You probably know that Java uses a “default character encoding” to convert binary data to Strings. To read or write text using another encoding you can use an InputStreamReader or OutputStreamWriter. But for data-to-text conversions deep in the API you have no choice but to change the default encoding.

Java reads the default character encoding from the system language settings. On Unix this means LANG and LC_CTYPE environment variables; changing one of these is sufficient. For example, to make Java use latin1 you could start the JVM with the following command:

$ LANG=en_US.iso88591 java Ls
ni�o: true

Or, if you want all programs you start from the terminal to use this locale:

export LANG=en_US.iso88591

The locale en_US.iso88591 has to be installed on the system for these to work, though. You can use the following command to list locales that are available on your system.

locale -a

Defining and installing a new locale

If you don’t have a locale with the appropriate encoding installed you can define and install a new one with the localedef program. For example, to create locale with the Windows Western character encoding you could use the following command.

sudo localedef -f CP1252 -i en_US en_US.cp1252

Under this locale Java would correctly process files with all kinds of names, including those whose name contains curly quotes or the euro character €.

What about file.encoding?

The file.encoding system property can also be used to set the default character encoding that Java uses for I/O. Unfortunately it seems to have no effect on how file names are decoded into Strings.